Python Pip Usage

Basic PIP install
pip install somepackage

Basic PIP uninstall
pip uninstall somepackage

Requirements file
# cat requirements.txt
MyApp
Framework==0.0.1
Library>=0.2

Installing using a requirements file
pip install -r requirements.pip

PIP Freezing requirements
pip freeze # lists all packages and the specific version installed. Useful for migrating

PIP install into virtualenv env
pip install -E ./env
pip install -E ./env -r requirements.pip # using a requirements file

Basic script for creating a virtualenv and installing requirements
virtualenv --distribute --no-site-packages ve
source ./ve/bin/activate
pip install -E ./ve -r requirements.pip
pip install -E ./ve -r requirements-test.pip

http://heisel.org/blog/2009/11/21/django-hudson/
http://pip.openplans.org/#freezing-requirements

Case Study Notes: install djangobb in virtualenv

Ubuntu 10.04 python 2.6 django 1.1

sudo apt-get build-dep python-psycopg2
sudo aptitude install python-dev

cd Sites
virtualenv --no-site-packages env
. ./env/bin/activate
pip install pil
pip install markdown2 Markdown
pip install django-registration
pip install djapian
pil install xapian
pip install psycopg2
easy_install -i http://downloads.egenix.com/python/index/ucs4/ egenix-mx-base
pip install http://code.djangoproject.com/svn/django/tags/releases/1.1.2/
hg clone http://hg.djangobb.org/djangobb/ djangobb
cd env/lib/python2.6/site-packages/
cp /usr/lib/python2.6/dist-packages/_xapian.so .
cp /usr/lib/python2.6/dist-packages/xapian.py .

# configure settings.py to use postgresql_psycopg2
# comment out the openid stuff

## http://www.saltycrane.com/blog/2009/07/using-psycopg2-virtualenv-ubuntu-jaunty/

Postgresql Basic Commands

Login to postgresql:
psql -d mydb -U myuser -W
psql -h myhost -d mydb -U myuser -W
psql -U myuser -h myhost "dbname=mydb sslmode=require" # ssl connection

Default Admin Login:
sudo -u postgres psql -U postgres
sudo -u postgres psql

List databases on postgresql server:
psql -l [-U myuser] [-W]

Turn off line pager pagination in psql:
\pset pager

Determine system tables:
select * from pg_tables where tableowner = 'postgres';

List databases from within a pg shell:
\l

List databases from UNIX command prompt:
psql -U postgres -l

Describe a table:
\d tablename

Quit psql:
\q

Switch postgres database within admin login shell:
\connect databasename

Reset a user password as admin:
alter user usertochange with password 'new_passwd';

Show all tables:
\dt

List all Schemas:
\dn

List all users:
\du

Load data into posgresql:
psql -W -U username -H hostname < file.sql

Dump (Backup) Data into file:
pg_dump -W -U username -h hostname database_name > file.sql

Increment a sequence:
SELECT nextval('my_id_seq');

Create new user:
CREATE USER jjasinski WITH PASSWORD 'myPassword';
# or
sudo -u postgres createuser jjasinski -W

Change user password:
ALTER USER Postgres WITH PASSWORD 'mypass';

Grant user createdb privilege:
ALTER USER myuser WITH createdb;

Create a superuser user:
create user mysuper with password '1234' SUPERUSER
# or even better
create user mysuper with password '1234' SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN REPLICATION;
# or
sudo -u postgres createuser jjasinski -W -s

Upgrade an existing user to superuser: 
alter user mysuper with superuser;
# or even better
alter user mysuper with SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN REPLICATION

Show Database Version:
SELECT version();

Change Database Owner:
alter database database_name owner to new_owner;

Copy a database:
CREATE DATABASE newdb WITH TEMPLATE originaldb;
http://www.commandprompt.com/ppbook/x14316

View Database Connections:
SELECT * FROM pg_stat_activity;

View show data directory (works on 9.1+; not on 7.x):
show data_directory;

Show run-time parameters:

show all;
select * from pg_settings;

Show the block size setting:
# show block_size;
 block_size
------------
 8192
(1 row)

Show stored procedure source:
SELECT prosrc FROM pg_proc WHERE proname = 'procname'

Grant examples:
# readonly to all tables for myuser
grant select on all tables in schema public to myuser;
# all privileges on table1 and table2 to myuser
grant all privileges on table1, table2, table3 to myuser;

Restore Postgres .dump file:
pg_restore --verbose --clean --no-acl --no-owner -h localhost -U myuser -d mydb latest.dump
source

Find all active sessions and kill them (i.e. for when needing to drop or rename db)
Source: http://stackoverflow.com/questions/5408156/how-to-drop-a-postgresql-database-if-there-are-active-connections-to-it

# Postgres 9.2 and above
SELECT pg_terminate_backend(pg_stat_activity.pid) 

FROM pg_stat_activity 

WHERE pg_stat_activity.datname = 'TARGET_DB' 

 AND pid <> pg_backend_pid();

# Postgres 9.1 and below
SELECT pg_terminate_backend(pg_stat_activity.procpid) 

FROM pg_stat_activity 

WHERE pg_stat_activity.datname = 'TARGET_DB' 

AND procpid <> pg_backend_pid();


Re-read postgres config without dropping connections (i.e. if postgres.conf or pg_hba.conf changes)
Source: http://www.heatware.net/databases/postgresql-reload-config-without-restarting/

/usr/bin/pg_ctl reload
or
SELECT pg_reload_conf();

Per user query logging (logs to to postgres logs):
alter role myuser set log_statement = 'all';


Sources:
http://www.devdaily.com/blog/post/postgresql/log-in-postgresql-database
http://forums.devshed.com/postgresql-help-21/how-do-you-turn-off-more-scroll-lock-at-psql-174831.htm
http://www.cyberciti.biz/faq/howto-add-postgresql-user-account/
http://archives.postgresql.org/pgsql-general/1998-08/msg00050.php
http://stackoverflow.com/questions/876522/creating-a-copy-of-a-database-in-postgres
http://stackoverflow.com/questions/1137060/where-does-postgresql-store-the-database

Recycle bin script

crm()
{
# pass this function the name of the directory that
# you want to backup and remove 
if [ -d "$1" ]
then
  cd "${1}/../"
else
  echo "Directory does not exist"
  return 1
fi

PWD=`pwd`
BKPFILES=`find $PWD -type -f`
BKPDIR="$HOME/.Trash"
daten=`date +%Y.%m.%d-%H.%M.%S`

for i in $BKPFILES
do
  echo "$i"
  DIRNAME="$BKPDIR"`echo ${i%/*}`
  FILENAME=`echo ${i##*/}`
  mkdir -p "$DIRNAME"
  cp -pf "$i" "${DIRNAME}/${FILENAME}.${daten}"
  #NEWFILE=`echo "$i" | sed 's|[ \/]|_|g'`
  print "$i"  "$NEWFILE"   " $DIRNAME"
done
rm -rf "$1"
cd - 
}

OSX Color LS

Color LS for Mac OX
alias ls='ls -G'

sftp notes

sftp using alternate port:
sftp -oPort=2222 username@host.com

sftp using alternate private key file:

sftp -o IdentityFile=/custom/file/location.pem username@host.com

http://www.unix.com/shell-programming-scripting/43334-sftp-scripting-help-required.html

Basic Python Development Server setup with Ubuntu 10.04

Quick setup guide

# config vi command line
echo "set -o vi" >> ~/.bashrc
. ~/.bashrc

# install packages
aptitude install vim openssh-server apache2 python-virtualenv python-mysqldb mysql-server libapache2-mod-php5 libapache2-mod-wsgi eclipse subversion nmap ubuntu-restricted-extras g++ git-gui virtualbox-ose php5-cli mdadm fabric python-dev mercurial

# eclipse development plugins
install pydev for eclipse
install subclipse for eclipse

# firefox development plugins
install firebug for firefox @ http://getfirebug.com/
install web developer for firefox @ https://addons.mozilla.org/en-US/firefox/addon/60/

# add google repo
deb http://dl.google.com/linux/deb/ stable non-free
aptitude update

cd /opt/
wget http://www.djangoproject.com/download/1.1.2/tarball/
wget http://www.djangoproject.com/download/1.2.1/tarball/
tar -xvf Django-1.1.2.tar.gz
tar -xvf Django-1.2.1.tar.gz
cd /usr/lib/python2.6/dist-packages
ln -s /opt/Django-1.2.1/django/ django

# mysql workbench download (64 bit version)
aptitude install libzip1 python-pysqlite2 # deps needed
http://dev.mysql.com/get/Downloads/MySQLGUITools/mysql-workbench-oss-5.2.22-1ubu1004-amd64.deb/from/http://mirror.services.wisc.edu/mysql/
dpkg -i mysql-workbench-oss-5.2.22-1ubu1004-amd64.deb

# setup postgresql
aptitude install postgresql pgadmin3 python-psycopg2

sudo su -
passwd postgres
su postgres
psql template1

The last instruction should open the psql shell, where you can run the following:

ALTER USER postgres WITH ENCRYPTED PASSWORD 'mypassword';

# Setup NX server (Google NeatX server)
sudo apt-get install python-software-properties && sudo add-apt-repository ppa:freenx-team
sudo apt-get update
sudo apt-get install neatx-server

## NOTE: if neatx gives a weird error, delete the session dirs on the server:
## /var/lib/neatx/sessions/some_dir/

https://help.ubuntu.com/community/FreeNX
http://programmingzen.com/2007/12/26/installing-django-with-postgresql-on-ubuntu/

Simple Network Test Script

This is a simple script to log network outages on a local network

root@ubuntu:/mnt/root/usr/local/bin# more lifeline
#!/bin/bash

PING='/bin/ping'
EXTHOST1='www.google.com'
EXTHOST2='sun.iwu.edu'
INTHOST='192.168.1.1'
LOG='/var/log/network_outages'
WAITTIME=120

echo "NETWORK STATUS SCRIPT: Started " `date` >> $LOG
while [ 1=1 ]
do
   dater=`date +%Y.%m.%d-%H.%M.%S`
   $PING -q -c1 $EXTHOST1
   ret=$?
   echo "RET: $ret"
   if [ $ret -ne 0 ]
   then
     echo "EXTERNAL_PING(1): outage detected $dater" >> $LOG
   fi

   dater=`date +%Y.%m.%d-%H.%M.%S`
   $PING -q -c1 $EXTHOST2
   ret=$?
   echo "RET: $ret"
   if [ $ret -ne 0 ]
   then
     echo "EXTERNAL_PING(2): outage detected $dater" >> $LOG
   fi

   $PING -q -c1 $INTHOST
   ret=$?
   echo "RET: $ret"
   if [ $ret -ne 0 ]
   then
     echo "INTERNAL_PING: outage detected $dater" >> $LOG
   fi

   sleep $WAITTIME
done

Simple HEX Editor

View file in Hex Editor
od -cx filename

Bash String Operatinos

Remove the last four characters in a string
echo "somefile.txt" | awk 'sub("....$","")

Remove file extension from a string
ls -1 | sed 's/\(.*\)\..*/\1/'

Remove PREFIX from SOMEPATH
SOMEPATH="/home/myuser/usr/bin/"
PREFIX="/home/myuser/"
echo ${SOMEPATH#$PREFIX}
Returns: usr/bin/

Remove shortest match of PREFIX from SOMEPATH
SOMEPATH="/home/sub/subhome/myuser/usr/bin/"
PREFIX="/*/myuser/"
echo ${SOMEPATH#$PREFIX}
Returns: usr/bin/

Parsing parts of a file:
foo=/tmp/my.dir/filename.tar.gz
To get: /tmp/my.dir (like dirname)
path = ${foo%/*}
To get: filename.tar.gz (like basename)
file = ${foo##*/}
To get: filename
base = ${file%%.*}
To get: tar.gz
ext = ${file#*.}

Removing first 8 characters of a string:
echo $var | cut -c9-

http://unstableme.blogspot.com/2008/03/printremove-first-some-characters-of.html
http://tldp.org/LDP/LGNET/18/bash.html
http://docstore.mik.ua/orelly/unix/upt/ch09_07.htm
http://unstableme.blogspot.com/2007/12/removing-last-two-characters-bash.html
http://www.unix.com/shell-programming-scripting/40360-remove-file-extension.html

Oracle Login Context Information

Find the Database instance you are logged into
SELECT sys_context('USERENV', 'DB_NAME') FROM dual;
SELECT sys_context('USERENV', 'INSTANCE_NAME') FROM dual;

Find the operating system user hosting the Oracle session
SELECT sys_context('USERENV', 'OS_USER') FROM dual;

Find the Oracle user currently logged into
SELECT sys_context('USERENV', 'SESSION_USER') FROM dual;

http://www.psoug.org/reference/sys_context.html

Python Virtual Env

Download and install via apt-get, ports, easy_install, etc.

Locate the install directory (usually site-packages) and find the virtualenv.py script.
Alternatively, the package manager may have installed a virtualenv script into your PATH.

I will use ENV to refer to /path/to/python/virtual/env
I will use virtualenv.py to refer to /path/to/script/virtualenv.py

Create a new Virtual Environment
python virtualenv.py ENV

(optional) Activate (set PATH and PYTHONPATH) for new Virtual ENV
source ENV/bin/activate

Location of Virtual Env python binary
ENV/bin/python

Location of Virtual ENV python site-packages dir
ENV/lib/python2.x/site-packages/

Location of Virtual ENV python easy_install script
ENV/bin/easy_install


http://pypi.python.org/pypi/virtualenv

curl Usage

Send a POST

curl -u joe:1234 -X POST http://localhost:8000/api/1.0/NU1yWdRM5JBjnZpZX/message/ -d "message=test"

-u = username and password

-d = post parameters

-X = request method

Send a POST using data from standard in:
echo "mykey=myvalue" | curl -X POST -d @-  https://mysite.com/

Show verbose output: 
curl -v http://mysite.com/

Ignore SSL key errors:
curl -k http://mysite.com/

Combining them: 
echo "TEST=1
TEST2=2
" curl -X POST -vk -d @- http://mysite.com/

netbios name lookups

Find NetBIOS name from IP
Windows:
netstat -a 192.168.1.22

Linux:
nbtstat 192.168.1.22 # ntbstat separate package

Find IP from NetBIOS name
Windows:
nbtstat -a host.example.com

Linux:
nmblookup host.example.com

http://www.irongeek.com/i.php?page=security/ipinfo

MacPorts Usage

Update MacPorts itself
sudo port selfupdate
sudo port -d selfupdate # debug

Updates the port tree with new versions definitions
sudo port sync

List available ports
sudo port list

Search ports
port search [keyword]

Lookup package info (desc, maintainer, etc)
port info [package]

Find package dependencies
port deps [package]

Install package
sudo port install [package]
sudo port -v install [package] #verbose

Clean out build files and tarballs
port clean --all [pakcage]

Uninstall a package
sudo port uninstall [package]

Show port contents
port contents [package]

List installed packages
port installed

List outdated ports
port outdated

Upgrade specific packages
port upgrade [package]
port upgrade outdated #updates all outdated packages


http://guide.macports.org/chunked/using.html

DJango Model Inheritance with subclasses

http://www.djangosnippets.org/snippets/1034/
http://www.djangosnippets.org/snippets/1031/
http://adam.gomaa.us/blog/2009/feb/16/subclassing-django-querysets/
http://docs.djangoproject.com/en/dev/ref/contrib/contenttypes/

Python List Operations (map, for comprehensions)

EXAMPLE 1
# Create a list
>>> l = [1,2,3,4,5,6]

# iterate through the list using a for comprehension
>>> [i for i in l]
[1, 2, 3, 4, 5, 6]

# square elements in the list using a for comprehension
>>> [i**2 for i in l]
[1, 4, 9, 16, 25, 36]

# square only even elements in the list using a for comprehension
>>> [i**2 for i in l if i % 2 == 0]
[4, 16, 36]

# iterate through the list using map
>>> map(lambda w: w, l)
[1, 2, 3, 4, 5, 6]

# square each element of the list using map
>>> map(lambda w: w**2, l)
[1, 4, 9, 16, 25, 36]

# or use map to call a separately defined function to# iterate through the list
>>> def squa(x):
...... return x**2

>>> map(squa, l)
[1, 4, 9, 16, 25, 36]

# use map function to call a separately defined function that takes 2 args
>>> l2 = map(lambda w: (w,2),l)
>>> l2
[(1, 2), (2, 2), (3, 2), (4, 2), (5, 2), (6, 2)]

>>> def pow(base, expo):
...... return base**expo

>>> map(lambda (x,y): pow(x,y), l2)
[1, 4, 9, 16, 25, 36]

# filter a list
>>> filter(lambda w: w > 2, l)
[3, 4, 5, 6]

EXAMPLE 2
# define a simple class to play with
>>> class A(object):
...... def __init__(self, x):
........ self.x = x

# create simple list of object instances of the class
>>> l = [A(1), A(2), A(3)]

# use a for (list) comprehension to iterate through the list
>>> [i.x for i in l]
[1, 2, 3]

# use a for comprehension to iterate with a condition
>>> [i.x for i in l if i.x > 1]
[2, 3]

# use map to apply a function to every element in the list
>>> map(lambda w: w.x, l)
[1, 2, 3]

>>> map(lambda w: w.x * w.x, l)
[1, 4, 9]

EXAMPLE 1

# Create a list

>>> l = [1,2,3,4,5,6]

# iterate through the list using a for comprehension

>>> [i for i in l]

[1, 2, 3, 4, 5, 6]

# square elements in the list using a for comprehension

>>> [i**2 for i in l]

[1, 4, 9, 16, 25, 36]

# square only even elements in the list using a for comprehension

>>> [i**2 for i in l if i % 2 == 0]

[4, 16, 36]

# iterate through the list using map

>>> map(lambda w: w, l)

[1, 2, 3, 4, 5, 6]

# square each element of the list using map

>>> map(lambda w: w**2, l)

[1, 4, 9, 16, 25, 36]

# or use map to call a separately defined function to# iterate through the list

>>> def squa(x):

...... return x**2

>>> map(squa, l)

[1, 4, 9, 16, 25, 36]

# use map function to call a separately defined function that takes 2 args

>>> l2 = map(lambda w: (w,2),l)

>>> l2

[(1, 2), (2, 2), (3, 2), (4, 2), (5, 2), (6, 2)]

>>> def pow(base, expo):

...... return base**expo

>>> map(lambda (x,y): pow(x,y), l2)

[1, 4, 9, 16, 25, 36]

# filter a list

>>> filter(lambda w: w > 2, l)

[3, 4, 5, 6]

EXAMPLE 3

# split a list into chunks

>>> map(None, *(iter(range(10)),) * 3)

[(0, 1, 2), (3, 4, 5), (6, 7, 8), (9, None, None)]

http://stackoverflow.com/questions/1335392/iteration-over-list-slices

Django South Migrations

Install Django South

- Place the untared 'south' folder into your Django applicaion

- Modify the project's settings.py and add 'south' to the INSTALLED_APPS list

- For new Django projects:

---1) initialize any Django apps or models to monitor with South.

----a) See Initializing Applications below

---2) run ./manage.py syncdb

---3) run ./manage.py migrate

To view help on startmigration:  # South 7.x+ uses the schemamigration command instead

./manage.py startmigration

Usage: ./manage.py startmigration appname migrationname [--initial] [--auto]

[--model ModelName] [--add-field ModelName.field_name] [--freeze] [--stdout]

Initializing Applications - needed whenever you want to track more Django apps

# this creates a 'migrations' folder in your Django app directory

./manage startmigration app_name comment --initial  # updates for South 7.x+
./manage schemamigration app_name comment --initial # Init's all models in application
./manage startmigration app_name --initial # updates for South 7.x+

./manage schemamigration app_name --initial # comment is optional here

Adding South with no data and no existing models.py objects:
# 1) syncdb as normal
python manage.py syncdb
# 2) Start an application if needed
python manage.py startapp appname
# 3) create a migration file with a blueprint to create tables
python manage.py startmigration appname --initial  # updates for South 7.x+
python manage.py schemamigration appname --initial
# 4) Apply the blueprint and create the tables
python manage.py migrate appname

# 5) Repeat steps 3 and 4 for each additional app if needed

Adding south with no data and existing models.py objects:
# 1) syncdb as normal if needed
python manage.py syncdb
# 2) create a migration file with a blueprint to create tables
python manage.py startmigration appname --initial # updates for South 7.x+
python manage.py schemamigration appname --initial
# 3) Apply the blueprint but it doesn't actually create the tables
python manage.py migrate appname --fake
# 4) Repeat steps 2 and 3 for each additional app if needed

Applying changes after making changes to a model (repeat for each app):
# Creates a migration 'blueprint' file and guesses what should take place
python manage.py startmigration appname migrationname --auto # updates for South 7.x+
python manage.py schemamigration appname migrationname --auto
# OR create the blueprint file but it will be empty
python manage.py startmigration appname migrationname  # updates for South 7.x+
python manage.py schemamigration appname migrationname
# make any modifications needed
# migrate the tables in the app
python manage.py migrate appname
# OR migrate all tables in all apps
python manage.py migrate

Adding a new model to an app already managed with south:
./manage startmigration app_name comment --model m1 --model m2 # updates for South 7.x+
./manage schemamigration app_name comment --model m1 --model m2 # inits specific models

List available migrations
python manage.py migrate --list

http://south.aeracode.org/wiki/ConvertingAnApp

Oracle Space queries

Get Used space for current user
SELECT sum(bytes)/(1024 * 1024) as "Used (MB)"
FROM user_SEGMENTS
ORDER BY 1 desc

Get tablespace usage for all users using that tablespace
select owner, sum(bytes)/power(2,20)mb
from dba_extents
where tablespace_name = 'TABLESPACE'
group by owner
order by 2 desc;

Get Total tablespace available for current user
select tablespace_name,round(sum(bytes) / (1024 * 1024),2) "Tablespace SIZE (MB)"
from user_free_space
group by tablespace_name

Get space of individual tables/indexes (objects) in a schema
select sum(bytes) / (1024 * 1024) as MB,owner,segment_type, segment_name
from dba_segments s
where owner = 'SCHEMA'
group by owner,segment_type, segment_name
order by MB desc

Get USER_ dictornary Tables
SELECT table_name, comments
FROM dictionary
WHERE table_name LIKE 'USER_%'
ORDER BY table_name;

http://snipplr.com/view/4748/get-a-list-of-all-the-user-tables-oracle/
http://www.freelists.org/post/oracle-l/dba-extents-vs-dba-segments,8

CSS Pre Tag formatting

Make pre tags wrap as expected

pre {
white-space: pre-wrap;       /* css-3 should we be so lucky... */
white-space: -moz-pre-wrap;  /* Mozilla, since 1999 */
white-space: -pre-wrap;      /* Opera 4-6  */
white-space: -o-pre-wrap;    /* Opera 7  */
word-wrap: break-word;       /* Internet Explorer 5.5+ */
_white-space: pre;   /* IE only hack to re-specify in addition to
word-wrap  */
}

http://archivist.incutio.com/viewlist/css-discuss/55677
http://bavotasan.com/tutorials/how-to-wrap-text-within-the-pre-tag-using-css/