Drill through a Login with Python

import urllib2
import urllib
import BeautifulSoup

# build opener with HTTPCookieProcessor
o = urllib2.build_opener( urllib2.HTTPCookieProcessor() )
urllib2.install_opener( o )

# assuming the site expects 'user' and 'pass' as query params
p = urllib.urlencode( { 'username': 'myusername', 'password': 'mypassword' } )

# perform login with params
f = o.open( 'https://www.mysite.com/login', p )
data = f.read()
f.close()

# second request should automatically pass back any
# cookies received during login... thanks to the HTTPCookieProcessor
f = o.open( 'http://www.mysite.com/home/' )
data = f.read()
f.close()

soup = BeautifulSoup.BeautifulSoup(data)

http://www.nomadjourney.com/2009/03/automatic-site-login-using-python-urllib2/

Python String format

Python Format dates into strings
from datetime import date
d1 = date(2010,01,23)
d2 = date(2010,01,23)
"%s - %s" % (d1.strftime("%m / %d / %Y"), d1.strftime("%m / %d / %Y"))

Python Unicode Convert

Convert weird unicode string characters to string:
import unicodedata
unicodedata.normalize('NFKD',unicode_string).encode('ascii','ignore')

Django module for helping convert things to ascii strings:
from django.utils.encoding import smart_str
smart_str('weird string goes here')

Another hack for removing unicode errors:
unicode('my weird string', errors='ignore')


http://docs.python.org/howto/unicode.html#the-unicode-type

matplotlib basics

Simple Plot
import matplotlib.pyplot as plt
plt.plot([1,2,3,4])
# plot a set of points (1,1) and (3,2)
plt.plot([1,3],[1,2],'ro')
# Or plot as a bunch of connect line segments
plt.plot([2,4,6],[2,4,6])
# set a label for the x and y axis
plt.ylabel('some numbers')
plt.xlabel('joe rocks')
# set the axis [xmin, xmax, ymin, ymax ]
plt.axis([0,6,0,5])
# 'show' the graph
plt.show()


Plot a function
As far as I can tell matplotlib cannot plot a continuous function.  
Instead, create a range of distinct numbers:

import numpy as np
import matplotlib.pyplot as plt
# create a range of numbers from 0 to 5. increment by .1
x = np.arange(0.,5.,.1)
# apply a function to that 
y = np.sin(x)
# plot with extra display params
plt.plot(x,y,linewidth=2.0, label='joeplot', color='blue')
plt.title(r'$\alpha_i > \beta_i$', fontsize=20)
plt.axis([0,5,-1,1])
plt.grid(True)
plt.show()

Make a cool heatmap
import numpy as np
import matplotlib.pyplot as plt
#x = y = np.linspace(-5, 5, 12)
# create a grid of 12 X coordinates and 12 Y coordinates 
# these coordinates will be used to represent specific locations on 
# the grid.  the meshgrid() basically creates a nice uniform coordinate grid
X,Y = np.meshgrid([1,2,3,4],[1,2,3])
# X,Y = np.meshgrid(x, y)
# these ravel functions basically make the 2d arrays created above into lists
x = X.ravel()
y = Y.ravel()
plt.subplot(111)
plt.hexbin(x,y,C=[1,2,3,3,2,2,3,3,2,3,3,4], gridsize=30)
cb = plt.colorbar()
cb.set_label('Heat Value')
plt.axis([x.min(), x.max(), y.min(), y.max()])
plt.grid(True)


http://stackoverflow.com/questions/2369492/generate-a-heatmap-in-matplotlib-using-a-scatter-data-set

Case Study Notes: install pyqrcode in a virtualenv on OSX

Kind of a buggy install.

mkdir qrcode; cd qrcode
virtualenv --distribute --no-site-packages ve
source ./ve/bin/activate
echo "pil" > requirements.pip
echo "http://svn.apache.org/repos/asf/lucene/pylucene/trunk/jcc" >> requirements.pip

wget http://downloads.sourceforge.net/pyqrcode/pyqrcode-0.2.1.tar.gz?use_mirror=

cd pyqrcode-0.2.1

# EDIT pyqrcode-0.2.1/Makefile
# change the call to jcc from
GENERATE=python -m jcc --jar $(LIBFILE) \
# TO
GENERATE=python -m jcc.__main__ --jar $(LIBFILE) \

make
make egg

# it will create qrcode-0.2.1-py2.6-macosx-10.6-universal.egg
# Edit qrcode-0.2.1-py2.6-macosx-10.6-universal.egg/qrcode/__init__.py
# change the line
_qrcode._setExceptionTypes(JavaError, InvalidArgsError)
# TO
_qrcode._set_exception_types(JavaError, InvalidArgsError)

pip -E ./ve/ install qrcode-0.2.1-py2.6-macosx-10.6-universal.egg

# http://pyqrcode.sourceforge.net/
# http://www.mail-archive.com/pythonmac-sig@python.org/msg09864.html
# http://mail-archives.apache.org/mod_mbox/lucene-pylucene-dev/200904.mbox/%3C49EEECAC.7070606@cheimes.de%3E

Installing MySQL-python (MySQLdb) in a virtualenv on OSX 10.6

There were two problems with the MySQL-Python package that I had when installing on OS X 10.6
1) First, an error saying "EnvironmentError: mysql_config not found" when setup.py is run (through pip or easy_install)
2) Second, an error saying "ImportError: dynamic module does not define init function(init_mysql) when importing MySQLdb" when 'import MySQLdb" is issued

Causes:
1) This is caused by the build script not being able to find a MySQL program called mysql_config. This program is used to determine metadata about the mysql install. For example, the command "mysql_config --cflags" reports the flags used to build mysql.
2) This is caused by a mismatch between the architectures that MySQL was built with and the architecture that MySQL-python installer is trying to build and install MySQL-python as. To find what arch MySQL was built as, run "mysql_config --cflags" (this was i386 in my case). OSX tries to build MySQL-python as x86_64. Therefore, there is a arch mismatch.

Solution:
1) Locate the mysql_config binary location and add the path to the MySQL-python/site.cfg file as a config directive: "mysql_config = /opt/local/bin/mysql_config5"
2) Use the arch x86_64 (64 bit) version of MySQL compiled through MacPorts, not a i386 (32 bit) version downloaded off the internet and installed as a dmg.

Summary:
Assuming the MacPorts version of MySQL is installed, both issues can be solved by adding the mysql_config = /opt/local/bin/mysql_config5 config directive to the MySQL-python/site.cfg file.

# SCRIPTED SOLUTION: Create a script called setup.sh and add the following lines
virtualenv --distribute --no-site-packages ve
source ./ve/bin/activate
pip install -E ./ve -r requirements.1.pip # other deps
export ARCHFLAGS="-arch x86_64"
pip install -E ./ve MySQL-python
echo "mysql_config = /opt/local/bin/mysql_config5" >> ./ve/build/MySQL-python/site.cfg
pip install -E ./ve MySQL-python


Resources:
http://groups.google.com/group/python-virtualenv/msg/cf4f3117faea476b?pli=1
http://birdhouse.org/blog/2009/02/21/python-mysql-connections-on-mac-os/
http://stackoverflow.com/questions/2111283/how-to-build-64-bit-python-on-os-x-10-6-only-64-bit-no-universal-nonsense

Python Pip Usage

Basic PIP install
pip install somepackage

Basic PIP uninstall
pip uninstall somepackage

Requirements file
# cat requirements.txt
MyApp
Framework==0.0.1
Library>=0.2

Installing using a requirements file
pip install -r requirements.pip

PIP Freezing requirements
pip freeze # lists all packages and the specific version installed. Useful for migrating

PIP install into virtualenv env
pip install -E ./env
pip install -E ./env -r requirements.pip # using a requirements file

Basic script for creating a virtualenv and installing requirements
virtualenv --distribute --no-site-packages ve
source ./ve/bin/activate
pip install -E ./ve -r requirements.pip
pip install -E ./ve -r requirements-test.pip

http://heisel.org/blog/2009/11/21/django-hudson/
http://pip.openplans.org/#freezing-requirements

Case Study Notes: install djangobb in virtualenv

Ubuntu 10.04 python 2.6 django 1.1

sudo apt-get build-dep python-psycopg2
sudo aptitude install python-dev

cd Sites
virtualenv --no-site-packages env
. ./env/bin/activate
pip install pil
pip install markdown2 Markdown
pip install django-registration
pip install djapian
pil install xapian
pip install psycopg2
easy_install -i http://downloads.egenix.com/python/index/ucs4/ egenix-mx-base
pip install http://code.djangoproject.com/svn/django/tags/releases/1.1.2/
hg clone http://hg.djangobb.org/djangobb/ djangobb
cd env/lib/python2.6/site-packages/
cp /usr/lib/python2.6/dist-packages/_xapian.so .
cp /usr/lib/python2.6/dist-packages/xapian.py .

# configure settings.py to use postgresql_psycopg2
# comment out the openid stuff

## http://www.saltycrane.com/blog/2009/07/using-psycopg2-virtualenv-ubuntu-jaunty/

Postgresql Basic Commands

Login to postgresql:
psql -d mydb -U myuser -W
psql -h myhost -d mydb -U myuser -W
psql -U myuser -h myhost "dbname=mydb sslmode=require" # ssl connection

Default Admin Login:
sudo -u postgres psql -U postgres
sudo -u postgres psql

List databases on postgresql server:
psql -l [-U myuser] [-W]

Turn off line pager pagination in psql:
\pset pager

Determine system tables:
select * from pg_tables where tableowner = 'postgres';

List databases from within a pg shell:
\l

List databases from UNIX command prompt:
psql -U postgres -l

Describe a table:
\d tablename

Quit psql:
\q

Switch postgres database within admin login shell:
\connect databasename

Reset a user password as admin:
alter user usertochange with password 'new_passwd';

Show all tables:
\dt

List all Schemas:
\dn

List all users:
\du

Load data into posgresql:
psql -W -U username -H hostname < file.sql

Dump (Backup) Data into file:
pg_dump -W -U username -h hostname database_name > file.sql

Increment a sequence:
SELECT nextval('my_id_seq');

Create new user:
CREATE USER jjasinski WITH PASSWORD 'myPassword';
# or
sudo -u postgres createuser jjasinski -W

Change user password:
ALTER USER Postgres WITH PASSWORD 'mypass';

Grant user createdb privilege:
ALTER USER myuser WITH createdb;

Create a superuser user:
create user mysuper with password '1234' SUPERUSER
# or even better
create user mysuper with password '1234' SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN REPLICATION;
# or
sudo -u postgres createuser jjasinski -W -s

Upgrade an existing user to superuser: 
alter user mysuper with superuser;
# or even better
alter user mysuper with SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN REPLICATION

Show Database Version:
SELECT version();

Change Database Owner:
alter database database_name owner to new_owner;

Copy a database:
CREATE DATABASE newdb WITH TEMPLATE originaldb;
http://www.commandprompt.com/ppbook/x14316

View Database Connections:
SELECT * FROM pg_stat_activity;

View show data directory (works on 9.1+; not on 7.x):
show data_directory;

Show run-time parameters:

show all;
select * from pg_settings;

Show the block size setting:
# show block_size;
 block_size
------------
 8192
(1 row)

Show stored procedure source:
SELECT prosrc FROM pg_proc WHERE proname = 'procname'

Grant examples:
# readonly to all tables for myuser
grant select on all tables in schema public to myuser;
# all privileges on table1 and table2 to myuser
grant all privileges on table1, table2, table3 to myuser;

Restore Postgres .dump file:
pg_restore --verbose --clean --no-acl --no-owner -h localhost -U myuser -d mydb latest.dump
source

Find all active sessions and kill them (i.e. for when needing to drop or rename db)
Source: http://stackoverflow.com/questions/5408156/how-to-drop-a-postgresql-database-if-there-are-active-connections-to-it

# Postgres 9.2 and above
SELECT pg_terminate_backend(pg_stat_activity.pid) 

FROM pg_stat_activity 

WHERE pg_stat_activity.datname = 'TARGET_DB' 

 AND pid <> pg_backend_pid();

# Postgres 9.1 and below
SELECT pg_terminate_backend(pg_stat_activity.procpid) 

FROM pg_stat_activity 

WHERE pg_stat_activity.datname = 'TARGET_DB' 

AND procpid <> pg_backend_pid();


Re-read postgres config without dropping connections (i.e. if postgres.conf or pg_hba.conf changes)
Source: http://www.heatware.net/databases/postgresql-reload-config-without-restarting/

/usr/bin/pg_ctl reload
or
SELECT pg_reload_conf();

Per user query logging (logs to to postgres logs):
alter role myuser set log_statement = 'all';


Sources:
http://www.devdaily.com/blog/post/postgresql/log-in-postgresql-database
http://forums.devshed.com/postgresql-help-21/how-do-you-turn-off-more-scroll-lock-at-psql-174831.htm
http://www.cyberciti.biz/faq/howto-add-postgresql-user-account/
http://archives.postgresql.org/pgsql-general/1998-08/msg00050.php
http://stackoverflow.com/questions/876522/creating-a-copy-of-a-database-in-postgres
http://stackoverflow.com/questions/1137060/where-does-postgresql-store-the-database

Recycle bin script

crm()
{
# pass this function the name of the directory that
# you want to backup and remove 
if [ -d "$1" ]
then
  cd "${1}/../"
else
  echo "Directory does not exist"
  return 1
fi

PWD=`pwd`
BKPFILES=`find $PWD -type -f`
BKPDIR="$HOME/.Trash"
daten=`date +%Y.%m.%d-%H.%M.%S`

for i in $BKPFILES
do
  echo "$i"
  DIRNAME="$BKPDIR"`echo ${i%/*}`
  FILENAME=`echo ${i##*/}`
  mkdir -p "$DIRNAME"
  cp -pf "$i" "${DIRNAME}/${FILENAME}.${daten}"
  #NEWFILE=`echo "$i" | sed 's|[ \/]|_|g'`
  print "$i"  "$NEWFILE"   " $DIRNAME"
done
rm -rf "$1"
cd - 
}

OSX Color LS

Color LS for Mac OX
alias ls='ls -G'

sftp notes

sftp using alternate port:
sftp -oPort=2222 username@host.com

sftp using alternate private key file:

sftp -o IdentityFile=/custom/file/location.pem username@host.com

http://www.unix.com/shell-programming-scripting/43334-sftp-scripting-help-required.html

Basic Python Development Server setup with Ubuntu 10.04

Quick setup guide

# config vi command line
echo "set -o vi" >> ~/.bashrc
. ~/.bashrc

# install packages
aptitude install vim openssh-server apache2 python-virtualenv python-mysqldb mysql-server libapache2-mod-php5 libapache2-mod-wsgi eclipse subversion nmap ubuntu-restricted-extras g++ git-gui virtualbox-ose php5-cli mdadm fabric python-dev mercurial

# eclipse development plugins
install pydev for eclipse
install subclipse for eclipse

# firefox development plugins
install firebug for firefox @ http://getfirebug.com/
install web developer for firefox @ https://addons.mozilla.org/en-US/firefox/addon/60/

# add google repo
deb http://dl.google.com/linux/deb/ stable non-free
aptitude update

cd /opt/
wget http://www.djangoproject.com/download/1.1.2/tarball/
wget http://www.djangoproject.com/download/1.2.1/tarball/
tar -xvf Django-1.1.2.tar.gz
tar -xvf Django-1.2.1.tar.gz
cd /usr/lib/python2.6/dist-packages
ln -s /opt/Django-1.2.1/django/ django

# mysql workbench download (64 bit version)
aptitude install libzip1 python-pysqlite2 # deps needed
http://dev.mysql.com/get/Downloads/MySQLGUITools/mysql-workbench-oss-5.2.22-1ubu1004-amd64.deb/from/http://mirror.services.wisc.edu/mysql/
dpkg -i mysql-workbench-oss-5.2.22-1ubu1004-amd64.deb

# setup postgresql
aptitude install postgresql pgadmin3 python-psycopg2

sudo su -
passwd postgres
su postgres
psql template1

The last instruction should open the psql shell, where you can run the following:

ALTER USER postgres WITH ENCRYPTED PASSWORD 'mypassword';

# Setup NX server (Google NeatX server)
sudo apt-get install python-software-properties && sudo add-apt-repository ppa:freenx-team
sudo apt-get update
sudo apt-get install neatx-server

## NOTE: if neatx gives a weird error, delete the session dirs on the server:
## /var/lib/neatx/sessions/some_dir/

https://help.ubuntu.com/community/FreeNX
http://programmingzen.com/2007/12/26/installing-django-with-postgresql-on-ubuntu/

Simple Network Test Script

This is a simple script to log network outages on a local network

root@ubuntu:/mnt/root/usr/local/bin# more lifeline
#!/bin/bash

PING='/bin/ping'
EXTHOST1='www.google.com'
EXTHOST2='sun.iwu.edu'
INTHOST='192.168.1.1'
LOG='/var/log/network_outages'
WAITTIME=120

echo "NETWORK STATUS SCRIPT: Started " `date` >> $LOG
while [ 1=1 ]
do
   dater=`date +%Y.%m.%d-%H.%M.%S`
   $PING -q -c1 $EXTHOST1
   ret=$?
   echo "RET: $ret"
   if [ $ret -ne 0 ]
   then
     echo "EXTERNAL_PING(1): outage detected $dater" >> $LOG
   fi

   dater=`date +%Y.%m.%d-%H.%M.%S`
   $PING -q -c1 $EXTHOST2
   ret=$?
   echo "RET: $ret"
   if [ $ret -ne 0 ]
   then
     echo "EXTERNAL_PING(2): outage detected $dater" >> $LOG
   fi

   $PING -q -c1 $INTHOST
   ret=$?
   echo "RET: $ret"
   if [ $ret -ne 0 ]
   then
     echo "INTERNAL_PING: outage detected $dater" >> $LOG
   fi

   sleep $WAITTIME
done

Simple HEX Editor

View file in Hex Editor
od -cx filename

Bash String Operatinos

Remove the last four characters in a string
echo "somefile.txt" | awk 'sub("....$","")

Remove file extension from a string
ls -1 | sed 's/\(.*\)\..*/\1/'

Remove PREFIX from SOMEPATH
SOMEPATH="/home/myuser/usr/bin/"
PREFIX="/home/myuser/"
echo ${SOMEPATH#$PREFIX}
Returns: usr/bin/

Remove shortest match of PREFIX from SOMEPATH
SOMEPATH="/home/sub/subhome/myuser/usr/bin/"
PREFIX="/*/myuser/"
echo ${SOMEPATH#$PREFIX}
Returns: usr/bin/

Parsing parts of a file:
foo=/tmp/my.dir/filename.tar.gz
To get: /tmp/my.dir (like dirname)
path = ${foo%/*}
To get: filename.tar.gz (like basename)
file = ${foo##*/}
To get: filename
base = ${file%%.*}
To get: tar.gz
ext = ${file#*.}

Removing first 8 characters of a string:
echo $var | cut -c9-

http://unstableme.blogspot.com/2008/03/printremove-first-some-characters-of.html
http://tldp.org/LDP/LGNET/18/bash.html
http://docstore.mik.ua/orelly/unix/upt/ch09_07.htm
http://unstableme.blogspot.com/2007/12/removing-last-two-characters-bash.html
http://www.unix.com/shell-programming-scripting/40360-remove-file-extension.html

Oracle Login Context Information

Find the Database instance you are logged into
SELECT sys_context('USERENV', 'DB_NAME') FROM dual;
SELECT sys_context('USERENV', 'INSTANCE_NAME') FROM dual;

Find the operating system user hosting the Oracle session
SELECT sys_context('USERENV', 'OS_USER') FROM dual;

Find the Oracle user currently logged into
SELECT sys_context('USERENV', 'SESSION_USER') FROM dual;

http://www.psoug.org/reference/sys_context.html

Python Virtual Env

Download and install via apt-get, ports, easy_install, etc.

Locate the install directory (usually site-packages) and find the virtualenv.py script.
Alternatively, the package manager may have installed a virtualenv script into your PATH.

I will use ENV to refer to /path/to/python/virtual/env
I will use virtualenv.py to refer to /path/to/script/virtualenv.py

Create a new Virtual Environment
python virtualenv.py ENV

(optional) Activate (set PATH and PYTHONPATH) for new Virtual ENV
source ENV/bin/activate

Location of Virtual Env python binary
ENV/bin/python

Location of Virtual ENV python site-packages dir
ENV/lib/python2.x/site-packages/

Location of Virtual ENV python easy_install script
ENV/bin/easy_install


http://pypi.python.org/pypi/virtualenv

curl Usage

Send a POST

curl -u joe:1234 -X POST http://localhost:8000/api/1.0/NU1yWdRM5JBjnZpZX/message/ -d "message=test"

-u = username and password

-d = post parameters

-X = request method

Send a POST using data from standard in:
echo "mykey=myvalue" | curl -X POST -d @-  https://mysite.com/

Show verbose output: 
curl -v http://mysite.com/

Ignore SSL key errors:
curl -k http://mysite.com/

Combining them: 
echo "TEST=1
TEST2=2
" curl -X POST -vk -d @- http://mysite.com/

netbios name lookups

Find NetBIOS name from IP
Windows:
netstat -a 192.168.1.22

Linux:
nbtstat 192.168.1.22 # ntbstat separate package

Find IP from NetBIOS name
Windows:
nbtstat -a host.example.com

Linux:
nmblookup host.example.com

http://www.irongeek.com/i.php?page=security/ipinfo