Pretty Print JSON with python

Pretty Print JSON
echo '{"foo": "lorem", "bar": "ipsum"}' | python -mjson.tool

Dump Django data as pretty JSON
./manage.py dumpdata mymodule | python -mjson.tool



Prety Print JSON from Python Shell
import json
json_data = [1,2,3,{'a','b',c'}]
json.dumps(json_data, sort_keys=True, indent=4)


http://stackoverflow.com/questions/352098/how-to-pretty-print-json-script
http://docs.python.org/library/json.html

Django: Random Image/File Field name

By default, Django image/file fields place uploaded files in the MEDIA_ROOT directory, under a subdirectory given by the "upload_to" model field parameter.  However, the name of uploaded file is the same name as the original filename uploaded by the client.  The filename can be changed by creating a custom storage manager. 

---- myapp/storage.py
import os, itertools, random, string
from django.core.files.storage import FileSystemStorage

def rand_key(size):
    return "".join([random.choice(string.letters + string.digits)
                          for i in range(size)])

class RandomFileSystemStorage(FileSystemStorage):

    def get_valid_name(self, name):
        file_root, file_ext = os.path.splitext(name)
        return "%s%s" % (rand_key(32), file_ext)

    def get_available_name(self, name):
        dir_name, file_name = os.path.split(name)
        file_root, file_ext = os.path.splitext(file_name)
        count = itertools.count(1)
        while self.exists(name):
            name = os.path.join(dir_name, "%s_%s%s" % (
                             rand_key(32),  count.next(), file_ext))
        return name

---- myapp/models.py

from django.db import models

from myapp.storage import RandomFileSystemStorage

class Photo(models.Model):

    image = models.ImageField(upload_to='media',
                  storage=RandomFileSystemStorage())

Setup Django RabbitMQ and Celery

Rabbit Message Queue is a separate server that remotely executes tasks given to it. 
Celery is a Python client program that sends tasks to the RabbitMQ.  
Django-celery is a Django wrapper for Celery that makes it 
work with Django more nicely.  Tasks are executed in Django 
view code or wherever.  The tasks can be defined in the Django app, 
are sent to a Celery client daemon executed by ./manage.py and
Celery serializes the task and sends it to RabbitMQ for processing.
RabbitMQ notifies Celery when it is done with each task. 


1) Install deps
UBUNTU 10.04 NOTE
sudo aptitude install python-virtualenv  #(recommended but not required)
sudo aptitude install rabbitmq-server

2) Create rabbitmq user and vhost.  These settings must be set again if you change your server hostname.  Or you can set the NODENAME to rabbit@localhost in the rabbitmq configuration. 
sudo rabbitmqctl add_user my_rabbit_user mypasswd
sudo rabbitmqctl add_vhost my_vhost
sudo rabbitmqctl set_permissions -p my_vhost my_rabbit_user ".*" ".*" ".*"

3) Setup python environment
virtualenv --no-site-packages testrabbitmq  #(create a virtualenv, not required)
cd testrabbitmq
. ./bin/activate # (activate virtualenv, not required)
pip install django
pip install celery
pip install django-celery

4) Setup Django project
django-admin.py startproject testc  # create a test project
cd testc
python ./manage.py startapp cel  # create a test app

5) Create a test model
Edit cel/models.py:
from django.db import models
class MyModel(models.Model):
    field1 = models.CharField(max_length=12)
    field2 = models.CharField(max_length=12)

6) Create some test tasks:
Edit cel/tasks.py
from celery.decorators import task
@task
def add(x, y)
    return x + y

from cel import models
@task
def addmodel(x, y):
    record = models.MyModel.objects.create(field1=x, field2=y)
    record.save()
    return record

from cel import models

@task(ignore_result=True)  # Celery will ignore results sent back to it

def addmodel2(x, y):

    record = models.MyModel.objects.create(field1=x, field2=y)

    record.save()

    return record

7) configure django settings
Edit settings.py


# set django-celery autoloader 
import djcelery
djcelery.setup_loader()

...

# Set database settings

...

# set information to connect to rabbitmq (broker) 
BROKER_HOST = "127.0.0.1"
BROKER_PORT = 5672
BROKER_VHOST = "/my_vhost"
BROKER_USER = "my_rabbit_user"
BROKER_PASSWORD = "1234"

# add to installed apps
INSTALLED_APPS = (
    ...
    'djcelery',  # django-celery
    'cel',
)

8) Syncdb
python ./manage syncdb

9) Restart the rabbitmq server (optional)
UBUNTU 10.04 NOTE - Ubuntu starts the rabbitmq 
server by default and installs an init script 
(/etc/init.d/rabbitmq-server start|stop).  For testing, 
let's stop the server and restart manually run it in the 
foreground to see more output. 
sudo /etc/init.d/rabbitmq-server stop
sudo rabbitmq-server start
# will display lots of output and say broker is running.  
# The terminal will wait.


10) Start the celery client process
In another terminal (if using virtualenv, be sure to activate it), 
in the testrabbitmq Django project, execute the following:

python ./manage.py celeryd -l info

# this will hang the terminal and set the Celery (the message client) waiting. 

11) Send a message
In another terminal (if using virtualenv, be sure to activate it), 
in the testrabbitmq Django project, execute the following:
python ./manage.py shell
>>> from cel import tasks
>>> result = tasks.add.delay(1, 2)
>>> result.ready()  # waits until task is done
True
>>> result.state
u'SUCCESS'
>>> result.successful()
True
>>> result = tasks.add.delay(1 + 2) # will cause an error
>>> result.successful()
False
>>> result = tasks.addmodel.delay('a','b')
>>> result.successful()
True
>>> for i in range(0,1000):  # stresstest
....            result = tasks.addmodel.delay('a','b')

# alternate syntax (more flexible - can pass args)
>>> result = tasks.addmodel.apply_asyc(args=['c','d'])
>>> result.successful()

# execute at a given time

>>> from datetime import datetime, timedelta

>>> result = tasks.addmodel.apply_asyc(args=['c','d'],
                             eta=datetime.now() + timedelta(minutes=1))

>>> result.successful()

# execute after a given number of seconds

>>> result = tasks.addmodel.apply_asyc(args=['c','d'],

                             countdown=3)

>>> result.successful()

# alternate syntax  (can queue functions you don't own)
>>> from celery.execute import send_task
>>> tasks.addmodel.name
'cel.tasks.addmodel'
>>> result = send_task("cel.tasks.addmodel", args=['e', 'f'])
>>> result.get()

>>> from djcelery.models import TaskMeta
>>> TaskMeta.objects.count()  # this table contains all results meta info 
                                                  # for tasks not defined with ignore_result=True

APPENDIX - rabbitmq commands

sudo rabbitmqctl stop  # stop the rabbitmq
sudo rabbitmqctl list_users # list the available users
sudo rabbitmqctl list_vhosts # list the available vhosts
sudo rabbitmqctl list_queues
Listing queues ...
celery 544   # by default, lists the queues in the server 
                   # and how many messages in them
sudo rabbitmqctl list_queues [options]
    common list_queues options 
         name = name of queue
         durable = queue survives server restarts
         pid = erlang process id  
         messages_ready = ready to be delivered to clients
         messages = queue depth
         memory = bytes of mem consumed by erlang process

# NOTES, for any changes to the code, be sure to restart the celery client (./manage.py celeryd)


UPDATE: I did a talk about this at Chicago Djangonaughts.  See here: 
https://groups.google.com/forum/?fromgroups#!topic/django-chicago/iJr9nStrM-U


http://www.turnkeylinux.org/blog/django-celery-rabbitmq
http://ask.github.com/celery/userguide/executing.html

Tell Browser not to cache forms on back

This works in IE and Firefox but not Chrome or Opera


<meta http-equiv="cache-control" content="no-cache"> <!-- tells browser not to cache -->
<meta http-equiv="cache-control" content="no-store">
<meta http-equiv="expires" content="0"> <!-- says that the cache expires 'now' -->
<meta http-equiv="pragma" content="no-cache"> <!-- says not to use cached stuff, if there is any -->

Django List Installed Tables and Models

Get Database connection
# Django 1.2
from django.db import connections

connection = connections['default']

# Django 1.1.1<=

from django.db import connection

# Get Database tables

tables = connection.introspection.table_names()

# Get Database models

modelclasses = connection.introspection.installed_models(tables)

# Get Django model name and app name
for model in list(modelclasses):
    print model._meta.app_label, model._meta.module_name

# A better way to get Django models
from django.db.models.loading import get_models, get_app
get_app('app_label')  # returns an app class given a label
get_models()  # all models 
get_models(application)  # models belonging to app class

# Get a model from a string
from django.db import models
models.get_model(*"myapp.mymodel".split('.')) # returns model class

# Get all Python modules

import sys
sys.modules

http://www.slideshare.net/ubernostrum/django-in-depth?src=related_normal&rel=3017822

Use alternate Mirror for Python pypi installs (when pypi goes down)

Specify alternate mirror from the command line:
pip install -i http://d.pypi.python.org/simple $PACKAGE

Specify alternate mirror in ~/.pip/pip.conf:
[global]
index-url = http://d.pypi.python.org/simple

More mirrors:
http://b.pypi.python.org/
http://c.pypi.python.org/
http://d.pypi.python.org/
http://e.pypi.python.org/
http://f.pypi.python.org/

Lists of mirrors:
http://pypi.python.org/mirrors/
http://www.pypi-mirrors.org/


Source:
http://jacobian.org/writing/when-pypi-goes-down/

Query Django Admin Log

Show Django Admin Log entries


select u.username, u.first_name, u.last_name, l.action_time, c.name, c.app_label,
       l.object_id as "Object Modified", l.object_repr, l.action_flag, l.change_message
from (auth_user u inner join django_admin_log l on u.id = l.user_id )
      inner join django_content_type c on c.id = l.content_type_id
where l.object_id = '9'  /* filter by object id if desired */
       and l.action_time > to_date('2010-10-04 01:00','YYYY-MM-DD HH:MI')  /* filter by date range */
order by l.action_time;

Change Default Linux Shell

Needs sudo privileges: to change the default shell:

chsh

http://forums.devshed.com/unix-help-35/how-to-change-default-shell-52749.html

Host a Test email server for Django header debugging

Set the following in settings.py
EMAIL_HOST="localhost"
EMAIL_PORT="1025"

Run the following to host the test webserver

python -m smtpd -n -c DebuggingServer localhost:1025

http://docs.djangoproject.com/en/dev/topics/email/ 

Django Management Commands

Django syncdb without prompts
python manage.py syncdb --noinput

http://docs.djangoproject.com/en/dev/ref/django-admin/

Bash customize autocomplete

http://www.linuxjournal.com/content/more-using-bash-complete-command

Dynamic Fieldsets in Django Admin

Problem: the fieldsets option for the admin interface (classes defined in admin.py) requires that you explicitly list every single field in the model in the fieldset list that you pass to it.  This gets tedious if you add a field to a model, as you also have to add it to this admin.py.  Not very DRY.

Solution:
In models.py
from django.db import models
class My(models.Model):
    a = models.CharField(max_length=10)
    b = models.CharField(max_length=10)
    c = models.CharField(max_length=10)
    d = models.CharField(max_length=10)
    e = models.CharField(max_length=10)
    f = models.CharField(max_length=10)
    g = models.CharField(max_length=10)
    h = models.CharField(max_length=10)

In admin.py
from django.contrib import admin
from django.forms.models import fields_for_model
from f import models
class MyAdmin(admin.ModelAdmin):
    def __init__(self, *args, **kwargs):
        super(MyAdmin, self).__init__(*args, **kwargs)
        all_fields = set(fields_for_model(models.My))
        fieldset1_fields = ('e', 'f',)
        fieldset2_fields = ('g', 'h',)
        fieldset_fields = set(fieldset1_fields) | set(fieldset2_fields)
        rest_fields = list(all_fields - fieldset_fields)
        self.fieldsets = (
            (None, {
                'fields': rest_fields
            }),
            ('Fieldset 1', {
                'classes': ('collapse',),
                'fields': fieldset1_fields
            }),
            ('Fieldset 2', {
                'classes': ('collapse',),
                'fields': fieldset2_fields
            }),
        )
    models = models.My
admin.site.register(models.My, MyAdmin)


Thanks for the help lorochka85

Drill through a Login with Python

import urllib2
import urllib
import BeautifulSoup

# build opener with HTTPCookieProcessor
o = urllib2.build_opener( urllib2.HTTPCookieProcessor() )
urllib2.install_opener( o )

# assuming the site expects 'user' and 'pass' as query params
p = urllib.urlencode( { 'username': 'myusername', 'password': 'mypassword' } )

# perform login with params
f = o.open( 'https://www.mysite.com/login', p )
data = f.read()
f.close()

# second request should automatically pass back any
# cookies received during login... thanks to the HTTPCookieProcessor
f = o.open( 'http://www.mysite.com/home/' )
data = f.read()
f.close()

soup = BeautifulSoup.BeautifulSoup(data)

http://www.nomadjourney.com/2009/03/automatic-site-login-using-python-urllib2/

Python String format

Python Format dates into strings
from datetime import date
d1 = date(2010,01,23)
d2 = date(2010,01,23)
"%s - %s" % (d1.strftime("%m / %d / %Y"), d1.strftime("%m / %d / %Y"))

Python Unicode Convert

Convert weird unicode string characters to string:
import unicodedata
unicodedata.normalize('NFKD',unicode_string).encode('ascii','ignore')

Django module for helping convert things to ascii strings:
from django.utils.encoding import smart_str
smart_str('weird string goes here')

Another hack for removing unicode errors:
unicode('my weird string', errors='ignore')


http://docs.python.org/howto/unicode.html#the-unicode-type

matplotlib basics

Simple Plot
import matplotlib.pyplot as plt
plt.plot([1,2,3,4])
# plot a set of points (1,1) and (3,2)
plt.plot([1,3],[1,2],'ro')
# Or plot as a bunch of connect line segments
plt.plot([2,4,6],[2,4,6])
# set a label for the x and y axis
plt.ylabel('some numbers')
plt.xlabel('joe rocks')
# set the axis [xmin, xmax, ymin, ymax ]
plt.axis([0,6,0,5])
# 'show' the graph
plt.show()


Plot a function
As far as I can tell matplotlib cannot plot a continuous function.  
Instead, create a range of distinct numbers:

import numpy as np
import matplotlib.pyplot as plt
# create a range of numbers from 0 to 5. increment by .1
x = np.arange(0.,5.,.1)
# apply a function to that 
y = np.sin(x)
# plot with extra display params
plt.plot(x,y,linewidth=2.0, label='joeplot', color='blue')
plt.title(r'$\alpha_i > \beta_i$', fontsize=20)
plt.axis([0,5,-1,1])
plt.grid(True)
plt.show()

Make a cool heatmap
import numpy as np
import matplotlib.pyplot as plt
#x = y = np.linspace(-5, 5, 12)
# create a grid of 12 X coordinates and 12 Y coordinates 
# these coordinates will be used to represent specific locations on 
# the grid.  the meshgrid() basically creates a nice uniform coordinate grid
X,Y = np.meshgrid([1,2,3,4],[1,2,3])
# X,Y = np.meshgrid(x, y)
# these ravel functions basically make the 2d arrays created above into lists
x = X.ravel()
y = Y.ravel()
plt.subplot(111)
plt.hexbin(x,y,C=[1,2,3,3,2,2,3,3,2,3,3,4], gridsize=30)
cb = plt.colorbar()
cb.set_label('Heat Value')
plt.axis([x.min(), x.max(), y.min(), y.max()])
plt.grid(True)


http://stackoverflow.com/questions/2369492/generate-a-heatmap-in-matplotlib-using-a-scatter-data-set

Case Study Notes: install pyqrcode in a virtualenv on OSX

Kind of a buggy install.

mkdir qrcode; cd qrcode
virtualenv --distribute --no-site-packages ve
source ./ve/bin/activate
echo "pil" > requirements.pip
echo "http://svn.apache.org/repos/asf/lucene/pylucene/trunk/jcc" >> requirements.pip

wget http://downloads.sourceforge.net/pyqrcode/pyqrcode-0.2.1.tar.gz?use_mirror=

cd pyqrcode-0.2.1

# EDIT pyqrcode-0.2.1/Makefile
# change the call to jcc from
GENERATE=python -m jcc --jar $(LIBFILE) \
# TO
GENERATE=python -m jcc.__main__ --jar $(LIBFILE) \

make
make egg

# it will create qrcode-0.2.1-py2.6-macosx-10.6-universal.egg
# Edit qrcode-0.2.1-py2.6-macosx-10.6-universal.egg/qrcode/__init__.py
# change the line
_qrcode._setExceptionTypes(JavaError, InvalidArgsError)
# TO
_qrcode._set_exception_types(JavaError, InvalidArgsError)

pip -E ./ve/ install qrcode-0.2.1-py2.6-macosx-10.6-universal.egg

# http://pyqrcode.sourceforge.net/
# http://www.mail-archive.com/pythonmac-sig@python.org/msg09864.html
# http://mail-archives.apache.org/mod_mbox/lucene-pylucene-dev/200904.mbox/%3C49EEECAC.7070606@cheimes.de%3E

Installing MySQL-python (MySQLdb) in a virtualenv on OSX 10.6

There were two problems with the MySQL-Python package that I had when installing on OS X 10.6
1) First, an error saying "EnvironmentError: mysql_config not found" when setup.py is run (through pip or easy_install)
2) Second, an error saying "ImportError: dynamic module does not define init function(init_mysql) when importing MySQLdb" when 'import MySQLdb" is issued

Causes:
1) This is caused by the build script not being able to find a MySQL program called mysql_config. This program is used to determine metadata about the mysql install. For example, the command "mysql_config --cflags" reports the flags used to build mysql.
2) This is caused by a mismatch between the architectures that MySQL was built with and the architecture that MySQL-python installer is trying to build and install MySQL-python as. To find what arch MySQL was built as, run "mysql_config --cflags" (this was i386 in my case). OSX tries to build MySQL-python as x86_64. Therefore, there is a arch mismatch.

Solution:
1) Locate the mysql_config binary location and add the path to the MySQL-python/site.cfg file as a config directive: "mysql_config = /opt/local/bin/mysql_config5"
2) Use the arch x86_64 (64 bit) version of MySQL compiled through MacPorts, not a i386 (32 bit) version downloaded off the internet and installed as a dmg.

Summary:
Assuming the MacPorts version of MySQL is installed, both issues can be solved by adding the mysql_config = /opt/local/bin/mysql_config5 config directive to the MySQL-python/site.cfg file.

# SCRIPTED SOLUTION: Create a script called setup.sh and add the following lines
virtualenv --distribute --no-site-packages ve
source ./ve/bin/activate
pip install -E ./ve -r requirements.1.pip # other deps
export ARCHFLAGS="-arch x86_64"
pip install -E ./ve MySQL-python
echo "mysql_config = /opt/local/bin/mysql_config5" >> ./ve/build/MySQL-python/site.cfg
pip install -E ./ve MySQL-python


Resources:
http://groups.google.com/group/python-virtualenv/msg/cf4f3117faea476b?pli=1
http://birdhouse.org/blog/2009/02/21/python-mysql-connections-on-mac-os/
http://stackoverflow.com/questions/2111283/how-to-build-64-bit-python-on-os-x-10-6-only-64-bit-no-universal-nonsense

Python Pip Usage

Basic PIP install
pip install somepackage

Basic PIP uninstall
pip uninstall somepackage

Requirements file
# cat requirements.txt
MyApp
Framework==0.0.1
Library>=0.2

Installing using a requirements file
pip install -r requirements.pip

PIP Freezing requirements
pip freeze # lists all packages and the specific version installed. Useful for migrating

PIP install into virtualenv env
pip install -E ./env
pip install -E ./env -r requirements.pip # using a requirements file

Basic script for creating a virtualenv and installing requirements
virtualenv --distribute --no-site-packages ve
source ./ve/bin/activate
pip install -E ./ve -r requirements.pip
pip install -E ./ve -r requirements-test.pip

http://heisel.org/blog/2009/11/21/django-hudson/
http://pip.openplans.org/#freezing-requirements

Case Study Notes: install djangobb in virtualenv

Ubuntu 10.04 python 2.6 django 1.1

sudo apt-get build-dep python-psycopg2
sudo aptitude install python-dev

cd Sites
virtualenv --no-site-packages env
. ./env/bin/activate
pip install pil
pip install markdown2 Markdown
pip install django-registration
pip install djapian
pil install xapian
pip install psycopg2
easy_install -i http://downloads.egenix.com/python/index/ucs4/ egenix-mx-base
pip install http://code.djangoproject.com/svn/django/tags/releases/1.1.2/
hg clone http://hg.djangobb.org/djangobb/ djangobb
cd env/lib/python2.6/site-packages/
cp /usr/lib/python2.6/dist-packages/_xapian.so .
cp /usr/lib/python2.6/dist-packages/xapian.py .

# configure settings.py to use postgresql_psycopg2
# comment out the openid stuff

## http://www.saltycrane.com/blog/2009/07/using-psycopg2-virtualenv-ubuntu-jaunty/

Postgresql Basic Commands

Login to postgresql:
psql -d mydb -U myuser -W
psql -h myhost -d mydb -U myuser -W
psql -U myuser -h myhost "dbname=mydb sslmode=require" # ssl connection

Default Admin Login:
sudo -u postgres psql -U postgres
sudo -u postgres psql

List databases on postgresql server:
psql -l [-U myuser] [-W]

Turn off line pager pagination in psql:
\pset pager

Determine system tables:
select * from pg_tables where tableowner = 'postgres';

List databases from within a pg shell:
\l

List databases from UNIX command prompt:
psql -U postgres -l

Describe a table:
\d tablename

Quit psql:
\q

Switch postgres database within admin login shell:
\connect databasename

Reset a user password as admin:
alter user usertochange with password 'new_passwd';

Show all tables:
\dt

List all Schemas:
\dn

List all users:
\du

Load data into posgresql:
psql -W -U username -H hostname < file.sql

Dump (Backup) Data into file:
pg_dump -W -U username -h hostname database_name > file.sql

Increment a sequence:
SELECT nextval('my_id_seq');

Create new user:
CREATE USER jjasinski WITH PASSWORD 'myPassword';
# or
sudo -u postgres createuser jjasinski -W

Change user password:
ALTER USER Postgres WITH PASSWORD 'mypass';

Grant user createdb privilege:
ALTER USER myuser WITH createdb;

Create a superuser user:
create user mysuper with password '1234' SUPERUSER
# or even better
create user mysuper with password '1234' SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN REPLICATION;
# or
sudo -u postgres createuser jjasinski -W -s

Upgrade an existing user to superuser: 
alter user mysuper with superuser;
# or even better
alter user mysuper with SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN REPLICATION

Show Database Version:
SELECT version();

Change Database Owner:
alter database database_name owner to new_owner;

Copy a database:
CREATE DATABASE newdb WITH TEMPLATE originaldb;
http://www.commandprompt.com/ppbook/x14316

View Database Connections:
SELECT * FROM pg_stat_activity;

View show data directory (works on 9.1+; not on 7.x):
show data_directory;

Show run-time parameters:

show all;
select * from pg_settings;

Show the block size setting:
# show block_size;
 block_size
------------
 8192
(1 row)

Show stored procedure source:
SELECT prosrc FROM pg_proc WHERE proname = 'procname'

Grant examples:
# readonly to all tables for myuser
grant select on all tables in schema public to myuser;
# all privileges on table1 and table2 to myuser
grant all privileges on table1, table2, table3 to myuser;

Restore Postgres .dump file:
pg_restore --verbose --clean --no-acl --no-owner -h localhost -U myuser -d mydb latest.dump
source

Find all active sessions and kill them (i.e. for when needing to drop or rename db)
Source: http://stackoverflow.com/questions/5408156/how-to-drop-a-postgresql-database-if-there-are-active-connections-to-it

# Postgres 9.2 and above
SELECT pg_terminate_backend(pg_stat_activity.pid) 

FROM pg_stat_activity 

WHERE pg_stat_activity.datname = 'TARGET_DB' 

 AND pid <> pg_backend_pid();

# Postgres 9.1 and below
SELECT pg_terminate_backend(pg_stat_activity.procpid) 

FROM pg_stat_activity 

WHERE pg_stat_activity.datname = 'TARGET_DB' 

AND procpid <> pg_backend_pid();


Re-read postgres config without dropping connections (i.e. if postgres.conf or pg_hba.conf changes)
Source: http://www.heatware.net/databases/postgresql-reload-config-without-restarting/

/usr/bin/pg_ctl reload
or
SELECT pg_reload_conf();

Per user query logging (logs to to postgres logs):
alter role myuser set log_statement = 'all';


Sources:
http://www.devdaily.com/blog/post/postgresql/log-in-postgresql-database
http://forums.devshed.com/postgresql-help-21/how-do-you-turn-off-more-scroll-lock-at-psql-174831.htm
http://www.cyberciti.biz/faq/howto-add-postgresql-user-account/
http://archives.postgresql.org/pgsql-general/1998-08/msg00050.php
http://stackoverflow.com/questions/876522/creating-a-copy-of-a-database-in-postgres
http://stackoverflow.com/questions/1137060/where-does-postgresql-store-the-database

Recycle bin script

crm()
{
# pass this function the name of the directory that
# you want to backup and remove 
if [ -d "$1" ]
then
  cd "${1}/../"
else
  echo "Directory does not exist"
  return 1
fi

PWD=`pwd`
BKPFILES=`find $PWD -type -f`
BKPDIR="$HOME/.Trash"
daten=`date +%Y.%m.%d-%H.%M.%S`

for i in $BKPFILES
do
  echo "$i"
  DIRNAME="$BKPDIR"`echo ${i%/*}`
  FILENAME=`echo ${i##*/}`
  mkdir -p "$DIRNAME"
  cp -pf "$i" "${DIRNAME}/${FILENAME}.${daten}"
  #NEWFILE=`echo "$i" | sed 's|[ \/]|_|g'`
  print "$i"  "$NEWFILE"   " $DIRNAME"
done
rm -rf "$1"
cd - 
}

OSX Color LS

Color LS for Mac OX
alias ls='ls -G'

sftp notes

sftp using alternate port:
sftp -oPort=2222 username@host.com

sftp using alternate private key file:

sftp -o IdentityFile=/custom/file/location.pem username@host.com

http://www.unix.com/shell-programming-scripting/43334-sftp-scripting-help-required.html

Basic Python Development Server setup with Ubuntu 10.04

Quick setup guide

# config vi command line
echo "set -o vi" >> ~/.bashrc
. ~/.bashrc

# install packages
aptitude install vim openssh-server apache2 python-virtualenv python-mysqldb mysql-server libapache2-mod-php5 libapache2-mod-wsgi eclipse subversion nmap ubuntu-restricted-extras g++ git-gui virtualbox-ose php5-cli mdadm fabric python-dev mercurial

# eclipse development plugins
install pydev for eclipse
install subclipse for eclipse

# firefox development plugins
install firebug for firefox @ http://getfirebug.com/
install web developer for firefox @ https://addons.mozilla.org/en-US/firefox/addon/60/

# add google repo
deb http://dl.google.com/linux/deb/ stable non-free
aptitude update

cd /opt/
wget http://www.djangoproject.com/download/1.1.2/tarball/
wget http://www.djangoproject.com/download/1.2.1/tarball/
tar -xvf Django-1.1.2.tar.gz
tar -xvf Django-1.2.1.tar.gz
cd /usr/lib/python2.6/dist-packages
ln -s /opt/Django-1.2.1/django/ django

# mysql workbench download (64 bit version)
aptitude install libzip1 python-pysqlite2 # deps needed
http://dev.mysql.com/get/Downloads/MySQLGUITools/mysql-workbench-oss-5.2.22-1ubu1004-amd64.deb/from/http://mirror.services.wisc.edu/mysql/
dpkg -i mysql-workbench-oss-5.2.22-1ubu1004-amd64.deb

# setup postgresql
aptitude install postgresql pgadmin3 python-psycopg2

sudo su -
passwd postgres
su postgres
psql template1

The last instruction should open the psql shell, where you can run the following:

ALTER USER postgres WITH ENCRYPTED PASSWORD 'mypassword';

# Setup NX server (Google NeatX server)
sudo apt-get install python-software-properties && sudo add-apt-repository ppa:freenx-team
sudo apt-get update
sudo apt-get install neatx-server

## NOTE: if neatx gives a weird error, delete the session dirs on the server:
## /var/lib/neatx/sessions/some_dir/

https://help.ubuntu.com/community/FreeNX
http://programmingzen.com/2007/12/26/installing-django-with-postgresql-on-ubuntu/

Simple Network Test Script

This is a simple script to log network outages on a local network

root@ubuntu:/mnt/root/usr/local/bin# more lifeline
#!/bin/bash

PING='/bin/ping'
EXTHOST1='www.google.com'
EXTHOST2='sun.iwu.edu'
INTHOST='192.168.1.1'
LOG='/var/log/network_outages'
WAITTIME=120

echo "NETWORK STATUS SCRIPT: Started " `date` >> $LOG
while [ 1=1 ]
do
   dater=`date +%Y.%m.%d-%H.%M.%S`
   $PING -q -c1 $EXTHOST1
   ret=$?
   echo "RET: $ret"
   if [ $ret -ne 0 ]
   then
     echo "EXTERNAL_PING(1): outage detected $dater" >> $LOG
   fi

   dater=`date +%Y.%m.%d-%H.%M.%S`
   $PING -q -c1 $EXTHOST2
   ret=$?
   echo "RET: $ret"
   if [ $ret -ne 0 ]
   then
     echo "EXTERNAL_PING(2): outage detected $dater" >> $LOG
   fi

   $PING -q -c1 $INTHOST
   ret=$?
   echo "RET: $ret"
   if [ $ret -ne 0 ]
   then
     echo "INTERNAL_PING: outage detected $dater" >> $LOG
   fi

   sleep $WAITTIME
done

Simple HEX Editor

View file in Hex Editor
od -cx filename

Bash String Operatinos

Remove the last four characters in a string
echo "somefile.txt" | awk 'sub("....$","")

Remove file extension from a string
ls -1 | sed 's/\(.*\)\..*/\1/'

Remove PREFIX from SOMEPATH
SOMEPATH="/home/myuser/usr/bin/"
PREFIX="/home/myuser/"
echo ${SOMEPATH#$PREFIX}
Returns: usr/bin/

Remove shortest match of PREFIX from SOMEPATH
SOMEPATH="/home/sub/subhome/myuser/usr/bin/"
PREFIX="/*/myuser/"
echo ${SOMEPATH#$PREFIX}
Returns: usr/bin/

Parsing parts of a file:
foo=/tmp/my.dir/filename.tar.gz
To get: /tmp/my.dir (like dirname)
path = ${foo%/*}
To get: filename.tar.gz (like basename)
file = ${foo##*/}
To get: filename
base = ${file%%.*}
To get: tar.gz
ext = ${file#*.}

Removing first 8 characters of a string:
echo $var | cut -c9-

http://unstableme.blogspot.com/2008/03/printremove-first-some-characters-of.html
http://tldp.org/LDP/LGNET/18/bash.html
http://docstore.mik.ua/orelly/unix/upt/ch09_07.htm
http://unstableme.blogspot.com/2007/12/removing-last-two-characters-bash.html
http://www.unix.com/shell-programming-scripting/40360-remove-file-extension.html

Oracle Login Context Information

Find the Database instance you are logged into
SELECT sys_context('USERENV', 'DB_NAME') FROM dual;
SELECT sys_context('USERENV', 'INSTANCE_NAME') FROM dual;

Find the operating system user hosting the Oracle session
SELECT sys_context('USERENV', 'OS_USER') FROM dual;

Find the Oracle user currently logged into
SELECT sys_context('USERENV', 'SESSION_USER') FROM dual;

http://www.psoug.org/reference/sys_context.html

Python Virtual Env

Download and install via apt-get, ports, easy_install, etc.

Locate the install directory (usually site-packages) and find the virtualenv.py script.
Alternatively, the package manager may have installed a virtualenv script into your PATH.

I will use ENV to refer to /path/to/python/virtual/env
I will use virtualenv.py to refer to /path/to/script/virtualenv.py

Create a new Virtual Environment
python virtualenv.py ENV

(optional) Activate (set PATH and PYTHONPATH) for new Virtual ENV
source ENV/bin/activate

Location of Virtual Env python binary
ENV/bin/python

Location of Virtual ENV python site-packages dir
ENV/lib/python2.x/site-packages/

Location of Virtual ENV python easy_install script
ENV/bin/easy_install


http://pypi.python.org/pypi/virtualenv