Saturday, November 19, 2011


Convert all TIFs in a directory to JPEGs
mogrify -format jpg -quality 50 *.tiff

Thursday, November 10, 2011

Python: lxml insert element before and after given element

from lxml import etree
xml = etree.parse('old_xml.xml')
count = 1
for pp in xml.xpath('/response/items/item'):
   prod_id = etree.Element('new_element')  # this element will be inserted
   prod_id.text = "%s" % count
   nav = pp.find('name')  # we will insert before/after this element
   div = nav.getparent()
   pp.insert(pp.index(nav), prod_id) # insert before 'name'
   pp.insert(pp.index(nav) + 1, prod_id) # insert after 'name'
   count += 1

out = open('new_xml.xml','w')
out.write(etree.tostring(xml, pretty_print=True))

Python: Detect Charset and convert to Unicode

Converting data to Unicode via chardet

import chardet
from lxml import html

f = open('my.xml')
content =

encoding = chardet.detect(content)['encoding']
if encoding != 'utf-8':
    content = content.decode(encoding, 'replace').encode('utf-8')

Converting Data to Unicode via UnicodeDammit

from BeautifulSoup import UnicodeDammit

converted = UnicodeDammit(content)
   if not converted.unicode: 
    raise UnicodeDecodeError( 
       "Failed to detect encoding, tried [%s]", 
        ', '.join(converted.triedEncodings))  
        # print converted.originalEncoding
         return converted.unicode

Credit goes to Ian Bicking and others on the lxml team

Tuesday, November 08, 2011

Using lxml xpath to get elements with a default namespace

<a xmlns="">

from lxml import etree
xml = etree.fromstring(xml_string)
xml.xpath('/w3:a/w3:b/w3:r/w3:Value/text()', namespaces={'w3':''})

Thursday, September 22, 2011

SQLAlchemy Notes

Summarized From

This covers the basics of the Non-orm usage of SQLAlchemy

# Define connection engine
from sqlalchemy import create_engine
engine = create_engine('sqlite:///:memory:', echo=True)

# Create some database tables 
from sqlalchemy import Table, Column, Integer, String, MetaData, ForeignKey
metadata = MetaData()
users = Table('users', metadata,
    Column('id', Integer, primary_key=True),
    Column('name', String),
    Column('fullname', String),

addresses = Table('addresses', metadata,
  Column('id', Integer, primary_key=True),
  Column('user_id', None, ForeignKey('')),
  Column('email_address', String, nullable=False)


# Insert into the tables
ins = users.insert()
ins.values(name='jack', fullname='Jack the Pumpkin king')

conn = engine.connect()
result = conn.execute(ins)

# Multiple insert into the tables
conn.execute(addresses.insert(), [
   {'user_id': 1, 'email_address' : ''},
   {'user_id': 1, 'email_address' : ''},
   {'user_id': 2, 'email_address' : ''},
   {'user_id': 2, 'email_address' : ''},

result = engine.execute(users.insert(), name='fred', fullname="Fred Flintstone")

# Select from the tables
from sqlalchemy.sql import select

# Select all fields in user table
s = select([users])
result = conn.execute(s)

# Select name and fullname columns from user table
s = select([, users.c.fullname])

# Join two user and address tables
#  Cartesian product 
list(conn.execute(select([users, addresses])))
# Join on id columns
list(conn.execute(select([users, addresses],

# create conditions for the where clause
from sqlalchemy.sql import and_, or_, not_
print and_('j%'),, )

s = select([(users.c.fullname + "--" + addresses.c.email_address).label('title')])
print conn.execute(s).fetchall()

print users.join(addresses)
print users.join(addresses, + '%'))

s = select([users.c.fullname], from_obj=[
   users.join(addresses, + '%'))
print conn.execute(s).fetchall()

# Delete records based on a condition
conn.execute(users.delete().where( > 'm'))

Thursday, September 01, 2011

Django: Quickly testing an authenticated ajax

Make a authenticated post with the django test client and mark simulate an ajax post:
from django.test import Client
c = Client()
c.login(username='jjasinski', password='xxxx')

Wednesday, August 10, 2011

Self-signed SSL Certificate

Generate self-signed certificate
# Generate Encrypted RSA Private Key with passphrase
openssl genrsa -des3 -out myssl.key 2048

# Generate Unencrypted RSA Private Key 
openssl genrsa -out myssl.key 2048

# Remove Passphrase from key
mv myssl.key
openssl rsa -in -out myssl.key

# Generate Certificate Signing Request with an existing Private Key
openssl req -sha256 -new -key myssl.key -out myssl.csr

# Create SSL certificate
openssl x509 -req -days 365 -in myssl.csr -signkey myssl.key -out myssl.crt

Generating a key and csr in one command
# for providing to your certificate provider
openssl req -sha256 -new -newkey rsa:2048 -nodes -keyout yourdomain.key -out yourdomain.csr

   req = create PKCS#10 X.509 Certificate Signing Request
   -sha256 = adds support for SHA-2

Decode PEM encoded SSL certificate meta information
openssl x509 -in certificate.crt -text -noout
openssl x509 -noout -text -modulus -in

Decode DER encoded SSL certificate meta information
openssl x509 -in certificate.crt -inform der -text -noout

Decode SSL key meta information
openssl rsa -noout -text -modulus -in

Convert PEM encoded SSL certificate to DER encoded:
openssl x509 -in certificate.crt -outform der -out certificate.der

Convert DER encoded SSL certificate to PEM encoded:
openssl x509 -in certfile.crt -inform der -outform pem -out certificate.pem

Note: in the meta info, the Modulus should be the same for the key and cert.


Friday, July 01, 2011

Using Rsync to copy file Structure

Simple rsync copy on local filesystem:
rsync -r -t -v -l source/ dest

rsync over ssh:
rsync -r -t -v -l /path/to/dest

rsync copy shortcut to archive:
rsync -avz /path/to/dest
rsync -avz --progress /path/to/dest

rsync copy archive without caring about owner and group:
rsync -rlptD /path/to/dest
rsync -a --no-o --no-g /path/to/dest

Common Flags:
 -a = archive copy.  Equivalent to flags -rlptgoD
 -r = recursive copy
 -t = save timestamp
 -v = verbose copy
 -vv = more verbose
 -u = don't copy over existing files newer than source
 -l = preserve symlinks
 -p = preserve premissions
 -g = preserve group
 -o = preserve owner
 -D = preserve devices (superuser only)
 -z = compress transfer
 --progress = show copy progress
--delete = delete stuff on the receiving side that doesn't exist on the sending side

Thursday, June 02, 2011

Git INFO Common Commands

Configure Git client options:
# show config options
git config --list # show all
git config --global --list # show only global
git config --local --list # show only local

# --global option impacts options at a global level
# --local option impacts options at a local level (.git/config)

# set username/email associated with client
git config --global ""
git config --global "Joe Jaz"

# add color syntax highlighting
git config --global color.status auto
git config --global color.branch auto

# set default pager and editor
git config --global core.editor vim # or set the GIT_EDITOR environment variable
git config --global core.pager less # or set the GIT_PAGER environment variable

# display an option value
git config group.value
git config

Start a new repository and import:
# In the root folder to add to version control:
git init
git init --bare  # creates a new repository (with no working copy)
git add . # add everything in directory to repo
git commit -m 'initial commit'

Making commits:
# commit specific files
git commit file1 file2 -m 'comment'

# commit all 'added' files
git commit -a -m 'comment'

# change a commit message
git commit --amend -m 'New Message'

Create new repository by cloning existing one:
git clone --bare . path_or_url_to_git_dir.git

Setting the remote repository (to push and pull from):
git remote add short_name path_or_url_to_git_dir.git
git remote add origin git://

Show remote repositories available:
git remote
git remote -v # show the url
git remote show origin  # show detail information about origin

Remove a remote repository: 
git remove repo_name

Rename a remote repository: 
git rename old_repo_name new_repo_name

Push and Pull from remote repository:
git push origin # push recent commits to remote repository called "origin"
git pull origin # pull updates from origin

Revert changes in working copy:
git clean -n
git clean --dry-run

Show file differences:
git diff   # show differences between working copy and any files staged for commit
git diff --staged   # compares staged changes with last commit
git diff --cached  # same as above for git older than 1.6.1

git difftool -t [meld|kdiff3|vimdiff|etc.] [diff criteria]
git difftool -t meld --staged
git difftool --tool=meld --staged
git difftool origin/master  --dir-diff # use difftool to diff entire dir tree

git diff -U999999999 master.. -- file.txt # shows the full file.txt, with diff markup, in the master/branch diff results

git diff --name-status brancha..branchb # shows list of changed files

Show File/s History:
git log filename # Show single file versions

# Show number of lines changed in file/s
git log -stat filename # single file
git log -stat # multiple files

# show log summaries on one line
git log --pretty=oneline # Pre-defined format
git log --pretty=format:"%h - %cD (%an) %s" # custom format (see below)
git log --pretty=format:"%h %s" --graph
git log --graph --full-history --all --color \
    Pre-Defined Pretty formats:
    format:string # where string is

        Format Strings
        %H: commit hash
        %h: abbreviated commit hash
        %T: tree hash
        %t: abbreviated tree hash
        %P: parent hashes
        %p: abbreviated parent hashes
        %an: author name
        %aN: author name (respecting .mailmap, see git-shortlog(1) or git-blame(1))
        %ae: author email
        %aE: author email (respecting .mailmap, see git-shortlog(1) or git-blame(1))
        %ad: author date (format respects --date= option)
        %aD: author date, RFC2822 style
        %ar: author date, relative
        %at: author date, UNIX timestamp
        %ai: author date, ISO 8601 format
        %cn: committer name
        %cN: committer name (respecting .mailmap, see git-shortlog(1) or git-blame(1))
        %ce: committer email
        %cE: committer email (respecting .mailmap, see git-shortlog(1) or git-blame(1))
        %cd: committer date
        %cD: committer date, RFC2822 style
        %cr: committer date, relative
        %ct: committer date, UNIX timestamp
        %ci: committer date, ISO 8601 format
        %d: ref names, like the --decorate option of git-log(1)
        %e: encoding
        %s: subject
        %f: sanitized subject line, suitable for a filename
        %b: body
        %B: raw body (unwrapped subject and body)
        %N: commit notes
        %gD: reflog selector, e.g., refs/stash@{1}
        %gd: shortened reflog selector, e.g., stash@{1}
        %gs: reflog subject
        %Cred: switch color to red
        %Cgreen: switch color to green
        %Cblue: switch color to blue
        %Creset: reset color
        %C(…): color specification, as described in color.branch.* config option
        %m: left, right or boundary mark
        %n: newline
        %%: a raw %
        %x00: print a byte from a hex code
        %w([[,[,]]]): switch line wrapping, like the -w option of git-shortlog(1).

Show git branches available:
git branch

Show all branches including remote branches:
git branch -a

Create new branch:
git branch new_branch

Change branches:
git checkout new_branch

Delete branch:
git branch -d new_branch # delete only if all changes are merged
git branch -D new_branch  # delete even if there are unmerged changes

Merge branch into current copy:
git merge --no-ff branch_to_merge_in

Resolve a Merge:
1) Edit file to resolve
2) git add conflict_file
3) git commit

Using Mergetool
git mergetool
git mergetool --tool=meld
git mergetool -t meld

Abort a merge conflict before committing:
git merge --abort

Undo a Merge:  # restores state to pre-merge
git reset --heard HEAD

Undo a local commit: 
git reset --soft HEAD^

Find the commit where two branches diverge:
git merge-base branch1 branch2

Find what branches were merged into a given branch:
git branch --merged master  # local branches only
git branch -a --merged master   # local and remote branches

Find what branches were NOT merged into a given branch:
git branch --no-merged master  # local branches only
git branch -a --no-merged master   # local and remote branches

Find what branches are merged into develop, but not master:
comm -12 <(sort <(git branch -a --no-merged origin/master)) <(sort <(git branch -a --merged origin/develop))

Copy all branches from origin to new_remote: 
git push new_remote refs/remotes/origin/*:refs/heads/*

Get number of commits per author
apt-get install git-extras

Sunday, May 15, 2011

Python Process Info

Pipe the output of one process into the input of another via python:
import subprocess
p1 = subprocess.Popen(['echo','joe'], shell=False, stdout=subprocess.PIPE)
p2 = subprocess.Popen(['cat','-n'],
    stdin=p1.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# Or 

Pipe standard input to a program
import subprocess
p1 = subprocess.Popen(['node', 'index.js'], shell=False,
    stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p1.communicate('standard input here')

Convenience method for splitting apart shell commands for use in the above:
>>> import shlex
>>> shlex.split('ps -ef')
['ps', '-ef']

Common Open ID Urls

Taken From


Tuesday, May 03, 2011

SVN List Codes

Codes Displayed when doing an "svn stat"
U = Working file was updated
G = Changes on the repository were automatically merged into the working copy
M = Working Copy is modified
C = This file conflicts with the version in the repo
? = This file is not under version control
! = This file is under version control but is missing or incomplete
A = This file will be added to version control after commit
A+ = This file will be moved after commit
D = This file will be deleted after commit
I = This item is being ignored (due to the svn:ignore porperty)
L = This file is locked
X =  Item is unversioned, but used by an externals definition
~ = Messed up item (file when it should be dir). resolve by moving and doing "svn up"

Tuesday, April 26, 2011

Whoosh Python Search Notes

Baisc Whoosh Search Walkthrough 

# Import required 
import os
from whoosh.index import create_in
from whoosh.fields import *

# Create a "Schema" object to store text to search
# You can create custom search fields such as "title", "domain", "path"
# and "content". 
schema = Schema(title=TEXT(stored=True), domain=ID(stored=True),
                              path=ID(stored=True), content=TEXT)

# Check to see if a search index directory exists; if not, create one
if not os.path.exists("search_test"):

# Create a search index in the search index directory
ix = create_in("search_test", schema, indexname="testindex")
writer = ix.writer()

# Index some documents
writer.add_document(title=u"Document 1", domain=u"",
                         path=u"/mywebsite", content=u'Article')
writer.add_document(title=u"Document 2", domain=u"",
                         path=u"/mywebsite/2/article/1", content=u'Document a')

# Running search queries
from whoosh.qparser import QueryParser
import whoosh.index as index
# Check to see if the given search index exists
if index.exists_in("search_test", indexname="testindex"):
    # open the search index
    ix = index.open_dir("search_test", indexname="testindex")
    # generate a search query based on some string.
    # Here is creates a query looking for "Document" in the "content"
    # search index field. 
    query = QueryParser("content", ix.schema).parse(u'*Document*')
    with ix.searcher() as searcher:  
        # do the actual search
        results =

# Running search queries
from whoosh.qparser import QueryParser
import whoosh.index as index
if index.exists_in("search_test", indexname="testindex"):
    ix = index.open_dir("search_test", indexname="testindex"))
    query = QueryParser("domain", ix.schema).parse(u'')
    with ix.searcher() as searcher:
        results =

# Running search queries
from whoosh.qparser import QueryParser, MultifieldParser
import whoosh.index as index
from whoosh.query import *
if index.exists_in("search_test", indexname="testindex"):
    ix = index.open_dir("search_test")
    query = MultifieldParser(["domain","title","content"], schema=ix.schema).parse(u"Document OR Article")
    with ix.searcher() as searcher:
        results =

Different way to alter a Postgres table

Alter Postgres Tables via metadata 
(Thanks Louis R.) 

select atttypmod from pg_attribute where attrelid = 'TABLE1'::regclass and attname = 'title';
# then
update pg_attribute set atttypmod = 200+4 where attrelid = 'TABLE1'::regclass and attname = 'title';

Thursday, April 21, 2011

Quick Setup Instructions for SVN Compile and Local Install

Steps to make a custom build of SVN and place it in non-system directory.

Make a place to store the new copy:
mkdir -p /home/myuser/customsvn/src/
cd /home/myuser/customsvn/src/

Download from Apache Subversion; place in local install src directory:
ls -la /home/myuser/customsvn/src/

Untar subversion then subversion-deps:
tar -jxvf subversion-1.6.xx.tar.bz2
tar -jxvf subversion-deps-1.6.xx.tar.bz2

Configure Command (Configure without Apache stuff):
./configure --prefix=/home/myuser/customsvn/bin/svn/ --with-apxs=no --disable-mod-activation --with-serf=no --with-berkeley-db=no  | tee configure.jjj.log
make | tee make.jjj.log
make install | tee make_install.jjj.log

Tuesday, April 19, 2011

SVN Merging For Version 1.4-

Merging for svn 1.4- is a bit different than for newer versions.  
This is what I have found that works for merging a trunk to a branch:
svn merge \
  svn+ssh:// \
  svn+ssh:// \

*NOTE: seems weird that the trunk is the argument after the branch, but my trunk changes weren't being merged in when I did it the other way.

It seems that with svn version pre 1.4, you must specify the version where your branch was last copied or merged from.  If your branch was created at version 234, then the following command would merge it. 
svn merge --dry-run \
  svn+ssh:// \
  svn+ssh:// .

Friday, April 15, 2011

Django Content Type Quick Reference

from django.contrib.contenttypes.models import ContentType

# Get the content type for a given model
ct = ContentType.objects.get(app_label="myapp", model="mymodel")
# or
ct = ContentType.objects.get_for_model(MyModel)

# Get the class associated with the content type
model_cls = ct.model_class()

# Filter the queryset as normal              
queryset = model_cls.objects.filter()

# Get a specific object instance given a content type

Wednesday, April 13, 2011

Cool Python Modules

String Similarity

google-diff-match-patch - The Diff Match and Patch libraries offer robust algorithms to perform the operations required for synchronizing plain text.

pylevenshtein - Levenshtein (edit) distance, and edit operations, string similarity...

difflib - classes and functions for comparing sequences

Database Tools

uuid - Python built-in module for creating unique ids

django-command-extensions: UUIDField -  uuid field distributed with django-comamnd-extensions.

django-uuidfield - stand-alone Django UUIDField implementation.

System Tools

Imp - get python internals

Django Tools

AutoSlugify - automate slug field

Postgresql Sequences

Create a new sequence:
CREATE SEQUENCE mytable_myid_seq;

Add the sequence as the incrementor for a table column:
SET DEFAULT NEXTVAL('mytable_myid_seq');

Update existing rows in table with the sequence:
UPDATE mytable
SET myid = NEXTVAL('mytable_myid_seq');

Display the sequence information:
Select * from mytable_myid_seq;

Update a sequence to be one greater than a table's id column:
select setval('mytable_myid_seq', (select max(id) + 1 from mytable)); (thanks Louis)

Friday, April 01, 2011

Django News/Code At

I've created a more formal blog called, focused around Django.  I'll continue to post code snippets and information on this site, but will additionally post more formal write-ups at  Check it out!

Wednesday, March 16, 2011

Pretty Print XML with Python

Pretty Print XML

import xml.dom.minidom

# Read XML from File
xml = xml.dom.minidom.parse(xml_filename) 

# Read XML from String
xml = xml.dom.minidom.parseString(xml_string)

Friday, February 25, 2011

Django Chain QuerySet

Based off of:

Find Open Ports

Find open ports on local Linux machine:
sudo netstat -lnptu

Sunday, January 30, 2011

Django Testing Views

Setting up the example:

--- -------------------------------------
# define a view for testing
def index(request, template_name=''):
    context = {'things':MyThing.objects.all(),}
    return render_to_response(
           context, context_instance=RequestContext(request))

--- ---------------------------------------
# define a url for testing
urlpatterns = patterns('',
    url('^$', 'core.views.index',
        {'template_name': 'someindex.html'},

Testing from a Dango shell:
import django
from django.test.client import Client
django.test.utils.setup_test_environment() # needed to see response context
client = Client()
response = client.get('/')
# we now have access to the response context and can use this
# test code in a unittest file.  (not yet shown)

Testing in a file:
class SimpleTest(TestCase):

    def test_view_list(self):
        resp = self.client.get(reverse('some_named_url'))
        self.assertEqual(resp.status_code, 200)
        self.assertTrue('somevar' in resp.context[-1])

Thursday, January 06, 2011

Favorite Plugins

Firefox Plugins for Web Development
- ColorZilla # color picker
- Firebug # CSS / HTML / JS debugger
- Firecookie # cookie info for firebug
- JSONView # Pretty JSON viewer
- RESTClient # GET and POST to restful interfaces
- Selenium IDE # Create tests you can replay in the browser
- Tamper Data # Allow one to tamper with headers and POST vars the client sends
- View Source Chart # Pretty HTML viewer
- Web Developer # Web developer tools
- GMarks # google bookmars sync
- LIVE HTTP headers # view http headers
- SQLite Manager # manage SQLite db files
- YSlow # Yahoo performance tool
- SearchStatus # find website rankings

Thunderbird Plugins for Google Sync
- Lightening # Calendar plugin
- google_contacts # sync google contacts
- CalDev sync google calendar #

Eclipse Plugins
- Pydev # python web development
- Subclipse # subversion for eclipse
- Egit # git plugin for eclipse
- m2eclipse # maven plugin for Java package management
- IndentGuide  # nice guidelines for code indenting

Wednesday, January 05, 2011

Python Date Operations

Get current time:
>>> import datetime
>>>, 1, 12)
datetime.datetime(2011, 1, 12, 20, 36, 21, 430985)

Adding time and timedelta:
>>> datetime.timedelta(hours=7) + datetime.datetime(2011,1,1)
datetime.datetime(2011, 1, 1, 7, 0)
>>> datetime.timedelta(days=50) +,1,1), 2, 20)

Subtracting dates:
>>>,1,2) -,1,1)

Convert date and time into datetime:
>>> import datetime
>>> d =,01,05)
>>> t = datetime.time(10,25)
datetime.datetime(2010, 1, 5, 10, 25)

Convert String to datetime:
>>> datetime.datetime.strptime("2011-01-05","%Y-%m-%d")
datetime.datetime(2011, 1, 5, 0, 0)

Convert datetime to individual components:
 >>> dt = datetime.datetime(2011,01,05,12,00)
>>> (dt.year, dt.month,, dt.hour, dt.minute)
(2011, 1, 5, 12, 0)

Convert datetime to date:
 >>> dt = datetime.datetime(2011,01,05,12,00)
>>>, 1, 5)

Add one day to a date:
>>> dt = datetime.datetime(2011,01,05,12,00)
>>> delta = datetime.timedelta(days=1)
>>> dt + delta
datetime.datetime(2011, 1, 6, 12, 0)

Number of days in a month:
>>> import calendar
>>> calendar.monthrange(2011,1)
(5, 31)

from pytz import timezone as tz
localtz = tz('US/Central')
#dt_aware = localtz.localize(dt_unaware)

more to come...