Postgres Access

In pg_hba.conf, set the following: 

To allow all remote users to log into a postgres database:

In pg_hba.conf, set the following: 

# remote hosts    Database    User      From any IP       Using PG md5 auth

host                      all                 all         0.0.0.0/0            md5

In the postgres.conf, set the following:
listen_addresses = '*'

To allow local UNIX users with the same name as PG users to auth automatically:
local                     all                 all           trust

Redis

Login to Redis:
redis-cli

Login to Remote Redis:
redis-cli -h 1.2.3.4
auth mypassword

Switch database:
select 1  # where one is the database name

List keys in database:
redis> keys *

Persist everything to disk right away:
save

Get the location of the redis data dump dir:
config get dir
1) "dir"
2) "/var/lib/redis/"

Backup the Redis database datafile:
cp /var/lib/redis/dump.rdb /my/backup/location/

Drop all data in db:
flushdb

List of Installed Pacakges on Debian Based System

List packages:
dpkg --get-selections > outfile.txt

Reinstall:
dpkg --get-selections < outfile.txt

http://ubuntuforums.org/showthread.php?t=261366

Get IP Address from command line

wget -q -O - checkip.dyndns.org

http://go2linux.garron.me/what-is-my-public-ip-address-with-linux

Django/Web Modules

Modernizer - Feature Detection for JS
http://modernizr.com/

Django-flash - Rails-like django flash messages:
https://github.com/danielfm/django-flash

Facebook-Graph - Django facebook graph API
https://github.com/feinheit/django-facebook-graph

django-cumulus - Rackspace manager

http://packages.python.org/django-cumulus/

Django-ittybitty - Django based URL Shortener:

http://code.google.com/p/django-ittybitty/

django-ittybitty

Django Storages

http://django-storages.readthedocs.org/en/latest/
django-storages

Python-Postmark - API for http://postmarkapp.com/

https://github.com/themartorana/python-postmark

Django Cache Utils

http://pypi.python.org/pypi/django-cache-utils

stripe-python - API for Stripe.com (like Braintree)

https://github.com/stripe/stripe-python

FrogBugz - online ticket tracking system

http://www.fogcreek.com/fogbugz/pricing.html

Form Builder
http://www.wufoo.com

Factual - service for data information and places. GeoData
http://v2.factual.com/

CartoDB - open source data browser db
https://github.com/Vizzuality/cartodb

GeoCoda - Online Geocoding tool
https://geocoda.com/

Census Tiger Dataset
http://www.census.gov

SendGrid - Email Web applications
http://sendgrid.com/

RapidSSL - Cheap SSL Host

django-tables2 - manipulate html tables programmatically similar to Django forms.
http://django-tables2.readthedocs.org/en/latest/index.html

django-message-extends - extensions to Django messages such as sticky messages and assigned messages
https://github.com/AliLozano/django-messages-extends

Simple Logging

Simple Logging

In someplace like python shell, __init__.py, or settings.py (pre Django 1.3):

import logging
logger = logging.getLogger('mymodule.test')
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s - %(name)s - %(lineno)s - %(levelname)s - %(message)s")

# Configure log to file
filelog = logging.FileHandler("mylogfile.log", 'a')
filelog.setLevel(logging.INFO)
filelog.setFormatter(formatter)
logger.addHandler(filelog)

# Configure log to stdout
conlog = logging.StreamHandler()
conlog.setLevel(logging.DEBUG)
conlog.setFormatter(formatter)
logger.addHandler(conlog)

In place you want to log in, like views.py:

import logging
logger = logging.getLogger('mymodule.test')
logger.debug("My Log text")
logger.info("My info text")

Another example of simple logging:

import logging
logger = logging.getLogger('mymodulename')
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
logger.addHandler(ch)

---------------------------------------------------

A simple example:

import logging
logging.basicConfig(filename='test.log', level=logging.DEBUG)
logging.info("INFO")
logging.debug("DEBUG")
logging.error("ERROR")


Sources:
http://dancingpenguinsoflight.com/2009/03/simple-and-effective-python-logging/
http://www.blog.pythonlibrary.org/2012/08/02/python-101-an-intro-to-logging/

Dulwich Command Notes

# Load Git repo into object

from dulwich.repo import Repo

repo = Repo("/home/jjasinsk/Sites/env/proj/gitdjango/")

# Find sha hash of Git repo head

repo.head()

# 'e80b3ada92d2a02ce353c83683a6533e25240a1a'

# Find the Commit object of the repo head

repo[repo.head()]

# Commit e80b3ada92d2a02ce353c83683a6533e25240a1a

# Find all the repo tags

repo.refs.subkeys('refs/tags')

# set(['v0.0.1'])

# Find all the repo branches

repo.refs.subkeys('refs/heads')

# set(['master', 'develop'])

# Find all the repo refs and their sha hashes

repo.get_refs()

# {'HEAD': 'e80b3ada92d2a02ce353c83683a6533e25240a1a', 

   'refs/heads/develop': 'b53fe83f58f293decae39a70106a5ae78886ba5e', 

   'refs/tags/v0.0.1': '5554750c44f8dc17ab9dc5926129cf409d06d0bb', 

   'refs/heads/master': 'e80b3ada92d2a02ce353c83683a6533e25240a1a'}

# Find all the commits tracing back to head Commit 

repo.revision_history(repo.head())

# [Commit fcc1e1eb645f6731c4a8701b5facf852738850a9, 

    Commit 39eaa48156e3df2fd2cbe9922e33787cbafdad90, 

    Commit e80b3ada92d2a02ce353c83683a6533e25240a1a,

   ...

TAG OBJECTS

# Get the Tag object by looking up it's ref path
repo['refs/tags/v0.0.1']

# Tag 5554750c44f8dc17ab9dc5926129cf409d06d0b;

# Find Tag by sha hash

repo.get_object('5554750c44f8dc17ab9dc5926129cf409d06d0bb')

# Tag 5554750c44f8dc17ab9dc5926129cf409d06d0bb

repo.tag("5554750c44f8dc17ab9dc5926129cf409d06d0bb")

# Tag 5554750c44f8dc17ab9dc5926129cf409d06d0bb

repo['5554750c44f8dc17ab9dc5926129cf409d06d0bb']

# Tag 5554750c44f8dc17ab9dc5926129cf409d06d0bb

# Find information about a tag

t = repo['5554750c44f8dc17ab9dc5926129cf409d06d0bb']

# Find tagger 
t.tagger

# 'Joe Jasinski joe.jasinski@example.com'

# Find tag date

from datetime import datetime

datetime.utcfromtimestamp(t.tag_time)

datetime.datetime(2012, 3, 30, 6, 58, 53)

# Print tag contents

print t.as_pretty_string()

# ...

# Find Tag sha hash (id)

t.id

# '5554750c44f8dc17ab9dc5926129cf409d06d0bb'

   OR
s = t.sha()

s.hexdigest()

# '5554750c44f8dc17ab9dc5926129cf409d06d0bb'

# Find Tag message

t.message

# 'test tag\n'

# Find Tag Type

t.get_type()

# 4

COMMIT OBJECTS

# Find Commit object from ref spec
repo['refs/heads/master']

# Commit e80b3ada92d2a02ce353c83683a6533e25240a1a

# Look up commit several different ways

repo.get_object('b53fe83f58f293decae39a70106a5ae78886ba5e')

# Commit b53fe83f58f293decae39a70106a5ae78886ba5e

repo.commit("b53fe83f58f293decae39a70106a5ae78886ba5e")

# Commit b53fe83f58f293decae39a70106a5ae78886ba5e

repo['b53fe83f58f293decae39a70106a5ae78886ba5e']

# Commit b53fe83f58f293decae39a70106a5ae78886ba5e

# Find Commit hash 

repo.ref('refs/heads/master')

# 'e80b3ada92d2a02ce353c83683a6533e25240a1a'

# Find commit author 

repo['refs/heads/master'].author

# 'Joe Jasinski joe.jasinski@example.com'

# Find committer 

repo['refs/heads/master'].committer

# 'Joe Jasinski joe.jasinski@example.com'

# Find Commit message

repo['refs/heads/master'].message

# 'updated style\n'

# Find commit time

from datetime import datetime

datetime.utcfromtimestamp(repo['refs/heads/master'].commit_time)

# Find Trees associated with commit

repo['b53fe83f58f293decae39a70106a5ae78886ba5e'].tree

'b3437c8329fc3f6306936c7ca1986cefc1629862'

repo[c.tree].items()

[TreeEntry(path='.gitignore', mode=33188, sha='13609a506289eba22b61df5eab0f89b7aede323d'), 

TreeEntry(path='example', mode=16384, sha='f8e95712570c4316868ee9af32088324e8d3e498'), 

TreeEntry(path='gitdjango', mode=16384, sha='32b164fac85e5b40dd8d4265a0a8df765bf57300'), 

TreeEntry(path='manage.py', mode=33261, sha='2605e3768e2bab92ef0550d452036472e9a6a210'), 

...

 repo[repo[c.tree].items()[0].sha]

TREE OBJECTS

# Find Tree associated with hash 
repo.get_object('b3437c8329fc3f6306936c7ca1986cefc1629862')

# Tree b3437c8329fc3f6306936c7ca1986cefc1629862

# Find all items under a Tree

repo[b3437c8329fc3f6306936c7ca1986cefc1629862].items()

BLOB OBJECTS

# Find Blob associated with hash 

repo.get_blob("2605e3768e2bab92ef0550d452036472e9a6a210")

# Blob 2605e3768e2bab92ef0550d452036472e9a6a210

repo['2605e3768e2bab92ef0550d452036472e9a6a210']

# Blob 2605e3768e2bab92ef0550d452036472e9a6a210

MISC

# Iterate through all objects under a gree
for entry in repo.object_store.iter_tree_contents(repo['refs/heads/master'].tree):

    print entry

# TreeEntry(path='.gitignore', mode=33188, sha='13609a506289eba22b61df5eab0f89b7aede323d')

# TreeEntry(path='example/__init__.py', mode=33188, sha='e69de29bb2d1d6434b8b29ae775ad8c2e48c5391')

# TreeEntry(path='example/__init__.pyc', mode=33188, sha='ced988bb5f3f22f4f1956bf852f106b23eea5c09')

# ...

# Find a the hash of an object by specifying a filesystem path

from dulwich.object_store import tree_lookup_path  

(mode, hash) = tree_lookup_path(repo.get_object, repo['refs/heads/master'].tree, 'gitdjango/models.py')

# Find an object filehandle relative to inside the .git directory

repo.get_named_file('../requirements.pip')

# open file '/home/jjasinsk/Sites/env/proj/gitdjango/.git/../requirements.pip', mode 'rb' at 0x101cdf938

f.read()

# 'Django==1.4\nGitPython==0.3.2.RC1\nasync==0.6.1\ngitdb==0.5.4\ninclude-server==3.1-toolwhip.1\nsmmap==0.8.2\nwsgiref==0.1.2\n'

Monitoring Connections

Find IPs that have the most connections
netstat -plan|grep :80|awk {'print $5'}|cut -d: -f 1|sort|uniq -c|sort -nk 1
http://www.webhostingtalk.com/showthread.php?t=673467

Find Listening Connections (open ports)
netstat -an | grep -i listening

Other Netstat Commands:

netstat -ntl
netstat -nputw
netstat -a -n -p
netstat -tulpn
netstat -tunapl

http://unix.stackexchange.com/questions/56453/how-can-i-monitor-all-outgoing-requests-connections-from-my-machine

http://www.webhostingtalk.com/showthread.php?t=991911

Basic Vagrant Usage

Doing a 'vagrant box add boxname.box' creates a .vagrant.d/ directory in your home directory. This directory stores base boxes installed. Doing a 'vagrant init' in some other directory creates a Vagrantfile inside of that directory which specifies which base box to use and provides any other provisioning information. 

List vagrant base boxes installed:
vagrant box list
# returns list of boxnames and type
mybox (virtualbox)

Add a new vagrant base box named "my_box" based on box image:
vagrant box add my_box http://files.vagrantup.com/lucid32.box

Remove a base box:
vagrant box remove my_box type
# for a virtualbox use virtualbox as the type

Location of vagrant base boxes:
cd ~/.vagrant.d/

Create a directory where customizations off the of the base box should take place.  then run the following: 
cd ~/Sites/VM/test/
vagrant init
vagrant init lucid32  # as an alternative to explicitly indicate which base box to use

Example Vagrant file in ~/Sites/VM/test/Vagrantfile

Vagrant::Config.run do |config|

     config.vm.box = "lucid32"  # specify which base box to use

end

Run a virtual machine.  Builds a virtual machine using box and Vagrantfile and runs it.  You must be inside the given vagrant directory (~/Sites/VM/test) in order for this to work.

vagrant up

Destroy a virtual machine. Destroy's the virtual machine:
vagrant destroy

SSH into a virtual machine:
vagrant ssh

Running a packaged virtual machine
vagrant box add my_box /path/to/the/package.box
vagrant init my_box
vagrant up

Suspend a virtual machine (save state to disk):
vagrant suspend

Resume a suspended virtual machine:
vagrant resume

Some Customizations to the default config (Pre-vagrant 1.1): 
Vagrant::Config.run do |config|
    # this is basically a way to programmatically invoke the
    # 'VBoxManage modifyvm' command from within Vagrant 

    # NOTE: this requires a fairly new version of vagrant 
    # to use this syntax. 
    config.vm.customize ["modifyvm", :id, "--memory", 1024]

    # By default vagrant boots boxes headless.  This enables the 
    # Virtualbox GUI
    config.vm.boot_mode = :gui

    # Path to ssh private key used to log into box with 
    # vagrant ssh. If it's a relative path, it will be relative 
    # to the dir holding the Vagrantfile

    config.ssh.private_key_path = 'vagrant_id_dsa'

    # Creates a shared folder on host called myshared in the 
    # same directory as the Vagrantfile.  On the guest, it 
    # mounts the shared folder as /my/shared.  The :create 
    # setting creates the directory on the host if it doesn't
    # exist.   

    config.vm.share_folder 'my-shared', '/my/shared/', 'myshared', {:create, true}

end

Some Customizations to the default config (Post-vagrant 1.1): 

Vagrant.configure("2") do |config|
   # setup port forwarding 
  config.vm.network :forwarded_port, guest: 8000, host: 18000, auto_correct: true
  config.vm.network :forwarded_port, guest: 5432, host: 15432, auto_correct: true

  config.vm.provider :virtualbox do |vb|
      # Don't boot with headless mode
      vb.gui = true

      # Use VBoxManage to customize the VM. For example to change memory:
      vb.customize ["modifyvm", :id, "--memory", "1024"]

     # Create a shared folder   (host path, guest path)
     config.vm.synced_folder "../data", "/vagrant_data"
  end


Turn on Debugging
export VAGRANT_LOG=debug


List of vagrant base boxes:
http://vagrantbox.es/

Get SSH configuration for machines:

vagrant ssh-config
Host swarm_manager
  HostName 127.0.0.1
  User vagrant
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /Users/jjasinski/Sites/test_docker_swarm/.vagrant/machines/swarm_manager/virtualbox/private_key
  IdentitiesOnly yes
  LogLevel FATAL

Host swarm_node1
  HostName 127.0.0.1
  User vagrant
  Port 2200
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /Users/jjasinski/Sites/test_docker_swarm/.vagrant/machines/swarm_node1/virtualbox/private_key
  IdentitiesOnly yes
  LogLevel FATAL

Email Web Applications 

http://postmarkapp.com/
http://cloudmailin.com/
http://mailchimp.com
http://myemma.com/

Finding table sizes in postgres

Finding table sizes in postgres

Taken from: http://wiki.postgresql.org/wiki/Disk_Usage

SELECT nspname || '.' || relname AS "relation",
    pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size"
  FROM pg_class C
  LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
  WHERE nspname NOT IN ('pg_catalog', 'information_schema')
    AND C.relkind <> 'i'
    AND nspname !~ '^pg_toast'
  ORDER BY pg_total_relation_size(C.oid) DESC
  LIMIT 20;

Copy File through Compressed Tar

Copy and Compress File through an ssh connection
tar jcvf - myfile.txt | ssh myhost.com "cat > myfile.tar.bz2"

Django Session Introspection

Get Data Stored in a Session
from django.contrib.sessions import models
s = models.Session.objects.get(session_key='05adfa345a54525cad912')
s.get_decoded()

ImageMatik


Convert all TIFs in a directory to JPEGs
mogrify -format jpg -quality 50 *.tiff

http://linuxuser32.wordpress.com/2007/06/16/batch-image-convert-scale-thumbnail-jpegs-pdf/

Python: lxml insert element before and after given element

Python: lxml insert element before and after given element

Based on: http://stackoverflow.com/questions/7474972/python-lxml-append-element-after-another-element

from lxml import etree

xml = etree.parse('old_xml.xml')

count = 1

for pp in xml.xpath('/response/items/item'):

   prod_id = etree.Element('new_element')  # this element will be inserted

   prod_id.text = "%s" % count

   nav = pp.find('name')  # we will insert before/after this element

   div = nav.getparent()

   pp.insert(pp.index(nav), prod_id) # insert before 'name'

   pp.insert(pp.index(nav) + 1, prod_id) # insert after 'name'

   count += 1

out = open('new_xml.xml','w')

out.write(etree.tostring(xml, pretty_print=True))

out.close()

Python: Detect Charset and convert to Unicode


Converting data to Unicode via chardet 
http://stackoverflow.com/questions/2686709/encoding-in-python-with-lxml-complex-solution

import chardet
from lxml import html

f = open('my.xml')

content = f.read()

f.close()

encoding = chardet.detect(content)['encoding']
if encoding != 'utf-8':
    content = content.decode(encoding, 'replace').encode('utf-8')

Converting Data to Unicode via UnicodeDammit 

http://lxml.de/elementsoup.html

from BeautifulSoup import UnicodeDammit

converted = UnicodeDammit(content)

   if not converted.unicode: 

    raise UnicodeDecodeError( 

       "Failed to detect encoding, tried [%s]", 

        ', '.join(converted.triedEncodings))  

        # print converted.originalEncoding

         return converted.unicode

Credit goes to Ian Bicking and others on the lxml team

Using lxml xpath to get elements with a default namespace

xml_string="""

<a xmlns="http://www.w3.org/1999/xhtml">

  <b>
    <r>
      <Value>string</Value>

    </r>

    <r>

      <Value>string</Value>

    </r>

  </b>

  <e>

    <string>string1</string>

    <string>string2</string>

  </e>

</a>

"""

from lxml import etree

xml = etree.fromstring(xml_string)

xml.xpath('/w3:a/w3:b/w3:r/w3:Value/text()', namespaces={'w3':'http://www.w3.org/1999/xhtml'})

SQLAlchemy Notes

Summarized From 
http://www.sqlalchemy.org/docs/core/tutorial.html


This covers the basics of the Non-orm usage of SQLAlchemy

# Define connection engine
from sqlalchemy import create_engine
engine = create_engine('sqlite:///:memory:', echo=True)

# Create some database tables 
from sqlalchemy import Table, Column, Integer, String, MetaData, ForeignKey
metadata = MetaData()
users = Table('users', metadata,
    Column('id', Integer, primary_key=True),
    Column('name', String),
    Column('fullname', String),
)

addresses = Table('addresses', metadata,
  Column('id', Integer, primary_key=True),
  Column('user_id', None, ForeignKey('users.id')),
  Column('email_address', String, nullable=False)
)

metadata.create_all(engine)

# Insert into the tables
ins = users.insert()
ins.values(name='jack', fullname='Jack the Pumpkin king')

conn = engine.connect()
result = conn.execute(ins)
result.inserted_primary_key

# Multiple insert into the tables
conn.execute(addresses.insert(), [
   {'user_id': 1, 'email_address' : 'jack@yahoo.com'},
   {'user_id': 1, 'email_address' : 'jack@msn.com'},
   {'user_id': 2, 'email_address' : 'www@www.org'},
   {'user_id': 2, 'email_address' : 'wendy@aol.com'},
])

result = engine.execute(users.insert(), name='fred', fullname="Fred Flintstone")

# Select from the tables
from sqlalchemy.sql import select

# Select all fields in user table
s = select([users])
result = conn.execute(s)
list(result)

# Select name and fullname columns from user table
s = select([users.c.name, users.c.fullname])

# Join two user and address tables
#  Cartesian product 
list(conn.execute(select([users, addresses])))
# Join on id columns
list(conn.execute(select([users, addresses], users.c.id==addresses.c.user_id)))

# create conditions for the where clause
from sqlalchemy.sql import and_, or_, not_
print and_(users.c.name.like('j%'), users.c.id==addresses.c.user_id, )


s = select([(users.c.fullname + "--" + addresses.c.email_address).label('title')])
print conn.execute(s).fetchall()

print users.join(addresses)
print users.join(addresses, addresses.c.email_address.like(users.c.name + '%'))


s = select([users.c.fullname], from_obj=[
   users.join(addresses, addresses.c.email_address.like(users.c.name + '%'))
   ])
print conn.execute(s).fetchall()

# Delete records based on a condition
conn.execute(users.delete().where(users.c.name > 'm'))

Django: Quickly testing an authenticated ajax

Make a authenticated post with the django test client and mark simulate an ajax post:
from django.test import Client

c = Client()

c.login(username='jjasinski', password='xxxx')

print c.post('/user/posts/1/',{},HTTP_X_REQUESTED_WITH='XMLHttpRequest')

Self-signed SSL Certificate

Generate self-signed certificate
# Generate Encrypted RSA Private Key with passphrase
openssl genrsa -des3 -out myssl.key 2048

# Generate Unencrypted RSA Private Key 
openssl genrsa -out myssl.key 2048

# Remove Passphrase from key

mv myssl.key myssl.key.org
openssl rsa -in myssl.key.org -out myssl.key

# Generate Certificate Signing Request with an existing Private Key

openssl req -sha256 -new -key myssl.key -out myssl.csr

# Create SSL certificate
openssl x509 -req -days 365 -in myssl.csr -signkey myssl.key -out myssl.crt

Generating a key and csr in one command

# for providing to your certificate provider

openssl req -sha256 -new -newkey rsa:2048 -nodes -keyout yourdomain.key -out yourdomain.csr

   req = create PKCS#10 X.509 Certificate Signing Request
   -sha256 = adds support for SHA-2

Decode PEM encoded SSL certificate meta information
openssl x509 -in certificate.crt -text -noout
OR
openssl x509 -noout -text -modulus -in https://example.com.crt

Decode DER encoded SSL certificate meta information
openssl x509 -in certificate.crt -inform der -text -noout

Decode SSL key meta information
openssl rsa -noout -text -modulus -in https://example.com.key

Convert PEM encoded SSL certificate to DER encoded:
openssl x509 -in certificate.crt -outform der -out certificate.der

Convert DER encoded SSL certificate to PEM encoded:
openssl x509 -in certfile.crt -inform der -outform pem -out certificate.pem


Note: in the meta info, the Modulus should be the same for the key and cert.


Sources
http://articles.slicehost.com/2007/12/19/ubuntu-gutsy-self-signed-ssl-certificates-and-nginx
http://support.godaddy.com/help/article/3601/generating-a-certificate-signing-request-nginx
http://www.sslshopper.com/certificate-decoder.html
https://www.digitalocean.com/community/tutorials/openssl-essentials-working-with-ssl-certificates-private-keys-and-csrs
https://support.ssl.com/Knowledgebase/Article/View/19/0/der-vs-crt-vs-cer-vs-pem-certificates-and-how-to-convert-them