Using Rsync to copy file Structure

Simple rsync copy on local filesystem:
rsync -r -t -v -l source/ dest

rsync over ssh:
rsync -r -t -v -l username@source.com:/path/to/src/ /path/to/dest

rsync copy shortcut to archive:
rsync -avz username@source.com:/path/to/src/ /path/to/dest
rsync -avz --progress username@source.com:/path/to/src/ /path/to/dest

rsync copy archive without caring about owner and group:
rsync -rlptD  username@source.com:/path/to/src/ /path/to/dest
rsync -a --no-o --no-g username@source.com:/path/to/src/ /path/to/dest

Common Flags:
 -a = archive copy.  Equivalent to flags -rlptgoD
 -r = recursive copy
 -t = save timestamp
 -v = verbose copy
 -vv = more verbose
 -u = don't copy over existing files newer than source
 -l = preserve symlinks
 -p = preserve premissions
 -g = preserve group
 -o = preserve owner
 -D = preserve devices (superuser only)
 -z = compress transfer
 --progress = show copy progress
--delete = delete stuff on the receiving side that doesn't exist on the sending side

http://www.cyberciti.biz/tips/linux-use-rsync-transfer-mirror-files-directories.html
http://en.wikipedia.org/wiki/Rsync
http://serverfault.com/questions/364709/how-to-keep-rsync-from-chowning-transfered-files

Git INFO Common Commands

Configure Git client options:
# show config options
git config --list # show all
git config --global --list # show only global
git config --local --list # show only local

# --global option impacts options at a global level
# --local option impacts options at a local level (.git/config)
# set username/email associated with client
git config --global user.email "joe@example.com"
git config --global user.name "Joe Jaz"

# add color syntax highlighting
git config --global color.status auto
git config --global color.branch auto

# set default pager and editor
git config --global core.editor vim # or set the GIT_EDITOR environment variable
git config --global core.pager less # or set the GIT_PAGER environment variable

# display an option value
git config group.value
git config user.name

Start a new repository and import:
# In the root folder to add to version control:
git init
git init --bare  # creates a new repository (with no working copy)
git add . # add everything in directory to repo
git commit -m 'initial commit'

Making commits:
# commit specific files
git commit file1 file2 -m 'comment'

# commit all 'added' files
git commit -a -m 'comment'

# change a commit message
git commit --amend -m 'New Message'

Create new repository by cloning existing one:
git clone --bare . path_or_url_to_git_dir.git

Setting the remote repository (to push and pull from):
git remote add short_name path_or_url_to_git_dir.git
git remote add origin git://github.com/imagescape/iscape-authlog.git

Show remote repositories available:
git remote
git remote -v # show the url
git remote show origin  # show detail information about origin

Remove a remote repository: 
git remove repo_name

Rename a remote repository: 
git rename old_repo_name new_repo_name

Push and Pull from remote repository:
git push origin # push recent commits to remote repository called "origin"
git pull origin # pull updates from origin

Revert changes in working copy:
git clean -n
git clean --dry-run

Show file differences:
git diff   # show differences between working copy and any files staged for commit
git diff --staged   # compares staged changes with last commit
git diff --cached  # same as above for git older than 1.6.1

git difftool -t [meld|kdiff3|vimdiff|etc.] [diff criteria]
git difftool -t meld --staged
git difftool --tool=meld --staged
git difftool origin/master  --dir-diff # use difftool to diff entire dir tree

git diff -U999999999 master.. -- file.txt # shows the full file.txt, with diff markup, in the master/branch diff results

git diff --name-status brancha..branchb # shows list of changed files

Show File/s History:
git log filename # Show single file versions

# Show number of lines changed in file/s
git log -stat filename # single file
git log -stat # multiple files

# show log summaries on one line
git log --pretty=oneline # Pre-defined format
git log --pretty=format:"%h - %cD (%an) %s" # custom format (see below)
git log --pretty=format:"%h %s" --graph
git log --graph --full-history --all --color \

           --pretty=format:"%x1b[31m%h%x09%x1b[32m%d%x1b[0m%x20%s"
    Pre-Defined Pretty formats:
    oneline
    short
    medium
    full
    fuller
    email
    raw
    format:string # where string is

        Format Strings
        %H: commit hash
        %h: abbreviated commit hash
        %T: tree hash
        %t: abbreviated tree hash
        %P: parent hashes
        %p: abbreviated parent hashes
        %an: author name
        %aN: author name (respecting .mailmap, see git-shortlog(1) or git-blame(1))
        %ae: author email
        %aE: author email (respecting .mailmap, see git-shortlog(1) or git-blame(1))
        %ad: author date (format respects --date= option)
        %aD: author date, RFC2822 style
        %ar: author date, relative
        %at: author date, UNIX timestamp
        %ai: author date, ISO 8601 format
        %cn: committer name
        %cN: committer name (respecting .mailmap, see git-shortlog(1) or git-blame(1))
        %ce: committer email
        %cE: committer email (respecting .mailmap, see git-shortlog(1) or git-blame(1))
        %cd: committer date
        %cD: committer date, RFC2822 style
        %cr: committer date, relative
        %ct: committer date, UNIX timestamp
        %ci: committer date, ISO 8601 format
        %d: ref names, like the --decorate option of git-log(1)
        %e: encoding
        %s: subject
        %f: sanitized subject line, suitable for a filename
        %b: body
        %B: raw body (unwrapped subject and body)
        %N: commit notes
        %gD: reflog selector, e.g., refs/stash@{1}
        %gd: shortened reflog selector, e.g., stash@{1}
        %gs: reflog subject
        %Cred: switch color to red
        %Cgreen: switch color to green
        %Cblue: switch color to blue
        %Creset: reset color
        %C(…): color specification, as described in color.branch.* config option
        %m: left, right or boundary mark
        %n: newline
        %%: a raw %
        %x00: print a byte from a hex code
        %w([[,[,]]]): switch line wrapping, like the -w option of git-shortlog(1).

Show git branches available:
git branch

Show all branches including remote branches:
git branch -a

Create new branch:
git branch new_branch

Change branches:
git checkout new_branch

Delete branch:
git branch -d new_branch # delete only if all changes are merged
git branch -D new_branch  # delete even if there are unmerged changes

Merge branch into current copy:
git merge --no-ff branch_to_merge_in

Resolve a Merge:
1) Edit file to resolve
2) git add conflict_file
3) git commit

Using Mergetool
git mergetool
git mergetool --tool=meld
git mergetool -t meld

Abort a merge conflict before committing:
git merge --abort

Undo a Merge:  # restores state to pre-merge
git reset --heard HEAD

Undo a local commit: 
git reset --soft HEAD^

Find the commit where two branches diverge:
git merge-base branch1 branch2

    http://stackoverflow.com/questions/1549146/find-common-ancestor-of-two-branches

Find what branches were merged into a given branch:
git branch --merged master  # local branches only
git branch -a --merged master   # local and remote branches

Find what branches were NOT merged into a given branch:
git branch --no-merged master  # local branches only
git branch -a --no-merged master   # local and remote branches

Find what branches are merged into develop, but not master:
comm -12 <(sort <(git branch -a --no-merged origin/master)) <(sort <(git branch -a --merged origin/develop))

    https://stackoverflow.com/questions/8071079/git-list-branches-merged-into-a-branch-but-not-into-another

Copy all branches from origin to new_remote: 
git push new_remote refs/remotes/origin/*:refs/heads/*

Get number of commits per author
apt-get install git-extras

git summary -n

References:
http://jonas.nitro.dk/git/quick-reference.html
http://www.vogella.de/articles/Git/article.html#git_stagingindex
http://book.git-scm.com/3_basic_branching_and_merging.html
http://stackoverflow.com/questions/1838873/visualizing-branch-topology-in-git
http://stackoverflow.com/questions/226976/how-can-i-know-in-git-if-a-branch-has-been-already-merged-into-master
http://stackoverflow.com/questions/7818927/git-push-branch-from-one-remote-to-another

Python Process Info

Pipe the output of one process into the input of another via python:
import subprocess
p1 = subprocess.Popen(['echo','joe'], shell=False, stdout=subprocess.PIPE)
p2 = subprocess.Popen(['cat','-n'],
    stdin=p1.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p2.stdout.read()

# Or 
out,err=p2.communicate()

Pipe standard input to a program
import subprocess
p1 = subprocess.Popen(['node', 'index.js'], shell=False,
    stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p1.communicate('standard input here')
p1.returncode

Convenience method for splitting apart shell commands for use in the above:
>>> import shlex
>>> shlex.split('ps -ef')
['ps', '-ef']

Common Open ID Urls

Taken From http://stackoverflow.com/questions/1116743/where-can-i-find-a-list-of-openid-provider-urls

Google https://www.google.com/accounts/o8/id
Yahoo https://me.yahoo.com
Flickr http://www.flickr.com/username
AOL http://openid.aol.com/username
Blogspot https://www.blogspot.com/
LiveJournal http://username.livejournal.com/
Wordpress https://username.wordpress.com/
VerisignLabs https://pip.verisignlabs.com/
MyOpenID https://www.myopenid.com/
MyVidoop https://myvidoop.com/
ClaimID https://claimid.com/username
Technorati https://technorati.com/people/technorati/username/

SVN List Codes

Codes Displayed when doing an "svn stat"
U = Working file was updated
G = Changes on the repository were automatically merged into the working copy
M = Working Copy is modified
C = This file conflicts with the version in the repo
? = This file is not under version control
! = This file is under version control but is missing or incomplete
A = This file will be added to version control after commit
A+ = This file will be moved after commit
D = This file will be deleted after commit
I = This item is being ignored (due to the svn:ignore porperty)
L = This file is locked
X =  Item is unversioned, but used by an externals definition
~ = Messed up item (file when it should be dir). resolve by moving and doing "svn up"

http://knaddison.com/technology/svn-status-code-cheat-sheet

Whoosh Python Search Notes

Baisc Whoosh Search Walkthrough 


# Import required 
import os
from whoosh.index import create_in
from whoosh.fields import *

# Create a "Schema" object to store text to search
# You can create custom search fields such as "title", "domain", "path"
# and "content". 
schema = Schema(title=TEXT(stored=True), domain=ID(stored=True),
                              path=ID(stored=True), content=TEXT)

# Check to see if a search index directory exists; if not, create one
if not os.path.exists("search_test"):
    os.mkdir("search_test")

# Create a search index in the search index directory
ix = create_in("search_test", schema, indexname="testindex")
writer = ix.writer()

# Index some documents
writer.add_document(title=u"Document 1", domain=u"www.mysearch.com",
                         path=u"/mywebsite", content=u'Article')
writer.add_document(title=u"Document 2", domain=u"www.mysearch.com",
                         path=u"/mywebsite/2/article/1", content=u'Document a')
writer.commit()

# Running search queries
from whoosh.qparser import QueryParser
import whoosh.index as index
# Check to see if the given search index exists
if index.exists_in("search_test", indexname="testindex"):
    # open the search index
    ix = index.open_dir("search_test", indexname="testindex")
    # generate a search query based on some string.
    # Here is creates a query looking for "Document" in the "content"
    # search index field. 
    query = QueryParser("content", ix.schema).parse(u'*Document*')
    with ix.searcher() as searcher:  
        # do the actual search
        results = searcher.search(query)
        results

# Running search queries
from whoosh.qparser import QueryParser
import whoosh.index as index
if index.exists_in("search_test", indexname="testindex"):
    ix = index.open_dir("search_test", indexname="testindex"))
    query = QueryParser("domain", ix.schema).parse(u'www.mydomain.com')
    with ix.searcher() as searcher:
        results = searcher.search(query)
        results

# Running search queries
from whoosh.qparser import QueryParser, MultifieldParser
import whoosh.index as index
from whoosh.query import *
if index.exists_in("search_test", indexname="testindex"):
    ix = index.open_dir("search_test")
    query = MultifieldParser(["domain","title","content"], schema=ix.schema).parse(u"Document OR Article")
    with ix.searcher() as searcher:
        results = searcher.search(query)
        results

Different way to alter a Postgres table

Alter Postgres Tables via metadata 
(Thanks Louis R.) 

select atttypmod from pg_attribute where attrelid = 'TABLE1'::regclass and attname = 'title';
# then
update pg_attribute set atttypmod = 200+4 where attrelid = 'TABLE1'::regclass and attname = 'title';

http://sniptools.com/databases/resize-a-column-in-a-postgresql-table-without-changing-data

Quick Setup Instructions for SVN Compile and Local Install

Steps to make a custom build of SVN and place it in non-system directory.

Make a place to store the new copy:
mkdir -p /home/myuser/customsvn/src/
cd /home/myuser/customsvn/src/

Download from Apache Subversion; place in local install src directory:
ls -la /home/myuser/customsvn/src/
subversion-1.6.xx.tar.bz2
subversion-deps-1.6.xx.tar.bz2

Untar subversion then subversion-deps:
tar -jxvf subversion-1.6.xx.tar.bz2
tar -jxvf subversion-deps-1.6.xx.tar.bz2

Configure Command (Configure without Apache stuff):
./configure --prefix=/home/myuser/customsvn/bin/svn/ --with-apxs=no --disable-mod-activation --with-serf=no --with-berkeley-db=no  | tee configure.jjj.log
make | tee make.jjj.log
make install | tee make_install.jjj.log

SVN Merging For Version 1.4-

Merging for svn 1.4- is a bit different than for newer versions.  
This is what I have found that works for merging a trunk to a branch:
svn merge \
  svn+ssh://svn.example.com/repo/myproject/branches/newbranch@HEAD \
  svn+ssh://svn.example.com/repo/myproject/trunk@HEAD \
  /sites/myproject/src/myproject-workingcopy


*NOTE: seems weird that the trunk is the argument after the branch, but my trunk changes weren't being merged in when I did it the other way.

It seems that with svn version pre 1.4, you must specify the version where your branch was last copied or merged from.  If your branch was created at version 234, then the following command would merge it. 
svn merge --dry-run \
  svn+ssh://svn.example.com/export/myproject/branches/newbranchh@234 \
  svn+ssh://svn.example.com/export/myproject/trunk/@HEAD .


http://www.svnforum.org/threads/38394-1.4-merge

Django Content Type Quick Reference

from django.contrib.contenttypes.models import ContentType

# Get the content type for a given model
ct = ContentType.objects.get(app_label="myapp", model="mymodel")
# or
ct = ContentType.objects.get_for_model(MyModel)

# Get the class associated with the content type
model_cls = ct.model_class()

# Filter the queryset as normal              
queryset = model_cls.objects.filter()

# Get a specific object instance given a content type
ct.get_object_for_this_type(username='Guido')

http://docs.djangoproject.com/en/dev/ref/contrib/contenttypes/

Cool Python Modules

String Similarity

google-diff-match-patch - The Diff Match and Patch libraries offer robust algorithms to perform the operations required for synchronizing plain text.

http://code.google.com/p/google-diff-match-patch/wiki/API

pylevenshtein - Levenshtein (edit) distance, and edit operations, string similarity...

http://code.google.com/p/pylevenshtein/

difflib - classes and functions for comparing sequences

http://docs.python.org/library/difflib.html

Database Tools

uuid - Python built-in module for creating unique ids
http://docs.python.org/library/uuid.html

django-command-extensions: UUIDField -  uuid field distributed with django-comamnd-extensions.
http://code.google.com/p/django-command-extensions/

django-uuidfield - stand-alone Django UUIDField implementation.
https://github.com/dcramer/django-uuidfield

System Tools

Imp - get python internals
http://docs.python.org/library/imp.html

Django Tools

AutoSlugify - automate slug field
http://packages.python.org/django-autoslug/

Postgresql Sequences

Create a new sequence:
CREATE SEQUENCE mytable_myid_seq;

Add the sequence as the incrementor for a table column:
ALTER TABLE mytable
ALTER COLUMN myid
SET DEFAULT NEXTVAL('mytable_myid_seq');

Update existing rows in table with the sequence:
UPDATE mytable
SET myid = NEXTVAL('mytable_myid_seq');

Display the sequence information:
Select * from mytable_myid_seq;

Update a sequence to be one greater than a table's id column:
select setval('mytable_myid_seq', (select max(id) + 1 from mytable));
http://railspikes.com/2009/3/6/duplicate-key-violates-unique-constraint (thanks Louis)
http://pointbeing.net/weblog/2008/03/mysql-versus-postgresql-adding-an-auto-increment-column-to-a-table.html

Django News/Code At www.djangocurrent.com

I've created a more formal blog called www.djangocurrent.com, focused around Django.  I'll continue to post code snippets and information on this site, but will additionally post more formal write-ups at djangocurrent.com.  Check it out!

Pretty Print XML with Python

Pretty Print XML

import xml.dom.minidom

# Read XML from File

xml = xml.dom.minidom.parse(xml_filename) 

# Read XML from String

xml = xml.dom.minidom.parseString(xml_string)

xml.toprettyxml()

http://stackoverflow.com/questions/749796/pretty-printing-xml-in-python

Django Chain QuerySet

Based off of:
http://zmsmith.com/2010/04/using-custom-django-querysets/

Find Open Ports

Find open ports on local Linux machine:
sudo netstat -lnptu

http://www.techrepublic.com/blog/security/list-open-ports-and-listening-services/443

Django Testing Views

Setting up the example:

--- views.py -------------------------------------
# define a view for testing
def index(request, template_name=''):
    context = {'things':MyThing.objects.all(),}
    return render_to_response(
           template_name,
           context, context_instance=RequestContext(request))

--- urls.py ---------------------------------------
# define a url for testing
urlpatterns = patterns('',
...
    url('^$', 'core.views.index',
        {'template_name': 'someindex.html'},
         name="index"),
)

Testing from a Dango shell:
import django
from django.test.client import Client
django.test.utils.setup_test_environment() # needed to see response context
client = Client()
response = client.get('/')
response.context[-1]['things']

# we now have access to the response context and can use this

# test code in a unittest tests.py file.  (not yet shown)

Testing in a test.py file:
class SimpleTest(TestCase):

    def test_view_list(self):
        resp = self.client.get(reverse('some_named_url'))
        self.assertEqual(resp.status_code, 200)
        self.assertTrue('somevar' in resp.context[-1])

http://ericholscher.com/blog/2008/nov/11/practical-examples-test-django/

Favorite Plugins

Firefox Plugins for Web Development
- ColorZilla # color picker
- Firebug # CSS / HTML / JS debugger
- Firecookie # cookie info for firebug
- JSONView # Pretty JSON viewer
- RESTClient # GET and POST to restful interfaces
- Selenium IDE # Create tests you can replay in the browser
- Tamper Data # Allow one to tamper with headers and POST vars the client sends
- View Source Chart # Pretty HTML viewer
- Web Developer # Web developer tools
- GMarks # google bookmars sync
- LIVE HTTP headers # view http headers
- SQLite Manager # manage SQLite db files
- YSlow # Yahoo performance tool
- SearchStatus # find website rankings

Thunderbird Plugins for Google Sync
- Lightening # Calendar plugin
- google_contacts # sync google contacts
- CalDev sync google calendar # http://www.google.com/support/calendar/bin/answer.py?hl=en&answer=99358

Eclipse Plugins
- Pydev # python web development
- Subclipse # subversion for eclipse
- Egit # git plugin for eclipse
- m2eclipse # maven plugin for Java package management
- IndentGuide  # nice guidelines for code indenting http://atlanto.github.io/IndentGuide/

Python Date Operations

Get current time:
>>> import datetime
>>> datetime.date.today()
datetime.date(2011, 1, 12)
>>> datetime.datetime.now()
datetime.datetime(2011, 1, 12, 20, 36, 21, 430985)

Adding time and timedelta:
>>> datetime.timedelta(hours=7) + datetime.datetime(2011,1,1)
datetime.datetime(2011, 1, 1, 7, 0)
>>> datetime.timedelta(days=50) + datetime.date(2011,1,1)
datetime.date(2011, 2, 20)

Subtracting dates:
>>> datetime.date(2011,1,2) - datetime.date(2011,1,1)
datetime.timedelta(1)

Convert date and time into datetime:
>>> import datetime
>>> d = datetime.date(2010,01,05)
>>> t = datetime.time(10,25)

>>>datetime.datetime.combine(d,t)

datetime.datetime(2010, 1, 5, 10, 25)

Convert String to datetime:

>>> datetime.datetime.strptime("2011-01-05","%Y-%m-%d")

datetime.datetime(2011, 1, 5, 0, 0)

Convert datetime to individual components:

 >>> dt = datetime.datetime(2011,01,05,12,00)

>>> (dt.year, dt.month, dt.day, dt.hour, dt.minute)

(2011, 1, 5, 12, 0)

Convert datetime to date:

 >>> dt = datetime.datetime(2011,01,05,12,00)

>>> dt.date()

datetime.date(2011, 1, 5)

Add one day to a date:
>>> dt = datetime.datetime(2011,01,05,12,00)
>>> delta = datetime.timedelta(days=1)

>>> dt + delta

datetime.datetime(2011, 1, 6, 12, 0)

Number of days in a month:
>>> import calendar
>>> calendar.monthrange(2011,1)
(5, 31)

Timezone
from pytz import timezone as tz
localtz = tz('US/Central')
#dt_aware = localtz.localize(dt_unaware)
dt_unaware.replace(tzinfo=localtz)

more to come...

http://stackoverflow.com/questions/1937622/convert-date-to-datetime-in-python

http://www.linuxtopia.org/online_books/programming_books/python_programming/python_ch32s03.html

Pretty Print JSON with python

Pretty Print JSON
echo '{"foo": "lorem", "bar": "ipsum"}' | python -mjson.tool

Dump Django data as pretty JSON
./manage.py dumpdata mymodule | python -mjson.tool



Prety Print JSON from Python Shell
import json
json_data = [1,2,3,{'a','b',c'}]
json.dumps(json_data, sort_keys=True, indent=4)


http://stackoverflow.com/questions/352098/how-to-pretty-print-json-script
http://docs.python.org/library/json.html