AB Testing parameters

ab -e out.csv -k -n 5000 -c 100 http://example.com/

-e = write to csv file
-k = enable http keepalive
-n = total number of requests to make
-c = number of concurrent requests
-i = do HEAD requests instead of GET
-A  auth-username:password 


Benchmarking example.com (be patient)
Completed 500 requests
Completed 1000 requests
Completed 1500 requests
Completed 2000 requests
Completed 2500 requests
Completed 3000 requests
Completed 3500 requests
Completed 4000 requests
Completed 4500 requests
Completed 5000 requests
Finished 5000 requests


Server Software:        nginx/1.1.19
Server Hostname:        example.com
Server Port:            80

Document Path:          /
Document Length:        185 bytes

Concurrency Level:      100
Time taken for tests:   8.359 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Non-2xx responses:      5003
Keep-Alive requests:    5000
Total transferred:      1951170 bytes
HTML transferred:       925555 bytes
Requests per second:    598.19 [#/sec] (mean)
Time per request:       167.172 [ms] (mean)
Time per request:       1.672 [ms] (mean, across all concurrent requests)
Transfer rate:          227.96 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    4  31.1      0     228
Processing:    38  158 239.9     96    3477
Waiting:       38  158 239.9     96    3477
Total:         38  163 253.6     96    3477

Percentage of the requests served within a certain time (ms)
  50%     96
  66%    120
  75%    145
  80%    159
  90%    237
  95%    434
  98%    951
  99%   1339
 100%   3477 (longest request)

Python Crontab Simple usage

pip install python-dateutil python-crontab

from crontab import CronTab
user_cron = CronTab('myuser')
command = "/usr/bin/ls -la"
comment = "Run"
job = user_cron.new(command=command, comment=comment)
job.minute.every(5)
job.is_valid()
user_cron.write("outfile.tab")

Redhat/Debian Update Support

Preventing Kernel Updates in Linux
Ubuntu/Debian
http://www.howtogeek.com/howto/10606/how-to-hide-kernel-updates-in-ubuntu/

RedHat
http://www.howtogeek.com/50898/how-to-prevent-yum-from-updating-the-kernel/

List Available updates in Linux
Debian
aptitude search '?upgradable'

Redhat
sudo yum list updates
http://yum.baseurl.org/wiki/YumCommands


Automatic Security Updates
Ubuntu
https://help.ubuntu.com/community/AutomaticSecurityUpdates


Download only packages without installing
Redhat
yum install -y yum-downloadonly
yum install httpd -y --downloadonly --downloaddir=/my/download/dir/
http://www.cyberciti.biz/faq/yum-downloadonly-plugin/

Ubuntu
apt-get -d, --download-only ...

Screenshots from Python and Phantomjs

Copied from Pykler's Stackoverflow answer:
http://stackoverflow.com/questions/13287490/is-there-a-way-to-use-phantomjs-in-python

1. Install NodeJS
2. npm -g install phantomjs
3. pip install selenium 

from selenium import webdriver
driver = webdriver.PhantomJS()
driver.set_window_size(1024, 768) # optional
driver.get('https://google.com/')
driver.save_screenshot('screen.png') # save a screenshot to disk
sbtn = driver.find_element_by_css_selector('button.gbqfba')
sbtn.click()

Supervisord Common Commands

Invoke the supervisor shell:
sudo supervisorctl

# all of the following commands can be run within the supervisor shell or from the UNIX CLI

View processes managed by supervisord:
sudo supervisorctl status

Start/Stop/Restart processes from the UNIX command line:
sudo supervisorctl start processname
sudo supervisorctl stop processname
sudo supervisorctl restart processname

View standard error:
supervisorctl tail -f  processname stderr

Load changes to configuration without restarting unaffected processes:
supervisorctl reread  # reads the configuration changes
supervisorctl update  # restarts changed process groups

Reload supervisor processes and configurations, but doesn't restart supervisor itself:
supervisorctl reload

Sources:
http://stackoverflow.com/questions/3792081/will-reloading-supervisord-cause-the-process-under-its-to-stop

Postgres Configuration on RedHat

This will install Postgresql on a RedHat 6.4 or CentOS 6.4 system and will allow password-protected remote login access. 

# Open the firewall to allow remote postgres connections
sudo /sbin/iptables -A INPUT -p tcp --dport 5432 -j ACCEPT

sudo /sbin/service iptables save
sudo /sbin/service iptables restart

# Install packages through yum
sudo yum install vim-enhanced
sudo yum install postgresql-server

# Initialize a new database directory
sudo service postgresql initdb 

# Start the database server
sudo service postgresql start 

# Add postgres to the default runlevel 
sudo chkconfig postgresql on
sudo chkconfig postgresql --list
sudo chmod 755 /home/idcuser/

# Edit the postgres.conf to allow connections from all ips

sudo vim /var/lib/pgsql/data/postgres.conf
listen_addresses="*"

# Edit the pg_hba.conf file to allow password auth for any ip

sudo vim /var/lib/pgsql/data/pg_hba.conf
host all all 0.0.0.0/0 md5

# Restart postgres
sudo /etc/init.d/postgres restart

# Configure a postgresql superuser and new schema. 
sudo -u postgres psql
psql> create role idcuser superuser createdb createrole inherit login;
psql> alter user idcuser with password 'password';
psql> create database sample;
psql> \c sample
psql> create schema sample;
psql> alter user idcuser set search_path to sample,public;

Change Keybindings for Emacs-like Editing

Move left-ctrl to left-alt

Contents of .Xmodmap

clear Mod1

clear Control 

clear Mod2 

keycode 64 = Control_L Control_L Control_L Control_L 

keycode 37 = Alt_L Alt_L Alt_L Alt_L 

keycode 108 = Alt_R Alt_R Alt_R Alt_R 

keycode 127 = Num_Lock 

add Mod1 = Alt_L Alt_R 

add Control = Control_L Control_R 

add Mod2 = Num_Lock

Make effective without loging in
xmodmap ~/.Xmodmap

Save current keybindings
xmodmap -pke > ~/.Xmodmap-original

Source:

Dustin Lacewell http://ldlework.com/

iptables

Listing Rules
# Rules are evaluated from top to bottom
# list the iptable rules
iptables -list
iptables -L

# list the rules with high verbosity 
iptables -L -v

# list the rules with high verbosity and line numbers
iptables -L -v --line-numbers

# list the rules with high verbosity and line numbers and show the raw IP addresses and ports
iptables -L -v --line-numbers -n

# list rules as a list of rules in the conf file.
iptables -S

Adding/Editing/Removing Rules 
# -I Option: Insert a rule at a specific point (to be the 4th rule). 
iptables -I INPUT 4 -p tcp  --dport 3306 -m state --state NEW,ESTABLISHED -j ACCEPT

# -A Option: Append a rule to the end of the rule list
iptables -A INPUT -p tcp  --dport 3306 -m state --state NEW,ESTABLISHED -j ACCEPT

# -D Option: Delete the INPUT rule at line 4
iptables -D INPUT 4

# -R Option: Replace a rule at 4th line
iptables -R INPUT 4 -p tcp  --dport 3306 -m state --state NEW,ESTABLISHED -j ACCEPT

Change the iptables policy

# Change the policy to drop all traffic if no rules matched.  
# Important: You'll want to setup an ssh rule before doing this
iptables -P INPUT DROP

# Change the policy to accept all traffic unless specific rules are matched
iptables -P INPUT ACCEPT

IPTables modules

# Modules are loaded using the -m flag
# Some common examples include: "-m tcp", "-m mac", and "-m state"
# Many are loaded by default. For example, if you specify "-p tcp --dport 22", I think it automatically loads the tcp module and assumes "-m tcp".

# List Loaded IPTables modules
cat /proc/net/ip_tables_matches

Jump option parameters (-j option) "jump to the specified target"
  ACCEPT - Accept the packet and stop processing rules
  REJECT - reject the packet and notify sender
  DROP - silently drop the packet and stop processing rules
  LOG - log the packet and continue processing

State option parameters (--cstate and --state options) 
   NEW - new connection that has not been seen
   RELATED - new connection but related to another connection
   ESTABLISHED - connection is already established
   INVALID - traffic couldn't be identified

Save the firewall rules on a RedHat system
/sbin/service iptables save

Clear rule statistics counters
iptables -Z

# good howto for Ubuntu
https://help.ubuntu.com/community/IptablesHowTo

# A good howto for Centos and DROP policy
http://wiki.centos.org/HowTos/Network/IPTables

Ping with Timestamp

Ping an IP and print the time:
ping 198.178.132.10 | perl -nle 'print scalar(localtime), " ", $_'


http://stackoverflow.com/questions/10679807/how-to-timestamp-every-ping-result

Synergy Linux Setup

Install
sudo aptitude install synergy

Given the following computers:
computerA  (Synergy server)
computerB  (Synergy client)

And given: 
computerA is the name returned by the 'hostname' command on the server; ip is 192.168.0.102
computerB is an alias (setup in synergy.conf) to 192.168.0.101


Edit /etc/synergy.conf on the Synergy Server
section: screens
    computerA:
    computerB:
end

# we might not want to go through the effort of setting up DNS for computerB, so we can alias here instead.
section: aliases
    computerB:
        192.168.0.101
end

section: links
    computerA:
       right = computerB
    computerB:
       left = computerA
end

On the server, run the following: 
 synergys -f --config /etc/synergy.conf

On the client, run the following: 

synergyc -f 192.168.0.102

http://www.mattcutts.com/blog/how-to-configure-synergy-in-six-steps/

launchctl / launchd on OSX

Launch definitions are in the form of plist documents

List jobs managed by launchctl (shows job label)
launchctl list

Show detail information about launchctl managed job
launchctl list job_label
i.e. launchctl list homebrew.mxcl.postgresql
# output will look like:
{
"Label" = "homebrew.mxcl.postgresql";
"LimitLoadToSessionType" = "Aqua";
"OnDemand" = false;
"LastExitStatus" = 0;
"PID" = 17312;
"TimeOut" = 30;
"StandardErrorPath" = "/usr/local/var/log/postgres/stderror.log";
"ProgramArguments" = (
"/opt/homebrew/bin/postgres";
"-D";
"/usr/local/var/postgres";
"-r";
"/usr/local/var/log/postgres/server.log";
);

};

Stop a job managed by launchctl
launchctl stop job_label

Good tool for making launchctl plists
http://sourceforge.net/projects/lingon/

Plists located in one of the following places
FILES

 ~/Library/LaunchAgents -  Per-user agents provided by the user. 

 /Library/LaunchAgents   -  Per-user agents provided by the administrator. 

 /Library/LaunchDaemons  -  System-wide daemons provided by the administrator. /System/Library/LaunchAgents -  Per-user agents provided by OS X.

 /System/Library/LaunchDaemons -  System-wide daemons provided by OS X.


https://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man5/launchd.plist.5.html

DB2 command reference

Invoke interpreter:
shell> db2

List available databases:
db2_shell> list database directory

List DB2 shell history:
db2_shell> history
# or
db2_shell> h

Edit and Execute command from DB2 shell history:
# command_num is from the above history command
db2_shell> edit command_num
# or
db2_shell> e command_num

Connect to database from within the db2 interpreter (CLP):
db2_shell> connect to databasename

Connect to database from UNIX shell:
# lasts for duration of current shell session.
db2 connect to database_name

Run SQL command from UNIX shell:
# first connect to database via UNIX shell (above)
# then run the following
db2 -vf scriptname.sql

List tables in database:
db2_shell> list tables

List indexes on a table:
SELECT indschema,INDNAME from syscat.indexes where tabname = 'MYTABLE' and tabschema = 'MYSCHEMA'

Analyze table:
runstats on table schema.tablename

List connections to database:
unix> db2 list applications
# or
db2_shell> list applications
# for detail information
unix> db2 list applications show detail

Get current user:
SELECT CURRENT USER FROM SYSIBM.SYSDUMMY1

Disconnect all connections:
force applications all
#or 
db2 terminate # this might work


Drop database:
drop database databasename

Create database:
create database databasename

Create the explain tables:
connect to databasename
db2 -vtf /opt/ibm/db2/V9.7/misc/EXPLAIN.DDL

Stop DB2:
db2stop

Start DB2: 
db2start

Allow TCP connections (probably unsafe):
update database manager configuration using svcename 3700
get database manager configuration    # verify change
db2set DB2COMM=tcpip
db2stop

db2start
sudo /sbin/iptables -A INPUT -p tcp --dport 7300 -j ACCEPT

sudo /sbin/service iptables save

sudo /sbin/service iptables restart

Update config params:
update db cfg for database_name using logfilsiz 8000
 # or
update database configuration for database_name using logprimary 9 logfilsiz 8000
force applications all

Sources:
http://www.tek-tips.com/viewthread.cfm?qid=128876

NewRelic quick Django setup

pip install newrelic

newrelic-admin generate-config LICENSE-KEY newrelic.ini

# add your key to generated config

NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program python manage.py run_gunicorn

or

NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program python manage.py runserver

https://newrelic.com/docs/python/python-agent-quick-start

Postgres Access

In pg_hba.conf, set the following: 

To allow all remote users to log into a postgres database:

In pg_hba.conf, set the following: 

# remote hosts    Database    User      From any IP       Using PG md5 auth

host                      all                 all         0.0.0.0/0            md5

In the postgres.conf, set the following:
listen_addresses = '*'

To allow local UNIX users with the same name as PG users to auth automatically:
local                     all                 all           trust

Redis

Login to Redis:
redis-cli

Login to Remote Redis:
redis-cli -h 1.2.3.4
auth mypassword

Switch database:
select 1  # where one is the database name

List keys in database:
redis> keys *

Persist everything to disk right away:
save

Get the location of the redis data dump dir:
config get dir
1) "dir"
2) "/var/lib/redis/"

Backup the Redis database datafile:
cp /var/lib/redis/dump.rdb /my/backup/location/

Drop all data in db:
flushdb

List of Installed Pacakges on Debian Based System

List packages:
dpkg --get-selections > outfile.txt

Reinstall:
dpkg --get-selections < outfile.txt

http://ubuntuforums.org/showthread.php?t=261366

Get IP Address from command line

wget -q -O - checkip.dyndns.org

http://go2linux.garron.me/what-is-my-public-ip-address-with-linux

Django/Web Modules

Modernizer - Feature Detection for JS
http://modernizr.com/

Django-flash - Rails-like django flash messages:
https://github.com/danielfm/django-flash

Facebook-Graph - Django facebook graph API
https://github.com/feinheit/django-facebook-graph

django-cumulus - Rackspace manager

http://packages.python.org/django-cumulus/

Django-ittybitty - Django based URL Shortener:

http://code.google.com/p/django-ittybitty/

django-ittybitty

Django Storages

http://django-storages.readthedocs.org/en/latest/
django-storages

Python-Postmark - API for http://postmarkapp.com/

https://github.com/themartorana/python-postmark

Django Cache Utils

http://pypi.python.org/pypi/django-cache-utils

stripe-python - API for Stripe.com (like Braintree)

https://github.com/stripe/stripe-python

FrogBugz - online ticket tracking system

http://www.fogcreek.com/fogbugz/pricing.html

Form Builder
http://www.wufoo.com

Factual - service for data information and places. GeoData
http://v2.factual.com/

CartoDB - open source data browser db
https://github.com/Vizzuality/cartodb

GeoCoda - Online Geocoding tool
https://geocoda.com/

Census Tiger Dataset
http://www.census.gov

SendGrid - Email Web applications
http://sendgrid.com/

RapidSSL - Cheap SSL Host

django-tables2 - manipulate html tables programmatically similar to Django forms.
http://django-tables2.readthedocs.org/en/latest/index.html

django-message-extends - extensions to Django messages such as sticky messages and assigned messages
https://github.com/AliLozano/django-messages-extends

Simple Logging

Simple Logging

In someplace like python shell, __init__.py, or settings.py (pre Django 1.3):

import logging
logger = logging.getLogger('mymodule.test')
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s - %(name)s - %(lineno)s - %(levelname)s - %(message)s")

# Configure log to file
filelog = logging.FileHandler("mylogfile.log", 'a')
filelog.setLevel(logging.INFO)
filelog.setFormatter(formatter)
logger.addHandler(filelog)

# Configure log to stdout
conlog = logging.StreamHandler()
conlog.setLevel(logging.DEBUG)
conlog.setFormatter(formatter)
logger.addHandler(conlog)

In place you want to log in, like views.py:

import logging
logger = logging.getLogger('mymodule.test')
logger.debug("My Log text")
logger.info("My info text")

Another example of simple logging:

import logging
logger = logging.getLogger('mymodulename')
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
logger.addHandler(ch)

---------------------------------------------------

A simple example:

import logging
logging.basicConfig(filename='test.log', level=logging.DEBUG)
logging.info("INFO")
logging.debug("DEBUG")
logging.error("ERROR")


Sources:
http://dancingpenguinsoflight.com/2009/03/simple-and-effective-python-logging/
http://www.blog.pythonlibrary.org/2012/08/02/python-101-an-intro-to-logging/

Dulwich Command Notes

# Load Git repo into object

from dulwich.repo import Repo

repo = Repo("/home/jjasinsk/Sites/env/proj/gitdjango/")

# Find sha hash of Git repo head

repo.head()

# 'e80b3ada92d2a02ce353c83683a6533e25240a1a'

# Find the Commit object of the repo head

repo[repo.head()]

# Commit e80b3ada92d2a02ce353c83683a6533e25240a1a

# Find all the repo tags

repo.refs.subkeys('refs/tags')

# set(['v0.0.1'])

# Find all the repo branches

repo.refs.subkeys('refs/heads')

# set(['master', 'develop'])

# Find all the repo refs and their sha hashes

repo.get_refs()

# {'HEAD': 'e80b3ada92d2a02ce353c83683a6533e25240a1a', 

   'refs/heads/develop': 'b53fe83f58f293decae39a70106a5ae78886ba5e', 

   'refs/tags/v0.0.1': '5554750c44f8dc17ab9dc5926129cf409d06d0bb', 

   'refs/heads/master': 'e80b3ada92d2a02ce353c83683a6533e25240a1a'}

# Find all the commits tracing back to head Commit 

repo.revision_history(repo.head())

# [Commit fcc1e1eb645f6731c4a8701b5facf852738850a9, 

    Commit 39eaa48156e3df2fd2cbe9922e33787cbafdad90, 

    Commit e80b3ada92d2a02ce353c83683a6533e25240a1a,

   ...

TAG OBJECTS

# Get the Tag object by looking up it's ref path
repo['refs/tags/v0.0.1']

# Tag 5554750c44f8dc17ab9dc5926129cf409d06d0b;

# Find Tag by sha hash

repo.get_object('5554750c44f8dc17ab9dc5926129cf409d06d0bb')

# Tag 5554750c44f8dc17ab9dc5926129cf409d06d0bb

repo.tag("5554750c44f8dc17ab9dc5926129cf409d06d0bb")

# Tag 5554750c44f8dc17ab9dc5926129cf409d06d0bb

repo['5554750c44f8dc17ab9dc5926129cf409d06d0bb']

# Tag 5554750c44f8dc17ab9dc5926129cf409d06d0bb

# Find information about a tag

t = repo['5554750c44f8dc17ab9dc5926129cf409d06d0bb']

# Find tagger 
t.tagger

# 'Joe Jasinski joe.jasinski@example.com'

# Find tag date

from datetime import datetime

datetime.utcfromtimestamp(t.tag_time)

datetime.datetime(2012, 3, 30, 6, 58, 53)

# Print tag contents

print t.as_pretty_string()

# ...

# Find Tag sha hash (id)

t.id

# '5554750c44f8dc17ab9dc5926129cf409d06d0bb'

   OR
s = t.sha()

s.hexdigest()

# '5554750c44f8dc17ab9dc5926129cf409d06d0bb'

# Find Tag message

t.message

# 'test tag\n'

# Find Tag Type

t.get_type()

# 4

COMMIT OBJECTS

# Find Commit object from ref spec
repo['refs/heads/master']

# Commit e80b3ada92d2a02ce353c83683a6533e25240a1a

# Look up commit several different ways

repo.get_object('b53fe83f58f293decae39a70106a5ae78886ba5e')

# Commit b53fe83f58f293decae39a70106a5ae78886ba5e

repo.commit("b53fe83f58f293decae39a70106a5ae78886ba5e")

# Commit b53fe83f58f293decae39a70106a5ae78886ba5e

repo['b53fe83f58f293decae39a70106a5ae78886ba5e']

# Commit b53fe83f58f293decae39a70106a5ae78886ba5e

# Find Commit hash 

repo.ref('refs/heads/master')

# 'e80b3ada92d2a02ce353c83683a6533e25240a1a'

# Find commit author 

repo['refs/heads/master'].author

# 'Joe Jasinski joe.jasinski@example.com'

# Find committer 

repo['refs/heads/master'].committer

# 'Joe Jasinski joe.jasinski@example.com'

# Find Commit message

repo['refs/heads/master'].message

# 'updated style\n'

# Find commit time

from datetime import datetime

datetime.utcfromtimestamp(repo['refs/heads/master'].commit_time)

# Find Trees associated with commit

repo['b53fe83f58f293decae39a70106a5ae78886ba5e'].tree

'b3437c8329fc3f6306936c7ca1986cefc1629862'

repo[c.tree].items()

[TreeEntry(path='.gitignore', mode=33188, sha='13609a506289eba22b61df5eab0f89b7aede323d'), 

TreeEntry(path='example', mode=16384, sha='f8e95712570c4316868ee9af32088324e8d3e498'), 

TreeEntry(path='gitdjango', mode=16384, sha='32b164fac85e5b40dd8d4265a0a8df765bf57300'), 

TreeEntry(path='manage.py', mode=33261, sha='2605e3768e2bab92ef0550d452036472e9a6a210'), 

...

 repo[repo[c.tree].items()[0].sha]

TREE OBJECTS

# Find Tree associated with hash 
repo.get_object('b3437c8329fc3f6306936c7ca1986cefc1629862')

# Tree b3437c8329fc3f6306936c7ca1986cefc1629862

# Find all items under a Tree

repo[b3437c8329fc3f6306936c7ca1986cefc1629862].items()

BLOB OBJECTS

# Find Blob associated with hash 

repo.get_blob("2605e3768e2bab92ef0550d452036472e9a6a210")

# Blob 2605e3768e2bab92ef0550d452036472e9a6a210

repo['2605e3768e2bab92ef0550d452036472e9a6a210']

# Blob 2605e3768e2bab92ef0550d452036472e9a6a210

MISC

# Iterate through all objects under a gree
for entry in repo.object_store.iter_tree_contents(repo['refs/heads/master'].tree):

    print entry

# TreeEntry(path='.gitignore', mode=33188, sha='13609a506289eba22b61df5eab0f89b7aede323d')

# TreeEntry(path='example/__init__.py', mode=33188, sha='e69de29bb2d1d6434b8b29ae775ad8c2e48c5391')

# TreeEntry(path='example/__init__.pyc', mode=33188, sha='ced988bb5f3f22f4f1956bf852f106b23eea5c09')

# ...

# Find a the hash of an object by specifying a filesystem path

from dulwich.object_store import tree_lookup_path  

(mode, hash) = tree_lookup_path(repo.get_object, repo['refs/heads/master'].tree, 'gitdjango/models.py')

# Find an object filehandle relative to inside the .git directory

repo.get_named_file('../requirements.pip')

# open file '/home/jjasinsk/Sites/env/proj/gitdjango/.git/../requirements.pip', mode 'rb' at 0x101cdf938

f.read()

# 'Django==1.4\nGitPython==0.3.2.RC1\nasync==0.6.1\ngitdb==0.5.4\ninclude-server==3.1-toolwhip.1\nsmmap==0.8.2\nwsgiref==0.1.2\n'