Python Tokenlib

Make an expiring token using a known secret.

Source:
https://github.com/mozilla-services/tokenlib

Make a token, hashed with a secret, and storing an IP, that expires in 12 seconds
>>> t = tokenlib.make_token({"ip":"1.1.1.1."}, secret="1234567890", timeout=12)

Parse a token before expiration time is up
>>> tokenlib.parse_token(t, secret="1234567890")
{u'ip': u'1.1.1.1.', u'expires': 1428098130.01954, u'salt': u'fbd08e'}

Parse a token after expiration time is up
>>> tokenlib.parse_token(t, secret="1234567890")
Traceback (most recent call last):
  File "", line 1, in
  File "tokenlib/__init__.py", line 186, in parse_token
    return TokenManager(**kwds).parse_token(token, now=now)
  File "/tokenlib/__init__.py", line 143, in parse_token
    raise errors.ExpiredTokenError()
tokenlib.errors.ExpiredTokenError: token has expired

Simple Python Http Server

Serves up files in the current directory.

Python 3
python3 -m http.server
or
python3 -m http.server 8080

Python 2
python -m SimpleHTTPServer
or 
python -m SimpleHTTPServer 8080

Source:
http://www.reddit.com/r/Python/comments/2ysd91/what_are_some_nifty_python_snippets_that_you_have/

Nodejs Hue Quicksheet

Summary of the following article:
https://github.com/peter-murray/node-hue-api

// load the module
var hue = require('node-hue-api');

// define some display functions
var displayResult = function(result) {
    console.log("Created user: " + JSON.stringify(result));
};

// define some display functions
var displayError = function(err) {
    console.error(err);
};

// search for bridges

timeout = 6000  // 6 seconds

hue.upnpSearch(timeout).then(displayBridges).done();

// register a new user

hostname = '192.168.1.112';

newUserName = null;

userDescription = "some description";

hue.registerUser(hostname, newUserName, userDescription)

   .then(displayResult).fail(displayError).done();

// define connection information
var hostname = "192.168.1.112",
    username="*************************",
    api;

// create the api object
api = new hue.HueApi(hostname, username);

// get basic information about the gateway api
api.description().then(displayResult).fail(displayError).done();

// get the full state of the gateway
api.fullState().then(displayResult).fail(displayError).done();

// see the users that are registered
api.registeredUsers().then(displayResult).fail(displayError).done();

// show the lights and their states
api.lights().then(displayResult).fail(displayError).done();

// define a state and set it to light #1
state = hue.lightState.create().on().white(500, 100);
api.setLightState(1, state).then(displayResult).fail(displayError).done();

// turn light #1 on
api.setLightState(1, state.on()).then(displayResult).fail(displayError).done();

// turn light #1 off
api.setLightState(1, state.off()).then(displayResult).fail(displayError).done();

// get the status of light #1
api.lightStatus(1).then(displayResult).fail(displayError).done();

// see the light groups 
api.groups().then(displayResult).fail(displayError).done();

// see the available scenes  
api.scenes().then(displayResult).fail(displayError).done();

// create a new scene assign light #1-7 to it.  Will include a scene 'id' in the result

lightIds = [1,2,3,4,5,6,7]

sceneName = "Demo Scene"

api.createScene(lightIds, sceneName).then(displayResults).done();

// update an existing scene

lightIds = [1,2,3,4,5,6,]

sceneName = "Updated Demo Scene"

sceneId = "id-generated-by-gateway"

api.updateScene(sceneId, lightIds, sceneName).then(displayResults).done();

// set states for light #1 in a scene

state = lightState.create().on().hue(2000)

sceneId = "id-generated-by-gateway"

lightId = 1

api.setSceneLightState(sceneId, lightId, state).then(displayResults).done();

// recall scene
api.recallScene(sceneId).then(displayResults).done();

Docker Elastisearch

Simple Configuration

1) Create a directory for local data:

mkdir `pwd`/data-dir

2) Create an elasticsearch config inside of the local data dir:

vim data-dir/elasticsearch.yml

path:

  logs: /data/log 

  data: /data/data

3) Create a script as such:

CONTAINER_NAME="elasticsearch_server"
ELASTIC_DATA_DIR="`pwd`/data-dir"
HOST_PORT_1="9200"
HOST_PORT_2="9300"

if [ ! -d "${ELASTIC_DATA_DIR}" ]
then
    mkdir -p "${ELASTIC_DATA_DIR}"
fi

docker rm ${CONTAINER_NAME}
sudo docker run -it --name ${CONTAINER_NAME} --rm \
    -p ${HOST_PORT_1}:9200 \
    -p ${HOST_PORT_2}:9300 \
    -v ${ELASTIC_DATA_DIR}:/data \
    dockerfile/elasticsearch \
    /elasticsearch/bin/elasticsearch -Des.config=/data/elasticsearch.yml

Source:
https://github.com/dockerfile/elasticsearch

Postgres As Docker

CONTAINER_NAME="postgres_server"
CONTAINER_DATA_DIR="/var/lib/postgresql/data"
HOST_PORT="55432"
POSTGRES_DOCKER_DIR=`pwd`
HOST_DATA_DIR="${POSTGRES_DOCKER_DIR}/data-dir/"

if [ ! -d "${HOST_DATA_DIR}" ]
then
    mkdir -p "${HOST_DATA_DIR}"
fi

sudo docker run --rm --name ${CONTAINER_NAME} \
            -v ${HOST_DATA_DIR}:${CONTAINER_DATA_DIR} \
            -p ${HOST_PORT}:5432 \
            -it postgres $1

Ubuntu: Apt find all outdated packages

apt-get --just-print upgrade | grep "Inst"|  while read a b c d;
do echo "\"$a\",\"$b\",\"$c\",\"$d\"";
done

Git Copy Branches Locally

Source: https://coderwall.com/p/0ypmka

#!/bin/bash for branch in `git branch -a | grep remotes | grep -v HEAD | grep -v master `; do git branch --track ${branch#remotes/origin/} $branch done

Django Annotation: Users with duplicate emails

User.objects.values("email").annotate(num_emails=Count("email")
                    ).order_by("-num_emails").filter(
                    num_emails__count__gt=1):

Reference:
http://stackoverflow.com/questions/8989221/django-select-only-rows-with-duplicate-field-values

Docker Notes

CASE STUDY

# see the list of docker images as a tree view. Determine IMAGE_ID
sudo docker images --tree

# get list of currently running containers. Determine CONTAINER_ID
sudo docker ps

# remove a docker image tree
sudo docker rmi IMAGE_ID/TAG

# get list of all containers
sudo docker ps -a

# get last executed container info
sudo docker ps -al

# get all containers without the command being truncated
sudo docker ps -a --no-trunc

# commit changes to a docker container and tag the commit
sudo docker commit CONTAINER_ID feature/nginx

# tag an image without running
sudo docker tag IMAGE_ID TAG_NAME

# run a tag or IMAGE_ID as an interactive bash shell
sudo docker run -t -i -p 80 feature/nginx /bin/bash

# remove the container after running
sudo docker run -rm ….

# name a container
sudo docker run --name …

# run a tag or IMAGE_ID as an interactive ssh process
sudo docker run -t -i -p 22 -p 80 feature/nginx /usr/sbin/sshd -D

# run a tag or IMAGE_ID as a background ssh process
sudo docker run -d -p 22 -p 80 feature/nginx /usr/sbin/sshd -D

# view docker logs
sudo docker logs CONTAINER_ID/NAME
sudo docker logs -f CONTAINER_ID/NAME  # follow

# stop a running container in the background or foreground
sudo docker stop CONTAINER_ID/NAME

# build and tag an image using a Dockerfile
docker build -t tag/name .

# run a docker process and open up ports 22 and 80 on the docker instance. The host ports are generated automatically.
sudo docker run -d -p 22 -p 80 feature/nginx /usr/bin/supervisord -n

# Get port on the host that is NATed to the given port on the image
sudo docker port CONTAINER_ID 22
sudo docker port CONTAINER_ID 80

# run a docker process and map host port 8888 to docker instance port 80 and host port 2222 to docker instance port 22
sudo docker run -d -p 2222:22 -p 8888:80 feature/nginx /usr/bin/supervisord -n

# Same as above but additionally creates a shared folder between the host and guest
sudo docker run -d -p 2222:22 -p 8888:80 -v /home/:/home/:rw feature/nginx /usr/bin/supervisord -n

# clean up extra containers
sudo docker rm `sudo docker ps -a | grep Exit | awk '{ print $1}'`
sudo docker rm $(sudo docker ps -a -q)

# NEW: clean up exited containers 
sudo docker ps -a -q --filter status=exited | xargs docker rm

# NEW: clean up untagged images
docker images -q --filter "dangling=true" | xargs docker rmi

# delete all untagged images
sudo docker rmi $( sudo docker images | grep '' | tr -s ' ' | cut -d ' ' -f 3)

# In Ubuntu, change the docker daemon listen address for Shipyard
DOCKER_OPTS="-H tcp://127.0.0.1:4243 -H unix:///var/run/docker.sock"
service restart docker

# images files stored in
/var/lib/docker/graph/

# get the host gateway from within the container
netstat -nr | grep '^0\.0\.0\.0' | awk '{print $2}'

# find metatdata bout the container
sudo docker inspect CONTAINER_ID/NAME

# Setup shipyard-agent
curl https://github.com/shipyard/shipyard-agent/releases/download//shipyard-agent -L -o /usr/local/bin/shipyard-agent
chmod +x /usr/local/bin/shipyard-agent

shipyard-agent -url http://myshipyardhost:shipyardport -register # register with shipyard
shipyard-agent -url http://myshipyardhost:shipyardport -key 1234567890qwertyuiop # run the agent


SETUP AN IMAGE
sudo aptitude install nginx vim openssh-server supervisord psmisc

vim /etc/supervisor/conf.d/nginx.conf
[program:nginx]
command=/usr/sbin/nginx -c /etc/nginx/nginx.conf
directory=/etc/nginx
user=root
autostart=true
autorestart=true
redirect_stderr=True

vim /etc/supervisor/conf.d/ssh.conf
[program:sshd]
command=/usr/sbin/sshd -D


FLATTEN IMAGE
http://www.hnwatcher.com/r/650964/Slimming-down-Docker-containers-Intercity-Blog

CONTAINER_ID=$(docker run -d image-name /bin/bash)
docker export $CONTAINER_ID | docker import – flat-image-name

CONTAINER_ID=$(docker run -d image-name /bin/bash)
(docker export $CONTAINER_ID | gzip -c > image.tgz)
gzip -dc image.tgz | docker import - flat-image-name


OTHER
lxc-attach
--links option
--env-file
https://gist.github.com/wsargent/7049221#images

Remove images article
http://mikehadlow.blogspot.com/2014/04/docker-bulk-remove-images-and-containers.html

Linking Containers
http://docs.docker.com/userguide/dockerlinks/

Dockerfile Reference
http://docs.docker.com/reference/builder/

FLYNN - Open Source PaaS
https://github.com/flynn/flynn

ETCD - Configuration Tool
https://github.com/coreos/etcd

Ubuntu 14.04 / Centos 7 Postgres 9.3 and Postgis

CENTOS 7

Reference:

http://www.postgresonline.com/journal/archives/329-An-almost-idiots-guide-to-install-PostgreSQL-9.3,-PostGIS-2.1-and-pgRouting-with-Yum.html

Pick a release from here:

   http://yum.postgresql.org/repopackages.php

Add the following postgres repo:

   sudo rpm -ivh  http://yum.postgresql.org/9.3/redhat/rhel-7-x86_64/pgdg-centos93-9.3-1.noarch.rpm

Install ifconfig:

   sudo yum install net-tools

Install packages:

   sudo yum install postgresql93 postgresql93-server postgresql93-libs postgresql93-contrib postgresql93-devel

Install needed epel repo:

   sudo rpm -ivh http://dl.fedoraproject.org/pub/epel/beta/7/x86_64/epel-release-7-0.2.noarch.rpm

Install postgis and pgrouting extensions:

   sudo yum install postgis2_93

   sudo yum install pgrouting_93

Edit pg_hba.conf to allow network users to login:

   vim /var/lib/pgsql/9.3/data/pg_hba.conf

   # add the following

   host all all  0.0.0.0/0   md5

Edit the postgresql.conf to allow hosts to connect from anywhere:

   vim /var/lib/pgsql/9.3/data/postgresql.conf

   # change the listen address

   listen_addresses = '*'

Init the database:

   /usr/pgsql-9.3/bin/postgresql93-setup initdb

Restart the service:

   sudo systemctl restart  postgresql-9.3.service

Get status:

   sudo service postgresql-9.3 status

Create a database:

   [vagrant@localhost ~]$ sudo -u postgres psql

   postgres=# create user testa with password '1234';

   CREATE ROLE

   postgres=# create database testa with owner testa;

   CREATE DATABASE

   postgres=# \c testa

   You are now connected to database "testa" as user "postgres".

   testa=# create extension postgis;

   CREATE EXTENSION

   testa=# create extension postgis_topology;

   CREATE EXTENSION

   testa=# SELECT postgis_full_version();

    POSTGIS="2.1.3 r12547" GEOS="3.4.2-CAPI-1.8.2 r3921" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.11.0, released

             2014/04/16" LIBXML="2.9.1" LIBJSON="UNKNOWN" TOPOLOGY RASTER

UBUNTU 14.04

Ubuntu Installation:

   sudo aptitude update

   sudo apt-get install build-essential postgresql-9.3 postgresql-server-dev-9.3 libxml2-dev libgdal-dev libproj-dev libjson0-dev xsltproc docbook-xsl docbook-mathml

   sudo aptitude install postgresql-9.3-postgis-2.1

Edit pg_hba.conf to allow network users to login:

   vim /etc/postgresql/9.3/main/pg_hba.conf

   # add the following

   host all all  0.0.0.0/0   md5

Edit the postgresql.conf to allow hosts to connect from anywhere:

   vim /etc/postgresql/9.3/main/postgresql.conf

   # change the listen address

   listen_addresses = '*'

Restart the service:

   sudo service postgresql restart

Create a database:

   [vagrant@localhost ~]$ sudo -u postgres psql

   postgres=# create user testa with password '1234';

   CREATE ROLE

   postgres=# create database testa with owner testa;

   CREATE DATABASE

   postgres=# \c testa

   You are now connected to database "testa" as user "postgres".

   testa=# create extension postgis;

   CREATE EXTENSION

   testa=# create extension postgis_topology;

   CREATE EXTENSION

   testa=# SELECT postgis_full_version();

    POSTGIS="2.1.3 r12547" GEOS="3.4.2-CAPI-1.8.2 r3921" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.11.0, released

             2014/04/16" LIBXML="2.9.1" LIBJSON="UNKNOWN" TOPOLOGY RASTER

vim: get full path of current file

Get the full path of the current file
:echo expand('%:p')


http://vim.wikia.com/wiki/Get_the_name_of_the_current_file

Upstart Basic Commands

Disables Nginx so that dpkg does not re-enable it when Nginx is upgraded:

sudo update-rc.d -f nginx disable

Add Nginx to default runlevels:
update-rc.d nginx defaults

Disable Service

sudo sh -c "echo 'manual' > /etc/init/nginx.override"

Manual Service Control
sudo service nginx start

sudo service nginx restart

sudo service nginx stop

Or

sudo start nginx

sudo stop nginx
sudo status nginx


http://askubuntu.com/questions/177041/nginx-disable-autostart

Creating differential backups with hard links and rsync

Creating differential backups with hard links and rsync:


From above:
backup@backup_server> DAY2=`date +%Y%m%d%H%M%S` 

backup@backup_server> cp -al /var/backups/$DAY1 /var/backups/$DAY2 

backup@backup_server> rsync -av -e ssh --delete earl@192.168.1.20:/home/earl/ /var/backups/$DAY2/

Find IP addresses with a lot of open connections:
netstat -atun | awk '{print $5}' | cut -d: -f1 | sed -e '/^$/d' |sort | uniq -c | sort -n

Source:
http://www.cyberciti.biz/tips/netstat-command-tutorial-examples.html

LSOF Quick Commands

http://stackoverflow.com/questions/106234/lsof-survival-guide

http://www.akadia.com/services/lsof_quickstart.txt

Bzip2 PGDump

To backup:

pg_dump -hlocalhost -Upostgres -a -t table database|bzip2 > tabledata.bz2

To restore:
bzip2 -c -d tabledata.bz2 | psql -hlocalhost -Upostgres database


Source: http://www.solomonson.com/content/using-bzip-pgdump

AB Testing parameters

ab -e out.csv -k -n 5000 -c 100 http://example.com/

-e = write to csv file
-k = enable http keepalive
-n = total number of requests to make
-c = number of concurrent requests
-i = do HEAD requests instead of GET
-A  auth-username:password 


Benchmarking example.com (be patient)
Completed 500 requests
Completed 1000 requests
Completed 1500 requests
Completed 2000 requests
Completed 2500 requests
Completed 3000 requests
Completed 3500 requests
Completed 4000 requests
Completed 4500 requests
Completed 5000 requests
Finished 5000 requests


Server Software:        nginx/1.1.19
Server Hostname:        example.com
Server Port:            80

Document Path:          /
Document Length:        185 bytes

Concurrency Level:      100
Time taken for tests:   8.359 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Non-2xx responses:      5003
Keep-Alive requests:    5000
Total transferred:      1951170 bytes
HTML transferred:       925555 bytes
Requests per second:    598.19 [#/sec] (mean)
Time per request:       167.172 [ms] (mean)
Time per request:       1.672 [ms] (mean, across all concurrent requests)
Transfer rate:          227.96 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    4  31.1      0     228
Processing:    38  158 239.9     96    3477
Waiting:       38  158 239.9     96    3477
Total:         38  163 253.6     96    3477

Percentage of the requests served within a certain time (ms)
  50%     96
  66%    120
  75%    145
  80%    159
  90%    237
  95%    434
  98%    951
  99%   1339
 100%   3477 (longest request)

Python Crontab Simple usage

pip install python-dateutil python-crontab

from crontab import CronTab
user_cron = CronTab('myuser')
command = "/usr/bin/ls -la"
comment = "Run"
job = user_cron.new(command=command, comment=comment)
job.minute.every(5)
job.is_valid()
user_cron.write("outfile.tab")

Redhat/Debian Update Support

Preventing Kernel Updates in Linux
Ubuntu/Debian
http://www.howtogeek.com/howto/10606/how-to-hide-kernel-updates-in-ubuntu/

RedHat
http://www.howtogeek.com/50898/how-to-prevent-yum-from-updating-the-kernel/

List Available updates in Linux
Debian
aptitude search '?upgradable'

Redhat
sudo yum list updates
http://yum.baseurl.org/wiki/YumCommands


Automatic Security Updates
Ubuntu
https://help.ubuntu.com/community/AutomaticSecurityUpdates


Download only packages without installing
Redhat
yum install -y yum-downloadonly
yum install httpd -y --downloadonly --downloaddir=/my/download/dir/
http://www.cyberciti.biz/faq/yum-downloadonly-plugin/

Ubuntu
apt-get -d, --download-only ...

Screenshots from Python and Phantomjs

Copied from Pykler's Stackoverflow answer:
http://stackoverflow.com/questions/13287490/is-there-a-way-to-use-phantomjs-in-python

1. Install NodeJS
2. npm -g install phantomjs
3. pip install selenium 

from selenium import webdriver
driver = webdriver.PhantomJS()
driver.set_window_size(1024, 768) # optional
driver.get('https://google.com/')
driver.save_screenshot('screen.png') # save a screenshot to disk
sbtn = driver.find_element_by_css_selector('button.gbqfba')
sbtn.click()