Django Manipulating the Session without a request

Django: Lookup a session without a request

    from django.conf import settings
    from importlib import import_module
    engine = import_module(settings.SESSION_ENGINE)
    session_store = engine.SessionStore("new_or_existing_sessionID")
    uid = session_store.get('_auth_user_id')

    try:
        del session_store['session_key']
    except IndexError:
        pass

    session_store.save()

GlusterFS Configuration

Assume node1 is: 192.168.33.10
Assume node2 is: 192.168.33.11

Sources:
https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers
http://www.jamescoyle.net/how-to/457-glusterfs-firewall-rules

On BOTH nodes

# set hosts file entry (or configure DNS)
vim /etc/hosts
192.168.33.10 gluster1
192.168.33.11 gluster2

# On Ubuntu: install requirements 
sudo add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.5
sudo apt-get update
sudo apt-get install glusterfs-server
sudo apt-get install glusterfs-client

# On Centos: install requirements

wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/RHEL/glusterfs-epel.repo

yum install glusterfs-server

yum install glusterfs-client

# make storage location for gluster data
sudo mkdir /gluster-storage

# make a mountpoint
sudo mkdir /mnt/media/

ON GLUSTER 1

# connect node 1 and node 2
sudo gluster peer probe gluster2
sudo gluster peer status

# create a Gluster replica volume
sudo gluster volume create volume1 replica 2 transport tcp \

      gluster1:/gluster-storage \

      gluster2:/gluster-storage force

# list volumes
sudo gluster volume list

# start the gluster volume
sudo gluster volume start volume1

# mount the volume
sudo mount -t glusterfs gluster1:/volume1 /mnt/media/

# only allow given ips
sudo gluster volume set volume1 auth.allow gluster1,gluster2

sudo gluster volume info
sudo gluster volume profile volume1 start
sudo gluster volume profile volume1 info
sudo gluster volume status

# stop the volume
sudo gluster volume stop volume1

FSTAB Entries

** ON Gluster 1
gluster1:/volume1 /mnt/media/ glusterfs defaults,_netdev,backupvolfile-server=gluster2 0 0

** ON Gluster 2
gluster2:/volume1 /mnt/media/ glusterfs defaults,_netdev,backupvolfile-server=gluster1 0 0

systemd quickstart

Source:
https://fedoraproject.org/wiki/SysVinit_to_Systemd_Cheatsheet
https://wiki.ubuntu.com/SystemdForUpstartUsers
http://wiki.nginx.org/FedoraSystemdServiceFile

Start/stop/reload a service:
service foo start   # or
systemctl start foo

service foo stop   # or 
systemctl stop foo

service foo restart   # or 
systemctl restart foo

Reload a service using reload signal:
service foo reload

See if service is running:
service foo status   # or 
systemctl status foo

See tree listing of all services:
systemctl status

See job log: 
sudo journalctl -u foo -f

Enable a service (to start on boot):
systemctl enable foo

Disable a service:
systemctl disable foo

Service Definition Directory:
/lib/systemd/system/

Example Nginx Service: /lib/systemd/system/nginx.service 

[Unit]
Description=The nginx HTTP and reverse proxy server
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target


Links:
https://scottlinux.com/2014/12/08/how-to-create-a-systemd-service-in-linux-centos-7/

Simple ClamAV Usage

Source: Nice Resource
http://askubuntu.com/questions/250290/how-do-i-scan-for-viruses-with-clamav

Install:
sudo apt-get install clamav

Refresh virus definition database:
sudo freshclam

Scan all files on computer showing each
clamscan -r /

Only display infected files and ring bell
clamscan -r --bell -i /

Scan all files and move them to infected dir
clamscan -r --move=/home/USER/VIRUS /home/USER

Scan all files and remove bad ones
clamscan -r --remove /home/USER

Python Tokenlib

Make an expiring token using a known secret.

Source:
https://github.com/mozilla-services/tokenlib

Make a token, hashed with a secret, and storing an IP, that expires in 12 seconds
>>> t = tokenlib.make_token({"ip":"1.1.1.1."}, secret="1234567890", timeout=12)

Parse a token before expiration time is up
>>> tokenlib.parse_token(t, secret="1234567890")
{u'ip': u'1.1.1.1.', u'expires': 1428098130.01954, u'salt': u'fbd08e'}

Parse a token after expiration time is up
>>> tokenlib.parse_token(t, secret="1234567890")
Traceback (most recent call last):
  File "", line 1, in
  File "tokenlib/__init__.py", line 186, in parse_token
    return TokenManager(**kwds).parse_token(token, now=now)
  File "/tokenlib/__init__.py", line 143, in parse_token
    raise errors.ExpiredTokenError()
tokenlib.errors.ExpiredTokenError: token has expired

Simple Python Http Server

Serves up files in the current directory.

Python 3
python3 -m http.server
or
python3 -m http.server 8080

Python 2
python -m SimpleHTTPServer
or 
python -m SimpleHTTPServer 8080

Source:
http://www.reddit.com/r/Python/comments/2ysd91/what_are_some_nifty_python_snippets_that_you_have/

Nodejs Hue Quicksheet

Summary of the following article:
https://github.com/peter-murray/node-hue-api

// load the module
var hue = require('node-hue-api');

// define some display functions
var displayResult = function(result) {
    console.log("Created user: " + JSON.stringify(result));
};

// define some display functions
var displayError = function(err) {
    console.error(err);
};

// search for bridges

timeout = 6000  // 6 seconds

hue.upnpSearch(timeout).then(displayBridges).done();

// register a new user

hostname = '192.168.1.112';

newUserName = null;

userDescription = "some description";

hue.registerUser(hostname, newUserName, userDescription)

   .then(displayResult).fail(displayError).done();

// define connection information
var hostname = "192.168.1.112",
    username="*************************",
    api;

// create the api object
api = new hue.HueApi(hostname, username);

// get basic information about the gateway api
api.description().then(displayResult).fail(displayError).done();

// get the full state of the gateway
api.fullState().then(displayResult).fail(displayError).done();

// see the users that are registered
api.registeredUsers().then(displayResult).fail(displayError).done();

// show the lights and their states
api.lights().then(displayResult).fail(displayError).done();

// define a state and set it to light #1
state = hue.lightState.create().on().white(500, 100);
api.setLightState(1, state).then(displayResult).fail(displayError).done();

// turn light #1 on
api.setLightState(1, state.on()).then(displayResult).fail(displayError).done();

// turn light #1 off
api.setLightState(1, state.off()).then(displayResult).fail(displayError).done();

// get the status of light #1
api.lightStatus(1).then(displayResult).fail(displayError).done();

// see the light groups 
api.groups().then(displayResult).fail(displayError).done();

// see the available scenes  
api.scenes().then(displayResult).fail(displayError).done();

// create a new scene assign light #1-7 to it.  Will include a scene 'id' in the result

lightIds = [1,2,3,4,5,6,7]

sceneName = "Demo Scene"

api.createScene(lightIds, sceneName).then(displayResults).done();

// update an existing scene

lightIds = [1,2,3,4,5,6,]

sceneName = "Updated Demo Scene"

sceneId = "id-generated-by-gateway"

api.updateScene(sceneId, lightIds, sceneName).then(displayResults).done();

// set states for light #1 in a scene

state = lightState.create().on().hue(2000)

sceneId = "id-generated-by-gateway"

lightId = 1

api.setSceneLightState(sceneId, lightId, state).then(displayResults).done();

// recall scene
api.recallScene(sceneId).then(displayResults).done();

Docker Elastisearch

Simple Configuration

1) Create a directory for local data:

mkdir `pwd`/data-dir

2) Create an elasticsearch config inside of the local data dir:

vim data-dir/elasticsearch.yml

path:

  logs: /data/log 

  data: /data/data

3) Create a script as such:

CONTAINER_NAME="elasticsearch_server"
ELASTIC_DATA_DIR="`pwd`/data-dir"
HOST_PORT_1="9200"
HOST_PORT_2="9300"

if [ ! -d "${ELASTIC_DATA_DIR}" ]
then
    mkdir -p "${ELASTIC_DATA_DIR}"
fi

docker rm ${CONTAINER_NAME}
sudo docker run -it --name ${CONTAINER_NAME} --rm \
    -p ${HOST_PORT_1}:9200 \
    -p ${HOST_PORT_2}:9300 \
    -v ${ELASTIC_DATA_DIR}:/data \
    dockerfile/elasticsearch \
    /elasticsearch/bin/elasticsearch -Des.config=/data/elasticsearch.yml

Source:
https://github.com/dockerfile/elasticsearch

Postgres As Docker

CONTAINER_NAME="postgres_server"
CONTAINER_DATA_DIR="/var/lib/postgresql/data"
HOST_PORT="55432"
POSTGRES_DOCKER_DIR=`pwd`
HOST_DATA_DIR="${POSTGRES_DOCKER_DIR}/data-dir/"

if [ ! -d "${HOST_DATA_DIR}" ]
then
    mkdir -p "${HOST_DATA_DIR}"
fi

sudo docker run --rm --name ${CONTAINER_NAME} \
            -v ${HOST_DATA_DIR}:${CONTAINER_DATA_DIR} \
            -p ${HOST_PORT}:5432 \
            -it postgres $1

Ubuntu: Apt find all outdated packages

apt-get --just-print upgrade | grep "Inst"|  while read a b c d;
do echo "\"$a\",\"$b\",\"$c\",\"$d\"";
done

Git Copy Branches Locally

Source: https://coderwall.com/p/0ypmka

#!/bin/bash for branch in `git branch -a | grep remotes | grep -v HEAD | grep -v master `; do git branch --track ${branch#remotes/origin/} $branch done

Django Annotation: Users with duplicate emails

User.objects.values("email").annotate(num_emails=Count("email")
                    ).order_by("-num_emails").filter(
                    num_emails__count__gt=1):

Reference:
http://stackoverflow.com/questions/8989221/django-select-only-rows-with-duplicate-field-values

Docker Notes

CASE STUDY

# see the list of docker images as a tree view. Determine IMAGE_ID
sudo docker images --tree

# get list of currently running containers. Determine CONTAINER_ID
sudo docker ps

# remove a docker image tree
sudo docker rmi IMAGE_ID/TAG

# get list of all containers
sudo docker ps -a

# get last executed container info
sudo docker ps -al

# get all containers without the command being truncated
sudo docker ps -a --no-trunc

# commit changes to a docker container and tag the commit
sudo docker commit CONTAINER_ID feature/nginx

# tag an image without running
sudo docker tag IMAGE_ID TAG_NAME

# run a tag or IMAGE_ID as an interactive bash shell
sudo docker run -t -i -p 80 feature/nginx /bin/bash

# remove the container after running
sudo docker run -rm ….

# name a container
sudo docker run --name …

# run a tag or IMAGE_ID as an interactive ssh process
sudo docker run -t -i -p 22 -p 80 feature/nginx /usr/sbin/sshd -D

# run a tag or IMAGE_ID as a background ssh process
sudo docker run -d -p 22 -p 80 feature/nginx /usr/sbin/sshd -D

# view docker logs
sudo docker logs CONTAINER_ID/NAME
sudo docker logs -f CONTAINER_ID/NAME  # follow

# stop a running container in the background or foreground
sudo docker stop CONTAINER_ID/NAME

# build and tag an image using a Dockerfile
docker build -t tag/name .

# run a docker process and open up ports 22 and 80 on the docker instance. The host ports are generated automatically.
sudo docker run -d -p 22 -p 80 feature/nginx /usr/bin/supervisord -n

# Get port on the host that is NATed to the given port on the image
sudo docker port CONTAINER_ID 22
sudo docker port CONTAINER_ID 80

# run a docker process and map host port 8888 to docker instance port 80 and host port 2222 to docker instance port 22
sudo docker run -d -p 2222:22 -p 8888:80 feature/nginx /usr/bin/supervisord -n

# Same as above but additionally creates a shared folder between the host and guest
sudo docker run -d -p 2222:22 -p 8888:80 -v /home/:/home/:rw feature/nginx /usr/bin/supervisord -n

# clean up extra containers
sudo docker rm `sudo docker ps -a | grep Exit | awk '{ print $1}'`
sudo docker rm $(sudo docker ps -a -q)

# NEW: clean up exited containers 
sudo docker ps -a -q --filter status=exited | xargs docker rm

# NEW: clean up untagged images
docker images -q --filter "dangling=true" | xargs docker rmi

# delete all untagged images
sudo docker rmi $( sudo docker images | grep '' | tr -s ' ' | cut -d ' ' -f 3)

# In Ubuntu, change the docker daemon listen address for Shipyard
DOCKER_OPTS="-H tcp://127.0.0.1:4243 -H unix:///var/run/docker.sock"
service restart docker

# images files stored in
/var/lib/docker/graph/

# get the host gateway from within the container
netstat -nr | grep '^0\.0\.0\.0' | awk '{print $2}'

# find metatdata bout the container
sudo docker inspect CONTAINER_ID/NAME

# Setup shipyard-agent
curl https://github.com/shipyard/shipyard-agent/releases/download//shipyard-agent -L -o /usr/local/bin/shipyard-agent
chmod +x /usr/local/bin/shipyard-agent

shipyard-agent -url http://myshipyardhost:shipyardport -register # register with shipyard
shipyard-agent -url http://myshipyardhost:shipyardport -key 1234567890qwertyuiop # run the agent


SETUP AN IMAGE
sudo aptitude install nginx vim openssh-server supervisord psmisc

vim /etc/supervisor/conf.d/nginx.conf
[program:nginx]
command=/usr/sbin/nginx -c /etc/nginx/nginx.conf
directory=/etc/nginx
user=root
autostart=true
autorestart=true
redirect_stderr=True

vim /etc/supervisor/conf.d/ssh.conf
[program:sshd]
command=/usr/sbin/sshd -D


FLATTEN IMAGE
http://www.hnwatcher.com/r/650964/Slimming-down-Docker-containers-Intercity-Blog

CONTAINER_ID=$(docker run -d image-name /bin/bash)
docker export $CONTAINER_ID | docker import – flat-image-name

CONTAINER_ID=$(docker run -d image-name /bin/bash)
(docker export $CONTAINER_ID | gzip -c > image.tgz)
gzip -dc image.tgz | docker import - flat-image-name


OTHER
lxc-attach
--links option
--env-file
https://gist.github.com/wsargent/7049221#images

Remove images article
http://mikehadlow.blogspot.com/2014/04/docker-bulk-remove-images-and-containers.html

Linking Containers
http://docs.docker.com/userguide/dockerlinks/

Dockerfile Reference
http://docs.docker.com/reference/builder/

FLYNN - Open Source PaaS
https://github.com/flynn/flynn

ETCD - Configuration Tool
https://github.com/coreos/etcd

Ubuntu 14.04 / Centos 7 Postgres 9.3 and Postgis

CENTOS 7

Reference:

http://www.postgresonline.com/journal/archives/329-An-almost-idiots-guide-to-install-PostgreSQL-9.3,-PostGIS-2.1-and-pgRouting-with-Yum.html

Pick a release from here:

   http://yum.postgresql.org/repopackages.php

Add the following postgres repo:

   sudo rpm -ivh  http://yum.postgresql.org/9.3/redhat/rhel-7-x86_64/pgdg-centos93-9.3-1.noarch.rpm

Install ifconfig:

   sudo yum install net-tools

Install packages:

   sudo yum install postgresql93 postgresql93-server postgresql93-libs postgresql93-contrib postgresql93-devel

Install needed epel repo:

   sudo rpm -ivh http://dl.fedoraproject.org/pub/epel/beta/7/x86_64/epel-release-7-0.2.noarch.rpm

Install postgis and pgrouting extensions:

   sudo yum install postgis2_93

   sudo yum install pgrouting_93

Edit pg_hba.conf to allow network users to login:

   vim /var/lib/pgsql/9.3/data/pg_hba.conf

   # add the following

   host all all  0.0.0.0/0   md5

Edit the postgresql.conf to allow hosts to connect from anywhere:

   vim /var/lib/pgsql/9.3/data/postgresql.conf

   # change the listen address

   listen_addresses = '*'

Init the database:

   /usr/pgsql-9.3/bin/postgresql93-setup initdb

Restart the service:

   sudo systemctl restart  postgresql-9.3.service

Get status:

   sudo service postgresql-9.3 status

Create a database:

   [vagrant@localhost ~]$ sudo -u postgres psql

   postgres=# create user testa with password '1234';

   CREATE ROLE

   postgres=# create database testa with owner testa;

   CREATE DATABASE

   postgres=# \c testa

   You are now connected to database "testa" as user "postgres".

   testa=# create extension postgis;

   CREATE EXTENSION

   testa=# create extension postgis_topology;

   CREATE EXTENSION

   testa=# SELECT postgis_full_version();

    POSTGIS="2.1.3 r12547" GEOS="3.4.2-CAPI-1.8.2 r3921" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.11.0, released

             2014/04/16" LIBXML="2.9.1" LIBJSON="UNKNOWN" TOPOLOGY RASTER

UBUNTU 14.04

Ubuntu Installation:

   sudo aptitude update

   sudo apt-get install build-essential postgresql-9.3 postgresql-server-dev-9.3 libxml2-dev libgdal-dev libproj-dev libjson0-dev xsltproc docbook-xsl docbook-mathml

   sudo aptitude install postgresql-9.3-postgis-2.1

Edit pg_hba.conf to allow network users to login:

   vim /etc/postgresql/9.3/main/pg_hba.conf

   # add the following

   host all all  0.0.0.0/0   md5

Edit the postgresql.conf to allow hosts to connect from anywhere:

   vim /etc/postgresql/9.3/main/postgresql.conf

   # change the listen address

   listen_addresses = '*'

Restart the service:

   sudo service postgresql restart

Create a database:

   [vagrant@localhost ~]$ sudo -u postgres psql

   postgres=# create user testa with password '1234';

   CREATE ROLE

   postgres=# create database testa with owner testa;

   CREATE DATABASE

   postgres=# \c testa

   You are now connected to database "testa" as user "postgres".

   testa=# create extension postgis;

   CREATE EXTENSION

   testa=# create extension postgis_topology;

   CREATE EXTENSION

   testa=# SELECT postgis_full_version();

    POSTGIS="2.1.3 r12547" GEOS="3.4.2-CAPI-1.8.2 r3921" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.11.0, released

             2014/04/16" LIBXML="2.9.1" LIBJSON="UNKNOWN" TOPOLOGY RASTER

vim: get full path of current file

Get the full path of the current file
:echo expand('%:p')


http://vim.wikia.com/wiki/Get_the_name_of_the_current_file

Upstart Basic Commands

Disables Nginx so that dpkg does not re-enable it when Nginx is upgraded:

sudo update-rc.d -f nginx disable

Add Nginx to default runlevels:
update-rc.d nginx defaults

Disable Service

sudo sh -c "echo 'manual' > /etc/init/nginx.override"

Manual Service Control
sudo service nginx start

sudo service nginx restart

sudo service nginx stop

Or

sudo start nginx

sudo stop nginx
sudo status nginx


http://askubuntu.com/questions/177041/nginx-disable-autostart

Creating differential backups with hard links and rsync

Creating differential backups with hard links and rsync:


From above:
backup@backup_server> DAY2=`date +%Y%m%d%H%M%S` 

backup@backup_server> cp -al /var/backups/$DAY1 /var/backups/$DAY2 

backup@backup_server> rsync -av -e ssh --delete earl@192.168.1.20:/home/earl/ /var/backups/$DAY2/

Find IP addresses with a lot of open connections:
netstat -atun | awk '{print $5}' | cut -d: -f1 | sed -e '/^$/d' |sort | uniq -c | sort -n

Source:
http://www.cyberciti.biz/tips/netstat-command-tutorial-examples.html

LSOF Quick Commands

http://stackoverflow.com/questions/106234/lsof-survival-guide

http://www.akadia.com/services/lsof_quickstart.txt

Bzip2 PGDump

To backup:

pg_dump -hlocalhost -Upostgres -a -t table database|bzip2 > tabledata.bz2

To restore:
bzip2 -c -d tabledata.bz2 | psql -hlocalhost -Upostgres database


Source: http://www.solomonson.com/content/using-bzip-pgdump