Docker Elastisearch

Simple Configuration

1) Create a directory for local data:

mkdir `pwd`/data-dir

2) Create an elasticsearch config inside of the local data dir:

vim data-dir/elasticsearch.yml

path:

  logs: /data/log 

  data: /data/data

3) Create a script as such:

CONTAINER_NAME="elasticsearch_server"
ELASTIC_DATA_DIR="`pwd`/data-dir"
HOST_PORT_1="9200"
HOST_PORT_2="9300"

if [ ! -d "${ELASTIC_DATA_DIR}" ]
then
    mkdir -p "${ELASTIC_DATA_DIR}"
fi

docker rm ${CONTAINER_NAME}
sudo docker run -it --name ${CONTAINER_NAME} --rm \
    -p ${HOST_PORT_1}:9200 \
    -p ${HOST_PORT_2}:9300 \
    -v ${ELASTIC_DATA_DIR}:/data \
    dockerfile/elasticsearch \
    /elasticsearch/bin/elasticsearch -Des.config=/data/elasticsearch.yml

Source:
https://github.com/dockerfile/elasticsearch

Postgres As Docker

CONTAINER_NAME="postgres_server"
CONTAINER_DATA_DIR="/var/lib/postgresql/data"
HOST_PORT="55432"
POSTGRES_DOCKER_DIR=`pwd`
HOST_DATA_DIR="${POSTGRES_DOCKER_DIR}/data-dir/"

if [ ! -d "${HOST_DATA_DIR}" ]
then
    mkdir -p "${HOST_DATA_DIR}"
fi

sudo docker run --rm --name ${CONTAINER_NAME} \
            -v ${HOST_DATA_DIR}:${CONTAINER_DATA_DIR} \
            -p ${HOST_PORT}:5432 \
            -it postgres $1

Ubuntu: Apt find all outdated packages

apt-get --just-print upgrade | grep "Inst"|  while read a b c d;
do echo "\"$a\",\"$b\",\"$c\",\"$d\"";
done

Git Copy Branches Locally

Source: https://coderwall.com/p/0ypmka

#!/bin/bash for branch in `git branch -a | grep remotes | grep -v HEAD | grep -v master `; do git branch --track ${branch#remotes/origin/} $branch done

Django Annotation: Users with duplicate emails

User.objects.values("email").annotate(num_emails=Count("email")
                    ).order_by("-num_emails").filter(
                    num_emails__count__gt=1):

Reference:
http://stackoverflow.com/questions/8989221/django-select-only-rows-with-duplicate-field-values

Docker Notes

CASE STUDY

# see the list of docker images as a tree view. Determine IMAGE_ID
sudo docker images --tree

# get list of currently running containers. Determine CONTAINER_ID
sudo docker ps

# remove a docker image tree
sudo docker rmi IMAGE_ID/TAG

# get list of all containers
sudo docker ps -a

# get last executed container info
sudo docker ps -al

# get all containers without the command being truncated
sudo docker ps -a --no-trunc

# commit changes to a docker container and tag the commit
sudo docker commit CONTAINER_ID feature/nginx

# tag an image without running
sudo docker tag IMAGE_ID TAG_NAME

# run a tag or IMAGE_ID as an interactive bash shell
sudo docker run -t -i -p 80 feature/nginx /bin/bash

# remove the container after running
sudo docker run -rm ….

# name a container
sudo docker run --name …

# run a tag or IMAGE_ID as an interactive ssh process
sudo docker run -t -i -p 22 -p 80 feature/nginx /usr/sbin/sshd -D

# run a tag or IMAGE_ID as a background ssh process
sudo docker run -d -p 22 -p 80 feature/nginx /usr/sbin/sshd -D

# view docker logs
sudo docker logs CONTAINER_ID/NAME
sudo docker logs -f CONTAINER_ID/NAME  # follow

# stop a running container in the background or foreground
sudo docker stop CONTAINER_ID/NAME

# build and tag an image using a Dockerfile
docker build -t tag/name .

# run a docker process and open up ports 22 and 80 on the docker instance. The host ports are generated automatically.
sudo docker run -d -p 22 -p 80 feature/nginx /usr/bin/supervisord -n

# Get port on the host that is NATed to the given port on the image
sudo docker port CONTAINER_ID 22
sudo docker port CONTAINER_ID 80

# run a docker process and map host port 8888 to docker instance port 80 and host port 2222 to docker instance port 22
sudo docker run -d -p 2222:22 -p 8888:80 feature/nginx /usr/bin/supervisord -n

# Same as above but additionally creates a shared folder between the host and guest
sudo docker run -d -p 2222:22 -p 8888:80 -v /home/:/home/:rw feature/nginx /usr/bin/supervisord -n

# clean up extra containers
sudo docker rm `sudo docker ps -a | grep Exit | awk '{ print $1}'`
sudo docker rm $(sudo docker ps -a -q)

# NEW: clean up exited containers 
sudo docker ps -a -q --filter status=exited | xargs docker rm

# NEW: clean up untagged images
docker images -q --filter "dangling=true" | xargs docker rmi

# delete all untagged images
sudo docker rmi $( sudo docker images | grep '' | tr -s ' ' | cut -d ' ' -f 3)

# In Ubuntu, change the docker daemon listen address for Shipyard
DOCKER_OPTS="-H tcp://127.0.0.1:4243 -H unix:///var/run/docker.sock"
service restart docker

# images files stored in
/var/lib/docker/graph/

# get the host gateway from within the container
netstat -nr | grep '^0\.0\.0\.0' | awk '{print $2}'

# find metatdata bout the container
sudo docker inspect CONTAINER_ID/NAME

# Setup shipyard-agent
curl https://github.com/shipyard/shipyard-agent/releases/download//shipyard-agent -L -o /usr/local/bin/shipyard-agent
chmod +x /usr/local/bin/shipyard-agent

shipyard-agent -url http://myshipyardhost:shipyardport -register # register with shipyard
shipyard-agent -url http://myshipyardhost:shipyardport -key 1234567890qwertyuiop # run the agent


SETUP AN IMAGE
sudo aptitude install nginx vim openssh-server supervisord psmisc

vim /etc/supervisor/conf.d/nginx.conf
[program:nginx]
command=/usr/sbin/nginx -c /etc/nginx/nginx.conf
directory=/etc/nginx
user=root
autostart=true
autorestart=true
redirect_stderr=True

vim /etc/supervisor/conf.d/ssh.conf
[program:sshd]
command=/usr/sbin/sshd -D


FLATTEN IMAGE
http://www.hnwatcher.com/r/650964/Slimming-down-Docker-containers-Intercity-Blog

CONTAINER_ID=$(docker run -d image-name /bin/bash)
docker export $CONTAINER_ID | docker import – flat-image-name

CONTAINER_ID=$(docker run -d image-name /bin/bash)
(docker export $CONTAINER_ID | gzip -c > image.tgz)
gzip -dc image.tgz | docker import - flat-image-name


OTHER
lxc-attach
--links option
--env-file
https://gist.github.com/wsargent/7049221#images

Remove images article
http://mikehadlow.blogspot.com/2014/04/docker-bulk-remove-images-and-containers.html

Linking Containers
http://docs.docker.com/userguide/dockerlinks/

Dockerfile Reference
http://docs.docker.com/reference/builder/

FLYNN - Open Source PaaS
https://github.com/flynn/flynn

ETCD - Configuration Tool
https://github.com/coreos/etcd

Ubuntu 14.04 / Centos 7 Postgres 9.3 and Postgis

CENTOS 7

Reference:

http://www.postgresonline.com/journal/archives/329-An-almost-idiots-guide-to-install-PostgreSQL-9.3,-PostGIS-2.1-and-pgRouting-with-Yum.html

Pick a release from here:

   http://yum.postgresql.org/repopackages.php

Add the following postgres repo:

   sudo rpm -ivh  http://yum.postgresql.org/9.3/redhat/rhel-7-x86_64/pgdg-centos93-9.3-1.noarch.rpm

Install ifconfig:

   sudo yum install net-tools

Install packages:

   sudo yum install postgresql93 postgresql93-server postgresql93-libs postgresql93-contrib postgresql93-devel

Install needed epel repo:

   sudo rpm -ivh http://dl.fedoraproject.org/pub/epel/beta/7/x86_64/epel-release-7-0.2.noarch.rpm

Install postgis and pgrouting extensions:

   sudo yum install postgis2_93

   sudo yum install pgrouting_93

Edit pg_hba.conf to allow network users to login:

   vim /var/lib/pgsql/9.3/data/pg_hba.conf

   # add the following

   host all all  0.0.0.0/0   md5

Edit the postgresql.conf to allow hosts to connect from anywhere:

   vim /var/lib/pgsql/9.3/data/postgresql.conf

   # change the listen address

   listen_addresses = '*'

Init the database:

   /usr/pgsql-9.3/bin/postgresql93-setup initdb

Restart the service:

   sudo systemctl restart  postgresql-9.3.service

Get status:

   sudo service postgresql-9.3 status

Create a database:

   [vagrant@localhost ~]$ sudo -u postgres psql

   postgres=# create user testa with password '1234';

   CREATE ROLE

   postgres=# create database testa with owner testa;

   CREATE DATABASE

   postgres=# \c testa

   You are now connected to database "testa" as user "postgres".

   testa=# create extension postgis;

   CREATE EXTENSION

   testa=# create extension postgis_topology;

   CREATE EXTENSION

   testa=# SELECT postgis_full_version();

    POSTGIS="2.1.3 r12547" GEOS="3.4.2-CAPI-1.8.2 r3921" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.11.0, released

             2014/04/16" LIBXML="2.9.1" LIBJSON="UNKNOWN" TOPOLOGY RASTER

UBUNTU 14.04

Ubuntu Installation:

   sudo aptitude update

   sudo apt-get install build-essential postgresql-9.3 postgresql-server-dev-9.3 libxml2-dev libgdal-dev libproj-dev libjson0-dev xsltproc docbook-xsl docbook-mathml

   sudo aptitude install postgresql-9.3-postgis-2.1

Edit pg_hba.conf to allow network users to login:

   vim /etc/postgresql/9.3/main/pg_hba.conf

   # add the following

   host all all  0.0.0.0/0   md5

Edit the postgresql.conf to allow hosts to connect from anywhere:

   vim /etc/postgresql/9.3/main/postgresql.conf

   # change the listen address

   listen_addresses = '*'

Restart the service:

   sudo service postgresql restart

Create a database:

   [vagrant@localhost ~]$ sudo -u postgres psql

   postgres=# create user testa with password '1234';

   CREATE ROLE

   postgres=# create database testa with owner testa;

   CREATE DATABASE

   postgres=# \c testa

   You are now connected to database "testa" as user "postgres".

   testa=# create extension postgis;

   CREATE EXTENSION

   testa=# create extension postgis_topology;

   CREATE EXTENSION

   testa=# SELECT postgis_full_version();

    POSTGIS="2.1.3 r12547" GEOS="3.4.2-CAPI-1.8.2 r3921" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.11.0, released

             2014/04/16" LIBXML="2.9.1" LIBJSON="UNKNOWN" TOPOLOGY RASTER

vim: get full path of current file

Get the full path of the current file
:echo expand('%:p')


http://vim.wikia.com/wiki/Get_the_name_of_the_current_file

Upstart Basic Commands

Disables Nginx so that dpkg does not re-enable it when Nginx is upgraded:

sudo update-rc.d -f nginx disable

Add Nginx to default runlevels:
update-rc.d nginx defaults

Disable Service

sudo sh -c "echo 'manual' > /etc/init/nginx.override"

Manual Service Control
sudo service nginx start

sudo service nginx restart

sudo service nginx stop

Or

sudo start nginx

sudo stop nginx
sudo status nginx


http://askubuntu.com/questions/177041/nginx-disable-autostart

Creating differential backups with hard links and rsync

Creating differential backups with hard links and rsync:


From above:
backup@backup_server> DAY2=`date +%Y%m%d%H%M%S` 

backup@backup_server> cp -al /var/backups/$DAY1 /var/backups/$DAY2 

backup@backup_server> rsync -av -e ssh --delete earl@192.168.1.20:/home/earl/ /var/backups/$DAY2/

Find IP addresses with a lot of open connections:
netstat -atun | awk '{print $5}' | cut -d: -f1 | sed -e '/^$/d' |sort | uniq -c | sort -n

Source:
http://www.cyberciti.biz/tips/netstat-command-tutorial-examples.html