ssh-copy-id: add authorized_keys entry automatically

Copy authorized key to remote server:
ssh-copy-id -i ~/.ssh/id_rsa example.com

-i = private key associated with the public key that you would like to copy

Postgresql: Read-Only User

create user xxxxx with password 'yyyy';
GRANT connect ON DATABASE mydb1 TO xxxxx;
GRANT connect ON DATABASE mydb2 TO xxxxx;

\c mydb1
GRANT usage ON SCHEMA public TO xxxxx;
GRANT select ON ALL TABLES IN SCHEMA public TO xxxxx;
GRANT select ON ALL SEQUENCES IN SCHEMA public TO xxxxx;

\c mydb2
GRANT usage ON SCHEMA public TO xxxxx;
GRANT select ON ALL TABLES IN SCHEMA public TO xxxxx;
GRANT select ON ALL SEQUENCES IN SCHEMA public TO xxxxx;

Open Source Webservices

Version Control
Gitlab - https://about.gitlab.com/
Phabricator - http://phabricator.org/

Ticket Management (kanban)
Gitlab Kanban Board - http://kanban.leanlabs.io/
Taiga - https://taiga.io/
Kanboard - https://kanboard.net/

Group Chat
Mattermost - mattermost.org/ - also bundled with gitlab

Rocket.Chat - https://rocket.chat/

Let's Chat - http://sdelements.github.io/lets-chat/

Friends - http://moose-team.github.io/friends/

Zulip - https://www.zulip.org/

Compare: https://blog.okturtles.com/2015/11/five-open-source-slack-alternatives/
Amazon S3 Compatible Alternatives
Minio - https://www.minio.io/
Red Hat Ceph - http://ceph.com/

Password Generator
Diceware - https://www.rempe.us/diceware/#eff

Make Image out of code:
Cabon - https://carbon.now.sh

PaaS
openstack
Kubernetes
Kel - http://docs.kelproject.com/
Doku - http://dokku.viewdocs.io/dokku/
Flynn - https://flynn.io/
   More: https://www.quora.com/Is-there-open-source-software-that-implements-Amazon-S3-plug-compatible-storage

Malicatcher
Send email to here for debugging
https://hub.docker.com/r/schickling/mailcatcher/
https://mailcatcher.me/

Make Unix Password Hash

mkpasswd --method=SHA-512

Webservices

Email Spam Checker
http://www.mail-tester.com/

Git Hosting
Gitlab - https://about.gitlab.com/ (open source)
Phabricator - http://phabricator.org/ (open source)

Group Chat
Mattermost - http://www.mattermost.org/ (open source)
Slack - http://www.json-generator.com/
HipChat - https://www.hipchat.com/
Rocket - https://rocket.chat/features (open source)

Javascript Editor
jsfiddle https://jsfiddle.net/
Codepen - http://codepen.io/
jsbin - https://jsbin.com/ (open source)

JSON Generator
http://www.json-generator.com/

Online IDE
Cloudnine - https://c9.io/ (open source)

HTML/CS/JS Cleanup
http://www.dirtymarkup.com/

SSL Certificate Checker
https://www.ssllabs.com/ssltest

SSL Check Intermediate Certificate Chain
https://www.sslshopper.com/ssl-checker.html

Fake Email Tester (Send emails to here on dev and view them)
https://mailtrap.io/

Terminal Emulator:
https://hyperterm.org/ (open source)

Note Taking Desktop App
Boostnote - https://b00st.io/ (open source)

Python/Django Storage Backend Notes

Inspired by:
https://django-storages.readthedocs.org/en/latest/backends/amazon-S3.html#storage

Interacting directly with storage backend
from django.conf import settings
from django.core.files.storage import get_storage_class
STORAGE_CLASS_STRING = getattr(settings, "MY_STORAGE_CLASS",  \
        settings.DEFAULT_FILE_STORAGE)
sc = get_storage_class(STORAGE_CLASS_STRING)
s = sc()
s.url("jjj-test/JOE.jpg")
# '/media/jjj-test/JOE.jpg'
s.path("jjj-test/JOE.jpg")
# u'/home/jjasinski/Sites/mysite/htdocs/media/jjj-test/JOE.jpg'
s.exists("jjj-test/JOE.jpg")
# False
f = s.open("jjj-test/JOE.jpg", 'w')
f.write("joe test")
f.close()
s.exists("jjj-test/JOE.jpg")
# True
s.delete("jjj-test/JOE.jpg")
s.exists("jjj-test/JOE.jpg")
# False

Interacting with a Model's storage
from django.core.files.base import ContentFile
obj = MyModel()
obj.photo.save('django_test.txt', ContentFile('content'))
obj.photo.size
obj.photo.read()
obj.delete()

Django: staticfiles

Locate static file location in codebase:

>>> from django.contrib.staticfiles.finders import find

>>> find("img/logo.png", all=True)
[u'/sites/static_in_pro/our_static/img/logo.png']

>>> find("img/logo.png",)
u'/sites/static_in_pro/our_static/img/logo.png'

LetsEncrypt with HAProxy or Nginx

At this time, LetsEncrypt is in public beta, but I suspect that it will continue to evolve. 

# all commands must be done as root
sudo su

# Download the letsencrypt repo 
git clone https://github.com/letsencrypt/letsencrypt.git /top/letsencrypt/

# change to the desired keys directory.  All commands following are relative to this dir.
cd /jaz/sites/common/etc/keys/

# Generate a 4096 bit ssl private key
openssl genrsa 4096 > jazstudios.com.key

# Generate the certificate signing request.  The following allows lets you specify a SAN (Subject Alternative Name) which allows www and non-www versions of the same domain.  The output needs to be in "der" format. 

openssl req -new -sha256 \
    -key joejasinski.com.key \
    -subj "/C=US/ST=IL/L=Chicago/O=Jazstudios/OU=Information Technology/CN=www.joejasinski.com" \
    -reqexts SAN \
    -outform der \
    -config <(cat /etc/ssl/openssl.cnf \
        <(printf "[SAN]\nsubjectAltName=DNS:joejasinski.com,DNS:www.joejasinski.com")) \
    -out joejasinski.com.csr

# execute the letsencrypt command.  This will prompt you through a few actions.  The most important is that you will need to stop any server running on port 80 and run the python script that they provide in the output.  This will serve up a specific secret file at a specific location, allowing letsencrypt to authenticate the server.  (You could also host the secret file with your webserver)

/opt/letsencrypt/letsencrypt-auto --email example@gmail.com --text \
    --authenticator manual \
    --work-dir /tmp/work/ \
    --config-dir /tmp/config/ \
    --logs-dir /tmp/logs/ auth \
    --cert-path /tmp/certs/ \
    --chain-path /tmp/chains/ \
    --csr joejasinski.com.csr

# --text = use the text based 'wizard' installer instead of an ncurses one

# --authenticator manual = the admin must manually host the verification file

# --csr = path to the previously generated csr file

# The command that it will have you run looks something like this:

mkdir -p /tmp/letsencrypt/public_html/.well-known/acme-challenge

cd /tmp/letsencrypt/public_html

printf "%s" asdfkjasfdasfdasfdasfdasdf > .well-known/acme-challenge/asdfasdfasfd

# run only once per server:

$(command -v python2 || command -v python2.7 || command -v python2.6) -c \

"import BaseHTTPServer, SimpleHTTPServer; \

s = BaseHTTPServer.HTTPServer(('', 80), SimpleHTTPServer.SimpleHTTPRequestHandler); \

s.serve_forever()"

# The output of the of the letsencrypt-auto command will be a file called 0000_chain.pem.  This file contains the host certificate and the intermediate certificate.  It will look something like this.  

-----BEGIN CERTIFICATE-----

       Host certificate contents

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

       letsencrypt intermediate certificate

-----END CERTIFICATE-----

# Unrelated, but a good idea: generate a dhparam used for perfect forward security 

openssl dhparam -out dhparam.pem 4096

For hosting with Nginx, this file can be set to the ssl_certificate parameter. The ssl_certificate_key setting would be set to the location of the key file.

For hosting with HAProxy, you want to modify the file so it looks something like this:

-----BEGIN CERTIFICATE-----

       Host certificate contents

-----END CERTIFICATE-----

-----BEGIN RSA PRIVATE KEY-----

       Private key contents

-----END RSA PRIVATE KEY-----

-----BEGIN CERTIFICATE-----

       letsencrypt intermediate certificate

-----END CERTIFICATE-----

-----BEGIN DH PARAMETERS-----

       contents of dhparam.pem

-----END DH PARAMETERS-----

Dig Commands

Look up A record example.com as seen by 8.8.8.8 nameserver
dig @8.8.8.8 exampe.com

Show the MX record for a yahoo domain
dig yahoo.com MX

Show the TTL for a given domain
dig imagescape.com TTL

Show the TXT record for an yahoo domain
dig yahoo.com TXT

Show only the answer

dig yahoo.com +nocomments +noquestion +noauthority +noadditional +nostats

All Records

dig @8.8.8.8 imagescape.com ANY +noall +answer

Reverse Lookup

dig -x 198.178.132.10

Nice output that looks like the domain record

dig @ns.imagescape.com imagescape.com any +multiline +noall +answer

Find the authoritative nameserver for a domain:

dig -t ns imagescape.com +short

Find the SOA of the domain:
dig joejasinski.com soa +noall +answer


dig @ns1.linode.com joejasinski.com soa

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @ns1.linode.com joejasinski.com soa
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- 64217="" id:="" noerror="" opcode:="" p="" query="" status:="">;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 5, ADDITIONAL: 11
;; WARNING: recursion requested but not available

;; ANSWER SECTION:
joejasinski.com. 86400 IN SOA ns1.linode.com. joe\.jasinski.gmail.com. 2014092859 14400 14400 1209600 86400

The presence "aa" indicates if the nameserver is the authoritative server.

The "2014092859" is the serial

Download a zone file (zone transfer):
dig axfr domain.com


Common Options:
  +short  = show only the results
  +[no]comments = show/hide the comments
  +[no]question = show/hide the question section
  +[no]authority = show/hide the authority section
  +[no]stats = show/hide the stats section
  +[no]answer = show/hide the answer section
  +[no]all = show/hide everything

https://neverendingsecurity.wordpress.com/2015/04/13/dig-commands-cheatsheet/

https://www.madboa.com/geek/dig/

Nice articles:

Commands
http://anouar.adlani.com/2011/12/useful-dig-command-to-troubleshot-your-domains.html

Create DNS Slave
http://www.microhowto.info/howto/configure_bind_as_a_slave_dns_server.html

Python Email Send

Simple email test from python

import smtplib
from email.mime.text import MIMEText
f = "no-reply@example.com"
t = "joe@example.com"
s = smtplib.SMTP('localhost')
msg = MIMEText("this is a test")
msg["Subject"] = "This is a test - ignore"
msg["To"] = t
msg["From"] = f
s.sendmail(f, [t], msg.as_string())

Docker Gitlab

1) Create the following file
vim run_gitlab.sh

CONTAINER_NAME="gitlab_server"
GITLAB_DATA_DIR="`pwd`/gitlab/"
GITLAB_SSH_PORT="2222"
GITLAB_HTTP_PORT="8080"
GITLAB_HTTPS_PORT="8443"

docker rm ${CONTAINER_NAME}
sudo docker run --detach \
    --publish ${GITLAB_HTTPS_PORT}:443 \
    --publish ${GITLAB_HTTP_PORT}:8000 \
    --publish ${GITLAB_SSH_PORT}:22 \
    --name ${CONTAINER_NAME} \
    --restart always \
    --volume ${GITLAB_DATA_DIR}config:/etc/gitlab \
    --volume ${GITLAB_DATA_DIR}logs:/var/log/gitlab \
    --volume ${GITLAB_DATA_DIR}gitlab/data:/var/opt/gitlab \
    gitlab/gitlab-ce:latest

2) Run to create the initial directories

   sudo ./run_gitlab.sh 

3) Configure: modify the following settings

    sudo vim gitlab/config/gitlab.rb 

         external_url 'http://localhost:8000'

         gitlab_rails['gitlab_ssh_host'] = 'localhost'

         gitlab_rails['gitlab_shell_ssh_port'] = 2222

4) Restart docker

    sudo docker restart gitlab_server

5) Visit http://localhost:8080 and login with

    root 5iveL!fe


Source:
https://gitlab.com/gitlab-org/gitlab-ce/tree/master/docker

Autostart:
https://docs.docker.com/articles/host_integration/

Docker Redis

1) Create a script as such:
CONTAINER_NAME="redis_server"
REDIS_DATA_DIR="`pwd`/data-dir/redis/"
HOST_PORT_1="6379"

if [ ! -d "${REDIS_DATA_DIR}" ]
then
    mkdir -p "${REDIS_DATA_DIR}"
fi

docker rm ${CONTAINER_NAME}
docker run --name ${CONTAINER_NAME} --rm \
   -p ${HOST_PORT_1}:6379 \
   -v ${REDIS_DATA_DIR}:/data \
    redis

Postgres Query: Given an index find associated table

select
    t.relname as table_name,
    i.relname as index_name,
    array_to_string(array_agg(a.attname), ', ') as column_names
from
    pg_class t,
    pg_class i,
    pg_index ix,
    pg_attribute a
where
    t.oid = ix.indrelid
    and i.oid = ix.indexrelid
    and a.attrelid = t.oid
    and a.attnum = ANY(ix.indkey)
    and t.relkind = 'r'
    and i.relname = 'Index_to_search_for'
group by
    t.relname,
    i.relname
order by
    t.relname,
    i.relname;
 
 
Source:
http://stackoverflow.com/questions/2204058/list-columns-with-indexes-in-postgresql

Docker MongoDB

1) Create a directory for local data:

mkdir `pwd`/data-dir

2) Create a script as such:

CONTAINER_NAME="mongodb_server"
MONGO_DATA_DIR="`pwd`/data-dir"
HOST_PORT_1="27017"

if [ ! -d "${MONGO_DATA_DIR}" ]
then
    mkdir -p "${MONGO_DATA_DIR}"
fi

docker rm ${CONTAINER_NAME}
docker run --name ${CONTAINER_NAME} --rm \
   -p ${HOST_PORT_1}:27017 \
   -v ${MONGO_DATA_DIR}:/data/db \
   mongo

Django Manipulating the Session without a request

Django: Lookup a session without a request

    from django.conf import settings
    from importlib import import_module
    engine = import_module(settings.SESSION_ENGINE)
    session_store = engine.SessionStore("new_or_existing_sessionID")
    uid = session_store.get('_auth_user_id')

    try:
        del session_store['session_key']
    except IndexError:
        pass

    session_store.save()

GlusterFS Configuration

Assume node1 is: 192.168.33.10
Assume node2 is: 192.168.33.11

Sources:
https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers
http://www.jamescoyle.net/how-to/457-glusterfs-firewall-rules

On BOTH nodes

# set hosts file entry (or configure DNS)
vim /etc/hosts
192.168.33.10 gluster1
192.168.33.11 gluster2

# On Ubuntu: install requirements 
sudo add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.5
sudo apt-get update
sudo apt-get install glusterfs-server
sudo apt-get install glusterfs-client

# On Centos: install requirements

wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/RHEL/glusterfs-epel.repo

yum install glusterfs-server

yum install glusterfs-client

# make storage location for gluster data
sudo mkdir /gluster-storage

# make a mountpoint
sudo mkdir /mnt/media/

ON GLUSTER 1

# connect node 1 and node 2
sudo gluster peer probe gluster2
sudo gluster peer status

# create a Gluster replica volume
sudo gluster volume create volume1 replica 2 transport tcp \

      gluster1:/gluster-storage \

      gluster2:/gluster-storage force

# list volumes
sudo gluster volume list

# start the gluster volume
sudo gluster volume start volume1

# mount the volume
sudo mount -t glusterfs gluster1:/volume1 /mnt/media/

# only allow given ips
sudo gluster volume set volume1 auth.allow gluster1,gluster2

sudo gluster volume info
sudo gluster volume profile volume1 start
sudo gluster volume profile volume1 info
sudo gluster volume status

# stop the volume
sudo gluster volume stop volume1

FSTAB Entries

** ON Gluster 1
gluster1:/volume1 /mnt/media/ glusterfs defaults,_netdev,backupvolfile-server=gluster2 0 0

** ON Gluster 2
gluster2:/volume1 /mnt/media/ glusterfs defaults,_netdev,backupvolfile-server=gluster1 0 0

systemd quickstart

Source:
https://fedoraproject.org/wiki/SysVinit_to_Systemd_Cheatsheet
https://wiki.ubuntu.com/SystemdForUpstartUsers
http://wiki.nginx.org/FedoraSystemdServiceFile

Start/stop/reload a service:
service foo start   # or
systemctl start foo

service foo stop   # or 
systemctl stop foo

service foo restart   # or 
systemctl restart foo

Reload a service using reload signal:
service foo reload

See if service is running:
service foo status   # or 
systemctl status foo

See tree listing of all services:
systemctl status

See job log: 
sudo journalctl -u foo -f

Enable a service (to start on boot):
systemctl enable foo

Disable a service:
systemctl disable foo

Service Definition Directory:
/lib/systemd/system/

Example Nginx Service: /lib/systemd/system/nginx.service 

[Unit]
Description=The nginx HTTP and reverse proxy server
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target


Links:
https://scottlinux.com/2014/12/08/how-to-create-a-systemd-service-in-linux-centos-7/

Simple ClamAV Usage

Source: Nice Resource
http://askubuntu.com/questions/250290/how-do-i-scan-for-viruses-with-clamav

Install:
sudo apt-get install clamav

Refresh virus definition database:
sudo freshclam

Scan all files on computer showing each
clamscan -r /

Only display infected files and ring bell
clamscan -r --bell -i /

Scan all files and move them to infected dir
clamscan -r --move=/home/USER/VIRUS /home/USER

Scan all files and remove bad ones
clamscan -r --remove /home/USER

Python Tokenlib

Make an expiring token using a known secret.

Source:
https://github.com/mozilla-services/tokenlib

Make a token, hashed with a secret, and storing an IP, that expires in 12 seconds
>>> t = tokenlib.make_token({"ip":"1.1.1.1."}, secret="1234567890", timeout=12)

Parse a token before expiration time is up
>>> tokenlib.parse_token(t, secret="1234567890")
{u'ip': u'1.1.1.1.', u'expires': 1428098130.01954, u'salt': u'fbd08e'}

Parse a token after expiration time is up
>>> tokenlib.parse_token(t, secret="1234567890")
Traceback (most recent call last):
  File "", line 1, in
  File "tokenlib/__init__.py", line 186, in parse_token
    return TokenManager(**kwds).parse_token(token, now=now)
  File "/tokenlib/__init__.py", line 143, in parse_token
    raise errors.ExpiredTokenError()
tokenlib.errors.ExpiredTokenError: token has expired

Simple Python Http Server

Serves up files in the current directory.

Python 3
python3 -m http.server
or
python3 -m http.server 8080

Python 2
python -m SimpleHTTPServer
or 
python -m SimpleHTTPServer 8080

Source:
http://www.reddit.com/r/Python/comments/2ysd91/what_are_some_nifty_python_snippets_that_you_have/