Wednesday, August 07, 2024

systemd advanced topics

Basic Commands

# start / stop
systemctl start myservice
systemctl stop service

# reload after making changes to systemd unit files
# note: a reboot also does this
systemctl daemon-reload

# list dependencies of everything
systemctl  list-dependencies

# list dependencies for a given unit file
systemctl list-dependencies myservice.service

# list reverse dependencies  for a given unit file
systemctl list-dependencies myservice.service --reverse --all

Locations

Location of unit files
/etc/systemd/system/

Location of OS installed unit files (so users don't break the system)
/usr/lib/systemd/system/

Location of  runtime systemd unit files
/run/systemd/system

Various types of unit files ".target", ".service", ".path", ".mount", etc

.target = is like a "runlevel" - it lets you tie many unit files under it so you can enable/disable entire groups of things at once

.service = is a running process.

.path = lets you watch for changes to a path using inotify, and then trigger another unit file

.mount = lets you mount filesystems using systemd. /etc/fstab is automatically turned into unit files

and more...

Sections of a unit file

[Unit]  Metadata and trigger ordering dependencies

Description = description about the unit file

Before = ensures that units specified here are triggered before this unit

After = units listed here are triggered after this unit

Requires = list of units that must be running/completed before this one

Wants = similar to requires but ignores units that are not found or fail

[Install]  Metadata and trigger ordering dependencies

WantedBy = reverse dependency for "wants". units listed here must wait for this unit to run/complete before they can run

RequriedBy = reverse dependency for "requires".

Activating the Install section

Items listed in the [Install] section do not take affect until you run "systemctl enable myservice.service". Or conversely "systemctl disable myservice.service".  For example, say you have a unit file as such:

[Unit]
Description = myservice.service

[Install]
WantedBy = multi-user.target
RequiredBy = cloud-final.target


When you run "systemctl enable myservice.service", a symlink to this unit file will be created in /etc/systemd/system/multi-user.target.wants/ and in /etc/systemd/system/cloud-final.target/. This will make the dependency take effect.

Note: if the /etc/systemd/system/multi-user.target.wants/ dir does not exist, it will be created
Note: this also works for all types unit files (e.g, you can run 'systemctl enable myservice.path')

[Target | Service | Path | Mount | etc] Type specific unit file configuration

Services
 
[Service]
# Types:
# simple = "will consider the unit started immediately after the main service process has been forked off"
# oneshot = "is similar to simple; however, the service manager will consider the unit up after the main process exits."
Type = simple
 
# If True, service shall be considered active even when all its processes exited.
RemainAfterExit =
true

# command to run
ExecStart = /bin/bash -c 'echo hi'
 
ExecStartPre=...
ExecStartPost=...

Path
Waits for a path to create or update
https://www.freedesktop.org/software/systemd/man/latest/systemd.path.html


Conditionally starting a service

Two types: 
- Conditions fail (quietly) if a unit is run and a condition is not met.  
- Asserts fail and return an error

Run if file exists and not empty 

[Unit]
ConditionPathExists = /opt/empty.txt
ConditionFileNotEmpty = /opt/empty.txt

Run if file exists and not empty 

[Unit] ConditionPathExists =! /opt/empty.txt 

Run if either abc.txt or def.txt exists and user is root

[Unit] ConditionPathExists=| /tmp/abc.txt ConditionPathExists=| /tmp/def.txt ConditionUser = root

 
List of conditions
https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#Conditions%20and%20Asserts 

Path unit: Running unit based on path

Example unit file

[Unit]
Description="Trigger unit based on /tmp/test path"

[Path]
PathModified=/tmp/test
# other options PathExists=, PathExistsGlob=, PathChanged=, PathModified=, DirectoryNotEmpty=

Unit=unit-to-trigger.service

[Install]
WantedBy=multi-user.target 
 

Friday, November 03, 2017

ssh-copy-id: add authorized_keys entry automatically


Copy authorized key to remote server:
ssh-copy-id -i ~/.ssh/id_rsa example.com

-i = private key associated with the public key that you would like to copy

Saturday, June 17, 2017

Postgresql: Read-Only User



create user xxxxx with password 'yyyy';
GRANT connect ON DATABASE mydb1 TO xxxxx;
GRANT connect ON DATABASE mydb2 TO xxxxx;

\c mydb1
GRANT usage ON SCHEMA public TO xxxxx;
GRANT select ON ALL TABLES IN SCHEMA public TO xxxxx;
GRANT select ON ALL SEQUENCES IN SCHEMA public TO xxxxx;

\c mydb2
GRANT usage ON SCHEMA public TO xxxxx;
GRANT select ON ALL TABLES IN SCHEMA public TO xxxxx;
GRANT select ON ALL SEQUENCES IN SCHEMA public TO xxxxx;

Sunday, June 12, 2016

Open Source Webservices


Version Control
Gitlab - https://about.gitlab.com/
Phabricator - http://phabricator.org/

Ticket Management (kanban)
Gitlab Kanban Board - http://kanban.leanlabs.io/
Taiga - https://taiga.io/
Kanboard - https://kanboard.net/

Group Chat
Mattermost - mattermost.org/ - also bundled with gitlab
Rocket.Chat - https://rocket.chat/
Let's Chat - http://sdelements.github.io/lets-chat/
Friends - http://moose-team.github.io/friends/
Zulip - https://www.zulip.org/

Compare: https://blog.okturtles.com/2015/11/five-open-source-slack-alternatives/
Amazon S3 Compatible Alternatives
Minio - https://www.minio.io/
Red Hat Ceph - http://ceph.com/

Password Generator
Diceware - https://www.rempe.us/diceware/#eff

Make Image out of code:
Cabon - https://carbon.now.sh

PaaS
openstack
Kubernetes
Kel - http://docs.kelproject.com/
Doku - http://dokku.viewdocs.io/dokku/
Flynn - https://flynn.io/
   More: https://www.quora.com/Is-there-open-source-software-that-implements-Amazon-S3-plug-compatible-storage

Malicatcher
Send email to here for debugging
https://hub.docker.com/r/schickling/mailcatcher/
https://mailcatcher.me/



Friday, March 04, 2016

Make Unix Password Hash

mkpasswd --method=SHA-512

Webservices


Email Spam Checker
http://www.mail-tester.com/

Git Hosting
Gitlab - https://about.gitlab.com/ (open source)
Phabricator - http://phabricator.org/ (open source)

Group Chat
Mattermost - http://www.mattermost.org/ (open source)
Slack - http://www.json-generator.com/
HipChat - https://www.hipchat.com/
Rocket - https://rocket.chat/features (open source)

Javascript Editor
jsfiddle https://jsfiddle.net/
Codepen - http://codepen.io/
jsbin - https://jsbin.com/ (open source)

JSON Generator
http://www.json-generator.com/

Online IDE
Cloudnine - https://c9.io/ (open source)

HTML/CS/JS Cleanup
http://www.dirtymarkup.com/

SSL Certificate Checker
https://www.ssllabs.com/ssltest

SSL Check Intermediate Certificate Chain
https://www.sslshopper.com/ssl-checker.html

Fake Email Tester (Send emails to here on dev and view them)
https://mailtrap.io/

Terminal Emulator:
https://hyperterm.org/ (open source)

Note Taking Desktop App
Boostnote - https://b00st.io/ (open source)

Python/Django Storage Backend Notes

Inspired by:
https://django-storages.readthedocs.org/en/latest/backends/amazon-S3.html#storage

Interacting directly with storage backend
from django.conf import settings
from django.core.files.storage import get_storage_class
STORAGE_CLASS_STRING = getattr(settings, "MY_STORAGE_CLASS",  \
        settings.DEFAULT_FILE_STORAGE)
sc = get_storage_class(STORAGE_CLASS_STRING)
s = sc()
s.url("jjj-test/JOE.jpg")
# '/media/jjj-test/JOE.jpg'
s.path("jjj-test/JOE.jpg")
# u'/home/jjasinski/Sites/mysite/htdocs/media/jjj-test/JOE.jpg'
s.exists("jjj-test/JOE.jpg")
# False
f = s.open("jjj-test/JOE.jpg", 'w')
f.write("joe test")
f.close()
s.exists("jjj-test/JOE.jpg")
# True
s.delete("jjj-test/JOE.jpg")
s.exists("jjj-test/JOE.jpg")
# False

Interacting with a Model's storage
from django.core.files.base import ContentFile
obj = MyModel()
obj.photo.save('django_test.txt', ContentFile('content'))
obj.photo.size
obj.photo.read()
obj.delete()

Wednesday, December 09, 2015

Django: staticfiles

Locate static file location in codebase:

>>> from django.contrib.staticfiles.finders import find

>>> find("img/logo.png", all=True)
[u'/sites/static_in_pro/our_static/img/logo.png']

>>> find("img/logo.png",)
u'/sites/static_in_pro/our_static/img/logo.png'

Monday, December 07, 2015

LetsEncrypt with HAProxy or Nginx


At this time, LetsEncrypt is in public beta, but I suspect that it will continue to evolve. 

# all commands must be done as root
sudo su

# Download the letsencrypt repo 
git clone https://github.com/letsencrypt/letsencrypt.git /top/letsencrypt/

# change to the desired keys directory.  All commands following are relative to this dir.
cd /jaz/sites/common/etc/keys/

# Generate a 4096 bit ssl private key
openssl genrsa 4096 > jazstudios.com.key

# Generate the certificate signing request.  The following allows lets you specify a SAN (Subject Alternative Name) which allows www and non-www versions of the same domain.  The output needs to be in "der" format. 

openssl req -new -sha256 \
    -key joejasinski.com.key \
    -subj "/C=US/ST=IL/L=Chicago/O=Jazstudios/OU=Information Technology/CN=www.joejasinski.com" \
    -reqexts SAN \
    -outform der \
    -config <(cat /etc/ssl/openssl.cnf \
        <(printf "[SAN]\nsubjectAltName=DNS:joejasinski.com,DNS:www.joejasinski.com")) \
    -out joejasinski.com.csr

# execute the letsencrypt command.  This will prompt you through a few actions.  The most important is that you will need to stop any server running on port 80 and run the python script that they provide in the output.  This will serve up a specific secret file at a specific location, allowing letsencrypt to authenticate the server.  (You could also host the secret file with your webserver)

/opt/letsencrypt/letsencrypt-auto --email example@gmail.com --text \
    --authenticator manual \
    --work-dir /tmp/work/ \
    --config-dir /tmp/config/ \
    --logs-dir /tmp/logs/ auth \
    --cert-path /tmp/certs/ \
    --chain-path /tmp/chains/ \
    --csr joejasinski.com.csr

# --text = use the text based 'wizard' installer instead of an ncurses one
# --authenticator manual = the admin must manually host the verification file
# --csr = path to the previously generated csr file


# The command that it will have you run looks something like this:

mkdir -p /tmp/letsencrypt/public_html/.well-known/acme-challenge
cd /tmp/letsencrypt/public_html
printf "%s" asdfkjasfdasfdasfdasfdasdf > .well-known/acme-challenge/asdfasdfasfd
# run only once per server:
$(command -v python2 || command -v python2.7 || command -v python2.6) -c \
"import BaseHTTPServer, SimpleHTTPServer; \
s = BaseHTTPServer.HTTPServer(('', 80), SimpleHTTPServer.SimpleHTTPRequestHandler); \
s.serve_forever()"

# The output of the of the letsencrypt-auto command will be a file called 0000_chain.pem.  This file contains the host certificate and the intermediate certificate.  It will look something like this.  

-----BEGIN CERTIFICATE-----
       Host certificate contents
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
       letsencrypt intermediate certificate
-----END CERTIFICATE-----


# Unrelated, but a good idea: generate a dhparam used for perfect forward security 

openssl dhparam -out dhparam.pem 4096

For hosting with Nginx, this file can be set to the ssl_certificate parameter. The ssl_certificate_key setting would be set to the location of the key file.

For hosting with HAProxy, you want to modify the file so it looks something like this:

-----BEGIN CERTIFICATE-----
       Host certificate contents
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
       Private key contents
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
       letsencrypt intermediate certificate
-----END CERTIFICATE-----
-----BEGIN DH PARAMETERS-----
       contents of dhparam.pem
-----END DH PARAMETERS-----

Friday, November 13, 2015

Dig Commands


Look up A record example.com as seen by 8.8.8.8 nameserver
dig @8.8.8.8 exampe.com

Show the MX record for a yahoo domain
dig yahoo.com MX

Show the TTL for a given domain
dig imagescape.com TTL

Show the TXT record for an yahoo domain
dig yahoo.com TXT

Show only the answer
dig yahoo.com +nocomments +noquestion +noauthority +noadditional +nostats

All Records
dig @8.8.8.8 imagescape.com ANY +noall +answer

Reverse Lookup
dig -x 198.178.132.10

Nice output that looks like the domain record
dig @ns.imagescape.com imagescape.com any +multiline +noall +answer

Find the authoritative nameserver for a domain:
dig -t ns imagescape.com +short

Find the SOA of the domain:
dig joejasinski.com soa +noall +answer


dig @ns1.linode.com joejasinski.com soa

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @ns1.linode.com joejasinski.com soa
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- 64217="" id:="" noerror="" opcode:="" p="" query="" status:="">;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 5, ADDITIONAL: 11
;; WARNING: recursion requested but not available

;; ANSWER SECTION:
joejasinski.com. 86400 IN SOA ns1.linode.com. joe\.jasinski.gmail.com. 2014092859 14400 14400 1209600 86400


The presence "aa" indicates if the nameserver is the authoritative server.
The "2014092859" is the serial


Download a zone file (zone transfer):
dig axfr domain.com


Common Options:
  +short  = show only the results
  +[no]comments = show/hide the comments
  +[no]question = show/hide the question section
  +[no]authority = show/hide the authority section
  +[no]stats = show/hide the stats section
  +[no]answer = show/hide the answer section
  +[no]all = show/hide everything


https://neverendingsecurity.wordpress.com/2015/04/13/dig-commands-cheatsheet/
https://www.madboa.com/geek/dig/


Nice articles:

Commands
http://anouar.adlani.com/2011/12/useful-dig-command-to-troubleshot-your-domains.html

Create DNS Slave
http://www.microhowto.info/howto/configure_bind_as_a_slave_dns_server.html



Monday, November 02, 2015

Python Email Send


Simple email test from python

import smtplib
from email.mime.text import MIMEText
f = "no-reply@example.com"
t = "joe@example.com"
s = smtplib.SMTP('localhost')
msg = MIMEText("this is a test")
msg["Subject"] = "This is a test - ignore"
msg["To"] = t
msg["From"] = f
s.sendmail(f, [t], msg.as_string())

Sunday, August 23, 2015

Docker Gitlab

1) Create the following file
vim run_gitlab.sh

CONTAINER_NAME="gitlab_server"
GITLAB_DATA_DIR="`pwd`/gitlab/"
GITLAB_SSH_PORT="2222"
GITLAB_HTTP_PORT="8080"
GITLAB_HTTPS_PORT="8443"

docker rm ${CONTAINER_NAME}
sudo docker run --detach \
    --publish ${GITLAB_HTTPS_PORT}:443 \
    --publish ${GITLAB_HTTP_PORT}:8000 \
    --publish ${GITLAB_SSH_PORT}:22 \
    --name ${CONTAINER_NAME} \
    --restart always \
    --volume ${GITLAB_DATA_DIR}config:/etc/gitlab \
    --volume ${GITLAB_DATA_DIR}logs:/var/log/gitlab \
    --volume ${GITLAB_DATA_DIR}gitlab/data:/var/opt/gitlab \
    gitlab/gitlab-ce:latest

2) Run to create the initial directories
   sudo ./run_gitlab.sh 

3) Configure: modify the following settings

    sudo vim gitlab/config/gitlab.rb 

         external_url 'http://localhost:8000'
         gitlab_rails['gitlab_ssh_host'] = 'localhost'
         gitlab_rails['gitlab_shell_ssh_port'] = 2222

4) Restart docker

    sudo docker restart gitlab_server

5) Visit http://localhost:8080 and login with

    root 5iveL!fe


Source:
https://gitlab.com/gitlab-org/gitlab-ce/tree/master/docker

Autostart:
https://docs.docker.com/articles/host_integration/

Friday, July 24, 2015

Docker Redis

1) Create a script as such:
CONTAINER_NAME="redis_server"
REDIS_DATA_DIR="`pwd`/data-dir/redis/"
HOST_PORT_1="6379"

if [ ! -d "${REDIS_DATA_DIR}" ]
then
    mkdir -p "${REDIS_DATA_DIR}"
fi

docker rm ${CONTAINER_NAME}
docker run --name ${CONTAINER_NAME} --rm \
   -p ${HOST_PORT_1}:6379 \
   -v ${REDIS_DATA_DIR}:/data \
    redis

Postgres Query: Given an index find associated table

select
    t.relname as table_name,
    i.relname as index_name,
    array_to_string(array_agg(a.attname), ', ') as column_names
from
    pg_class t,
    pg_class i,
    pg_index ix,
    pg_attribute a
where
    t.oid = ix.indrelid
    and i.oid = ix.indexrelid
    and a.attrelid = t.oid
    and a.attnum = ANY(ix.indkey)
    and t.relkind = 'r'
    and i.relname = 'Index_to_search_for'
group by
    t.relname,
    i.relname
order by
    t.relname,
    i.relname;
 
 
Source:
http://stackoverflow.com/questions/2204058/list-columns-with-indexes-in-postgresql

Docker MongoDB

1) Create a directory for local data:

mkdir `pwd`/data-dir

2) Create a script as such:

CONTAINER_NAME="mongodb_server"
MONGO_DATA_DIR="`pwd`/data-dir"
HOST_PORT_1="27017"

if [ ! -d "${MONGO_DATA_DIR}" ]
then
    mkdir -p "${MONGO_DATA_DIR}"
fi

docker rm ${CONTAINER_NAME}
docker run --name ${CONTAINER_NAME} --rm \
   -p ${HOST_PORT_1}:27017 \
   -v ${MONGO_DATA_DIR}:/data/db \
   mongo

Wednesday, June 24, 2015

Django Manipulating the Session without a request



Django: Lookup a session without a request

    from django.conf import settings
    from importlib import import_module
    engine = import_module(settings.SESSION_ENGINE)
    session_store = engine.SessionStore("new_or_existing_sessionID")
    uid = session_store.get('_auth_user_id')

    try:
        del session_store['session_key']
    except IndexError:
        pass

    session_store.save()

Tuesday, May 12, 2015

GlusterFS Configuration

Assume node1 is: 192.168.33.10
Assume node2 is: 192.168.33.11

Sources:
https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers
http://www.jamescoyle.net/how-to/457-glusterfs-firewall-rules


On BOTH nodes

# set hosts file entry (or configure DNS)
vim /etc/hosts
192.168.33.10 gluster1
192.168.33.11 gluster2

# On Ubuntu: install requirements 
sudo add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.5
sudo apt-get update
sudo apt-get install glusterfs-server
sudo apt-get install glusterfs-client

# On Centos: install requirements
wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/RHEL/glusterfs-epel.repo
yum install glusterfs-server
yum install glusterfs-client

# make storage location for gluster data
sudo mkdir /gluster-storage

# make a mountpoint
sudo mkdir /mnt/media/

ON GLUSTER 1

# connect node 1 and node 2
sudo gluster peer probe gluster2
sudo gluster peer status

# create a Gluster replica volume
sudo gluster volume create volume1 replica 2 transport tcp \
      gluster1:/gluster-storage \
      gluster2:/gluster-storage force

# list volumes
sudo gluster volume list

# start the gluster volume
sudo gluster volume start volume1

# mount the volume
sudo mount -t glusterfs gluster1:/volume1 /mnt/media/

# only allow given ips
sudo gluster volume set volume1 auth.allow gluster1,gluster2

sudo gluster volume info
sudo gluster volume profile volume1 start
sudo gluster volume profile volume1 info
sudo gluster volume status

# stop the volume
sudo gluster volume stop volume1

FSTAB Entries

** ON Gluster 1
gluster1:/volume1 /mnt/media/ glusterfs defaults,_netdev,backupvolfile-server=gluster2 0 0

** ON Gluster 2
gluster2:/volume1 /mnt/media/ glusterfs defaults,_netdev,backupvolfile-server=gluster1 0 0

systemd quickstart


Source:
https://fedoraproject.org/wiki/SysVinit_to_Systemd_Cheatsheet
https://wiki.ubuntu.com/SystemdForUpstartUsers
http://wiki.nginx.org/FedoraSystemdServiceFile

Start/stop/reload a service:
service foo start   # or
systemctl start foo

service foo stop   # or 
systemctl stop foo

service foo restart   # or 
systemctl restart foo

Reload a service using reload signal:
service foo reload

See if service is running:
service foo status   # or 
systemctl status foo

See tree listing of all services:
systemctl status

See job log: 
sudo journalctl -u foo -f

Enable a service (to start on boot):
systemctl enable foo

Disable a service:
systemctl disable foo

Service Definition Directory:
/lib/systemd/system/

Example Nginx Service: /lib/systemd/system/nginx.service 
[Unit]
Description=The nginx HTTP and reverse proxy server
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target


Links:
https://scottlinux.com/2014/12/08/how-to-create-a-systemd-service-in-linux-centos-7/

Saturday, May 09, 2015

Simple ClamAV Usage


Source: Nice Resource
http://askubuntu.com/questions/250290/how-do-i-scan-for-viruses-with-clamav

Install:
sudo apt-get install clamav

Refresh virus definition database:
sudo freshclam

Scan all files on computer showing each
clamscan -r /

Only display infected files and ring bell
clamscan -r --bell -i /

Scan all files and move them to infected dir
clamscan -r --move=/home/USER/VIRUS /home/USER

Scan all files and remove bad ones
clamscan -r --remove /home/USER

Friday, April 03, 2015

Python Tokenlib

Make an expiring token using a known secret.

Source:
https://github.com/mozilla-services/tokenlib

Make a token, hashed with a secret, and storing an IP, that expires in 12 seconds
>>> t = tokenlib.make_token({"ip":"1.1.1.1."}, secret="1234567890", timeout=12)

Parse a token before expiration time is up
>>> tokenlib.parse_token(t, secret="1234567890")
{u'ip': u'1.1.1.1.', u'expires': 1428098130.01954, u'salt': u'fbd08e'}

Parse a token after expiration time is up
>>> tokenlib.parse_token(t, secret="1234567890")
Traceback (most recent call last):
  File "", line 1, in
  File "tokenlib/__init__.py", line 186, in parse_token
    return TokenManager(**kwds).parse_token(token, now=now)
  File "/tokenlib/__init__.py", line 143, in parse_token
    raise errors.ExpiredTokenError()
tokenlib.errors.ExpiredTokenError: token has expired