Linux – Sort IPv4 addresses numerically

A novice user’s first attempt to sort a list of IP addresses would be to use ‘sort -n’, that is a numeric-sort option for sort command. Unfortunately, this will sort only the first quadrant of the IP address preceding the initial dot(‘.’). Definitely the GNU sort command does support sorting IPv4 addresses in numeric order, we just have to specify the right options.

Question to answer –

1. What is our delimiter for IPv4? dot.
2. What type of sorting? numeric.
3. How many fields? four.

Reading the man page for sort provides an option for each – 1) -t. 2) -n 3)-k
The third part might need clarification – since we have dot as a separator, the IP address will have four fields. We need to give sort a key specification (-k), with start and stop positions i.e to story by first quadrant(-k1,1), followed by second(-k2,2), followed by third(-k3,3) and finally by fourth(-k4,4).

The full command looks like this –

sort -t. -n -k1 -k2 -k3 -k4 /tmp/ipv4_file.txt

Let us use ForgeryPy to generate random Ipv4 addresses, we will write a simple python script to generate these random IPs to a file.

First install ForgeryPY –

pip install ForgeryPY

Script to generate IPv4 addresses –

$cat ipv4_generator.py

#!/usr/bin/env python

import forgery_py

uniq_ipv4=set()
for i in range(50):
    uniq_ipv4.add(forgery_py.internet.ip_v4())

with open('/tmp/ipv4_addresses.txt', 'w') as fp:
     for line in uniq_ipv4:
         fp.writelines(line+'\n')

Output –

daniel@linubuvma:/tmp$ cat /tmp/ipv4_addresses.txt
cat: /tmp/ipv4_addresses.txt: No such file or directory
daniel@linubuvma:/tmp$ python ipv4_generator.py
daniel@linubuvma:/tmp$ cat /tmp/ipv4_addresses.txt
222.21.147.97
187.234.9.45
144.101.36.131
31.192.196.59
24.16.131.84
8.52.22.181
17.40.228.224
58.164.169.156
234.78.147.45
254.150.145.225
167.111.243.3
168.168.248.227
68.104.225.196
55.138.152.3
223.30.151.183
235.245.57.76
226.122.222.107
176.199.0.130
13.68.133.125
14.157.155.254
11.155.170.92
249.0.112.141
228.209.60.62
246.130.20.235
113.17.65.20
120.76.166.133
81.191.49.37
17.226.209.151
81.184.136.140
9.172.35.65
129.205.96.54
181.130.8.142
21.78.73.162
5.216.102.88
91.140.115.96
134.140.243.193
177.148.152.60
175.37.63.212
60.175.123.112
176.250.114.170
54.62.22.255
182.78.64.216
238.92.143.140
181.206.65.80
11.139.192.62
38.158.146.36
241.236.161.184
30.223.32.242
233.107.53.70
36.222.68.164
daniel@linubuvma:/tmp$

Let us sort it –

daniel@linubuvma:/tmp$ sort -n -t. -k1,1 -k2,2 -k3,3 -k4,4 /tmp/ipv4_addresses.txt
5.216.102.88
8.52.22.181
9.172.35.65
11.139.192.62
11.155.170.92
13.68.133.125
14.157.155.254
17.40.228.224
17.226.209.151
21.78.73.162
24.16.131.84
30.223.32.242
31.192.196.59
36.222.68.164
38.158.146.36
54.62.22.255
55.138.152.3
58.164.169.156
60.175.123.112
68.104.225.196
81.184.136.140
81.191.49.37
91.140.115.96
113.17.65.20
120.76.166.133
129.205.96.54
134.140.243.193
144.101.36.131
167.111.243.3
168.168.248.227
175.37.63.212
176.199.0.130
176.250.114.170
177.148.152.60
181.130.8.142
181.206.65.80
182.78.64.216
187.234.9.45
222.21.147.97
223.30.151.183
226.122.222.107
228.209.60.62
233.107.53.70
234.78.147.45
235.245.57.76
238.92.143.140
241.236.161.184
246.130.20.235
249.0.112.141
254.150.145.225

Hope this help.

http://man7.org/linux/man-pages/man1/sort.1.html
https://pypi.python.org/pypi/ForgeryPy

How to be your own Certificate Authority(CA) with self signed certificates

This is a hands on tutorial on how you can setup your own Certificate Authority(CA) for internal network use. Once the CA certs are setup, you will generate certificate request(CSR) for your clients and sign them with your CA certs to create SSL certs for your internal network use. If you import your CA certs to your browser, you will be able to visit all internal sites using https without any browser warning, as long as the certs the your internal services are using are signed by your internal CA.

*Demo – Own CA for the home.net internal domain

1. Prepare certificate environment and default parameters to use when creating CSR –

# mkdir /etc/ssl/CA
# mkdir /etc/ssl/newcerts
# sh -c "echo '100000' > /etc/ssl/CA/serial"
# touch /etc/ssl/CA/index.txt

# cat /etc/ssl/openssl.cnf
 dir		= /etc/ssl		# Where everything is kept
 database	= $dir/CA/index.txt	# database index file.
 certificate	= $dir/certs/home_cacert.pem 	# The CA certificate
 serial		= $dir/CA/serial 		# The current serial number
 private_key	= $dir/private/home_cakey.pem  # The private key
 default_days	= 1825			# how long to certify for
 default_bits		= 2048
 countryName_default		= US
 stateOrProvinceName_default	= California
 0.organizationName_default	= Home Ltd

2. Create self signed root certificate and install the root certificate and key –

# openssl req -new -x509 -extensions v3_ca -keyout home_cakey.pem -out home_cacert.pem -days 3650
# mv home_cakey.pem /etc/ssl/private/
# mv home_cacert.pem /etc/ssl/certs/

3. Generate a CSR for the domain you want to issue a certificate –

# openssl genrsa -des3 -out home_server.key 2048
# openssl rsa -in home_server.key -out server.key.insecure
# mv server.key server.key.secure
# mv server.key.insecure server.key

4. Create the CSR now and generate a CA signed certificate –

# openssl req -new -key server.key -out server.csr
# openssl ca -in server.csr -config /etc/ssl/openssl.cnf

Directory structure after signing and issuing certificates –

# ls -l /etc/ssl/CA/
total 24
-rw-r--r-- 1 root root 444 Aug 29 18:20 index.txt
-rw-r--r-- 1 root root  21 Aug 29 18:20 index.txt.attr
-rw-r--r-- 1 root root  21 Aug 29 18:16 index.txt.attr.old
-rw-r--r-- 1 root root 328 Aug 29 18:18 index.txt.old
-rw-r--r-- 1 root root   7 Aug 29 18:20 serial
-rw-r--r-- 1 root root   7 Aug 29 18:19 serial.old

# ls -l /etc/ssl/newcerts/
total 32
-rw-r--r-- 1 root root 4612 Aug 29 16:24 100000.pem
-rw-r--r-- 1 root root 4613 Aug 29 16:51 100001.pem
-rw-r--r-- 1 root root 4574 Aug 29 17:50 100002.pem
-rw-r--r-- 1 root root 4619 Aug 29 18:20 100003.pem

# cat /etc/ssl/CA/index.txt
V	190828202443Z		100000	unknown	/C=US/ST=California/O=Home Ltd/OU=Home/CN=www.home.net/emailAddress=daniel@home.net
V	190828205127Z		100001	unknown	/C=US/ST=California/O=Home Ltd/OU=Home/CN=wiki.home.net/emailAddress=daniel@home.net
V	190828215006Z		100002	unknown	/C=US/ST=California/O=Home Ltd/CN=home.net/emailAddress=daniel@home.net
V	190828222038Z		100003	unknown	/C=US/ST=California/O=Home Ltd/OU=Home/CN=homevm.home.net/emailAddress=daniel@home.net

# cat /etc/ssl/CA/serial
10411A

Now that you have your certificate, in this example /etc/ssl/certs/home_cacert.pem, you can import it to your web client such as a web browser, LDAP client etc.

References –

https://help.ubuntu.com/12.04/serverguide/certificates-and-security.html

Server refused to allocate pty : pseudoterminal in use reached maximum allowed limit.

You are unlikely to encounter this error in most cases, as the default maximum number of pseudoterminal(pty) in a Linux environment is large enough for typical use cases. The error might occur though under either an admin lowering the pty limit or unusual high number of connections to the system, using ssh or GUI terminal. Under those circumstances, you will see the below error during ssh interaction –

$ssh daniel@192.168.10.103
daniel@192.168.10.103's password:
Server refused to allocate pty

GUI terminal error –

There was an error creating the child process for this terminal
getpt failed: No such file or directory

Per the man page –

” The Linux kernel imposes a limit on the number of available UNIX 98
pseudoterminals. In kernels up to and including 2.6.3, this limit is
configured at kernel compilation time (CONFIG_UNIX98_PTYS), and the
permitted number of pseudoterminals can be up to 2048, with a default
setting of 256. Since kernel 2.6.4, the limit is dynamically
adjustable via /proc/sys/kernel/pty/max, and a corresponding file,
/proc/sys/kernel/pty/nr, indicates how many pseudoterminals are
currently in use.
”

To resolve this, get a count of pty currently allocated using either of the below commands –

[root@kauai tmp]# sysctl kernel.pty.nr
kernel.pty.nr = 10

[root@kauai tmp]# cat /proc/sys/kernel/pty/nr 
10

You can list the allocated pts names –

# ps aux |grep -o -P '\s+pts/\d+\s+' |sort -u
 pts/0    
 pts/1    
 pts/2    
 pts/3    
 pts/4    
 pts/5    
 pts/6    
 pts/8    
 pts/9    

If the currently allocated count is closer or less than to the limit, which you can find using

/proc/sys/kernel/pty/max

, go ahead increase the max limit as follows, say to 4096 in this example –

sysctl -w kernel.pty.max=4096

References –

http://man7.org/linux/man-pages/man7/pty.7.html

AIDE (Advanced Intrusion Detection Environment) setup

AIDE is a host-based file and directory integrity checking tool, similar to Tripwire. It creates a snapshot of file details during initialization and stores them in a database. The files that AIDE monitors are user-defined rules, where the admin can specify which directories/files to keep an eye on. The snapshot is basically a message digest of the files/directories information returned by stat command. One AIDE is initialized, it can detect any changes in the future and alert the admin of such changes. AIDE can be configured to run on a scheduled based using cron jobs for instance.

Installation

yum list aide
yum install aide

Initialization

Create AIDE DB – stores snapshot of file or directory stats by scanning the monitored resources.

$ /usr/sbin/aide --init 
$ mv /var/lib/aide/aidb.db.new.gz /var/lib/aide/aide.db.gz

To minimize false positives – Set PRELINKING=no in /etc/sysconfig/prelink and run

 /usr/sbin/prelink -ua 

to restore the binaries to their prelinked state.

Scheduled integrity checks
Add a cron job to check file integrity, say every morning at 8 AM –

echo '0 8 * * * /usr/sbin/aide --check' >> /etc/crontab

Updating DB after making changes or verifying any changes reported during change –

$ aide -c aide.conf --update

References –

AIDE (Advanced Intrusion Detection Environment)

Linux – run a scheduled command once

When we think of running scheduled tasks in Linux, the first tool which comes to mind to most Linux users and admins is cron. Cron is very popular and useful when you want to run a task regularly – say after a given interval, hourly, weekly or even every time the system reboots. The scheduled tasks are faithfully executed by the crond daemon based on the scheduling we set, if somehow crond missed the task because the machine was not running 24/7, then anacron takes care of it. My topic today though is about at which executes a scheduled task only ones at a later time.

1. Adding future commands interactively –

Let us schedule to run a specific command 10 minutes from now, press CTRL+D once you have entered the command –

daniel@lindell:~$ at now +10 minutes
at> ps aux &> /tmp/at.log
[[PRESS CTRL+D HERE]]
job 4 at Wed Mar  1 21:24:00 2017

Now the above command ‘ps aux’ is scheduled to run 10 minutes from now, only once. We can check the pending jobs using atq command –

daniel@lindell:~$ atq
4	Wed Mar  1 21:24:00 2017 a daniel

2. Remove scheduled jobs from queue using atrm or at -r –

daniel@lindell:~$ at now +1 minutes
at> ps aux > /tmp/atps.logs
at> <EOT>
job 8 at Wed Mar  1 21:25:00 2017
daniel@lindell:~$ atq
8	Wed Mar  1 21:25:00 2017 a daniel
daniel@lindell:~$ atrm 8
daniel@lindell:~$ atq
daniel@lindell:~$ 

3. Run jobs from a script or file.

In some cases the job you want to run is a script –

daniel@lindell:~$ at -f /tmp/myscript.sh 8:00 AM tomorrow
daniel@lindell:~$ atq
11	Thu Mar  2 08:00:00 2017 a daniel

4. Embed shell commands inline –

at now +10 minutes <<-EOF
if [ -d ~/pythonscripts ]; then
 find ~/pythonscripts/ -type f -iname '*.pyc' -delete
fi
EOF

5. View contents of scheduled task using ‘at -c JOBNUMBER’ :

daniel@lindell:~$ at now +10 minutes <<-EOF
> if [ -d ~/pythonscripts ]; then
>  find ~/pythonscripts/ -type f -iname '*.pyc' -delete
> fi
> EOF
job 13 at Wed Mar  1 21:51:00 2017

daniel@lindell:~$ atq
11	Thu Mar  2 08:00:00 2017 a daniel
12	Wed Mar  1 21:45:00 2017 a daniel
13	Wed Mar  1 21:51:00 2017 a daniel


daniel@lindell:~$ at -c 13
 [[ TRUNCATED ENVIRONMENTAL STUFF ]]
cd /home/daniel || {
	 echo 'Execution directory inaccessible' >&2
	 exit 1
}
if [ -d ~/pythonscripts ]; then
 find ~/pythonscripts/ -type f -iname '*.pyc' -delete
fi

In this small tutorial about at utility, we saw some of the use cases for at – especially where we had to execute a scheduled task only once. The time specification it uses is human friendly, example it supports time specs such as midnight, noon, teatime or today. Feel free to read the man pages for details.

References –

https://linux.die.net/man/1/at

How to run playbooks against a host running ssh on a port other than port 22.

Ansible is a simple automation or configuration management tool, which allows to execute a command/script on remote hosts in an adhoc or using playbooks. It is push based, and uses ssh to run the playbooks against remote hosts. The below steps are on how to run ansible playbooks on a host running ssh on port 2222.

One of the hosts managed by ansible is running in a non-default port. It is a docker container listening on port 2222. Actually ssh in container listens on port 22, but the host redirect port 2222 on host to port 22 on container.

1. Use environment variable –

 ansible-playbook tasks/app-deployment.yml --check -e ansible_ssh_port=2222

2. specify the port in the inventory or hosts file –

Under hosts file set the hostname to have the format ‘server:port’ –

[docker-hosts]
docker1:2222

Let us run the playbook now –

root@linubuvma:/tmp/ansible# cat tasks/app-deployment.yml
- hosts: docker-hosts
  vars:
    app_version: 1.1.0
  tasks:
  - name: install git
    apt: name=git state=latest
  - name: Checkout the application from git
    git: repo=https://github.com/docker/docker-py.git dest=/srv/www/myapp version={{ app_version }}
    register: app_checkout_result


root@linubuvma:/tmp/ansible# ansible-playbook tasks/app-deployment.yml

PLAY [docker-hosts] ************************************************************

TASK: [install git] ***********************************************************
changed: [docker1]

TASK: [Checkout the application from git] *************************************
changed: [docker1]

PLAY RECAP ********************************************************************
docker1                    : ok=2    changed=2    unreachable=0    failed=0

References –

http://docs.ansible.com/
http://docs.ansible.com/ansible/intro_inventory.html