Archive for the ‘ Linux ’ Category

This script is written based on the list of U.S. federal holidays I found in Wikipedia – Wikipedia – U.S. Federal holidays. Some of the dates, such as New Year, are straight forward, as the date and month are fixed. While others require some effort, take for instance Thanksgiving, which is on the fourth Thursday of November OR Memorial day – last Monday of May.

The script is written in bash, and tested in 32 and 64 bit Ubuntu box. It will exit with an error message if you try to get the holidays for the year 2038 or above. This is a know issue with UNIX dates on 32 bit Operating Systems – UNIX: Year 2038 problem

Sample output

daniel@dnetbook:~$ /usr/local/bin/federalholidays.sh
Usage: federalholidays.sh Year
Eg. federalholidays.sh 2014


daniel@linubuvma:~$ ./federalholidays.sh 1500
New Year's Day:               Monday, January 01, 1500
Martin Luther King, Jr. Day:  Monday, January 15, 1500
Washington's Birthday:        Monday, February 19, 1500
Memorial Day:                 Monday, May 28, 1500
Independence Day:             Wednesday, July 04, 1500
Labor Day:                    Monday, September 03, 1500
Columbus Day:                 Monday, October 08, 1500
Veteran's Day:                Sunday, November 11, 1500
Thanksgiving:                 Thursday, November 22, 1500
Christmas Day:                Tuesday, December 25, 1500


daniel@linubuvma:~$ ./federalholidays.sh 2014
New Year's Day:               Wednesday, January 01, 2014
Martin Luther King, Jr. Day:  Monday, January 20, 2014
Washington's Birthday:        Monday, February 17, 2014
Memorial Day:                 Monday, May 26, 2014
Independence Day:             Friday, July 04, 2014
Labor Day:                    Monday, September 01, 2014
Columbus Day:                 Monday, October 13, 2014
Veteran's Day:                Tuesday, November 11, 2014
Thanksgiving:                 Thursday, November 27, 2014
Christmas Day:                Thursday, December 25, 2014


daniel@linubuvma:~$ ./federalholidays.sh 2500
New Year's Day:               Friday, January 01, 2500
Martin Luther King, Jr. Day:  Monday, January 18, 2500
Washington's Birthday:        Monday, February 15, 2500
Memorial Day:                 Monday, May 31, 2500
Independence Day:             Sunday, July 04, 2500
Labor Day:                    Monday, September 06, 2500
Columbus Day:                 Monday, October 11, 2500
Veteran's Day:                Thursday, November 11, 2500
Thanksgiving:                 Thursday, November 25, 2500
Christmas Day:                Saturday, December 25, 2500

Here is the whole script, feel free to modify it or report any problem –


#!/bin/bash

ARCH=$(arch)
ARGC=$#

function Usage
{

echo "Usage: $(basename $0) Year"
echo "Eg. $(basename $0) 2014"
exit 1

}

# we will need the year as argument in YYYY format
[[ $ARGC -ne 1 ]] &&  Usage

myyear="$1"
dformat='+%A, %B %d, %Y'

[[ "$myyear" -ge 2038 ]] && [[ "$ARCH" = "i686" ]] && echo 'Year 2038 problem : http://en.wikipedia.org/wiki/Year_2038_problem ' && exit 1

#We will ignore any year below 1902
[[ "$myyear" -lt 1902 ]] && [[ "$ARCH" = "i686" ]] && exit 1

##Function to get the nth day week of the month, for instance, Third Monday of March.

function nth_xday_of_month
{

my_nth=$1
my_xday=$2
my_month=$3
my_year=$4

case "$my_nth" in

1)  mydate=$(echo {01..07})
    ;;
2)  mydate=$(echo {08..14})
    ;;
3)  mydate=$(seq 15 21)
    ;;
4)  mydate=$(seq 22 28)
   ;;
5)  mydate=$(seq 29 31)
    ;;
*) echo "Echo wrong day of the week"
   exit 1
   ;;
esac


for x in $mydate; do
  nthday=$(date '+%u' -d "${my_year}${my_month}${x}")
  if [ "$nthday" -eq "$my_xday" ]; then
   date "${dformat}" -d "${my_year}${my_month}${x}"
  fi
done
}


##Memorial day - Last Monday of May.

for x in {31..01}; do y=$(date '+%u' -d "${myyear}05${x}"); if [ "$y" -eq 1 ]; then memday="${x}" ; break; fi ; done

echo "New Year's Day:              " $(date "${dformat}"  -d "${myyear}0101")
echo "Martin Luther King, Jr. Day: " $(nth_xday_of_month 3 1 01 ${myyear})
echo "Washington's Birthday:       " $(nth_xday_of_month 3 1 02 ${myyear})
echo "Memorial Day:                " $(date "${dformat}" -d "${myyear}05${memday}")
echo "Independence Day:            " $(date "${dformat}" -d "${myyear}0704")
echo "Labor Day:                   " $(nth_xday_of_month 1 1 09 ${myyear})
echo "Columbus Day:                " $(nth_xday_of_month 2 1 10 ${myyear})
echo "Veteran's Day:               " $(date "${dformat}" -d "${myyear}1111")
echo "Thanksgiving:                " $(nth_xday_of_month 4 4 11 ${myyear})
echo "Christmas Day:               " $(date "${dformat}" -d "${myyear}1225")

: <<'federal_holidays_comment'

http://en.wikipedia.org/wiki/Federal_holidays_in_the_United_States

Jan 1 - New Year's Day - 1st day of the year
Third Monday of January - Martin Luther King, Jr. Day 
Third Monday of February - Washington's Birthday
Last Monday of May - Memorial Day.
July 4 - Independence Day.
First Monday of September - Labor Day.
Second Monday of October - Columbus Day.
November 11 - Veteran's Day.
Fourth Thursday of November - Thanksgiving
December 25 - Christmas Day
federal_holidays_comment

The strace command allows us to trace the system calls made by a program. In this blog, I will show you how you can use strace to capture some of the syscalls made by Apache when clients make http requests. strace has several options, but here we will consider only the following options –

<code>-p : attach to the process with the process ID pid
-o filename : write the strace output to the file filename rather than to stderr
-ff : If the -o filename option is in effect, each processes trace is written to filename.pid where pid is the numeric process id of each process. 
-e expr : to filter only specific syscalls (eg. open, fstat etc.)</code>

We will attach strace to the parent process for the apache threads. With -ff specified, strace will trace all children of the parent process and saves the trace output to a file named filename.PID. We will be using ab(Apache HTTP server benchmarking tool) to generate traffic to the web server and see which files apache opens during client requests by explicitly looking for open syscall.

1. Let us find the parent process –

ns1 strace # ps xo comm,pid,ppid | grep apache2
apache2          2062     1

The PID to trace in this case is 2062.

2. Run strace command, while this is running, launch another session and run the ab command –

ns1 strace # strace -ff -o wiki.home.net -e trace=open,close -p 2062
Process 2062 attached - interrupt to quit
Process 18526 attached
Process 18531 attached
Process 18532 attached
Process 18536 attached
Process 18537 attached
Process 18538 attached
Process 18539 attached
Process 18526 detached
Process 18531 detached
Process 18532 detached
Process 18538 detached
^CProcess 2062 detached
Process 18536 detached
Process 18537 detached
Process 18539 detached


140706133405:root:homevm:/home/daniel:# ab -n 25 -c 10 http://wiki.home.net./
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking wiki.home.net. (be patient).....done


Server Software:        Apache/2.2.22
Server Hostname:        wiki.home.net.
Server Port:            80

Document Path:          /
Document Length:        0 bytes

Concurrency Level:      10
Time taken for tests:   20.595 seconds
Complete requests:      25
Failed requests:        0
Write errors:           0
Non-2xx responses:      25
Total transferred:      11300 bytes
HTML transferred:       0 bytes
Requests per second:    1.21 [#/sec] (mean)
Time per request:       8237.856 [ms] (mean)
Time per request:       823.786 [ms] (mean, across all concurrent requests)
Transfer rate:          0.54 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1  282 459.3      2    1004
Processing:  5372 6551 733.6   6846    7631
Waiting:      367 1546 734.1   1842    2629
Total:       5374 6833 991.9   7199    8049

Percentage of the requests served within a certain time (ms)
  50%   7170
  66%   7616
  75%   7819
  80%   7933
  90%   8005
  95%   8013
  98%   8049
  99%   8049
 100%   8049 (longest request)

3. Once ab completes, stop the strace command and do ls in current directory to see the output of strace command for each apache thread which was serving the http request as well as the strace output for the parent process

ns1 strace #  ls -l
total 88
-rw-r--r-- 1 root root 10617 Jul  6 13:34 wiki.home.net.18526
-rw-r--r-- 1 root root 10617 Jul  6 13:34 wiki.home.net.18531
-rw-r--r-- 1 root root 10617 Jul  6 13:34 wiki.home.net.18532
-rw-r--r-- 1 root root 10441 Jul  6 13:34 wiki.home.net.18536
-rw-r--r-- 1 root root 10441 Jul  6 13:34 wiki.home.net.18537
-rw-r--r-- 1 root root 10617 Jul  6 13:34 wiki.home.net.18538
-rw-r--r-- 1 root root 10441 Jul  6 13:34 wiki.home.net.18539
-rw-r--r-- 1 root root   581 Jul  6 13:34 wiki.home.net.2062

4. As you can see the apache parent process doesn’t serve any client requests, the child threads are the ones serving the client requests and in each strace output for the child threads we can see the files accesses/opend –

strace output for parent process - 

ns1 strace # cat wiki.home.net.2062
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---


ns1 strace # head -25 wiki.home.net.18526
open("/proc/sys/kernel/ngroups_max", O_RDONLY) = 20
close(20)                               = 0
open("/etc/group", O_RDONLY|O_CLOEXEC)  = 20
close(20)                               = 0
open("/.htaccess", O_RDONLY|O_CLOEXEC)  = -1 ENOENT (No such file or directory)
open("/var/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/var/www/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(22)                               = 0
close(22)                               = 0
close(22)                               = 0
open("/var/www/wiki/index.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/WebStart.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/Init.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/AutoLoader.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/profiler/Profiler.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/Defines.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/normal/UtfNormalDefines.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/DefaultSettings.php", O_RDONLY) = 22

The Zen of Python

In [1]: import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Linux security tip of the day

Users accounts usually get created and removed on most Development or Production servers. It is not uncommon to simply delete the users and yet not either delete or change the ownership of all files and directories associate with that user or user/group id. Some of the files might not be in the home directory of that user, so it is a good idea to search the whole file system for any files not owned by non-existent user or group. This is a big security issue, as an account might be created in the future with the same user or group id of the deleted account and end up having complete ownership of the files which don’t belong to them.

Solution – search ‘un-owned’ files and either change their ownership to ‘root:root’ or move them to some backup storage.


[root@danasmera ~]# declare -a no_user_files
[root@kauai ~]# for myfile in $(egrep '(ext2|ext3|ext4)' /etc/fstab | awk '{print $2}')
do
find $myfile -xdev \( -type f -o -type d \) -nouser -print
done

[root@danasmera ~]#for myfile in ${no_user_files[@]}; do chown  root:root $myfile;done

Follow similar steps for files/directories owned by non-existent domains.

[root@danasmera ~]# declare -a no_group_files
[root@danasmera ~]# for myfile in $(egrep '(ext2|ext3|ext4)' /etc/fstab | awk '{print $2}')
do
find $myfile -xdev \( -type f -o -type d \) -nogroup -print
done

[root@danasmera ~]#for myfile in ${no_group_files[@]}; do chown  root:root $myfile;done

For more information on hardening your Operating system or application, go to the Center for Internet Security website, an download the freely available Benchmarks. The Benchmarks are ‘scorable’, easy to follow steps by step instructions on how to secure you box.

Problem: every time a user logs in, they get “Could not chdir to home directory….Permission denied” error, although they can login to the system and change to their home directories without any problem.

Cause in this particular case: The system had a separate LVM partition for /home, and the partition crashed at one point, and was gone for good. I had to create a new LVM for the /home directory, and apparently SELinux doesn’t seem to like the security context as shown below.

-See the error below

[daniel@danasmera.com ~]$ ssh daniel@localhost
daniel@localhost's password:
Last login: Wed Dec 11 09:48:56 2013 from localhost.localdomain
Could not chdir to home directory /home/daniel: Permission denied

-No login or changing to home directory issue here.

[daniel@danasmera.com /]$ cd /home/daniel/
[daniel@danasmera.com ~]$ pwd
/home/daniel

-SELinux is enabled and in enforcing mode

[daniel@danasmera.com ~]$ sudo sestatus
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   enforcing
Mode from config file:          enforcing
Policy version:                 24
Policy from config file:        targeted

-Let us set SELinux into permissive mode to see if that is the cause.

[root@danasmera.com ~]# setenforce 0
 
 
[root@danasmera.com ~]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   permissive
Mode from config file:          enforcing
Policy version:                 24
Policy from config file:        targeted

[daniel@danasmera.com ~]$ ssh daniel@localhost
daniel@localhost's password:
Last login: Wed Dec 11 09:50:11 2013 from localhost.localdomain

(No error message anymore!)..Now let us try to resolve the SELinux issue

-Let us display the security context for home

[root@danasmera.com ~]# ls -dZ /home
drwxr-xr-x. root root system_u:object_r:file_t:s0      /home

-Time to restore to default SELinux security context

[root@danasmera.com ~]# restorecon -v /home
restorecon reset /home context system_u:object_r:file_t:s0->system_u:object_r:home_root_t:s0

-Let us enable SELinux

[root@danasmera.com ~]# setenforce 1

-Error message disappears!

[daniel@danasmera.com ~]$ ssh daniel@localhost
daniel@localhost's password:
Last login: Wed Dec 11 09:52:11 2013 from localhost.localdomain

View all posts in this blog – https://linuxfreelancer.com/all-posts

This is my first attempt in response to a question posed in one of StackExchange sites for Unix/Linux – How do you compare two folders and copy the difference to a third folder?. The scripts compares the latest directory, given as argument one, to an old directory, argument two, and creates a difference directory if it doesn’t exist, third argument, and copies the files and directories which exist only in latest directory into the difference directory. It also copies files which are different in latest directory as compared to the old one, to the difference directory. Make sure to put the arguments in the right order – latest directory first, old directory next, and the difference directory last.

Sample usage:

daniel@linubuvma:~/scripts/python$ python copy_difference.py /tmp/test/current /tmp/test/old /tmp/test/difference

(Silent output is good).

daniel@linubuvma:~/practice/python$ ls -1R /tmp/test/current/
/tmp/test/current/:
dirc
extra
newone
one
three
two

/tmp/test/current/dirc:

/tmp/test/current/extra:
extra2
fourth

/tmp/test/current/extra/extra2:

/tmp/test/current/newone:
file2
fileone
daniel@linubuvma:~/practice/python$ ls -1R /tmp/test/old
/tmp/test/old:
extra
newone
one
two

/tmp/test/old/extra:

/tmp/test/old/newone:
file2
daniel@linubuvma:~/practice/python$ ls -1R /tmp/test/difference
ls: cannot access /tmp/test/difference: No such file or directory
daniel@linubuvma:~/practice/python$ python copy_difference.py /tmp/test/current /tmp/test/old /tmp/test/difference
daniel@linubuvma:~/practice/python$ ls -1R /tmp/test/difference
/tmp/test/difference:
extra
newone
three
two

/tmp/test/difference/extra:
fourth

/tmp/test/difference/newone:
fileone

Here is the Python script.


#!/usr/bin/env python

import os, sys
import filecmp
import re
import shutil
holderlist=[]

def compareme(dir1, dir2):
    dircomp=filecmp.dircmp(dir1,dir2)
    only_in_one=dircomp.left_only
    diff_in_one=dircomp.diff_files
    dirpath=os.path.abspath(dir1)
    [holderlist.append(os.path.abspath( os.path.join(dir1,x) )) for x in only_in_one]
    [holderlist.append(os.path.abspath( os.path.join(dir1,x) )) for x in diff_in_one]
    if len(dircomp.common_dirs) > 0:
        for item in dircomp.common_dirs:
            compareme(os.path.abspath(os.path.join(dir1,item)), os.path.abspath(os.path.join(dir2,item)))
        return holderlist

def main():
 if len(sys.argv) > 3:
   dir1=sys.argv[1]
   dir2=sys.argv[2]
   dir3=sys.argv[3]
 else:
   print "Usage: ", sys.argv[0], "currentdir olddir difference"
   sys.exit(1)

 if not dir3.endswith('/'): dir3=dir3+'/'

 source_files=compareme(dir1,dir2)
 dir1=os.path.abspath(dir1)
 dir3=os.path.abspath(dir3)
 destination_files=[]
 new_dirs_create=[]
 for item in source_files:
   destination_files.append(re.sub(dir1, dir3, item) )
 for item in destination_files:
  new_dirs_create.append(os.path.split(item)[0])
 for mydir in set(new_dirs_create):
   if not os.path.exists(mydir): os.makedirs(mydir)
#copy pair
 copy_pair=zip(source_files,destination_files)
 for item in copy_pair:
   if os.path.isfile(item[0]):
    shutil.copyfile(item[0], item[1])

if __name__ == '__main__':
 main()