Archive for December, 2013

The Zen of Python

In [1]: import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Yahoo Mail fail

After not being able to access your yahoo email account for 3 days, and losing 2 weeks worth of email from your mail folders (including inbox,drafs, sent…), what do you get from Yahoo? A free Premium Yahoo Mail servers for a month.

This is the message from yahoo —

————————————–BEGIN———————————
We're working to restore your inbox.

We want to apologize again for the frustration last week's outage caused and give you an update on your Yahoo Mail account.

Our engineering team is working around the clock to restore the messages and folders that still may not be appearing in your inbox. We will send you an update once the process is complete for your account.

While we know we can't make up for the inconvenience we've caused you, we're upgrading your account to our Premium Yahoo Mail service for January 2014, for free.

If you have questions or want more help with your account, please reach out to Customer Care or call 1-800-318-0612.

The Yahoo Mail Team

P.S. If you already have a Premium account, you will receive an extra month of free service.
————————————–END———————————

P.S. Yahoo has finally recovered all the user emails, which is a good thing.

Linux security tip of the day

Users accounts usually get created and removed on most Development or Production servers. It is not uncommon to simply delete the users and yet not either delete or change the ownership of all files and directories associate with that user or user/group id. Some of the files might not be in the home directory of that user, so it is a good idea to search the whole file system for any files not owned by non-existent user or group. This is a big security issue, as an account might be created in the future with the same user or group id of the deleted account and end up having complete ownership of the files which don’t belong to them.

Solution – search ‘un-owned’ files and either change their ownership to ‘root:root’ or move them to some backup storage.


[root@danasmera ~]# declare -a no_user_files
[root@kauai ~]# for myfile in $(egrep '(ext2|ext3|ext4)' /etc/fstab | awk '{print $2}')
do
find $myfile -xdev \( -type f -o -type d \) -nouser -print
done

[root@danasmera ~]#for myfile in ${no_user_files[@]}; do chown  root:root $myfile;done

Follow similar steps for files/directories owned by non-existent domains.

[root@danasmera ~]# declare -a no_group_files
[root@danasmera ~]# for myfile in $(egrep '(ext2|ext3|ext4)' /etc/fstab | awk '{print $2}')
do
find $myfile -xdev \( -type f -o -type d \) -nogroup -print
done

[root@danasmera ~]#for myfile in ${no_group_files[@]}; do chown  root:root $myfile;done

For more information on hardening your Operating system or application, go to the Center for Internet Security website, an download the freely available Benchmarks. The Benchmarks are ‘scorable’, easy to follow steps by step instructions on how to secure you box.

Problem: every time a user logs in, they get “Could not chdir to home directory….Permission denied” error, although they can login to the system and change to their home directories without any problem.

Cause in this particular case: The system had a separate LVM partition for /home, and the partition crashed at one point, and was gone for good. I had to create a new LVM for the /home directory, and apparently SELinux doesn’t seem to like the security context as shown below.

-See the error below

[daniel@danasmera.com ~]$ ssh daniel@localhost
daniel@localhost's password:
Last login: Wed Dec 11 09:48:56 2013 from localhost.localdomain
Could not chdir to home directory /home/daniel: Permission denied

-No login or changing to home directory issue here.

[daniel@danasmera.com /]$ cd /home/daniel/
[daniel@danasmera.com ~]$ pwd
/home/daniel

-SELinux is enabled and in enforcing mode

[daniel@danasmera.com ~]$ sudo sestatus
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   enforcing
Mode from config file:          enforcing
Policy version:                 24
Policy from config file:        targeted

-Let us set SELinux into permissive mode to see if that is the cause.

[root@danasmera.com ~]# setenforce 0
 
 
[root@danasmera.com ~]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   permissive
Mode from config file:          enforcing
Policy version:                 24
Policy from config file:        targeted

[daniel@danasmera.com ~]$ ssh daniel@localhost
daniel@localhost's password:
Last login: Wed Dec 11 09:50:11 2013 from localhost.localdomain

(No error message anymore!)..Now let us try to resolve the SELinux issue

-Let us display the security context for home

[root@danasmera.com ~]# ls -dZ /home
drwxr-xr-x. root root system_u:object_r:file_t:s0      /home

-Time to restore to default SELinux security context

[root@danasmera.com ~]# restorecon -v /home
restorecon reset /home context system_u:object_r:file_t:s0->system_u:object_r:home_root_t:s0

-Let us enable SELinux

[root@danasmera.com ~]# setenforce 1

-Error message disappears!

[daniel@danasmera.com ~]$ ssh daniel@localhost
daniel@localhost's password:
Last login: Wed Dec 11 09:52:11 2013 from localhost.localdomain

View all posts in this blog – https://linuxfreelancer.com/all-posts

This is my first attempt in response to a question posed in one of StackExchange sites for Unix/Linux – How do you compare two folders and copy the difference to a third folder?. The scripts compares the latest directory, given as argument one, to an old directory, argument two, and creates a difference directory if it doesn’t exist, third argument, and copies the files and directories which exist only in latest directory into the difference directory. It also copies files which are different in latest directory as compared to the old one, to the difference directory. Make sure to put the arguments in the right order – latest directory first, old directory next, and the difference directory last.

Sample usage:

daniel@linubuvma:~/scripts/python$ python copy_difference.py /tmp/test/current /tmp/test/old /tmp/test/difference

(Silent output is good).

daniel@linubuvma:~/practice/python$ ls -1R /tmp/test/current/
/tmp/test/current/:
dirc
extra
newone
one
three
two

/tmp/test/current/dirc:

/tmp/test/current/extra:
extra2
fourth

/tmp/test/current/extra/extra2:

/tmp/test/current/newone:
file2
fileone
daniel@linubuvma:~/practice/python$ ls -1R /tmp/test/old
/tmp/test/old:
extra
newone
one
two

/tmp/test/old/extra:

/tmp/test/old/newone:
file2
daniel@linubuvma:~/practice/python$ ls -1R /tmp/test/difference
ls: cannot access /tmp/test/difference: No such file or directory
daniel@linubuvma:~/practice/python$ python copy_difference.py /tmp/test/current /tmp/test/old /tmp/test/difference
daniel@linubuvma:~/practice/python$ ls -1R /tmp/test/difference
/tmp/test/difference:
extra
newone
three
two

/tmp/test/difference/extra:
fourth

/tmp/test/difference/newone:
fileone

Here is the Python script.


#!/usr/bin/env python

import os, sys
import filecmp
import re
import shutil
holderlist=[]

def compareme(dir1, dir2):
    dircomp=filecmp.dircmp(dir1,dir2)
    only_in_one=dircomp.left_only
    diff_in_one=dircomp.diff_files
    dirpath=os.path.abspath(dir1)
    [holderlist.append(os.path.abspath( os.path.join(dir1,x) )) for x in only_in_one]
    [holderlist.append(os.path.abspath( os.path.join(dir1,x) )) for x in diff_in_one]
    if len(dircomp.common_dirs) > 0:
        for item in dircomp.common_dirs:
            compareme(os.path.abspath(os.path.join(dir1,item)), os.path.abspath(os.path.join(dir2,item)))
        return holderlist

def main():
 if len(sys.argv) > 3:
   dir1=sys.argv[1]
   dir2=sys.argv[2]
   dir3=sys.argv[3]
 else:
   print "Usage: ", sys.argv[0], "currentdir olddir difference"
   sys.exit(1)

 if not dir3.endswith('/'): dir3=dir3+'/'

 source_files=compareme(dir1,dir2)
 dir1=os.path.abspath(dir1)
 dir3=os.path.abspath(dir3)
 destination_files=[]
 new_dirs_create=[]
 for item in source_files:
   destination_files.append(re.sub(dir1, dir3, item) )
 for item in destination_files:
  new_dirs_create.append(os.path.split(item)[0])
 for mydir in set(new_dirs_create):
   if not os.path.exists(mydir): os.makedirs(mydir)
#copy pair
 copy_pair=zip(source_files,destination_files)
 for item in copy_pair:
   if os.path.isfile(item[0]):
    shutil.copyfile(item[0], item[1])

if __name__ == '__main__':
 main()

Getting the URLs in your favorites or bookmarks as a plain list.

I have tons of pages that i bookmarked in my Firefox browser in a Linux box and wanted to get a simple listing of these URLs with titles.

1. Export books marks to a JSON file
2. Extract JSON file to get a simple list

1. How to Export bookmars in Firefox as JSON.
Go to Bookmarks menu
Show All Bookmarks
Import and Backup (click the down arrow to expand it)
Backup
Save (Make sure JSON is selected at the right bottom corner)

The file will be saved something like ‘bookmarks-2013-12-07.json’, the format is ‘bookmarks-yyyy-mm-dd.json’. Write down the path where you saved this file, we will need it for the next step.

2. Get a simple list out of the JSON format file

We are going to use the json module for python to load the file into a python list object and print the lines containing URLs. Make sure you set the ‘bookmarks_path’ variable to the path where you saved the bookmarks file.


#!/usr/bin/env python
'''extract a list of URLs from Firefox exported bookmars JSON file '''

import sys
import os
import json
import io

def Usage():
    print "{0} Path-to-bookmarks-file".format(sys.argv[0])
    sys.exit(1)

if len(sys.argv) < 2:
    Usage()

bookmark_file = sys.argv[1]

#Does the file exist?
if not os.path.isfile(bookmark_file):
    print "{0} not found.".format(bookmark_file)
    sys.exit(1)

# Load JSON file
fp_data = io.open(bookmark_file, encoding='utf-8')
try:
    jdata = json.load(fp_data)
except ValueError:
    print "{0} not valid JSON file".format(bookmark_file)
    sys.exit(1)
fp_data.close()


#Recursive function to get the title and URL keys from JSON file

def grab_keys(bookmarks_data, bookmarks_list=[]):
  if 'children' in bookmarks_data:
    for item in bookmarks_data['children']:
      bookmarks_list.append({'title': item.get('title', 'No title'),
                             'uri': item.get('uri', 'None')})
      grab_keys(item, bookmarks_list)
  return bookmarks_list


def main():
  mydata=grab_keys(jdata)
  for item in mydata:
    myurl = item['uri']
    if myurl.startswith('http') or myurl.startswith('ftp'):
      print item['uri'], "  ", item['title']

if __name__=="__main__":
  main()

Save this file, say as ‘get_bookmars.py’, and running it will give an output similar to the one below –

[root@localhost]# python get_bookmarks.py
https://www.google.com/ Google
https://aws.amazon.com/ Amazon Web Services, Cloud Computing: Compute, Storage, Database
http://docs.python.org/3/py-modindex.html Python Module Index รข Python v3.3.3 documentation
http://www.linuxhomenetworking.com/wiki/#.UqMjHddn21E Linux Home Networking
http://www.zytrax.com/books/dns/ DNS for Rocket Scientists - Contents
http://www.centos.org/ Centos
http://wiki.centos.org/ Wiki
http://www.centos.org/docs/6/ Documentation
http://www.centos.org/modules/newbb/ Forums

Another way of approaching the problem is to export the bookmarks as HTML file and then dump it as text file. Here I used ‘lynx’ (Install it using ‘yum install lynx’ in CentOS/RHEL/Fedora) to dump the file and grepped for the URLs –

[root@localhost]# lynx –dump bookmarks.html | egrep ‘[0-9]+\.[[:space:]]+http’
3. https://www.google.com/
4. https://aws.amazon.com/
5. http://docs.python.org/3/py-modindex.html
6. http://www.linuxhomenetworking.com/wiki/#.UqMjHddn21E
7. http://www.zytrax.com/books/dns/
9. http://www.centos.org/
10. http://wiki.centos.org/
11. http://www.centos.org/docs/6/
12. http://www.centos.org/modules/newbb/

[root@localhost]# lynx –dump bookmarks.html | egrep ‘[0-9]+\.[[:space:]]+http’ | awk ‘{print $2}’
https://www.google.com/
https://aws.amazon.com/
http://docs.python.org/3/py-modindex.html
http://www.linuxhomenetworking.com/wiki/#.UqMjHddn21E
http://www.zytrax.com/books/dns/
http://www.centos.org/
http://wiki.centos.org/
http://www.centos.org/docs/6/
http://www.centos.org/modules/newbb/