The strace command allows us to trace the system calls made by a program. In this blog, I will show you how you can use strace to capture some of the syscalls made by Apache when clients make http requests. strace has several options, but here we will consider only the following options –

<code>-p : attach to the process with the process ID pid
-o filename : write the strace output to the file filename rather than to stderr
-ff : If the -o filename option is in effect, each processes trace is written to filename.pid where pid is the numeric process id of each process. 
-e expr : to filter only specific syscalls (eg. open, fstat etc.)</code>

We will attach strace to the parent process for the apache threads. With -ff specified, strace will trace all children of the parent process and saves the trace output to a file named filename.PID. We will be using ab(Apache HTTP server benchmarking tool) to generate traffic to the web server and see which files apache opens during client requests by explicitly looking for open syscall.

1. Let us find the parent process –

ns1 strace # ps xo comm,pid,ppid | grep apache2
apache2          2062     1

The PID to trace in this case is 2062.

2. Run strace command, while this is running, launch another session and run the ab command –

ns1 strace # strace -ff -o wiki.home.net -e trace=open,close -p 2062
Process 2062 attached - interrupt to quit
Process 18526 attached
Process 18531 attached
Process 18532 attached
Process 18536 attached
Process 18537 attached
Process 18538 attached
Process 18539 attached
Process 18526 detached
Process 18531 detached
Process 18532 detached
Process 18538 detached
^CProcess 2062 detached
Process 18536 detached
Process 18537 detached
Process 18539 detached


140706133405:root:homevm:/home/daniel:# ab -n 25 -c 10 http://wiki.home.net./
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking wiki.home.net. (be patient).....done


Server Software:        Apache/2.2.22
Server Hostname:        wiki.home.net.
Server Port:            80

Document Path:          /
Document Length:        0 bytes

Concurrency Level:      10
Time taken for tests:   20.595 seconds
Complete requests:      25
Failed requests:        0
Write errors:           0
Non-2xx responses:      25
Total transferred:      11300 bytes
HTML transferred:       0 bytes
Requests per second:    1.21 [#/sec] (mean)
Time per request:       8237.856 [ms] (mean)
Time per request:       823.786 [ms] (mean, across all concurrent requests)
Transfer rate:          0.54 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1  282 459.3      2    1004
Processing:  5372 6551 733.6   6846    7631
Waiting:      367 1546 734.1   1842    2629
Total:       5374 6833 991.9   7199    8049

Percentage of the requests served within a certain time (ms)
  50%   7170
  66%   7616
  75%   7819
  80%   7933
  90%   8005
  95%   8013
  98%   8049
  99%   8049
 100%   8049 (longest request)

3. Once ab completes, stop the strace command and do ls in current directory to see the output of strace command for each apache thread which was serving the http request as well as the strace output for the parent process

ns1 strace #  ls -l
total 88
-rw-r--r-- 1 root root 10617 Jul  6 13:34 wiki.home.net.18526
-rw-r--r-- 1 root root 10617 Jul  6 13:34 wiki.home.net.18531
-rw-r--r-- 1 root root 10617 Jul  6 13:34 wiki.home.net.18532
-rw-r--r-- 1 root root 10441 Jul  6 13:34 wiki.home.net.18536
-rw-r--r-- 1 root root 10441 Jul  6 13:34 wiki.home.net.18537
-rw-r--r-- 1 root root 10617 Jul  6 13:34 wiki.home.net.18538
-rw-r--r-- 1 root root 10441 Jul  6 13:34 wiki.home.net.18539
-rw-r--r-- 1 root root   581 Jul  6 13:34 wiki.home.net.2062

4. As you can see the apache parent process doesn’t serve any client requests, the child threads are the ones serving the client requests and in each strace output for the child threads we can see the files accesses/opend –

strace output for parent process - 

ns1 strace # cat wiki.home.net.2062
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(20)                               = 0
--- SIGCHLD (Child exited) @ 0 (0) ---


ns1 strace # head -25 wiki.home.net.18526
open("/proc/sys/kernel/ngroups_max", O_RDONLY) = 20
close(20)                               = 0
open("/etc/group", O_RDONLY|O_CLOEXEC)  = 20
close(20)                               = 0
open("/.htaccess", O_RDONLY|O_CLOEXEC)  = -1 ENOENT (No such file or directory)
open("/var/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/var/www/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(22)                               = 0
close(22)                               = 0
close(22)                               = 0
open("/var/www/wiki/index.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/WebStart.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/Init.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/AutoLoader.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/profiler/Profiler.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/Defines.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/normal/UtfNormalDefines.php", O_RDONLY) = 22
close(22)                               = 0
open("/var/www/wiki/includes/DefaultSettings.php", O_RDONLY) = 22

After not being able to access your yahoo email account for 3 days, and losing 2 weeks worth of email from your mail folders (including inbox,drafs, sent…), what do you get from Yahoo? A free Premium Yahoo Mail servers for a month.

This is the message from yahoo —

————————————–BEGIN———————————
We're working to restore your inbox.

P.S. If you already have a Premium account, you will receive an extra month of free service.
————————————–END———————————

P.S. Yahoo has finally recovered all the user emails, which is a good thing.

Getting the URLs in your favorites or bookmarks as a plain list.

I have tons of pages that i bookmarked in my Firefox browser in a Linux box and wanted to get a simple listing of these URLs with titles.

1. Export books marks to a JSON file
2. Extract JSON file to get a simple list

1. How to Export bookmars in Firefox as JSON.
Go to Bookmarks menu
Show All Bookmarks
Import and Backup (click the down arrow to expand it)
Backup
Save (Make sure JSON is selected at the right bottom corner)

The file will be saved something like ‘bookmarks-2013-12-07.json’, the format is ‘bookmarks-yyyy-mm-dd.json’. Write down the path where you saved this file, we will need it for the next step.

2. Get a simple list out of the JSON format file

We are going to use the json module for python to load the file into a python list object and print the lines containing URLs. Make sure you set the ‘bookmarks_path’ variable to the path where you saved the bookmarks file.

#!/usr/bin/env python
'''extract a list of URLs from Firefox exported bookmars JSON file '''

import sys
import os
import json
import io

def Usage():
    print "{0} Path-to-bookmarks-file".format(sys.argv[0])
    sys.exit(1)

if len(sys.argv) < 2:
    Usage()

bookmark_file = sys.argv[1]

#Does the file exist?
if not os.path.isfile(bookmark_file):
    print "{0} not found.".format(bookmark_file)
    sys.exit(1)

# Load JSON file
fp_data = io.open(bookmark_file, encoding='utf-8')
try:
    jdata = json.load(fp_data)
except ValueError:
    print "{0} not valid JSON file".format(bookmark_file)
    sys.exit(1)
fp_data.close()


#Recursive function to get the title and URL keys from JSON file

def grab_keys(bookmarks_data, bookmarks_list=[]):
  if 'children' in bookmarks_data:
    for item in bookmarks_data['children']:
      bookmarks_list.append({'title': item.get('title', 'No title'),
                             'uri': item.get('uri', 'None')})
      grab_keys(item, bookmarks_list)
  return bookmarks_list


def main():
  mydata=grab_keys(jdata)
  for item in mydata:
    myurl = item['uri']
    if myurl.startswith('http') or myurl.startswith('ftp'):
      print item['uri'], "  ", item['title']

if __name__=="__main__":
  main()

Save this file, say as ‘get_bookmars.py’, and running it will give an output similar to the one below –

[root@localhost]# python get_bookmarks.py
https://www.google.com/ Google
https://aws.amazon.com/ Amazon Web Services, Cloud Computing: Compute, Storage, Database
http://docs.python.org/3/py-modindex.html Python Module Index â Python v3.3.3 documentation
http://www.linuxhomenetworking.com/wiki/#.UqMjHddn21E Linux Home Networking
http://www.zytrax.com/books/dns/ DNS for Rocket Scientists - Contents
http://www.centos.org/ Centos
http://wiki.centos.org/ Wiki
http://www.centos.org/docs/6/ Documentation
http://www.centos.org/modules/newbb/ Forums

Another way of approaching the problem is to export the bookmarks as HTML file and then dump it as text file. Here I used ‘lynx’ (Install it using ‘yum install lynx’ in CentOS/RHEL/Fedora) to dump the file and grepped for the URLs –

[root@localhost]# lynx –dump bookmarks.html | egrep ‘[0-9]+\.[[:space:]]+http’
3. https://www.google.com/
4. https://aws.amazon.com/
5. http://docs.python.org/3/py-modindex.html
6. http://www.linuxhomenetworking.com/wiki/#.UqMjHddn21E
7. http://www.zytrax.com/books/dns/
9. http://www.centos.org/
10. http://wiki.centos.org/
11. http://www.centos.org/docs/6/
12. http://www.centos.org/modules/newbb/

[root@localhost]# lynx –dump bookmarks.html | egrep ‘[0-9]+\.[[:space:]]+http’ | awk ‘{print $2}’
https://www.google.com/
https://aws.amazon.com/
http://docs.python.org/3/py-modindex.html
http://www.linuxhomenetworking.com/wiki/#.UqMjHddn21E
http://www.zytrax.com/books/dns/
http://www.centos.org/
http://wiki.centos.org/
http://www.centos.org/docs/6/
http://www.centos.org/modules/newbb/

In order to use this script, you need to do certain things in advance –

1. Download youtube-dl, a script which allows you to download videos

https://github.com/rg3/youtube-dl

2. Install ffmpet: an audio/video conversion tool.
Ubuntu users can run the following commands –

  apt-get install ffmpeg libavcodec-extra-53

Note: More details can be found here.

Usage Example: –

 ./musicdownloader.sh http://www.youtube.com/watch?v=8tHu-OwzwPg BereketMengstead-mizerey.mp3

#!/bin/bash

downloader=`which youtube-dl`
ffmpeg=`which ffmpeg`
bitrate=192000

ARGC=$#
LINK=$1
FILENAME=$2
SAVEDFILE=$(basename $0)_mymusic123.mp4

if [ $ARGC -ne 2 ]; then
  echo "Usage: $(basename $0) url-link output-file"
  echo "Example: $(basename  $0)  http://www.youtube.com/watch?v=fQZNiMckKbI Azmari-ethio01.mp3"
  exit
fi

$downloader -f 18 $LINK -o $SAVEDFILE  &&  $ffmpeg -i $SAVEDFILE -f mp3 -ab $bitrate -vn $FILENAME

if [ $? -eq 0 ];
then
 echo "File saved in " $FILENAME
 rm $SAVEDFILE
fi

Eritrean music , Tigrigna music.

A link for Internet Radio – mainly Eritrean and Ethiopian Tigrigna music. Some Amharic too.

http://linuxfreelancer.com:8000/listen.pls?sid=1

Link here.

How to search Internet radio channels –

1. Go to shoutcast.com
http://www.shoutcast.com/

2. Search using a key phrase

http://www.shoutcast.com/Internet-Radio/eritrean

http://www.shoutcast.com/Internet-Radio/ethiopian

Tag – Eritrean music.

Recently I was trying to download numerous files from a certain website using a shell script I wrote. With in the script, first I used wget to retrieve the files, but I kept on getting the following error message –

HTTP request sent, awaiting response... 403 Forbidden
2012-12-30 06:17:45 ERROR 403: Forbidden.

Then hoping that this was just a wget problem, I replaced wget with curl. It turned out that Curl would actually create a file with the same name as the one being download, but to my surprise the file was not downloaded. Instead, it contained an html file with 403 Forbidden message.

403 Forbidden
Forbidden


You don't have permission to access /dir/names.txt on this server.


What was surprising is that I could download the files using Firefox, Internet Explorer, elinks and even text based browser ‘lynx’. It seems that the website was blocking access from client browsers with certain ‘User-Agent’ header field. So the trick was to simply modify the User-Agent to a ‘legitimate’ one. Both curl and wget support the altering of User-Agent header field. You can use below commands to change the User-Agent parameter –

USER_AGENT="User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12"

wget --user-agent="$USER_AGENT" -c http://linuxfreelancer.com/status.html

curl -A "$USER_AGENT" -O http://linuxfreelancer.com/status.html

In addition to wget or curl, a much easier to use CLI HTTP client httpie can be used. Passing custom HTTP headers is intuitive using httpie, installation and usage details can be accessed here. Modifying the User-Agent header using httpie is shown below –
USER_AGENT="User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12"
http http://linuxfreelancer.com/ "$USER_AGENT"

All commands –
USER_AGENT="User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12"
wget --user-agent="$USER_AGENT" -c http://linuxfreelancer.com/status.html

curl -A "$USER_AGENT" -O http://linuxfreelancer.com/status.html

http http://linuxfreelancer.com/ "$USER_AGENT"




View all posts in this blog – https://linuxfreelancer.com/all-posts