tcpdump – how to grep or save output in real time

Tcpdump is a handy tool for capturing network packets. It will keep on capturing packets until it receives a SIGINT or SIGTERM signal, or the specified number of packets have been processed. If you have tried to pipe the output of tcpdump to a file or tried to grep it, you will notice a significant delay before you even see an output. The reason behind that is, tcpdump buffers output in 4k byte chunks and it doesn’t flush it until 4k of data is captured.

To get around the buffering, you can use ‘-l’ option to see the packets captured in real time in order to ‘grep’ or ‘tee’ output to a file. From the man page –

-l     Make stdout line buffered.  Useful if you want to see the data while capturing it.  
     E.g. "tcpdump  -l  |  tee dat" or "tcpdump  -l   > dat  &  tail  -f  dat"

Send output to a file while watching the captured packets in real time –

root@linubuvma:~# tcpdump -l -i any -qn port 53 | tee -a /tmp/dnslogs
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
09:02:48.772892 IP 192.168.10.206.29185 > 192.168.10.109.53: UDP, length 33
09:02:48.773196 IP 192.168.10.206.35333 > 192.168.10.109.53: UDP, length 33
09:02:48.775062 IP 192.168.10.109.53 > 192.168.10.206.29185: UDP, length 78
09:02:48.775085 IP 192.168.10.109.53 > 192.168.10.206.35333: UDP, length 117
09:02:50.274318 IP 192.168.10.206.46983 > 192.168.10.109.53: UDP, length 33
09:02:50.274695 IP 192.168.10.206.55061 > 192.168.10.109.53: UDP, length 33
09:02:50.275531 IP 192.168.10.109.53 > 192.168.10.206.46983: UDP, length 78
09:02:50.276384 IP 192.168.10.109.53 > 192.168.10.206.55061: UDP, length 117

Grep text pattern in real time –
root@linubuvma:~# tcpdump -l -i any -vv |grep --color -i google
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
linubuvma.home.net.34647 > ns1.home.net.domain: [bad udp cksum 0x96c1  0x4797!] 34365+ A? google.com. (28)
linubuvma.home.net.34647 > ns1.home.net.domain: [bad udp cksum 0x96c1  0x9bf1!] 12744+ AAAA? google.com. (28)
ns1.home.net.domain > linubuvma.home.net.34647: [udp sum ok] 12744 q: AAAA? google.com. 1/0/0 google.com. AAAA 2607:f8b0:4002:c07::66 (56)
ns1.home.net.domain > linubuvma.home.net.34647: [udp sum ok] 34365 q: A? google.com. 6/0/0 google.com. A 74.125.196.139, google.com. A 74.125.196.100, google.com. A 74.125.196.101, google.com. A 74.125.196.102, google.com. A 74.125.196.113, google.com. A 74.125.196.138 (124)
173 packets captured
240 packets received by filter
0 packets dropped by kernel

A handy cheat sheet for tcpdump – https://comparite.ch/tcpdumpcs
References –

http://www.tcpdump.org/tcpdump_man.html

http://unix.stackexchange.com/questions/15989/how-to-process-pipe-tcpdumps-output-in-realtime
        

Redhat satellite or Spacewalk – real time push to clients.

By default, a client waits for a set of interval (minutes) configured in /etc/sysconfig/rhn/rhnsd to pull scheduled tasks from satellite server. For instance, if a remote command is set to be executed on client or a patch is waiting to be applied, rhn_check has to wait at least for 60 minutes to pick up the task.

For real time command execution or patch or configuration deployment, the following steps have to be performed on server and client –

1. Server : Install osa-dispatcher

root:homevm:~:# rpm -q osa-dispatcher
osa-dispatcher-5.11.43-1.el6.noarch

root:homevm:~:# service osa-dispatcher status

root:homevm:~:# chkconfig osa-dispatcher on

root:homevm:~:# chkconfig osa-dispatcher --list
osa-dispatcher  0:off   1:off   2:on    3:on    4:on    5:on    6:off

2. Client : Install and enable osad (OSA daemon).

# yum install osad -y
# chkconfig osad on
# /etc/init.d/osad restart

3. Client : Make sure the deploy and run options are enabled –

# rhn-actions-control --enable-run
# rhn-actions-control --enable-deploy

# rhn-actions-control --report
deploy is enabled
diff is enabled
upload is enabled
mtime_upload is enabled
run is enabled

Extra steps in case you encounter SSL certificate issues –
OSA is picky on SSL certificte verification, make sure the right CA cert is deployed on client, and the serverURL on up2date should match with the CN on the server certificate.

1. Copy RHN certificate from satellite server to client, make sure the cert has not expired and the CN matches server name.

root:homevm:~:# openssl x509 -in /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT -noout -subject
subject= /C=US/ST=CA/L=SanFrancisco/O=home.net/OU=spacewalk.home.net/CN=homevm.home.net

root:homevm:~:# openssl x509 -in /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT -noout -dates
notBefore=Aug  2 06:04:05 2014 GMT
notAfter=Jul 27 06:04:05 2036 GMT

root:homevm:~:# scp /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT root@client:/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT

[root@blackhat rpm-gpg]# grep -i serverurl /etc/sysconfig/rhn/up2date 
serverURL=http://homevm.home.net/XMLRPC

2. If you get certificate error, during package deployment, copy the RPM GPG public keys from satellite to the clients
On Server side –

root:homevm:/etc/pki/rpm-gpg:# ls -al RPM-GPG-KEY-*
-rw-r--r-- 1 root root 1706 Nov 30  2013 RPM-GPG-KEY-CentOS-6
-rw-r--r-- 1 root root 1730 Nov 30  2013 RPM-GPG-KEY-CentOS-Debug-6
-rw-r--r-- 1 root root 1730 Nov 30  2013 RPM-GPG-KEY-CentOS-Security-6
-rw-r--r-- 1 root root 1734 Nov 30  2013 RPM-GPG-KEY-CentOS-Testing-6
-rw-r--r-- 1 root root 1649 Nov  4  2012 RPM-GPG-KEY-EPEL-6
-rw-r--r-- 1 root root 1011 Feb  5  2011 RPM-GPG-KEY-oracle

root:homevm:/etc/pki/rpm-gpg:# scp RPM-GPG-KEY-* root@client:/etc/pki/rpm-gpg

On client side -
[bash]
# rpm --import RPM-GPG-KEY-CentOS-*

References –
https://access.redhat.com/documentation/en-US/Red_Hat_Network_Satellite/5.3/html/Installation_Guide/s1-maintenance-push-clients.html

Reduce or shrink the size of non root LVM mount.

In a system with limited disk size, you might run out of disk space in one LVM mount while having plenty of space in another mount. If both LVMs are in the same volume group (VGs), you can easily take away some of the free space from one LVM and add it to the one with low disk space. Both lvreduce and lvresize commands can be used to shrink the LVM. In this example, we will use lvresize.

Note – the steps below have to be done with care, there is a potential for losing data. If the data in the existing partition is critical, make sure you take a backup.

Shrink LVM by example – we will reduce the LVM for /usr/local file system mount from 2.0G to approximately 1.5G.

1. Unmount partition after confirming no file is in use from the partition.

root:homevm:~:# df -Pvh /usr/local
/dev/mapper/vg00-lvol04  2.0G   68M  1.9G   4% /usr/local

root:homevm:~:# lsof /dev/mapper/vg00-lvol04 

root:homevm:~:# umount /usr/local/

2. Do a file system consistency check –

root:homevm:~:# e2fsck -f /dev/mapper/vg00-lvol04 
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/vg00-lvol04: 46/131072 files (0.0% non-contiguous), 25423/524288 blocks

3. Reduce the file system first, so that the logical volume is always at least as large as the file system expects it to be.

root:homevm:~:# resize2fs /dev/mapper/vg00-lvol04 1400M
resize2fs 1.41.12 (17-May-2010)
Resizing the filesystem on /dev/mapper/vg00-lvol04 to 358400 (4k) blocks.
The filesystem on /dev/mapper/vg00-lvol04 is now 358400 blocks long.

root:homevm:~:# mount /usr/local/

root:homevm:~:# lvresize -L 1500M /dev/mapper/vg00-lvol04 
  Rounding size to boundary between physical extents: 1.47 GiB
  WARNING: Reducing active and open logical volume to 1.47 GiB
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce lvol04? [y/n]: y
  Reducing logical volume lvol04 to 1.47 GiB
  Logical volume lvol04 successfully resized

root:homevm:~:# resize2fs /dev/mapper/vg00-lvol04 
resize2fs /dev/mapper/vg00-lvol04
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/vg00-lvol04 is mounted on /usr/local; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 1
Performing an on-line resize of /dev/mapper/vg00-lvol04 to 385024 (4k) blocks.
The filesystem on /dev/mapper/vg00-lvol04 is now 385024 blocks long.

root:homevm:~:# df -Pvh /usr/local
/dev/mapper/vg00-lvol04  1.5G   68M  1.4G   5% /usr/local

References –
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/Logical_Volume_Manager_Administration/

When will the SSL certificate for a site expire or in how many days will an SSL certificate expire?

If you are a system administrator, at least once in your career you might have worked with managing SSL certificates as well as making sure that SSL certificates are renewed before they expire. I have seen Linux admins using Nagios to monitor SSL certificates and get notified a few days before expiry and in some cases admins setup a cron job which polls the sites to be monitored and send out an email if any of the certs for a site are going to expire soon.

Googling for information on how to check SSL certificate expiration for a site might return results like this one on openssl s_client.

My favorite tool for getting certificate expiry is the Nagios plugin utility – check_http. The check_http script displays the exact date/time the SSL certificate for a given site expires as well as how many days are left before expiry.

Installation –

apt-get install nagios-plugins
yum install nagios-plugins-all

In my system, the plugins were installed under /usr/lib/nagios/plugins directory –

root@linubuvma:/usr/lib/nagios/plugins# cat /etc/issue
Ubuntu 14.04.5 LTS \n \l

root@linubuvma:/usr/lib/nagios/plugins# pwd
/usr/lib/nagios/plugins

root@linubuvma:/usr/lib/nagios/plugins# ls
check_apt      check_dbi       check_dns       check_host       check_ifoperstatus  check_ldap   check_mrtg         check_nntp      check_ntp_time  check_ping   check_rta_multi  check_spop   check_time   negate
check_breeze   check_dhcp      check_dummy     check_hpjd       check_ifstatus      check_ldaps  check_mrtgtraf     check_nntps     check_nwstat    check_pop    check_sensors    check_ssh    check_udp    urlize
check_by_ssh   check_dig       check_file_age  check_http       check_imap          check_load   check_mysql        check_nt        check_oracle    check_procs  check_simap      check_ssmtp  check_ups    utils.pm
check_clamd    check_disk      check_flexlm    check_icmp       check_ircd          check_log    check_mysql_query  check_ntp       check_overcr    check_real   check_smtp       check_swap   check_users  utils.sh
check_cluster  check_disk_smb  check_ftp       check_ide_smart  check_jabber        check_mailq  check_nagios       check_ntp_peer  check_pgsql     check_rpc    check_snmp       check_tcp    check_wave

How to get the expiry information?

The -C option of check_http is what we are looking for. The help page for check_http explains the -C option as below –

-C, --certificate=INTEGER
Minimum number of days a certificate has to be valid. Port defaults to 443
(when this option is used the URL is not checked.)


Let us test it if any of the sites below have certificates which expire in the coming 30 days –

root@linubuvma:/usr/lib/nagios/plugins# ./check_http -t 60 -H yahoo.com -C 30
OK - Certificate 'www.yahoo.com' will expire on 10/30/2017 23:59.

root@linubuvma:/usr/lib/nagios/plugins# ./check_http -t 60 -H gmail.com -C 30
OK - Certificate 'mail.google.com' will expire on 03/09/2017 13:34.

root@linubuvma:/usr/lib/nagios/plugins# ./check_http -t 60 -H linuxfreelancer.com -C 30
OK - Certificate 'linuxfreelancer.com' will expire on 08/12/2017 03:01.

In order for check_http to show us how many days are left before the SSL certificate expires, we give it a much longer number of days (-C) –

root@linubuvma:/usr/lib/nagios/plugins# ./check_http -t 60 -H yahoo.com -C 1000
WARNING - Certificate 'www.yahoo.com' expires in 298 day(s) (10/30/2017 23:59).

root@linubuvma:/usr/lib/nagios/plugins# ./check_http -t 60 -H gmail.com -C 1000
WARNING - Certificate 'mail.google.com' expires in 63 day(s) (03/09/2017 13:34).

root@linubuvma:/usr/lib/nagios/plugins# ./check_http -t 60 -H linuxfreelancer.com -C 1000
WARNING - Certificate 'linuxfreelancer.com' expires in 219 day(s) (08/12/2017 03:01).

If the output doesn’t show the number of days left or the status is ‘OK’, keep on increasing the number of days. The ‘-t’ option is the connection timeout in seconds. In addition to running it interactively, check_http is very useful for scripting as well as automated monitoring.

yum – dump all yum repos configuration directives

Per the man page, the yum-config-manager is “a program that can manage main yum configuration options, toggle which repositories are enabled or disabled, and add new repositories.” The details on how to use the command is in the Official Redhat documentation.

One feature that the man page does not list is how you can use the yum-config-manager to display the yum repo configuration sections/directives and options. Not only can you use it to just show the configuration in your system, but it can also help you with displaying all the options supported by yum configuration. It might be useful for scripting as well.

Installation – identify the package name:

yum whatprovides */yum-config-manager

Install package –

yum install yum-utils

Once the package is installed, the command yum-config-manager should be available –

[root@kauai /tmp]# which yum-config-manager
/usr/bin/yum-config-manager

Running yum-config-manager will dump a list of all repositories in the server, and for each repository it will list all directives, including the hidden ones.

Below is just a the truncated version of the output, the output is much more longer depending on the number of yum repositories in your system –

[root@kauai /tmp]# yum-config-manager
===================================== main =====================================
[main]
alwaysprompt = True
assumeno = False
assumeyes = False
bandwidth = 0
bugtracker_url = http://bugs.centos.org/set_project.php?project_id=19&ref=http://bugs.centos.org/bug_report_page.php?category=yum
cache = 0
cachedir = /var/cache/yum/x86_64/6
clean_requirements_on_remove = False
color = auto
color_list_available_downgrade = dim,cyan
color_list_available_install = normal
color_list_available_reinstall = bold,underline,green
color_list_available_upgrade = bold,blue
color_list_installed_extra = bold,red
color_list_installed_newer = bold,yellow
color_list_installed_older = bold
color_list_installed_reinstall = normal
color_search_match = bold
color_update_installed = normal
color_update_local = bold
color_update_remote = normal
commands = 
debuglevel = 2
depsolve_loop_limit = 100
diskspacecheck = True
distroverpkg = centos-release
downloaddir = 
downloadonly = 
enable_group_conditionals = True
enabled = True
enablegroups = True
errorlevel = 2
exactarch = True
exactarchlist = 
exclude = 
exit_on_lock = False
failovermethod = priority
ftp_disable_epsv = False
gaftonmode = False
gpgcheck = True
group_package_types = mandatory,
   default
groupremove_leaf_only = False
history_list_view = users
history_record = True
history_record_packages = yum,
   rpm
http_caching = all
installonly_limit = 5
installonlypkgs = kernel,
   kernel-bigmem,
   installonlypkg(kernel-module),
   installonlypkg(vm),
   kernel-enterprise,
   kernel-smp,
   kernel-debug,
   kernel-unsupported,
   kernel-source,
   kernel-devel,
   kernel-PAE,
   kernel-PAE-debug
installroot = /
keepalive = True
keepcache = False
kernelpkgnames = kernel,
   kernel-smp,
   kernel-enterprise,
   kernel-bigmem,
   kernel-BOOT,
   kernel-PAE,
   kernel-PAE-debug
loadts_ignoremissing = False
loadts_ignorerpm = False
localpkg_gpgcheck = False
logfile = /var/log/yum.log
mdpolicy = group:primary
metadata_expire = 21600
mirrorlist_expire = 86400
multilib_policy = best
obsoletes = True
overwrite_groups = False
password = 
persistdir = /var/lib/yum
pluginconfpath = /etc/yum/pluginconf.d
pluginpath = /usr/share/yum-plugins,
   /usr/lib/yum-plugins
plugins = True
progess_obj = 
protected_multilib = True
protected_packages = yum
proxy = False
proxy_password = 
proxy_username = 
query_install_excludes = False
recent = 7
recheck_installed_requires = True
repo_gpgcheck = False
reposdir = /etc/yum/repos.d,
   /etc/yum.repos.d
reset_nice = True
retries = 10
rpm_check_debug = True
rpmverbosity = info
showdupesfromrepos = False
skip_broken = False
ssl_check_cert_permissions = True
sslcacert = 
sslclientcert = 
sslclientkey = 
sslverify = True
syslog_device = /dev/log
syslog_facility = LOG_USER
syslog_ident = 
throttle = 0
timeout = 30.0
tolerant = True
tsflags = 
username = 
================================== repo: base ==================================
[base]
bandwidth = 0
base_persistdir = /var/lib/yum/repos/x86_64/6
baseurl = 
cache = 0
cachedir = /var/cache/yum/x86_64/6/base
cost = 1000
enabled = True
enablegroups = True
exclude = 
failovermethod = priority
ftp_disable_epsv = False
gpgcadir = /var/lib/yum/repos/x86_64/6/base/gpgcadir
gpgcakey = 
gpgcheck = True
gpgdir = /var/lib/yum/repos/x86_64/6/base/gpgdir
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
hdrdir = /var/cache/yum/x86_64/6/base/headers
http_caching = all
includepkgs = 
keepalive = True
mdpolicy = group:primary
mediaid = 
metadata_expire = 21600
metalink = 
mirrorlist = http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os&infra=stock
mirrorlist_expire = 86400
name = CentOS-6 - Base
old_base_cache_dir = 
password = 
persistdir = /var/lib/yum/repos/x86_64/6/base
pkgdir = /var/cache/yum/x86_64/6/base/packages
proxy = False
proxy_dict = 
proxy_password = 
proxy_username = 
repo_gpgcheck = False
retries = 10
skip_if_unavailable = False
ssl_check_cert_permissions = True
sslcacert = 
sslclientcert = 
sslclientkey = 
sslverify = True
throttle = 0
timeout = 30.0
username = 

References –

Redhat official documentation