Archive for the ‘ Miscellaneous ’ Category

Google cloud platform – NEXT 2017

As of the beginning of 2017, Amazon Web Services(AWS) is the leader in Cloud based infrastructure as a service(IAS), followed by Microsoft. The cloud business is still competitive and many enterprises have yet to migrate fully to the cloud. Cloud service providers are continuously competing in the quality of service, diversity and range of services provided, price etc.

A new entrant to the Cloud business is Google, which has recently started targeting big enterprises as well as individual developers and small businesses. The core infrastructure Google had used for years internally to service global users on such services as Gmail, Google maps, Google search is now being offered to users. The Gartner magic quadrant for 2016 has put it in the visionaries quadrant

Follow NEXT in twitter
Google cloud in Facebook

To get started with Google cloud platform(GCP), go to the documentation page for GCP.

For list of solution and products offered by GCP – GCP products.

Linux – run a scheduled command once

When we think of running scheduled tasks in Linux, the first tool which comes to mind to most Linux users and admins is cron. Cron is very popular and useful when you want to run a task regularly – say after a given interval, hourly, weekly or even every time the system reboots. The scheduled tasks are faithfully executed by the crond daemon based on the scheduling we set, if somehow crond missed the task because the machine was not running 24/7, then anacron takes care of it. My topic today though is about at which executes a scheduled task only ones at a later time.

1. Adding future commands interactively

Let us schedule to run a specific command 10 minutes from now, press CTRL+D once you have entered the command –

daniel@lindell:~$ at now +10 minutes
at> ps aux &> /tmp/at.log
[[PRESS CTRL+D HERE]]
job 4 at Wed Mar  1 21:24:00 2017

Now the above command ‘ps aux’ is scheduled to run 10 minutes from now, only once. We can check the pending jobs using atq command –

daniel@lindell:~$ atq
4	Wed Mar  1 21:24:00 2017 a daniel

2. Remove scheduled jobs from queue using atrm or at -r

daniel@lindell:~$ at now +1 minutes
at> ps aux > /tmp/atps.logs
at> <EOT>
job 8 at Wed Mar  1 21:25:00 2017
daniel@lindell:~$ atq
8	Wed Mar  1 21:25:00 2017 a daniel
daniel@lindell:~$ atrm 8
daniel@lindell:~$ atq
daniel@lindell:~$ 

3. Run jobs from a script or file.

In some cases the job you want to run is a script –

daniel@lindell:~$ at -f /tmp/myscript.sh 8:00 AM tomorrow
daniel@lindell:~$ atq
11	Thu Mar  2 08:00:00 2017 a daniel

4. Embed shell commands inline –

at now +10 minutes <<-EOF
if [ -d ~/pythonscripts ]; then
 find ~/pythonscripts/ -type f -iname '*.pyc' -delete
fi
EOF

5. View contents of scheduled task using ‘at -c JOBNUMBER’ :

daniel@lindell:~$ at now +10 minutes <<-EOF
> if [ -d ~/pythonscripts ]; then
>  find ~/pythonscripts/ -type f -iname '*.pyc' -delete
> fi
> EOF
job 13 at Wed Mar  1 21:51:00 2017

daniel@lindell:~$ atq
11	Thu Mar  2 08:00:00 2017 a daniel
12	Wed Mar  1 21:45:00 2017 a daniel
13	Wed Mar  1 21:51:00 2017 a daniel


daniel@lindell:~$ at -c 13
 [[ TRUNCATED ENVIRONMENTAL STUFF ]]
cd /home/daniel || {
	 echo 'Execution directory inaccessible' >&2
	 exit 1
}
if [ -d ~/pythonscripts ]; then
 find ~/pythonscripts/ -type f -iname '*.pyc' -delete
fi

In this small tutorial about at utility, we saw some of the use cases for at – especially where we had to execute a scheduled task only once. The time specification it uses is human friendly, example it supports time specs such as midnight, noon, teatime or today. Feel free to read the man pages for details.

References –

https://linux.die.net/man/1/at

Ansible non standard ssh port

How to run playbooks against a host running ssh on a port other than port 22.

Ansible is a simple automation or configuration management tool, which allows to execute a command/script on remote hosts in an adhoc or using playbooks. It is push based, and uses ssh to run the playbooks against remote hosts. The below steps are on how to run ansible playbooks on a host running ssh on port 2222.

One of the hosts managed by ansible is running in a non-default port. It is a docker container listening on port 2222. Actually ssh in container listens on port 22, but the host redirect port 2222 on host to port 22 on container.

1. Use environment variable –


 ansible-playbook tasks/app-deployment.yml --check -e ansible_ssh_port=2222

2. specify the port in the inventory or hosts file –

Under hosts file set the hostname to have the format ‘server:port’ –

[docker-hosts]
docker1:2222

Let us run the playbook now –

root@linubuvma:/tmp/ansible# cat tasks/app-deployment.yml
- hosts: docker-hosts
  vars:
    app_version: 1.1.0
  tasks:
  - name: install git
    apt: name=git state=latest
  - name: Checkout the application from git
    git: repo=https://github.com/docker/docker-py.git dest=/srv/www/myapp version={{ app_version }}
    register: app_checkout_result


root@linubuvma:/tmp/ansible# ansible-playbook tasks/app-deployment.yml

PLAY [docker-hosts] ************************************************************

TASK: [install git] ***********************************************************
changed: [docker1]

TASK: [Checkout the application from git] *************************************
changed: [docker1]

PLAY RECAP ********************************************************************
docker1                    : ok=2    changed=2    unreachable=0    failed=0

References –

http://docs.ansible.com/
http://docs.ansible.com/ansible/intro_inventory.html

yum : show all yum directives

yum – dump all yum repos configuration directives

Per the man page, the yum-config-manager is “a program that can manage main yum configuration options, toggle which repositories are enabled or disabled, and add new repositories.” The details on how to use the command is in the Official Redhat documentation.

One feature that the man page does not list is how you can use the yum-config-manager to display the yum repo configuration sections/directives and options. Not only can you use it to just show the configuration in your system, but it can also help you with displaying all the options supported by yum configuration. It might be useful for scripting as well.

Installation – identify the package name:

yum whatprovides */yum-config-manager

Install package –

yum install yum-utils

Once the package is installed, the command yum-config-manager should be available –

[root@kauai /tmp]# which yum-config-manager
/usr/bin/yum-config-manager

Running yum-config-manager will dump a list of all repositories in the server, and for each repository it will list all directives, including the hidden ones.

Below is just a the truncated version of the output, the output is much more longer depending on the number of yum repositories in your system –

[root@kauai /tmp]# yum-config-manager
===================================== main =====================================
[main]
alwaysprompt = True
assumeno = False
assumeyes = False
bandwidth = 0
bugtracker_url = http://bugs.centos.org/set_project.php?project_id=19&ref=http://bugs.centos.org/bug_report_page.php?category=yum
cache = 0
cachedir = /var/cache/yum/x86_64/6
clean_requirements_on_remove = False
color = auto
color_list_available_downgrade = dim,cyan
color_list_available_install = normal
color_list_available_reinstall = bold,underline,green
color_list_available_upgrade = bold,blue
color_list_installed_extra = bold,red
color_list_installed_newer = bold,yellow
color_list_installed_older = bold
color_list_installed_reinstall = normal
color_search_match = bold
color_update_installed = normal
color_update_local = bold
color_update_remote = normal
commands = 
debuglevel = 2
depsolve_loop_limit = 100
diskspacecheck = True
distroverpkg = centos-release
downloaddir = 
downloadonly = 
enable_group_conditionals = True
enabled = True
enablegroups = True
errorlevel = 2
exactarch = True
exactarchlist = 
exclude = 
exit_on_lock = False
failovermethod = priority
ftp_disable_epsv = False
gaftonmode = False
gpgcheck = True
group_package_types = mandatory,
   default
groupremove_leaf_only = False
history_list_view = users
history_record = True
history_record_packages = yum,
   rpm
http_caching = all
installonly_limit = 5
installonlypkgs = kernel,
   kernel-bigmem,
   installonlypkg(kernel-module),
   installonlypkg(vm),
   kernel-enterprise,
   kernel-smp,
   kernel-debug,
   kernel-unsupported,
   kernel-source,
   kernel-devel,
   kernel-PAE,
   kernel-PAE-debug
installroot = /
keepalive = True
keepcache = False
kernelpkgnames = kernel,
   kernel-smp,
   kernel-enterprise,
   kernel-bigmem,
   kernel-BOOT,
   kernel-PAE,
   kernel-PAE-debug
loadts_ignoremissing = False
loadts_ignorerpm = False
localpkg_gpgcheck = False
logfile = /var/log/yum.log
mdpolicy = group:primary
metadata_expire = 21600
mirrorlist_expire = 86400
multilib_policy = best
obsoletes = True
overwrite_groups = False
password = 
persistdir = /var/lib/yum
pluginconfpath = /etc/yum/pluginconf.d
pluginpath = /usr/share/yum-plugins,
   /usr/lib/yum-plugins
plugins = True
progess_obj = 
protected_multilib = True
protected_packages = yum
proxy = False
proxy_password = 
proxy_username = 
query_install_excludes = False
recent = 7
recheck_installed_requires = True
repo_gpgcheck = False
reposdir = /etc/yum/repos.d,
   /etc/yum.repos.d
reset_nice = True
retries = 10
rpm_check_debug = True
rpmverbosity = info
showdupesfromrepos = False
skip_broken = False
ssl_check_cert_permissions = True
sslcacert = 
sslclientcert = 
sslclientkey = 
sslverify = True
syslog_device = /dev/log
syslog_facility = LOG_USER
syslog_ident = 
throttle = 0
timeout = 30.0
tolerant = True
tsflags = 
username = 
================================== repo: base ==================================
[base]
bandwidth = 0
base_persistdir = /var/lib/yum/repos/x86_64/6
baseurl = 
cache = 0
cachedir = /var/cache/yum/x86_64/6/base
cost = 1000
enabled = True
enablegroups = True
exclude = 
failovermethod = priority
ftp_disable_epsv = False
gpgcadir = /var/lib/yum/repos/x86_64/6/base/gpgcadir
gpgcakey = 
gpgcheck = True
gpgdir = /var/lib/yum/repos/x86_64/6/base/gpgdir
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
hdrdir = /var/cache/yum/x86_64/6/base/headers
http_caching = all
includepkgs = 
keepalive = True
mdpolicy = group:primary
mediaid = 
metadata_expire = 21600
metalink = 
mirrorlist = http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os&infra=stock
mirrorlist_expire = 86400
name = CentOS-6 - Base
old_base_cache_dir = 
password = 
persistdir = /var/lib/yum/repos/x86_64/6/base
pkgdir = /var/cache/yum/x86_64/6/base/packages
proxy = False
proxy_dict = 
proxy_password = 
proxy_username = 
repo_gpgcheck = False
retries = 10
skip_if_unavailable = False
ssl_check_cert_permissions = True
sslcacert = 
sslclientcert = 
sslclientkey = 
sslverify = True
throttle = 0
timeout = 30.0
username = 

References –

Redhat official documentation

How to locate broadband Internet service providers in your area.

The FCC keeps a database of national broadband providers and it is publicly accessible at www.broadbandmap.gov. Just enter your full address or Zip code, and it will the broadband providers in your area as well as the advertised speed. One caveat is the data was last updated on June 2014, thus you might get latest information.

I checked the database for an area which had Google Fiber for the last 9 or 10 months, and it didn’t show Google Fiber as available in that area. The database has Google Fiber Inc. as a provider listed though.

If you want to check if Google Fiber is available or coming soon to your area check https://fiber.google.com/about/.

Once nice thing about the National broadband Map is the open standards API they made available to the public. It is well documented and very easy to pull data from programmatically. The API also gives you access to Census data and demographic information.

Note – most of the queries require the FIPS state and/or county codes (Federal Information Processing Standard state code). For instance, for New York state, the FIPS code is 36. Any county within a state will have FIPS county code of state FIPS code + county FIPS code. Bronx county’s (FIPS 005) full code would be 36005, for instance.

Here is a simple python script on how to interact with the API, will use Bronx county and/or NY as an example.

Let us get the overall broadband ranking within New York state –


import requests
url='https://www.broadbandmap.gov/broadbandmap/almanac/jun2014/rankby/state/36/population/wirelineproviderequals0/county?format=json&order=asc'
r=requests.get(url).json().get('Results').get('All')
for item in r:
    print item.get('rank'), item.get('geographyName')

Output based on ranking would look like this –
1 Franklin
2 Cattaraugus
3 Allegany
4 Schoharie
5 Otsego
6 Lewis
7 Washington
8 Hamilton
9 Yates
10 Delaware
11 Steuben
12 Wyoming
13 Cayuga
14 Jefferson
15 Herkimer
16 Schuyler
17 Essex
18 Seneca
19 St. Lawrence
20 Clinton
21 Montgomery
22 Chautauqua
23 Wayne
24 Columbia
25 Greene
26 Tioga
27 Livingston
28 Tompkins
29 Rensselaer
30 Chemung
31 Genesee
32 Cortland
33 Oswego
34 Sullivan
35 Albany
36 Oneida
37 Chenango
38 Orleans
39 Fulton
40 Madison
41 Niagara
42 Ontario
43 Warren
44 Schenectady
45 Ulster
46 Erie
47 Putnam
48 Onondaga
49 Saratoga
50 Broome
51 Suffolk
52 Monroe
53 Kings
54 Queens
55 New York
56 Bronx
57 Nassau
58 Westchester
59 Richmond
60 Orange
61 Rockland
62 Dutchess

Bronx county is ranked 56 out of 62, and the data for Bronx would be –

for item in r:
    if item.get('geographyId') == '36005':
        print item
        break


{u'anyWireline': 1.0,
 u'anyWirelineError': 0.0,
 u'downloadSpeedGreaterThan3000k': 1.0,
 u'downloadSpeedGreaterThan3000kError': 0.0,
 u'geographyId': u'36005',
 u'geographyName': u'Bronx',
 u'myAreaIndicator': False,
 u'population': 1482311,
 u'providerGreaterThan3': 1.0,
 u'rank': 56,
 u'stateFips': u'36',
 u'wirelineProviderEquals0': 0.0}

There is lots more you can do with the data, feel free to dig further.

The future of Net Neutrality

FCC chairman Tom Wheeler will be resigning next month, and this is not a good news for the proponents of Net Neutrality. Tom Wheeler was the driving force behind the reclassification of broadband internet access as a telecom service. The open internet rules that the FCC approved were a great victory for the supporters of Net Neutrality. With the chairman’s departure, the fight will still continue. The opponents of Net Neutrality, primarily cable/telecom companies, did not like the reclassification of broadband Internet as a telecom service, as that will impose more government and public oversight of the Internet.

For many people, the idea of Net Neutrality is still not clear. In its simplest form, it is the principle that all packets or network traffic should be treated equally. If you pay a monthly fee of $50 to your cable company for a given bandwidth, the cable company should not interfere with your browsing. As far as the cable company is concerned, whether you visit to site A or B, or use your bandwidth to download some media, they should treat it equally. Of course, illegal sites can be blocked per the legality of content in a given country. Basically the pipe has to be agnostic of the type of traffic, source or destination of traffic etc.

That is how the Internet is right now in practice, but the telecom companies want to change it. In my view, it is how the Internet should be. An open Internet encourages innovation at the application/content layer, as any new entrants whether start ups or a kid in a basement can launch a successful product without negotiating with the cable companies. Without Net Neutrality, the cable companies can pick and chose the winners, as they will practically be the gate keepers of content. There is no limit to the amount of control they will have over the Internet, a future with no Net Neutrality is a future of multi-tiered and multi-priced Internet. What that mean is the price as well as quality of your Internet service might vary on any of the following reasons –

1. What site are you visiting? May be Comcast made a deal to priority google.com.
2. What is your source IP address? This identifies your location, AT&T might have a deal with a certain municipality or owner of an IP block.
3. From which country are your browsing? Cable company has a deal with a foreign government.
4. What type of media or content are you viewing? text/audio/video? Cable company want to block a competitor’s streaming video.
5. What time of the day or day of the week are you browsing? Cable company has a popular show that it streams through a recently acquired media company during a certain time of the day.
6. What browser are you using? Internet explorer, Firefox, Safari, Chrome? Browser maker software company has a deal with cable company.
etc.

Any publicly identifiable information that the cable companies can get from your browsing can be used for pricing purposes.

This is how I analogize Net Neutrality – imagine all the Interstate roads were owned by private companies, say companies X, Y and Z. Without any regulatory rules, the Interstate owners can negotiate with car manufacturers on what types of cars take the fastest or even safest lanes. If Ford pays company X more, the “road owner” would allow only Ford cars to take the HOV lanes, or reserve more lanes for Ford cars while limiting drivers of other car types to the slowest lanes. As a prospective car owner, you won’t just pick a car based on just mileage or driving habits, you have to do extra research to find out what kind of deals the car manufacturer has made with the road owners. Travelling long distance would be a nightmare, as the various segments of the Interstate would be owned by different companies, and companies X would charge you at a different rate than companies Y and Z. So by supporting Net Neutrality, we are agreeing to the principle that the type of car you drive should not matter, we should all abide by the same rules. This does not mean that someone who can afford a high quality car can drive faster than an older car, in the same manner that if you have a lower bandwidth package with your cable company, you might not be able to view high quality movies smoothly.

Per opensecrets.org, here is the list of the top spenders on lobbying for the year 2016, surely enough the telecom and cable companies or associations are in the top list.

 

US Chamber of Commerce $79,205,000
National Assn of Realtors $45,255,769
Blue Cross/Blue Shield $19,058,109
American Hospital Assn $15,454,734
American Medical Assn $15,290,000
Pharmaceutical Research & Manufacturers of America $14,717,500
Boeing Co $12,870,000
AT&T Inc $12,660,000
National Assn of Broadcasters $12,118,000
Alphabet Inc $11,850,000
Business Roundtable $11,530,000
Comcast Corp $10,510,000
Lockheed Martin $10,380,488
Dow Chemical $10,295,982
Southern Co $10,090,000
Northrop Grumman $9,420,000
National Cable & Telecommunications Assn $9,230,000
FedEx Corp $9,221,000
Exxon Mobil $8,840,000
Amazon.com $8,624,000

Support Net Neutrality at Freepress.net.

Check link below for a great resource which shows lobbying spend of 51 internet service providers (ISPs) dating back to 1998 –

How much does your ISP spend on lobbying?