How to locate broadband Internet service providers in your area.

The FCC keeps a database of national broadband providers and it is publicly accessible at www.broadbandmap.gov. Just enter your full address or Zip code, and it will the broadband providers in your area as well as the advertised speed. One caveat is the data was last updated on June 2014, thus you might get latest information.

I checked the database for an area which had Google Fiber for the last 9 or 10 months, and it didn’t show Google Fiber as available in that area. The database has Google Fiber Inc. as a provider listed though.

If you want to check if Google Fiber is available or coming soon to your area check https://fiber.google.com/about/.

Once nice thing about the National broadband Map is the open standards API they made available to the public. It is well documented and very easy to pull data from programmatically. The API also gives you access to Census data and demographic information.

Note – most of the queries require the FIPS state and/or county codes (Federal Information Processing Standard state code). For instance, for New York state, the FIPS code is 36. Any county within a state will have FIPS county code of state FIPS code + county FIPS code. Bronx county’s (FIPS 005) full code would be 36005, for instance.

Here is a simple python script on how to interact with the API, will use Bronx county and/or NY as an example.

Let us get the overall broadband ranking within New York state –

import requests
url='https://www.broadbandmap.gov/broadbandmap/almanac/jun2014/rankby/state/36/population/wirelineproviderequals0/county?format=json&order=asc'
r=requests.get(url).json().get('Results').get('All')
for item in r:
    print item.get('rank'), item.get('geographyName')

Output based on ranking would look like this –
1 Franklin
2 Cattaraugus
3 Allegany
4 Schoharie
5 Otsego
6 Lewis
7 Washington
8 Hamilton
9 Yates
10 Delaware
11 Steuben
12 Wyoming
13 Cayuga
14 Jefferson
15 Herkimer
16 Schuyler
17 Essex
18 Seneca
19 St. Lawrence
20 Clinton
21 Montgomery
22 Chautauqua
23 Wayne
24 Columbia
25 Greene
26 Tioga
27 Livingston
28 Tompkins
29 Rensselaer
30 Chemung
31 Genesee
32 Cortland
33 Oswego
34 Sullivan
35 Albany
36 Oneida
37 Chenango
38 Orleans
39 Fulton
40 Madison
41 Niagara
42 Ontario
43 Warren
44 Schenectady
45 Ulster
46 Erie
47 Putnam
48 Onondaga
49 Saratoga
50 Broome
51 Suffolk
52 Monroe
53 Kings
54 Queens
55 New York
56 Bronx
57 Nassau
58 Westchester
59 Richmond
60 Orange
61 Rockland
62 Dutchess

Bronx county is ranked 56 out of 62, and the data for Bronx would be –

for item in r:
    if item.get('geographyId') == '36005':
        print item
        break


{u'anyWireline': 1.0,
 u'anyWirelineError': 0.0,
 u'downloadSpeedGreaterThan3000k': 1.0,
 u'downloadSpeedGreaterThan3000kError': 0.0,
 u'geographyId': u'36005',
 u'geographyName': u'Bronx',
 u'myAreaIndicator': False,
 u'population': 1482311,
 u'providerGreaterThan3': 1.0,
 u'rank': 56,
 u'stateFips': u'36',
 u'wirelineProviderEquals0': 0.0}

There is lots more you can do with the data, feel free to dig further.

In some cases, you might want to block all users from logging in to the system or just after you login, you want to prevent everyone else from connecting to the server. During server maintenance, this could be helpful or there are use cases where only one actively logged in user has to do some work if the username is a shared account.

Solution – create the /etc/nologin file, and put the text notice as the body of the file. If a user attempts to log in to a system where this file exists, the contents of the nologin file is displayed, and the user login is terminated.

[root@kauai ~]# echo 'System is under maintenance till Dec. 24, 2PM EST.' > /etc/nologin

Now try to login to the server under non super user –

daniel@linubuvma:~$ ssh ns2
System is under maintenance till Dec. 24, 2PM EST.
Connection closed by 192.168.10.103

If your ssh configuration allows it, root user can login to the server though, the root user will still be greeted with the contents of /etc/nologin file though –

daniel@linubuvma:~$ ssh root@ns2
root@ns2's password:
System is under maintenance till Dec. 24, 2PM EST.
Last login: Sat Dec 12 01:11:35 2015 from linubuvma.home.net
[root@kauai ~]# 

Reference – https://docs.oracle.com/cd/E19683-01/806-4078/6jd6cjs3v/index.html

During user login, a Linux box might show message of the day(motd), new email, or package updates information. This is particularly common in Ubuntu boxes. In some cases, you want to prevent all these messages from being displayed as it could be delaying your login for instance.

Solution – Create a file named .hushlogin in the user’s home directory.

A typical login to an Ubuntu box might look like this –

[daniel@kauai etc]$ ssh practice
daniel@practice's password: 
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-39-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Sat Jan 10 11:37:24 EST 2015

  System load:  0.0                Processes:           290
  Usage of /:   46.1% of 45.15GB   Users logged in:     1
  Memory usage: 13%                IP address for eth0: 192.168.10.206
  Swap usage:   0%

  Graph this data and manage this system at:
    https://landscape.canonical.com/

168 packages can be updated.
63 updates are security updates.

You have new mail.
Last login: Sat Jan 10 11:37:26 2015 from linux.local

To suppress all this information, create a .hushlogin file in the users home directory and log out and login back –

daniel@linubuvma:~$ touch ~/.hushlogin

daniel@linubuvma:~$ exit
logout
Connection to practice closed.

[daniel@kauai etc]$ ssh practice
daniel@practice's password: 

daniel@linubuvma:~$ 

How to be financially independent.

For clarity I am going to define financial independence by what it is not first. There are different meanings of financial independence out there, for the sake of this post I will narrow down the scope. It is not objective definition but hopefully you get the gist of it once you finish reading this article. Financial independence does not mean full retirement where you spend the rest of your life travelling or relaxing in some pristine beaches. Neither does it mean just being out of debt or having a few months of emergency fund.

You might consider yourself to be financially independence once you get to the point where you no longer have to panic if you were to lose your job right now, or you don’t lose sleep about upcoming day to day expenses. You will reach it once you have your basic needs easily met from what your assets generate. It is possible that you might have to work, but you will no longer pick jobs on which one pays more, or keep on staying in a toxic work environment. You will have plenty of time to focus on your hobbies and personal development, and might actually end up doing creative work from your imagination, not because someone told you to do so under a deadline. With some of the tips given below, it is achievable and it is worth it.

Rule of thumb – Multiply your annual expenses by 25. If that number is closer to or better yet less than your net worth, then consider yourself financially independent. Or if 4% (7% market gain – 3% inflation) of your net worth is closer to (or less higher than) how much you need per year, you are financially independent.

1. Pay all your debt and stay out of it – financial independence and debt do not go together. There are exceptions of course, where the debt is considered as an investment such as going to medical school. No credit card debt, no mortgage for a house you can’t afford, no auto loans, no student debt for a worthless degree. Make sure to pay the ones with high interest rates, especially credit card debt. Pay it all and stay out of it! Good debt vs. bad debt.

2. Live below your means – this is no-brainer. If you are spending more than what you are bringing it, then say bye bye to financial independence. Frugal living is not death sentence.

3. Cut expenses – go check all your expenses and eliminate the ones you don’t need. Pay special attention to multiple small fee subscriptions or recurring ones. This is death by thousand cuts, they add up. Think of all the expenses that you can cut.

4. Stay healthy – Health care is expensive, I have read somewhere that sixty percent of personal bankruptcies are due to health care related expenses. Besides, you won’t enjoy your free time if you are not healthy. So take good care of your health – eat healthy food, exercise, sleep well and don’t stress out. Be healthy.

5. Have a marketable skill – one way to earn more money is to be skillful in an area which is in high demand. Finding a job will be easier and the pay would be better. This does not necessarily require college degree, there are many professions with plenty of self taught professionals. Continuously update your skills, all you need is access to the Internet and by this I don’t mean Facebook! Fastest growing occupations.

6. Take advantage of 401k, Roth IRA or traditional IRA – If your employer provides 401k, contribute to the maximum allowed. Similarly with Roth IRA, you can do this one on your own. Create an account with Vanguard or Fidelity, and start investing in low cost mutual funds, the risk is minimal. If you are adventurous you can invest in real estate, peer to peer lending etc. How to open an IRA.

7. Build passive sources of income – Put your money where it works for you, it should generate some dividend or interest. Don’t stash it in a zero interest saving account, eventually inflation alone will wipe it out. So be thoughtful of where you put your money. CD rates.

8. Have multiple income streams – Diversifying is good, make sure you don’t put all your eggs in one basket. If you have any free time – besides what you set aside for fun, family time, socializing etc. – you can use it to do freelancing, tutoring, selling stuff online and more. The passive sources of income mentioned above can fall into this category as well. Some tips on how to generate extra income.

9. Don’t give in to peer or social pressure – Stick to your plan even if your immediate circle of friends or family members have a different life style and pressure to make decisions that hinder your financial independence journely. This means no new car every two or three years, in fact buying a new car is a bad financial decision in most cases. Save for a car and pay cash for a slightly used car. Avoid keeping Up With The Joneses’, it might actually keep you broke.

10. Give – this might sound non sequitur, but remember the objective of financial independence is not the accumulation of money for the sake of money. Use money as a means to an end – in this case to personal freedom, free time, accomplishing your dreams – not hoarding money. Giving releases us from the power of money.Help the less fortunate folks. Volunteer.You are not an island. No one did it on their own, we all had someone in the past who steered us to the right direction be it a parent, relative, teacher, stranger etc. Be part of something by giving your money, time etc.

Note – these are rules of thumb and generalizations. If you have a justifiable reason and you surely know what you are doing, you can break some of these rules.

Some other sites to visit, I don’t fully agree with all the opinions on these sites, but I found them helpful in one way or another –
Mr. Money Mustache
Retire by 40
Can I retire yet.
Financial samurai
The Millionaire next door.

How to interact with web services.

Curl is the defacto CLI tool for interacting with web services and other non-HTTP services such as FTP or LDAP. Linux or Unix system administrators as well as developers love it for its ease of use and debugging capabilities. When you want to interact with web services from within scripts, curl is the number one choice.For downloading files from the web, wget is commonly used as well, but curl can way more.

Since enough has been written about curl, this post is about a tool which takes interaction with web services a lot more human friendly, with nicely formatted and colored output – httpie. It is written in Python.

Installation –

apt-get  install httpie     #(Debian/Ubuntu)
yum install httpie          #(Redhat/CentOS)

Note – although the package name is httpie, the binary file is installed as http.

When troubleshooting web services, the first thing we check is usually http request and response headers –

daniel@lindell:/$ http -p hH  httpbin.org
GET / HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: httpbin.org
User-Agent: HTTPie/0.9.2

HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 12150
Content-Type: text/html; charset=utf-8
Date: Thu, 22 Dec 2016 01:32:13 GMT
Server: nginx

Where -H is for Request headers, -h is for response headers. Similarly, -B is for request body and -b is for response body.

We can also pass more complex HTTP headers, in this case “If-Modified-Since”, the web server will return 304 if the static content i am requesting has not been modified. Moving the date a few years back, it will respond with 200 status code.

daniel@lindell:/$ http -p hH http://linuxfreelancer.com/wp-content/themes/soulvision/images/texture.jpg "If-Modified-Since: Wed, 21 Dec 2016 20:51:14 GMT"
GET /wp-content/themes/soulvision/images/texture.jpg HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: linuxfreelancer.com
If-Modified-Since:  Wed, 21 Dec 2016 20:51:14 GMT
User-Agent: HTTPie/0.9.2

HTTP/1.1 304 Not Modified
Connection: Keep-Alive
Date: Thu, 22 Dec 2016 01:39:28 GMT
ETag: "34441c-f04-4858fcd6af900"
Keep-Alive: timeout=15, max=100
Server: Apache/2.2.14 (Ubuntu)

daniel@lindell:/$ http -p hH http://linuxfreelancer.com/wp-content/themes/soulvision/images/texture.jpg "If-Modified-Since: Wed, 21 Dec 2008 20:51:14 GMT"
GET /wp-content/themes/soulvision/images/texture.jpg HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: linuxfreelancer.com
If-Modified-Since:  Wed, 21 Dec 2008 20:51:14 GMT
User-Agent: HTTPie/0.9.2

HTTP/1.1 200 OK
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Length: 3844
Content-Type: image/jpeg
Date: Thu, 22 Dec 2016 01:39:37 GMT
ETag: "34441c-f04-4858fcd6af900"
Keep-Alive: timeout=15, max=100
Last-Modified: Sat, 01 May 2010 22:23:00 GMT
Server: Apache/2.2.14 (Ubuntu)

httpie also makes passing JSON encoding as well as POST/PUT methods a lot easier. No need for formatting your payload as JSON, it defaults to JSON. Debugging is easier to with -v option, which shows the raw wire data –

daniel@lindell:/$ http -v PUT httpbin.org/put name=JoeDoe email=joedoe@gatech.edu
PUT /put HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 48
Content-Type: application/json
Host: httpbin.org
User-Agent: HTTPie/0.9.2

{
    "email": "joedoe@gatech.edu", 
    "name": "JoeDoe"
}

HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 487
Content-Type: application/json
Date: Thu, 22 Dec 2016 01:44:20 GMT
Server: nginx

{
    "args": {}, 
    "data": "{\"name\": \"JoeDoe\", \"email\": \"joedoe@gatech.edu\"}", 
    "files": {}, 
    "form": {}, 
    "headers": {
        "Accept": "application/json", 
        "Accept-Encoding": "gzip, deflate", 
        "Content-Length": "48", 
        "Content-Type": "application/json", 
        "Host": "httpbin.org", 
        "User-Agent": "HTTPie/0.9.2"
    }, 
    "json": {
        "email": "joedoe@gatech.edu", 
        "name": "JoeDoe"
    }, 
    "origin": "192.1.1.2", 
    "url": "http://httpbin.org/put"
}

I have touched just the surface of httpie here, please feel free to get more detailed information on the github repo. It has built-in JSON support, form/file upload, HTTPS, proxies and authentication, custom headers, persistent sessions etc.

Article on wget and curl from previous post.

In these series of Docker tutorials, i will walk you through a hands on experimentation with Docker. The operating system I am working on is Ubuntu 16.04.

Docker is a containerization technology which allows deployment of applications in containers. Its advantage is speed, a docker container hosting an application would be up and running in a few milliseconds.

As opposed to Virtual machines, containers run on top of the host OS. They share the host kernel. Thus you can only run a Linux container on a Linux host or machine.

Docker Installation – use this link for instructions on how to install Docker.

Installation Limitation – Docker runs on 64-bit OS only and it supports Linux kernel version 3.10 and above. You can verify this using the commands below –

root@lindell:~# arch
x86_64
root@lindell:~# uname -r
4.4.0-47-generic

Docker – terminology

Images – are the building blocks of Docker. Once created or built, they can be shared, updated and used to launch containers. No image, no containers.

Containers – are images in action. Containers give images life, containers are image plus all the ecosystem the Operating system need to run the application.

Registry – where images are stored. They can be public or private. DockerHub is a typical example of public registry.

Data volumes – persistent storage used by containers.
Dockerfile – file containing instructions to be read by Docker for building a Docker image.

Node – physical or virtual machine running Docker engine.

Our first Docker container
After installing docker and making sure that the Docker engine is running, run the commands below to check the Docker images(‘docker images’) available or if any Docker containers are running(‘docker ps’). Both commands should not return any results if this is a first time installation.

root@lindell:~# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
root@lindell:~# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED
             STATUS              PORTS               NAMES

The next step is to get a Docker image from Docker Hub. For security reasons, we are going to use only official images –

root@lindell:~# docker search --filter=is-official=true ubuntu
NAME                 DESCRIPTION                                     STARS     OFFICIAL   AUTOMATED
ubuntu               Ubuntu is a Debian-based Linux operating s...   5238      [OK]
ubuntu-upstart       Upstart is an event-based replacement for ...   69        [OK]
ubuntu-debootstrap   debootstrap --variant=minbase --components...   27        [OK]


root@lindell:~# docker run -ti ubuntu /bin/bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu

b3e1c725a85f: Pull complete
4daad8bdde31: Pull complete
63fe8c0068a8: Pull complete
4a70713c436f: Pull complete
bd842a2105a8: Pull complete
Digest: sha256:7a64bc9c8843b0a8c8b8a7e4715b7615e4e1b0d8ca3c7e7a76ec8250899c397a
Status: Downloaded newer image for ubuntu:latest

root@d1b13e2c3d3f:/# docker images
bash: docker: command not found

root@d1b13e2c3d3f:/# hostname -f
d1b13e2c3d3f

root@d1b13e2c3d3f:/# uname -r
4.4.0-47-generic

We just downloaded an official Ubuntu image and started an Ubuntu container by running /bin/bash inside the newly started container. The ‘-ti’ option runs bash interactively(-i) by allocating a pseudo-TTY(-t).

Note that – the kernel version on the container is the same as the host’s kernel version. During first run, Docker will try to find Ubuntu image in our local storage, if it can’t find it, it downloads it from Docker Hub. On next runs, starting the containers will be much faster.

If we check the images and processes running now –

root@lindell:~# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu              latest              104bec311bcd        5 days ago          129 MB
root@lindell:~# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
d1b13e2c3d3f        ubuntu              "/bin/bash"         About a minute ago   Up About a minute              

At this point, if we exit from the container, docker ps will no longer show the container as it has been terminated. We use ‘docker ps -a’ instead to view it and then use ‘docker start’ command to start the container –

root@d1b13e2c3d3f:/# exit
exit


root@lindell:~# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

root@lindell:~# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
d1b13e2c3d3f        ubuntu              "/bin/bash"         5 minutes ago       Exited (0) 5 seconds ago

root@lindell:~# docker start d1b13e2c3d3f
root@lindell:~# docker exec -ti d1b13 /bin/bash
root@d1b13e2c3d3f:/# uptime
 01:39:46 up  1:18,  0 users,  load average: 0.39, 0.39, 0.37
root@d1b13e2c3d3f:/# 
               

On Part 2 of quick introduction to Docker, we will walk through using Dockerfile to automate image creation. We will see how quickly we can go from development to deployment.

You might also find some of the questions I answered in Stackoverflow about Docker.