Ngrep is a very user friendly packet sniffer, basically the “grep” equivalent at the network layer.

Here is a quick way of figuring out the http connections your browser is making even if you are browsing to a secure site, make sure that is the only site you are visiting as the command will capture all port 80 connections.

Installation –

apt-get install ngrep

Let us redirect all traffic ngrep captured to a file –

ngrep -d any -W byline port 80 | tee  /tmp/net_output

Now visit a secure site, say https://cnet.com, you will see nicely formated output

root@lindell:~# ngrep -d any -W byline port 80 | tee  /tmp/output
interface: any
filter: (ip or ip6) and ( port 80 )
####
T 17.31.198.19:33954 -> 72.21.91.29:80 [AP]
POST / HTTP/1.1.
Host: ocsp.digicert.com.
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0.
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8.
Accept-Language: en-US,en;q=0.5.
Accept-Encoding: gzip, deflate.
Content-Length: 83.
Content-Type: application/ocsp-request.
Connection: keep-alive.
..

From here, you can parse the /tmp/output file.

Similarly, you can parse the output file for the type of web server your favorite sites are using. Keep the ngrep command running, and visit all your favorite sites. Note, this works for http only, as https traffic is encrypted, for https only destination IP and port are shown.

In this case, I searched for the ‘Server:’ field in the HTTP response header from the web server. Apparently, nginx seems to be most popular, it is also interesting to see that AmazonS3 storage being used for hosting static content –

root@lindell:~# awk '/Server:/ {print $2}' /tmp/output |sort | uniq -c |sort -nr
    155 nginx.
     40 Apache.
     36 Apache-Coyote/1.1.
     20 Apache/2.2.3
     14 nginx/1.8.1.
      7 AmazonS3.
      6 Akamai
      5 ECS
      5 cloudflare-nginx.
      4 Omniture
      4 ESF.
      3 sffe.
      3 nginx/1.10.2.
      2 Microsoft-IIS/7.5.
      2 gws.
      2 AkamaiGHost.
      1 WildFly/8.
      1 Varnish.
      1 openresty.
      1 NetDNA-cache/2.2.
      1 Cowboy.
      1 ATS.
      1 Apache/2.2.14

References –
http://ngrep.sourceforge.net/usage.html
https://wiki.christophchamp.com/index.php?title=Ngrep

How to locate broadband Internet service providers in your area.

The FCC keeps a database of national broadband providers and it is publicly accessible at www.broadbandmap.gov. Just enter your full address or Zip code, and it will the broadband providers in your area as well as the advertised speed. One caveat is the data was last updated on June 2014, thus you might get latest information.

I checked the database for an area which had Google Fiber for the last 9 or 10 months, and it didn’t show Google Fiber as available in that area. The database has Google Fiber Inc. as a provider listed though.

If you want to check if Google Fiber is available or coming soon to your area check https://fiber.google.com/about/.

Once nice thing about the National broadband Map is the open standards API they made available to the public. It is well documented and very easy to pull data from programmatically. The API also gives you access to Census data and demographic information.

Note – most of the queries require the FIPS state and/or county codes (Federal Information Processing Standard state code). For instance, for New York state, the FIPS code is 36. Any county within a state will have FIPS county code of state FIPS code + county FIPS code. Bronx county’s (FIPS 005) full code would be 36005, for instance.

Here is a simple python script on how to interact with the API, will use Bronx county and/or NY as an example.

Let us get the overall broadband ranking within New York state –

import requests
url='https://www.broadbandmap.gov/broadbandmap/almanac/jun2014/rankby/state/36/population/wirelineproviderequals0/county?format=json&order=asc'
r=requests.get(url).json().get('Results').get('All')
for item in r:
    print item.get('rank'), item.get('geographyName')

Output based on ranking would look like this –
1 Franklin
2 Cattaraugus
3 Allegany
4 Schoharie
5 Otsego
6 Lewis
7 Washington
8 Hamilton
9 Yates
10 Delaware
11 Steuben
12 Wyoming
13 Cayuga
14 Jefferson
15 Herkimer
16 Schuyler
17 Essex
18 Seneca
19 St. Lawrence
20 Clinton
21 Montgomery
22 Chautauqua
23 Wayne
24 Columbia
25 Greene
26 Tioga
27 Livingston
28 Tompkins
29 Rensselaer
30 Chemung
31 Genesee
32 Cortland
33 Oswego
34 Sullivan
35 Albany
36 Oneida
37 Chenango
38 Orleans
39 Fulton
40 Madison
41 Niagara
42 Ontario
43 Warren
44 Schenectady
45 Ulster
46 Erie
47 Putnam
48 Onondaga
49 Saratoga
50 Broome
51 Suffolk
52 Monroe
53 Kings
54 Queens
55 New York
56 Bronx
57 Nassau
58 Westchester
59 Richmond
60 Orange
61 Rockland
62 Dutchess

Bronx county is ranked 56 out of 62, and the data for Bronx would be –

for item in r:
    if item.get('geographyId') == '36005':
        print item
        break


{u'anyWireline': 1.0,
 u'anyWirelineError': 0.0,
 u'downloadSpeedGreaterThan3000k': 1.0,
 u'downloadSpeedGreaterThan3000kError': 0.0,
 u'geographyId': u'36005',
 u'geographyName': u'Bronx',
 u'myAreaIndicator': False,
 u'population': 1482311,
 u'providerGreaterThan3': 1.0,
 u'rank': 56,
 u'stateFips': u'36',
 u'wirelineProviderEquals0': 0.0}

There is lots more you can do with the data, feel free to dig further.