How to interact with web services.

Curl is the defacto CLI tool for interacting with web services and other non-HTTP services such as FTP or LDAP. Linux or Unix system administrators as well as developers love it for its ease of use and debugging capabilities. When you want to interact with web services from within scripts, curl is the number one choice.For downloading files from the web, wget is commonly used as well, but curl can way more.

Since enough has been written about curl, this post is about a tool which takes interaction with web services a lot more human friendly, with nicely formatted and colored output – httpie. It is written in Python.

Installation –

apt-get  install httpie     #(Debian/Ubuntu)
yum install httpie          #(Redhat/CentOS)

Note – although the package name is httpie, the binary file is installed as http.

When troubleshooting web services, the first thing we check is usually http request and response headers –

daniel@lindell:/$ http -p hH  httpbin.org
GET / HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: httpbin.org
User-Agent: HTTPie/0.9.2

HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 12150
Content-Type: text/html; charset=utf-8
Date: Thu, 22 Dec 2016 01:32:13 GMT
Server: nginx

Where -H is for Request headers, -h is for response headers. Similarly, -B is for request body and -b is for response body.

We can also pass more complex HTTP headers, in this case “If-Modified-Since”, the web server will return 304 if the static content i am requesting has not been modified. Moving the date a few years back, it will respond with 200 status code.

daniel@lindell:/$ http -p hH http://linuxfreelancer.com/wp-content/themes/soulvision/images/texture.jpg "If-Modified-Since: Wed, 21 Dec 2016 20:51:14 GMT"
GET /wp-content/themes/soulvision/images/texture.jpg HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: linuxfreelancer.com
If-Modified-Since:  Wed, 21 Dec 2016 20:51:14 GMT
User-Agent: HTTPie/0.9.2

HTTP/1.1 304 Not Modified
Connection: Keep-Alive
Date: Thu, 22 Dec 2016 01:39:28 GMT
ETag: "34441c-f04-4858fcd6af900"
Keep-Alive: timeout=15, max=100
Server: Apache/2.2.14 (Ubuntu)

daniel@lindell:/$ http -p hH http://linuxfreelancer.com/wp-content/themes/soulvision/images/texture.jpg "If-Modified-Since: Wed, 21 Dec 2008 20:51:14 GMT"
GET /wp-content/themes/soulvision/images/texture.jpg HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: linuxfreelancer.com
If-Modified-Since:  Wed, 21 Dec 2008 20:51:14 GMT
User-Agent: HTTPie/0.9.2

HTTP/1.1 200 OK
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Length: 3844
Content-Type: image/jpeg
Date: Thu, 22 Dec 2016 01:39:37 GMT
ETag: "34441c-f04-4858fcd6af900"
Keep-Alive: timeout=15, max=100
Last-Modified: Sat, 01 May 2010 22:23:00 GMT
Server: Apache/2.2.14 (Ubuntu)

httpie also makes passing JSON encoding as well as POST/PUT methods a lot easier. No need for formatting your payload as JSON, it defaults to JSON. Debugging is easier to with -v option, which shows the raw wire data –

daniel@lindell:/$ http -v PUT httpbin.org/put name=JoeDoe email=joedoe@gatech.edu
PUT /put HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 48
Content-Type: application/json
Host: httpbin.org
User-Agent: HTTPie/0.9.2

{
    "email": "joedoe@gatech.edu", 
    "name": "JoeDoe"
}

HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 487
Content-Type: application/json
Date: Thu, 22 Dec 2016 01:44:20 GMT
Server: nginx

{
    "args": {}, 
    "data": "{\"name\": \"JoeDoe\", \"email\": \"joedoe@gatech.edu\"}", 
    "files": {}, 
    "form": {}, 
    "headers": {
        "Accept": "application/json", 
        "Accept-Encoding": "gzip, deflate", 
        "Content-Length": "48", 
        "Content-Type": "application/json", 
        "Host": "httpbin.org", 
        "User-Agent": "HTTPie/0.9.2"
    }, 
    "json": {
        "email": "joedoe@gatech.edu", 
        "name": "JoeDoe"
    }, 
    "origin": "192.1.1.2", 
    "url": "http://httpbin.org/put"
}

I have touched just the surface of httpie here, please feel free to get more detailed information on the github repo. It has built-in JSON support, form/file upload, HTTPS, proxies and authentication, custom headers, persistent sessions etc.

Article on wget and curl from previous post.