Archive for the ‘ Containers ’ Category

nf_conntrack: table full, dropping packet


I actually saw this error in a Docker host, and Docker uses iptables and allof Docker’s iptables rules are added to the DOCKER chain. In this case though, it wasn’t the Docker iptables rules that were a problem, it is just that limits were reached in the connection tracking of the netfilter module. You might see this error in /var/log/messages or /var/log/kern

The full error message looked like this –

May 29 09:10:37 docker kernel: [74350.150400] nf_conntrack: table full, dropping packet
May 29 09:10:37 docker kernel: [74350.155361] nf_conntrack: table full, dropping packet
May 29 09:10:37 docker kernel: [74350.160282] nf_conntrack: table full, dropping packet
May 29 09:10:37 docker kernel: [74350.181547] nf_conntrack: table full, dropping packet
May 29 09:10:37 docker kernel: [74350.184807] nf_conntrack: table full, dropping packet
May 29 09:10:37 docker kernel: [74350.184913] nf_conntrack: table full, dropping packet

Resolution – increase maximum number of connections being tracked and/or reduce tracking timeouts. Look for these run time kernel parameters –

[root@kauai /]# sysctl net.ipv4.netfilter.ip_conntrack_tcp_timeout_established
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 27000
[root@kauai /]# sysctl net.netfilter.nf_conntrack_generic_timeout
net.netfilter.nf_conntrack_generic_timeout = 60
[root@kauai /]# sysctl net.ipv4.netfilter.ip_conntrack_max
net.ipv4.netfilter.ip_conntrack_max = 64268

These are the settings which resolved my issue, simply doubled the values –

sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_established=54000
sysctl -w net.netfilter.nf_conntrack_generic_timeout=120
sysctl -w net.ipv4.netfilter.ip_conntrack_max=128536

To make this permanent, add the lines above to the /etc/sysctl.conf file.

 

References –

https://security.stackexchange.com/questions/43205/nf-conntrack-table-full-dropping-packet

https://docs.docker.com/network/iptables/

Ansible non standard ssh port

How to run playbooks against a host running ssh on a port other than port 22.

Ansible is a simple automation or configuration management tool, which allows to execute a command/script on remote hosts in an adhoc or using playbooks. It is push based, and uses ssh to run the playbooks against remote hosts. The below steps are on how to run ansible playbooks on a host running ssh on port 2222.

One of the hosts managed by ansible is running in a non-default port. It is a docker container listening on port 2222. Actually ssh in container listens on port 22, but the host redirect port 2222 on host to port 22 on container.

1. Use environment variable –


 ansible-playbook tasks/app-deployment.yml --check -e ansible_ssh_port=2222

2. specify the port in the inventory or hosts file –

Under hosts file set the hostname to have the format ‘server:port’ –

[docker-hosts]
docker1:2222

Let us run the playbook now –

root@linubuvma:/tmp/ansible# cat tasks/app-deployment.yml
- hosts: docker-hosts
  vars:
    app_version: 1.1.0
  tasks:
  - name: install git
    apt: name=git state=latest
  - name: Checkout the application from git
    git: repo=https://github.com/docker/docker-py.git dest=/srv/www/myapp version={{ app_version }}
    register: app_checkout_result


root@linubuvma:/tmp/ansible# ansible-playbook tasks/app-deployment.yml

PLAY [docker-hosts] ************************************************************

TASK: [install git] ***********************************************************
changed: [docker1]

TASK: [Checkout the application from git] *************************************
changed: [docker1]

PLAY RECAP ********************************************************************
docker1                    : ok=2    changed=2    unreachable=0    failed=0

References –

http://docs.ansible.com/
http://docs.ansible.com/ansible/intro_inventory.html

In part 1 of this series, we saw how to pull Docker images from Docker Hub and launch Docker containers. We interacted with a running Docker container by running some bash commands, in this tutorial we will see how to use Dockerfile to automate image building for quicker deployment of applications in a container.

Dockerfile is a text file containing a set of instructions or commands in order to build a Docker image.

Prerequisites

1. Complete the tutorial on part 1 before proceeding. You will need a Docker engine running and the latest official Ubuntu Docker images locally hosted.

2. Create directories

$mkdir ~/docker-flask 
$cd ~/docker-flask

3. Add Dockerfile : ~/docker-flask/Dockerfile
The commands below will be used to create the Docker image. It will pull the latest Ubuntu official Docker image as a first layer or base. Then it will resynchronize the apt package index files from their sources.

A /flask directory will be created in the image, followed by installing Flask and running our Flask app, which we will write in next step.

FROM ubuntu:latest
RUN apt-get update && apt-get install -y python-pip python-dev
COPY . /flask
WORKDIR /flask
RUN pip install -r requirements.txt
EXPOSE 80
ENTRYPOINT ["python"]
CMD ["app.py"]

4. Write flask app : ~/docker-flask/app.py
Let us write a practical app, rather than just printing hello world. The flask app will return the user agent information of the visitor if the index page is visited.

We will also have a URL under /status/ followed by a valid HTTP status code. Given this HTTP status code by the visitor, the flask web server will generate the same HTTP status code header. For instance, if the user visits http://localhost/status/502, the flask server will respond with ‘502 BAD GATEWAY’ HTTP header.

Let us write it under ~/docker-flask/app.py

from flask import Flask
from flask import request, jsonify

app = Flask(__name__)

@app.route('/')
def user_agent():
    user_agent = request.headers.get('User-Agent')
    return 'Your browser is %s.' % user_agent

@app.route('/status/<int:httpcode>')
def get_status(httpcode):
    httpcode = int(httpcode)
    if httpcode < 100 or httpcode >= 600:
        return jsonify({'Status': 'Invalid HTTP status code'})
    elif httpcode >= 100 and httpcode < 500:
        return jsonify({'Status': 'UP'}) , httpcode
    else:
        return jsonify({'Status': 'DOWN'}) , httpcode

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0', port=80)

5. requirements.txt : ~/docker-flask/requirements.txt

cat ~/docker-flask/requirements.txt
Flask==0.12

By now, your directory structure should look similar to this –

daniel@lindell:~/docker-flask$ pwd
/home/daniel/docker-flask

daniel@lindell:~/docker-flask$ ls
app.py  Dockerfile  requirements.txt

Time to build the Docker image –

sudo docker build -t flaskweb:latest .

This will execute the series of commands under Dockerfile. If successful, you will end up with a Docker image named flaskweb and tagged latest –

root@lindell:~# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
flaskweb            latest              6b45443b6380        22 minutes ago      440.4 MB
ubuntu              latest              104bec311bcd        2 weeks ago         129 MB

If you encounter any errors, validate you don’t have any syntax errors on Dockerfile.

It is time to run the container –

daniel@lindell:~/docker-flask$ sudo docker run -d -p 80:80 flaskweb
d9af9a1c92bff45b56fc97d13935972b65e3554bfe22ec2f3c102fd26bd20e4c

daniel@lindell:~/docker-flask$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS                NAMES
d9af9a1c92bf        flaskweb            "python app.py"     About a minute ago   Up About a minute   0.0.0.0:80->80/tcp   drunk_mcnulty

In this case, both the host and container will be listening on port 80, feel free to modify this according to your setup.

Test it, we will use httpie to query the web server, if you don’t have httpie installed, you can use ‘curl -I’ to get the full header –

daniel@lindell:~/blog/docker-flask$ http http://localhost/
HTTP/1.0 200 OK
Content-Length: 29
Content-Type: text/html; charset=utf-8
Date: Fri, 30 Dec 2016 14:35:53 GMT
Server: Werkzeug/0.11.13 Python/2.7.12

Your browser is HTTPie/0.9.2.

daniel@lindell:~/blog/docker-flask$ http http://localhost/status/404
HTTP/1.0 404 NOT FOUND
Content-Length: 21
Content-Type: application/json
Date: Fri, 30 Dec 2016 14:35:56 GMT
Server: Werkzeug/0.11.13 Python/2.7.12

{
    "Status": "UP"
}

daniel@lindell:~/blog/docker-flask$ http http://localhost/status/502
HTTP/1.0 502 BAD GATEWAY
Content-Length: 23
Content-Type: application/json
Date: Fri, 30 Dec 2016 14:35:58 GMT
Server: Werkzeug/0.11.13 Python/2.7.12

{
    "Status": "DOWN"
}

Full clean up – if you want to start all over again or want to delete the container and images we have created, i have outlined the steps below. The first step is to stop the running container using ‘docker stop’ command, pass it the first few digits of the container ID.

Once the container is stopped, use ‘docker rm’ to delete the container. At this point, we can proceed with deleting the image as the image is not attached to any running container. Use ‘docker rmi’ to delete the image. We will keep the base Ubuntu image for future use.

daniel@lindell:/tmp$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                NAMES
d9af9a1c92bf        flaskweb            "python app.py"     12 minutes ago      Up 12 minutes       0.0.0.0:80->80/tcp   drunk_mcnulty

daniel@lindell:/tmp$ sudo docker stop d9a
d9a

daniel@lindell:/tmp$ sudo docker rm d9a
d9a

daniel@lindell:/tmp$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

daniel@lindell:/tmp$ sudo docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
flaskweb            latest              6b45443b6380        39 minutes ago      440.4 MB
ubuntu              latest              104bec311bcd        2 weeks ago         129 MB

daniel@lindell:/tmp$ sudo docker rmi 6b45
Untagged: flaskweb:latest
Deleted: sha256:6b45443b63805583f41fbf60aaf5cf746b871fdcfa8fe1c6d5adfb52870e7c89
Deleted: sha256:02062a8ea251d993f54e15f9e5654e40894449430acd045476000cd9ebbdf459
Deleted: sha256:fa2439cd5bc8a53152877c1dc3b12a60ab808bcfe5078549ea5e945f462330da
Deleted: sha256:3bac38b223d80a4db6c4283fd56275fe05ceeab6a1dfd81871aa14c6cda387df
Deleted: sha256:d97357dc5d7454e3b7757f2c348323c84d1902dd806792c53d1fd0ca7813b091
Deleted: sha256:b55dd5bd3326ec4657dc389f4aae69c34a7ba222872f7b868eb8de69d7f69dab
Deleted: sha256:eab59ae84eb136339d08fbacd2905a1ee80a0c875e8e14a4d5184fac30445714
Deleted: sha256:588253a9066c49786fcd0121353e7f0f2cea05cebbc6b9cef67f0c823d23dce8
Deleted: sha256:fe9f27a1cb9165531a1f5149c16ebcd522422e4ac2610035bbbcada7fd0b7551
Deleted: sha256:18ca1bc40895f6f97cae28fa5707bde537ac27023762303f98912c11549431ae

daniel@lindell:/tmp$ sudo docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu              latest              104bec311bcd        2 weeks ago         129 MB

References –
https://docs.docker.com/engine/reference/builder/
https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/
http://flask.pocoo.org/docs/0.12/quickstart/

In these series of Docker tutorials, i will walk you through a hands on experimentation with Docker. The operating system I am working on is Ubuntu 16.04.

Docker is a containerization technology which allows deployment of applications in containers. Its advantage is speed, a docker container hosting an application would be up and running in a few milliseconds.

As opposed to Virtual machines, containers run on top of the host OS. They share the host kernel. Thus you can only run a Linux container on a Linux host or machine.

Docker Installationuse this link for instructions on how to install Docker.

Installation Limitation – Docker runs on 64-bit OS only and it supports Linux kernel version 3.10 and above. You can verify this using the commands below –

root@lindell:~# arch
x86_64
root@lindell:~# uname -r
4.4.0-47-generic

Docker – terminology

    Images – are the building blocks of Docker. Once created or built, they can be shared, updated and used to launch containers. No image, no containers.

    Containers – are images in action. Containers give images life, containers are image plus all the ecosystem the Operating system need to run the application.

    Registry – where images are stored. They can be public or private. DockerHub is a typical example of public registry.

    Data volumes – persistent storage used by containers.
    Dockerfile – file containing instructions to be read by Docker for building a Docker image.

    Node – physical or virtual machine running Docker engine.

Our first Docker container
After installing docker and making sure that the Docker engine is running, run the commands below to check the Docker images(‘docker images’) available or if any Docker containers are running(‘docker ps’). Both commands should not return any results if this is a first time installation.

root@lindell:~# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
root@lindell:~# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED
             STATUS              PORTS               NAMES

The next step is to get a Docker image from Docker Hub. For security reasons, we are going to use only official images –

root@lindell:~# docker search --filter=is-official=true ubuntu
NAME                 DESCRIPTION                                     STARS     OFFICIAL   AUTOMATED
ubuntu               Ubuntu is a Debian-based Linux operating s...   5238      [OK]
ubuntu-upstart       Upstart is an event-based replacement for ...   69        [OK]
ubuntu-debootstrap   debootstrap --variant=minbase --components...   27        [OK]


root@lindell:~# docker run -ti ubuntu /bin/bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu

b3e1c725a85f: Pull complete
4daad8bdde31: Pull complete
63fe8c0068a8: Pull complete
4a70713c436f: Pull complete
bd842a2105a8: Pull complete
Digest: sha256:7a64bc9c8843b0a8c8b8a7e4715b7615e4e1b0d8ca3c7e7a76ec8250899c397a
Status: Downloaded newer image for ubuntu:latest

root@d1b13e2c3d3f:/# docker images
bash: docker: command not found

root@d1b13e2c3d3f:/# hostname -f
d1b13e2c3d3f

root@d1b13e2c3d3f:/# uname -r
4.4.0-47-generic

We just downloaded an official Ubuntu image and started an Ubuntu container by running /bin/bash inside the newly started container. The ‘-ti’ option runs bash interactively(-i) by allocating a pseudo-TTY(-t).

Note that – the kernel version on the container is the same as the host’s kernel version. During first run, Docker will try to find Ubuntu image in our local storage, if it can’t find it, it downloads it from Docker Hub. On next runs, starting the containers will be much faster.

If we check the images and processes running now –

root@lindell:~# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu              latest              104bec311bcd        5 days ago          129 MB
root@lindell:~# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
d1b13e2c3d3f        ubuntu              "/bin/bash"         About a minute ago   Up About a minute              

At this point, if we exit from the container, docker ps will no longer show the container as it has been terminated. We use ‘docker ps -a’ instead to view it and then use ‘docker start’ command to start the container –

root@d1b13e2c3d3f:/# exit
exit


root@lindell:~# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

root@lindell:~# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
d1b13e2c3d3f        ubuntu              "/bin/bash"         5 minutes ago       Exited (0) 5 seconds ago

root@lindell:~# docker start d1b13e2c3d3f
root@lindell:~# docker exec -ti d1b13 /bin/bash
root@d1b13e2c3d3f:/# uptime
 01:39:46 up  1:18,  0 users,  load average: 0.39, 0.39, 0.37
root@d1b13e2c3d3f:/# 
               

On Part 2 of quick introduction to Docker, we will walk through using Dockerfile to automate image creation. We will see how quickly we can go from development to deployment.

You might also find some of the questions I answered in Stackoverflow about Docker.