How to Install and Use Gremlin with Docker on Ubuntu 16.04

Introduction

Gremlin is a simple, safe and secure way to use Chaos Engineering to improve system resilience. You can use Gremlin with Docker in a variety of ways. It is possible to attack Docker containers and it is also possible to run Gremlin in a container to create attacks against the host or other containers.

This tutorial will provide a walkthrough of the following:

  • How to install Docker
  • How to create a htop container to monitor the host and containers
  • How to create an Nginx Docker container to attack using Gremlin
  • How to install Gremlin on the host
  • How to create a CPU Attack from the host against an nginx Docker container using the Gremlin Control Panel

To run Gremlin in a Docker container, view the guide on How to Install and Use Gremlin in a Docker Container.

Prerequisites

Before you begin this tutorial, you’ll need the following:

  • An Ubuntu 16.04 server
  • A Gremlin account
  • The apt-transport-https package to be able to install gremlin from our repo via HTTPS.

Step 1 - Installing Docker

In this step, you’ll install Docker.

Add Docker’s official GPG key:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Use the following command to set up the stable repository.

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

Update the apt package index:

sudo apt-get update

Make sure you are about to install from the Docker repo instead of the default Ubuntu 16.04 repo:

apt-cache policy docker-ce

Install the latest version of Docker CE:

sudo apt-get install docker-ce

Docker should now be installed, the daemon started, and the process enabled to start on boot. Check that it's running:

sudo systemctl status docker

Make sure you are in the Docker usergroup, replace $USER with your username:

sudo usermod -aG docker $USER

Log out and back in for your permissions to take effect, or type the following:

su - ${USER}

Step 2 - Create an htop container for monitoring

htop is an interactive process viewer for unix.

Create the docker file:

vim Dockerfile

Add the following to the Dockerfile:

FROM alpine:latest
RUN apk add --update htop && rm -rf /var/cache/apk/*
ENTRYPOINT ["htop"]

Build the Dockerfile and tag the image:

docker build -t htop .

Run htop inside a container:

docker run -it --rm --pid=host htop

To exit htop, use the q key.

Next we will create an nginx container and monitor the new container directly by joining the nginx container’s pid namespace.

Step 3 - Create an nginx Docker container to be used for Gremlin Attacks

First we will create a directory for the html page we will serve using nginx:

mkdir -p ~/docker-nginx/html
cd ~/docker-nginx/html

Create a simple html page:

vim index.html

Paste in the content shown below:

<html>
    <head>
        <title>Docker nginx tutorial</title>
        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous">
    </head>
    <body>
        <div class="container">
            <h1>Hello this is your container speaking.</h1>
            <p>This page was created by your Docker container.</p>
            <p>Now it’s time to create a Gremlin attack.</p>
        </div>
    </body>
</html>

Create a container using the nginx Docker image:

sudo docker run -l service=nginx --name docker-nginx -p 80:80 -d -v ~/docker-nginx/html:/usr/share/nginx/html nginx

View the nginx Docker container

sudo docker ps -a

You will see the following:

CONTAINER ID        IMAGE               COMMAND                       CREATED                 STATUS                   PORTS                         NAMES
352609a67e95           nginx               "nginx -g 'daemon of…"   33 seconds ago      Up 32 seconds       0.0.0.0:80->80/tcp   docker-nginx

Step 4 - Use an htop container to monitor an nginx Docker container

htop can be used to monitor Gremlin attacks against the host and Gremlin attacks against individual containers.

Join the docker-nginx container’s pid namespace:

docker run -it --rm --pid=container:docker-nginx htop

Before the attack, htop will show you that CPU is not spiking:

  1  [|                                                               0.7%]   Tasks: 3, 0 thr; 1 running
  2  [||                                                              1.3%]   Load average: 0.07 0.05 0.06
  Mem[||||||||||||||||||||||||||                                141M/3.86G]   Uptime: 02:48:43
  Swp[                                                               0K/0K]

  PID USER      PRI  NI    VIRT   RES   SHR   S   CPU%   MEM%   TIME+   Command
   10 root      20    0    4324   1708  936   R   0.0    0.0    0:00.05  htop
    1 root      20    0    32428  5080  4400  S   0.0    0.1    0:00.03  nginx: master process nginx -g daemon off;
    5 101       20   0     32900  3060  1824  S   0.0    0.1    0:00.00  nginx: worker process

Next we are going to install Gremlin on the host to perform attacks.

Step 5 - **Installing the Gremlin Daemon and CLI **

First, add the Gremlin Debian repository:

echo "deb https://deb.gremlin.com/ release non-free" | sudo tee /etc/apt/sources.list.d/gremlin.list

Import the repo’s GPG key:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C81FC2F43A48B25808F9583BDFF170F324D41134 9CDB294B29A5B1E2E00C24C022E8EF3461A50EF6

Then install the Gremlin daemon and CLI:

sudo apt-get update && sudo apt-get install -y gremlind gremlin

Step 6 - ** Downloading your Gremlin client certificates**

After you have created your Gremlin account (sign up here) you will need to find your Gremlin Daemon credentials. Login to the Gremlin App using your Company name and sign-on credentials. These were emailed to you when you signed up to start using Gremlin.

Navigate to Team Settings and click on your Team. Click the blue Download button to save your certificates to your local computer. The downloaded certificate.zip contains both a public-key certificate and a matching private key.

certificates

Unzip the downloaded certificate.zip on your laptop and copy the files to the server you will be using with a Linux file transfer tool such as rsync, sftp or scp. Alternatively, you can store these certificates in a storage service such as AWS S3. For example:

rsync -avz /Users/tammybutow/Desktop/tammy-client.pub_cert.pem tammy@142.93.31.189:/var/lib/gremlin
rsync -avz /Users/tammybutow/Desktop/tammy-client.priv_key.pem tammy@142.93.31.189:/var/lib/gremlin

**Creating a gremlind file for your environment variables **

Next create the /etc/default/gremlind file:

sudo vim /etc/default/gremlind

Add your GREMLIN environment variables to the file, for example:

GREMLIN_TEAM_ID="3f242793-018a-5ad5-9211-fb958f8dc084"GREMLIN_TEAM_CERTIFICATE_OR_FILE="file:///var/lib/gremlin/tammy-client.pub_cert.pem"GREMLIN_TEAM_PRIVATE_KEY_OR_FILE="file:///var/lib/gremlin/tammy-client.priv_key.pem"GREMLIN_CLIENT_TAGS="service=prometheus"

Save the file. Restart the service:

sudo service gremlind restart

Confirming your gremlind configuration

Take a look at /var/log/gremlin/daemon.log to confirm:

tail /var/log/gremlin/daemon.log

You should see an output similar to below if it was successful:

2018-10-31 02:34:20 - Logging successfully initialized2018-10-31 02:34:23 - Using Team ID : 3f242793-018a-5ad5-9211-fb958f8dc0842018-10-31 02:34:23 - Using Identifier : 142.93.31.1892018-10-31 02:34:23 - Found GREMLIN_TEAM_CERTIFICATE_OR_FILE in file:///var/lib/gremlin/tammy-client.pub_cert.pem2018-10-31 02:34:23 - Found GREMLIN_TEAM_PRIVATE_KEY_OR_FILE in file:///var/lib/gremlin/tammy-client.priv_key.pem

Step 7 - Creating attacks using the Gremlin App

Example: Creating a CPU Attack from the host against the nginx Docker container using the App

You can use the Gremlin App or the Gremlin API to trigger Gremlin attacks. You can view the available range of Gremlin Attacks in Gremlin Help.

The “Hello World” of Chaos Engineering is the CPU Resource Attack. To create a CPU Resource Attack select “Resource” and then “CPU” in the dropdown menu.

gremlin cpu

The CPU Resource Attack will consume CPU resources based on the settings you select. The most popular default settings for a CPU Resource Attack are pre-selected, a default attack will utilize 1 core for 60 seconds. Before you can run the Gremlin attack you will need to click either Exact hosts to run the attack on or click the Random attack option.

Click Exact and select the host you created the nginx Docker container on, in this example that is 138.68.226.195.

Example: Using Container Labels to Attack Specific Containers

Container labels will enable you to choose containers on your host to attack.

Click to enable container labels, type in the label details of the container.

For this example, the Nginx Docker container label we created is set to service=nginx.

sudo docker run -l service=nginx --name docker-nginx -p 80:80 -d -v ~/docker-nginx/html:/usr/share/nginx/html nginx

Finally select ”Create” to kick off a random Gremlin CPU Resource Attack on the nginx Docker container.

Your attack will begin to run, you will be able to view its progress via Gremlin Attacks in the Gremlin Control Panel.

To view the results of the attack join the docker-nginx container’s pid namespace:

docker run -it --rm --pid=container:docker-nginx htop

You will see the following in htop:

  1  [                                                      0.0%]   Tasks: 4, 1 thr; 2 running
  2  [||||||||||||||||||||||||||||||||||||||||||||||||||||100.0%]   Load average: 0.61 0.17 0.06
  Mem[||||||||||||||||||||||                          176M/3.86G]   Uptime: 00:37:17
  Swp[                                                     0K/0K]
  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command          
   18 root       20   0 15456 13692  4112 S 100.  0.3  0:26.39 gremlin attack cpu -c 1 -l 60
   13 root       20   0  4324  1988   944 R  0.0  0.0  0:00.12 htop
    1 root       20   0 32428  5184  4504 S  0.0  0.1  0:00.04 nginx: master process nginx -g daemon off;
    8 101        20   0 32900  2948  1712 S  0.0  0.1  0:00.00 nginx: worker process

If you have the Gremlin Slackbot enabled you will also see the bot post that the Gremlin Attack has started. When it’s successful the Gremlin Slackbot will post again. To setup the Gremlin Slackbot follow the guide, How to Setup and Use the Gremlin Slackbot.

slackcpu

When your attack is finished it will move to Completed Attacks in the Gremlin App.

To view the logs of the Attack, click on the Attack in Completed Attacks.

Conclusion

You've installed Gremlin on a server running Docker and validated that Gremlin works by running the “Hello World” of Chaos Engineering for Docker Containers, the CPU Resource attack. You now possess tools that make it possible for you to explore additional Gremlin Attacks.

Gremlin’s Developer Guide is a great resource and reference for using Gremlin to do Chaos Engineering. You can also explore the Gremlin Community for more information on how to use Chaos Engineering with your infrastructure.

Avoid downtime. Use Gremlin to turn failure into resilience.

Gremlin empowers you to proactively root out failure before it causes downtime. Try Gremlin for free and see how you can harness chaos to build resilient systems.