Chaos Engineering For Prometheus
Gremlin is a simple, safe and secure service for performing Chaos Engineering experiments through a SaaS-based platform. Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Prometheus is now a standalone open source project and maintained independently of any company. Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes. Datadog is a monitoring service for cloud-scale applications, providing monitoring of servers, databases, tools, and services, through a SaaS-based data analytics platform.
This tutorial shows:
- How to install Docker
- How to install Prometheus for monitoring and alerting
- How to install Gremlin in a Docker container for Chaos Engineering experiments
- How to install Datadog in a container to monitor Prometheus
- How to create a Shutdown attack from a Gremlin Container against a Prometheus Container
- How to ensure your Docker containers automatically restart when they are killed
- Additional Chaos Engineering experiments to run on Prometheus
Chaos Engineering Hypothesis
Prometheus’ storage model groups ingested samples into blocks of two hours. Each two hour block contains one or more chunk files that contain all time series samples for that window of time. The block for currently incoming samples is kept in memory and not fully persisted. It is secured against crashes by a WAL (write-ahead-log) that can be replayed when the prometheus server restarts after a crash. The limitation of local storage is that it is not clustered or replicated. Thus, it is not scalable or durable in the face or disk or node outages. There is a open source project called Thanos which aims to be a highly available Prometheus setup with long term storage capabilities. For the purposes of this tutorial we will run Chaos Engineering experiments on the Prometheus server running inside a Docker container. The Chaos Engineering experiment we will perform is a shutdown of the Prometheus Docker container.
Image source: Prometheus Overview
- Cloud Infrastructure droplet running Ubuntu 16.04
- A Gremlin account (sign up here)
- A Datadog account(sign up here)
Step 1 - Installing Docker
In this step, you’ll install Docker.
Add Docker’s official GPG key:
Use the following command to set up the stable repository.
Update the apt package index
Make sure you are about to install from the Docker repo instead of the default Ubuntu 16.04 repo:
Install the latest version of Docker CE:
Docker should now be installed, the daemon started, and the process enabled to start on boot. Check that it is running:
Type q to return to the prompt.
Make sure you are in the Docker usergroup, replace tammy with your username:
Next we will create an Prometheus container and monitor the new container directly by joining the container’s pid namespace.
Step 2 - Installing Prometheus with Docker
Prometheus is an open-source systems monitoring and alerting toolkit.
First we'll create a directory in /etc for Prometheus' configuration files and a directory in /var/lib for its data.
First create a Prometheus configuration file on the host filesystem:
Add the following contents to the yml file, replace the IP address in red with your own IP address:
Start the Prometheus Docker container with your configuration file:
List all running Docker containers using the following command:
To start the Node Exporter on port 9100 using Docker:
Now you will be able to view your Prometheus metrics.
The Prometheus server should now automatically start scraping the Node Exporter.
Step 3 - View your Prometheus metrics
You can also verify that Prometheus is serving metrics about itself by navigating to its metrics endpoint, e.g. http://18.104.22.168:9090/metrics, replacing the IP Address with your own.
To graph expressions, navigate to http://22.214.171.124:9090/graph and use the "Graph" tab, replacing the IP Address with your own.
For example, enter the following expression to graph the per-second rate of chunks being created in the self-scraped Prometheus:
You will a graph similar to below:
Experiment with the graph range parameters and other settings.
Step 4 - Set up your Gremlin agent credentials
After you have created your Gremlin account (sign up here) you will need to find your Gremlin Daemon credentials. Login to the Gremlin App using your Company name and sign-on credentials. These were emailed to you when you signed up to start using Gremlin.
Navigate to Team Settings and click on your Team.
Store your Gremlin agent credentials as environment variables, for example:
Next run the Gremlin Daemon in a Container.
Use docker run to pull the official Gremlin Docker image and run the Gremlin daemon:
Use docker ps to see all running Docker containers:
Jump into your Gremlin container with an interactive shell (replace b281e749ac33 with the real ID of your Gremlin container):
From within the container, check out the available attack types:
Then exit the container.
Step 5 - Installing the Datadog agent in a Docker container
To install Datadog in a Docker container you can use the Datadog Docker easy one-step install.
Run the following command, replacing the item in red with your own API key:
It will take a few minutes for Datadog to spin up the Datadog container, collect metrics on your existing containers and display them in the Datadog App.
View your Docker Containers in Datadog, you should see the following on the host Dashboard:
Step 6 - Run a Shutdown Attack against the Prometheus container from a Gremlin container
We will use the Gremlin CLI attack command to create a CPU attack.
Now use the Gremlin CLI (gremlin) to run a CPU attack from within a Gremlin container:
This attack will shutdown the Prometheus container and it will not be automatically recreated.
Use docker ps to confirm that the Prometheus container is no longer running:
Step 7 - How to ensure your containers automatically restart when they are killed
To create Docker containers that restart automatically you will need to you docker’s restart functionality. Start Prometheus again using the following Docker command, this includes --restart unless-stopped which will ensure your Prometheus Docker container is restarted.
Use docker ps to confirm that the Prometheus container is now running again:
The average lifespan of a container is 2.5 days, visit the Docker guide on starting containers automatically to learn more. You should not be getting paged for containers or hosts being down. You should be handling this in an automated manner.
Step 8 - Additional Chaos Engineering experiments to run on Prometheus
There are many Chaos Engineering experiments you could possibly run on your Prometheus infrastructure:
- Time Travel Gremlin - will changing the clock time of the host impact how Prometheus processes metrics?
- Latency & Packet Loss Gremlins - will they impact the ability to serve the UI?
- Disk Gremlin - will filling up the disk crash the Prometheus host? Will the WAL perform as expected?
We encourage you to run these Chaos Engineering experiments and share your findings! You can get access to Gremlin.
This tutorial has explored how to install Prometheus and Gremlin in Docker containers for your Chaos Engineering experiments. We then ran a shutdown Chaos Engineering experiment on the Prometheus container using the Gremlin Shutdown attack.