
Gremlin is a simple, safe and secure service for performing Chaos Engineering experiments through a SaaS-based platform. Redis is an open source in-memory data structure store. Datadog is a monitoring service for cloud-scale applications, providing monitoring of servers, databases, tools, and services, through a SaaS-based data analytics platform. Datadog provides an integration to monitor Redis.
For the purposes of this tutorial we will run Chaos Engineering experiments on Redis. Our Chaos Engineering hypothesis is that we need to constantly ensure we are monitoring latency and crash frequency as these are common issues that can appear when running Redis in production. To begin with, view the guide on Problems With Redis published by the Redis team.
Latency measures the average time in milliseconds it takes the Redis server to respond. Typical Redis latency for a 1GBits/s network is about 200 μs. Tracking latency is the most useful way to see impacts to Redis performance.
We know from the official Redis Cluster documentation that a Redis Cluster does not guarantee strong consistency. This tutorial will focus on doing Chaos Engineering with one Redis instance. In a future tutorial we will focus on Redis Cluster.
To complete this tutorial you will need the following:
This tutorial will walk you through the required steps to do Chaos Engineering with Redis.
First update the server:
1sudo apt update
To compile Redis, run the following commands:
1sudo apt-get install redis-server
Now start redis by running the following command:
1redis-server --daemonize yes
A successful result will end with:
1Configuration loaded
Now use the redis cli to confirm you can connect:
1redis-cli
You will see the following as a successful result:
1redis 127.0.0.1:6379>
Now type the following at the redis prompt:
1ping
The successful result will be
1PONG
Now type the following to store data in Redis:
1set test "Time for Chaos Engineering with Redis!"
The successful result will be:
1OK
Now type the following to retrieve your stored data:
1get test
The successful result will be:
1"Time for Chaos Engineering with Redis!"
Now exit the Redis prompt:
1exit
Restart Redis:
1sudo systemctl restart redis
Now use the redis cli:
1redis-cli
Type the following at the redis cli prompt to return the data you stored previously:
1get test
You will see the following if it is successfully returned:
1Output"Time for Chaos Engineering with Redis!"
Now exit the Redis prompt:
1exit
In this step, you’ll install Docker.
Add Docker’s official GPG key:
1curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
Use the following command to set up the stable repository.
1sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
Update the apt package index:
1sudo apt-get update
Make sure you are about to install from the Docker repo instead of the default Ubuntu 16.04 repo:
1apt-cache policy docker-ce
Install the latest version of Docker CE:
1sudo apt-get install docker-ce
Docker should now be installed, the daemon started, and the process enabled to start on boot. Check that it is running:
1sudo systemctl status docker
Type q
to return to the prompt.
Make sure you are in the Docker usergroup, replace redis with your username:
1sudo usermod -aG docker redis
After you have created your Gremlin account (sign up here) you will need to find your Gremlin Daemon credentials. Login to the Gremlin App using your Company name and sign-on credentials. These were emailed to you when you signed up to start using Gremlin.
Navigate to Team Settings and click on your Team.
Store your Gremlin agent credentials as environment variables, for example:
1export GREMLIN_TEAM_ID=3f242793-018a-5ad5-9211-fb958f8dc084
1export GREMLIN_TEAM_SECRET=eac3a31b-4a6f-6778-1bdb813a6fdc
Next run the Gremlin Daemon in a Container.
Use docker run to pull the official Gremlin Docker image and run the Gremlin daemon:
1sudo docker run -d \ --net=host \ --pid=host \ --cap-add=NET_ADMIN \ --cap-add=SYS_BOOT \ --cap-add=SYS_TIME \ --cap-add=KILL \ -e GREMLIN_TEAM_ID="${GREMLIN_TEAM_ID}" \ -e GREMLIN_TEAM_SECRET="${GREMLIN_TEAM_SECRET}" \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /var/log/gremlin:/var/log/gremlin \ -v /var/lib/gremlin:/var/lib/gremlin \ gremlin/gremlin daemon
Use docker ps to see all running Docker containers:
1sudo docker ps
1CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESb281e749ac33 gremlin/gremlin "/entrypoint.sh daem…" 5 seconds ago Up 4 seconds relaxed_heisenberg
Jump into your Gremlin container with an interactive shell (replace b281e749ac33 with the real ID of your Gremlin container):
1sudo docker exec -it b281e749ac33 /bin/bash
From within the container, check out the available attack types:
1gremlin help attack-container
1Usage: gremlin attack-container CONTAINER TYPE [type-specific-options]Type "gremlin help attack-container TYPE" for more details: blackhole # An attack which drops all matching network traffic cpu # An attack which consumes CPU resources io # An attack which consumes IO resources latency # An attack which adds latency to all matching network traffic memory # An attack which consumes memory packet_loss # An attack which introduces packet loss to all matching network traffic shutdown # An attack which forces the target to shutdown dns # An attack which blocks access to DNS servers time_travel # An attack which changes the system time. disk # An attack which consumes disk resources process_killer # An attack which kills the specified process
To install Datadog in a Docker container you can use the Datadog Docker easy one-step install.
Run the following command, replacing the item in red with your own API key:
1sudo docker run -d --name dd-agent -v /var/run/docker.sock:/var/run/docker.sock:ro -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e DD_API_KEY=7cfe87ce1756aea datadog/agent:latest
It will take a few minutes for Datadog to spin up the Datadog container, collect metrics on your existing containers and display them in the Datadog App.
We will use the Gremlin CLI attack command to create a latency attack.
Now use the Gremlin CLI (gremlin) to run a latency attack against the host from a Gremlin container:
1sudo docker run -d \ --net=host \ --pid=host \ --cap-add=NET_ADMIN \ --cap-add=SYS_BOOT \ --cap-add=SYS_TIME \ --cap-add=KILL \ -e GREMLIN_TEAM_ID="${GREMLIN_TEAM_ID}" \ -e GREMLIN_TEAM_SECRET="${GREMLIN_TEAM_SECRET}" \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /var/log/gremlin:/var/log/gremlin \ -v /var/lib/gremlin:/var/lib/gremlin \ gremlin/gremlin attack latency
This attack will inject latency to the Redis host.
Now exit the container by running the following command:
1exit
There are many Chaos Engineering experiments you could possibly run on your Redis infrastructure:
We encourage you to run these Chaos Engineering experiments and share your findings! To get access to Gremlin, sign up here.
This tutorial has explored how to install Redis and Gremlin in Docker containers for your Chaos Engineering experiments. We then ran a shutdown Chaos Engineering experiment on the Redis container using the Gremlin Latency attack.
Gremlin empowers you to proactively root out failure before it causes downtime. See how you can harness chaos to build resilient systems by requesting a demo of Gremlin.
Get started