Chaos Engineering with Redis

Tammy Butow

Principal SRE

Last Updated: February 15, 2019

Categories: Chaos Engineering

Introduction

Gremlin is a simple, safe and secure service for performing Chaos Engineering experiments through a SaaS-based platform. Redis is an open source in-memory data structure store. Datadog is a monitoring service for cloud-scale applications, providing monitoring of servers, databases, tools, and services, through a SaaS-based data analytics platform. Datadog provides an integration to monitor Redis.

Chaos Engineering Hypothesis

For the purposes of this tutorial we will run Chaos Engineering experiments on Redis. Our Chaos Engineering hypothesis is that we need to constantly ensure we are monitoring latency and crash frequency as these are common issues that can appear when running Redis in production. To begin with, view the guide on Problems With Redis published by the Redis team.

Latency measures the average time in milliseconds it takes the Redis server to respond. Typical Redis latency for a 1GBits/s network is about 200 μs. Tracking latency is the most useful way to see impacts to Redis performance.

We know from the official Redis Cluster documentation that a Redis Cluster does not guarantee strong consistency. This tutorial will focus on doing Chaos Engineering with one Redis instance. In a future tutorial we will focus on Redis Cluster.

Prerequisites

To complete this tutorial you will need the following:

• 1 cloud infrastructure instance running Ubuntu 16.04
• A Gremlin account (sign up here)
• A Datadog account

Overview

This tutorial will walk you through the required steps to do Chaos Engineering with Redis.

• Step 1 - Creating a Redis server
• Step 2 - Installing Docker on each host
• Step 3 - Installing Gremlin in a Docker container on each host
• Step 4 - Installing Datadog in a Docker container on each host
• Step 5 - Running the Redis Latency Chaos Engineering experiment
• Step 6 - Additional Chaos Engineering experiments you can run with Gremlin

Step 1 - Creating a Redis server

First update the server:

1sudo apt update

To compile Redis, run the following commands:

1sudo apt-get install redis-server

Now start redis by running the following command:

1redis-server --daemonize yes

A successful result will end with:

1Configuration loaded

Now use the redis cli to confirm you can connect:

1redis-cli

You will see the following as a successful result:

1redis 127.0.0.1:6379>

Now type the following at the redis prompt:

1ping

The successful result will be

1PONG

Now type the following to store data in Redis:

1set test "Time for Chaos Engineering with Redis!"

The successful result will be:

1OK

Now type the following to retrieve your stored data:

1get test

The successful result will be:

1"Time for Chaos Engineering with Redis!"

Now exit the Redis prompt:

1exit

Restart Redis:

1sudo systemctl restart redis

Now use the redis cli:

1redis-cli

Type the following at the redis cli prompt to return the data you stored previously:

1get test

You will see the following if it is successfully returned:

1Output"Time for Chaos Engineering with Redis!"

Now exit the Redis prompt:

1exit

Step 2 - Installing Docker

In this step, you’ll install Docker.

Add Docker’s official GPG key:

1curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Use the following command to set up the stable repository.

1sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

Update the apt package index:

1sudo apt-get update

Make sure you are about to install from the Docker repo instead of the default Ubuntu 16.04 repo:

1apt-cache policy docker-ce

Install the latest version of Docker CE:

1sudo apt-get install docker-ce

Docker should now be installed, the daemon started, and the process enabled to start on boot. Check that it is running:

1sudo systemctl status docker

Type q to return to the prompt.

Make sure you are in the Docker usergroup, replace redis with your username:

1sudo usermod -aG docker redis

Step 3 - Installing Gremlin On Each Host

After you have created your Gremlin account (sign up here) you will need to find your Gremlin Daemon credentials. Login to the Gremlin App using your Company name and sign-on credentials. These were emailed to you when you signed up to start using Gremlin.

Navigate to Team Settings and click on your Team.

Store your Gremlin agent credentials as environment variables, for example:

1export GREMLIN_TEAM_ID=3f242793-018a-5ad5-9211-fb958f8dc084

1export GREMLIN_TEAM_SECRET=eac3a31b-4a6f-6778-1bdb813a6fdc

Next run the Gremlin Daemon in a Container.

Use docker run to pull the official Gremlin Docker image and run the Gremlin daemon:

1sudo docker run -d \      --net=host \      --pid=host \      --cap-add=NET_ADMIN \      --cap-add=SYS_BOOT \      --cap-add=SYS_TIME \     --cap-add=KILL \      -e GREMLIN_TEAM_ID="${GREMLIN_TEAM_ID}" \      -e GREMLIN_TEAM_SECRET="${GREMLIN_TEAM_SECRET}" \      -v /var/run/docker.sock:/var/run/docker.sock \      -v /var/log/gremlin:/var/log/gremlin \      -v /var/lib/gremlin:/var/lib/gremlin \    gremlin/gremlin daemon

Use docker ps to see all running Docker containers:

1sudo docker ps

1CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS              PORTS                    NAMESb281e749ac33        gremlin/gremlin      "/entrypoint.sh daem…"   5 seconds ago       Up 4 seconds                                 relaxed_heisenberg

Jump into your Gremlin container with an interactive shell (replace b281e749ac33 with the real ID of your Gremlin container):

1sudo docker exec -it b281e749ac33 /bin/bash

From within the container, check out the available attack types:

1gremlin help attack-container

1Usage: gremlin attack-container CONTAINER TYPE [type-specific-options]Type "gremlin help attack-container TYPE" for more details:  blackhole # An attack which drops all matching network traffic  cpu   # An attack which consumes CPU resources  io    # An attack which consumes IO resources  latency # An attack which adds latency to all matching network traffic  memory  # An attack which consumes memory  packet_loss # An attack which introduces packet loss to all matching network traffic    shutdown  # An attack which forces the target to shutdown  dns   # An attack which blocks access to DNS servers  time_travel # An attack which changes the system time.  disk    # An attack which consumes disk resources  process_killer  # An attack which kills the specified process

Step 4 - Installing the Datadog agent in a Docker container

To install Datadog in a Docker container you can use the Datadog Docker easy one-step install.

Run the following command, replacing the item in red with your own API key:

1sudo docker run -d --name dd-agent -v /var/run/docker.sock:/var/run/docker.sock:ro -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e DD_API_KEY=7cfe87ce1756aea datadog/agent:latest

It will take a few minutes for Datadog to spin up the Datadog container, collect metrics on your existing containers and display them in the Datadog App.

Step 5 - Running the Redis Latency Chaos Engineering Experiment

We will use the Gremlin CLI attack command to create a latency attack.

Now use the Gremlin CLI (gremlin) to run a latency attack against the host from a Gremlin container:

1sudo docker run -d \      --net=host \      --pid=host \      --cap-add=NET_ADMIN \      --cap-add=SYS_BOOT \      --cap-add=SYS_TIME \      --cap-add=KILL \      -e GREMLIN_TEAM_ID="${GREMLIN_TEAM_ID}" \      -e GREMLIN_TEAM_SECRET="${GREMLIN_TEAM_SECRET}" \      -v /var/run/docker.sock:/var/run/docker.sock \      -v /var/log/gremlin:/var/log/gremlin \      -v /var/lib/gremlin:/var/lib/gremlin \    gremlin/gremlin attack latency

This attack will inject latency to the Redis host.

Now exit the container by running the following command:

1exit

Step 6 - Additional Chaos Engineering experiments to run on Redis

There are many Chaos Engineering experiments you could possibly run on your Redis infrastructure:

• Monitoring - do your monitoring tools enable you to monitor changes in Redis latency?
• Time Travel Gremlin - will changing the clock time of the host impact how Redis processes data?
• Latency & Packet Loss Gremlins - will they impact the ability to use Redis
• Disk Gremlin - will filling up the disk crash the host?

We encourage you to run these Chaos Engineering experiments and share your findings! To get access to Gremlin, sign up here.

Conclusion

This tutorial has explored how to install Redis and Gremlin in Docker containers for your Chaos Engineering experiments. We then ran a shutdown Chaos Engineering experiment on the Redis container using the Gremlin Latency attack.