CPU Experiment Pack

CPU Experiment Pack

Description

This Gremlin Free CPU Experiment Pack shares how you can utilise the Gremlin CPU attack. CPU Attacks are available with your Gremlin Free account.

What’s Included

  • Ability to run a CPU attack on cloud infrastructure hosts & containers on AWS, Azure, GCP & more
  • Ability to run a CPU attack on cpu containers running locally with Docker

What we’ll break

With Gremlin Free, you have the ability to cpu attack any host or container wherever it may reside.

This pack includes 4 x 5 minute experiments:

  • Experiment 1: CPU Attack a cloud infrastructure host using Gremlin Free
  • Experiment 2: CPU Attack a cloud infrastructure Docker container using Gremlin Free
  • Experiment 3: CPU Attack a kubernetes pod using Gremlin Free
  • Experiment 4: CPU Attack a local Docker container using Gremlin Free
  • Experiment 5: CPU Attack a cloud infrastructure host using the Gremlin Free API

What you’ll need

Get ready to unleash chaos, get your credentials!

After you have created your Gremlin Free account (sign up here) you will need to get your Gremlin Free Daemon credentials.

Login to the Gremlin App using your Company name and sign-on credentials. These details were emailed to you when you signed up to start using Gremlin. Make a note of your Team ID and Secret.

Get a free Datadog trial to monitor your chaos 💥

Learn how to implement Chaos Engineering with Gremlin and Datadog.

Get your free 1 week Datadog trial account here: https://www.datadoghq.com/partner/gremlin-chaos-monitoring/.

Install your Datadog agent on each of your hosts or as a container by following the Datadog agent installation guide available within the Datadog App.

Experiment 1: CPU Attack a cloud infrastructure host using Gremlin Free

Step 1.0 - Installing the Gremlin Daemon and CLI

First, ssh into your server and add the Gremlin Debian repository:

$ echo "deb https://deb.gremlin.com/ release non-free" | sudo tee /etc/apt/sources.list.d/gremlin.list

Import the repo’s GPG key:

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C81FC2F43A48B25808F9583BDFF170F324D41134 9CDB294B29A5B1E2E00C24C022E8EF3461A50EF6

Then install the Gremlin daemon and CLI:

$ sudo apt-get update && sudo apt-get install -y gremlind gremlin

Step 1.1 - Configuring the Gremlin Daemon

You’ll need to register with the Gremlin control plane to create a new Gremlin client session.

Start by initializing Gremlin and assigning tags with the following command. Substitute your desired tag name for service=api:

gremlin init

Now you’re ready to run attacks using the Gremlin Free.

Step 1.2 - Run a CPU Attack Using Gremlin Free

Using your Gremlin login credentials (which were emailed to you when you created your account), log in to the Gremlin App. Then click Create Attack.

First choose your target by selecting the host you registered with Gremlin:

cpu target

Next we will use the Gremlin App to create a CPU Attack. Choose the Resource Category and Select the CPU Attack:

cpu attack

Click Unleash Gremlin and the Gremlin Free CPU Attack will consume CPU resources on your host.

You can now view the Gremlin Free CPU Attack in Datadog:

cpu infra attack datadog

Experiment 2 - CPU a cloud infrastructure container using Gremlin Free

Step 2.0 – Install Docker

In this step, you’ll install Docker.

Add Docker’s official GPG key:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Use the following command to set up the stable repository.

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

Update the apt package index:

sudo apt-get update

Make sure you are about to install from the Docker repo instead of the default Ubuntu 16.04 repo:

apt-cache policy docker-ce

Install the latest version of Docker CE:

sudo apt-get install docker-ce

Docker should now be installed, the daemon started, and the process enabled to start on boot. Check that it’s running:

sudo systemctl status docker

Make sure you are in the Docker usergroup, replace $USER with your username:

sudo usermod -aG docker $USER

Log out and back in for your permissions to take effect, or type the following:

su - ${USER}

Step 2.1 – Create an htop container for monitoring

Htop is an interactive process viewer for UNIX. We’ll use it to monitor the progress of our attacks.

First create the Dockerfile for your htop container:

$ vim Dockerfile

Add the following to the Dockerfile:

FROM alpine:latestRUN apk add --update htop && rm -rf /var/cache/apk/*ENTRYPOINT ["htop"]

Build the Dockerfile and tag the image:

$ sudo docker build -t htop .

Run htop inside a container, this will monitor the host:

$ sudo docker run -it --rm --pid=host htop

To exit htop, enter q.

Next we will create an NGINX container and monitor it directly by joining the container’s pid namespace.

Step 2.2 – Create an NGINX container to attack

First we will create a directory for the html page we will serve using nginx:

$ mkdir -p ~/docker-nginx/html
$ cd ~/docker-nginx/html

Create a simple HTML page:

$ vim index.html

Paste in this content:

<html>    <head>        <title>Docker nginx tutorial</title>        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous">    </head>    <body>        <div class="container">            <h1>Hello it is your container speaking</h1>            <p>This nginx page was created by your Docker container.</p>            <p>Now it’s time to create a Gremlin attack.</p>        </div>    </body></html>

Create a container using the nginx Docker image:

$ sudo docker run -l service=nginx --name docker-nginx -p 80:80 -d -v ~/docker-nginx/html:/usr/share/nginx/html nginx

Make sure the docker-nginx container is running:

$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                       CREATED                 STATUS                   PORTS                         NAMES352609a67e95        nginx               "nginx -g 'daemon of…"        33 seconds ago          Up 32 seconds       0.0.0.0:80->80/tcp              docker-nginx

Step 2.3 – Set up your Gremlin client credentials

The Gremlin daemon (gremlind) connects to the Gremlin backend and waits for attack orders from you. When it receives attack orders, it uses the CLI (gremlin) to run the attack.

$ export GREMLIN_TEAM_ID=your_team_ID
$ export GREMLIN_TEAM_SECRET=your_team_secret

Step 2.4 - Run the Gremlin Free Daemon in a Container

Use docker run to pull the official Gremlin Free Docker image and run the Gremlin Free daemon:

$ sudo docker run -d \    --net=host \    --pid=host \    --cap-add=NET_ADMIN \    --cap-add=SYS_BOOT \    --cap-add=SYS_TIME \    --cap-add=KILL \    -e GREMLIN_TEAM_ID="${GREMLIN_TEAM_ID}" \    -e GREMLIN_TEAM_SECRET="${GREMLIN_TEAM_SECRET}" \    -v /var/run/docker.sock:/var/run/docker.sock \    -v /var/log/gremlin:/var/log/gremlin \    -v /var/lib/gremlin:/var/lib/gremlin \    gremlin/gremlin daemon

Use docker ps to see all running Docker containers:

$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                NAMES7167cacb2536        gremlin/gremlin     "/entrypoint.sh daem…"   40 seconds ago      Up 39 seconds                            practical_benzfb58b77e5ef8        nginx               "nginx -g 'daemon of…"   10 minutes ago      Up 10 minutes       0.0.0.0:80->80/tcp   docker-nginx

Jump into your Gremlin container with an interactive shell (replace 7167cacb2536 with the real ID of your Gremlin container):

$ sudo docker exec -it 7167cacb2536 /bin/bash

From within the container, check out the available attack types:

$ gremlin help attack-container
Usage: gremlin attack-container CONTAINER TYPE [type-specific-options]Type "gremlin help attack-container TYPE" for more details:  blackhole # An attack which drops all matching network traffic  cpu   # An attack which consumes CPU resources  io    # An attack which consumes IO resources  latency # An attack which adds latency to all matching network traffic  memory  # An attack which consumes memory  packet_loss # An attack which introduces packet loss to all matching network traffic  shutdown  # An attack which forces the target to shutdown  dns   # An attack which blocks access to DNS servers  time_travel # An attack which changes the system time.  disk    # An attack which consumes disk resources  process_killer  # An attack which kills the specified process

Then exit the container.

Step 2.5 – Run a CPU Attack against the NGINX container from a Gremlin Container

In this step we will run gremlin attack-container to target the NGINX container by its ID and run a CPU Attack against it.

Before running the CPU attack, use htop to monitor the docker-nginx container (replace f291a040a6aa with your docker-nginx container ID):

sudo docker run -it --rm --pid=container:f291a040a6aa htop

Run the following to create the CPU container attack against the container (replace f291a040a6aa with your docker-nginx container ID):

sudo docker run -d -it \    --cap-add=NET_ADMIN \    -e GREMLIN_TEAM_ID="${GREMLIN_TEAM_ID}" \    -e GREMLIN_TEAM_SECRET="${GREMLIN_TEAM_SECRET}" \    -v /var/run/docker.sock:/var/run/docker.sock \    gremlin/gremlin attack-container f291a040a6aa cpu

Now use your htop container to monitor the results:

sudo docker run -it --rm --pid=container:f291a040a6aa htop

You will also see Gremlin Free consuming CPU resources in Datadog:

cpu attack datadog

Experiment 3 - CPU a kubernetes pod using Gremlin Free

Kubernetes is a container management system which is built with reliability in mind. Architecture is commonly 1 master and 2 or more nodes which are replicated from the master. When the master dies the nodes are ready to replace it. When one node dies another will be ready to replace it.

Step 3.0 - Downloading your Gremlin client certificates

After you have created your Gremlin account (sign up here) you will need to find your Gremlin Daemon credentials. Login to the Gremlin App using your Company name and sign-on credentials. These were emailed to you when you signed up to start using Gremlin.

Navigate to Team Settings and click on your Team. Click the blue Download button to save your certificates to your local computer. The downloaded certificate.zip contains both a public-key certificate and a matching private key.

certs gremlin

Step 3.1 – Set up your Gremlin credentials

Unzip the certificate.zip and save it to your gremlin folder on your desktop. Rename your certificate and key files to gremlin.cert and gremlin.key.

gremlin certs downloaded

Next create your secret as follows:

kubectl create secret generic gremlin-team-cert --from-file=./gremlin.cert --from-file=./gremlin.key

Installation with Helm

The simplest way to install the Gremlin client on your Kubernetes cluster is to use Helm. If you do not already have Helm installed, go here to get started. Once Helm is installed and configured, the next steps are to add the Gremlin repo and install the client.

To run the Helm install, you will need your Gremlin Team ID. It can be found in the Gremlin app on the Team Settings page, where you downloaded your certs earlier. Click on the name of your team in the list. The ID you’re looking for is found under Configuration as Team ID.

Export your Team ID as an environment variable:

export GREMLIN_TEAM_ID="YOUR_TEAM_ID"

Replace YOUR_TEAM_ID with the Team ID you obtained from the Gremlin UI.

Next, export your cluster ID, which is just a friendly name for your Kubernetes cluster. It can be whatever you want.

export GREMLIN_CLUSTER_ID="Your cluster id"

Now add the Gremlin Helm repo, and install Gremlin:

helm repo add gremlin https://helm.gremlin.com
helm install gremlin/gremlin \
\--namespace gremlin \
\--name gremlin \
\--set gremlin.teamID=$GREMLIN_TEAM_ID \
\--set gremlin.clusterID=$GREMLIN_CLUSTER_ID

For more information on the Gremlin Helm chart, including more configuration options, check out the chart on Github.

Step 3.2 - Creating attacks using the Gremlin App

Example: Creating a CPU Attack against a Kubernetes node using the Gremlin App

You can use the Gremlin App or the Gremlin API to trigger Gremlin attacks. You can view the available range of Gremlin Attacks in Gremlin Help.

The “Hello World” of Chaos Engineering is the CPU Resource Attack. To create a CPU Resource Attack, click Attacks in the left navigation bar and New Attack.

Host targeting should be selected by default. Click on “local-hostname” to expand the list of available hosts, and select one of them. You’ll see the Blast Radius for the attack is limited to 1 host.

Click on “Choose a Gremlin” and select “Resource” and then “CPU.”

The CPU Resource Attack will consume CPU resources based on the settings you select. The most popular default settings for a CPU Resource Attack are pre-selected, a default attack will utilize 1 core for 60 seconds.

When your attack is finished it will move to Completed Attacks in the Gremlin App. To view the logs of the Attack, click on the Attack in Completed Attacks then click to the arrow to view the logs.

Experiment 4 - CPU a local Docker container using Gremlin Free

Step 4.0 – Install Docker For Mac

First you will need to install Docker For Mac if you do not yet have it on your local computer, follow the instructions provided by Docker.

Step 4.1 – Set up your Gremlin credentials

The Gremlin daemon (gremlind) connects to the Gremlin backend and waits for attack orders from you. When it receives attack orders, it uses the CLI (gremlin) to run the attack.

$ export GREMLIN_TEAM_ID=your_org_id
$ export GREMLIN_TEAM_SECRET=your_org_secret

Step 4.2 – Create a Gremlin Free Docker container

Use docker pull to pull the official Gremlin Free Docker image:

$ docker pull gremlin/gremlin

Step 4.3 – Create an NGINX container to attack

First we will create a directory for the html page we will serve using nginx:

$ mkdir -p ~/docker-nginx/html
$ cd ~/docker-nginx/html

Create a simple HTML page:

$ vim index.html

Paste in this content:

<html>    <head>        <title>Docker Nginx tutorial</title>        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous">    </head>    <body>        <div class="container">            <h1>Hello it is your container speaking</h1>            <p>This nginx page was created by your Docker container.</p>            <p>Now it's time to create a Gremlin attack.</p>        </div>    </body></html>

Create a container using the nginx Docker image:

$ sudo docker run -l service=nginx --name docker-nginx -p 80:80 -d -v ~/docker-nginx/html:/usr/share/nginx/html nginx

Make sure the docker-nginx container is running:

$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                NAMES7167cacb2536        gremlin/gremlin     "/entrypoint.sh daem…"   40 seconds ago      Up 39 seconds                            practical_benzfb58b77e5ef8        nginx               "nginx -g 'daemon of…"   10 minutes ago      Up 10 minutes       0.0.0.0:80->80/tcp   docker-nginx

Step 4.4 - Run A Gremlin Free CPU Attack

Now use the Gremlin CLI (gremlin) to run a CPU attack from within a Gremlin container:

sudo docker run -i \    --cap-add=NET_ADMIN \    -e GREMLIN_TEAM_ID="${GREMLIN_TEAM_ID}" \    -e GREMLIN_TEAM_SECRET="${GREMLIN_TEAM_SECRET}" \    -v /var/run/docker.sock:/var/run/docker.sock \    gremlin/gremlin attack-container docker-nginx cpu

This attack will run a CPU attack on your Nginx container.

Experiment 5: CPU Attack on a cloud infrastructure host using the Gremlin Free API

It is also possible to run attacks programmatically using the Gremlin Free API. We will run these attacks from a Mac OS laptop.

Step 5.0 - Installing the Gremlin Daemon and CLI

First, ssh into your server and add the Gremlin Debian repository:

echo "deb https://deb.gremlin.com/ release non-free" | sudo tee /etc/apt/sources.list.d/gremlin.list

Import the repo’s GPG key:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C81FC2F43A48B25808F9583BDFF170F324D41134 9CDB294B29A5B1E2E00C24C022E8EF3461A50EF6

Then install the Gremlin daemon and CLI:

sudo apt-get update && sudo apt-get install -y gremlind gremlin

Step 5.1 - Register your server to the Gremlin control plane

Using your Gremlin login credentials (which were emailed to you when you created your account), log in to the Gremlin App. Open Settings and copy your Team ID and Secret.

Initialise Gremlin by running the following command and follow the prompts to enter your Gremlin Team ID and Secret:

gremlin init

Now you’re ready to obtain your Gremlin Free API token.

Step 5.2 - Obtain your Gremlin Free API token

First access your Gremlin Free API token by providing your gremlin username and password to /users/auth, replacing your email, password and company name in the curl request below:

curl -X POST --header 'Content-Type: application/x-www-form-urlencoded' \    --data-urlencode 'email=name@youremail.com' \    --data-urlencode 'password=changeit' \    --data-urlencode 'companyName=changeit' \    'https://api.gremlin.com/v1/users/auth'

Your API token will be displayed on the screen. The example below shows "Bearer Y4MGE3MGMzOmdyZW1saW5AZ3JlbWxpbmluYy5jb206NWNhMWFiMWU" as the API token that will be used for interacting with the Gremlin Free API:

[  {   "expires_at": "2099-01-00T00:00:00.000Z",   "header": "Bearer Y4MGE3MGMzOmdyZW1saW5AZ3JlbWxpbmluYy5jb206NWNhMWFiMWU",   "identifier": "yourname@email.com",   "org_id": "e7352a6b-a9a0-513c-8000-980f680a70c3",   "org_name": "My Org (Production)",   "renew_token": "5ca1ab1e-ffff-0000ffff0001",   "role": "USER",   "token": "5ca1ab1e-ffff-0000ffff0000"  },]

Step 5.3 - Store your Gremlin Free API token

On your local computer (Mac OS), store your Gremlin Free API token as an environment variable:

$ export bearertoken="Bearer Y4MGE3MGMzOmdyZW1saW5AZ3JlbWxpbmluYy5jb206NWNhMWFiMWU"

Check to ensure you have set your token correctly:

$ echo $bearertoken

Your API token will be displayed on your terminal screen:

Bearer Y4MGE3MGMzOmdyZW1saW5AZ3JlbWxpbmluYy5jb206NWNhMWFiMWU

Now you’re ready to run attacks using the Gremlin Free API .

Step 5.4 - Run a CPU Attack Using The Gremlin Free API

Now you’re ready to run a shutdown attack using the Gremlin Free API . Run the following command on your local computer (Mac OS):

curl --header "Content-Type: application/json" \    --header "Authorization: $bearertoken" \    https://api.gremlin.com/v1/attacks/new \    --data '    {        "command": { "type": "cpu" },        "target": { "type": "Random" }    }'

View the attack in progress using the Gremlin Free API:

curl -X GET "https://api.gremlin.com/v1/attacks/active" -H "Authorization: $bearertoken" -H "accept: application/json"

Your current attack will be displayed in terminal:

[  {    "target_type": "Host",    "targets": [      "172.31.21.178"    ],    "org_id": "3f242793-018a-5ad5-8000-fb958f8dc084",    "args": [      "cpu"    ],    "created_at": "2019-03-07T19:14:51.379Z",    "create_source": "Api",    "stage": "Pending",    "stage_lifecycle": "Active",    "guid": "483c1c53-410d-11e9-8165-0242b395dd29",    "start_time": "2019-03-07T19:14:51.379Z",    "create_user": "yourname@email.com",    "updated_at": "2019-03-07T19:14:51.404Z",    "kind": "Api"  }

Further Attacks

Gremlin free unlocks the ability to perform CPU and Shutdown attacks. To unlock further attacks upgrade your Gremlin account by contacting our team sales@gremlin.com.

Avoid downtime. Use Gremlin to turn failure into resilience.

Gremlin empowers you to proactively root out failure before it causes downtime. Use Gremlin for Free and see how you can harness chaos to build resilient systems.

Use For Free