Running Chaos Engineering Experiments With Gremlin Free And Spinnaker

Introduction

Spinnaker is an open source cloud-based Continuous Delivery platform for releasing software, built by Netflix and backed by Google, Microsoft, and Oracle. Gremlin is a simple, safe and secure service for performing Chaos Engineering experiments through a SaaS-based platform. Gremlin Free is a free version of Gremlin that can run on up to five hosts, and run two types of Chaos Engineering attacks.

Prerequisites

To complete this tutorial you will need:

  • A Mac with a minimum of 16GB Memory running Java 8 to install Spinnaker on. We tested this tutorial running Spinnaker locally on a Mac running macOS Mojave, but it may work on other operating systems. You can get the Java 8 JDK from Oracle here. This host also needs a git client, yarn and node installed. We recommend installing them with Homebrew.
  • A host running Ubuntu 16.04 to run the Chaos Engineering experiments on. This host will run the Gremlin agent. You need to be able to run commands as root with sudo on this host.
  • A Gremlin Free account (sign up here).

Overview

This tutorial will walk you through the steps to install Spinnaker from source code, and set it up to run Gremlin attacks.

  • Step 1 - Install the Gremlin agent
  • Step 2 - Create a Gremlin API key
  • Step 3 - Create a Target Template in Gremlin
  • Step 4 - Create an Attack Template in Gremlin
  • Step 5 - Install Redis on the Spinnaker host
  • Step 6 - Install Spinnaker components from GitHub
  • Step 7 - Add an application in Spinnaker
  • Step 8 - Run a Gremlin attack using Spinnaker

Step 1 - Install the Gremlin Agent

On your Ubuntu host that we’ll be running experiments on, install the Gremlin agent using the instructions here.

Step 2 - Create a Gremlin API Key

Sign into your Gremlin Free account. Go to the Company Settings menu:

Company settings

Click on the team name.

Click team name

On the next screen click on the API Keys tab and New API Key.

New API key

Fill in the name and a description for the key and hit Save.

Save API key

You should then see a screen that shows the API key that was created.

Step 3 - Create a Target Template in Gremlin

There are two types of templates in Gremlin, and we will need both to be able to run Chaos Engineering experiments from Spinnaker. Attack Templates defines the type of attack that will run. Target Templates define the hosts or containers those attacks will run on.

To create a Target Template, click Attacks in the upper left and New Attack. Select the Ubuntu host you installed the Gremlin agent on:

New Attack

In the Choose The Targets section click the three dots on the right and select “Save as template…”

Save as template

On the next screen fill in a name for the template and hit Done.

Ubuntu host

Step 4 - Create an Attack Template in Gremlin

Still on that same New Attack screen, scroll down to the part of the page that says Choose A Gremlin.

spinnaker choose a gremlin

This is where we select the type of attack to run. CPU should be selected by default, which is one of the attack types we can use with Gremlin Free. If it’s not selected click on it and use the default settings.

On the right of Choose A Gremlin you’ll see the three dots again. Click those and select “Save as template…”

CPU template

Step 5 - Install Redis on the Spinnaker host

Install Redis using the package manager for the operating system of your Spinnaker host. If you’re using macOS you can install it with Homebrew (“brew install redis”).

Launch the redis-server process:

redis-server

It will run by default on port 6379.

Step 6 - Install Spinnaker components from GitHub

We’re going to use git to clone repositories for the Spinnaker components from GithHub, as the changes to integrate with Gremlin are recent and may not be available if you install Spinnaker using another method.

Make a directory somewhere on the Spinnaker host to clone the files into, and cd to that directory.

mkdir ~/spinnaker
cd spinnaker

Next we’ll clone the repo for each component and then run the corresponding server. We recommend running them each in their own terminal window or tab so you can see their output isolated. Note that you may see errors in the output of some of the services until you have them all running, as some of the Spinnaker services depend on each other.

Before your proceed to the next step check that you have Java 8 installed:

java --version

If you don’t yet have Java 8 installed you can install it from the Oracle Java Downloads website.

Clone and run Orca

Orca is the orchestration engine for Spinnaker. It is responsible for taking a pipeline or task definition and managing the stages and tasks, coordinating the other Spinnaker services. Orca pipelines are composed of stages which in turn are composed of tasks. The tasks of a stage share a common context and can publish to a global context shared across the entire pipeline allowing multiple stages to co-ordinate. For example a bake stage publishes details of the image it creates which is then used by a deploy stage. Orca persists a running execution to Redis.

Clone the repo:

cd ~/spinnaker
git clone https://github.com/spinnaker/orca.git
cd orca

Edit the file orca-web/config/orca.yml. Find the setting for gremlin.enabled and change it to true.

integrations:
  gremlin:
    enabled: true
    baseUrl: https://api.gremlin.com/v1

Run the Orca server:

./gradlew

It will run on port 8083.

Clone and run Deck

Deck is the management UI for Spinnaker.

You will need to make sure node and yarn are installed on your system to install and run Deck. We tested this tutorial with node version 11.12.0 and yarn version 1.15.2.

Clone the repo:

cd ~/spinnaker
git clone https://github.com/spinnaker/deck.git
cd deck

Install the dependencies with yarn:

yarn

Run the Deck server:

API_HOST=http://localhost:8084 yarn run start

It will run on port 9000.

Clone and run Gate

Gate is the Spinnaker API Gateway. This service provides the Spinnaker REST API, servicing scripting clients as well as all actions from Deck.

Clone the repo:

cd ~/spinnaker
git clone https://github.com/spinnaker/gate.git
cd gate

Edit gate-web/config/gate.yml. Find the setting for gremlin.enabled and change it to true.

integrations:
  gremlin:
    enabled: true
    baseUrl: https://api.gremlin.com/v1

Run the Gate server:

./gradlew

It will run on port 8084.

Clone and run Front50

Front50 is a cross-application persistence service for Spinnaker. Front50 fronts a Spinnaker datastore and is written using Spring Boot.

Clone the repo:

cd ~/spinnaker
git clone https://github.com/spinnaker/front50.git
cd front50

Edit front50-web/config/front50.yml and set spinnaker.redis.enabled to true.

spinnaker:
  redis:
    enabled: true
    host: localhost
    port: 6379

Run the Front50 server:

./gradlew

It will run on port 8080.

Clone and run Clouddriver

Clone the repo:

cd ~/spinnaker
git clone https://github.com/spinnaker/clouddriver.git
cd clouddriver

Run the Clouddriver server:

./gradlew

It will run on port 7002.

Clone and run Echo

Echo serves as a router for events that happen within Spinnaker.

Clone the repo:

cd ~/spinnaker
git clone https://github.com/spinnaker/echo.git  
cd echo 

Run the Echo server:

./gradlew

It will run on port 8089.

Echo is the last component to install. It may take a few minutes for all of the components to get running.

Accessing the Spinnaker UI

At this point you should have a working Spinnaker instance running on the host. It will run on port 9000.

Access it locally on your Mac at:

http://localhost:9000

The UI will look like this:

Spinnaker UI

Step 7 - Create an application in Spinnaker

On the right side of the Spinnaker UI is a menu called Actions. Click on that and select Create Application.

You’ll then see a dialog box for the new application’s settings. Enter the application name and an email address, and select github as the repository type. You don’t need to enter the address for an actual GitHub repository in Repo Project, you can leave that blank. But you need to select the repository type for it to work, you’ll get an error otherwise. You can optionally add a description.

Then hit the Create button in the bottom right.

New application

If it succeeds you should end up at this next screen:

After creating application

Click on Pipelines and "Configure a new pipeline."

Configure a new pipeline

Enter a Name for the Pipeline and click Create.

On the next screen we’ll add a Stage in the pipeline for the Gremlin experiment to run in. Click on “Add stage” under Configuration:

Add stage

For Type select Gremlin:

Type

The next screen is where we’ll enter the API key that we created earlier in the Gremlin UI. This will allow Spinnaker to access the templates we created and use them to launch experiments.

We’ll retrieve the API key that we created earlier using the Gremlin UI. In the Gremlin UI click Company Settings again and then your Team name. Click the API Keys tab and you’ll see the API Key that you created earlier.

View API Key

There will be an icon that looks like an eye. Hover your mouse over it and you’ll see a small popup with the instruction to click on it to copy the API key. Click on that popup. It will turn green and the text will change to say Successfully Copied.

API key copied

Paste that API Key text into the API Key field in Spinnaker and hit the Fetch button below it.

Paste in API key

This will make an API call to Gremlin and retrieve the list of templates you have available. Select the names of the templates you created earlier under Target Template and Attack Template, and then hit the Save Changes button in the bottom right of the Spinnaker window. You can leave the other settings on that screen set to the defaults.

Spinnaker listing templates

The Save Changes button should change to a checkmark and the text “In sync with server.”

You now have a pipeline for running your Gremlin experiment in Spinnaker!

Step 8 - Run a Gremlin attack using Spinnaker

Click on Pipelines in the Spinnaker UI and you should see the pipeline you created listed. Click on the Start Manual Execution Link to the right and it will kick off a Gremlin attack using the templates you provided.

Pipeline running

Once the attack has completed you should see the status in Spinnaker change to Succeeded.

Pipeline succeeded

You can verify that the attack actually ran in the Gremlin UI by going to the Attacks screen. The top entry in the list should be the attack you just ran.

Completed attacks

Click it to see the details.

Attack details

The User will correspond to the name of the API Key you used.

You’ve now successfully run a Gremlin attack from Spinnaker! At this point if you’d like to explore more, you can create additional templates and pipelines for more experiments to run from Spinnaker.

Conclusion

In this tutorial you’ve learned how to create API keys and templates in Gremlin, how to clone and run Spinnaker from source code, and how to run Chaos Engineering experiments using Gremlin from Spinnaker. If you have any questions, check out our documentation or get in touch on the #support channel in our community Chaos Engineering Slack.

Avoid downtime. Use Gremlin to turn failure into resilience.

Gremlin empowers you to proactively root out failure before it causes downtime. Use Gremlin for Free and see how you can harness chaos to build resilient systems.

Use For Free