Scenarios

Overview

A Scenario is a set of Gremlin attacks that you define with a title, hypothesis, and detailed results. Once created and configured, a scenario is run to inject failure through a sequence of attacks with an expanding blast radius. After each run of a scenario, you can easily record your observations.

Scenarios can be used to recreate a past outage that impacted your system, or it can be used to automate a sequence of attacks to iteratively grow the blast radius of a Chaos Engineering experiment.

Creating a Scenario

Name and Hypothesis

Create a scenario by entering both a name and a hypothesis. For the hypothesis, think through how you expect your application or environment to behave when a sequence of attacks within your scenario is run. How will your system react to the failure modes you'll be exposing it to? Is your system designed to handle this type of failure? If so, is the system working as designed?

A hypothesis can be useful to record and validate expected behavior and assumptions about your system to see how actual behavior may differ from what is expected.

Attacks

Start crafting your scenario by either adding a completed attack or by creating a new attack. Selecting the option to add a completed attack will show you a list of all attacks that have run in the past. Find the attack you'd like to add to the scenario and choose targets to impact. Once you've selected the tags of hosts to impact, the attack will be added to your scenario.

Alternatively, to start with a new attack, add a new attack to the scenario. Select the tags of hosts to impact, followed by the attack type and configuration options.

Continue to add as many attacks to the scenario as you like, changing the tags of hosts to target and attack configuration as desired to grow the blast radius.

A scenario can only include one attack type.

A delay of 5 seconds is added between each attack you add to the scenario. This delay can be set to any number of seconds, minutes, or hours.

Targets

Use tags to select targets for attacks within your scenario. By default, all hosts or containers available to your team are selected. To narrow the blast radius of the attack, either select a specific tag of hosts, or decrease the percentage or number of hosts to impact.

Selecting individual hosts or containers is not an option for attacks within a scenario, tags must be used.

Drafts & Archive

When at least one attack has been added to a scenario, Save the scenario to have it enter a draft state. The scenario will be visible in the list of scenarios and the attack configuration can continue to be edited.

When a scenario is no longer relevant or needed, you can archive the scenario to remove it from the active set. Navigate to the Archived tab to view these scenarios. Unarchive a scenario if you'd like to return it to the set of active scenarios.

Running a Scenario

To start running through the sequence of attacks defined in a scenario, run the scenario from either the scenario configuration view or on each scenario card. A scenario will show that it's running with a message at the top along with a Halt Scenario button. Halting a scenario will halt the attack underway, along with the scenario itself so that no further attacks will begin. The active attack within the scenario will be visually indicated. As each attack completes, the state and logs of each attack will be available.

Results

The scenario details view shows on which day and when the scenario has been run. For each run of a scenario, the result of the scenario is available. You can enter observations for the scenario run, as well as indicating with checkboxes whether the scenario produced an expected result or if an incident was detected and/or mitigated.