Fault Injection


A GameDay is an organized team event to practice Chaos Engineering, test your incident response process, validate past outages, or find unknown issues in your services. Gremlin is your central playing field for all GameDay activities from planning, running, analyzing, and sharing the results with your organization.

In a Gremlin GameDay, you can:

  • Assign a goal for the GameDay
  • Link to your dashboards and runbooks
  • Invite team members to participate
  • Run up to 20 scenarios
  • Capture your learnings and attach screenshots from the GameDay
  • Create and track Jira issues for any issues found
  • Rerun the same GameDay

Creating GameDays

To create a GameDay in Gremlin:

  • On the navigation bar, click GameDays.
New GameDay button on GameDays landing page
  • On the GameDays page, click New GameDay.
  • On the New GameDays page, enter a name for the GameDay and complete the details:
  • Type: Select the type of GameDay you plan to run: Train Teams, Tune Observability, Validate Past Outages, Exploratory, or Other.
  • Goal: Depending on the selected GameDay type, this space will display a suggested goal. You can edit the text to your preference.
  • Scenarios: Add up to 20 scenarios with assigned targets to run during the GameDay. GameDays are limited to 20 scenarios to encourage a focused scope.
  • Environment: Indicate the environment to be used for the GameDay.
  • Dashboards: Link any relevant dashboards that you may need to reference during the GameDay.
  • Runbooks: Link any relevant runbooks that you may need to reference during the GameDay.
  • Participants: Select the GameDay participants and assign each to a position: Owner, Coordinator, Observer, or Reporter. You can have a maximum of 20 participants in a GameDay but Gremlin recommends only assigning only one Owner and one Coordinator per GameDay. The list contains existing Gremlin users. If you want to include a new user in a GameDay, invite them to Gremlin and then assign them a position.
New GameDay page

Running GameDays

  • When you are ready to run the GameDay, click the GameDays icon on the left navigation and then click the GameDay that you want to run.
  • On the GameDay Details page, you will see the configuration and scenarios. Click Run Scenario next to the scenario you want to run first.
GameDays Run Scenario
  • Gremlin will open the Scenario Runs tab where you can observe the live view of the scenario as it runs.
GameDay Details page - Scenario Runs tab
  • If you need to run additional scenarios, click the Configuration tab and select the next scenario you want to run. You can add results to every run and mark it as Passed or Failed once it has finished.
GameDay Summary page

Recording GameDay Results

While the GameDay is running, the Observer is gathering the data from Gremlin and other monitoring tools and can upload the necessary screenshots of the impact on the GameDay Summary page. The Reporter can do the same, and also record any notes or follow-up actions on the GameDay Summary page.

GameDay Summary page

After the GameDay is over, there might be a few action items that the team needs to take. These action items could be just actions recorded in plain text or some Jira issues that any participant can create and attach to the GameDay. To create Jira tickets from within Gremlin, the individual user will need to set up Jira integration. See Enabling Jira integration for more details.

No items found.
This is some text inside of a div block.
Installing the Gremlin Agent
Authenticating the Gremlin Agent
Configuring the Gremlin Agent
Managing the Gremlin Agent
User Management
Health Checks
Command Line Interface
Updating Gremlin
Reliability Management (RM) Quick Start Guide
Services and Dependencies
Detected Risks
Reliability Tests
Reliability Score
Deploying Failure Flags on AWS Lambda
Deploying Failure Flags on AWS ECS
Deploying Failure Flags on Kubernetes
Classes, methods, & attributes
API Keys
Container security
Additional Configuration for Helm
Amazon CloudWatch Health Check
AppDynamics Health Check
Blackhole Experiment
CPU Experiment
Certificate Expiry
Custom Health Check
Custom Load Generator
DNS Experiment
Datadog Health Check
Disk Experiment
Dynatrace Health Check
Grafana Cloud Health Check
Grafana Cloud K6
IO Experiment
Install Gremlin on Kubernetes manually
Install Gremlin on OpenShift 4
Installing Gremlin on AWS - Configuring your VPC
Installing Gremlin on Kubernetes with Helm
Installing Gremlin on Windows
Installing Gremlin on a virtual machine
Installing the Failure Flags SDK
Latency Experiment
Memory Experiment
Network Tags
New Relic Health Check
Packet Loss Attack
PagerDuty Health Check
Preview: Gremlin in Kubernetes Restricted Networks
Private Network Integration Agent
Process Collection
Process Killer Experiment
Prometheus Health Check
Configuring Role Based Access Control (RBAC)
Running Failure Flags experiments
Scheduling Scenarios
Shared Scenarios
Shutdown Experiment
Managing Teams
Time Travel Experiment
Troubleshooting Gremlin on OpenShift
User Authentication via SAML and Okta
Managing Users
Integration Agent for Linux
Test Suites
Restricting Testing Times
Process Exhaustion Experiment
Enabling DNS collection
Authenticating Users with Microsoft Entra ID (Azure Active Directory) via SAML
AWS Quick Start Guide
Installing Gremlin on Amazon ECS