Gremlin Scenario: Database secondary failover

How it works

How this Scenario works

This Scenario drops all network traffic to your database's primary node, making it unreachable to both your applications and the rest of the database cluster. That forces the cluster to detect the loss and promote a secondary, while your applications handle the interruption in database access.

Use cases

Why run this Scenario?

Dependencies are unreliable and can fail at any time. This is especially true for databases. Do you know if your database will reliably failover when the primary instance fails? And how does your application tolerate this failover process if it's slow?

Running this Scenario helps you:

Verify that your database cluster can reliably failover from a primary instance to a secondary.
Detect incomplete or corrupt data due to slow replication.
Verify that your application gracefully handles a temporary interruption in database access.

Result

What to expect when you run it

The database fails over to the secondary node with:

No interruption in service,
No data loss,
No impact to our applications and services.

Keep going

More Scenarios to try

Kubernetes packet loss

Ensure your Kubernetes systems can detect and route around resources that fail health checks. This Scenario gradually increases packet loss to validate liveness/readiness probe and load balancer behavior.

Container packet loss

Ensure your containerized systems can detect and route around resources that fail health checks. This Scenario gradually increases packet loss to validate health check and load balancer behavior.

Windows packet loss

Ensure your Windows systems can detect and route around resources that fail health checks. This Scenario gradually increases packet loss to validate health check and load balancer behavior.

Database secondary failover