Database secondary failover
Drop all network traffic to your database's primary node.
Experiment types
Blackhole
Targets
Dependencies
Length
5 minutes
How this Scenario works
This Scenario drops all network traffic to your database's primary node, making it unreachable to both your applications and the rest of the database cluster. That forces the cluster to detect the loss and promote a secondary, while your applications handle the interruption in database access.
Why run this Scenario?
Dependencies are unreliable and can fail at any time. This is especially true for databases. Do you know if your database will reliably failover when the primary instance fails? And how does your application tolerate this failover process if it's slow?
Running this Scenario helps you:
- Verify that your database cluster can reliably failover from a primary instance to a secondary.
- Detect incomplete or corrupt data due to slow replication.
- Verify that your application gracefully handles a temporary interruption in database access.
What to expect when you run it
The database fails over to the secondary node with:
- No interruption in service,
- No data loss,
- No impact to our applications and services.
More Scenarios to try
Kubernetes packet loss
Ensure your Kubernetes systems can detect and route around resources that fail health checks. This Scenario gradually increases packet loss to validate liveness/readiness probe and load balancer behavior.
Container packet loss
Ensure your containerized systems can detect and route around resources that fail health checks. This Scenario gradually increases packet loss to validate health check and load balancer behavior.
Windows packet loss
Ensure your Windows systems can detect and route around resources that fail health checks. This Scenario gradually increases packet loss to validate health check and load balancer behavior.
Avoid downtime. Use Gremlin to turn failure into resilience.
Gremlin empowers you to proactively root out failure before it causes downtime. See how you can harness chaos to build resilient systems by requesting a demo of Gremlin.
.svg)