Test how your application handles an AWS MSK outage. Run Gremlin network experiments and reliability tests to validate streaming pipeline resilience.
Streaming pipelines on Amazon MSK fail quietly and expensively. When a broker becomes unreachable, nothing crashes. Producers buffer, consumers stall, and lag accumulates at whatever rate your data arrives. The system looks alive the entire time.
The cost compounds rather than accruing evenly:
- A broker or partition becomes unreachable, and the consumers reading from it stop making progress.
- A consumer group rebalance pauses processing further while ownership moves.
- Lag builds at full production rate for the duration, and new traffic keeps arriving on top of the backlog.
- Catching up means processing the backlog alongside live traffic, which takes far longer than the disruption did.
Install the Gremlin Agent on the services producing and consuming, then run a blackhole experiment against broker connectivity to watch the rebalance happen. A latency experiment simulates a lagging broker, which is more common and harder to detect than a clean outage.
Lag is the metric everyone watches after an incident. How fast it drains, and whether your consumers can drain it at all, is worth establishing before the pipeline carries something time-sensitive.
Kafka broker load on Linux
Test the impact of I/O pressure on Kafka broker throughput and partition synchronization on Linux hosts. As disk I/O usage increases on the broker's host, pipeline performance will decrease, but should stay within expected bounds.
Dependencies: Failure Test
Simulate a failed dependency by dropping all network traffic to the dependency.
Dependencies: Latency Test
Recreate poor network conditions by delaying all network traffic to a dependency by 100ms.
* Testing a dependency doesn't require installing anything on it. Gremlin runs the experiment from the service that calls it, so compatibility depends on that host rather than on the dependency itself. Check our compatibility documentation for supported operating systems and platforms, or get in touch if you don't see yours.
Intelligent Health Checks - AWS
Automatically monitor your AWS services during testing in one-click with Intelligent Health Checks.
AWS integration
Integrate Gremlin with your AWS account for automatic Health Check creation, service discovery, and more.
Using Failure Flags by proxy
Use Failure Flags, Gremlin's application-level fault injection feature, without changing a single line of code.
Installing Gremlin on Amazon ECS
Learn how to install Gremlin on EC2-backed Amazon Elastic Container Service (ECS) deployments.
Installing Gremlin on AWS - Configuring your VPC
Amazon Web Services (AWS) has unique networking requirements that must be implemented for Gremlin to run successfully…
Avoid downtime. Use Gremlin to turn failure into resilience.
Gremlin empowers you to proactively root out failure before it causes downtime. See how you can harness chaos to build resilient systems by requesting a demo of Gremlin.
.svg)