Streaming pipelines on Amazon MSK fail quietly and expensively. When a broker becomes unreachable, nothing crashes. Producers buffer, consumers stall, and lag accumulates at whatever rate your data arrives. The system looks alive the entire time.

The cost compounds rather than accruing evenly:

1. A broker or partition becomes unreachable, and the consumers reading from it stop making progress.
2. A consumer group rebalance pauses processing further while ownership moves.
3. Lag builds at full production rate for the duration, and new traffic keeps arriving on top of the backlog.
4. Catching up means processing the backlog alongside live traffic, which takes far longer than the disruption did.

Install the Gremlin Agent on the services producing and consuming, then run a blackhole experiment against broker connectivity to watch the rebalance happen. A latency experiment simulates a lagging broker, which is more common and harder to detect than a clean outage.

Lag is the metric everyone watches after an incident. How fast it drains, and whether your consumers can drain it at all, is worth establishing before the pipeline carries something time-sensitive.