Test how your application handles a Pub/Sub outage. Run Gremlin network experiments and reliability tests to validate publisher and subscriber resilience.
Messaging services are supposed to decouple your services, and Google Cloud Pub/Sub does that well...until delivery stops behaving the way your code assumes.
Pub/Sub guarantees at-least-once delivery, which means duplicate messages are a normal operating condition rather than an exception. Google covers the redelivery semantics in its exactly-once delivery documentation:
"A subscriber that misses its acknowledgment deadline receives the same message again while it is still processing the first copy. If that handler was never made idempotent, one delayed message becomes two charges, two shipments, or two notifications to the same customer."
These failures are silent. Nothing errors, no alarm fires, and the damage surfaces days later during reconciliation. Backlogs behave the same way: a stalled subscriber collects messages faster than it can process them, and the recovery takes far longer than the disruption that caused it.
Gremlin helps you find these problems while the stakes are low. Disrupt the network path between your service and the topic while messages are flowing, and you learn exactly how your publishers and subscribers respond. A blackhole experiment makes the endpoint unreachable, letting you test your retention settings. A latency experiment pushes subscribers past their acknowledgment deadline so you can intentionally test redelivery and ensure your subscribers process their backlogs reliably. Then, automate these experiments as Reliability Tests so every consumer stays scored and validated, even as your system scales.
Keep your event-driven architecture reliable with Gremlin.
Dependencies: Failure Test
Simulate a failed dependency by dropping all network traffic to the dependency.
Dependencies: Latency Test
Recreate poor network conditions by delaying all network traffic to a dependency by 100ms.
Unreliable Networks - Dependencies
Simulate unreliable network conditions when communicating with dependencies by adding latency to API calls. Test whether your users are affected when dependency response times degrade.
* Testing a dependency doesn't require installing anything on it. Gremlin runs the experiment from the service that calls it, so compatibility depends on that host rather than on the dependency itself. Check our compatibility documentation for supported operating systems and platforms, or get in touch if you don't see yours.
Intelligent Health Checks - GCP
Automatically monitor your Google Cloud services during testing in one-click with Intelligent Health Checks.
Avoid downtime. Use Gremlin to turn failure into resilience.
Gremlin empowers you to proactively root out failure before it causes downtime. See how you can harness chaos to build resilient systems by requesting a demo of Gremlin.
.svg)