Test GCP Dataproc reliability with Gremlin. Simulate node failures and resource pressure to validate big-data job resilience.
Google Cloud Dataproc encourages ephemeral clusters: spin up, run the job, tear down. Clusters get built with preemptible workers to cut costs, which means node loss during a job isn't an accident. It's the expected behavior of the configuration you chose.
Whether that's acceptable depends entirely on the job, and most teams have never measured it:
- A preemptible worker disappears partway through a stage.
- Spark recomputes the lost partitions, assuming the lineage is still available.
- If intermediate state was lost, the stage restarts, and your cheap capacity has quietly become a longer runtime.
- Total compute cost rises above what on-demand instances would have cost, and nobody connects the two.
Install the Gremlin Agent on your cluster nodes and Shutdown experiments give you the real number: how much a mid-job interruption costs this specific workload. With that in hand, preemptible capacity becomes a deliberate choice per job rather than a default applied everywhere and hoped for.
Host failure - Linux
Prepare for Linux host failures by shutting down a percentage of your hosts and observing the results. Validate that your infrastructure handles host loss without impacting service availability.
CPU scalability - Linux
Test that your Linux-hosted service scales as expected when CPU capacity is limited. Gremlin consumes CPU in three stages—50%, 75%, and 90%—to validate scaling thresholds.
Scalability: Memory
Verify that your Linux-hosted service scales as expected when memory is limited. Gremlin increases memory utilization in three stages—50%, 75%, and 90%—to validate memory management.
Host redundancy - Linux
Test resilience to host failures by shutting down a randomly selected Linux host. Verify that your platform automatically restarts or replaces it.
* Gremlin is designed to work on any cloud platform that provides Linux or Windows hosts. We haven't individually tested every service we cover, and not all are officially supported. Check our compatibility documentation for tested operating systems and known caveats, or get in touch if you don't see yours.
Intelligent Health Checks - GCP
Automatically monitor your Google Cloud services during testing in one-click with Intelligent Health Checks.
Avoid downtime. Use Gremlin to turn failure into resilience.
Gremlin empowers you to proactively root out failure before it causes downtime. See how you can harness chaos to build resilient systems by requesting a demo of Gremlin.
.svg)