Test Amazon EKS worker node reliability with Gremlin. Simulate node failures and resource pressure to validate pod rescheduling and cluster resilience.
Amazon EKS is built for high reliability. ReplicaSets, liveness probes, and the cluster autoscaler are all designed to keep workloads running when a node disappears. That safety net is real, and it is also the reason most teams have never checked whether it holds for their own services.
Node loss on EKS is routine. Spot instance reclamation, version upgrades, and autoscaling events remove nodes every week, and your cluster absorbs them quietly until the day it doesn't. Install the Gremlin Agent on your worker nodes to prepare for that day. With Gremlin, you can:
- Prove your pods reschedule cleanly. Run a shutdown experiment to remove a node while you monitor your services. You'll find out whether pod disruption budgets were ever set, how long it takes Kubernetes to schedule replacements onto nodes with capacity, and how long your service runs degraded before recovering.
- Test the upgrades you can't opt out of. Surge upgrades cycle nodes faster than Kubernetes can migrate slow-starting containers. Applying CPU and memory pressure during a rolling replacement shows you whether your cluster has the headroom it needs before AWS schedules the next version bump.
- Validate redundancy across availability zones. Blackhole experiments isolate a zone so you can confirm the remaining zones carry your traffic. Multi-AZ node groups look correct in Terraform, but only a test proves your topology constraints distribute pods the way you expect.
CPU scalability - Kubernetes
Test that your service scales as expected when CPU capacity is limited. Gremlin will consume CPU in 3 stages: 50%, 75%, and 90%.
Kubernetes node redundancy
Test your Kubernetes cluster's node redundancy by dropping all network traffic to a node.
Zone redundancy - Kubernetes
Test your Kubernetes service's availability when a randomly selected availability zone becomes unreachable. Verify that traffic fails over to pods in secondary zones.
Region Evacuation - Kubernetes
Test your Kubernetes service's availability when an entire cloud region becomes unavailable. Verify that traffic automatically fails over to clusters in backup regions without impacting the user experience.
* Gremlin is designed to work on any cloud platform that provides Linux or Windows hosts. We haven't individually tested every service we cover, and not all are officially supported. Check our compatibility documentation for tested operating systems and known caveats, or get in touch if you don't see yours.
Intelligent Health Checks - AWS
Automatically monitor your AWS services during testing in one-click with Intelligent Health Checks.
AWS integration
Integrate Gremlin with your AWS account for automatic Health Check creation, service discovery, and more.
Troubleshooting Failure Flags
Diagnose and resolve common issues with Failure Flags, Gremlin's application-based fault injection feature.
Using Failure Flags by proxy
Use Failure Flags, Gremlin's application-level fault injection feature, without changing a single line of code.
Reliability Intelligence: Spread load across replicas to reduce error spikes
Reduce error spikes during replica termination by spreading traffic more evenly across a larger number of pods.
Installing Gremlin on Amazon ECS
Learn how to install Gremlin on EC2-backed Amazon Elastic Container Service (ECS) deployments.
Installing Gremlin on AWS - Configuring your VPC
Amazon Web Services (AWS) has unique networking requirements that must be implemented for Gremlin to run successfully…
Amazon CloudWatch Health Check
Learn how to monitor the health of your systems during testing by using your existing AWS CloudWatch alarms.
Deploying Failure Flags on AWS ECS
This document will walk you through setting up Failure-Flags-Sidecar for your ECS Tasks. Failure-Flags-Sidecar runs…
Deploying Failure Flags on AWS Lambda
This document will walk you through setting up the Failure Flags agent for Lambda Functions. The Failure Flags agent…
Failure Flags
Gremlin Failure Flags lets you run Chaos Engineering experiments and reliability tests on serverless workloads…
Scenarios
A Scenario is a set of Health Checks and Gremlin experiments that you can define, along with a name, description…
Experiments
An experiment is a method of injecting failure into a system in a simple, safe, and secure way. Learn how easy it is to run experiments in Gremlin.
Targets
A target is any infrastructure or application resource that you can run experiments on. This can include Amazon EC…
Services
Gremlin tests your services to ensure they're resilient to traffic surges, network latency, cloud provider blackouts, and more. Learn how in our docs.
Configuring the Gremlin Agent
This documentation page shows you how to configure the Gremlin Agent. You can configure Gremlin using either environment…
Avoid downtime. Use Gremlin to turn failure into resilience.
Gremlin empowers you to proactively root out failure before it causes downtime. See how you can harness chaos to build resilient systems by requesting a demo of Gremlin.
