Gremlin Scenario: Kubernetes node redundancy

How it works

How this Scenario works

This Scenario drops all network traffic to a targeted Kubernetes node, cutting it off from the rest of the cluster and from anything sending traffic to it. The control plane, your load balancers, and your API gateways all have to detect the loss and reschedule the affected pods elsewhere.

Use cases

Why run this Scenario?

Network connections can fail for any number of reasons, including:

Failures in downstream systems (dependencies) or networking hardware.
Misconfigured firewall and router rules, such as the 2021 Fastly outage.
Saturation caused by unexpected surges in user traffic, high-bandwidth data transfers, or other causes.
Applications with poorly configured connection, timeout, and/or retry logic.

With this Scenario, you can validate that:

Your Kubernetes cluster can gracefully handle losing a node.
Your master nodes are redundant and won't create a split-brain situation.
Load balancers and API gateways are configured properly.

Result

What to expect when you run it

When a node fails, Kubernetes automatically re-routes traffic to healthy nodes and recreates failed pods.

Keep going

More Scenarios to try

Kubernetes packet loss

Ensure your Kubernetes systems can detect and route around resources that fail health checks. This Scenario gradually increases packet loss to validate liveness/readiness probe and load balancer behavior.

Container packet loss

Ensure your containerized systems can detect and route around resources that fail health checks. This Scenario gradually increases packet loss to validate health check and load balancer behavior.

Windows packet loss

Ensure your Windows systems can detect and route around resources that fail health checks. This Scenario gradually increases packet loss to validate health check and load balancer behavior.

Kubernetes node redundancy