Gremlin Scenario: Kubernetes DNS redundancy

How it works

How this Scenario works

This Scenario blocks DNS traffic within your Kubernetes pods, simulating a primary DNS provider failure. This tests whether your Kubernetes DNS configuration—including CoreDNS and any external DNS providers—correctly handles failover.

Use cases

Why run this Scenario?

DNS can fail in many ways, including:

Failures in nameservers and/or recursive resolvers, causing DNS queries to time out or return errors.
Network saturation (or a DDoS attack) slowing down DNS queries or causing them to drop.
Misconfigured Quality of Service (QoS) rules, causing the network to de-prioritize DNS traffic.

Running this Scenario can help:

Verify that Kubernetes CoreDNS or kube-dns handles upstream DNS failures gracefully.
Validate that Kubernetes service discovery continues working when external DNS providers fail.
Test DNS policy and dnsConfig settings across your pods for resilience.
Ensure that DNS failures don't cascade into pod-to-pod or pod-to-service communication outages.

Result

What to expect when you run it

If the primary DNS provider fails, Kubernetes services continue resolving DNS queries through secondary providers with no interruption.

Keep going

More Scenarios to try

Kubernetes packet loss

Ensure your Kubernetes systems can detect and route around resources that fail health checks. This Scenario gradually increases packet loss to validate liveness/readiness probe and load balancer behavior.

Container packet loss

Ensure your containerized systems can detect and route around resources that fail health checks. This Scenario gradually increases packet loss to validate health check and load balancer behavior.

Windows packet loss

Ensure your Windows systems can detect and route around resources that fail health checks. This Scenario gradually increases packet loss to validate health check and load balancer behavior.

Kubernetes DNS redundancy