Reliability Intelligence: Route network calls to healthy availability zones

Sometimes, availability zones will encounter network disruption without a full-blown shutdown of the availability zone. Gremlin’s official Zone Redundancy test can be used to simulate such a situation.

Under these conditions, your application loses its ability to reach network dependencies in the targeted zone, but can remain running in the affected zone without the connectivity it needs to function correctly. Make sure your application can make network calls for its upstream dependencies both within its availability zone and across other availability zones. Patterns like circuit-breakers allow failing over from one endpoint to another, while configuring applications with even and adequate capacity across all zones ensures traffic spikes in any zone can be tolerated.

Related

• Best Practices for Testing Zone Redundancy
• Ensure all zones have enough capacity to handle a sudden increase in traffic from zone failover.

‍

Cross-Zone Load Balancing

Cross-zone load balancing reduces the need to maintain equivalent numbers of instances in each enabled Availability Zone, and improves your application's ability to handle the loss of one or more instances. When cross-zone load balancing is disabled, each load balancer node distributes traffic only across the registered targets in its Availability Zone.

For AWS users, Gremlin can automatically detect when a service is attached to a load balancer that has cross-zone load balancing disabled and raise it as a detected risk.

Example: Configure cross-zone load balancing for AWS Load Balancers

‍

Considerations for Kubernetes

ClusterIP Service Routing

Many configurations of kube-proxy will route traffic to service endpoints using an algorithm that achieves an overall even distribution to backends. While this is usually desirable, this behavior does not always account for crossing availability-zone boundaries. Furthermore, because services of type ClusterIP are represented by a virtual IP address, applications cannot perform any client-side failover when connections to some endpoints fail and others succeed. Service mesh technologies such as Istio, LinkerD, and Consul provide circuit-breaking features so that connections to endpoints that result in failure are avoided.

‍

Traffic Distribution

A service’s spec.trafficDistribution defines how traffic to services should be distributed over the cluster’s topology. In particular, the PreferSamezone (also known as PreferClose) option prioritizes sending traffic to endpoints in the same zone as the client.

It’s important to note however that this routing configuration means some zones may receive more traffic than others if the clients producing this traffic are not equally distributed across zones. To account for this, ensure that both your client and server applications are evenly distributed across availability zones (using Pod Topology Constraints), with enough capacity to handle situations where traffic from one zone must be routed to another. See Ensure all zones have enough capacity for more details.