Reliability Intelligence: Spread load across replicas to reduce error spikes

Applications cannot always terminate gracefully. Sometimes an application is forcibly terminated without a chance to finish any pending requests it is working on. This is expected during situations of hardware failure, power loss, or Out of Memory (OOM) events. In these situations, application redundancy not only helps ensure new traffic is routed to live replicas, but can also help reduce the number of errors that users encounter by reducing the number of pending requests on any given replica at a time.

The best way to spread request load across more replicas is to configure autoscaling based on a metric that measures the number of incoming requests per replica.

‍

Solution: Raise minimum number of replicas (Easy)

Raising the minimum number of replicas that make up an application is an easy, albeit imperfect way to control the number of active requests on each replica. Enforcing a minimum helps control the impact of active requests when a given replica fails.

Using this formula, you can gauge how many replicas you may need to drive targetRequestPerReplica down to an acceptable level:

totalRequests / minimumReplicas = targetRequestPerReplica

For example:

30 average active requests across the application / 3 application replicas = 10 active requests per replica

Keep in mind however that this approach has its downsides:

• Inability to scale down past this minimum replica threshold when request rates are low, consuming more resources than necessary.
• Inability to react to increased request rates, leading to overscaling (consuming more resources) or underscaling when demand is high.

‍

Solution: Scale replicas based on request load (Moderate)

Reacting to live request load is a much better way to ensure active requests across replicas are kept to a reasonable level. By configuring autoscaling events based on the number of active requests coming into the application, you can scale up and down to meet a specific target of active requests per replica. Implementing this will be specific to the autoscaling infrastructure your application uses.

‍

Example: Autoscaling based on AWS ElasticLoadBalancing using Prometheus

If your application’s incoming request load can be measured from an AWS LoadBalancer or TargetGroup and you’re using Prometheus to inform a Kubernetes Horizontal Pod Autoscaler, you can leverage the Prometheus CloudWatch Exporter.

‍

Create a cloudwatch-exporter.yaml deployment and config map:

Ensure the pod has AWS IAM permissions (via IRSA or node role) to access cloudwatch:GetMetricData.

# cloudwatch-exporter-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: cloudwatch-exporter-config
data:
  config.yml: |
    region: us-west-2
    metrics:
      - aws_namespace: AWS/ApplicationELB
        aws_metric_name: RequestCount
        dimensions: [LoadBalancer, TargetGroup]
        statistics: [Sum]
        period_seconds: 60
        range_seconds: 600
        delay_seconds: 60

# cloudwatch-exporter-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudwatch-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cloudwatch-exporter
  template:
    metadata:
      labels:
        app: cloudwatch-exporter
    spec:
      containers:
        - name: exporter
          image: prom/cloudwatch-exporter:latest
          args: ["--config.file=/config/config.yml"]
          ports:
            - containerPort: 9106
          volumeMounts:
            - name: config-volume
              mountPath: /config
      volumes:
        - name: config-volume
          configMap:
            name: cloudwatch-exporter-config

‍

Expose the Exporter to Prometheus

# cloudwatch-exporter-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: cloudwatch-exporter
  labels:
    app: cloudwatch-exporter
spec:
  ports:
    - name: metrics
      port: 9106
      targetPort: 9106
  selector:
    app: cloudwatch-exporter

‍

And configure your Prometheus scrape config:

# prometheus.yml
scrape_configs:
  - job_name: 'cloudwatch-exporter'
    static_configs:
      - targets: ['cloudwatch-exporter:9106']

‍

Setup Prometheus Adapter

Here is an example of configuring prometheus-adapter via Helm:

rules:
  custom:
    - seriesQuery: 'aws_applicationelb_request_count_sum{TargetGroup="targetgroup/my-app"}'
      resources:
        overrides:
          namespace: {resource: "namespace"}
      name:
        matches: "aws_applicationelb_request_count_sum"
        as: "request_count"
      metricsQuery: 'sum(rate(aws_applicationelb_request_count_sum{TargetGroup="targetgroup/my-app"}[2m]))'

‍

Apply these values and deploy/update the adapter. You should now be able to query custom/request_count.

‍

Configure HPA

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: External
      external:
        metric:
          name: request_count
        target:
          type: Value
          value: "100"  # target requests per second

‍

This example will scale the deployment my-app up/down based on whether the request rate (RPS) for its ALB Target Group exceeds 100 RPS.

Tips

• Use stabilization parameters of HPA to control how quickly scaleUp and scaleDown should happen to protect against flapping.