Managing slow container starts with Kubernetes readiness probes

Imagine if your workday started as soon as you woke up.

Before you can even start your coffee maker, email alerts are flooding in, coworkers are pinging you on Slack, and your phone is buzzing nonstop with reminders. You haven’t even pulled the covers back, and your boss is asking you about deliverables.

This is what Kubernetes pods deal with every day. Unless, that is, you use readiness probes. Readiness probes give your pods time to “wake up” before they start the heavy work of responding to traffic.

But how do readiness probes work, how do you configure them, and how do you keep track of which pods do and don’t have readiness probes defined? We’ll answer all of these questions in this blog.

‍

What are readiness probes, and why are they important?

Pods without readiness probes are like employees who start working the moment they wake up. Instead of making coffee and eating breakfast, they’re online responding to Slack messages and working on deliverables.

This might sound ideal from a productivity perspective, but the reality is more nuanced. The time between waking up and starting work isn’t wasted or unproductive: you’re preparing for the day ahead. Jumping straight into work would actually lower productivity, as you won’t be as prepared to respond to the day's challenges.

This is what happens to your pods when they don’t have readiness probes defined. The moment they come online, Kubernetes starts sending network traffic their way, even if your application is in the middle of its morning coffee. Readiness probes act as a flag that notify Kubernetes when a pod has finished its wakup routine and is ready to receive live traffic.

‍

When are pods considered “ready”?

A pod becomes “ready” once it finishes its setup process without errors, can respond successfully to requests, and is operating within its Service Level Objectives (SLOs).

‍

For example, consider a pod that serves a database. Before the pod can handle requests, it needs to:

1. Generate or download data to prime the database.
2. Open ports to allow network traffic.
3. Start the database process.

While technically you could send traffic to the pod while it’s starting, it can't properly respond until the database is running. It might hold the request in a queue or reject it outright, creating a poor user experience. Instead, let Kubernetes route requests to replicas instead.

When planning readiness probes, consider the final step in your pod or container’s setup process. When is your application ready to receive user traffic? What HTTP, gRPC, or console commands can you run to check this state? This determines how you willconfigure your readiness probe.

‍

How to find pods with missing readiness probes

An easy way to detect pods without readiness probes is to use the Kubernetes command-line tool and jq to parse its output. For example, to create a list of pods with missing probes, run:

kubectl get pods -o json | jq -r '.items[] | select(.spec.containers[].readinessProbe == null) | .metadata.name'

‍

Alternatively, Gremlin provides a built-in Detected Risk that automatically scans Kubernetes services for missing readiness probes.

‍

How to add a readiness probe to a pod

You can define a readiness probe using spec.containers[].readinessProbe.

‍

For example, this manifest creates an nginx container that hosts a file at http://nginx-readiness-demo/startup_file.txt. It downloads the file from GitHub using wget, saves it to Nginx’s default web root folder, then starts Nginx. We can verify that the file was downloaded using a readiness probe that checks the endpoint over HTTP:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx-readiness-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-readiness-demo
  template:
    metadata:
      labels:
        app: nginx-readiness-demo
    spec:
      containers:
      - name: nginx
        image: nginx:stable-alpine
        ports:
        - containerPort: 80
        # The startup command performs two steps:
        # 1. Download the file 
        # 2. Start Nginx in the foreground
        command: ["/bin/sh", "-c"]
        args:
          - |
            wget -qO- https://raw.githubusercontent.com/kubernetes/kubernetes/master/README.md > /usr/share/nginx/html/startup_file.txt;
            nginx -g 'daemon off;'
        # The Readiness Probe checks for the existence of the startup file
        readinessProbe:
          httpGet:
            path: /startup_file.txt
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 2
          failureThreshold: 3
          timeoutSeconds: 5

‍

• initialDelaySeconds is how long to wait after the container starts before starting the probe. The example checks after 5 seconds, but this setting defaults to 0 (i.e. Kubernetes will start checking immediately).
• periodSeconds is how long to wait before repeating the probe. The example repeats the probe every 5 seconds, but this defaults to 10.
• successThreshold is the minimum number of consecutive successes needed for the probe to be successful after having failed. For instance, if our Nginx container fails one probe, it will need to complete two successful probes in a row before being considered successful. This defaults to 1 (and must be set to 1 for liveness and startup probes).
• failureThreshold is the number of times the probe must fail consecutively for the check to be marked as failed. The example sets this to the default of 3.
• timeoutSeconds is how long to wait for a response before the probe times out. The example will wait 5 seconds, but this defaults to 1 second.

Start the container with kubectl apply -f nginx-startup-demo.yaml and monitor the state of your pods. 

NAME                     READY   STATUS              RESTARTS   AGE
nginx-5db4dfdc46-mxzr2   0/1     ContainerCreating   0          12s
nginx-5db4dfdc46-qpwtk   0/1     ContainerCreating   0          12s

‍

If all goes well, both pods will be marked as Ready...

NAME                     READY   STATUS    RESTARTS   AGE
nginx-5db4dfdc46-mxzr2   1/1     Running   0          29s
nginx-5db4dfdc46-qpwtk   1/1     Running   0          29s

‍

…and the readiness probe will appear when running kubectl describe pod [pod name]:

Readiness: http-get http://:80/startup_file.txt delay=5s timeout=1s period=5s #success=1 #failure=3

‍

Readiness probes vs. readiness gates

Kubernetes has another mechanism for tracking readiness called readinessGates. Where readiness probes are self-reported by the container, readiness gates use external mechanisms to validate readiness. Gates are typically updated by an external service: for example, AWS Load Balancers can update a pod’s readiness after adding the pod to its load balancing rules. Only when the readiness probes pass and readiness gates are set to true will the pod start receiving traffic. In short:

• Readiness probes check the pod’s internal state to determine whether to send traffic to it.
• Readiness gates use services outside of the pod (load balancers, gateways, firewalls, etc.) to determine whether to send traffic to it.

‍

Other Kubernetes risks to watch for

If you want to make sure your pod keeps running reliably after startup, consider pairing your readiness probe with a liveness probe. Liveness probes periodically check that your pod is still responsive by sending an HTTP, TCP, or gRPC request, or by executing a command in the container. The recommended approach is to set an initialDelaySeconds for the liveness probe to ensure the container has enough time to become ready. Unlike readiness probes, Kubernetes will kill the container if one of its liveness probes fails. Be sure to include enough retries and delays to prevent false positives, while not leaving so much delay that traffic slows down.

You can also use startup probes, which check to make sure the container process started successfully. These are useful for catching unexpected problems during startup, particularly long load times. Like liveness probes, startup probes will kill the container on failure.

To recap:

• Startup probes give your containers time to get running before liveness checks kick in.
• Readiness probes ensure your pods can handle traffic before they start receiving it.
• Liveness probes keep an eye on container health while they're running.

Want to find missing readiness probes and other common Kubernetes misconfigurations? Use Gremlin’s Detected Risks to scan your Kubernetes services and spot reliability issues automatically.