Safer Kubernetes rollouts with minReadySeconds

Picture the scene: you’ve just deployed a rolling update to your service. Half of your pods are running the new version, they all passed their readiness checks, and Kubernetes terminated the old replicas.

Suddenly, the new pods start throwing 503 errors. Thankfully, you still have pods on the old version, so you stop the update. If the rollout had been a little bit faster, you’d have an outage.

This is the failure mode minReadySeconds exists to prevent. It tells Kubernetes to hold a new pod at arm’s length for a set number of seconds before trusting it enough to tear down the pod it replaces. In this blog, we’ll look at minReadySeconds in detail, and explain how you can use it to deploy changes more safely and confidently.

‍

What does minReadySeconds actually do?

minReadySeconds sets the minimum amount of time that a newly created pod must be in its Ready state, without any of its containers crashing, before a workload controller considers it Available. Available is a workload controller's judgment that a pod has stayed Ready long enough to be trusted. Controllers use it to gate rollout progression and to satisfy pod disruption budgets.

minReadySeconds applies to Deployments, StatefulSets, DaemonSets, ReplicaSets, ReplicationControllers, and Argo Rollouts. By default, it’s set to 0, which means Kubernetes considers the pod Available immediately after it’s Ready.

‍

Kubernetes uses a pod’s availability to manage:

• Rolling updates: the Deployment controller won’t terminate old pods in a batch until the new ones are Available. A pod that passes readiness then crashes shortly after never becomes Available, so the rollout stops instead of terminating your stable replicas.
• Pod disruption budgets: PDBs count the number of Available pods when determining how many pods to preserve. minReadySeconds can prevent a node drain from evicting a pod while its replacement is still stabilizing.
• Load balancer propagation: External load balancers can take several seconds to register a new backend. minReadySeconds provides a grace period for this process to complete before the old pod disappears.

‍

Why passing a readiness check isn’t enough

Readiness probes ask a yes-or-no question at a single point in time. A pod can say it’s Ready while still doing startup work, like priming a cache, opening a connection pool, or syncing cluster state. Pods like this can pass their initial checks and still provide degraded performance.

Without minReadySeconds, Kubernetes will interpret that pass as permission to remove the previous pod, trading a healthy replica for one that’s still spinning up. If the new pod crashes, that failure will propagate. Readiness probes determine when the pod can serve traffic, and minReadySeconds determines when the rollout accepts the pod’s availability and moves on to the next.

‍

How to find workloads without minReadySeconds set

minReadySeconds applies to workloads, not individual pods. To check deployments that are missing it or have it set to zero, you can use the kubectl and jq command-line tools (swap deployments for statefulsets or daemonsets, depending on your workload type):

kubectl get deployments -A -o json | jq -r '
  .items[]
  | select((.spec.minReadySeconds // 0) == 0)
  | "\(.metadata.namespace)/\(.metadata.name)"'

‍

Gremlin checks for minReadySeconds automatically as a built-in Detected Risk. Once you've defined a Kubernetes service in Gremlin, it continuously monitors that service's workloads and flags any missing minReadySeconds, including ones that were configured correctly and later drifted.

‍

How to configure minReadySeconds

Adding minReadySeconds requires a single field. In this example, Kubernetes will wait 30 seconds before removing old Nginx containers:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 4
  minReadySeconds: 30
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:stable-alpine
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5

‍

The strategy block controls how Kubernetes handles updates to this deployment. With maxUnavailable: 0 and maxSurge: 1, Kubernetes will bring up one new pod and wait for it to stay Ready for 30 seconds before terminating the old one and moving onto the next. This ensures that we never fall below our replica count, and an unstable pod stops the rollout instead of propagating through it.

You can verify this by running:

kubectl describe deployment nginx-deployment | grep MinReadySeconds

‍

MinReadySeconds:  30

‍

Then, during a rollout or update, you can watch the gap between pod readiness and availability. In this rolling update example, Kubernetes creates the new pod (Ready shows 5/4, waits for minReadySeconds to elapse, then replaces the old pod and increments Up-to-Date:

kubectl get deployment nginx-deployment -w

‍

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   5/4     1            4           2m40s
nginx-deployment   4/4     2            4           3m10s

‍

That 30-second gap is minReadySeconds doing its job.

‍

How to choose a value for minReadySeconds

The value you set for minReadySeconds depends on your workload and environment.

Start by taking measurements of your pod startup times. Record the time from when the pod becomes Ready to when your application’s metrics look normal. Use your service level indicators (SLIs) and service level objectives (SLOs), or look for metrics such as steady latency, zero errors, and baseline resource saturation. Repeat this for several pod starts, ideally under load, and take the 95th percentile to get a solid baseline.

Remember to account for other conditions that need to take place for the pod to start serving production traffic. If your cloud load balancer needs 15 seconds to register a backend, set a minimum of 15 seconds regardless of your application startup time.

Generally speaking, 10–60 seconds will cover most services. The tradeoff is in rollout time. Replacing four replicas with minReadySeconds: 30 and maxSurge: 1 adds two minutes to a deployment. That’s worth it for workloads with strict reliability requirements (tier 0 or tier 1, for example), but might not be for workloads that don’t directly serve production traffic.

‍

Other Kubernetes risks to watch for

minReadySeconds is just one chapter in the Kubernetes startup story:

• Startup probes give slow-starting containers time to breathe before liveness checks.
• Readiness probes prevent traffic from reaching a pod until it’s ready to serve.
• minReadySeconds keeps a rollout from trusting a pod for a set amount of time.

Pair these with liveness probes for ongoing health, and with topology spread constraints so the pods you do have aren't all sitting on one node.

These fields can be easy to forget or miss when building a new service, during a refactor, or when templating a Helm chart. That’s why continuous detection is important: a manifest that was reliable on the last major release might not be reliable today.

Want to find missing minReadySeconds and other common Kubernetes misconfigurations? Use Gremlin's Detected Risks to scan your Kubernetes services and spot reliability issues automatically.

‍