A typo is not an outage
A rolling update retires old pods only as new ones become Ready, beyond a small allowance. Ship a broken image tag, watch the old version keep serving, and finish the rollout properly.
When you change a Deployment’s image, Kubernetes doesn’t stop everything and start again. It makes a new ReplicaSet for the new pod template and moves pods across a few at a time. That’s a rolling update, and its whole job is to never leave you with nothing serving.
Two knobs control the pace. maxSurge sets how many extra pods may exist during the update, and maxUnavailable sets how many may be missing. Both default to 25%. The docs put it as: at least 75% of the desired pods are up, and at most 125% exist.
The key rule: beyond the maxUnavailable allowance, an old pod is removed only when a new one is Ready. The allowance is 25% rounded down: zero for the three replicas below, one for four, two for ten. So if the new version never becomes Ready, at most that 25% of the old version is ever removed. With small replica counts, none is.
Ship a typo
Stuck is not the same as fixed
The Kubernetes docs walk through exactly this case, a mistyped image tag, and say plainly: the rollout gets stuck. After progressDeadlineSeconds (600 by default) the Deployment reports ProgressDeadlineExceeded. Then, again in the docs’ words, Kubernetes takes no action on a stalled Deployment other than to report a status condition. Nothing rolls back by itself. A person (or a tool like Flagger) has to.
Your options are what you just did: fix the tag and roll forward, or kubectl rollout undo to go back to the previous template.
When the safe default is the wrong one
Rolling updates assume old and new pods can run side by side. A database with one volume can’t. The tilt platform’s lab 07 reproduces a real incident: a MySQL Deployment on RollingUpdate, where the new pod waits forever for a file lock the old pod holds. 733 restarts, two days, zero alerts. The old pod kept serving the whole time, so everything looked green. For one replica on one ReadWriteOnce volume, strategy: Recreate, which stops the old pod first, is the only correct choice.
You deploy an image tag that doesn't exist. With the default strategy, what do users see?
Beyond maxUnavailable (25%, rounded down), old pods are retired only as new ones become Ready. A new pod stuck in ImagePullBackOff never is, so the rollout stalls with the old version still serving: all of it at small replica counts, at least 75% of it at large ones.
A rollout has been stuck for 20 minutes. What has Kubernetes done about it?
The docs say Kubernetes takes no action on a stalled Deployment beyond reporting the condition. Higher-level tools can act on it; plain Deployments don't.
A single-replica database Deployment with one ReadWriteOnce volume. Which strategy?
RollingUpdate starts the new pod first, and it can never get the lock or mount the old pod holds. Recreate accepts a short gap in exchange for an update that can finish. A StatefulSet is the fuller answer.
Everything above ran in your browser. Now do it on a real cluster with tilt lab 07-stateful-rollout (profile minimal). You need Docker, kind and Tilt, about 15 GB of free disk and 8 GB of spare RAM. Setup guide · Can my machine run it?
kubectl apply -f labs/07-stateful-rollout/mysql.yaml
./labs/07-stateful-rollout/break.sh
kubectl -n lab07 get pods -w
kubectl -n lab07 logs -l app=mysql --tail=5 | grep -i lock
./labs/07-stateful-rollout/fix.sh
./scripts/platform.sh lab 07Then try the typo for real on hello:
kubectl -n hello set image deploy/hello podinfo=ghcr.io/stefanprodan/podinfo:does-not-exist
kubectl -n hello get rs
kubectl -n hello rollout undo deploy/helloSources
Missions and quizzes above earn their own XP. This marks the lesson done.