Why did it restart?
Two pods crash forever, both with exit code 137, and they need opposite fixes. Learn which two fields tell them apart before you reach for more memory.
CrashLoopBackOff is the most famous status in Kubernetes, and it tells you almost nothing. It means the container keeps exiting and the kubelet keeps restarting it, waiting longer each time. The docs give the curve: 10s, 20s, 40s and so on, capped at 5 minutes, resetting once a container has run for 10 minutes without trouble.
So something is wrong. The status won’t say what.
The number that isn’t an answer
The usual next step is the exit code. Here’s the trap: both pods in this lesson exit with 137. That’s 128 + 9, and signal 9 is SIGKILL. It tells you something killed the process. It doesn’t tell you who did it or why.
This isn’t made up. The tilt platform hit both for real. Keycloak was OOMKilled at a 2Gi limit, eight restarts deep. Wazuh’s API was killed by its own liveness probe before it finished starting, and the error that surfaced pointed at a different component entirely.
The two fields that actually tell you
- The termination reason (
Last State: Terminated, Reason: ...inkubectl describe pod).OOMKilledmeans the kernel killed it for using more memory than its limit. - The events.
Liveness probe failedfollowed byKillingmeans the kubelet killed it because a health check gave up.
Why a startup probe, not a longer delay
The right fix for a slow start is a startupProbe. The Kubernetes docs say that while a startup probe is configured, liveness and readiness checks don’t run until it succeeds. Startup gets as long as it needs, and then liveness takes over with its normal tight interval.
The tempting alternative is a big initialDelaySeconds on the liveness probe. It works on day one, but then a real hang takes minutes longer to notice, forever.
A pod's last termination shows exit code 137. What do you know for certain?
137 only means killed by signal 9. OOMKilled is one cause and a liveness kill is another. Read the termination reason and the events before choosing a fix.
An app takes 60 seconds to start and its liveness probe begins checking after 15. What's the best fix?
A startup probe holds liveness off until the app has started, then liveness keeps its quick interval for real hangs. Deleting liveness means a hung process is never restarted.
A container exits with code 1 and its previous logs say 'DATABASE_URL is not set'. Which command showed you that?
After a restart, plain kubectl logs shows the new container. --previous shows the one that crashed, which is where the reason usually is. Try panda/needs-config:1.0 in the playground.
Everything above ran in your browser. Now do it on a real cluster with tilt lab 09-why-did-it-restart (profile minimal). You need Docker, kind and Tilt, about 15 GB of free disk and 8 GB of spare RAM. Setup guide · Can my machine run it?
kubectl apply -f labs/09-why-did-it-restart/workloads.yaml
kubectl -n lab09 get pods -w # wait a couple of minutes
kubectl -n lab09 get pods -o jsonpath='{range .items[*]}{.metadata.labels.app}{" "}{.status.containerStatuses[0].lastState.terminated.reason}{"\n"}{end}'
./labs/09-why-did-it-restart/fix.sh
./scripts/platform.sh lab 09Re-break it with break.sh, not kubectl apply. The lab README explains why apply leaves a half-broken state.
Sources
Missions and quizzes above earn their own XP. This marks the lesson done.