Canary, blue-green and the rollback
Every release strategy eventually puts the new version in front of users - the difference is how many, and who notices first. Ship a broken build three ways and count the failed requests.
Tests catch the bugs you thought of. Some only show up under real traffic: a slow query on production data, a config value that only exists in one region, a build that answers 6% of requests with an error. A release strategy decides how much of your traffic finds out first.
Three are worth knowing:
- Rolling update. Replace pods a few at a time. Each new pod waits only for its readiness probe. You met this in A typo is not an outage.
- Blue-green. Run the new version (green) beside the old one (blue), test it off to the side, then flip the router. Martin Fowler’s description: you switch the router so all requests go to green, and if anything goes wrong you switch it back to blue.
- Canary. Send a small slice of real traffic to the new version, measure it, and only widen the slice while the numbers hold. The Google SRE Workbook calls the part that gets the change “the canary” and the rest “the control”.
- Users reach a gateway that can split traffic by weight. Before the release, all of it goes to the primary.
- The new version starts as the canary and gets one step of traffic: 25% in tilt lab 06.
- Every interval the analysis reads the canary's success rate. Below 99% it halts and counts a failed check.
- Healthy: raise the weight, up to 50%, then promote. Three failed checks: all traffic back to the primary.
How Flagger decides
The Flagger docs list the canary’s loop step by step. Each interval it checks the canary’s request success rate and latency, and halts advancement if a metric is under its threshold, incrementing a failed-checks counter. When that counter reaches the threshold, it routes all traffic to the primary, scales the canary to zero and marks it failed. While the checks pass, it raises the weight by one step until it reaches the maximum, then promotes.
The tilt platform’s lab 06 configures exactly that: a step of 25%, a maximum of 50%, a success-rate floor of 99%, and a threshold of 3 failed checks. The widget below uses the same numbers.
Try it
What each one cost
With the same broken build, the rolling update fails more and more requests as each batch of pods turns over, until everyone is on v3. Blue-green puts everyone on v3 at once, so 6% of every user’s requests fail from the first second, and it recovers as fast as someone flips the switch back - which makes it great when a person is watching and painful at 3am. The canary showed v3 to a quarter of traffic for three intervals, then backed out on its own.
None of them prevented the bad version from meeting users. They decide how many users, and whether a human has to be awake.
A new version's pods pass their readiness probes but return errors on 6% of requests. Which strategy notices on its own?
Readiness says the pod can take traffic, not that it answers correctly. Only a strategy that measures real request outcomes - the canary's success rate - sees the 6%.
In tilt lab 06 the canary's success rate reads 94% three intervals in a row. What does Flagger do?
Each check under the 99% floor halts advancement and increments the failed-checks counter. At the threshold of 3, Flagger routes all traffic back to the primary and marks the canary failed.
What is blue-green's main advantage over a rolling update when things go wrong?
Fowler's description: if anything goes wrong you switch the router back to blue. The cost is that everyone sees green at once, and you run two full environments during the switch.
Everything above ran in your browser. Now do it on a real cluster with tilt lab 06-canary (profile gitops). You need Docker, kind and Tilt, about 15 GB of free disk and 8 GB of spare RAM. Setup guide · Can my machine run it?
Follow the lab’s setup (waypoint, metric template, load tester, Canary), then ship a good version and a bad one:
kubectl -n hello set env deploy/hello PODINFO_UI_MESSAGE="canary v2"
kubectl -n hello get canary hello -w # WEIGHT 25 -> 50, then Promoting, Succeeded
kubectl -n hello set image deploy/hello podinfo=ghcr.io/stefanprodan/podinfo:does-not-exist
kubectl -n hello get canary hello -w # never Ready: Failed, primary unchanged
kubectl -n hello describe canary hello | tail -15
./scripts/platform.sh lab 06The bad version here is a different failure from the widget’s v3: its image doesn’t exist, so the pod never becomes Ready and Flagger fails the release at progressDeadlineSeconds, before it shifts any traffic. The widget’s v3 is Ready and still wrong, which only the success-rate check can catch.
Read the lab’s “Leaving the lab” section before you move on: deleting the Canary alone leaves the route pointing at services that no longer exist.
Sources
Missions and quizzes above earn their own XP. This marks the lesson done.