SLOs and the error budget
How reliable is reliable enough? Turn "never go down" into a number you can spend, then build the alert that wakes someone for a real fire and sleeps through a blip.
“Never go down” sounds like the right goal. It’s the wrong one. The SRE book says it plainly: “100% is probably never the right reliability target.” Nobody can reach it, users can’t tell 99.99% from 100% through their own flaky Wi-Fi, and every step closer costs more and slows every change.
So reliability work starts by choosing a number, and the number gets three names.
- An SLI (service level indicator) is “a carefully defined quantitative measure of some aspect of the level of service”. For example: the share of checkout requests that succeed in under 500 ms.
- An SLO (service level objective) is “a target value or range of values for a service level that is measured by an SLI”. For example: 99.9% of them, measured over 30 days.
- An SLA (service level agreement) is a contract with users “that includes consequences of meeting (or missing) the SLOs”. Miss it and someone pays. That’s why an SLA is usually looser than the SLO the team works to.
The budget
If the target is 99.9%, then 0.1% failure is allowed. That allowance is the error budget. Thirty days is 30 × 24 × 60 = 43,200 minutes, and 0.1% of that is 43.2 minutes. That’s the budget for total outage in a month, or the same failure spread thinner: 86 minutes of half the requests failing spends it just as surely.
The budget turns an argument into arithmetic. Budget left: ship, experiment, take risks. Budget spent: the SRE book’s answer is that “releases are temporarily halted” until reliability recovers. Nobody has to win a debate about whether the product is “stable enough”.
- Start with what users feel, measured: of the requests that should have worked, how many did?
- The SLO is the target for that measurement - here 99.9% over a rolling 30 days.
- Whatever the target does not demand is the error budget: 0.1% of 43,200 minutes is 43.2 minutes to spend.
- The SLA is the promise to customers, set looser than the SLO, so the team notices long before a penalty is due.
- And the alert does not watch errors. It watches how fast the budget is going.
Your SLO is 99.9% over 30 days. Last week a bad deploy failed every request for 30 minutes. How much of the month's budget is left?
43.2 - 30 = 13.2 minutes, about 30% of the budget. The 70% is what the outage spent, not what is left. An SLO is a budget, not a promise of zero failures. The team can still ship, carefully - and if the next incident spends the rest, releases stop until the window rolls past.
The alert is the hard part
A budget that nobody watches is a number in a slide deck. The obvious alert is “page when the error rate goes above 0.1%”. The SRE Workbook uses exactly that as its first example, and explains why it fails: it “fires on any event that threatens the SLO”, which is good, but its “precision is low: The alert fires on many events that do not threaten the SLO.” Their example: “A 0.1% error rate for 10 minutes would alert, while consuming only 0.02% of the monthly error budget.”
The opposite mistake is to average over a long window so blips vanish. Now a real fire takes so long to move the average that the budget is half gone before anyone is told.
The workbook’s answer is the burn rate: “how fast, relative to the SLO, the service consumes the error budget.” A burn rate of 1 spends the budget in exactly 30 days. A burn rate of 14.4 spends 2% of it in a single hour. Its recommended policy for a 99.9% SLO:
| Burn rate | Long window | Short window | Budget spent | Action |
|---|---|---|---|---|
| 14.4 | 1 hour | 5 minutes | 2% | page |
| 6 | 6 hours | 30 minutes | 5% | page |
| 1 | 3 days | 6 hours | 10% | ticket |
The long window proves the burn is real, not a blip. The short window proves it is still happening, so the alert stops firing soon after the fire is out: “ceasing to fire five minutes later, rather than one hour later.”
Try all three alerts against all three incidents. Only one gets every call right.
Why does the burn-rate page need a short window as well as a long one?
The 1-hour window stays high for up to an hour after a fix. Requiring the last 5 minutes to be burning too means the page resolves as soon as the fire is out - the workbook's 'reset time'.
Everything above ran in your browser. Now do it on a real cluster withthe tilt platform. You need Docker, kind and Tilt, about 15 GB of free disk and 8 GB of spare RAM. Setup guide · Can my machine run it?
On the tilt platform, Prometheus scrapes Istio’s request metrics (istio_requests_total), the same series lab 06’s canary analysis reads. Send some traffic to hello, then in Grafana’s Explore (grafana.localhost/explore) try an availability SLI over the last hour:
sum(rate(istio_requests_total{destination_service_name="hello", response_code!~"5.."}[1h]))
/
sum(rate(istio_requests_total{destination_service_name="hello"}[1h]))Subtract that from 1 and divide by 0.001 - that is, (1 - SLI) / 0.001 - and you have the 1-hour burn rate for a 99.9% SLO. If the query returns nothing, check the label names first with istio_requests_total{destination_service_name=~"hello.*"}: labels differ between gateway and sidecar proxies, and an empty result is a missing match, not a perfect service.
Sources
Missions and quizzes above earn their own XP. This marks the lesson done.