Incidents and blameless postmortems

An incident is anything that breaks badly enough to need a coordinated response: checkout failing, data going missing, a security alert you can’t explain. The technical fix is usually done by one or two people. Whether it takes twenty minutes or three hours often depends on everything around them.

Declare a severity

Severity levels let everyone agree, quickly, how big this is and who needs to know. PagerDuty publishes the scale it uses for its own incidents; at the top end:

• SEV-1: “Critical issue that warrants public notification and liaison with executive teams.”
• SEV-2: “Critical system issue actively impacting many customers’ ability to use the product.”
• SEV-3: “Stability or minor customer-impacting issues that require immediate attention from service owners.”

And one rule worth copying exactly: “If you are unsure which level an incident is (e.g. not sure if SEV-2 or SEV-1), treat it as the higher one.” Downgrading later is easy. Discovering an hour in that the right people were never called isn’t.

Roles: one person coordinates, others fix

The most common way an incident goes wrong is the best engineer trying to fix it and run it. PagerDuty’s response guide splits the jobs:

• Incident commander: “acts as the single source of truth of what is currently happening”. Their rule: “Delegate all repair actions, the Incident Commander is NOT a resolver.”
• Resolvers (subject matter experts): the people who investigate and fix.
• Customer liaison: “responsible for interacting with customers, either directly, or via our public communication channels.”
• Scribe: “documents the timeline of an incident as it progresses”. That timeline becomes the postmortem.

1. 1One commander: the single source of truth about what is happening and what happens next. They do not fix anything.
2. 2The fix belongs to resolvers - the people who know the system. The commander asks, decides and unblocks.
3. 3A liaison keeps customers and the rest of the company informed, so the resolvers are not answering the same question twenty times.
4. 4A scribe records the timeline as it happens. Afterwards, that record is the start of the postmortem.

Status updates people can use

“We are investigating an issue” tells a customer nothing. A useful update says what’s broken and for whom, what you’re doing, what still works, and when the next update will come. Promising a fix time you don’t know just creates a second broken promise.

The postmortem, without blame

Google’s SRE book defines a postmortem as “a written record of an incident, its impact, the actions taken to mitigate or resolve it”, together with its causes and the follow-up actions to stop it happening again. It lists triggers for writing one, including “user-visible downtime or degradation beyond a certain threshold”, “data loss of any kind” and “a monitoring failure (which usually implies manual incident discovery)”.

The word that matters is blameless. A blameless postmortem focuses on “identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior”, and assumes people acted in good faith on the information they had. The reason is practical, not polite: under a culture of finger pointing, the book warns, “people will not bring issues to light for fear of punishment”. The near-miss that would have warned you next time goes unreported.

So the question is never “who pushed the bad change?” It’s “what allowed a bad change to reach every customer at once, and why did it take 25 minutes to notice?” Those questions have fixes. “Be more careful” doesn’t.

Run one incident, start to finish.