Yesterday I wrote up Google’s final report on the europe-west2 outage. Reading it again as a document rather than a story, it’s a very good postmortem. There’s a precise timeline in one timezone. The cause is specific enough to act on. Contributing conditions (the site was in a single-feed maintenance posture) are separated from the defect. And every action item has a status and a date.
Most internal postmortems don’t look like that. They’re written two weeks late by whoever drew the short straw, the timeline starts at the page instead of the first signal, the root cause is “human error,” and the action items say “be more careful.” Then they go into a folder nobody searches.
This is a practical guide to writing the other kind. It’s opinionated, and it ends with a small linter for the mistakes I’d otherwise point out in review.
Decide in advance when you write one
If writing a postmortem is a judgment call made after the incident, the answer will usually be no, because everyone’s tired and the fix is in. Set the triggers ahead of time. Google’s SRE book has a good default list:
- User-visible downtime or degradation beyond a threshold
- Data loss of any kind
- On-call intervention, such as a rollback or rerouting traffic
- Resolution time above a threshold
- A monitoring failure, which usually means a human found the problem before the alerts did
Add one more: anyone involved can ask for one. Some of the most useful postmortems are about incidents that nearly happened.
Blameless, defined properly
The SRE book’s definition is the one to use: the review focuses on contributing causes “without indicting any individual or team for bad or inappropriate behavior,” and it assumes everyone “had good intentions.”
Blameless doesn’t mean nameless. People can appear in the timeline: “14:09 Priya acknowledges the page.” What changes is the question you ask about their actions. Not “why didn’t Priya check the pool dashboard,” but “what did she see, what did the tools show her, and why did the next step make sense at the time?” If an action looks obviously wrong in hindsight but made sense to a competent person at 2pm with the information on their screen, then the information on the screen is your finding.
This isn’t about being nice. A review that ends at “Bob made a mistake” leaves every condition that set Bob up still in place, ready for the next person.
The sections, and what each one is for
Here’s the template I’d start from:
# <What users experienced, not the internal cause>
## Summary
Two or three sentences. What happened, how long, how bad, what fixed it.
A reader who stops here should know enough.
## Impact
User-facing numbers: failed requests, affected customers, duration,
data loss (or "none"), SLO budget consumed.
## Timeline
All times UTC. Start at the first signal, not the first page.
- 13:58 UTC First anomalous metric (in hindsight)
- 14:09 UTC Alert fires; on-call acknowledges
- ...
- 14:49 UTC Recovery confirmed
## Contributing factors
The conditions that made this possible, each one a thing you could change.
## What went well
## Where we got lucky
## Action items
| Action | Owner | Due | Ticket |
|---|---|---|---|The title describes what users experienced. “Checkout API 503s after a connection-pool change” is searchable six months from now. “Incident 2026-10-14” isn’t.
The summary is for the reader who’ll stop there, which is most of them. Write it last.
Impact goes in user-facing numbers. “Elevated error rates” means nothing to the business. “About 3,100 failed checkouts over 47 minutes” means something, and it lets you compare incidents. If you have SLOs, state how much error budget the incident consumed.
The timeline is in UTC (one timezone, always stated) and starts at the first signal, not the first page. The gap between “the metric started moving” and “a human knew” is often the most important finding in the whole document, and you can’t see it if the timeline starts at the alert. Pull timestamps from chat logs, deploy history and dashboards, not memory.
Contributing factors, plural. I avoid a single “root cause” heading for software incidents because it pushes you to stop at the first satisfying answer. Google’s report does name a root cause, and that’s fair for one physical wiring defect, but even there the incident only happened because of a contributing condition: the site was running on one feed during maintenance. Write each factor as a condition you could change.
What went well is not filler. If the alert fired in seven minutes, write it down, because it tells you what not to break while you fix everything else.
Where we got lucky is the section teams skip and the one I’d fight hardest to keep. “The bad config only reached one region because the deploy happened to fail halfway.” “It was a Tuesday afternoon, so the one person who knew the database was online.” Luck is an unfunded control. Every lucky break in this incident is a factor in the next one, unless you turn it into something deliberate.
Wording: the traps
Three phrases do most of the damage.
“Should have.” It’s a counterfactual, written by someone who knows how the story ends. “The on-call engineer should have checked the pool dashboard” tells you nothing about why they didn’t. Rewrite it as the condition: “The pool saturation metric wasn’t on the service dashboard or in the alert, so it was found 22 minutes in.”
“Failed to.” Same problem, and it reads as an accusation. “Bob failed to notice the pool size in review” becomes “The pool size change was one line in a 400-line config diff, and config changes didn’t go through canary.”
“Human error.” It’s where the analysis stopped, not where it ended. Every human error has conditions upstream of it: a tool that allowed the action without confirmation, an alert that didn’t exist, a review process that couldn’t catch it. Find those.
Action items: the part that actually prevents the next one
Every action item needs one named owner (a person, not a team), a due date and a ticket. Without the ticket it’ll be forgotten. Without a person, everyone will assume someone else has it.
Then rank them by strength. Roughly, from strongest to weakest:
- Eliminate the hazard. Remove the setting, delete the dangerous command, make the bad state impossible to represent.
- Guardrail or automate. The tool refuses, or the pipeline catches it, or a canary stops it before it spreads.
- Detect faster. An alert on the thing that actually broke.
- Document or train. Runbooks, checklists, onboarding.
Training has its place, but if all your action items sit in category 4, you’ve decided the system is fine and the people need fixing. “Be more careful reviewing config diffs” is a wish. “Route config changes through the canary pipeline” is an action item.
Last, track completion. Google’s report lists each preventative action with a status and a date, some marked complete and some with target dates in November. Do the same, and review the open ones monthly. A pile of unfinished action items is how the same incident happens twice, with an older postmortem that already described it.
The review meeting
A few things make the difference between a useful review and a performative one:
- Hold it within a week, while people remember why they did what they did.
- Have someone other than the incident commander facilitate. The IC is too close to it.
- Circulate the draft beforehand and spend the meeting on contributing factors and action items, not reading the timeline aloud.
- Invite the people who were paged and protect their account of events. If the review feels like a trial, the next postmortem will be less honest.
A linter for the usual mistakes
Most of the problems above are mechanical enough to check automatically. This is a small script I’d run in CI on the postmortems repo, or before the review meeting:
"""pmlint: catch the usual postmortem mistakes before the review meeting."""
import re, sys
REQUIRED = ["Summary", "Impact", "Timeline", "Contributing factors",
"What went well", "Where we got lucky", "Action items"]
BLAME = [r"\bshould have\b", r"\bfailed to\b", r"\bhuman error\b", r"\bcareless",
r"\bforgot to\b", r"\bobviously\b", r"\bjust needed to\b", r"\bneglected\b"]
WEAK_ACTIONS = [r"\bbe more careful\b", r"\bremind\b", r"\btraining\b",
r"\bmonitor closely\b", r"\binvestigate\b", r"\bdouble[- ]check\b"]
def lint(text):
issues, lines = [], text.splitlines()
headings = {m.group(1).strip().lower() for m in re.finditer(r"^##\s+(.+)$", text, re.M)}
for section in REQUIRED:
if section.lower() not in headings:
issues.append(f"missing section: {section}")
section = None
for n, line in enumerate(lines, 1):
if line.startswith("## "):
section = line[3:].strip().lower()
continue
for pat in BLAME:
if re.search(pat, line, re.I):
issues.append(f"line {n}: blame/hindsight wording '{re.search(pat, line, re.I).group(0)}'")
if section == "timeline" and re.match(r"^\s*[-*]\s*\d{1,2}:\d{2}", line) \
and not re.search(r"\bUTC\b|Z\b|[+-]\d{2}:?\d{2}\b", line):
issues.append(f"line {n}: timeline entry without a timezone")
if section == "action items" and line.startswith("|") and not set(line) <= set("|-: "):
cells = [c.strip() for c in line.strip("|").split("|")]
if cells[0].lower() == "action":
continue
action, owner, due, ticket = (cells + ["", "", "", ""])[:4]
for label, value in (("owner", owner), ("due date", due), ("ticket", ticket)):
if not value or value in ("-", "TBD", "?"):
issues.append(f"line {n}: action item has no {label}: '{action}'")
for pat in WEAK_ACTIONS:
if re.search(pat, action, re.I):
issues.append(f"line {n}: weak action '{action}' (prefer a guardrail or automation)")
return issues
if __name__ == "__main__":
problems = lint(open(sys.argv[1]).read())
print("\n".join(problems) or "clean")
sys.exit(1 if problems else 0)Here it is on a made-up first draft, the kind that tends to come out of a long day:
# Checkout API 503s after a connection-pool change
## Summary
For 47 minutes, 18% of checkout requests failed with 503s after a config
change lowered the database connection pool from 50 to 5.
## Impact
About 3,100 failed checkouts; no data loss.
## Timeline
- 14:02 Config change merged and rolled out to all regions at once.
- 14:09 UTC Error-rate alert fires; on-call acknowledges.
- 14:31 UTC Rollback started after the pool metric is found.
- 14:49 UTC Error rate back to baseline.
## Contributing factors
- The on-call engineer should have checked the pool dashboard first.
- Bob failed to notice the pool size in review.
- The config change skipped canary because config was exempt from it.
## What went well
- The alert fired within seven minutes.
## Action items
| Action | Owner | Due | Ticket |
|---|---|---|---|
| Be more careful reviewing config diffs | team | TBD | - |
| Route config changes through the canary pipeline | priya | 2026-11-14 | PLAT-2210 |
| Alert on pool saturation | | 2026-11-07 | PLAT-2211 |missing section: Where we got lucky
line 11: timeline entry without a timezone
line 17: blame/hindsight wording 'should have'
line 18: blame/hindsight wording 'failed to'
line 27: action item has no due date: 'Be more careful reviewing config diffs'
line 27: action item has no ticket: 'Be more careful reviewing config diffs'
line 27: weak action 'Be more careful reviewing config diffs' (prefer a guardrail or automation)
line 29: action item has no owner: 'Alert on pool saturation'It’s a regex, so it’ll have false positives. “Failed to connect” is a fine sentence about a service. Treat its output as review comments, not a gate. What it does well is catch the habits that make a postmortem read like a blame document before a human reviewer has to say so.
Notice what it can’t catch, too. The third contributing factor, “config was exempt from canary,” is the real finding in that draft, and no linter will tell you that. That part is still your job.
Where to keep them
Keep all postmortems in one searchable place, with consistent titles and a few tags for contributing-factor categories (config change, capacity, dependency, deploy, detection gap). The value compounds. After a year you can search “config change, exempt from canary” and find out it’s the fourth time, which is a much stronger argument for funding the fix than any single incident.
If agents are part of how your systems change, that applies to their incidents too. I wrote about what changes in postmortems when an agent made the change: the structure here still holds, but you need extra sections for the agent’s context at decision time, the identity it acted as, and the gates that didn’t apply to it.
The test of a postmortem isn’t whether it’s well written. It’s whether someone reading it next year can avoid repeating the incident.
Related: Observability and incident response — the SRE basics · Agents are deleting production and nobody is writing the postmortem · Progressive Delivery: Canary, Blue-Green, and the Rollback You Never Tested
Comments