1
# Blameless Postmortem
2
3
## 0. Wait until the incident is over
4
5
A postmortem written during an incident is a status update. Wait until the system is
6
stable, and hold it within about a week, while memory is fresh and interest survives.
7
8
## 1. Build the timeline from evidence
9
10
Not from memory. Memory reorders events to fit the explanation people arrived at.
11
12
Sources, in order of reliability: monitoring and logs with timestamps, deploy and
13
change records, the incident channel with its own timestamps, then interviews.
14
15
Record for each entry: time in UTC, what happened, and how you know. Include:
16
17
- When the fault was introduced, which is often long before the incident.
18
- When it began affecting users. This is when the incident started, not when it was noticed.
19
- When it was detected, and by what. A customer report means detection failed.
20
- Each mitigation attempt, including the ones that did not work.
21
- When impact ended, and when it was confirmed ended.
22
23
The gaps between introduced, started, detected and resolved are the most useful numbers
24
in the document. Each one is a different problem with a different fix.
25
26
## 2. Establish impact in numbers
27
28
Who, how many, how long, how much. "Some users saw errors" is not impact. "About 3% of
29
checkout requests failed for 47 minutes, roughly 1,200 orders" is impact, and it is what
30
justifies the effort of the fixes.
31
32
Include what you cannot measure, and say why you cannot. That gap is often itself an
33
action item.
34
35
## 3. Find contributing factors, not a root cause
36
37
Almost no real incident has one cause. It has a chain, and every link is a place it
38
could have been stopped.
39
40
For each step in the chain ask: what made this possible, and what would have caught it?
41
42
Look in four places:
43
44
- **Technical.** The bug, the missing limit, the untested path.
45
- **Detection.** Why did monitoring not catch it, or catch it sooner?
46
- **Response.** What slowed diagnosis? A missing runbook, an unclear owner, a dashboard
47
  nobody could find.
48
- **Systemic.** What made this class of mistake easy to make? A confusing interface, a
49
  deploy process with no staging, review that cannot catch this kind of error.
50
51
## 4. Stay blameless, which is a technique and not a courtesy
52
53
Whenever a sentence names a person, rewrite it to name the condition that let the action
54
have that consequence.
55
56
- Not "Sam deployed without running migrations."
57
- But "the deploy pipeline allowed a deploy to proceed with pending migrations."
58
59
The second is fixable. The first is only embarrassing. **Assume everyone acted
60
reasonably given what they knew at the time**, then ask why the wrong action looked
61
right. That question always has a systemic answer.
62
63
Also resist counterfactuals. "If only they had checked the dashboard" is not a finding.
64
Nobody knew to check it. Ask why the dashboard did not come to them.
65
66
## 5. Write actions that will actually be done
67
68
Every action needs an owner who is a named person, a date, and a definition of done that
69
someone else could verify.
70
71
Prioritize by what breaks the chain closest to the start. In order:
72
73
1. Make the failure impossible.
74
2. Make it detected automatically, and sooner.
75
3. Make it faster to diagnose.
76
4. Make it faster to mitigate.
77
78
Cap the list at what will genuinely be completed. Five actions that ship beat fifteen
79
that decorate a document. Delete anything you are not honestly going to do, rather than
80
leaving it there to be quietly abandoned.
81
82
An action that is only "add more monitoring" or "be more careful" is not an action.
83
84
## 6. Circulate and follow up
85
86
Publish where people outside the incident can read it. Review the actions on a schedule.
87
An unreviewed action list is how the same incident happens twice, and the second
88
postmortem has to explain why the first one changed nothing.
89