1
# Log Triage
2
3
## 0. Write down the symptom before reading anything
4
5
One sentence, in observable terms: what happened, to whom, starting when. "Checkout
6
returned 500 for about 3% of requests starting 14:20 UTC." Not "the database broke",
7
which is already a hypothesis.
8
9
If you skip this, every log line will look like evidence for whatever you read first.
10
11
## 1. Establish the timeline
12
13
Before interpreting anything, get the sequence:
14
15
- **First occurrence.** Not the first one you noticed. Search backwards until you find
16
  a window with none.
17
- **Rate over time.** Constant, growing, spiky, or a step change? A step change points
18
  at a deploy or a config change. A slow ramp points at a leak or a filling queue.
19
- **What else happened at that moment.** Deploys, config changes, feature flags, traffic
20
  shifts, upstream incidents, certificate expiries, cron jobs, month boundaries.
21
22
The first occurrence and what coincided with it usually contain the answer.
23
24
## 2. Separate signal from background
25
26
Most logs are always noisy. Get a baseline from a healthy window of the same length,
27
ideally the same hour on a previous day.
28
29
- Errors present in both windows are background. Set them aside, do not explain them.
30
- Errors only in the bad window are signal.
31
- Errors whose *rate* changed are signal even if present in both.
32
33
This one step removes most wrong turns. The scary-looking warning that has fired every
34
minute for a year is not your incident.
35
36
## 3. Group before you read
37
38
Do not read chronologically. Normalize away request ids, timestamps and other varying
39
parts, then count by shape. Twenty thousand lines usually collapse into six distinct
40
errors, and the counts tell you which matters.
41
42
Then, for each group: first seen, last seen, count, and whether it is background.
43
44
## 4. Follow one request end to end
45
46
Pick a single failing request and trace it across every service by its correlation id.
47
One complete trace beats a thousand fragments. You are looking for the first thing that
48
went wrong, not the loudest.
49
50
Note where the error is *first* raised, and what the caller did with it. Errors get
51
re-wrapped and re-logged at every layer, so the same failure appears five times with
52
five different messages, and the last one is usually the least informative.
53
54
## 5. Form hypotheses, ranked
55
56
At least three. Fewer than three means you anchored.
57
58
For each: what it claims, the evidence for it, the evidence against it, and the
59
cheapest test that would distinguish it from the others.
60
61
Rank by evidence, not by how satisfying the story is. Note explicitly when a hypothesis
62
explains the timing but not the shape, or the shape but not the timing. A hypothesis
63
that explains only some of the facts is not yet the answer.
64
65
## 6. Test the cheapest discriminating test first
66
67
Cheapest, not most likely. A test that takes 30 seconds and rules out two hypotheses
68
beats one that takes an hour and confirms one.
69
70
Write down what you expect to see before running it. If the result is what you
71
expected, that is weak evidence. If it is not, that is strong evidence, and update.
72
73
## Mistakes this exists to prevent
74
75
- **Anchoring on the first stack trace.** It is usually a symptom several layers from
76
  the cause.
77
- **Explaining background noise.** See step 2.
78
- **Confusing correlation with the deploy.** A deploy at 14:19 and errors at 14:20 is
79
  strong evidence, not proof. Check whether the deploy touched anything on the path.
80
- **Stopping at the first plausible cause.** Plausible is not confirmed. Run the test.
81
- **Reporting a guess as a finding.** Say which it is.
82