1
# Measuring without fooling yourself
2
3
## Repetitions and warm-up
4
5
- Never trust one run. Take at least 5, ideally 20 or more for short operations.
6
- Discard warm-up runs explicitly, and say how many. JIT compilation, connection pools
7
  and caches all make the first run unrepresentative.
8
- Report the distribution, not the mean alone. p50 and p95 tell different stories, and
9
  the complaint you received is almost always about p95.
10
11
## Realistic conditions
12
13
- **Production-shaped data.** An algorithm that is quadratic looks fine on 100 rows.
14
- **Production-shaped concurrency.** Contention only appears under load.
15
- **Honest cache state.** Measure both cold and warm, and label which is which.
16
- **A machine that is not doing other things.** Your laptop compiling in the background
17
  is not a benchmark environment.
18
19
## Traps
20
21
- **Measuring the wrong layer.** A fast function inside a slow request means the time is
22
  elsewhere. Measure end to end first, then narrow.
23
- **Optimizing a benchmark.** If the benchmark does not resemble the real workload, you
24
  will make the benchmark faster and the product slower.
25
- **Dead code elimination.** A compiler may remove work whose result you never use.
26
  Consume the result.
27
- **Confusing throughput with latency.** Batching improves one and worsens the other.
28
  Know which one the complaint was about.
29
- **Ignoring variance.** A 5% improvement inside 15% run-to-run noise is not an
30
  improvement, it is a coin flip you liked the result of.
31
32
## What to record with every measurement
33
34
So it can be reproduced and compared later:
35
36
- Command or code path, exactly.
37
- Input size and shape.
38
- Machine, cores, memory.
39
- Runtime and library versions.
40
- Concurrency level.
41
- Cache state.
42
- Repetition count and the discarded warm-up count.
43
- Raw numbers, not only the summary.
44