# Measuring without fooling yourself

## Repetitions and warm-up

- Never trust one run. Take at least 5, ideally 20 or more for short operations.
- Discard warm-up runs explicitly, and say how many. JIT compilation, connection pools
  and caches all make the first run unrepresentative.
- Report the distribution, not the mean alone. p50 and p95 tell different stories, and
  the complaint you received is almost always about p95.

## Realistic conditions

- **Production-shaped data.** An algorithm that is quadratic looks fine on 100 rows.
- **Production-shaped concurrency.** Contention only appears under load.
- **Honest cache state.** Measure both cold and warm, and label which is which.
- **A machine that is not doing other things.** Your laptop compiling in the background
  is not a benchmark environment.

## Traps

- **Measuring the wrong layer.** A fast function inside a slow request means the time is
  elsewhere. Measure end to end first, then narrow.
- **Optimizing a benchmark.** If the benchmark does not resemble the real workload, you
  will make the benchmark faster and the product slower.
- **Dead code elimination.** A compiler may remove work whose result you never use.
  Consume the result.
- **Confusing throughput with latency.** Batching improves one and worsens the other.
  Know which one the complaint was about.
- **Ignoring variance.** A 5% improvement inside 15% run-to-run noise is not an
  improvement, it is a coin flip you liked the result of.

## What to record with every measurement

So it can be reproduced and compared later:

- Command or code path, exactly.
- Input size and shape.
- Machine, cores, memory.
- Runtime and library versions.
- Concurrency level.
- Cache state.
- Repetition count and the discarded warm-up count.
- Raw numbers, not only the summary.