Flaky Test Triage

1. Establish the rate before touching anything

"Sometimes fails" is not a measurement. Run it runs times (default 50) and record the failure rate.

# Adapt to your runner. The point is repetition and a count.
for i in $(seq 1 50); do <run one test> >/dev/null 2>&1 || echo "fail $i"; done | wc -l

Run it three ways, because the differences are the diagnosis:

A test that only fails in one of the three has already told you which family of cause you are in.

2. Capture a failure with enough detail to read

Do not debug from the assertion message alone. On failure, capture:

• The full diff between expected and actual, not a truncated boolean.
• Timestamps at each step, to spot a timing edge.
• The random seed and the test execution order.
• Relevant environment: time zone, locale, machine, concurrency level.

If your runner cannot report the seed and the order, fix that first. Without it, an ordering bug is unreproducible by construction.

3. Classify the cause

Work through causes.md. Match on the tell, not on intuition. The families, roughly in order of frequency:

1. Order dependence and shared state
2. Time and clock assumptions
3. Concurrency and race conditions
4. Unseeded randomness
5. Resource contention
6. External dependencies
7. Genuine product bugs that only surface under a specific interleaving

The last one matters most: a flaky test is sometimes a correct test finding a real race. Rule that out before assuming the test is at fault. Silencing it would be deleting a bug report.

4. Prove the cause

A hypothesis is not confirmed until you can turn the failure on and off:

• Make it fail every time. Force the ordering, pin the clock, hold the lock, use the seed that fails. If you cannot make it fail deterministically, you have not found it.
• Then apply the fix and make it pass every time.

Skipping this step is how a test gets "fixed" three times and stays flaky.

5. Fix the cause, not the symptom

Ranked, best first:

1. Remove the nondeterminism. Inject the clock, seed the randomness, fix the shared state, await the actual condition.
2. Make the test wait for a condition, never a duration. sleep is a race with extra steps.
3. Isolate the resource. A unique port, a temp directory, a per-test schema.
4. Narrow the assertion if it was over-specifying, for example asserting an unordered collection in order.

Never acceptable as a fix: a retry annotation, a longer sleep, or a loosened assertion that would no longer catch the original bug.

6. Quarantine honestly, if you must

Sometimes the fix cannot land today. Then:

• Move the test out of the blocking suite, but keep running it and keep reporting it.
• File an issue with the failure rate you measured and everything you established.
• Put an expiry on it. A quarantine with no date is a deletion with extra steps.
• Never quarantine a test whose failure you have not yet explained. That is where a real bug goes to be forgotten.

7. Verify

Re-run the same three conditions from step 1, at the same repetition count. Report the before and after rates as numbers. "Seems better" is not a result.