Agent Eval Harness
Builds an eval suite that can actually catch a regression. The output is a case file, a grader per case type, a recorded baseline, and a gate that fails a change which makes things worse.
0. Decide what better means, before looking at any output
Write down the one or two things this agent must get right, in a sentence each. If you cannot state them, you cannot grade them, and the eval will drift into measuring whatever is easy to measure.
Bad: "responses are high quality."
Good: "the cited file path exists in the repo" and "the fix compiles."
1. Collect cases from reality, not imagination
Aim for n_cases (default 30). A suite of 30 real cases beats 300 synthetic ones.
In priority order:
- Bugs. Every reported failure becomes a case, with the expected behavior as the report described it. These are the highest-value cases you will ever have.
- Production traffic. Sample real inputs. Stratify. Take some from the head and some from the tail, not 30 of the same shape.
- Known-hard cases. Inputs already near the edge of capability.
- Synthetic cases, last. Only to cover a branch the first three missed.
Record each case in the templates/cases.jsonl format. Every case needs an id, an
input, and enough of an expected to grade against.
2. Split the suite
- dev (about 60%): you look at these, iterate against them, and overfit them. Fine.
- held-out (about 40%): you do not look at individual outputs, only the aggregate.
Report both numbers, always. A dev score that climbs while held-out stays flat means you tuned to the cases, not the task.
3. Write a grader per case type
See grading.md. The rule that matters: a grader that cannot judge a case must return
unknown, never pass. Silent passes are how eval suites rot into decoration. Track
unknown as its own bucket and drive it toward zero.
4. Take a baseline before changing anything
Run the current version against the full suite and record:
- pass, fail and unknown counts, for dev and held-out separately
- the score per case type, not just the total
- cost and p50/p95 latency per case
- the run date, model id, and prompt version
Commit this. A baseline you cannot reproduce is a rumor.
5. Gate changes on it
A change ships when held-out pass rate does not drop and no case type regresses. Two rules make this survive contact with reality:
- Investigate every newly failing case individually. A steady aggregate can hide two cases breaking while two unrelated ones start passing.
- A flaky case is a bug in the case or the grader. Run the suite twice against an unchanged target. Anything that flips is not measuring the target. Fix it or drop it.
6. Keep it alive
- Add the case first when a bug is reported, and watch it fail. Then fix.
- Re-baseline on a model or major prompt change, and note why in the commit.
- Prune cases that have never once failed and never will.
Reporting
State the score honestly:
held-out 24/30 pass, 4 fail, 2 unknown (baseline: 22/30, 6 fail, 2 unknown). Regression on tool-choice cases: 3/5 down to 2/5.
Never report a single percentage with no denominator, no held-out split, and no unknown count. That number is unfalsifiable and therefore useless.