1
# Agent Eval Harness
2
3
Most agent changes ship on vibes: someone tries three prompts by hand, likes the
4
third, and merges. This harness replaces that with a small, honest eval suite. Enough
5
cases to notice a regression, graders that fail loudly when they cannot judge, and a
6
recorded baseline so "it got better" becomes a number you can check.
7
8
## When to use
9
10
- Before changing a prompt, model, or tool set that something depends on.
11
- When a change feels better and you need to know whether it actually is.
12
- When a bug is reported that you want to keep fixed. Every bug becomes a case.
13
14
## Install
15
16
```bash
17
npx openagents-cli add openagents/eval-harness
18
```
19
20
| Runtime | Installed to |
21
|---|---|
22
| `claude-code` | `.claude/skills/eval-harness/` |
23
| `codex` | `.codex/skills/eval-harness/` |
24
| `openai-agents` | `agents/eval-harness/` |
25
| `langgraph` | `graphs/eval-harness/` |
26
| `generic` | `.openagents/eval-harness/` |
27
28
## What is in the package
29
30
- `HARNESS.md` - the procedure: collect cases, write graders, take a baseline, gate on it.
31
- `grading.md` - how to pick a grader per case type, and how to keep an LLM grader honest.
32
- `templates/cases.jsonl` - the case file format, with worked examples.
33
34
## License
35
36
MIT
37