# Agent Eval Harness

Most agent changes ship on vibes: someone tries three prompts by hand, likes the
third, and merges. This harness replaces that with a small, honest eval suite. Enough
cases to notice a regression, graders that fail loudly when they cannot judge, and a
recorded baseline so "it got better" becomes a number you can check.

## When to use

- Before changing a prompt, model, or tool set that something depends on.
- When a change feels better and you need to know whether it actually is.
- When a bug is reported that you want to keep fixed. Every bug becomes a case.

## Install

```bash
npx openagents-cli add openagents/eval-harness
```

| Runtime | Installed to |
|---|---|
| `claude-code` | `.claude/skills/eval-harness/` |
| `codex` | `.codex/skills/eval-harness/` |
| `openai-agents` | `agents/eval-harness/` |
| `langgraph` | `graphs/eval-harness/` |
| `generic` | `.openagents/eval-harness/` |

## What is in the package

- `HARNESS.md` - the procedure: collect cases, write graders, take a baseline, gate on it.
- `grading.md` - how to pick a grader per case type, and how to keep an LLM grader honest.
- `templates/cases.jsonl` - the case file format, with worked examples.

## License

MIT