Harness evals
Test an AGENTS.md, rule or skill change before you keep it: A/B it, keep safety lines, refine in rounds, check skill triggers and run repo-replay tasks.
The harness is everything around the model: AGENTS.md, rules, skills, the agent and its flags. These commands test a change to it before you keep it, on the sessions you already have or on tasks replayed from your repo.
A/B a harness edit on your sessions (built)
You changed a line in AGENTS.md and want to know if it helps. coding.sh exp ab finds the moments in your recent sessions
where the agent went wrong, such as saying it was done without running anything. It replays the next turn twice, once with your committed
files and once with your edit, and a judge scores both.
The noise floor is how often your committed files alone produce a different turn from the recorded one. A smaller difference may be replay noise. Responses are cached, so a rerun with unchanged files is free.
| Flag | What it does |
|---|---|
--dry-run | Shows the cases and a cost estimate; calls nothing. |
--max-cost | Caps the spend. |
--gate | Makes the verdict line the exit code. |
--watch | Reruns each time you save, one line per run. |
The walkthrough is Test an AGENTS.md change in five minutes.
Keep safety lines from being dropped (built)
A pass-rate A/B can't see the harm of deleting "Never run git push --force" if no case needed it.
coding.sh refine check compares a harness edit by text (default: git HEAD against your working tree) and exits 1
when it deletes, rewords or relaxes a protected line. exp ab runs the same check and warns before any model call.
A line is protected inside a <!-- coding.sh:protect --> block, when it matches a [refine] protect regex, or
when one sentence pairs a word like never or must not with a safety topic such as credentials, deploy or rm -rf.
refine protected lists them, and --allow-protected-edit AGENTS.md:3 allows one edit on purpose
(details).
Improve AGENTS.md in tested rounds (built)
refine test decides each candidate edit in stages: the protected-line check and two critics on the added lines, then an A/B
on the decision points exp ab mines, through the same ship gate. Last comes the guard: what earlier kept edits won must stay
solved.
Here the second edit wins its own decision points but drops the line the first edit added, so the guard rejects it.
| Command | What it does |
|---|---|
refine propose | Asks a model for lines that address your sessions' most severe flags. |
refine apply | Writes a kept candidate with a backup. Never commits. |
refine rollback | Restores the backup byte for byte. |
refine log | Every round with its verdict, effect, guard result, cost and reason. |
refine run | Loops propose, test and apply until --rounds, --max-cost or a plateau. |
See Refine loop in the evals guide.
Check that a skill loads (built)
A skill only helps when the agent loads it. eval trigger <skill-dir> reads a triggers.toml of
[[should]] prompts and [[should_not]] near misses. It runs each prompt k times with the agent (Claude Code by default)
and reads from the captured session whether the skill loaded. --description or --candidate DIR adds a second arm to compare.
The output above is the CI check with real Claude Code against a stub model. Eight prompts can't show a change of this size, so the verdict holds. --min-f1 and --fail-on regression set the
exit code for CI. See Does
the skill load?.
Replay tasks from your repo (partial)
A repo-replay task is the repository from before the agent ran, the first prompt as the instruction, and a verifier command.
compare runs two harness bundles on those tasks in isolated worktrees.
coding.sh eval new latest --verifier 'npm test' --hidden-test 'test/retry.spec.ts'
coding.sh eval validate <task> --k 5
coding.sh eval export <task> --out ./tasks/retry --format harbor
coding.sh harness snapshot --agent claude-code --name base
coding.sh compare base candidate --k 3Before a task counts, eval validate checks the verifier:
- Doing nothing fails.
- The reference solution passes
kruns out ofk(default 5), so flaky tests fail validation. - Near-miss variants of the reference, made by mutating its patch, fail.
- Nothing leaks: hidden tests are copied in only when the verifier runs.
A bundle is the agent, model, flags, env and files such as AGENTS.md, identified by a content hash. The report gives each arm's pass rate, the paired difference, McNemar's test, the cost, token and latency deltas, and the first step where the traces diverge.
| Command | What it adds |
|---|---|
eval crosscheck <task> | A reviewer model judges each run's diff and reports where it disagrees with the verifier. |
eval audit <task> --experiment <id> --witness <file> --witness-cmd <cmd> | A test for a requirement the verifier skips; it can refute passing runs. |
eval import tasks/*/ | Loads hand-written tasks (task.json, instruction.md, repo/, tests/) as drafts. Run them with compare A B --tag <tag> (walkthrough). |
Gate a change in CI
compare A B --gate answers "should I ship B?" in one line: Ship it: B wins 9 tasks, loses 0,
Hold: not enough evidence, or Don't ship: B breaks 3 tasks A solved. It exits 0 only for "Ship it".
A break costs two fixes (--lambda), the wins must clear a one-sided exact test (--gate-alpha, default 0.05), and
B may break at most 10% of the tasks A solved (--tau). This is the SAGE accept-or-abstain rule
(arXiv 2609.36043). Passes that tampered with tests count as fails.
More compare options
| Flag | What it adds |
|---|---|
--gate --by suite: | Also gates each group of tasks; any group's "Don't ship" blocks. |
--promote | Exits 0 only when B is credibly better, or cheaper within --eps-q of A's pass rate. |
--audited | Counts passes a witness refuted as fails. Needs every pass audited. |
--item-gate | Exits 0 on KEEP or ACCEPT from the one-item gate (when B adds or removes one item). |
--with plugin:terse | B is A plus a Claude Code plugin. Your Claude config is never changed. |
--reuse baseline | Copies A's finished runs from earlier comparisons, so the baseline is paid for once. |
--experiment exp_01 --retry-errors | Reruns, in place, the runs that errored or failed on infrastructure. |
--failing exp_01 | Runs only the tasks A failed in that comparison. |
--repair 2 | Tells the agent its verifier failed and lets it try again, up to 2 rounds. |
--max-cost 15 | Stops starting runs once agent and judge spend reach $15. |
What else compare reports
- Passing runs that edited, deleted or skipped tests, each linked to the tool call:
tampered passes B 3: disclosed 2, stealth 1. - Each arm's failed runs by the agent's own last test run. Green fails are silent misses: the agent's tests passed and the verifier's did not.
- A note when the arms ran on different agent releases.
--promotethen holds and exits 1. - After
eval audit: each arm's raw and audited pass rate, as inaudited pass B 60% → 30% (3 refuted, coverage 100%).
The method behind each line, and every flag above, is in Harness A/B and the compare reference.
Known gap. A hook-captured session starts from the commit that was checked out when it began, so uncommitted changes at
that point are missing and eval new warns. coding.sh run -- snapshots the whole working tree before and after.