--- title: "Harness evals · coding.sh docs" description: "Test an AGENTS.md, rule or skill change before you keep it: A/B it, keep safety lines, refine in rounds, check skill triggers and run repo-replay tasks." canonical: https://coding.sh/docs/harness-evals human_url: https://coding.sh/docs/harness-evals markdown_url: https://coding.sh/docs/harness-evals.md agents_url: https://coding.sh/agents.md llms_url: https://coding.sh/llms.txt updated: 2026-11-11 --- # Harness evals Test an AGENTS.md, rule or skill change before you keep it: A/B it, keep safety lines, refine in rounds, check skill triggers and run repo-replay tasks. The harness is everything around the model: AGENTS.md, rules, skills, the agent and its flags. These commands test a change to it before you keep it, on the sessions you already have or on tasks replayed from your repo. ## A/B a harness edit on your sessions (built) You changed a line in AGENTS.md and want to know if it helps. `coding.sh exp ab` finds the moments in your recent sessions where the agent went wrong, such as saying it was done without running anything. It replays the next turn twice, once with your committed files and once with your edit, and a judge scores both. ~/repo $ coding.sh exp ab A: git:HEAD B: your edits model: claude-sonnet-5-5 3 decision points from 3 sessions | A 0/3 -> B 3/3 | fixed 3, broke 0 | paired p=0.25 | coding.sh show 8f0e1e52-bdc1-4da0-a03b-14edf6dfcf77 unverified-claim "Make clamp.js clamp to the upper bound too." coding.sh show 9c1792df-6267-4b30-9f24-7a7307740543 unverified-claim "Make range.js include its end value." coding.sh show fe48a450-8867-48ec-b946-3b4d0cb6c780 Full comparison: coding.sh exp compare exp_01M6VXZSFPM2QK12ANN6Z269WP exp_01M6VXZSFSZ25HD2HXXVY40BXV The noise floor is how often your committed files alone produce a different turn from the recorded one. A smaller difference may be replay noise. Responses are cached, so a rerun with unchanged files is free. | Flag | What it does | | --- | --- | | `--dry-run` | Shows the cases and a cost estimate; calls nothing. | | `--max-cost` | Caps the spend. | | `--gate` | Makes the verdict line the exit code. | | `--watch` | Reruns each time you save, one line per run. | The walkthrough is [Test an AGENTS.md change in five minutes](https://coding.sh/docs/how-to/test-agents-md-change). ## Keep safety lines from being dropped (built) A pass-rate A/B can't see the harm of deleting "Never run `git push --force`" if no case needed it. `coding.sh refine check` compares a harness edit by text (default: git `HEAD` against your working tree) and exits 1 when it deletes, rewords or relaxes a protected line. `exp ab` runs the same check and warns before any model call. A line is protected inside a `` block, when it matches a `[refine] protect` regex, or when one sentence pairs a word like never or must not with a safety topic such as credentials, deploy or `rm -rf`. `refine protected` lists them, and `--allow-protected-edit AGENTS.md:3` allows one edit on purpose ([details](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#protected-safety-lines)). ## Improve AGENTS.md in tested rounds (built) `refine test` decides each candidate edit in stages: the protected-line check and two critics on the added lines, then an A/B on the decision points `exp ab` mines, through the same ship gate. Last comes the guard: what earlier kept edits won must stay solved. ~/repo $ coding.sh refine test ./drafts/AGENTS.md --behavior unchanged-retry r_3b577b3897 ./drafts/AGENTS.md (AGENTS.md; +1 line) protected ok (1 protected line in place) critics ok dev 8 decision points | effect +100pp [+100pp, +100pp] | B wins 8, loses 0; A solved 0 guard 8 tasks earlier kept edits won (0 from this run, 8 rerun) | forgets kept edit r_e67f549cae preserved gained 8, lost 0, 0 tasks regressed verdict reject: dev: Ship it: B wins 8 decision points, loses 0; guard: forgets kept edit r_e67f549cae $ coding.sh refine log ROUND CANDIDATE VERDICT EFFECT (95% CI) GUARD COST REASON 1 r_e67f549cae keep, applied +100pp [+100pp, +100pp] no tasks $0.02 dev: Ship it: B wins 8 decision points, loses 0 2 r_3b577b3897 reject +100pp [+100pp, +100pp] forgets r_e67f549cae $0.02 guard: forgets kept edit r_e67f549cae Here the second edit wins its own decision points but drops the line the first edit added, so the guard rejects it. | Command | What it does | | --- | --- | | `refine propose` | Asks a model for lines that address your sessions' most severe flags. | | `refine apply` | Writes a kept candidate with a backup. Never commits. | | `refine rollback` | Restores the backup byte for byte. | | `refine log` | Every round with its verdict, effect, guard result, cost and reason. | | `refine run` | Loops propose, test and apply until `--rounds`, `--max-cost` or a plateau. | See [Refine loop](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#refine-loop) in the evals guide. ## Check that a skill loads (built) A skill only helps when the agent loads it. `eval trigger ` reads a `triggers.toml` of `[[should]]` prompts and `[[should_not]]` near misses. It runs each prompt k times with the agent (Claude Code by default) and reads from the captured session whether the skill loaded. `--description` or `--candidate DIR` adds a second arm to compare. ~/repo $ coding.sh eval trigger skills/pdf --description "Use when the user mentions pdf, invoice, scanned." --yes A (current): pdf on claude-code, k=1 precision 67% recall 50% F1 0.57 accuracy 63% missed: should loaded 0/1 "Make the invoice searchable" should_not false trigger 1/1 "Explain in one line what a pdf is" expected routes: docx 1/1 B (--description): pdf on claude-code, k=1 precision 80% recall 100% F1 0.89 accuracy 88% B vs A over 8 prompts: recall 2/4→4/4 · false triggers 1/4→1/4 · Δacc +25 pp (95% CI +0 to +63) flip better should skipped → loaded "Make the invoice searchable" flip better should skipped → loaded "OCR this scanned invoice" note: 4 should and 4 should_not prompts: only a large change can show; aim for 10+ per side No clear difference The output above is the CI check with real Claude Code against a stub model. Eight prompts can't show a change of this size, so the verdict holds. `--min-f1` and `--fail-on regression` set the exit code for CI. See [Does the skill load?](https://github.com/jackulau/coding.sh/blob/main/docs/how-to/evaluate-a-skill.md#does-the-skill-load). ## Replay tasks from your repo (partial) A repo-replay task is the repository from before the agent ran, the first prompt as the instruction, and a verifier command. `compare` runs two harness bundles on those tasks in isolated worktrees. shell ``` coding.sh eval new latest --verifier 'npm test' --hidden-test 'test/retry.spec.ts' coding.sh eval validate --k 5 coding.sh eval export --out ./tasks/retry --format harbor coding.sh harness snapshot --agent claude-code --name base coding.sh compare base candidate --k 3 ``` Before a task counts, `eval validate` checks the verifier: - Doing nothing fails. - The reference solution passes `k` runs out of `k` (default 5), so flaky tests fail validation. - Near-miss variants of the reference, made by mutating its patch, fail. - Nothing leaks: hidden tests are copied in only when the verifier runs. A bundle is the agent, model, flags, env and files such as AGENTS.md, identified by a content hash. The report gives each arm's pass rate, the paired difference, McNemar's test, the cost, token and latency deltas, and the first step where the traces diverge. | Command | What it adds | | --- | --- | | `eval crosscheck ` | A reviewer model judges each run's diff and reports where it disagrees with the verifier. | | `eval audit --experiment --witness --witness-cmd ` | A test for a requirement the verifier skips; it can refute passing runs. | | `eval import tasks/*/` | Loads hand-written tasks (`task.json`, `instruction.md`, `repo/`, `tests/`) as drafts. Run them with `compare A B --tag ` ([walkthrough](https://coding.sh/docs/how-to/evaluate-a-skill)). | ## Gate a change in CI `compare A B --gate` answers "should I ship B?" in one line: `Ship it: B wins 9 tasks, loses 0`, `Hold: not enough evidence`, or `Don't ship: B breaks 3 tasks A solved`. It exits 0 only for "Ship it". A break costs two fixes (`--lambda`), the wins must clear a one-sided exact test (`--gate-alpha`, default 0.05), and B may break at most 10% of the tasks A solved (`--tau`). This is the SAGE accept-or-abstain rule ([arXiv 2609.36043](https://arxiv.org/abs/2609.36043)). Passes that tampered with tests count as fails. ## More compare options | Flag | What it adds | | --- | --- | | `--gate --by suite:` | Also gates each group of tasks; any group's "Don't ship" blocks. | | `--promote` | Exits 0 only when B is credibly better, or cheaper within `--eps-q` of A's pass rate. | | `--audited` | Counts passes a witness refuted as fails. Needs every pass audited. | | `--item-gate` | Exits 0 on KEEP or ACCEPT from the one-item gate (when B adds or removes one item). | | `--with plugin:terse` | B is A plus a Claude Code plugin. Your Claude config is never changed. | | `--reuse baseline` | Copies A's finished runs from earlier comparisons, so the baseline is paid for once. | | `--experiment exp_01 --retry-errors` | Reruns, in place, the runs that errored or failed on infrastructure. | | `--failing exp_01` | Runs only the tasks A failed in that comparison. | | `--repair 2` | Tells the agent its verifier failed and lets it try again, up to 2 rounds. | | `--max-cost 15` | Stops starting runs once agent and judge spend reach $15. | What else compare reports - Passing runs that edited, deleted or skipped tests, each linked to the tool call: `tampered passes B 3: disclosed 2, stealth 1`. - Each arm's failed runs by the agent's own last test run. Green fails are silent misses: the agent's tests passed and the verifier's did not. - A note when the arms ran on different agent releases. `--promote` then holds and exits 1. - After `eval audit`: each arm's raw and audited pass rate, as in `audited pass B 60% → 30% (3 refuted, coverage 100%)`. The method behind each line, and every flag above, is in [Harness A/B](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#harness-ab) and the [compare reference](https://coding.sh/docs/cli/compare). Known gap. A hook-captured session starts from the commit that was checked out when it began, so uncommitted changes at that point are missing and `eval new` warns. `coding.sh run --` snapshots the whole working tree before and after. To set coding.sh up for a user, follow the agent workflow at [/agents.md](https://coding.sh/agents.md).