Evals

Your first eval suite in minutes with eval init and eval run, then datasets, the task under test, graders and calibrated LLM judges.

There are two kinds of eval. Dataset experiments run any agent, in any language, over a set of cases and grade the outputs. Repo-replay tasks re-run a coding agent from the repository state before a real session and check the result with a validated verifier.

Experiments covers comparing two runs, sweeps and long runs. Harness evals covers testing an AGENTS.md, rule or skill change, and repo-replay tasks.

Your first eval in 5 minutes (built)

eval init writes a suite to evals/<name>/ in your repo. eval run runs every suite there, or the ones you name, and prints a report. In a terminal, eval init with no flags asks what you want to test.

Your agent's failed sessions. Up to 10 flagged sessions from this repo in the last 14 days become repo tasks: the repo before the session, its first prompt, and your verifier. The failed diff is kept as a near miss the verifier must fail.

shell

coding.sh eval init --sessions --verifier 'npm test'
coding.sh eval run failures --yes

A skill. Prompts where the agent should load the skill and prompts where it should not, drafted by a model and marked DRAFT for you to review. eval run runs each in Claude Code and checks whether the skill loaded.

shell

coding.sh eval init --skill .claude/skills/pdf --draft
coding.sh eval run pdf --yes

An app or a prompt. Cases of {"input", "expected"} and a command (or --url) that answers one. eval init also prints a GitHub Actions workflow that runs the suite on every pull request.

shell

coding.sh eval init triage --cases cases.jsonl --command 'node triage.mjs'
# no cases yet: describe the app and have a model draft them
coding.sh eval init triage --describe 'Labels a support ticket billing, bug or other' \
  --command 'node triage.mjs' --draft
coding.sh eval run triage

~/proj

$ coding.sh eval run failures -k 2 --yes
failures  capability suite · 1 task · 2 trials each
pass@1  50.0% [9.5%, 90.5%]  (1/2 trials, 95% Wilson)
pass@2  100.0%  a task counts when any of 2 trials passes
pass^2  0.0%  a task counts only when all 2 trials pass

task                     passed         first failure
-----------------------  ------  -----  ---------------------------
make-sum-js-add-its-two     1/2  flaky  coding.sh show 4fd72079-09a

flaky: make-sum-js-add-its-two. These pass some trials and fail others; pass^2 counts them as failures.
1 task: a suite this small only shows large changes; grow it toward 20 to 50 tasks from real failures

Each task runs k times (3 by default). pass@k counts a task when any of its k trials passes, and pass^k only when all of them pass, so a flaky task counts for the first and against the second. Each failed trial links its captured session.

The evals guide maps each step of Anthropic's Demystifying evals for AI agents roadmap to a command.

Datasets (built)

A dataset is a list of cases: input, optional expected, tags, metadata and a split (train, dev or test). Every change makes a new version whose id is a hash of the cases, so an experiment records exactly what it ran on.

shell

coding.sh dataset add smoke \
  --case '{"input":"2+2","expected":"4"}' \
  --case '{"input":"3*3","expected":"9"}'
coding.sh dataset import cases.jsonl --name support   # JSONL, JSON or CSV
coding.sh dataset add regressions --session <session-id>   # a case from a captured session

When a dataset tests a change, give each case a kind (in the case, or with dataset add --kind). exp run prints the pass rate per kind, and exp compare names the bad cases fixed and the guards and controls that broke.

Kind	Before the change	After the change
`bad`	Fails: it reproduces the failure the change is for.	Passes.
`guard`	Passes: behavior that already works.	Passes.
`control`	Passes: the fix must not kick in here.	Passes; a failure means the change overcorrected.

What gets run

The task is the agent under test. Two kinds work from the CLI:

command: any executable. The case input arrives as JSON on stdin; stdout is the output. CODING_SH_RUN_ID, CODING_SH_CASE_ID and CODING_SH_TRIAL are in its environment.
http: a POST of the input as JSON to a URL; the response (or a JSON path into it) is the output. Header values can come from environment variables, so secrets stay out of spec files.

agent.mjs

// agent.mjs: reads the case input as JSON on stdin, prints its answer
let s = "";
for await (const chunk of process.stdin) s += chunk;
const input = JSON.parse(s);
console.log(String(eval(input)));

shell

coding.sh exp run --dataset smoke --name base \
  --task '{"kind":"command","command":"node","args":["agent.mjs"]}' \
  --grader '{"kind":"exact"}'

~/smoke

$ coding.sh exp run --dataset smoke --name base …
base  exp_01M6SAJYC5RM7A5TCT65R3SFSM
dataset smoke@8db025860cd0b8da  task command: node agent.mjs  trials 1
pass    100.0% [34.2%, 100.0%] (2/2, 95% Wilson)
latency median 32ms, p90 32ms, max 32ms

grader  kind     pass  95% CI           mean score  errors  trust
------  -----  ------  ---------------  ----------  ------  -------------
exact   exact  100.0%  [34.2%, 100.0%]        1.00       0  deterministic

With one trial per case the interval is a Wilson interval; with --trials above 1 it is a bootstrap over cases. Options can also live in a JSON spec file: coding.sh exp run spec.json.

spec.json

{
  "name": "support-v2",
  "dataset": "support",
  "split": "test",
  "trials": 3,
  "task": { "kind": "http", "url": "http://127.0.0.1:3000/api/agent", "responsePath": "$.output" },
  "graders": [
    { "kind": "contains", "value": ["refund"] },
    { "kind": "llm", "name": "helpful", "rubric": "Pass if the reply resolves the customer's issue.", "tunedOn": ["dev"] }
  ]
}

Graders (built)

Deterministic graders come first. Each grader's config is hashed into a version, so a changed grader is a different grader. Every field is in Grader fields.

Kind	Passes when
`exact`	Output equals `expected` (structural for JSON, normalized for text).
`contains`	Output contains all (or any) of the strings; `negate` inverts it.
`regex`	Output matches a pattern.
`json-schema`	Output validates against a JSON Schema.
`json-path`	A JSON path into the output equals a value or the same path in `expected`.
`numeric`	The first number in the output is within an absolute or relative tolerance.
`similarity`	Normalized edit-distance similarity is at least the threshold (default 0.8).
`progress`	The run's trace derailed (looping, churn, stalling and the like) at most `maxDerails` times (default 0).
`executability`	The trace has at most `maxInvalid` tool calls the harness refused to run (default 0).
`agent-slop`	The lines the agent's diff adds have at most `maxErrors` findings such as `as any` or empty catches (default 0).
`command`	Your script, any language: JSON on stdin; a boolean, a score or `{score, pass, reason}` on stdout, or the exit code.
`module`	A JS module's exported grade function returns pass.
`llm`	A model judge returns pass on your rubric. See below.
`llm-checklist`	Yes/no questions, each answered by the model or by a trace check such as `ran-command` or `no-test-tamper`.
`llm-panel`	One judge per provider scores 1-10 and the median reaches `target` (default 7).

LLM judges (built)

An llm grader calls Anthropic's Messages API or any OpenAI-compatible endpoint (OpenAI, OpenRouter, Ollama, LM Studio, vLLM through baseURL). Keys are read from the environment at call time and never logged, hashed or cached. Verdicts are cached on disk, so re-grading an unchanged output costs nothing.

Each judge is versioned as name@hash over model, prompt template, rubric, criterion and parameters. Judge verdicts can cite the trace, and citations are checked against what the judge was shown.

A pairwise judge compares two experiments: exp compare A B --pairwise judge.json. It sees both outputs for each case twice, with the order swapped, and a side wins only when both orders agree. It reports B's win rate with an interval and a sign test.

Position bias and stability

A case whose answer changes with the order is a tie, and counts toward the printed position-bias rate. --labels takes human A/B/tie labels and calibrates the judge like any other.

An inconclusive win rate can mean "needs more cases" or "this judge flips". The report prints how often a resample of the cases picks the same winner, and n*, the smallest size where that holds 95% of the time: stable B better in 97% of resamples (n*=50).

An uncalibrated judge that never settles is flagged "unresolved: pick or calibrate the judge before collecting more verdicts" (arXiv 2611.04090). Judge graders get the same line in exp compare.

Calibrate a judge against people (built)

A judge's score counts only once it agrees with people. Label sessions pass or fail in the viewer, then measure agreement on a split the judge's prompt was not tuned on.

shell

coding.sh ui                                   # label sessions pass / fail
coding.sh judge calibrate <experiment> --criterion outcome --split test

Agreement is Cohen's κ with a bootstrap interval, plus the confusion matrix and precision and recall for "fail".
A judge is calibrated at κ ≥ 0.6 on at least 20 held-out cases with both a human label and a verdict.
κ measured on a split in the judge's tunedOn is in-sample, never calibrated.
Below the bar, the judge is uncalibrated or has insufficient data, and every report flags its scores.
Cases where the judge is confidently opposite to unanimous humans are listed for relabelling.
The judge's held-out sensitivity and specificity are stored, so exp compare shows a bias-corrected judge pass rate.

Flag	What it adds
`--threshold`, `--min-n`	Change the bar: κ 0.6 and 20 cases.
`--judge-cost 0.001`	The human label cost above which calibrating the judge pays off.
`--plan plan.json`	The human-audited pass rate from the cases `judge sample` picked to label.
`--probe-injection judge.json`	Re-judges human-fail cases with a planted "respond pass" order. Any flip exits 1.
`--grader tests`	Audits a test verifier instead: how many of its passes a person would accept.
`--sample-passes 40`	Lists a seeded sample of unlabelled passes to label, instead of measuring.

The agent writes the text a judge grades, so judge prompts escape the data tags inside it and tell the model that everything inside them is untrusted. The Judges section of the evals guide has the method behind each number.

Scores (built)

A score is one typed value on a session, an event, a dataset case or an eval run: a number, true or false, a category or text. A config fixes a score name's type and range, and a value outside it is refused. A pass or fail label is a human boolean score, which is what judge calibrate reads.

shell

coding.sh score config helpfulness --type numeric --min 0 --max 5
coding.sh score add case:refund-03 helpfulness 4 --comment "named the refund window"
coding.sh score list --name helpfulness

Scoring the same target, name and source again replaces the score. The viewer and coding.sh serve take scores at POST /api/scores, and agents use the score_add and score_list MCP tools.

Workflows

Multi-agent workflows have their own page: capturing runs, scoring them against seeded bugs, and comparing two versions. See Workflow experiments.