--- title: "Evals for AI agents and coding agents · coding.sh docs" description: "Your first eval suite in minutes with eval init and eval run, then datasets, the task under test, graders and calibrated LLM judges." canonical: https://coding.sh/docs/evals human_url: https://coding.sh/docs/evals markdown_url: https://coding.sh/docs/evals.md agents_url: https://coding.sh/agents.md llms_url: https://coding.sh/llms.txt updated: 2026-11-11 --- # Evals Your first eval suite in minutes with eval init and eval run, then datasets, the task under test, graders and calibrated LLM judges. There are two kinds of eval. Dataset experiments run any agent, in any language, over a set of cases and grade the outputs. Repo-replay tasks re-run a coding agent from the repository state before a real session and check the result with a validated verifier. [Experiments](https://coding.sh/docs/experiments) covers comparing two runs, sweeps and long runs. [Harness evals](https://coding.sh/docs/harness-evals) covers testing an AGENTS.md, rule or skill change, and repo-replay tasks. ## Your first eval in 5 minutes (built) `eval init` writes a suite to `evals//` in your repo. `eval run` runs every suite there, or the ones you name, and prints a report. In a terminal, `eval init` with no flags asks what you want to test. Your agent's failed sessions. Up to 10 flagged sessions from this repo in the last 14 days become repo tasks: the repo before the session, its first prompt, and your verifier. The failed diff is kept as a near miss the verifier must fail. shell ``` coding.sh eval init --sessions --verifier 'npm test' coding.sh eval run failures --yes ``` A skill. Prompts where the agent should load the skill and prompts where it should not, drafted by a model and marked DRAFT for you to review. `eval run` runs each in Claude Code and checks whether the skill loaded. shell ``` coding.sh eval init --skill .claude/skills/pdf --draft coding.sh eval run pdf --yes ``` An app or a prompt. Cases of `{"input", "expected"}` and a command (or `--url`) that answers one. `eval init` also prints a GitHub Actions workflow that runs the suite on every pull request. shell ``` coding.sh eval init triage --cases cases.jsonl --command 'node triage.mjs' # no cases yet: describe the app and have a model draft them coding.sh eval init triage --describe 'Labels a support ticket billing, bug or other' \ --command 'node triage.mjs' --draft coding.sh eval run triage ``` ~/proj ``` $ coding.sh eval run failures -k 2 --yes failures capability suite · 1 task · 2 trials each pass@1 50.0% [9.5%, 90.5%] (1/2 trials, 95% Wilson) pass@2 100.0% a task counts when any of 2 trials passes pass^2 0.0% a task counts only when all 2 trials pass task passed first failure ----------------------- ------ ----- --------------------------- make-sum-js-add-its-two 1/2 flaky coding.sh show 4fd72079-09a flaky: make-sum-js-add-its-two. These pass some trials and fail others; pass^2 counts them as failures. 1 task: a suite this small only shows large changes; grow it toward 20 to 50 tasks from real failures ``` Each task runs k times (3 by default). pass@k counts a task when any of its k trials passes, and pass^k only when all of them pass, so a flaky task counts for the first and against the second. Each failed trial links its captured session. The [evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#your-first-eval-in-5-minutes) maps each step of Anthropic's [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) roadmap to a command. ## Datasets (built) A dataset is a list of cases: `input`, optional `expected`, `tags`, `metadata` and a `split` (`train`, `dev` or `test`). Every change makes a new version whose id is a hash of the cases, so an experiment records exactly what it ran on. shell ``` coding.sh dataset add smoke \ --case '{"input":"2+2","expected":"4"}' \ --case '{"input":"3*3","expected":"9"}' coding.sh dataset import cases.jsonl --name support # JSONL, JSON or CSV coding.sh dataset add regressions --session # a case from a captured session ``` When a dataset tests a change, give each case a `kind` (in the case, or with `dataset add --kind`). `exp run` prints the pass rate per kind, and [`exp compare`](https://coding.sh/docs/experiments#compare) names the bad cases fixed and the guards and controls that broke. | Kind | Before the change | After the change | | --- | --- | --- | | `bad` | Fails: it reproduces the failure the change is for. | Passes. | | `guard` | Passes: behavior that already works. | Passes. | | `control` | Passes: the fix must not kick in here. | Passes; a failure means the change overcorrected. | ## What gets run The task is the agent under test. Two kinds work from the CLI: - `command`: any executable. The case input arrives as JSON on stdin; stdout is the output. `CODING_SH_RUN_ID`, `CODING_SH_CASE_ID` and `CODING_SH_TRIAL` are in its environment. - `http`: a POST of the input as JSON to a URL; the response (or a JSON path into it) is the output. Header values can come from environment variables, so secrets stay out of spec files. agent.mjs ``` // agent.mjs: reads the case input as JSON on stdin, prints its answer let s = ""; for await (const chunk of process.stdin) s += chunk; const input = JSON.parse(s); console.log(String(eval(input))); ``` shell ``` coding.sh exp run --dataset smoke --name base \ --task '{"kind":"command","command":"node","args":["agent.mjs"]}' \ --grader '{"kind":"exact"}' ``` ~/smoke ``` $ coding.sh exp run --dataset smoke --name base … base exp_01M6SAJYC5RM7A5TCT65R3SFSM dataset smoke@8db025860cd0b8da task command: node agent.mjs trials 1 pass 100.0% [34.2%, 100.0%] (2/2, 95% Wilson) latency median 32ms, p90 32ms, max 32ms grader kind pass 95% CI mean score errors trust ------ ----- ------ --------------- ---------- ------ ------------- exact exact 100.0% [34.2%, 100.0%] 1.00 0 deterministic ``` With one trial per case the interval is a Wilson interval; with `--trials` above 1 it is a bootstrap over cases. Options can also live in a JSON spec file: `coding.sh exp run spec.json`. spec.json ``` { "name": "support-v2", "dataset": "support", "split": "test", "trials": 3, "task": { "kind": "http", "url": "http://127.0.0.1:3000/api/agent", "responsePath": "$.output" }, "graders": [ { "kind": "contains", "value": ["refund"] }, { "kind": "llm", "name": "helpful", "rubric": "Pass if the reply resolves the customer's issue.", "tunedOn": ["dev"] } ] } ``` ## Graders (built) Deterministic graders come first. Each grader's config is hashed into a version, so a changed grader is a different grader. Every field is in [Grader fields](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#grader-fields). | Kind | Passes when | | --- | --- | | `exact` | Output equals `expected` (structural for JSON, normalized for text). | | `contains` | Output contains all (or any) of the strings; `negate` inverts it. | | `regex` | Output matches a pattern. | | `json-schema` | Output validates against a JSON Schema. | | `json-path` | A JSON path into the output equals a value or the same path in `expected`. | | `numeric` | The first number in the output is within an absolute or relative tolerance. | | `similarity` | Normalized edit-distance similarity is at least the threshold (default 0.8). | | `progress` | The run's trace derailed (looping, churn, stalling and the like) at most `maxDerails` times (default 0). | | `executability` | The trace has at most `maxInvalid` tool calls the harness refused to run (default 0). | | `agent-slop` | The lines the agent's diff adds have at most `maxErrors` findings such as `as any` or empty catches (default 0). | | `command` | Your script, any language: JSON on stdin; a boolean, a score or `{score, pass, reason}` on stdout, or the exit code. | | `module` | A JS module's exported grade function returns pass. | | `llm` | A model judge returns pass on your rubric. See below. | | `llm-checklist` | Yes/no questions, each answered by the model or by a trace check such as `ran-command` or `no-test-tamper`. | | `llm-panel` | One judge per provider scores 1-10 and the median reaches `target` (default 7). | ## LLM judges (built) An `llm` grader calls Anthropic's Messages API or any OpenAI-compatible endpoint (OpenAI, OpenRouter, Ollama, LM Studio, vLLM through `baseURL`). Keys are read from the environment at call time and never logged, hashed or cached. Verdicts are cached on disk, so re-grading an unchanged output costs nothing. Each judge is versioned as `name@hash` over model, prompt template, rubric, criterion and parameters. Judge verdicts can cite the trace, and citations are checked against what the judge was shown. A `pairwise` judge compares two experiments: `exp compare A B --pairwise judge.json`. It sees both outputs for each case twice, with the order swapped, and a side wins only when both orders agree. It reports B's win rate with an interval and a sign test. Position bias and stability A case whose answer changes with the order is a tie, and counts toward the printed position-bias rate. `--labels` takes human A/B/tie labels and calibrates the judge like any other. An inconclusive win rate can mean "needs more cases" or "this judge flips". The report prints how often a resample of the cases picks the same winner, and n*, the smallest size where that holds 95% of the time: `stable B better in 97% of resamples (n*=50)`. An uncalibrated judge that never settles is flagged "unresolved: pick or calibrate the judge before collecting more verdicts" (arXiv 2611.04090). Judge graders get the same line in `exp compare`. ## Calibrate a judge against people (built) A judge's score counts only once it agrees with people. Label sessions pass or fail in the viewer, then measure agreement on a split the judge's prompt was not tuned on. shell ``` coding.sh ui # label sessions pass / fail coding.sh judge calibrate --criterion outcome --split test ``` - Agreement is Cohen's κ with a bootstrap interval, plus the confusion matrix and precision and recall for "fail". - A judge is calibrated at κ ≥ 0.6 on at least 20 held-out cases with both a human label and a verdict. - κ measured on a split in the judge's `tunedOn` is in-sample, never calibrated. - Below the bar, the judge is uncalibrated or has insufficient data, and every report flags its scores. - Cases where the judge is confidently opposite to unanimous humans are listed for relabelling. - The judge's held-out sensitivity and specificity are stored, so `exp compare` shows a bias-corrected judge pass rate. | Flag | What it adds | | --- | --- | | `--threshold`, `--min-n` | Change the bar: κ 0.6 and 20 cases. | | `--judge-cost 0.001` | The human label cost above which calibrating the judge pays off. | | `--plan plan.json` | The human-audited pass rate from the cases [`judge sample`](https://coding.sh/docs/cli/judge#cmd-judge-sample) picked to label. | | `--probe-injection judge.json` | Re-judges human-fail cases with a planted "respond pass" order. Any flip exits 1. | | `--grader tests` | Audits a test verifier instead: how many of its passes a person would accept. | | `--sample-passes 40` | Lists a seeded sample of unlabelled passes to label, instead of measuring. | The agent writes the text a judge grades, so judge prompts escape the data tags inside it and tell the model that everything inside them is untrusted. The [Judges](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#judges) section of the evals guide has the method behind each number. ## Scores (built) A score is one typed value on a session, an event, a dataset case or an eval run: a number, true or false, a category or text. A config fixes a score name's type and range, and a value outside it is refused. A pass or fail label is a human boolean score, which is what `judge calibrate` reads. shell ``` coding.sh score config helpfulness --type numeric --min 0 --max 5 coding.sh score add case:refund-03 helpfulness 4 --comment "named the refund window" coding.sh score list --name helpfulness ``` Scoring the same target, name and source again replaces the score. The viewer and `coding.sh serve` take scores at `POST /api/scores`, and agents use the `score_add` and `score_list` MCP tools. ## Workflows Multi-agent workflows have their own page: capturing runs, scoring them against seeded bugs, and comparing two versions. See [Workflow experiments](https://coding.sh/docs/workflows). To set coding.sh up for a user, follow the agent workflow at [/agents.md](https://coding.sh/agents.md).