coding.sh eval

coding.sh eval <action>

  Build eval suites from sessions, skills or cases, and run them

Commands
  init          Scaffold an eval suite in evals/<name>/: from failed sessions, a skill, or an app's cases
  run           Run eval suites and report pass@1, pass@k, pass^k, flaky tasks and saturation
  candidates    Rank captured sessions in this repo by how good an eval task each would make
  new           Draft a task from a captured session
  import        Add tasks from task directories (hand-written, or from eval export)
  validate      Check the reference passes, near-misses fail and nothing leaks
  crosscheck    Ask a model whether stored solutions are valid and count where the verifier disagrees
  audit         Replay a witness test on an experiment's passing runs and count the passes it refutes
  trigger       Check that the agent loads a skill when it should and not when it shouldn't
  list          List eval tasks
  export        Write a task to a directory

Notes
  eval run exits 1 when a regression suite has a task that fails most of its trials. A capability suite reports its score and exits 0.
  pass@k is the chance at least one of k trials passes; pass^k is the chance all k pass. Both are estimated from every trial run.

coding.sh eval init [name] [options]

  Scaffold an eval suite in evals/<name>/: from failed sessions, a skill, or an app's cases

Arguments
  name          Suite name (default: the skill's or cases file's name, else failures or app)

Options
      --sessions                 Tasks from failed or flagged sessions in this repo (newest first)
      --session <ID>             Tasks from these sessions instead: an id, a prefix or latest (repeatable)
      --since <AGE>              With --sessions: how far back to look (30m, 12h, 7d, 2w) (default: 14d)
      --limit <N>                With --sessions: at most N tasks (default: 10)
      --repo <DIR|all>           With --sessions: sessions under this directory, or all, for another checkout or worktree of the project (default: this git repo)
      --verifier <CMD>           With --sessions: command that passes only when a task is solved (default: guessed per session)
      --hidden-test <GLOB>       With --sessions: test files to hide from the agent (repeatable)
      --where <QUERY>            Tasks from the sessions this query matches (sessions --where syntax), instead of every failed or flagged one
      --validate                 With --sessions or --where: validate each task before it goes into the suite. A session diff that passes the verifier becomes the reference and the task is marked validated; one that fails it, or that changes the tests the verifier runs, is kept as a near miss
      --validate-k <N>           With --validate: the session's end state must pass N of N verifier runs (default: 1)
      --skill <DIR>              Should / should-not trigger prompts for this skill
      --should <PROMPT>          With --skill: a prompt that should load it (repeatable)
      --should-not <PROMPT>      With --skill: a prompt that should not (repeatable)
      --cases <FILE>             Cases for an app or prompt: JSONL, JSON or CSV of {"input", "expected"}
      --describe <TEXT|@FILE>    What the app or skill should do: the judge's rubric when cases have no expected answer, and the brief for --draft
      --command <CMD>            Command that answers one case: the input arrives as JSON on stdin, the answer is what it prints
      --url <URL>                Or an HTTP endpoint that answers one case (POST)
      --draft [<PROVIDER|JSON>]  Have a model write the prompts or cases you didn't give, marked draft (default provider: anthropic)
      --draft-model <MODEL>      With --draft: the model
      --purpose <PURPOSE>        capability: a hill to climb (sessions default). regression: should pass nearly always, eval run exits 1 when it doesn't (skill and cases default)
  -k, --trials <N>               Trials per task, case or prompt (default: 3)
  -a, --agent <AGENT>            Agent that runs the suite (default: the sessions' agent, else claude-code)
      --force                    Replace an existing suite; refused when it holds files eval init did not write, or that you changed
      --force-all                Replace an existing suite and delete those files too

Examples
  coding.sh eval init
  coding.sh eval init --sessions --verifier 'npm test'
  coding.sh eval init --skill .claude/skills/pdf --draft

coding.sh eval run [suite]... [options]

  Run eval suites and report pass@1, pass@k, pass^k, flaky tasks and saturation

Arguments
  suite         Suite name under evals/, or its dir (default: every suite in evals/)

Options
  -k, --trials <N>            Trials per task, case or prompt (default: the suite's trials, else 3)
  -c, --concurrency <N>       Parallel runs
  -a, --agent <AGENT>         Agent for task and skill suites (default: the suite's, else claude-code)
      --model <MODEL>         Model for the agent
      --harness <BUNDLE>      Task suites: run this stored harness bundle (default: a snapshot of the project's current setup)
      --summary <FILE>        Also append the report to FILE as Markdown, e.g. "$GITHUB_STEP_SUMMARY" in GitHub Actions
      --retries <N>           Retries per trial after a rate limit, 5xx or network error, with backoff; a failed setup step is retried once and a refused login never (failures are never retried) (default: 3)
      --stop-after-infra <N>  Stop once N trials in a row fail on infrastructure after their retries (a refused login, a failed setup, rate limits, 5xx, network): no new trial starts and the run is no quality verdict; 0 never stops (default: 5)
      --trial-timeout <SEC>   Time limit per trial, retries included; a trial over it fails
  -y, --yes                   Answer yes to confirmation prompts. An agent passes it only once the user approves

Examples
  coding.sh eval run
  coding.sh eval run failures --trials 5 --yes
  coding.sh eval run triage --json

coding.sh eval candidates [options]

  Rank captured sessions in this repo by how good an eval task each would make

Options
  --since <AGE>     How far back to look (30m, 12h, 7d, 2w) (default: 14d)
  --limit <N>       Show the best N (default: 10)
  --repo <DIR|all>  Sessions under this directory, or all (default: this git repo)
  --flagged         Only failed or flagged sessions, the ones eval init --sessions drafts from
  --all             Also list the sessions that can't become a task, with the reason

Examples
  coding.sh eval candidates
  coding.sh eval candidates --since 30d --all
  coding.sh eval candidates --repo all --json

coding.sh eval new <SESSION> [options]

  Draft a task from a captured session

Arguments
  session       Session id, or 'latest'

Options
  --name <NAME>               Task name
  --instruction <TEXT|@FILE>  What the agent is asked to do (default: the session's first prompt)
  --statement <MODE>          How to write the instruction when --instruction is not given. first-prompt: the session's first prompt. ledger: every user turn and approved proposal as a list of obligations, with the ones a later turn replaced left out (default: first-prompt)
  --judge <JSON|FILE>         With --statement ledger: the model that merges turns into obligations, e.g. {"provider":"anthropic"} (default: one obligation per turn, no model)
  --verifier <CMD>            Command that passes only when the task is solved
  --setup <CMD>               Setup command (repeatable)
  --hidden-test <GLOB>        Test files to hide from the agent (repeatable)
  --tag <TAG>                 Tag (repeatable)

Examples
  coding.sh eval new latest --verifier 'npm test'
  coding.sh eval new latest --instruction @prompt.md --verifier 'npm test'
  coding.sh eval new latest --statement ledger --judge '{"provider":"anthropic"}' --verifier 'npm test'

coding.sh eval import <DIR>...

  Add tasks from task directories (hand-written, or from eval export)

Arguments
  dir           Task directory: task.json, instruction.md, optional repo/ (the start tree) and tests/ (hidden verifier files)

Examples
  coding.sh eval import tasks/*/

coding.sh eval validate [task]... [options]

  Check the reference passes, near-misses fail and nothing leaks

Arguments
  task          Task id or name (repeatable; or use --tag)

Options
  --tag <TAG>        Validate every task with this tag, e.g. suite:<name> after eval import or eval run
  --k <N>            Reference must pass N of N runs (default: 5)
  --keep             Keep work directories
  --strict-coupling  Fail validation when a hidden test names an identifier or file only the reference introduces

Examples
  coding.sh eval validate my-task
  coding.sh eval validate --tag suite:failures --k 1

coding.sh eval crosscheck <TASK> [options]

  Ask a model whether stored solutions are valid and count where the verifier disagrees

Arguments
  task          Task id or name

Options
  --experiment <ID>    Only runs from this experiment (default: every stored run of the task)
  --max <N>            Runs to review (one model call each) (default: 5)
  --judge <JSON|FILE>  Reviewer model: {"provider":"anthropic"|"openai"|"gateway"|"claude-code"|"codex"|"mock","model":…} (default: {"provider":"anthropic"})
  --solver <BUNDLE>    First solve the task once with this harness bundle and review that attempt too

Examples
  coding.sh eval crosscheck my-task --experiment exp_01J…
  coding.sh eval crosscheck my-task --solver claude-default --max 3

coding.sh eval audit <TASK> [options]

  Replay a witness test on an experiment's passing runs and count the passes it refutes

Arguments
  task          Task id or name

Options
  --experiment <ID>           Experiment whose passing runs to replay (required)
  --witness <FILE|PATH=FILE>  Test for a requirement the verifier misses; written at its path relative to here, or at PATH
  --witness-cmd <CMD>         Command that runs the witness from the repo root
  --requirement <TEXT>        The requirement it checks, in the instruction's words
  --promote                   Add the witnesses to the verifier afterwards (the task needs validating again)
  --count-ambiguous           Also count refutations by a witness that refutes over half the passing runs

Examples
  coding.sh eval audit add-neg --experiment exp_01J… --witness test/neg.test.js --witness-cmd 'node test/neg.test.js' --requirement 'handle negative numbers'
  coding.sh eval audit add-neg --experiment exp_01J…

coding.sh eval trigger <SKILL> [options]

  Check that the agent loads a skill when it should and not when it shouldn't

Arguments
  skill         Skill directory, or its SKILL.md; prompts come from triggers.toml next to it

Options
      --spec <FILE>               Trigger spec: [[should]] and [[should_not]] prompts (default: triggers.toml in the skill dir)
      --candidate <DIR>           A/B: another version of the skill, installed under the same name
      --description <TEXT|@FILE>  A/B: the same skill with only its frontmatter description replaced
  -a, --agent <AGENT>             Agent to probe (default: the --harness bundle's, else claude-code)
      --model <MODEL>             Model for the agent
      --harness <BUNDLE>          Start each arm from this harness bundle (its files, flags, env and model)
  -k, --k <N>                     Runs per prompt and arm (default: the spec's k, else 3)
  -c, --concurrency <N>           Parallel runs (default: 4)
      --seed <N>                  Seed for the run order and the bootstrap (default: 1)
      --timeout <SEC>             Seconds per run (default: the spec's timeout_s, else 120)
      --siblings <DIR>            Install the skills in this dir in every arm (default: the skills next to <SKILL>)
      --no-siblings               Install only the skill under test
      --min-f1 <X>                Exit 1 when the tested arm's F1 is under X (0 to 1)
      --fail-on <WHAT>            Exit 1 when B answers fewer prompts right than A: the 95% CI below 0 and an exact sign-flip p below 0.05
      --draft [<PROVIDER|JSON>]   Write triggers.toml instead of running it: prompts of captured sessions that loaded the skill (should) or a sibling (should_not), then a model fills each side (default provider: anthropic)
      --draft-model <MODEL>       With --draft: the model
      --n <N>                     With --draft: prompts per side (default: 8)
      --since <AGE>               With --draft: how far back to look for sessions (default: 90d)
      --repo <DIR|all>            With --draft: sessions under this directory, or all (default: this git repo)
      --no-traces                 With --draft: draft from SKILL.md only, without reading captured sessions
      --force                     With --draft: replace an existing triggers.toml. An agent passes it only once the user approves
      --dry-run                   With --draft: print the draft without writing it
  -y, --yes                       Answer yes to confirmation prompts. An agent passes it only once the user approves

Examples
  coding.sh eval trigger .claude/skills/pdf --draft --dry-run
  coding.sh eval trigger .claude/skills/pdf -k 3 --yes
  coding.sh eval trigger .claude/skills/pdf --description @new-description.txt --fail-on regression --yes

coding.sh eval list [options]

  List eval tasks

Options
  --status <STATUS>  Only tasks with this status
  --tag <TAG>        Only tasks with this tag

coding.sh eval export <TASK> [options]

  Write a task to a directory

Arguments
  task          Task id or name

Options
  -o, --out <DIR>        Output directory (required)
      --format <FORMAT>  coding.sh: task dir. harbor: a Harbor task (task.toml schema 1.4, the source session in solution/trajectory.json). atif: the source session as an ATIF-v1.8 trajectory.json (default: coding.sh)
      --force            Replace the named target when it already exists. An agent passes it only once the user approves