coding.sh exp
Run a task over a dataset, compare experiments and sweep configs. Every option of coding.sh exp and its subcommands, from its own --help.
coding.sh exp <action> Run a task over a dataset, compare experiments and sweep configs Commands ab A/B an AGENTS.md, rule or skill edit on the decision points your agent recently got wrong replay Replay logged model calls against the real API, with your own key, varying the model, system prompt or a field run Run an experiment, two tasks as one interleaved A/B, or a matrix of cells against a baseline cell session Group runs into a session: new, note <text>, summary. exp run in it prints the delta against the previous run compare Compare two experiments case by case regrade Grade a stored experiment's outputs again with new graders, without calling the task, and show what flipped aa Run one config twice (A/A) and store its run-to-run noise sweep Race task configs with sequential halving, then re-score the winner on held-out cases metrics A/B two metric commands (ms, bytes) in interleaved pairs plateau Should an iteration loop stop? A restart e-detector over its experiments, and the version to ship (exit 3 stop, 2 too few, 0 continue) show Show a stored experiment: pass rate, graders and failures list List experiments status Progress, ETA and cost so far of long runs (exp, compare, bench, workflows), and how to resume them Notes exp ab and refine call a model API with ANTHROPIC_API_KEY or OPENAI_API_KEY. With neither set they use a logged-in Claude Code (`claude -p`) or Codex (`codex exec`) instead and say so in one line; --provider claude-code or codex asks for one. Each call runs with no hooks, plugins, MCP servers or settings of yours, in an empty directory, with a throwaway CLAUDE_CONFIG_DIR or CODEX_HOME holding only a copy of the login. claude -p gets the session's own Claude Code tools and refuses every call, so none runs. On a subscription, costs are API-equivalent (what the same tokens would cost on the API), there is no spending cap unless you pass --max-cost, calls run 2 at a time, rate limits back off and retry, and a spent usage limit stops the run and keeps the finished trials. With no uncommitted harness edits, exp ab tests your last commit that changed AGENTS.md (or another instructions file), a rule or a skill (B) against the version before it (A).
coding.sh exp ab
A/B an AGENTS.md, rule or skill edit on the decision points your agent recently got wrong.
coding.sh exp ab [spec] [options]
A/B an AGENTS.md, rule or skill edit on the decision points your agent recently got wrong
Arguments
spec JSON file with the same fields as the options (behavior, since, a, b, model, …); options override it
Options
--a <FILE|DIR|git:REV|BUNDLE|none> Baseline harness (default: git HEAD when B is omitted, else the repo's current files; with no uncommitted edits, the version before your last harness commit)
--b <FILE|DIR|git:REV|BUNDLE|none> Candidate harness; a file replaces the file of the same path in A (default: your uncommitted edits, else your last harness commit)
--behavior <ID[,ID]> Decision points to test: unverified-claim, done-while-failing, unchanged-retry (default: all)
--anchor <FILE> Module exporting your own anchor(s) (see the evals guide)
--since <DURATION|DATE> Sessions started since (default: 14d)
--agent <AGENT> Only sessions of this agent (repeatable, or a comma list)
--repo <DIR|all> Only sessions under this directory (default: this git repo)
-n, --limit <N> Maximum decision points (default: 40)
--provider <NAME> Who answers the next turn and judges it: an API, or claude-code / codex to run `claude -p` / `codex exec` on your subscription (default: anthropic; with no API key set, a logged-in Claude Code, else Codex)
--model <MODEL> Model for the next turn (default: the one the decision points were recorded with, else claude-sonnet-5-5 or gpt-6-sol)
--judge-model <MODEL> Model for the judge (default: --model)
--base-url <URL> API base URL (default: ANTHROPIC_BASE_URL / OPENAI_BASE_URL, else the provider's)
--trials <N> Next turns per case and arm (default: 1)
-c, --concurrency <N> Parallel model calls (default: 4)
--max-cost <USD> Refuse to start when the estimate is over this, and stop once the spend passes it (API-equivalent dollars with claude-code or codex)
--dry-run Show the decision points and the cost estimate; make no model calls
--watch Rerun when B's files change, one line per run; Ctrl-C to stop
--cache Reuse stored model responses (--no-cache to ask the model again) (default: true)
--name <NAME> Experiment name (default: the dataset's)
--gate Ship-or-hold decision: exit 0 when B wins enough without breaking what A solved, 1 otherwise
--lambda <N> With --gate: what breaking one solved item costs, in fixed items (default: 2)
--gate-alpha <A> With --gate: one-sided significance level (default: 0.05)
--tau <T> With --gate: largest share of solved items B may break (default: 0.10)
--allow-protected-edit <PATH:LINE> Allow the edit to delete, reword or relax this protected line (repeatable; never implied by --force)
--resume [<ID>] Continue an interrupted A/B: finished trials are kept, only the rest run (no ID: the newest unfinished one with the same config)
--retries <N> Retries per trial after a rate limit, 5xx or network error, with backoff; a failed setup step is retried once and a refused login never (failures are never retried) (default: 3)
--stop-after-infra <N> Stop once N trials in a row fail on infrastructure after their retries (a refused login, a failed setup, rate limits, 5xx, network): no new trial starts and the run is no quality verdict; 0 never stops (default: 5)
--trial-timeout <SEC> Time limit per trial, retries included; a trial over it fails
Examples
coding.sh exp ab
coding.sh exp ab --b ./drafts/AGENTS.md --behavior unverified-claim
coding.sh exp ab --a git:main --b . --since 30d --dry-run coding.sh exp replay
Replay logged model calls against the real API, with your own key, varying the model, system prompt or a field.
coding.sh exp replay [call]... [options]
Replay logged model calls against the real API, with your own key, varying the model, system prompt or a field
Arguments
call Calls logged by coding.sh proxy: an id, a 6+ character prefix, or <session>#<n> (latest#-1 is the last call)
Options
--vary <KEY=B|KEY=A,B> A request field to vary: model, system, tools or any body field (dots for nested, @file reads a file). One value: A sends the recorded request, B this. Two: A and B
--session <SESSION> Also replay this session's proxy calls (the last --limit)
--limit <N> With --session: the last N calls (default: 20)
--n <N> Replays per call and arm (default: 5)
-g, --grader <JSON|FILE> More grader specs (same-action always runs: does the reply take the first action the agent took?)
--criterion <TEXT> Also grade each reply with an LLM judge on this criterion
--judge-model <MODEL> Model for --criterion
--upstream <URL> API base URL (default: the proxy's upstream for the call's provider)
-c, --concurrency <N> Parallel model calls (default: 4)
--max-cost <USD> Refuse to start when the estimate is over this
--dry-run Show the calls, the arms and a cost estimate; make no model calls and need no key
--name <NAME> Experiment name (default: replay-<first call id>)
--resume [<ID>] Continue an interrupted replay: finished trials are kept, only the rest run (no ID: the newest unfinished one with the same config)
--retries <N> Retries per trial after a rate limit, 5xx or network error, with backoff; a failed setup step is retried once and a refused login never (failures are never retried) (default: 3)
--stop-after-infra <N> Stop once N trials in a row fail on infrastructure after their retries (a refused login, a failed setup, rate limits, 5xx, network): no new trial starts and the run is no quality verdict; 0 never stops (default: 5)
--trial-timeout <SEC> Time limit per trial, retries included; a trial over it fails
Examples
coding.sh exp replay latest#-1 --vary model=claude-haiku-4-5 --dry-run
coding.sh exp replay latest#-1 --vary model=claude-sonnet-4-5,claude-haiku-4-5 --n 5
coding.sh exp replay --session latest --vary system=@prompt.md --n 3 coding.sh exp run
Run an experiment, two tasks as one interleaved A/B, or a matrix of cells against a baseline cell.
coding.sh exp run [spec] [options]
Run an experiment, two tasks as one interleaved A/B, or a matrix of cells against a baseline cell
Arguments
spec Experiment spec JSON file (options override its fields; --grader adds graders); with "matrix" or "variants" it runs every cell
Options
-d, --dataset <ID> Dataset id
--task <JSON|FILE> Task spec: {"kind":"command","command":"…"} (gets each case input as JSON on stdin) or {"kind":"http","url":"…"}
-g, --grader <JSON|FILE> Grader spec, e.g. {"kind":"exact"} (repeatable); added to the spec file's graders
--replace-graders Use only the --grader graders, not the spec file's
--name <NAME> Experiment name
--trials <N> Attempts per case (default: 1)
-c, --concurrency <N> Parallel case runs (default: 4)
--split <SPLIT> Only cases in this split
--samples <MODE> adaptive: 2 trials each, a 3rd when they disagree; it fails at half or more failing. Reports no CI (default: fixed)
--stop-on-regression Stop and exit 1 once a case that passed every trial of --baseline fails
--baseline <EXPERIMENT> With --stop-on-regression: the stored experiment to hold the run to
--arm <NAME=JSON|FILE> One arm's task spec; give two (A then B) to interleave them case by case in one run
--stop-early With --arm: stop once a group sequential test (O'Brien-Fleming) is decisive
--looks <N> With --stop-early: planned looks over the case budget (default: 4)
--max-cases <N> With --arm: run at most N cases (a random subset)
--seed <N> With --arm or a matrix spec: seed for the case and arm order (default random, recorded). With --sample: seed for the draw (default 0)
--failing <EXPERIMENT> Only the cases this experiment failed (compare warns and holds when the other side is that experiment: regression to the mean)
--errors-from <EXPERIMENT> Only the cases with a task or infrastructure error in this experiment
--case <PATTERN> Only cases whose id matches (* and ? wildcards; repeatable)
--sample <N|SHARE> A seeded random sample of the cases: a count (10) or a share (0.2); the same seed picks the same cases
--first-n <N> Only the first N cases, in dataset order (after the other selections)
--retry-errors With --resume <EXPERIMENT>: also run its task-errored case-trials again (a finished experiment too); the rest are kept
--seq <K> Sequential attempts: a case-trial that fails is tried again with feedback, up to K attempts (2-10); the pass rate stays the first attempt's (Seq@1) and the run reports Seq@1..K. Command and http tasks need "envelope": true
--feedback <MODE> With --seq: bit (one fixed sentence), reasons (the failed graders' reasons, at most 3, 300 characters each) or critic=<GRADER JSON|FILE> (a grader's reason on the failed output; it never gates). The expected answer is never shown (default: reasons)
--seq-history <MODE> With --seq: show each new attempt every earlier attempt, or only the last (default: full)
--plan With a matrix spec: print the cells and what each changes, check them, and run nothing
--cell <NAME> With a matrix spec: run only this cell (repeatable); the baseline cell always runs
--reuse <CELLS> With a matrix spec: copy finished case-trials of the same case, task, graders, trial and timeout from earlier experiments instead of running them again (default: none)
--reuse-max-age <AGE> With --reuse: only case-trials that started this recently (30m, 12h, 7d, 2w) (default: 7d)
--control A continuation task or a --replay needs a control arm (A, the recorded config) to measure replay noise; --no-control runs one task alone and marks the result (default: true)
--record Record the task's model calls (Anthropic and OpenAI, through a local proxy the task's ANTHROPIC_BASE_URL/OPENAI_BASE_URL point at), so a later run can --replay them
--replay <EXPERIMENT> Serve this recorded experiment's model calls again, up to each case-trial's first changed call; later calls run live. Adds the recorded config as a control arm (A) unless --no-control
--replay-until <N> With --replay: serve only calls 1..N from the recording; call N+1 onward runs live
--series <NAME> Tag the experiment as the next round of an iteration loop and print whether the loop should stop (see exp plateau)
--watch Rerun on every change to the spec, the task's script, grader files or --watch-path; a grader-only change regrades the stored outputs. Enter reruns, q quits
--watch-path <GLOB> With --watch: more files whose change reruns the task (repeatable; also the spec's "watch" field)
--session In an experiment session (exp session new), add the run to its timeline and print the delta against the previous run; --no-session skips it (default: true)
--resume [<ID>] Continue an interrupted experiment: finished trials are kept, only the rest run (no ID: the newest unfinished one with the same config)
--retries <N> Retries per trial after a rate limit, 5xx or network error, with backoff; a failed setup step is retried once and a refused login never (failures are never retried) (default: 3)
--stop-after-infra <N> Stop once N trials in a row fail on infrastructure after their retries (a refused login, a failed setup, rate limits, 5xx, network): no new trial starts and the run is no quality verdict; 0 never stops (default: 5)
--trial-timeout <SEC> Time limit per trial, retries included; a trial over it fails
Examples
coding.sh exp run spec.json --watch
coding.sh exp run -d smoke -g '{"kind":"exact"}' --arm base=base.json --arm cand=cand.json --stop-early --max-cases 400
coding.sh exp run spec.json --resume coding.sh exp session
Group runs into a session: new, note <text>, summary. exp run in it prints the delta against the previous run.
coding.sh exp session <COMMAND> [text]... [options] Group runs into a session: new, note <text>, summary. exp run in it prints the delta against the previous run Arguments command new, note or summary text With note: the note (hypothesis before a run, what you saw after it) Options --name <TOPIC> With new: the session's name Examples coding.sh exp session new --name shorter-prompt coding.sh exp session note "hypothesis: the shorter prompt fixes c3" coding.sh exp session summary --json
coding.sh exp compare
Compare two experiments case by case.
coding.sh exp compare <A> <B> [options]
Compare two experiments case by case
Arguments
a Baseline experiment id or name (newest)
b Candidate experiment id or name (newest)
Options
--force Compare a cancelled (partial) experiment too
--cluster <KEY> Case metadata field (e.g. repo, session) whose clusters the bootstrap resamples instead of single cases
--pairwise <JSON|FILE> Pairwise judge spec, {"kind":"pairwise","name":…,"rubric":…}: judges A and B's outputs side by side, in both orders. Repeat it for more judges: compare then reports how far they agree, and --gate holds unless every judge finds B better
--labels <FILE> With --pairwise: human labels (JSON array or JSONL of {"caseId","winner":"a"|"b"|"tie"}) to calibrate each judge on the test split
--matrix Also print the success-count matrix: cases by passes out of k in A (rows) and B (columns)
--margin <D> Non-inferiority: exit 1 unless the 95% CI lower bound of B − A is above −D (0.05 = 5pp)
--labels-a <FILE> Human labels of A's outputs (JSON array or JSONL of {"caseId","value":"pass"|"fail"|"skip","criterion"?}): LLM judge graders get a PPI++ corrected B − A
--labels-b <FILE> Human labels of B's outputs (required with --labels-a; labels are never shared across arms)
--min-labels <N> With --labels-a/-b: fewest usable labels per arm for a corrected delta (default: 20)
--verdict <ON> What the verdict reads: the first-attempt pass rate, or Seq@K when both runs used the same --seq and --feedback (default: pass)
--gate Ship-or-hold decision: exit 0 when B wins enough without breaking what A solved, 1 otherwise
--lambda <N> With --gate: what breaking one solved item costs, in fixed items (default: 2)
--gate-alpha <A> With --gate: one-sided significance level (default: 0.05)
--tau <T> With --gate: largest share of solved items B may break (default: 0.10)
Examples
coding.sh exp compare base cand
coding.sh exp compare base cheaper --margin 0.03
coding.sh exp compare base cand --cluster repo coding.sh exp regrade
Grade a stored experiment's outputs again with new graders, without calling the task, and show what flipped.
coding.sh exp regrade <EXPERIMENT> [spec] [options]
Grade a stored experiment's outputs again with new graders, without calling the task, and show what flipped
Arguments
experiment Experiment id or name (newest)
spec Experiment spec JSON file whose graders to use (--grader adds to them)
Options
-g, --grader <JSON|FILE> Grader spec (repeatable); added to the spec file's graders
--replace-graders Use only the --grader graders, not the spec file's
--name <NAME> Name of the regraded experiment (default: the original's)
-c, --concurrency <N> Parallel grader runs (default: 4)
Examples
coding.sh exp regrade base spec.json
coding.sh exp regrade base -g strict.json --json coding.sh exp aa
Run one config twice (A/A) and store its run-to-run noise.
coding.sh exp aa [a] [b] [options]
Run one config twice (A/A) and store its run-to-run noise
Arguments
a Spec JSON file to run twice, or the first of two stored runs of one config
b The second stored run (compares the two instead of running)
Options
-d, --dataset <ID> Dataset id
--task <JSON|FILE> Task spec: {"kind":"command","command":"…"} (gets each case input as JSON on stdin) or {"kind":"http","url":"…"}
-g, --grader <JSON|FILE> Grader spec, e.g. {"kind":"exact"} (repeatable); added to the spec file's graders
--replace-graders Use only the --grader graders, not the spec file's
--name <NAME> Experiment name
--trials <N> Attempts per case (default: 1)
-c, --concurrency <N> Parallel case runs (default: 4)
--split <SPLIT> Only cases in this split
--resume [<ID>] Continue an interrupted A/A run: finished trials are kept, only the rest run (no ID: the newest unfinished one with the same config)
--retries <N> Retries per trial after a rate limit, 5xx or network error, with backoff; a failed setup step is retried once and a refused login never (failures are never retried) (default: 3)
--stop-after-infra <N> Stop once N trials in a row fail on infrastructure after their retries (a refused login, a failed setup, rate limits, 5xx, network): no new trial starts and the run is no quality verdict; 0 never stops (default: 5)
--trial-timeout <SEC> Time limit per trial, retries included; a trial over it fails
Examples
coding.sh exp aa --dataset smoke --task '{"kind":"command","command":"node agent.mjs"}' --grader '{"kind":"exact"}'
coding.sh exp aa smoke-aa1 smoke-aa2 coding.sh exp sweep
Race task configs with sequential halving, then re-score the winner on held-out cases.
coding.sh exp sweep <CONFIGS>... [options]
Race task configs with sequential halving, then re-score the winner on held-out cases
Arguments
configs Task configs, [NAME=]<JSON|FILE>; the first is the start config
Options
-d, --dataset <ID> Dataset id
-g, --grader <JSON|FILE> Grader spec, e.g. {"kind":"exact"} (repeatable); added to the spec file's graders
--name <NAME> Experiment name
-c, --concurrency <N> Parallel case runs (default: 4)
--budget <N> Case runs for the race, over all configs (default: each race case used once by the last round)
--split <SPLIT> Split the race runs on (default: train)
--held-out <SPLIT> Split the winner is re-scored on (a seeded 30% when the dataset has no splits) (default: test)
--seed <N> Seed for the case and job order (default random, recorded)
--guard <DATASET[@VER]|prior> An earlier suite the winner must not forget: start and winner run on it (its test split, else every case), and a dont-ship gate blocks adoption and exits 1. prior: every dataset a sweep adopted a config on (repeatable)
--guard-tau <T> With --guard: largest share of the start's solved guard cases the winner may break (default: 0.10)
--guard-lambda <L> With --guard: loss weight of the guard gate (default: 2)
--resume [<ID>] Continue an interrupted sweep: finished trials are kept, only the rest run (no ID: the newest unfinished one with the same config)
--retries <N> Retries per trial after a rate limit, 5xx or network error, with backoff; a failed setup step is retried once and a refused login never (failures are never retried) (default: 3)
--stop-after-infra <N> Stop once N trials in a row fail on infrastructure after their retries (a refused login, a failed setup, rate limits, 5xx, network): no new trial starts and the run is no quality verdict; 0 never stops (default: 5)
--trial-timeout <SEC> Time limit per trial, retries included; a trial over it fails
Examples
coding.sh exp sweep base=base.json t2=temp2.json t5=temp5.json -d smoke -g '{"kind":"exact"}' --budget 300
coding.sh exp sweep base=base.json cand=cand.json -d swe -g '{"kind":"exact"}' --guard prior coding.sh exp metrics
A/B two metric commands (ms, bytes) in interleaved pairs.
coding.sh exp metrics [options] A/B two metric commands (ms, bytes) in interleaved pairs Options --arm <NAME=CMD|JSON|FILE> One arm: a command line, a task spec or a spec file; give two (A then B). The command prints a JSON object of numbers --pairs <N> Interleaved pairs (each runs A and B once, in balanced random order) (default: 11) --warmup <N> Runs of each arm before the pairs, not counted (default: 0) --higher <METRIC> Metric where higher is better (repeatable, * wildcard); lower is better otherwise --metric <METRIC> Only report these metrics (repeatable, * wildcard) --correction <METHOD> Multiple-comparison correction across metrics (default: holm) --timeout <SEC> Per-run timeout for arms that don't set timeoutMs (default: 600) --seed <N> Seed for the arm order (default random, recorded) --name <NAME> Experiment name --fail-on-worse Exit 1 when any metric is worse in B Examples coding.sh exp metrics --arm base='node bench.mjs --dir a' --arm cand='node bench.mjs --dir b' coding.sh exp metrics --arm base=base.json --arm cand=cand.json --pairs 21 --higher score --json
coding.sh exp plateau
Should an iteration loop stop? A restart e-detector over its experiments, and the version to ship (exit 3 stop, 2 too few, 0 continue).
coding.sh exp plateau [experiments]... [options]
Should an iteration loop stop? A restart e-detector over its experiments, and the version to ship (exit 3 stop, 2 too few, 0 continue)
Arguments
experiments Experiment ids or names in loop order, baseline first (default: --series or --dataset, oldest first)
Options
--series <NAME> The experiments tagged by exp run --series NAME
-d, --dataset <ID> The completed experiments on this dataset
--baseline <ID> Version 0 of the loop; earlier experiments are left out
--eps <GAIN> Smallest gain per round worth another round (0.05 = 5pp) (default: 0.05)
--delta <P> Stop once the evidence reaches 1/δ (false alarms average no sooner than every 1/δ rounds) (default: 0.05)
--gate <GATE> greedy keeps a candidate only when it beats the incumbent; chain keeps every candidate (metadata.accepted overrides both) (default: greedy)
--bet <BET> Betting strategy of the detector (default: agrapa)
Examples
coding.sh exp plateau --series prompt-loop
coding.sh exp plateau v0 v1 v2 v3 v4 --gate chain --json
coding.sh exp run spec.json --name v5 --series prompt-loop coding.sh exp show
Show a stored experiment: pass rate, graders and failures.
coding.sh exp show <EXPERIMENT> Show a stored experiment: pass rate, graders and failures Arguments experiment Experiment id or name (newest, cancelled ones included)
coding.sh exp list
List experiments.
coding.sh exp list [options] List experiments Options -n, --limit <N> Maximum rows (default: 20)
coding.sh exp status
Progress, ETA and cost so far of long runs (exp, compare, bench, workflows), and how to resume them.
coding.sh exp status [run] [options] Progress, ETA and cost so far of long runs (exp, compare, bench, workflows), and how to resume them Arguments run One run: its id, or the id of an experiment it produced Options -n, --limit <N> Maximum rows (default: 20) Examples coding.sh exp status coding.sh exp status exp_01JC3T6Z2Q8W1S5V0R7N4K9M2D --json