Experiments
Compare experiments with paired statistics, sweep and matrix configs, stop a loop at its plateau, resume runs, A/B metrics and replay a model call.
An experiment is one graded run of a task over a dataset (see Evals). This page covers what comes next: compare two runs, pick the best of several configs, know when a loop should stop, and keep long runs going. Every flag is in the exp reference.
Compare two runs (built)
exp compare A B pairs the two runs by case. It prints B − A with a paired bootstrap interval, McNemar's exact test on the
cases that flipped, and the cases that regressed. When the interval includes zero, the verdict is inconclusive.
$ coding.sh exp compare exp_01M6SAJYC5RM7A5TCT65R3SFSM exp_01M6SAK893Z1DFH770KVNQC4BZ A base exp_01M6SAJYC5RM7A5TCT65R3SFSM pass 100.0% B cand exp_01M6SAK893Z1DFH770KVNQC4BZ pass 50.0% B − A -50.0pp [-100.0pp, 0.0pp] (2 paired cases, 95% paired bootstrap) mcnemar p=1.000 1 regressed, 0 improved verdict inconclusive (CI includes 0) regressions 1 case A B Δ ------------------ ---- -- ------ c_756966685591eb9b 100% 0% -100pp
| Flag | What it adds |
|---|---|
--gate | A ship-or-hold line; exits 0 only for "Ship it" (how it decides). |
--margin 0.03 | Non-inferiority: exits 1 unless the interval's lower end is above −3pp. |
--cluster repo | Resamples groups of related cases instead of single cases; warns below 10 groups. |
--matrix | The cases by passes out of k in A and in B. |
--pairwise judge.json | A judge that sees both outputs in both orders. Repeat it to ask several. |
--labels-a, --labels-b | Human labels that correct each LLM judge's B − A (PPI++). |
--verdict seq | The verdict from Seq@K, when both runs used the same --seq and --feedback. |
What else compare reports
- With case kinds: each kind's pass rate, bad cases that do not show the fix, and broken guards and controls.
- Below 100 paired cases: P(B>A) from a Bayesian posterior. The bootstrap interval stays.
- Trials lost to errors, split into infra and agent errors, with a warning when the split is uneven.
- After
exp aa(one config run twice): the A/A flip rate. The minimum detectable effect never goes below it. - The exact McNemar power:
no 80% power at this n (exact McNemar ceiling 62%), orexact MDE 10.6pp. - A warning when the two runs ran one after the other.
- With 2+ trials per case: the cases B gained and lost (
lost coverage). A net gain can hide lost cases. - A warning when a grader changed version between A and B, or graded one side only. Ungraded cases are left out.
- When B ran with
exp run --failing A: a regression-to-the-mean warning. The verdict is inconclusive and--gateholds.
The evals guide has the method behind each line.
Grade again with new graders (built)
exp regrade <experiment> spec.json grades a stored experiment's outputs again with the spec's graders, without calling
the task. It prints the pass rate before and after, and every run whose outcome flipped. Regrade A with B's graders, then compare like
with like.
Run two configs as one A/B (built)
exp run --arm base=a.json --arm cand=b.json runs each case in both arms back to back, in random order, so drift during the
run hits both arms the same. --stop-early stops once the answer is clear (an O'Brien-Fleming group sequential test, never
before 20 cases). --max-cases caps the run.
Pick the best of several configs (built)
coding.sh exp sweep base=base.json t2=temp2.json t5=temp5.json -d smoke -g '{"kind":"exact"}' --budget 300
The race uses sequential halving: each round runs the surviving configs on the same fresh train cases and keeps the better half. The
winner and the first config then run on the test split, which the race never saw. The output prints adopt <winner>
or keep <start>, and the winner's optimism gap (train minus held-out score).
| Flag | What it adds |
|---|---|
--budget N | Caps the race's case runs. |
--guard old-suite | An earlier suite the winner must not forget. If it does, the sweep keeps the start and exits 1. |
--guard-tau 0.10 | The share of the start's solved guard cases the winner may break. |
--guard prior | Guards every suite a sweep adopted a config on before. |
With three or more configs, a line names the plausibly best set: the configs not significantly worse than the best. It holds the truly best config 95% of the time, so a dropped config still in it may have lost on noise.
Run every combination (built)
A spec with "matrix" runs every combination of its axes against one baseline cell. Each axis is a JSON Pointer into the spec
and a list of values ("variants" names cells by hand instead). A cell may change only /task/… and
/trials; anything else is refused before it runs.
{
"name": "shout-matrix",
"dataset": "qa",
"task": { "kind": "command", "command": "node agent.mjs", "env": { "MODEL": "stub-small", "PROMPT": "plain.md" } },
"graders": [{ "kind": "regex", "pattern": "^[A-Z]+$" }],
"matrix": {
"model": { "path": "/task/env/MODEL", "values": [{ "small": "stub-small" }, { "big": "stub-big" }] },
"prompt": { "path": "/task/env/PROMPT", "values": [{ "plain": "plain.md" }, { "shout": "shout.md" }] }
},
"baseline": { "model": "small", "prompt": "plain" }
} exp run matrix.json runs the cells interleaved and compares each with the baseline. The p-values are Holm-adjusted over the
cells, so a big matrix doesn't produce a winner by chance.
cell pass B − A p p (Holm) verdict
model=small,prompt=plain 0.0% baseline
model=small,prompt=shout 100.0% +100.0pp 0.008 0.023 better
model=big,prompt=plain 50.0% +50.0pp 0.125 0.125 inconclusive
model=big,prompt=shout 100.0% +100.0pp 0.008 0.023 better| Flag | What it adds |
|---|---|
--plan | Prints the cells and what each changes; runs nothing. |
--cell NAME | Runs only these cells. The baseline always runs. |
--reuse baseline | Copies the baseline's finished case-trials from earlier experiments (all: any cell's). |
--reuse-max-age 7d | Copies only case-trials that started this recently. |
--resume | Picks up an interrupted matrix. |
Know when to stop a loop (built)
Tag each round with exp run … --series NAME. It prints series NAME: continue, incumbent v4 … or
series NAME: stop, ship v2 … (the plateau began at round 3). exp plateau --series NAME prints one row per round
and exits 0 (continue), 3 (stop) or 2 (too few experiments).
How it decides
Each round compares the candidate with the incumbent on the cases both ran. A restart e-detector (Yin, Liu and Qi, arXiv 2610.04756)
stops once the gains stay below ε (--eps, default 0.05) and names the incumbent before the plateau as the version to ship.
--gate greedy keeps a candidate only when it beats the incumbent; --gate chain keeps every one. False alarms
come on average no sooner than every 1/δ rounds (20 at --delta 0.05): an average run length, not an error rate.
Resume a long run (built)
exp run, exp aa, exp sweep, compare, bench run and
workflows run save each trial as it finishes. After Ctrl-C, a crash or a reboot, run the same command with
--resume. Only the unfinished trials run, and the verdict matches a run that was never interrupted. If the dataset, graders,
task or agent script changed, it refuses and names what changed.
| Flag | What it does |
|---|---|
--resume <experiment> | Runs what is left of a stored run that exp list marks (partial). |
--retry-errors | With --resume: also runs its task errors again. |
--retries N | Retries rate limits, 5xx and network errors (default 3). Task failures are never retried. |
--trial-timeout SEC | Caps each trial, retries included. |
exp status shows each run's progress, ETA and cost so far.
Run part of a dataset, or retry with feedback (built)
| Flag | What it does |
|---|---|
--failing, --case, --sample, --first-n | Run part of a dataset. |
--seq 3 --feedback reasons | Tries a failing case-trial again with feedback, up to 3 attempts. The pass rate stays the first attempt's (modes). |
See each run's delta as you iterate (built)
exp session new --name <topic> starts a session in the current folder. Each exp run there then prints its
delta against the previous run on the same dataset, and names the files that changed. exp session note records a hypothesis
or a reading, and exp session summary prints the timeline.
--watch reruns whenever the spec, the agent script or a grader file changes. A change to the graders alone regrades the last
outputs with no agent calls. Enter reruns, q quits.
Compare two numeric commands (built)
exp metrics --arm base='node bench.mjs a' --arm cand='node bench.mjs b' compares two commands that print numbers, such as
{"lcp_ms": 2310}. The arms run in interleaved pairs (11 by default). For each metric it prints both medians, the
paired difference with a 95% interval, and a verdict.
Lower is better unless you name the metric in --higher. A change counts only when the interval excludes zero and a corrected
sign test agrees, so an A/A run stays quiet. --fail-on-worse exits 1 for CI.
Replay one model call (built)
exp replay sends a call that the proxy logged again, with the model, the system
prompt or any request field changed. One --vary value changes arm B and keeps arm A as recorded; two values set both.
(exp ab replays a whole turn instead.)
coding.sh exp replay latest#-1 --vary model=claude-sonnet-4-5,claude-haiku-4-5 --n 5 --dry-run
coding.sh exp replay latest#-1 --vary model=claude-sonnet-4-5,claude-haiku-4-5 --n 5
coding.sh exp replay --session latest --limit 10 --vary system=@new-prompt.md
A reply passes when its first action (a tool call by name, or text) matches the agent's; --criterion adds an LLM judge. The
report is the paired comparison of exp compare. It uses your own ANTHROPIC_API_KEY or
OPENAI_API_KEY, never the agent's login. --dry-run prints the call count and a cost estimate without a key.