coding.sh compare

coding.sh compare <A> [b] [options]

  A/B two harness bundles on eval tasks

Arguments
  a             Baseline bundle id or name (with --parity: a URL, or a command that prints the page)
  b             Candidate bundle id or name (omit with --without or --with; with --parity: the other URL or command)

Options
      --parity                  Instead: diff what a crawler reads from two pages (status, redirects, title, description, canonical, robots, Open Graph and Twitter tags, hreflang count, JSON-LD types, lang). Exits 1 when they differ
      --user-agent <UA>         With --parity: the User-Agent to fetch as (default: a coding.sh crawler agent)
  -t, --task <ID>               Task id (repeatable; default: all validated tasks)
      --tag <TAG>               Run every non-archived task with this tag, drafts included (e.g. the tag given to eval import)
      --k <N>                   Attempts per task and arm (default: 1)
  -c, --concurrency <N>         Parallel runs
      --max-cost <USD>          Spend cap: once finished runs (agent plus judge grade calls) reach it, no new run starts
      --pick <N>                Run the N tasks that best tell harnesses apart in stored runs; tasks every run passed go last
      --pick-by <HOW>           With --pick or --budget: rank by outcome-suite correlation, or by item information from an IRT fit of stored runs (mid difficulty for A, high discrimination first; needs 3+ configs) (default: separation)
      --budget <USD>            Run the most informative tasks whose estimated cost (2 arms x k runs each) fits; also the spend cap unless --max-cost is set
      --cost-per-run <USD>      With --budget: the cost of one run when no stored run, source session or other task gives one
      --sequential              Run each arm's tasks in order, one stream per attempt, carrying the agent's HOME state from task to task; the report adds a learning curve
      --carry <PATH>            With --sequential: what to carry, a path in the run's HOME such as ~/.claude/projects (repeatable; default: the whole HOME)
      --otel                    Give each agent an OTLP endpoint, a TRACEPARENT and coding_sh.run_id, so its spans join the run's session
      --isolation               Grade each run in a fresh copy of the start tree plus the agent's source changes (default on); --no-isolation grades the agent's own tree
      --experiment <ID>         Compare the stored runs of this experiment instead of running
      --retry-errors            With --experiment: run again the runs that errored or hit infrastructure errors, in place; the old runs are kept, marked superseded
      --failing <EXP>[@BUNDLE]  Run only the tasks A (or BUNDLE) failed in that stored comparison
      --failing-mode <MODE>     With --failing: a task counts as failed in any attempt, or in more than half (default: any)
      --sample <N>              Run a random sample of N of the selected tasks
      --seed <N>                With --sample: seed for the draw (default random, printed in the report). With --false-alarm: seed for which runs get it (default 0)
      --repair <N>              After a failed verifier, tell the agent and let it try again in its own session, up to N rounds (1-8); hidden tests stay out of its tree
      --repair-signal <SIGNAL>  With --repair: say only that verification failed, or also name the failing tests (parsed from the verifier output). (default: bit)
      --false-alarm <P>         With --repair: tell this share (0-1) of passing runs they failed, to measure what a wrong signal breaks
      --repair-verdict          Verdict, gates and pass rates on the pass after repair rounds instead of the first attempt
      --reuse <ARMS>            Copy finished runs of the same task, bundle, agent version, attempt and timeout from earlier comparisons and eval runs instead of running them again (default: none)
      --reuse-max-age <AGE>     With --reuse: only runs that finished this recently (30m, 12h, 7d, 2w) (default: 7d)
      --dry-run                 Print how many runs each arm would reuse and run, and run nothing
      --without <PATH>          Ablation: B is A without this file, dir or glob, or without plugin:<NAME> (repeatable)
      --with plugin:<PLUGIN>    B is A plus this Claude Code plugin, an installed one (name or name@marketplace) or a plugin directory (repeatable; claude-code bundles)
      --gateway [<NAME>]        Route both arms through a gateway. Without a name: [gateway] default, else the one with a key, else vercel
      --gate                    Ship-or-hold decision: exit 0 when B wins enough without breaking what A solved, 1 otherwise
      --lambda <N>              With --gate: what breaking one solved item costs, in fixed items (default: 2)
      --gate-alpha <A>          With --gate: one-sided significance level (default: 0.05)
      --tau <T>                 With --gate: largest share of solved items B may break (default: 0.10)
      --by <PREFIX>             With --gate: also gate each task group (first tag with this prefix, e.g. suite:); any group's don't-ship blocks
      --promote                 Pareto promotion gate: exit 0 when B is credibly better on quality or cheaper (cost, else tokens), 1 otherwise
      --gamma <G>               With --promote: support needed to promote; the exact sign-flip test over tasks must also have p at most 1 − G (default: 0.7)
      --eps-q <Q>               With --promote: pass rate a cheaper B may give up (default: 0.1)
      --eps-c <C|inf>           With --promote: relative cost rise a better B may take (0.2 = +20%) (default: inf)
      --draws <N>               With --promote: bootstrap draws (default: 10000)
      --item-gate               One-item gate (shown automatically when B adds or removes one rule, skill or context file): exit 0 on KEEP or ACCEPT, 1 otherwise; --no-item-gate hides it
      --audited                 Count passes a witness refuted (eval audit) as fails in the verdict; needs every pass in both arms audited
      --signals                 Check each failure flag and tool errors against the outcome: pooled, within-task and within-task-and-arm AUROC beside difficulty and length references (needs k >= 2)
      --resume [<ID>]           Continue an interrupted comparison: finished trials are kept, only the rest run (no ID: the newest unfinished one with the same config)
      --retries <N>             Retries per trial after a rate limit, 5xx or network error, with backoff; a failed setup step is retried once and a refused login never (failures are never retried) (default: 3)
      --stop-after-infra <N>    Stop once N trials in a row fail on infrastructure after their retries (a refused login, a failed setup, rate limits, 5xx, network): no new trial starts and the run is no quality verdict; 0 never stops (default: 5)
      --trial-timeout <SEC>     Time limit per trial, retries included; a trial over it fails
  -y, --yes                     Answer yes to confirmation prompts. An agent passes it only once the user approves

Examples
  coding.sh compare base candidate --k 3
  coding.sh compare base candidate --k 3 --resume
  coding.sh compare base candidate2 --k 3 --reuse baseline --dry-run