coding.sh bench
Dynamic benchmarks: fresh cases every run, same set for every contender. Every option of coding.sh bench and its subcommands, from its own --help.
Alias: benchmark.
coding.sh bench <action> Dynamic benchmarks: fresh cases every run, same set for every contender Commands new Write a benchmark definition (bench/<name>.toml) run Sample fresh cases, freeze them, run every contender and grade regrade Re-score stored outputs with the current graders (no contender calls) list Benchmarks with their last run and leader show Leaderboard, head-to-head and failures for one run (checks a secret seed's commitment) history One metric per contender over time report Write a self-contained HTML report plus report.json schedule Print a daily cron line and a GitHub Actions workflow (installing needs budget_usd) Notes Definitions live in ./bench/<name>.toml (searched upward from the current directory) or ~/.coding.sh/benchmarks/<name>.toml. Each run is stored in ~/.coding.sh/benchmarks/<name>/runs/, its sample as dataset bench-<name>, and one experiment per contender. History counts the first complete plain `bench run` per date. --seed, --cases, --date and same-day reruns are adhoc. Guide: docs/guides/benchmarks.md Aliases benchmark
coding.sh bench new
Write a benchmark definition (bench/<name>.toml).
coding.sh bench new <NAME> [options]
Write a benchmark definition (bench/<name>.toml)
Arguments
name Benchmark name (lowercase, digits, . _ -)
Options
--source <KIND> Where fresh cases come from (default: command)
--command <CMD> command source: prints JSONL cases (gets CODING_SH_BENCH_SEED and _DATE)
--repo <OWNER/NAME> github source: repository
--sitemap <URL> urls source: sitemap URL (repeatable)
--dataset <NAME> dataset source: dataset to sample from
-c, --contender <NAME=CMD|URL|agent:ID[:MODEL]>
A contender (repeatable, at least two)
-g, --grader <JSON> Grader spec, e.g. {"kind":"exact"} (repeatable)
-n, --cases <N> Cases per run (default: 50)
--global Write to ~/.coding.sh/benchmarks instead of ./bench
--force Overwrite an existing definition
Examples
coding.sh bench new answers --command 'node gen.mjs' -c fast='node fast.mjs' -c big='node big.mjs' -g '{"kind":"exact"}'
coding.sh bench new triage --source github --repo owner/name -c claude=agent:claude-code -c codex=agent:codex
coding.sh bench new fetchers --source urls --sitemap https://example.com/sitemap.xml -c a=https://a.example/v1 -c b=https://b.example/v1 coding.sh bench run
Sample fresh cases, freeze them, run every contender and grade.
coding.sh bench run <NAME> [options]
Sample fresh cases, freeze them, run every contender and grade
Arguments
name Benchmark name, or a path to its .toml
Options
--date <YYYY-MM-DD> Sample as if on this UTC date (adhoc: kept out of history)
--seed <SEED> Fixed seed instead of the definition's (adhoc)
-n, --cases <N> Cases this run (adhoc; default: the definition's)
--adhoc Keep this run out of history and the freshness ledger
--concurrency <N> Parallel calls (default: the definition's)
--dry-run Sample and print the cases, the detectable gap and the cost estimate; run nothing
--resume [<RUN>] Continue an incomplete run (budget stop, Ctrl-C, crash): same seed and cases, finished calls kept. RUN: id or YYYY-MM-DD (default: the latest)
--run <RUN> With --resume: which incomplete run, by id or YYYY-MM-DD (default: the latest)
--fresh Sessions benchmarks: wait for every pending capture to be ingested (default: answer after 1.5 s)
Examples
coding.sh bench run answers
coding.sh bench run answers --dry-run
coding.sh bench run answers --seed try-1 --cases 10 coding.sh bench regrade
Re-score stored outputs with the current graders (no contender calls).
coding.sh bench regrade <NAME> [options] Re-score stored outputs with the current graders (no contender calls) Arguments name Benchmark name, or a path to its .toml Options --run <RUN> A specific run: id or YYYY-MM-DD (default: latest) --all Regrade every counted run Examples coding.sh bench regrade answers coding.sh bench regrade answers --all
coding.sh bench list
Benchmarks with their last run and leader.
coding.sh bench list Benchmarks with their last run and leader
coding.sh bench show
Leaderboard, head-to-head and failures for one run (checks a secret seed's commitment).
coding.sh bench show <NAME> [options] Leaderboard, head-to-head and failures for one run (checks a secret seed's commitment) Arguments name Benchmark name, or a path to its .toml Options --run <RUN> A specific run: id or YYYY-MM-DD (default: latest) --cases Also list the frozen cases
coding.sh bench history
One metric per contender over time.
coding.sh bench history <NAME> [options]
One metric per contender over time
Arguments
name Benchmark name, or a path to its .toml
Options
-m, --metric <METRIC> Metric (default: pass)
-n, --limit <N> Most recent runs (default: 30)
--vs <CONTENDER> Show each contender's paired pass-rate gap to this one, plus the gap pooled over the window
--anchor Numbers from the anchor set only (the same cases every day)
Examples
coding.sh bench history answers
coding.sh bench history answers --metric latency --json
coding.sh bench history answers --vs baseline coding.sh bench report
Write a self-contained HTML report plus report.json.
coding.sh bench report <NAME> [options]
Write a self-contained HTML report plus report.json
Arguments
name Benchmark name, or a path to its .toml
Options
-o, --out <DIR> Output directory (default: ~/.coding.sh/benchmarks/<name>/report)
--days <N> Days of history to include (default: 90)
--viewer-url <URL> Viewer base URL for session links (default: http://127.0.0.1:4319)
--reference <CONTENDER> Contender for the gap chart (default: the definition's reference)
--include-private Show case contents, outputs and session links of a private benchmark (local copies only)
Examples
coding.sh bench report answers --out ./public/bench coding.sh bench schedule
Print a daily cron line and a GitHub Actions workflow (installing needs budget_usd).
coding.sh bench schedule <NAME> [options]
Print a daily cron line and a GitHub Actions workflow (installing needs budget_usd)
Arguments
name Benchmark name, or a path to its .toml
Options
--at <HH:MM> Time of day (cron: local; Actions: UTC) (default: 06:15)
--runner <LABELS> Actions runs-on label(s), comma-separated (default: self-hosted)
--report-dir <DIR> Also write the report here after each run
--install-cron Add the line to your crontab (replaces this benchmark's earlier line only)
--workflow-out <FILE> Write the workflow to this file, e.g. .github/workflows/bench-<name>.yml
-y, --yes Don't ask before editing the crontab
Examples
coding.sh bench schedule answers
coding.sh bench schedule answers --install-cron --yes
coding.sh bench schedule answers --workflow-out .github/workflows/bench-answers.yml