Nov 7, 2026 · 38 changes

coding.sh v0.1.1

Daily benchmarks for any AI agent: coding.sh bench runs every contender on fresh cases each day, and experiments add cluster bootstrap, an A/A noise floor, interleaved runs, early stopping and compare --without. Seven more agents are verified on a real install, the viewer and the site work on phones, and the installer uses npm once the package is there.

Benchmarks 6

# Added coding.sh bench samples fresh cases each day from a command, failed sessions, GitHub issues, a sitemap or a dataset, freezes them as a dataset version and runs every contender on that set in random order. Benchmarks.
# Added Shared timeout and retries, a budget cap, quota and rate-limit cases dropped for every contender, and bench run --resume to finish a stopped run with the same seed and cases.
# Added bench report writes a static HTML page and JSON: trend, gaps to a reference pooled over a fixed window, head-to-head stats, difficulty and failures. bench regrade re-scores stored outputs and bench schedule prints a cron line and a GitHub Actions workflow.
# Added An [anchor] set of fixed cases runs every day next to the fresh ones and is scored apart, so its trend has no day-to-day difficulty in it.
# Added seed = "secret" publishes the hash of each next day's salt and bench show checks the chain. Content hashes keep the same case under a new key from counting as fresh.
# Added Input and output tokens are kept apart, and each cost says where it came from: reported, costPerRun, the shared price table, or unknown.

MCP 1

# Added A benchmarks tool lists benchmarks and reads results and history.

Experiments 8

# Added exp compare --cluster <key> resamples by repo or session and warns under 10 clusters. Evals.
# Added exp aa measures the run-to-run noise floor, and power planning uses it.
# Added Compare flags uneven attrition between arms (Fisher exact) and splits infra errors from agent errors.
# Added Under 100 paired cases, compare also reports P(B>A) from a paired posterior.
# Added exp run --arm a=<task> --arm b=<task> interleaves both arms case by case in a seeded random order.
# Added --stop-early checks at planned looks with O'Brien-Fleming alpha spending and reports a repeated confidence interval.
# Added Pairwise judges see both outputs in both orders. exp compare --pairwise reports win rate, ties and position bias.
# Added exp sweep races task configs on a train split and reports the winner's optimism gap on held-out cases.

Compare 3

# Added compare <bundle> --without <path> measures what one skill, AGENTS.md or MCP config is worth. Compare agents.
# Added With 2+ attempts per task, failed runs split into bad runs, arm limits and tasks no arm solves, with variance shares.
# Added Cost rows show how many runs had a known cost, and "cheaper" needs a known cost for 80% of each arm.

Agents 2

# Added Verified on a real install against a local stub model: opencode, copilot-cli, droid, goose, pi, t3code and aider. Agent matrix.
# Added Antigravity backfill reads agy transcripts and conversation DBs for model, tokens, cache reads and latency per call. Tool steps missing from the transcript count as failures, and hooks reply {}.

Viewer 2

# Added Phone layouts: 44px targets, filters in a bottom sheet, a compact session header and safe-area insets.
# Added The Benchmarks tab shows anchors, pooled gaps, difficulty, cost sources and tokens, and ranks follow the pooled gaps.

Site 2

# Added Mobile pass: touch targets, safe areas, stacked agent cards and 16px body text on phones.
# Added A new landing page, a changelog page and docs pages.

Install 1

# Added install.sh installs from npm when the registry has the version, else from the GitHub release. CODING_SH_SOURCE=npm|github picks one.

Release 2

# Added A tag creates its GitHub release as soon as the checks pass, so the installer gets it right away.
# Added The release workflow can publish all five packages to npm and PyPI with trusted publishing. Nothing is on npm or PyPI yet.

Docs 1

# Added A new README, reference pages, how-to guides and a code of conduct.

Fixes 10

# Fixed Safer installs: symlinked and JSONC agent configs survive, init --dry-run prints a short plan, and blockers show before the Y/n.
# Fixed Each SQLite migration runs once when the hook and the CLI open a new index at the same time.
# Fixed Experiments stop cleanly on Ctrl-C, kill their child processes and keep every finished run.
# Fixed Capture, ingest and storage state bugs from an audit of each state machine. Sessions are priced locally at ingest.
# Fixed Adapter fixes for opencode, goose, copilot-cli, droid, aider and t3code, found running each on a real install.
# Fixed coding.sh run --agent <name> records sessions for names with no adapter.
# Fixed exp aa refuses to compare a run with itself.
# Fixed coding.sh ui prints the viewer URL once in a terminal.
# Fixed Muted text in the viewer passes WCAG AA in both themes.
# Fixed Timing tests compare against a baseline, so a loaded machine no longer fails them.

From CHANGELOG.md · RSS · All releases