--- title: "coding.sh v0.1.1 · Changelog" description: "Daily benchmarks for any AI agent: coding.sh bench runs every contender on fresh cases each day, and experiments add cluster bootstrap, an A/A noise…" canonical: https://coding.sh/changelog/v0-1-1 human_url: https://coding.sh/changelog/v0-1-1 markdown_url: https://coding.sh/changelog/v0-1-1.md agents_url: https://coding.sh/agents.md llms_url: https://coding.sh/llms.txt updated: 2026-11-11 --- [Changelog](https://coding.sh/changelog) Nov 7, 2026 · 38 changes # coding.sh v0.1.1 Daily benchmarks for any AI agent: `coding.sh bench` runs every contender on fresh cases each day, and experiments add cluster bootstrap, an A/A noise floor, interleaved runs, early stopping and `compare --without`. Seven more agents are verified on a real install, the viewer and the site work on phones, and the installer uses npm once the package is there. ## Benchmarks - Added `coding.sh bench` samples fresh cases each day from a command, failed sessions, GitHub issues, a sitemap or a dataset, freezes them as a dataset version and runs every contender on that set in random order. [Benchmarks](https://github.com/jackulau/coding.sh/blob/main/docs/guides/benchmarks.md). - Added Shared timeout and retries, a budget cap, quota and rate-limit cases dropped for every contender, and `bench run --resume` to finish a stopped run with the same seed and cases. - Added `bench report` writes a static HTML page and JSON: trend, gaps to a reference pooled over a fixed window, head-to-head stats, difficulty and failures. `bench regrade` re-scores stored outputs and `bench schedule` prints a cron line and a GitHub Actions workflow. - Added An `[anchor]` set of fixed cases runs every day next to the fresh ones and is scored apart, so its trend has no day-to-day difficulty in it. - Added `seed = "secret"` publishes the hash of each next day's salt and `bench show` checks the chain. Content hashes keep the same case under a new key from counting as fresh. - Added Input and output tokens are kept apart, and each cost says where it came from: reported, `costPerRun`, the shared price table, or unknown. ## MCP - Added A `benchmarks` tool lists benchmarks and reads results and history. ## Experiments - Added `exp compare --cluster ` resamples by repo or session and warns under 10 clusters. [Evals](https://coding.sh/docs/evals). - Added `exp aa` measures the run-to-run noise floor, and power planning uses it. - Added Compare flags uneven attrition between arms (Fisher exact) and splits infra errors from agent errors. - Added Under 100 paired cases, compare also reports P(B>A) from a paired posterior. - Added `exp run --arm a= --arm b=` interleaves both arms case by case in a seeded random order. - Added `--stop-early` checks at planned looks with O'Brien-Fleming alpha spending and reports a repeated confidence interval. - Added Pairwise judges see both outputs in both orders. `exp compare --pairwise` reports win rate, ties and position bias. - Added `exp sweep` races task configs on a train split and reports the winner's optimism gap on held-out cases. ## Compare - Added `compare --without ` measures what one skill, AGENTS.md or MCP config is worth. [Compare agents](https://coding.sh/docs/how-to/compare-agents). - Added With 2+ attempts per task, failed runs split into bad runs, arm limits and tasks no arm solves, with variance shares. - Added Cost rows show how many runs had a known cost, and "cheaper" needs a known cost for 80% of each arm. ## Agents - Added Verified on a real install against a local stub model: opencode, copilot-cli, droid, goose, pi, t3code and aider. [Agent matrix](https://coding.sh/docs/agents). - Added Antigravity backfill reads agy transcripts and conversation DBs for model, tokens, cache reads and latency per call. Tool steps missing from the transcript count as failures, and hooks reply `{}`. ## Viewer - Added Phone layouts: 44px targets, filters in a bottom sheet, a compact session header and safe-area insets. - Added The Benchmarks tab shows anchors, pooled gaps, difficulty, cost sources and tokens, and ranks follow the pooled gaps. ## Site - Added Mobile pass: touch targets, safe areas, stacked agent cards and 16px body text on phones. - Added A new landing page, a changelog page and docs pages. ## Install - Added `install.sh` installs from npm when the registry has the version, else from the GitHub release. `CODING_SH_SOURCE=npm|github` picks one. ## Release - Added A tag creates its GitHub release as soon as the checks pass, so the installer gets it right away. - Added The release workflow can publish all five packages to npm and PyPI with trusted publishing. Nothing is on npm or PyPI yet. ## Docs - Added A new README, reference pages, how-to guides and a code of conduct. ## Fixes - Fixed Safer installs: symlinked and JSONC agent configs survive, `init --dry-run` prints a short plan, and blockers show before the Y/n. - Fixed Each SQLite migration runs once when the hook and the CLI open a new index at the same time. - Fixed Experiments stop cleanly on Ctrl-C, kill their child processes and keep every finished run. - Fixed Capture, ingest and storage state bugs from an audit of each state machine. Sessions are priced locally at ingest. - Fixed Adapter fixes for opencode, goose, copilot-cli, droid, aider and t3code, found running each on a real install. - Fixed `coding.sh run --agent ` records sessions for names with no adapter. - Fixed `exp aa` refuses to compare a run with itself. - Fixed `coding.sh ui` prints the viewer URL once in a terminal. - Fixed Muted text in the viewer passes WCAG AA in both themes. - Fixed Timing tests compare against a baseline, so a loaded machine no longer fails them. [Newer release v0.1.2](https://coding.sh/changelog/v0-1-2) [Older release v0.1.0](https://coding.sh/changelog/v0-1-0) From [CHANGELOG.md](https://github.com/jackulau/coding.sh/blob/main/CHANGELOG.md) · [RSS](https://coding.sh/changelog.xml) · [All releases](https://coding.sh/changelog) To set coding.sh up for a user, follow the agent workflow at [/agents.md](https://coding.sh/agents.md).