---
title: "coding.sh v0.1.2 (page 4) · Changelog"
description: "2, page 4 of 5: every change to Experiments, Gateways, SDKs, Durable workflows, Viewer, Cost, Workflows and Production, one line each."
canonical: https://coding.sh/changelog/v0-1-2/4
human_url: https://coding.sh/changelog/v0-1-2/4
markdown_url: https://coding.sh/changelog/v0-1-2/4.md
agents_url: https://coding.sh/agents.md
llms_url: https://coding.sh/llms.txt
updated: 2026-11-11
---

[Changelog](https://coding.sh/changelog)

Nov 8, 2026 · 177 changes · page 4 of 5

# coding.sh v0.1.2

coding.sh runs on Windows, and CI tests it there.

## Experiments

- Added `exp replay <call> --vary model=a,b --vary system=@file.md --n 5` resends a logged model call with your own API key, grades whether each reply takes the agent's first action (or with a judge), and reports the paired comparison of `exp compare`. `--dry-run` prints the cost estimate. [Experiments](https://coding.sh/docs/experiments#exp-replay).
- Added `compare` records the agent release each run started on (`agentVersion`, checked at most once a minute, so an auto-update mid-run shows). When the arms ran on different releases, the report adds `agent claude-code 2.1.316 ×12 2.1.319 ×12 release differs` and a note that the delta mixes the change with the release, `--gate` repeats it, and `--promote` holds with `release_confound`. `--json` carries `report.agentVersions` (arXiv 2610.37332). [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#agent-release).
- Added `exp compare` and `compare` give the exact McNemar power next to the normal MDE: `power MDE 15.6pp · no 80% power at this n (exact McNemar ceiling 62%)` or `· exact MDE 10.6pp`. The exact test needs 6 flipped cases to reject at α 0.05, so a small suite can have no 80% power for any gap; a `no-power` warning says so, and `--json` carries `power.exact` (arXiv 2610.04433). [Experiments](https://coding.sh/docs/experiments#compare).
- Added `exp run --series NAME` tags each round of an iteration loop and prints whether to stop. `exp plateau` runs a restart e-detector over the series and names the version to ship. It exits 3 to stop, 0 to continue and 2 with too few experiments (Yin et al., arXiv 2610.04756). [Experiments](https://coding.sh/docs/experiments#plateau).
- Added `exp metrics` A/Bs two commands that print numbers (ms, bytes) in interleaved pairs, with a 95% paired bootstrap CI and a sign test per metric. `--fail-on-worse` exits 1 for CI. [Experiments](https://coding.sh/docs/experiments#metrics).
- Added `compare` notes the runs whose model calls another model answered: after a refusal Claude Code retries on an older model, so a run can measure a model its arm did not ask for. The note counts those runs per arm and names the model; `--json` carries `report.modelMix`. [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#agent-release).
- Added `exp compare --gate` gives the same ship-or-hold answer per case and exits 1 unless it is "Ship it". `exp sweep` ends with `adopt <winner>` or `keep <start>` and the reason. The MCP `experiment_compare` tool returns the gate too. With 2+ trials, `exp compare` also lists the cases B gained and lost (passed in any trial), counts cases that got more or less reliable, and warns when the two halves of the trials disagree; `--matrix` prints the success counts (EvoAudit, arXiv 2611.07624). [Experiments](https://coding.sh/docs/experiments#compare).
- Added Long runs survive interruptions. `exp run`, `exp aa`, `exp sweep`, `compare`, `bench run` and `workflows run` save each trial as it finishes, and `--resume [<id>]` re-runs only the unfinished ones, with the same seed, order and `--stop-early` decision as an uninterrupted run. A changed config refuses to resume and names what changed. The first Ctrl-C lets trials in flight finish. Rate limits, 5xx and network errors are retried with backoff (`--retries`, default 3), task failures never are. `--trial-timeout` caps a trial, and `exp status` shows progress, ETA and cost so far. [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#long-runs-resume-retries-status).
- Added `coding.sh exp ab` tests an AGENTS.md, rule or skill edit on your own recent sessions, with no eval tasks to write. It finds the moments your agent went wrong (said done without running anything, stopped with tests failing, retried a failed command unchanged), replays the next turn there with your committed files and with your edit, and has a judge score both. It prints the fixed and broken cases, the same ship gate as `compare --gate`, and the replay noise floor from the control arm. `--dry-run` shows the cost first, `--max-cost` caps it, reruns are free from the cache, and `--watch` reruns on every save. CI runs it on real Claude Code sessions against a stub model. [Test an AGENTS.md change in five minutes](https://coding.sh/docs/how-to/test-agents-md-change).
- Added The pieces under `exp ab` work on their own: a `continuation` task replays a frozen session's next turn with given context files, and a decision-point miner turns sessions into a dataset of those moments, each with its own rubric. A continuation experiment needs a control arm: run it as an A/B, or pass `--no-control` and get a warning that there is no noise floor to read the difference against. [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#ab-a-harness-edit-on-your-own-sessions).
- Added `exp compare --margin 0.03` checks that B is at most 3pp worse than A (paired non-inferiority on the 95% interval's lower bound) and exits 1 otherwise: `non-inf B − A ≥ -3.0pp: yes (95% CI lower bound -1.2pp > -3.0pp)`. It lists cases A passed on every trial that B now mostly fails. [Experiments](https://coding.sh/docs/experiments#non-inferiority).
- Added `exp run` and `toolevals run` take `--samples adaptive`: 2 trials per case, a 3rd when they disagree, and a case fails at half or more failing; the report gives case verdicts and no CI. `--stop-on-regression --baseline <experiment>` stops the run and exits 1 once a case that passed every baseline trial fails. [Gating runs](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#gating-runs-tie-breaks-and-the-regression-stop).
- Added `exp regrade <experiment> spec.json` grades a stored experiment's outputs again with new graders, without calling the task, and prints the pass rate before and after, each grader's version change and every run that flipped (`flips 1 of 4 runs: 1 pass → fail`). `exp compare` pins grader versions: a grader that changed version or graded one side only gets a warning above the verdict, and cases left ungraded on one side are listed. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#regrade).
- Added `exp run matrix.json` runs a spec with `"matrix"` (JSON Pointer axes, every combination) or `"variants"` (named cells plus `"vary"`) against a baseline cell: the cells run interleaved, each is stored as its own experiment with `metadata.matrix`, each is compared with the baseline, and the McNemar p-values are Holm-adjusted over the cells. A cell that differs from the baseline outside `vary`, or touches graders or the dataset, is refused before any call; `--plan` prints the cells and runs nothing, `--cell` runs a subset, `--resume` continues an interrupted matrix. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#matrix).
- Added `compare A B --reuse baseline` copies A's finished runs from earlier comparisons instead of running them again (`--reuse both` copies B's too), so several candidates against one baseline pay for it once. A run is reused only when its task, verifier and hidden files, bundle, agent version, attempt, timeout and route hash the same, it finished without an error, and it is newer than `--reuse-max-age` (default 7d). `--dry-run` prints how many runs each arm would reuse and run, and the report notes that reused runs did not run interleaved. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#reuse-finished-runs).
- Added `exp run matrix.json --reuse baseline` copies the baseline cell's finished case-trials from earlier experiments instead of running them again, so several matrices against one baseline pay for it once (`--reuse all` copies any cell's). A case-trial is copied only when its case, task spec and files, graders, trial number and timeout all match, within `--reuse-max-age` (default 7d). `--plan` prints how many each cell would copy, and the comparisons warn that the copies did not run interleaved. [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#matrix).
- Added `exp ab` and `refine` run on a Claude or ChatGPT subscription with `--provider claude-code` (`claude -p`) or `--provider codex` (`codex exec`), and pick one on their own when no API key is set and the CLI is logged in, saying so in one line. Each call runs without your hooks, plugins, MCP servers or settings, in a throwaway config dir; with `claude -p` the model gets the session's own Claude Code tools and a hook of ours refuses every call, so none runs. Calls run 2 at a time, rate limits back off and retry, and a spent usage limit stops the run with the finished trials kept. Costs show as API-equivalent, with no cap unless you pass `--max-cost`; `improve` and `improve fix` suggest the test on a subscription too. [Replaying on a subscription](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#replaying-on-a-subscription).
- Added `exp ab` with no uncommitted harness edits tests your last commit that changed AGENTS.md, a rule or a skill against the version before it, `--dry-run` included, instead of stopping with "No uncommitted changes". [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#ab-a-harness-edit-on-your-own-sessions).
- Changed The item gate no longer accepts removing a skill or rule when the agent fails the exposed tasks without it; such items read UNCOVERED (neither arm solves them) or UNMEASURED (pass rate without the item under 0.8). The gate line shows `U=` and `A=` and a count of internalized, works, uncovered, hurts and between tasks (Wang et al., arXiv 2610.32322). [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#harness-ab).
- Changed `exp run`, `exp aa` and `exp regrade` add each `--grader` to the spec file's graders instead of replacing them, so `exp run spec.json -g judge.json` grades with both; `--replace-graders` keeps the old behavior. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#datasets-and-experiments)
- Fixed `exp ab` with no `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` stops before the first trial with `missing_key` and points at `coding.sh improve check`, instead of running every case into an error and reporting "Hold: not enough evidence". `refine propose` without a key names the proposer, not the judge. [Improve your agent every week](https://github.com/jackulau/coding.sh/blob/main/docs/guides/improve.md#3-test-the-edit-before-you-commit-it-optional).
- Fixed When no attempt in either arm solves a task, `compare` suggests `coding.sh eval validate <task>` instead of an `eval curate` command that does not exist.
- Fixed A grader error leaves a run ungraded only when no other grader failed it. With `exact` and a judge that is down, the runs `exact` failed count as failures and only the runs it passed are left out, instead of every run reading ungraded and the pass rate going blank. `exp regrade` follows the same rule. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md)

## Gateways

- Added OpenRouter, LiteLLM, Cloudflare AI Gateway, Portkey, Helicone and Ramp Router (router.com) join Vercel: sessions get a `gateway:<name>` tag, `gateway enrich` fetches billed cost where the gateway has a lookup API, `gateway status` shows one row per gateway, and `compare --gateway <name>` routes runs through it. [AI gateways](https://coding.sh/docs/ai-gateway).

## SDKs

- Added The TypeScript and Python SDKs detect all twelve gateways, not only Vercel: by base URL, a self-hosted gateway's URL env var, a response header (`cf-aig-log-id`, `helicone-id`, `x-litellm-call-id`, `x-portkey-trace-id`, …) or the response id. Each model call records the gateway and its request id, so SDK sessions get the `gateway:<name>` tag and `gateway enrich` can fetch their billed cost. The SDK tables are generated from the server's gateway list. [AI gateways](https://coding.sh/docs/ai-gateway#sdk).

## Durable workflows

- Added A Vercel Workflow, Temporal, Inngest, Restate, DBOS, LangGraph or Cloudflare Workflows run (Temporal, Restate and DBOS in both TypeScript and Python) is one session across processes, traces, crashes and deploys (keyed by run id). `show`, `workflows show` and the viewer list each step with its attempts, failed and lost ones included, show sleeps and LangGraph interrupts as waiting rather than work, mark where the run resumed after a crash, deploy or restart, and leave replayed model calls out of tokens. An attempt lost with its process counts as busy only while its process was still seen; `workflows stats` shows the rest as "lost to crashes" instead of agent time. A span with `workflow.wait.name` around a sleep is a wait in any engine, for those that export none (Temporal, Inngest, DBOS TS). New flags `step-retried` and `resumed`. Each engine is checked in CI against a real run with a failed step, a killed worker, a sleep and a restart. [Trace a durable workflow](https://coding.sh/docs/how-to/trace-durable-workflow).

## Viewer

- Added Cost shows in the session list, the session header and on each model call. Estimates read `~$0.04` (hover: "Estimated from list prices", and which price a call used); billed and reported costs have no mark; a call with no price shows `—` and says why.
- Added The session list has one filter bar for the `--where` language: it completes keys and values (with session counts), runs as you type, and on Enter says what is wrong with a bad query and where. Clauses show as removable chips. The sidebar counts sessions per agent, model, repository, branch, flag, tool, MCP server and event kind; a click adds a value, Alt-click excludes it, and groups fold shut. The query is in the URL, and old `?agent=` and `?q=` links still open. Full-text hits moved out of the list; the list matches sessions by their text. [Explore](https://github.com/jackulau/coding.sh/blob/main/docs/guides/explore.md#viewer)
- Added An Events tab next to Sessions lists the events that match the query, newest first, with the session each belongs to and the matched words marked. Enter opens the event's step in its session, and `n` and `N` step through that session's other matches. `GET /api/events` takes `scope=top`. [Explore](https://github.com/jackulau/coding.sh/blob/main/docs/guides/explore.md#viewer)
- Fixed API sessions with a failed request are red in the session list again, with the reason: `1 failed request (429)`, the first error on hover. Sessions carry `stats.llmErrors`, counted at ingest (existing indexes are filled in on upgrade), and a list page costs the same store queries at any length.
- Fixed The top bar fits from 320px up, in any font (on Linux without Geist it ran 2px past a 320px screen; a tab label now ends in an ellipsis before the page scrolls sideways): between 641 and 820px search folds to its icon instead of squeezing the theme button, and every top bar control has a 44px hit area.
- Fixed The session title in the list keeps at least 12rem on any desktop width (at 1024px, next to the filter sidebar, it was 20px): tokens, then duration, then repo, then activity give way first.
- Fixed A big ingest no longer blanks reads: `coding.sh ui` opens at once (0.3 s, was 41 s on a 25k-line spool) and indexes in a background worker with an "Indexing N files…" pill, and `status`, `sessions` and `search` answer within 1.5 s, noting what is still pending (`--fresh` waits). [CLI reference](https://github.com/jackulau/coding.sh/blob/main/docs/reference/cli.md).
- Fixed The top bar fits a 360px phone.

## Cost

- Added Estimates price each model call at the rates it ran at: long-context rates past the model's threshold (OpenAI over 272K, Gemini Pro and Claude Sonnet 4.5 1M over 200K, Haiku 5.5 over 100K), and batch, flex, priority, fast or Claude fast-mode rates from the call's recorded service tier. The call's `payload.priceTier` says which, and shows in `show` and the viewer's cost tooltip. `usage --json` adds `estimatedCostByTier`. `[prices]` takes `long_context` and tier tables. Claude 1-hour cache writes are priced at 2x input instead of 1.25x, from Claude Code's `cache_creation.ephemeral_1h_input_tokens` and OpenTelemetry `cache_creation_1h_tokens`, and Claude 4.6+ calls with `inference_geo: "us"` at 1.1x (`[prices]` takes `cache_write_1h` and a `geo` table). Stored estimates reprice by themselves. [Where cost comes from](https://github.com/jackulau/coding.sh/blob/main/docs/how-to/expensive-sessions.md#where-cost-comes-from).
- Added Two kinds of waste per session. Re-fetches are read-only calls (Read, Grep, `cat`, `git diff`, …) that returned the same result as an earlier call in the same context; three or more add the `duplicate-fetch` flag (arXiv 2611.07238). Cache breaks are model calls that found the prompt cache gone and paid for the prompt again, by cause (compaction, model switch, idle, other), priced as cache write minus cache read (arXiv 2610.35496). `show` prints `Cache breaks 2: 31k prompt tokens billed again, $0.1086 extra (other 1, compaction 1)`, `usage` totals both per group, and the viewer header, `--json` and MCP carry `stats.waste` and `stats.cacheBreaks`. [API reference](https://github.com/jackulau/coding.sh/blob/main/docs/reference/api.md).
- Added `coding.sh usage` marks a cost that counts only some of a row's sessions as a lower bound (`≥$1.50`, replacing the `+` suffix), prints how many sessions have tokens but no cost, and names up to five unpriced models with the `[prices."<model>"]` fix. `--json` is unchanged. [Find expensive sessions](https://github.com/jackulau/coding.sh/blob/main/docs/how-to/expensive-sessions.md).
- Fixed `usage` `TOK/TURN` (`tokensPerTurn`) is input, output and cache-write tokens per user turn. Cache reads are left out, because each model call re-reads the whole context, and `TURNS` counts only top-level sessions' turns, because a subagent's prompt comes from its parent's turn. [Expensive sessions](https://github.com/jackulau/coding.sh/blob/main/docs/how-to/expensive-sessions.md).
- Fixed `coding.sh collector` prices model calls with tokens and no cost, as `serve` and `ingest` do, using your `[prices]`.

## Workflows

- Added `workflows compare` gives each variant P(front), the share of bootstrap draws in which no other variant beats it on quality and cost (`front 0.93`, `dominated by base (front 0.41)`). Near-equal noisy variants no longer get a confident "dominated". `--select min-cost --q-drop 10`, `--select weighted --lambda 2` or `--select pareto` picks a variant to deploy and prints `deploy: cand (…)` with the reason. `--seed` fixes the bootstrap (ParetoSaddler, arXiv 2611.02079). [Workflows guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/workflows.md#pick-a-variant-to-deploy)
- Added `workflows compare` checks replay experiments for a control fork. When the candidate resumes from recorded runs (`resumeFromRunId`) and no baseline run resumes from the same ones, it notes "no control fork", since the difference would mix the edit with the replay itself. With a control, it reports how often the control's outcome changed from its source run as the replay noise floor (`control.drift` in `--json`). [Workflows guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#workflows).

## Production

- Added `coding.sh-store-clickhouse` runs serve, the CLI and MCP on ClickHouse: `--store 'clickhouse://user:pass@host:8123/db'` (`clickhouses://` for TLS). It has no runtime dependencies and passes the store conformance suite; blobs stay on `file://` or S3. The same page shows how to send sessions to an OpenTelemetry Collector that writes to ClickHouse, with SQL for cost, slow tools and failed sessions. [Self-host for a team](https://github.com/jackulau/coding.sh/blob/main/docs/how-to/self-host-team.md#clickhouse).

[Previous page v0.1.2, page 3](https://coding.sh/changelog/v0-1-2/3) [Next page v0.1.2, page 5](https://coding.sh/changelog/v0-1-2/5)

From [CHANGELOG.md](https://github.com/jackulau/coding.sh/blob/main/CHANGELOG.md) · [RSS](https://coding.sh/changelog.xml) · [All releases](https://coding.sh/changelog)

To set coding.sh up for a user, follow the agent workflow at [/agents.md](https://coding.sh/agents.md).