GitHub
Changelog

Nov 8, 2026 · 177 changes · page 4 of 5

coding.sh v0.1.2

coding.sh runs on Windows, and CI tests it there.

Experiments 23

  • # Added exp replay <call> --vary model=a,b --vary system=@file.md --n 5 resends a logged model call with your own API key, grades whether each reply takes the agent's first action (or with a judge), and reports the paired comparison of exp compare. --dry-run prints the cost estimate. Experiments.
  • # Added compare records the agent release each run started on (agentVersion, checked at most once a minute, so an auto-update mid-run shows). When the arms ran on different releases, the report adds agent claude-code 2.1.316 ×12 2.1.319 ×12 release differs and a note that the delta mixes the change with the release, --gate repeats it, and --promote holds with release_confound. --json carries report.agentVersions (arXiv 2610.37332). Evals guide.
  • # Added exp compare and compare give the exact McNemar power next to the normal MDE: power MDE 15.6pp · no 80% power at this n (exact McNemar ceiling 62%) or · exact MDE 10.6pp. The exact test needs 6 flipped cases to reject at α 0.05, so a small suite can have no 80% power for any gap; a no-power warning says so, and --json carries power.exact (arXiv 2610.04433). Experiments.
  • # Added exp run --series NAME tags each round of an iteration loop and prints whether to stop. exp plateau runs a restart e-detector over the series and names the version to ship. It exits 3 to stop, 0 to continue and 2 with too few experiments (Yin et al., arXiv 2610.04756). Experiments.
  • # Added exp metrics A/Bs two commands that print numbers (ms, bytes) in interleaved pairs, with a 95% paired bootstrap CI and a sign test per metric. --fail-on-worse exits 1 for CI. Experiments.
  • # Added compare notes the runs whose model calls another model answered: after a refusal Claude Code retries on an older model, so a run can measure a model its arm did not ask for. The note counts those runs per arm and names the model; --json carries report.modelMix. Evals guide.
  • # Added exp compare --gate gives the same ship-or-hold answer per case and exits 1 unless it is "Ship it". exp sweep ends with adopt <winner> or keep <start> and the reason. The MCP experiment_compare tool returns the gate too. With 2+ trials, exp compare also lists the cases B gained and lost (passed in any trial), counts cases that got more or less reliable, and warns when the two halves of the trials disagree; --matrix prints the success counts (EvoAudit, arXiv 2611.07624). Experiments.
  • # Added Long runs survive interruptions. exp run, exp aa, exp sweep, compare, bench run and workflows run save each trial as it finishes, and --resume [<id>] re-runs only the unfinished ones, with the same seed, order and --stop-early decision as an uninterrupted run. A changed config refuses to resume and names what changed. The first Ctrl-C lets trials in flight finish. Rate limits, 5xx and network errors are retried with backoff (--retries, default 3), task failures never are. --trial-timeout caps a trial, and exp status shows progress, ETA and cost so far. Evals guide.
  • # Added coding.sh exp ab tests an AGENTS.md, rule or skill edit on your own recent sessions, with no eval tasks to write. It finds the moments your agent went wrong (said done without running anything, stopped with tests failing, retried a failed command unchanged), replays the next turn there with your committed files and with your edit, and has a judge score both. It prints the fixed and broken cases, the same ship gate as compare --gate, and the replay noise floor from the control arm. --dry-run shows the cost first, --max-cost caps it, reruns are free from the cache, and --watch reruns on every save. CI runs it on real Claude Code sessions against a stub model. Test an AGENTS.md change in five minutes.
  • # Added The pieces under exp ab work on their own: a continuation task replays a frozen session's next turn with given context files, and a decision-point miner turns sessions into a dataset of those moments, each with its own rubric. A continuation experiment needs a control arm: run it as an A/B, or pass --no-control and get a warning that there is no noise floor to read the difference against. Evals guide.
  • # Added exp compare --margin 0.03 checks that B is at most 3pp worse than A (paired non-inferiority on the 95% interval's lower bound) and exits 1 otherwise: non-inf B − A ≥ -3.0pp: yes (95% CI lower bound -1.2pp > -3.0pp). It lists cases A passed on every trial that B now mostly fails. Experiments.
  • # Added exp run and toolevals run take --samples adaptive: 2 trials per case, a 3rd when they disagree, and a case fails at half or more failing; the report gives case verdicts and no CI. --stop-on-regression --baseline <experiment> stops the run and exits 1 once a case that passed every baseline trial fails. Gating runs.
  • # Added exp regrade <experiment> spec.json grades a stored experiment's outputs again with new graders, without calling the task, and prints the pass rate before and after, each grader's version change and every run that flipped (flips 1 of 4 runs: 1 pass → fail). exp compare pins grader versions: a grader that changed version or graded one side only gets a warning above the verdict, and cases left ungraded on one side are listed. Evals.
  • # Added exp run matrix.json runs a spec with "matrix" (JSON Pointer axes, every combination) or "variants" (named cells plus "vary") against a baseline cell: the cells run interleaved, each is stored as its own experiment with metadata.matrix, each is compared with the baseline, and the McNemar p-values are Holm-adjusted over the cells. A cell that differs from the baseline outside vary, or touches graders or the dataset, is refused before any call; --plan prints the cells and runs nothing, --cell runs a subset, --resume continues an interrupted matrix. Evals.
  • # Added compare A B --reuse baseline copies A's finished runs from earlier comparisons instead of running them again (--reuse both copies B's too), so several candidates against one baseline pay for it once. A run is reused only when its task, verifier and hidden files, bundle, agent version, attempt, timeout and route hash the same, it finished without an error, and it is newer than --reuse-max-age (default 7d). --dry-run prints how many runs each arm would reuse and run, and the report notes that reused runs did not run interleaved. Evals.
  • # Added exp run matrix.json --reuse baseline copies the baseline cell's finished case-trials from earlier experiments instead of running them again, so several matrices against one baseline pay for it once (--reuse all copies any cell's). A case-trial is copied only when its case, task spec and files, graders, trial number and timeout all match, within --reuse-max-age (default 7d). --plan prints how many each cell would copy, and the comparisons warn that the copies did not run interleaved. Evals guide.
  • # Added exp ab and refine run on a Claude or ChatGPT subscription with --provider claude-code (claude -p) or --provider codex (codex exec), and pick one on their own when no API key is set and the CLI is logged in, saying so in one line. Each call runs without your hooks, plugins, MCP servers or settings, in a throwaway config dir; with claude -p the model gets the session's own Claude Code tools and a hook of ours refuses every call, so none runs. Calls run 2 at a time, rate limits back off and retry, and a spent usage limit stops the run with the finished trials kept. Costs show as API-equivalent, with no cap unless you pass --max-cost; improve and improve fix suggest the test on a subscription too. Replaying on a subscription.
  • # Added exp ab with no uncommitted harness edits tests your last commit that changed AGENTS.md, a rule or a skill against the version before it, --dry-run included, instead of stopping with "No uncommitted changes". Evals.
  • # Changed The item gate no longer accepts removing a skill or rule when the agent fails the exposed tasks without it; such items read UNCOVERED (neither arm solves them) or UNMEASURED (pass rate without the item under 0.8). The gate line shows U= and A= and a count of internalized, works, uncovered, hurts and between tasks (Wang et al., arXiv 2610.32322). Evals guide.
  • # Changed exp run, exp aa and exp regrade add each --grader to the spec file's graders instead of replacing them, so exp run spec.json -g judge.json grades with both; --replace-graders keeps the old behavior. Evals
  • # Fixed exp ab with no ANTHROPIC_API_KEY or OPENAI_API_KEY stops before the first trial with missing_key and points at coding.sh improve check, instead of running every case into an error and reporting "Hold: not enough evidence". refine propose without a key names the proposer, not the judge. Improve your agent every week.
  • # Fixed When no attempt in either arm solves a task, compare suggests coding.sh eval validate <task> instead of an eval curate command that does not exist.
  • # Fixed A grader error leaves a run ungraded only when no other grader failed it. With exact and a judge that is down, the runs exact failed count as failures and only the runs it passed are left out, instead of every run reading ungraded and the pass rate going blank. exp regrade follows the same rule. Evals

Gateways 1

  • # Added OpenRouter, LiteLLM, Cloudflare AI Gateway, Portkey, Helicone and Ramp Router (router.com) join Vercel: sessions get a gateway:<name> tag, gateway enrich fetches billed cost where the gateway has a lookup API, gateway status shows one row per gateway, and compare --gateway <name> routes runs through it. AI gateways.

SDKs 1

  • # Added The TypeScript and Python SDKs detect all twelve gateways, not only Vercel: by base URL, a self-hosted gateway's URL env var, a response header (cf-aig-log-id, helicone-id, x-litellm-call-id, x-portkey-trace-id, …) or the response id. Each model call records the gateway and its request id, so SDK sessions get the gateway:<name> tag and gateway enrich can fetch their billed cost. The SDK tables are generated from the server's gateway list. AI gateways.

Durable workflows 1

  • # Added A Vercel Workflow, Temporal, Inngest, Restate, DBOS, LangGraph or Cloudflare Workflows run (Temporal, Restate and DBOS in both TypeScript and Python) is one session across processes, traces, crashes and deploys (keyed by run id). show, workflows show and the viewer list each step with its attempts, failed and lost ones included, show sleeps and LangGraph interrupts as waiting rather than work, mark where the run resumed after a crash, deploy or restart, and leave replayed model calls out of tokens. An attempt lost with its process counts as busy only while its process was still seen; workflows stats shows the rest as "lost to crashes" instead of agent time. A span with workflow.wait.name around a sleep is a wait in any engine, for those that export none (Temporal, Inngest, DBOS TS). New flags step-retried and resumed. Each engine is checked in CI against a real run with a failed step, a killed worker, a sleep and a restart. Trace a durable workflow.

Viewer 8

  • # Added Cost shows in the session list, the session header and on each model call. Estimates read ~$0.04 (hover: "Estimated from list prices", and which price a call used); billed and reported costs have no mark; a call with no price shows — and says why.
  • # Added The session list has one filter bar for the --where language: it completes keys and values (with session counts), runs as you type, and on Enter says what is wrong with a bad query and where. Clauses show as removable chips. The sidebar counts sessions per agent, model, repository, branch, flag, tool, MCP server and event kind; a click adds a value, Alt-click excludes it, and groups fold shut. The query is in the URL, and old ?agent= and ?q= links still open. Full-text hits moved out of the list; the list matches sessions by their text. Explore
  • # Added An Events tab next to Sessions lists the events that match the query, newest first, with the session each belongs to and the matched words marked. Enter opens the event's step in its session, and n and N step through that session's other matches. GET /api/events takes scope=top. Explore
  • # Fixed API sessions with a failed request are red in the session list again, with the reason: 1 failed request (429), the first error on hover. Sessions carry stats.llmErrors, counted at ingest (existing indexes are filled in on upgrade), and a list page costs the same store queries at any length.
  • # Fixed The top bar fits from 320px up, in any font (on Linux without Geist it ran 2px past a 320px screen; a tab label now ends in an ellipsis before the page scrolls sideways): between 641 and 820px search folds to its icon instead of squeezing the theme button, and every top bar control has a 44px hit area.
  • # Fixed The session title in the list keeps at least 12rem on any desktop width (at 1024px, next to the filter sidebar, it was 20px): tokens, then duration, then repo, then activity give way first.
  • # Fixed A big ingest no longer blanks reads: coding.sh ui opens at once (0.3 s, was 41 s on a 25k-line spool) and indexes in a background worker with an "Indexing N files…" pill, and status, sessions and search answer within 1.5 s, noting what is still pending (--fresh waits). CLI reference.
  • # Fixed The top bar fits a 360px phone.

Cost 5

  • # Added Estimates price each model call at the rates it ran at: long-context rates past the model's threshold (OpenAI over 272K, Gemini Pro and Claude Sonnet 4.5 1M over 200K, Haiku 5.5 over 100K), and batch, flex, priority, fast or Claude fast-mode rates from the call's recorded service tier. The call's payload.priceTier says which, and shows in show and the viewer's cost tooltip. usage --json adds estimatedCostByTier. [prices] takes long_context and tier tables. Claude 1-hour cache writes are priced at 2x input instead of 1.25x, from Claude Code's cache_creation.ephemeral_1h_input_tokens and OpenTelemetry cache_creation_1h_tokens, and Claude 4.6+ calls with inference_geo: "us" at 1.1x ([prices] takes cache_write_1h and a geo table). Stored estimates reprice by themselves. Where cost comes from.
  • # Added Two kinds of waste per session. Re-fetches are read-only calls (Read, Grep, cat, git diff, …) that returned the same result as an earlier call in the same context; three or more add the duplicate-fetch flag (arXiv 2611.07238). Cache breaks are model calls that found the prompt cache gone and paid for the prompt again, by cause (compaction, model switch, idle, other), priced as cache write minus cache read (arXiv 2610.35496). show prints Cache breaks 2: 31k prompt tokens billed again, $0.1086 extra (other 1, compaction 1), usage totals both per group, and the viewer header, --json and MCP carry stats.waste and stats.cacheBreaks. API reference.
  • # Added coding.sh usage marks a cost that counts only some of a row's sessions as a lower bound (≥$1.50, replacing the + suffix), prints how many sessions have tokens but no cost, and names up to five unpriced models with the [prices."<model>"] fix. --json is unchanged. Find expensive sessions.
  • # Fixed usage TOK/TURN (tokensPerTurn) is input, output and cache-write tokens per user turn. Cache reads are left out, because each model call re-reads the whole context, and TURNS counts only top-level sessions' turns, because a subagent's prompt comes from its parent's turn. Expensive sessions.
  • # Fixed coding.sh collector prices model calls with tokens and no cost, as serve and ingest do, using your [prices].

Workflows 2

  • # Added workflows compare gives each variant P(front), the share of bootstrap draws in which no other variant beats it on quality and cost (front 0.93, dominated by base (front 0.41)). Near-equal noisy variants no longer get a confident "dominated". --select min-cost --q-drop 10, --select weighted --lambda 2 or --select pareto picks a variant to deploy and prints deploy: cand (…) with the reason. --seed fixes the bootstrap (ParetoSaddler, arXiv 2611.02079). Workflows guide
  • # Added workflows compare checks replay experiments for a control fork. When the candidate resumes from recorded runs (resumeFromRunId) and no baseline run resumes from the same ones, it notes "no control fork", since the difference would mix the edit with the replay itself. With a control, it reports how often the control's outcome changed from its source run as the replay noise floor (control.drift in --json). Workflows guide.

Production 1

  • # Added coding.sh-store-clickhouse runs serve, the CLI and MCP on ClickHouse: --store 'clickhouse://user:pass@host:8123/db' (clickhouses:// for TLS). It has no runtime dependencies and passes the store conformance suite; blobs stay on file:// or S3. The same page shows how to send sessions to an OpenTelemetry Collector that writes to ClickHouse, with SQL for cost, slow tools and failed sessions. Self-host for a team.

From CHANGELOG.md · RSS · All releases