--- title: "coding.sh v0.1.2 (page 3) · Changelog" description: "What changed in coding.sh v0.1.2, page 3 of 5: every change to Harness A/B, Skills, Self-host, Replay, Install and Serve, one line each." canonical: https://coding.sh/changelog/v0-1-2/3 human_url: https://coding.sh/changelog/v0-1-2/3 markdown_url: https://coding.sh/changelog/v0-1-2/3.md agents_url: https://coding.sh/agents.md llms_url: https://coding.sh/llms.txt updated: 2026-11-11 --- [Changelog](https://coding.sh/changelog) Nov 8, 2026 · 177 changes · page 3 of 5 # coding.sh v0.1.2 coding.sh runs on Windows, and CI tests it there. ## Harness A/B - Added Each run keeps the end of its verifier's output (the last 4096 characters of stdout and stderr, colour codes removed, redacted with `capture.redact`), each repair round too. `compare` and `eval run` print the last lines of failed runs under the report, and `--json` lists them in `failedRuns`. [Verifier output](https://coding.sh/docs/evals#verifier-output). - Added Harness bundles carry Claude Code plugins (hooks, skills, agents, commands, MCP servers: the whole plugin directory) with `harness snapshot --plugin ` or `plugins = [...]` in a harness TOML, and `compare A --with plugin:` runs "with the plugin vs without" in one command (`--without plugin:` drops one). Each run loads the plugin with `--plugin-dir` from its own home; your Claude config is never changed. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#claude-code-plugins), [Does this plugin help me?](https://coding.sh/docs/how-to/does-this-plugin-help). - Added `compare --repair N` reports Seq@1..N+1 per arm: the share of tasks solved by the k-th attempt with verifier feedback, with intervals, where the gain lands (EGS, LGS), rounds that changed no file, and the agent's spend per solved run. With `--k 2` or more it adds a cost front of k attempts with feedback against k independent attempts (unbiased pass@k), and a note when the two arms rank one way on the first attempt and the other way after the rounds (Seq@K, arXiv 2610.37859). [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#repair-rounds). - Added `compare --repair N` tells the agent "Verification failed." in its own session (Claude Code `--resume`, `codex exec resume`) after a failed verifier, up to N rounds, with the hidden tests out of its tree. The report adds the repair rate, mean rounds and the rounds' cost per arm. `--repair-signal names` adds the failing test names from the verifier output; a round that edits, deletes or skips tests is not a repair; `--false-alarm P` tells a seeded share of passing runs they failed; `--repair-verdict` decides on the outcome after repair. Repair settings are part of the `--reuse` key (BOBA, arXiv 2611.02207). [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#repair-rounds). - Added `compare --gate` answers "ship B or not": `Ship it: B wins 9 tasks, loses 0`, `Hold: not enough evidence`, or `Don't ship: B breaks 3 tasks A solved` with the broken tasks listed. It exits 0 only for "Ship it" (SAGE, arXiv 2609.36043). `--by suite:` gates each tagged suite too, and one broken suite blocks a pooled "Ship it". Each arm's failed runs are split by the agent's own last test run: `A fails 12: own tests green 7 (58%), red 3, stale 1, none 1`, where green fails are silent misses (BOBA, arXiv 2611.02207). `compare --promote` also keeps a B that is cheaper at the same quality: `pareto: support 0.93 ≥ 0.70 → promote (cost-cut: …)` or `→ hold`, exit 0 only on promote (ParetoSaddler, arXiv 2611.02079). When B adds or removes one rule, skill or context file, the report adds a one-item gate scored only on tasks where the item reached the agent's context: `item gate k=3 τ=.34 m=2: KEEP · …`, or RECHECK, KILL, RERUN, and ACCEPT, REJECT, UNMEASURED for removals; `--item-gate` sets the exit code (Cortex, arXiv 2610.14216). CI checks the gate and the own-tests split on real Claude Code and Codex sessions against a stub model. The MCP `harness_compare` tool lists these A/B results and returns one with its gates and own-tests split. [Compare agents](https://github.com/jackulau/coding.sh/blob/main/docs/how-to/compare-agents.md#ship-it-or-not). - Added `coding.sh refine check` exits 1 on an AGENTS.md, rule or skill edit that deletes or rewords a protected safety line, or adds an exception that relaxes one (``protected-line: AGENTS.md:3 "Never run `git push --force`." (auto)``), since a pass-rate A/B can't see that harm (Masked Erosion, arXiv 2610.38392). A line is protected by a `` block, a `[refine] protect` regex, or automatically when one sentence pairs a gate word with a safety topic; `refine protected` lists them. `exp ab` runs the same check before any model call and fails `--gate` on it. `--allow-protected-edit AGENTS.md:3` allows one on purpose. [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#protected-safety-lines). - Added `compare` links each tamper flag to the tool call that made the change and splits tampered passes by whether the agent said it was changing the test: `tampered passes B 3: disclosed 2, stealth 1`, then one line per pass with the run, file, tool, event id and the agent's words. `--json` adds `tamperAttribution` per run, and the MCP `harness_compare` tool a `tamper` field. [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#harness-ab). - Added After `eval audit`, `compare` shows each audited arm's raw and audited pass rate, `audited pass B 60% → 30% (3 refuted, coverage 100%)`. `--audited` counts refuted passes as fails and labels the verdict `verdict (audited)`, with a note when the audit changes it. It needs every pass in both arms replayed, or the raw verdict stands with a coverage note. `--json` and the MCP `harness_compare` tool add the per-arm counts. [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#harness-ab). - Added `compare` adds a `surface` row: the bytes of instruction text each bundle puts in front of the agent (AGENTS.md and the other instructions files, skills, rules, system-prompt files, `~/` ones too). When B's is more than 20% bigger, a note asks for a success CI above 0 rather than a tie, since those bytes cost tokens on every turn. `--json` has the sizes per file in `report.surface`. [Compare agents](https://github.com/jackulau/coding.sh/blob/main/docs/how-to/compare-agents.md#why-runs-failed). - Changed The `harness changes` block of a `compare` report collapses a file list longer than 20 lines into directories with file counts (`+ .claude/skills/ (412 files)`). Before, comparing a full `~/.claude` against plain Claude Code printed one line per file, 1,500 of them. `--json` still lists every path in `report.bundleDiff`. [Harness A/B](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#harness-ab). - Fixed `harness snapshot` follows linked skill, command and agent directories (`npx skills`, a dotfiles repo) and stores the files they point to. Before, a `~/.claude/skills` made of links snapshotted no skills, so the arm ran without them. [Harness A/B](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#harness-ab). - Fixed `compare` and `eval run` say why a failed run's agent failed: its exit code or timeout and the last lines of its stderr, redacted, under the report and in `failedRuns[].agent`. Before, an agent that couldn't start (Claude Code refusing bypass permissions as root, a bad flag) failed every run in seconds with only the verifier's output shown. [Verifier output](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#verifier-output). - Fixed Two tasks from the same repo no longer error when they start at once: the first run of a repo set up its snapshot store while the second was reading it. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md). - Fixed The tamper check no longer flags a run that only added new test cases to an existing test file, so a regression test written next to the fix keeps the task in the gate's evidence. Changed, removed or skipped tests, early returns, hooks, mocks and shadowing definitions still flag. The run lists those files in `tamper.addedTests`. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#harness-ab). - Fixed `compare --gate` no longer says "Don't ship: B breaks a task A solved" when both arms passed it by editing tests. Tamper-flagged passes now count as fails for the gate in both arms, not only in B. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#harness-ab). - Fixed `compare` and `eval crosscheck` runs keep the agent's text from mid-turn: the run dir, which holds the agent's own transcript, was removed before it was ingested, so the session kept only the final reply. ## Skills - Added `eval trigger ` checks that the agent loads a skill when it should and not when it shouldn't. It runs the `should` and `should_not` prompts in `triggers.toml` k times each with the skill and its siblings installed, reads each load from the captured session, and reports precision, recall and F1 with Wilson intervals, the missed and flaky prompts, and near misses that went to the expected sibling. `--description` or `--candidate` A/Bs a new description with a paired bootstrap CI and the prompts that flipped. `--min-f1` and `--fail-on regression` set the exit code. CI checks it with real Claude Code and Codex against a stub model. [Evaluate an Agent Skill](https://coding.sh/docs/how-to/evaluate-a-skill#does-the-skill-load). ## Self-host - Added `docker run -p 4318:4318 -v coding-sh:/data ghcr.io/jackulau/coding.sh` starts a team server with no configuration: with no keys set, the first start makes a read key and an ingest key, keeps them in the volume and prints them with the connect command. Each tagged release publishes the image for amd64 and arm64, starting with this one. `deploy/compose/` has SQLite, Postgres and ClickHouse stacks that need no edits. The viewer's sign-in page takes the read key in a form. serve waits up to 30 s for its database to accept connections. [Self-host for a team](https://coding.sh/docs/how-to/self-host-team). - Added `coding.sh init --remote --key csh_ik_…` connects a machine to a team server in one command: it checks the key, installs capture hooks and adds the `team` sink. `--remote-mode full` also sends prompts and output. [Self-host for a team](https://coding.sh/docs/how-to/self-host-team#connect). ## Replay - Added `coding.sh replay ` and a Replay mode on the viewer's session page step through a session with the files the agent touched rebuilt as of each step: each step's diff, a file's content at that step, what a range of steps changed, and shell commands with exit code and output. Where content can't be rebuilt, it says which step and why. The viewer plays at 1×, 4× or 16× with a scrubber and `?step=N` links; the MCP `session_replay` tool returns the same. [Explore](https://github.com/jackulau/coding.sh/blob/main/docs/guides/explore.md#replay). - Added `exp replay` and `replay --fork` use `CLAUDE_CODE_OAUTH_TOKEN` (from `claude setup-token`) for Anthropic calls when `ANTHROPIC_API_KEY` is not set. [Log model calls](https://coding.sh/docs/how-to/capture-with-proxy#5-replay-a-call-with-another-model-or-prompt). ## Install - Added `init --proxy` points Claude Code (`ANTHROPIC_BASE_URL`) and Codex (a `coding-sh` provider) at the proxy and starts it; the hook restarts it when a session starts. `uninstall --proxy` undoes only that, and `doctor` shows whether it runs. CI checks it with real Claude Code and Codex. [Capture with the proxy](https://coding.sh/docs/how-to/capture-with-proxy#1-connect). - Added `init --proxy` keeps a base URL the agent already used (a gateway or CLIProxyAPI in Claude Code's `ANTHROPIC_BASE_URL`, your own Codex provider, `openai_base_url` or `OPENAI_BASE_URL`) as the proxy's upstream and chains through it, instead of asking for `--force`. `uninstall --proxy` puts the old value back. [Already using a gateway or CLIProxyAPI?](https://coding.sh/docs/how-to/capture-with-proxy#already-using-a-gateway-or-cliproxyapi) - Added Windows: run the installer from Git Bash. Hook commands use `C:/…` paths that cmd.exe, PowerShell and Git Bash all run. CI runs the tests and the installer on Windows. [Install guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/install.md#windows). ## Serve - Added `coding.sh serve` mounts the proxy at `/proxy/`: send an ingest key in `x-coding-sh-key` and the agent's own API key goes upstream unchanged. [REST API](https://coding.sh/docs/api#post-proxypath). [Previous page v0.1.2, page 2](https://coding.sh/changelog/v0-1-2/2) [Next page v0.1.2, page 4](https://coding.sh/changelog/v0-1-2/4) From [CHANGELOG.md](https://github.com/jackulau/coding.sh/blob/main/CHANGELOG.md) · [RSS](https://coding.sh/changelog.xml) · [All releases](https://coding.sh/changelog) To set coding.sh up for a user, follow the agent workflow at [/agents.md](https://coding.sh/agents.md).