GitHub
Changelog

Nov 8, 2026 · 177 changes · page 3 of 5

coding.sh v0.1.2

coding.sh runs on Windows, and CI tests it there.

Harness A/B 16

  • # Added Each run keeps the end of its verifier's output (the last 4096 characters of stdout and stderr, colour codes removed, redacted with capture.redact), each repair round too. compare and eval run print the last lines of failed runs under the report, and --json lists them in failedRuns. Verifier output.
  • # Added Harness bundles carry Claude Code plugins (hooks, skills, agents, commands, MCP servers: the whole plugin directory) with harness snapshot --plugin <name|dir> or plugins = [...] in a harness TOML, and compare A --with plugin:<name> runs "with the plugin vs without" in one command (--without plugin:<name> drops one). Each run loads the plugin with --plugin-dir from its own home; your Claude config is never changed. Evals, Does this plugin help me?.
  • # Added compare --repair N reports Seq@1..N+1 per arm: the share of tasks solved by the k-th attempt with verifier feedback, with intervals, where the gain lands (EGS, LGS), rounds that changed no file, and the agent's spend per solved run. With --k 2 or more it adds a cost front of k attempts with feedback against k independent attempts (unbiased pass@k), and a note when the two arms rank one way on the first attempt and the other way after the rounds (Seq@K, arXiv 2610.37859). Evals.
  • # Added compare --repair N tells the agent "Verification failed." in its own session (Claude Code --resume, codex exec resume) after a failed verifier, up to N rounds, with the hidden tests out of its tree. The report adds the repair rate, mean rounds and the rounds' cost per arm. --repair-signal names adds the failing test names from the verifier output; a round that edits, deletes or skips tests is not a repair; --false-alarm P tells a seeded share of passing runs they failed; --repair-verdict decides on the outcome after repair. Repair settings are part of the --reuse key (BOBA, arXiv 2611.02207). Evals.
  • # Added compare --gate answers "ship B or not": Ship it: B wins 9 tasks, loses 0, Hold: not enough evidence, or Don't ship: B breaks 3 tasks A solved with the broken tasks listed. It exits 0 only for "Ship it" (SAGE, arXiv 2609.36043). --by suite: gates each tagged suite too, and one broken suite blocks a pooled "Ship it". Each arm's failed runs are split by the agent's own last test run: A fails 12: own tests green 7 (58%), red 3, stale 1, none 1, where green fails are silent misses (BOBA, arXiv 2611.02207). compare --promote also keeps a B that is cheaper at the same quality: pareto: support 0.93 ≥ 0.70 → promote (cost-cut: …) or → hold, exit 0 only on promote (ParetoSaddler, arXiv 2611.02079). When B adds or removes one rule, skill or context file, the report adds a one-item gate scored only on tasks where the item reached the agent's context: item gate k=3 τ=.34 m=2: KEEP · …, or RECHECK, KILL, RERUN, and ACCEPT, REJECT, UNMEASURED for removals; --item-gate sets the exit code (Cortex, arXiv 2610.14216). CI checks the gate and the own-tests split on real Claude Code and Codex sessions against a stub model. The MCP harness_compare tool lists these A/B results and returns one with its gates and own-tests split. Compare agents.
  • # Added coding.sh refine check exits 1 on an AGENTS.md, rule or skill edit that deletes or rewords a protected safety line, or adds an exception that relaxes one (`protected-line: AGENTS.md:3 "Never run git push --force." (auto)), since a pass-rate A/B can't see that harm (Masked Erosion, arXiv 2610.38392). A line is protected by a <!-- coding.sh:protect --> block, a [refine] protect regex, or automatically when one sentence pairs a gate word with a safety topic; refine protected lists them. exp ab runs the same check before any model call and fails --gate on it. --allow-protected-edit AGENTS.md:3` allows one on purpose. Evals guide.
  • # Added compare links each tamper flag to the tool call that made the change and splits tampered passes by whether the agent said it was changing the test: tampered passes B 3: disclosed 2, stealth 1, then one line per pass with the run, file, tool, event id and the agent's words. --json adds tamperAttribution per run, and the MCP harness_compare tool a tamper field. Evals guide.
  • # Added After eval audit, compare shows each audited arm's raw and audited pass rate, audited pass B 60% → 30% (3 refuted, coverage 100%). --audited counts refuted passes as fails and labels the verdict verdict (audited), with a note when the audit changes it. It needs every pass in both arms replayed, or the raw verdict stands with a coverage note. --json and the MCP harness_compare tool add the per-arm counts. Evals guide.
  • # Added compare adds a surface row: the bytes of instruction text each bundle puts in front of the agent (AGENTS.md and the other instructions files, skills, rules, system-prompt files, ~/ ones too). When B's is more than 20% bigger, a note asks for a success CI above 0 rather than a tie, since those bytes cost tokens on every turn. --json has the sizes per file in report.surface. Compare agents.
  • # Changed The harness changes block of a compare report collapses a file list longer than 20 lines into directories with file counts (+ .claude/skills/ (412 files)). Before, comparing a full ~/.claude against plain Claude Code printed one line per file, 1,500 of them. --json still lists every path in report.bundleDiff. Harness A/B.
  • # Fixed harness snapshot follows linked skill, command and agent directories (npx skills, a dotfiles repo) and stores the files they point to. Before, a ~/.claude/skills made of links snapshotted no skills, so the arm ran without them. Harness A/B.
  • # Fixed compare and eval run say why a failed run's agent failed: its exit code or timeout and the last lines of its stderr, redacted, under the report and in failedRuns[].agent. Before, an agent that couldn't start (Claude Code refusing bypass permissions as root, a bad flag) failed every run in seconds with only the verifier's output shown. Verifier output.
  • # Fixed Two tasks from the same repo no longer error when they start at once: the first run of a repo set up its snapshot store while the second was reading it. Evals.
  • # Fixed The tamper check no longer flags a run that only added new test cases to an existing test file, so a regression test written next to the fix keeps the task in the gate's evidence. Changed, removed or skipped tests, early returns, hooks, mocks and shadowing definitions still flag. The run lists those files in tamper.addedTests. Evals.
  • # Fixed compare --gate no longer says "Don't ship: B breaks a task A solved" when both arms passed it by editing tests. Tamper-flagged passes now count as fails for the gate in both arms, not only in B. Evals.
  • # Fixed compare and eval crosscheck runs keep the agent's text from mid-turn: the run dir, which holds the agent's own transcript, was removed before it was ingested, so the session kept only the final reply.

Skills 1

  • # Added eval trigger <skill-dir> checks that the agent loads a skill when it should and not when it shouldn't. It runs the should and should_not prompts in triggers.toml k times each with the skill and its siblings installed, reads each load from the captured session, and reports precision, recall and F1 with Wilson intervals, the missed and flaky prompts, and near misses that went to the expected sibling. --description or --candidate A/Bs a new description with a paired bootstrap CI and the prompts that flipped. --min-f1 and --fail-on regression set the exit code. CI checks it with real Claude Code and Codex against a stub model. Evaluate an Agent Skill.

Self-host 2

  • # Added docker run -p 4318:4318 -v coding-sh:/data ghcr.io/jackulau/coding.sh starts a team server with no configuration: with no keys set, the first start makes a read key and an ingest key, keeps them in the volume and prints them with the connect command. Each tagged release publishes the image for amd64 and arm64, starting with this one. deploy/compose/ has SQLite, Postgres and ClickHouse stacks that need no edits. The viewer's sign-in page takes the read key in a form. serve waits up to 30 s for its database to accept connections. Self-host for a team.
  • # Added coding.sh init --remote <url> --key csh_ik_… connects a machine to a team server in one command: it checks the key, installs capture hooks and adds the team sink. --remote-mode full also sends prompts and output. Self-host for a team.

Replay 2

  • # Added coding.sh replay <session> and a Replay mode on the viewer's session page step through a session with the files the agent touched rebuilt as of each step: each step's diff, a file's content at that step, what a range of steps changed, and shell commands with exit code and output. Where content can't be rebuilt, it says which step and why. The viewer plays at 1×, 4× or 16× with a scrubber and ?step=N links; the MCP session_replay tool returns the same. Explore.
  • # Added exp replay and replay --fork use CLAUDE_CODE_OAUTH_TOKEN (from claude setup-token) for Anthropic calls when ANTHROPIC_API_KEY is not set. Log model calls.

Install 3

  • # Added init --proxy points Claude Code (ANTHROPIC_BASE_URL) and Codex (a coding-sh provider) at the proxy and starts it; the hook restarts it when a session starts. uninstall --proxy undoes only that, and doctor shows whether it runs. CI checks it with real Claude Code and Codex. Capture with the proxy.
  • # Added init --proxy keeps a base URL the agent already used (a gateway or CLIProxyAPI in Claude Code's ANTHROPIC_BASE_URL, your own Codex provider, openai_base_url or OPENAI_BASE_URL) as the proxy's upstream and chains through it, instead of asking for --force. uninstall --proxy puts the old value back. Already using a gateway or CLIProxyAPI?
  • # Added Windows: run the installer from Git Bash. Hook commands use C:/… paths that cmd.exe, PowerShell and Git Bash all run. CI runs the tests and the installer on Windows. Install guide.

Serve 1

  • # Added coding.sh serve mounts the proxy at /proxy/: send an ingest key in x-coding-sh-key and the agent's own API key goes upstream unchanged. REST API.

From CHANGELOG.md · RSS · All releases