---
title: "coding.sh v0.1.2 · Changelog"
description: "coding.sh runs on Windows, and CI tests it there. 177 changes, most to Experiments, Evals and Harness A/B."
canonical: https://coding.sh/changelog/v0-1-2
human_url: https://coding.sh/changelog/v0-1-2
markdown_url: https://coding.sh/changelog/v0-1-2.md
agents_url: https://coding.sh/agents.md
llms_url: https://coding.sh/llms.txt
updated: 2026-11-11
---

[Changelog](https://coding.sh/changelog)

Nov 8, 2026 · 177 changes · page 1 of 5

# coding.sh v0.1.2

coding.sh runs on Windows, and CI tests it there.

## Sessions

- Added A new `text-tool-call` flag marks a session whose model wrote a tool call out as text, so the call never ran. In Claude Code this is usually a subagent whose `tools:` frontmatter is a map (`edit: false`), which leaves it with no tools; `improve` names the fix. [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md#recording).
- Added A session whose main thread moved to another model mid-session, such as Claude Code's silent fallback to an older model after a refusal, gets the `model:switch` tag and `stats.modelSwitch` with both models, when, and the trigger (`refusal`, `error` or `/model`). `stats.models` counts calls per model (both show in a Models row in `show` and the MCP digest), a refused call adds the `refusal` flag, and a session's model is now the one most of its main-thread calls ran on instead of the last one seen. Helper models (Haiku titles) and subagents are not a switch. [Read API](https://github.com/jackulau/coding.sh/blob/main/docs/reference/api.md#get-apisessionsid).
- Added Text that Claude Code hooks add to the context (a plugin's SessionStart notes, a UserPromptSubmit reminder, PostToolUse additionalContext) is stored as `hook.context` events with its hook event, source and size. `stats.hookContext` counts the tokens by source, each source tags the session (`hook:plugin:terse`, `hook:settings`), `sessions --where hook:plugin:<name>` or `hook.tokens>500` filters on it, `--facets hook` counts sessions per source, and `show` lists it. [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md).
- Added Headless sessions (`claude -p`, an Agent SDK, `codex exec`) are marked `stats.launch: "headless"`, and terminal or IDE sessions `"interactive"`, from Claude Code's entrypoint and Codex's originator. `sessions --where launch:headless` filters on it and `--facets launch` counts it. [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md).
- Added `show` has a `Flags` row, and `sessions` ends with how many listed sessions are flagged and a pointer to `coding.sh improve`.
- Added `show <session>#n` (or a call id) prints one model call's system prompt, tools, settings, cache use and cost, and `diff-calls` compares two calls in prompt-cache order and names what broke the cache. The viewer shows both on a proxy call. [Capture with the proxy](https://coding.sh/docs/how-to/capture-with-proxy#4-why-did-the-cache-miss).
- Added `replay <session> --step N --fork` sends the model call behind step N again with a changed AGENTS.md, rule, skill, system prompt or model (`--with`, `--harness`, `--model`, `--system`, `-k`) and prints the recorded next action beside the forked one. `--run` carries on with Claude Code in a copy of the repo, the stored history spliced into its requests, and `--verify` runs a check there. It needs a session captured through the proxy. The viewer's Replay pane has a "Fork from here" tab, and the MCP `session_replay` tool takes `fork`. [Explore sessions](https://github.com/jackulau/coding.sh/blob/main/docs/guides/explore.md#fork-from-a-step).
- Added Each skill a session used, through the Skill tool or by reading or running a file in its folder, adds a `skill:<name>` tag, so `sessions --flag skill:pdf` finds those sessions. `show` lists them as `Skills pdf (project, invoked, first, before any edit) · pdf-tools (personal, read)`, and `show --json` and MCP `session_get` return `skills`. A new `skill-copy-conflict` flag marks a session that used two copies of one skill (arXiv 2610.11647). [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md).
- Added New `test-tamper` flag: the agent added a skip marker to a test, deleted a test file while tests failed, or changed an assertion and that edit alone turned a failing run green. It reads Edit/Write/apply_patch changes and shell writes (`sed -i`, `>`, `tee`, `rm`, `mv`, `git checkout`), and covers test-runner configs and CI files. New tests and files the agent created itself are not flagged. On 3,957 local sessions it flags 13 (EPISTEME, arXiv 2611.06563). [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md).
- Fixed Replay knows that a file a step created did not exist before that step, instead of calling it unknown, so `replay --fork --run` removes it from the fork's copy. [Replay a session](https://coding.sh/docs/use-cases/replay-agent-session#how-files-are-rebuilt).
- Fixed `premature-completion` no longer flags an agent that stopped on a failing test run and said so ("2 tests still fail", "I couldn't get test_x to pass", or a question about the failing test). It still flags one that says it is done, fixed or passing, or says nothing about the failure. Stored sessions get their flags recomputed once. [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md).
- Fixed Claude Code subagents get a title and a turn: the title is the Agent tool's `description`, else the first line of the prompt, and the prompt is recorded as the subagent's first message. Claude Code's internal forks (the compaction summarizer, prompt suggestions) no longer leave empty subagent sessions with no model, tools or title. Stored sessions are parsed again on the next ingest. [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md).
- Fixed `skill:<name>` tags and `skill-copy-conflict` count only installed skills: a `skills/` folder under an agent's dot folder or in a plugin. A `skills/` folder in source code (`skills/mod.rs`), a reference checkout, text inside a heredoc and a write to a skill file no longer tag a skill. Codex's `.system/` and `synced/<id>/` folders are skipped, so `skill:.system` and `skill:synced` become the skill's own name, and a plugin named for its one skill is tagged once (`skill:frontend-design`). [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md).
- Fixed `test-tamper` no longer flags adding tests: a skip guard inside a new test, fixing a test the agent added earlier, updating an expectation its own source edit broke, or a green run of fewer tests than the failing one. A file a shell command's diff picked up without the command naming it (a test run, a `git rebase`, another agent in the checkout) is never flagged, and BSD `sed -i ''` no longer counts `''` as a file. A skip marker inside a string literal (a diff or `sed` command a test holds as data) is not a skip. Stored sessions are re-checked in the background after the upgrade. [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md).
- Fixed `coding.sh sessions` leaves out sessions with no turn, tool call or model call, such as a process that started and ended a session and did nothing; `--empty` lists them too, and `--limit` still fills with sessions that have work. [CLI reference](https://github.com/jackulau/coding.sh/blob/main/docs/reference/cli.md).

## Evals

- Added `eval validate` takes several tasks, or `--tag <tag>` for every task with that tag (`suite:<name>` for a suite), prints one line per task and exits 1 if any did not validate. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#session-to-eval).
- Added `eval init --sessions --repo <DIR|all>` drafts tasks from sessions in another checkout or worktree of the project, not only the current one. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md).
- Added `eval validate` checks verifiers that grade the agent's answer or trace: a task dir's `reference/` (`output.md`, `stdout.jsonl`) is a correct run that must pass, and `near-misses/<label>/` hold wrong runs that must fail. The verifier reads them through `CODING_SH_AGENT_OUTPUT`, `CODING_SH_AGENT_STDOUT` and `CODING_SH_AGENT_DIFF`, as in a real run. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#hand-written-tasks).
- Added A hand-written task can start from a commit with `"start": {"commit": "<sha or ref>", "repo": "<path>"}` in `task.json`, so a suite can live in the repo it tests. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#hand-written-tasks).
- Added `coding.sh eval init` writes an eval suite to `evals/<name>/` from failed or flagged sessions in the repo (`--sessions`, `--session latest`), a skill's should and should-not prompts (`--skill`), or cases for an app (`--cases`, `--describe` with `--command` or `--url`). `--draft` has a model write the prompts or cases, marked draft. It picks a code grader when it can and a judge to calibrate when it can't, marks the suite capability or regression, and ends with the run command and, for cases and skills, a GitHub Actions workflow. On a terminal with no flags it asks. [Your first eval in 5 minutes](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#your-first-eval-in-5-minutes).
- Added `coding.sh eval run [suite…]` runs suites k times per task and reports pass@k, pass^k, flaky tasks, the session of each failed trial, small suites and saturation. A regression suite exits 1 when a task fails most of its trials or nothing could be graded; a capability suite never fails on its score. [The report](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#the-report).
- Added `eval import` warns when a verifier names a `tests/` path but the hidden files land at the work-tree root (`tests/check.js` is copied to `check.js`). Such a verifier finds no file, or runs zero tests and passes. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#hand-written-tasks).
- Added `replay --fork --save-case <suite>` keeps the step as a case in `evals/<suite>`: a `request` task that sends the stored request with the project's current context files, graded by a `next-action` grader that fails when the model repeats the recorded action. `eval run` runs it like any suite. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#fork-a-failing-step-into-a-case).
- Added Behaviour tasks can be written by hand and loaded with `eval import tasks/*/`; `compare --tag <tag>` runs them. Verifiers get the agent's final answer and full stdout (`CODING_SH_AGENT_OUTPUT`, `CODING_SH_AGENT_STDOUT`), and `judge grade --grader checklist.json` grades both from inside one. `compare` reports `agent spend` and `grader spend` rows, `--max-cost 15` stops starting runs at $15, per-task rows show task names, and a fresh store gets its hook script so runs are captured. [Evaluate an Agent Skill](https://coding.sh/docs/how-to/evaluate-a-skill).
- Added `eval validate` flags hidden tests that name a helper or file only the reference adds and the instruction never mentions, which fail valid fixes written differently (`--strict-coupling` makes it blocking). `eval crosscheck <task>` has a reviewer model judge stored runs' diffs against the instruction and lists the runs where it disagrees with the verifier, as false rejects or false accepts. `--solver <bundle>` solves the task once first. The counts are stored on the task in the same shape as a human verifier audit. [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#session-to-eval)
- Added `eval audit <task> --experiment <id> --witness <file> --witness-cmd <cmd>` replays a witness test, one that checks a requirement the verifier skips, on every passing run: `claude-tdd: passes 6/10, refuted 3, audited pass rate 30% (3/10)`. The witness must pass the reference 3/3 first. A replay whose verifier no longer passes is `replay-mismatch`, never refuted, and a witness refuting over half the passes is ambiguous and not counted. No model calls (TestJack, arXiv 2610.10619). `--promote` adds it to the verifier, and `eval export`/`import` carry witnesses. [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#witness-audits).
- Added Experiment sessions: `exp session new --name <topic>` starts one in a folder, and each `exp run` there prints its delta against the previous run on the same dataset as it finishes (`pass 0.0% → 100.0% Δ +100.0pp […] mcnemar p=0.125`, the grader deltas, the cases fixed and broken, the files changed), with no `exp compare`. `exp session note` records hypotheses and readings, `exp session summary [--json]` prints the timeline. `exp run --watch` reruns on every save of the spec, the agent script, grader files or `watch` globs, aborts a run made stale mid-way without storing it, and regrades the stored outputs with 0 agent calls when only the graders changed. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#sessions-deltas-notes-watch).
- Added An `executability` grader scores a run by how far it got before its first invalid tool call, one the harness refused to run (a bad argument, an unknown tool, an edit of a file not read yet, a string to replace that is not there, a missing file or command, a denied call). A test that exits 1 ran and is not invalid. The score is the share of steps before that call (its mean is AUC-PEC), and it passes with at most `maxInvalid` invalid calls (its pass rate is TER); `kinds` counts only some kinds (Prefix Executability, arXiv 2610.40444). [Explore guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/explore.md#executability).
- Fixed `eval validate` applies a reference or near-miss patch after setup when it only applies to what setup creates, as an agent's run sees it. Before, a task whose setup made the mess the agent cleans up could not validate: its reference "did not apply". [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#hand-written-tasks).
- Fixed A command task, grader or source that crashes names the error and its cause, such as `exited 1: TypeError: fetch failed (cause: connect ECONNREFUSED 127.0.0.1:8803)`, instead of the last three lines of stderr (for Node, the closing braces of the error object). A refused connection now reads as an infrastructure error, so the trial is retried and counted as infra, not as a failure. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md)
- Fixed A task drafted from a session names the files its setup was guessed from, such as `setup guessed from package.json (no lockfile): npm install`, instead of saying "from lockfiles" when there was none.
- Fixed `eval run` reports on a cases suite say "case" and on a triggers suite "prompt" (the header, the table, never-passed, saturation and the regression lines), and `--json` carries the report's `unit`.
- Fixed `eval validate` shows the last 12 lines of the verifier's output when a check goes the wrong way (the reference fails, the no-op or a near-miss passes, or setup fails), once per kind of check, and `--json` carries every step with its output in `steps`. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md)
- Fixed `eval init --force` no longer deletes your work: it stops when the suite holds a file `eval init` did not write or that you changed since (a `reference.patch` you added, a prompt you fixed) and lists them. `--force-all` deletes those too. `evals/<name>/.eval-init.json` records what `eval init` wrote. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#your-agents-failed-sessions)
- Fixed `eval init --skill <dir>` uses the skill's own `triggers.toml` when it has one and you give no prompts and no `--draft`: `suite.json` points at it instead of a TODO template. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#a-skill)
- Fixed `eval init --sessions` and `--session latest` include a session that just ended: they wait up to 30 seconds for the ingest its hook started, then ingest the rest, instead of skipping the ingest when its lock is busy after 1.5 seconds. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#your-agents-failed-sessions)
- Fixed On Windows, `eval init` and `eval run` show suite paths with forward slashes, like the file list under them: `Wrote … to evals/triage`, `evals/pdf/triggers.toml still say TODO` and the `--skill` path in the `--draft --force` hint read the same on every OS. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#a-skill)

## Docs

- Added Use-case pages for tracking Claude Code and Codex cost, replaying a session, testing an AGENTS.md or skill change, building evals from failures, self-hosting, and logging model calls through the proxy. Comparison pages for Langfuse, LangSmith, Braintrust, AgentOps, Laminar and ccusage, each with a feature table, where the other tool is the better choice, and its sources. [Compare](https://coding.sh/docs/compare).
- Added A Workflow experiments page: capture and score multi-agent runs, `workflows compare`, replays, the control fork and the replay noise floor. [Workflow experiments](https://coding.sh/docs/workflows)
- Added A grader field reference: every grader kind, its fields and defaults, and when it passes, with the provider fields judges share. [Grader fields](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#grader-fields)
- Changed Long docs are split into pages that load faster: the CLI reference has a page per command, a long changelog release continues over numbered pages, Evals continues on [Experiments](https://coding.sh/docs/experiments) and [Harness evals](https://coding.sh/docs/harness-evals), and the Read API has its own two pages. Old links to a section that moved still land on it. [CLI reference](https://coding.sh/docs/cli)
- Changed The vercel-security-audit case study has the live result: six read-only tasks against a real Vercel team, three runs per arm. Counting the 27 sessions that stayed on the requested model, graded with the fixed site check, 0/14 runs passed without the skill and 9/13 with it (+69.2 points, 95% CI +23.1 to +100, McNemar p = 0.25, gate Hold), at 6 times the cost per run. A Corrections section says what changed from the first 4/18 and 14/18: the `$LIVE_SITE` check, task 06's wording, and 9 sessions that Claude Code moved to an older model after a refusal. Aggregate numbers only. [Case study](https://github.com/jackulau/coding.sh/blob/main/docs/case-studies/vercel-security-audit.md#live-run).

## Site

- Added Every docs page has its own link preview image, the home page and FAQ carry FAQ markup, and the site build fails on a missing or too-long title or description, a wrong canonical URL, or a link to a page that doesn't exist.
- Added Pages are ready about 0.5 s sooner on a slow phone, with half the font bytes and less layout shift. A Lighthouse A/B with budgets runs on every site change. [Site speed](https://coding.sh/docs/perf).
- Added A blog at /blog: posts are Markdown files in `apps/site/src/content/blog`, each with its own page, title card and sitemap entry, and an Atom feed at /blog.xml. The first post explains why coding.sh keeps your sessions on your machine. [Blog](https://coding.sh/blog).
- Fixed A page that shows "±" no longer downloads a second font file for it. [Site speed](https://coding.sh/docs/perf)

[Newer release v0.1.3](https://coding.sh/changelog/v0-1-3) [Next page v0.1.2, page 2](https://coding.sh/changelog/v0-1-2/2)

From [CHANGELOG.md](https://github.com/jackulau/coding.sh/blob/main/CHANGELOG.md) · [RSS](https://coding.sh/changelog.xml) · [All releases](https://coding.sh/changelog)

To set coding.sh up for a user, follow the agent workflow at [/agents.md](https://coding.sh/agents.md).