GitHub
Changelog

Nov 8, 2026 · 177 changes · page 1 of 5

coding.sh v0.1.2

coding.sh runs on Windows, and CI tests it there.

Sessions 15

  • # Added A new text-tool-call flag marks a session whose model wrote a tool call out as text, so the call never ran. In Claude Code this is usually a subagent whose tools: frontmatter is a map (edit: false), which leaves it with no tools; improve names the fix. Concepts.
  • # Added A session whose main thread moved to another model mid-session, such as Claude Code's silent fallback to an older model after a refusal, gets the model:switch tag and stats.modelSwitch with both models, when, and the trigger (refusal, error or /model). stats.models counts calls per model (both show in a Models row in show and the MCP digest), a refused call adds the refusal flag, and a session's model is now the one most of its main-thread calls ran on instead of the last one seen. Helper models (Haiku titles) and subagents are not a switch. Read API.
  • # Added Text that Claude Code hooks add to the context (a plugin's SessionStart notes, a UserPromptSubmit reminder, PostToolUse additionalContext) is stored as hook.context events with its hook event, source and size. stats.hookContext counts the tokens by source, each source tags the session (hook:plugin:terse, hook:settings), sessions --where hook:plugin:<name> or hook.tokens>500 filters on it, --facets hook counts sessions per source, and show lists it. Concepts.
  • # Added Headless sessions (claude -p, an Agent SDK, codex exec) are marked stats.launch: "headless", and terminal or IDE sessions "interactive", from Claude Code's entrypoint and Codex's originator. sessions --where launch:headless filters on it and --facets launch counts it. Concepts.
  • # Added show has a Flags row, and sessions ends with how many listed sessions are flagged and a pointer to coding.sh improve.
  • # Added show <session>#n (or a call id) prints one model call's system prompt, tools, settings, cache use and cost, and diff-calls compares two calls in prompt-cache order and names what broke the cache. The viewer shows both on a proxy call. Capture with the proxy.
  • # Added replay <session> --step N --fork sends the model call behind step N again with a changed AGENTS.md, rule, skill, system prompt or model (--with, --harness, --model, --system, -k) and prints the recorded next action beside the forked one. --run carries on with Claude Code in a copy of the repo, the stored history spliced into its requests, and --verify runs a check there. It needs a session captured through the proxy. The viewer's Replay pane has a "Fork from here" tab, and the MCP session_replay tool takes fork. Explore sessions.
  • # Added Each skill a session used, through the Skill tool or by reading or running a file in its folder, adds a skill:<name> tag, so sessions --flag skill:pdf finds those sessions. show lists them as Skills pdf (project, invoked, first, before any edit) · pdf-tools (personal, read), and show --json and MCP session_get return skills. A new skill-copy-conflict flag marks a session that used two copies of one skill (arXiv 2610.11647). Concepts.
  • # Added New test-tamper flag: the agent added a skip marker to a test, deleted a test file while tests failed, or changed an assertion and that edit alone turned a failing run green. It reads Edit/Write/apply_patch changes and shell writes (sed -i, >, tee, rm, mv, git checkout), and covers test-runner configs and CI files. New tests and files the agent created itself are not flagged. On 3,957 local sessions it flags 13 (EPISTEME, arXiv 2611.06563). Concepts.
  • # Fixed Replay knows that a file a step created did not exist before that step, instead of calling it unknown, so replay --fork --run removes it from the fork's copy. Replay a session.
  • # Fixed premature-completion no longer flags an agent that stopped on a failing test run and said so ("2 tests still fail", "I couldn't get test_x to pass", or a question about the failing test). It still flags one that says it is done, fixed or passing, or says nothing about the failure. Stored sessions get their flags recomputed once. Concepts.
  • # Fixed Claude Code subagents get a title and a turn: the title is the Agent tool's description, else the first line of the prompt, and the prompt is recorded as the subagent's first message. Claude Code's internal forks (the compaction summarizer, prompt suggestions) no longer leave empty subagent sessions with no model, tools or title. Stored sessions are parsed again on the next ingest. Concepts.
  • # Fixed skill:<name> tags and skill-copy-conflict count only installed skills: a skills/ folder under an agent's dot folder or in a plugin. A skills/ folder in source code (skills/mod.rs), a reference checkout, text inside a heredoc and a write to a skill file no longer tag a skill. Codex's .system/ and synced/<id>/ folders are skipped, so skill:.system and skill:synced become the skill's own name, and a plugin named for its one skill is tagged once (skill:frontend-design). Concepts.
  • # Fixed test-tamper no longer flags adding tests: a skip guard inside a new test, fixing a test the agent added earlier, updating an expectation its own source edit broke, or a green run of fewer tests than the failing one. A file a shell command's diff picked up without the command naming it (a test run, a git rebase, another agent in the checkout) is never flagged, and BSD sed -i '' no longer counts '' as a file. A skip marker inside a string literal (a diff or sed command a test holds as data) is not a skip. Stored sessions are re-checked in the background after the upgrade. Concepts.
  • # Fixed coding.sh sessions leaves out sessions with no turn, tool call or model call, such as a process that started and ended a session and did nothing; --empty lists them too, and --limit still fills with sessions that have work. CLI reference.

Evals 22

  • # Added eval validate takes several tasks, or --tag <tag> for every task with that tag (suite:<name> for a suite), prints one line per task and exits 1 if any did not validate. Evals.
  • # Added eval init --sessions --repo <DIR|all> drafts tasks from sessions in another checkout or worktree of the project, not only the current one. Evals.
  • # Added eval validate checks verifiers that grade the agent's answer or trace: a task dir's reference/ (output.md, stdout.jsonl) is a correct run that must pass, and near-misses/<label>/ hold wrong runs that must fail. The verifier reads them through CODING_SH_AGENT_OUTPUT, CODING_SH_AGENT_STDOUT and CODING_SH_AGENT_DIFF, as in a real run. Evals.
  • # Added A hand-written task can start from a commit with "start": {"commit": "<sha or ref>", "repo": "<path>"} in task.json, so a suite can live in the repo it tests. Evals.
  • # Added coding.sh eval init writes an eval suite to evals/<name>/ from failed or flagged sessions in the repo (--sessions, --session latest), a skill's should and should-not prompts (--skill), or cases for an app (--cases, --describe with --command or --url). --draft has a model write the prompts or cases, marked draft. It picks a code grader when it can and a judge to calibrate when it can't, marks the suite capability or regression, and ends with the run command and, for cases and skills, a GitHub Actions workflow. On a terminal with no flags it asks. Your first eval in 5 minutes.
  • # Added coding.sh eval run [suite…] runs suites k times per task and reports pass@k, pass^k, flaky tasks, the session of each failed trial, small suites and saturation. A regression suite exits 1 when a task fails most of its trials or nothing could be graded; a capability suite never fails on its score. The report.
  • # Added eval import warns when a verifier names a tests/ path but the hidden files land at the work-tree root (tests/check.js is copied to check.js). Such a verifier finds no file, or runs zero tests and passes. Evals.
  • # Added replay --fork --save-case <suite> keeps the step as a case in evals/<suite>: a request task that sends the stored request with the project's current context files, graded by a next-action grader that fails when the model repeats the recorded action. eval run runs it like any suite. Evals.
  • # Added Behaviour tasks can be written by hand and loaded with eval import tasks/*/; compare --tag <tag> runs them. Verifiers get the agent's final answer and full stdout (CODING_SH_AGENT_OUTPUT, CODING_SH_AGENT_STDOUT), and judge grade --grader checklist.json grades both from inside one. compare reports agent spend and grader spend rows, --max-cost 15 stops starting runs at $15, per-task rows show task names, and a fresh store gets its hook script so runs are captured. Evaluate an Agent Skill.
  • # Added eval validate flags hidden tests that name a helper or file only the reference adds and the instruction never mentions, which fail valid fixes written differently (--strict-coupling makes it blocking). eval crosscheck <task> has a reviewer model judge stored runs' diffs against the instruction and lists the runs where it disagrees with the verifier, as false rejects or false accepts. --solver <bundle> solves the task once first. The counts are stored on the task in the same shape as a human verifier audit. Evals guide
  • # Added eval audit <task> --experiment <id> --witness <file> --witness-cmd <cmd> replays a witness test, one that checks a requirement the verifier skips, on every passing run: claude-tdd: passes 6/10, refuted 3, audited pass rate 30% (3/10). The witness must pass the reference 3/3 first. A replay whose verifier no longer passes is replay-mismatch, never refuted, and a witness refuting over half the passes is ambiguous and not counted. No model calls (TestJack, arXiv 2610.10619). --promote adds it to the verifier, and eval export/import carry witnesses. Evals guide.
  • # Added Experiment sessions: exp session new --name <topic> starts one in a folder, and each exp run there prints its delta against the previous run on the same dataset as it finishes (pass 0.0% → 100.0% Δ +100.0pp […] mcnemar p=0.125, the grader deltas, the cases fixed and broken, the files changed), with no exp compare. exp session note records hypotheses and readings, exp session summary [--json] prints the timeline. exp run --watch reruns on every save of the spec, the agent script, grader files or watch globs, aborts a run made stale mid-way without storing it, and regrades the stored outputs with 0 agent calls when only the graders changed. Evals.
  • # Added An executability grader scores a run by how far it got before its first invalid tool call, one the harness refused to run (a bad argument, an unknown tool, an edit of a file not read yet, a string to replace that is not there, a missing file or command, a denied call). A test that exits 1 ran and is not invalid. The score is the share of steps before that call (its mean is AUC-PEC), and it passes with at most maxInvalid invalid calls (its pass rate is TER); kinds counts only some kinds (Prefix Executability, arXiv 2610.40444). Explore guide.
  • # Fixed eval validate applies a reference or near-miss patch after setup when it only applies to what setup creates, as an agent's run sees it. Before, a task whose setup made the mess the agent cleans up could not validate: its reference "did not apply". Evals.
  • # Fixed A command task, grader or source that crashes names the error and its cause, such as exited 1: TypeError: fetch failed (cause: connect ECONNREFUSED 127.0.0.1:8803), instead of the last three lines of stderr (for Node, the closing braces of the error object). A refused connection now reads as an infrastructure error, so the trial is retried and counted as infra, not as a failure. Evals
  • # Fixed A task drafted from a session names the files its setup was guessed from, such as setup guessed from package.json (no lockfile): npm install, instead of saying "from lockfiles" when there was none.
  • # Fixed eval run reports on a cases suite say "case" and on a triggers suite "prompt" (the header, the table, never-passed, saturation and the regression lines), and --json carries the report's unit.
  • # Fixed eval validate shows the last 12 lines of the verifier's output when a check goes the wrong way (the reference fails, the no-op or a near-miss passes, or setup fails), once per kind of check, and --json carries every step with its output in steps. Evals
  • # Fixed eval init --force no longer deletes your work: it stops when the suite holds a file eval init did not write or that you changed since (a reference.patch you added, a prompt you fixed) and lists them. --force-all deletes those too. evals/<name>/.eval-init.json records what eval init wrote. Evals
  • # Fixed eval init --skill <dir> uses the skill's own triggers.toml when it has one and you give no prompts and no --draft: suite.json points at it instead of a TODO template. Evals
  • # Fixed eval init --sessions and --session latest include a session that just ended: they wait up to 30 seconds for the ingest its hook started, then ingest the rest, instead of skipping the ingest when its lock is busy after 1.5 seconds. Evals
  • # Fixed On Windows, eval init and eval run show suite paths with forward slashes, like the file list under them: Wrote … to evals/triage, evals/pdf/triggers.toml still say TODO and the --skill path in the --draft --force hint read the same on every OS. Evals

Docs 5

  • # Added Use-case pages for tracking Claude Code and Codex cost, replaying a session, testing an AGENTS.md or skill change, building evals from failures, self-hosting, and logging model calls through the proxy. Comparison pages for Langfuse, LangSmith, Braintrust, AgentOps, Laminar and ccusage, each with a feature table, where the other tool is the better choice, and its sources. Compare.
  • # Added A Workflow experiments page: capture and score multi-agent runs, workflows compare, replays, the control fork and the replay noise floor. Workflow experiments
  • # Added A grader field reference: every grader kind, its fields and defaults, and when it passes, with the provider fields judges share. Grader fields
  • # Changed Long docs are split into pages that load faster: the CLI reference has a page per command, a long changelog release continues over numbered pages, Evals continues on Experiments and Harness evals, and the Read API has its own two pages. Old links to a section that moved still land on it. CLI reference
  • # Changed The vercel-security-audit case study has the live result: six read-only tasks against a real Vercel team, three runs per arm. Counting the 27 sessions that stayed on the requested model, graded with the fixed site check, 0/14 runs passed without the skill and 9/13 with it (+69.2 points, 95% CI +23.1 to +100, McNemar p = 0.25, gate Hold), at 6 times the cost per run. A Corrections section says what changed from the first 4/18 and 14/18: the $LIVE_SITE check, task 06's wording, and 9 sessions that Claude Code moved to an older model after a refusal. Aggregate numbers only. Case study.

Site 4

  • # Added Every docs page has its own link preview image, the home page and FAQ carry FAQ markup, and the site build fails on a missing or too-long title or description, a wrong canonical URL, or a link to a page that doesn't exist.
  • # Added Pages are ready about 0.5 s sooner on a slow phone, with half the font bytes and less layout shift. A Lighthouse A/B with budgets runs on every site change. Site speed.
  • # Added A blog at /blog: posts are Markdown files in apps/site/src/content/blog, each with its own page, title card and sitemap entry, and an Atom feed at /blog.xml. The first post explains why coding.sh keeps your sessions on your machine. Blog.
  • # Fixed A page that shows "±" no longer downloads a second font file for it. Site speed

From CHANGELOG.md · RSS · All releases