# coding.sh — full text Every page of coding.sh as plain text, for LLMs and agents. Index: https://coding.sh/llms.txt --- # coding.sh: open-source observability and evals for AI agents URL: https://coding.sh/ # See what your AI agents did, and what it cost coding.sh records every step your AI agents take, from Claude Code, Codex and Cursor to the ones you build. `curl -fsSL https://coding.sh/install | sh` [Get started](https://coding.sh/docs/quickstart) Free and open source under MIT. Your sessions stay on your machine. ![A Claude Code session in the coding.sh viewer, on the coding.sh repo: its title, agent, duration and estimated cost, then its steps in order, with what each model call cost. An npm test run is red because one test failed. A few steps later the agent edits src/store/sqlite.ts to fix it.] Works with - [Claude Code](https://coding.sh/docs/agents) - [Codex](https://coding.sh/docs/agents) - [Cursor](https://coding.sh/docs/agents) - [OpenCode](https://coding.sh/docs/agents) - [Antigravity CLI](https://coding.sh/docs/agents) - [GitHub Copilot](https://coding.sh/docs/agents) - [Amp](https://coding.sh/docs/agents) - [Goose](https://coding.sh/docs/agents) - [Pi](https://coding.sh/docs/agents) - [Factory Droid](https://coding.sh/docs/agents) - [Kiro CLI](https://coding.sh/docs/agents) - [T3 Code](https://coding.sh/docs/agents) ## How it works - 01 ### Install. Run the command above. It finds the agents you use and asks before it changes their settings. - 02 ### Keep working. Use your agents the way you do now. Each session is saved on your machine as it runs. - 03 ### Open the viewer. Run `coding.sh ui` to see every session and where it broke. Record ## Find the step that broke Open any session to see each prompt, tool call and file edit in order. When a command fails, the lines that say why are marked red. ![The failed npm test step in the coding.sh viewer, with the lines that say why in red: the test applies each migration once failed with SqliteError: duplicate column name: cost_usd, thrown at src/store/sqlite.ts line 428.] Measure ## See what each agent costs Run `coding.sh usage` to add up tokens and dollars by agent, model, repo or day. A `~` means the cost is estimated from list prices. [Track cost per session](https://coding.sh/docs/use-cases/claude-code-cost) [Compared with ccusage](https://coding.sh/docs/compare/ccusage) ~/coding.sh ``` $ coding.sh usage --since 7d AGENT SESSIONS TOKENS COST claude-code 29 9.8M ~$6.92 codex 14 4.0M ~$2.08 opencode 5 1.1M ~$1.07 ``` Compare ## You changed AGENTS.md. Did your agent get better? Run the same 20 tasks on the old and new version, 5 times each. coding.sh tells you which one passed more often. [Test an AGENTS.md or skill change](https://coding.sh/docs/use-cases/test-agent-instructions) tasks passed Tasks passed went from 64% before to 79% after. Better by 15 points. That's more than luck would explain. biggest gain Retry when the database is locked: 1 of 5 runs passed before, 5 of 5 after [How we check it's not luck](https://coding.sh/docs/experiments#compare) Production ## Track the AI calls in your app Wrap your OpenAI or Anthropic client with the SDK. Each request shows up with its model, tokens, cost and time. A failed call is marked red, with the error that came back. ``` import { observe } from "@codingdotsh/sdk"; const openai = observe(new OpenAI()); ``` [Read the SDK guide](https://github.com/jackulau/coding.sh/tree/main/packages/sdk#readme) [Python SDK](https://github.com/jackulau/coding.sh/tree/main/packages/sdk-python#readme) ![The coding.sh serve viewer listing requests from coding-sh-triage, a bot that labels issues on the coding.sh repo, each with its model, tokens, estimated cost and time.] ## Questions Does anything leave my machine? No. Sessions are saved in ~/.coding.sh. Nothing is sent anywhere unless you set up an export yourself. Will it slow my agent down? No. The hook adds one line to a file and exits. The rest of the work happens after the session. Does it change my agent's settings? Only after you say yes. It backs up each file first, and coding.sh uninstall takes its hooks out again. What does it cost? Nothing. It is open source under the MIT license. Your model spend stays what your agents already cost. How do you say coding.sh? “coding shell” — the .sh is a shell script. People also write it coding sh or codingsh; the CLI runs as coding.sh or codingsh. It is all one tool. [More questions](https://coding.sh/docs/faq) [How it compares with Langfuse, LangSmith and others](https://coding.sh/docs/compare) ## Try it on your next session `curl -fsSL https://coding.sh/install | sh` [Get started](https://coding.sh/docs/quickstart) --- # Blog · coding.sh URL: https://coding.sh/blog # Blog Notes on how coding.sh works and why, newest first. 2 posts · [RSS](https://coding.sh/blog.xml) Nov 8, 2026 ## [Coding agents run in your shell](https://coding.sh/blog/agents-run-in-your-shell) Claude Code, Codex and Cursor do their real work as shell commands. coding.sh captures them at the shell, through a hook under 50 ms. [Read the post](https://coding.sh/blog/agents-run-in-your-shell) Nov 7, 2026 ## [Why coding.sh is local-first](https://coding.sh/blog/why-local-first) Agent sessions hold your prompts, your code and sometimes your keys, so coding.sh keeps them on your machine unless you choose to send them. [Read the post](https://coding.sh/blog/why-local-first) --- # Coding agents run in your shell · coding.sh blog URL: https://coding.sh/blog/agents-run-in-your-shell [Blog](https://coding.sh/blog) Nov 8, 2026 # Coding agents run in your shell Claude Code, Codex and Cursor do their real work as shell commands. coding.sh captures them at the shell, through a hook under 50 ms. The `.sh` in coding.sh is not a pun. A coding agent does its real work in a shell. It reads files, runs tests, greps the tree, stages a commit, starts a server — all of it as shell commands the model decides to run. The model's prose is a plan; the shell is where the plan becomes an action that can pass, fail, or change your repo. That is the layer worth watching, and it is the layer coding.sh records. ## Where the work and the risk are Open any agent session and the shape is the same. A few prompts, a lot of tool calls, and most of the tool calls are shell commands. `npm test`. `git diff`. `rg TODO`. A one-liner that edits a file in place. The interesting moments are all here: the command that failed and got retried without a change, the test that went green only after a test file was edited, the `rm` that was wider than it looked, the key that ended up on a command line. coding.sh reads each shell command the agent ran, pairs it with its result, and keeps the diff of what changed. From that it can tell you plain things that are tedious to reconstruct by hand: which commands failed, which ones changed files without a test running afterward, and whether a passing test run was earned or edited into passing. ## A hook, not an integration coding.sh captures through a hook — a small process the agent's own harness runs after each step. The rules it follows are shell-citizen rules. It exits 0 so it can never break the agent. It prints nothing unless the protocol needs a reply. It does no network and no git work. It writes the raw payload to a spool file and returns, well under 50 ms, so the agent never waits on it. That is the whole install for Claude Code, Codex, Gemini CLI and the rest: a hook line in the agent's config, added by `coding.sh init`. No SDK, no wrapper around your commands, no proxy in the default path. The agent keeps running exactly as it did. ## It is a CLI, so it lives where you do The tool is a command. You read sessions, usage and diffs in your terminal, next to the agent that produced them. `coding.sh sessions`, `coding.sh usage --by model`, `coding.sh show `. The viewer is the same binary: `coding.sh ui` serves a local page on 127.0.0.1 when you want to click through a replay, and closes when you are done. Underneath, coding.sh has zero runtime dependencies. Everything is Node's own built-ins — the SQLite index through `node:sqlite`, compression through `node:zlib`. Install is one line: shell ``` curl -fsSL https://coding.sh/install.sh | sh ``` A tool for agents that work in the shell should be a shell tool. That is the one coding.sh tries to be. [Older post Why coding.sh is local-first](https://coding.sh/blog/why-local-first) [All posts](https://coding.sh/blog) · [RSS](https://coding.sh/blog.xml) --- # Why coding.sh is local-first · coding.sh blog URL: https://coding.sh/blog/why-local-first [Blog](https://coding.sh/blog) Nov 7, 2026 # Why coding.sh is local-first Agent sessions hold your prompts, your code and sometimes your keys, so coding.sh keeps them on your machine unless you choose to send them. An agent session is a record of your work. It holds your prompts, the model's replies, every tool call with its input and output, and the files that changed. Sometimes it also holds a key that ended up in a shell command. coding.sh records all of that, so where the record lives matters. ## One folder on your machine Everything coding.sh captures goes into `~/.coding.sh`. Set `CODING_SH_HOME` to put it somewhere else. Sessions and events go into a SQLite index, `index.db`, through Node's built-in `node:sqlite`. Large prompts and tool output go into `blobs/`, compressed with zstd. The viewer runs from the same CLI. It binds to 127.0.0.1 and asks for the token in the URL it prints. ## Nothing is sent unless you ask With no sinks configured, no session data leaves your machine. The one default network request is a daily update check: a GET to `api.github.com`, falling back to `registry.npmjs.org`. Set `CODING_SH_NO_UPDATE_CHECK=1` or `[update] check = false` to turn it off. The hook that agents run writes the raw payload to `~/.coding.sh/spool` with mode 600 and exits. It does no network work. To send events somewhere else, add a sink. A sink sends metadata unless you set `mode = "full"`: ids, timestamps, kinds, durations, token counts and short labels. Prompts, code and tool output stay on your machine. config.toml ``` # ~/.coding.sh/config.toml [[sinks]] kind = "otlp" # or "http" endpoint = "http://localhost:4318" mode = "metadata" # default; "full" sends content ``` ## Secrets are redacted before they are written Redaction runs at ingest, before anything reaches the index, the blobs or a sink. The built-in rules catch private keys, vendor API keys and tokens, JWTs, and bearer and basic auth. Redaction keeps a secret out of the store, but the agent and its model provider still saw it. So a session that held a vendor token gets the `secret:exposed` tag, and this lists the sessions whose credentials to rotate: shell ``` coding.sh sessions --flag secret:exposed ``` If a value got past redaction, `coding.sh purge --literal leaked.txt` replaces every stored copy with `[redacted:purged]`. ## A cloned repo cannot widen it A `.coding.sh/config.toml` inside a repository can add redaction rules or switch capture to metadata mode. It cannot add sinks or move the store. Opening a repository you cloned does not send your sessions anywhere. ## Read more - [Storage, sinks and privacy](https://github.com/jackulau/coding.sh/blob/main/docs/guides/privacy.md) lists everything that is captured and every setting that changes it. - [Concepts](https://coding.sh/docs/concepts) defines the words coding.sh uses: sessions, events, flags and the rest. [Newer post Coding agents run in your shell](https://coding.sh/blog/agents-run-in-your-shell) [All posts](https://coding.sh/blog) · [RSS](https://coding.sh/blog.xml) --- # Changelog · coding.sh URL: https://coding.sh/changelog # Changelog 7 releases since Nov 6, 2026 · [RSS](https://coding.sh/changelog.xml) · from [CHANGELOG.md](https://github.com/jackulau/coding.sh/blob/main/CHANGELOG.md) Unreleased ## [v0.1.6](https://coding.sh/changelog/v0-1-6) [Read the notes](https://coding.sh/changelog/v0-1-6) 146 changes in v0.1.6 - [Privacy 5](https://coding.sh/changelog/v0-1-6#privacy) - [Trace format 1](https://coding.sh/changelog/v0-1-6#trace-format) - [Serve 2](https://coding.sh/changelog/v0-1-6#serve) - [Sinks 6](https://coding.sh/changelog/v0-1-6#sinks) - [Evals 11](https://coding.sh/changelog/v0-1-6#evals) - [Tool evals 6](https://coding.sh/changelog/v0-1-6#tool-evals) - [Cache 1](https://coding.sh/changelog/v0-1-6#cache) - [Experiments 12](https://coding.sh/changelog/v0-1-6#experiments) - [Refine 10](https://coding.sh/changelog/v0-1-6/2#refine) - [Export 2](https://coding.sh/changelog/v0-1-6/2#export) - [Capture 14](https://coding.sh/changelog/v0-1-6/2#capture) - [Explore 1](https://coding.sh/changelog/v0-1-6/2#explore) - [Viewer 3](https://coding.sh/changelog/v0-1-6/2#viewer) - [API 1](https://coding.sh/changelog/v0-1-6/2#api) - [Harness A/B 5](https://coding.sh/changelog/v0-1-6/2#harness-a-b) - [Attribution 4](https://coding.sh/changelog/v0-1-6/2#attribution) - [Judges 7](https://coding.sh/changelog/v0-1-6/2#judges) - [Compare 2](https://coding.sh/changelog/v0-1-6/2#compare) - [Cost 9](https://coding.sh/changelog/v0-1-6/3#cost) - [OpenTelemetry 4](https://coding.sh/changelog/v0-1-6/3#opentelemetry) - [Collector 4](https://coding.sh/changelog/v0-1-6/3#collector) - [Docs 5](https://coding.sh/changelog/v0-1-6/3#docs) - [Benchmarks 4](https://coding.sh/changelog/v0-1-6/3#benchmarks) - [Sessions 4](https://coding.sh/changelog/v0-1-6/3#sessions) - [Workflows 4](https://coding.sh/changelog/v0-1-6/3#workflows) - [Store 6](https://coding.sh/changelog/v0-1-6/3#store) - [Improve 2](https://coding.sh/changelog/v0-1-6/3#improve) - [Search 1](https://coding.sh/changelog/v0-1-6/3#search) - [CLI 2](https://coding.sh/changelog/v0-1-6/4#cli) - [Query 2](https://coding.sh/changelog/v0-1-6/4#query) - [Harness 1](https://coding.sh/changelog/v0-1-6/4#harness) - [Doctor 1](https://coding.sh/changelog/v0-1-6/4#doctor) - [Gateways 4](https://coding.sh/changelog/v0-1-6/4#gateways) Nov 9, 2026 ## [v0.1.5](https://coding.sh/changelog/v0-1-5) [Read the notes](https://coding.sh/changelog/v0-1-5) 19 changes in v0.1.5 - [Sessions 5](https://coding.sh/changelog/v0-1-5#sessions) - [Refine 2](https://coding.sh/changelog/v0-1-5#refine) - [CLI 1](https://coding.sh/changelog/v0-1-5#cli) - [Store 2](https://coding.sh/changelog/v0-1-5#store) - [Evals 2](https://coding.sh/changelog/v0-1-5#evals) - [Privacy 2](https://coding.sh/changelog/v0-1-5#privacy) - [Workflows 1](https://coding.sh/changelog/v0-1-5#workflows) - [Capture 1](https://coding.sh/changelog/v0-1-5#capture) - [Cost 1](https://coding.sh/changelog/v0-1-5#cost) - [Harness A/B 2](https://coding.sh/changelog/v0-1-5#harness-a-b) Nov 8, 2026 ## [v0.1.4](https://coding.sh/changelog/v0-1-4) The 0.1.3 changes, published to npm. [Read the notes](https://coding.sh/changelog/v0-1-4) 1 change in v0.1.4 - [Release 1](https://coding.sh/changelog/v0-1-4#release) Nov 8, 2026 ## [v0.1.3](https://coding.sh/changelog/v0-1-3) Sessions read back from Claude Code and Codex transcripts keep more of what the agent recorded. [Read the notes](https://coding.sh/changelog/v0-1-3) 45 changes in v0.1.3 - [Docs 1](https://coding.sh/changelog/v0-1-3#docs) - [Workflows 8](https://coding.sh/changelog/v0-1-3#workflows) - [Improve 4](https://coding.sh/changelog/v0-1-3#improve) - [Sessions 3](https://coding.sh/changelog/v0-1-3#sessions) - [Capture 9](https://coding.sh/changelog/v0-1-3#capture) - [Cost 2](https://coding.sh/changelog/v0-1-3#cost) - [CI 1](https://coding.sh/changelog/v0-1-3#ci) - [MCP 1](https://coding.sh/changelog/v0-1-3#mcp) - [Harness 1](https://coding.sh/changelog/v0-1-3#harness) - [Refine 2](https://coding.sh/changelog/v0-1-3#refine) - [Cache 2](https://coding.sh/changelog/v0-1-3#cache) - [Site 1](https://coding.sh/changelog/v0-1-3#site) - [Setup 4](https://coding.sh/changelog/v0-1-3#setup) - [Evals 2](https://coding.sh/changelog/v0-1-3#evals) - [Storage 1](https://coding.sh/changelog/v0-1-3#storage) - [Proxy 2](https://coding.sh/changelog/v0-1-3#proxy) - [Ingest 1](https://coding.sh/changelog/v0-1-3/2#ingest) Nov 8, 2026 ## [v0.1.2](https://coding.sh/changelog/v0-1-2) coding.sh runs on Windows, and CI tests it there. [Read the notes](https://coding.sh/changelog/v0-1-2) 177 changes in v0.1.2 - [Sessions 15](https://coding.sh/changelog/v0-1-2#sessions) - [Evals 22](https://coding.sh/changelog/v0-1-2#evals) - [Docs 5](https://coding.sh/changelog/v0-1-2#docs) - [Site 4](https://coding.sh/changelog/v0-1-2#site) - [Capture 12](https://coding.sh/changelog/v0-1-2/2#capture) - [Examples 4](https://coding.sh/changelog/v0-1-2/2#examples) - [Improve 3](https://coding.sh/changelog/v0-1-2/2#improve) - [MCP 4](https://coding.sh/changelog/v0-1-2/2#mcp) - [Harness 2](https://coding.sh/changelog/v0-1-2/2#harness) - [Judges 11](https://coding.sh/changelog/v0-1-2/2#judges) - [Stores 1](https://coding.sh/changelog/v0-1-2/2#stores) - [Harness A/B 16](https://coding.sh/changelog/v0-1-2/3#harness-a-b) - [Skills 1](https://coding.sh/changelog/v0-1-2/3#skills) - [Self-host 2](https://coding.sh/changelog/v0-1-2/3#self-host) - [Replay 2](https://coding.sh/changelog/v0-1-2/3#replay) - [Install 3](https://coding.sh/changelog/v0-1-2/3#install) - [Serve 1](https://coding.sh/changelog/v0-1-2/3#serve) - [Experiments 23](https://coding.sh/changelog/v0-1-2/4#experiments) - [Gateways 1](https://coding.sh/changelog/v0-1-2/4#gateways) - [SDKs 1](https://coding.sh/changelog/v0-1-2/4#sdks) - [Durable workflows 1](https://coding.sh/changelog/v0-1-2/4#durable-workflows) - [Viewer 8](https://coding.sh/changelog/v0-1-2/4#viewer) - [Cost 5](https://coding.sh/changelog/v0-1-2/4#cost) - [Workflows 2](https://coding.sh/changelog/v0-1-2/4#workflows) - [Production 1](https://coding.sh/changelog/v0-1-2/4#production) - [Refine 4](https://coding.sh/changelog/v0-1-2/5#refine) - [Long sessions 1](https://coding.sh/changelog/v0-1-2/5#long-sessions) - [Tool evals 2](https://coding.sh/changelog/v0-1-2/5#tool-evals) - [Graders 2](https://coding.sh/changelog/v0-1-2/5#graders) - [Query 3](https://coding.sh/changelog/v0-1-2/5#query) - [Privacy 4](https://coding.sh/changelog/v0-1-2/5#privacy) - [Compare 2](https://coding.sh/changelog/v0-1-2/5#compare) - [SDK 1](https://coding.sh/changelog/v0-1-2/5#sdk) - [Rules 1](https://coding.sh/changelog/v0-1-2/5#rules) - [Doctor 1](https://coding.sh/changelog/v0-1-2/5#doctor) - [Windows 2](https://coding.sh/changelog/v0-1-2/5#windows) - [Aider 1](https://coding.sh/changelog/v0-1-2/5#aider) - [Codex 1](https://coding.sh/changelog/v0-1-2/5#codex) - [CLI 1](https://coding.sh/changelog/v0-1-2/5#cli) - [Store 1](https://coding.sh/changelog/v0-1-2/5#store) Nov 7, 2026 ## [v0.1.1](https://coding.sh/changelog/v0-1-1) Daily benchmarks for any AI agent: `coding.sh bench` runs every contender on fresh cases each day, and experiments add cluster bootstrap, an A/A noise floor, interleaved runs, early stopping and `compare --without`. Seven more agents are verified on a real install, the viewer and the site work on phones, and the installer uses npm once the package is there. [Read the notes](https://coding.sh/changelog/v0-1-1) 38 changes in v0.1.1 - [Benchmarks 6](https://coding.sh/changelog/v0-1-1#benchmarks) - [MCP 1](https://coding.sh/changelog/v0-1-1#mcp) - [Experiments 8](https://coding.sh/changelog/v0-1-1#experiments) - [Compare 3](https://coding.sh/changelog/v0-1-1#compare) - [Agents 2](https://coding.sh/changelog/v0-1-1#agents) - [Viewer 2](https://coding.sh/changelog/v0-1-1#viewer) - [Site 2](https://coding.sh/changelog/v0-1-1#site) - [Install 1](https://coding.sh/changelog/v0-1-1#install) - [Release 2](https://coding.sh/changelog/v0-1-1#release) - [Docs 1](https://coding.sh/changelog/v0-1-1#docs) - [Fixes 10](https://coding.sh/changelog/v0-1-1#fixes) Nov 6, 2026 ## [v0.1.0](https://coding.sh/changelog/v0-1-0) The first public release, still pre-1.0. coding.sh records what your AI agents do, with coding agents first-class, on your own machine, shows it in a local viewer, and turns those sessions into evals and A/B experiments. It also runs as a server on SQLite or Postgres, with SDKs for apps that call model APIs. [Read the notes](https://coding.sh/changelog/v0-1-0) 63 changes in v0.1.0 - [Install 6](https://coding.sh/changelog/v0-1-0#install) - [Capture 13](https://coding.sh/changelog/v0-1-0#capture) - [Agents 5](https://coding.sh/changelog/v0-1-0#agents) - [Cost 3](https://coding.sh/changelog/v0-1-0#cost) - [Storage 2](https://coding.sh/changelog/v0-1-0#storage) - [Viewer 1](https://coding.sh/changelog/v0-1-0#viewer) - [MCP 2](https://coding.sh/changelog/v0-1-0#mcp) - [CLI 1](https://coding.sh/changelog/v0-1-0#cli) - [Evals 7](https://coding.sh/changelog/v0-1-0#evals) - [Production 13](https://coding.sh/changelog/v0-1-0#production) - [SDKs 3](https://coding.sh/changelog/v0-1-0#sdks) - [Integrations 5](https://coding.sh/changelog/v0-1-0#integrations) - [Docs 2](https://coding.sh/changelog/v0-1-0#docs) --- # coding.sh v0.1.0 · Changelog URL: https://coding.sh/changelog/v0-1-0 [Changelog](https://coding.sh/changelog) Nov 6, 2026 · 63 changes # coding.sh v0.1.0 The first public release, still pre-1.0. coding.sh records what your AI agents do, with coding agents first-class, on your own machine, shows it in a local viewer, and turns those sessions into evals and A/B experiments. It also runs as a server on SQLite or Postgres, with SDKs for apps that call model APIs. ## Install - `curl -fsSL https://coding.sh/install | sh` checks for Node 22.15+, installs with npm, runs `init` and opens the viewer. - Or `npm install -g coding.sh && coding.sh init`. `init` shows one plan and asks one Y/n. [Install guide](https://coding.sh/docs/install). - `coding.sh` with no command opens the viewer, and sets up capture on its first run. - Commands print one dim line when a release is out, checked in the background at most once a day. - `coding.sh update` installs the new release. The check is off in CI and [can be turned off](https://github.com/jackulau/coding.sh/blob/main/docs/guides/install.md). - `coding.sh uninstall` restores each agent config byte for byte if nothing else has edited it since. ## Capture - Hooks record prompts, tool calls, file changes, turns and errors to a local spool, then ingest them into sessions. - Secrets are redacted before anything is stored. [Privacy](https://github.com/jackulau/coding.sh/blob/main/docs/guides/privacy.md). - `run` spools secret flag values (`--openai-api-key X`, `--api-key=X`) redacted, and ingest redacts them in any argv. - Pi, Goose, Copilot CLI, Droid, Gemini CLI and Antigravity sessions seen by hooks and by `ingest --backfill` store each call, reply and turn once, and count tokens once. - A session seen by hooks and by `ingest --backfill` keeps one session start and one end, for every agent. - A Claude Code or Codex compaction seen by hooks and by `ingest --backfill` is stored once; Codex transcripts no longer record each compaction twice. - Cursor prompts and replies seen by hooks and by `ingest --backfill` are stored once, though Cursor transcripts carry no timestamps. - `coding.sh run -- ` wraps any agent and snapshots the tree before and after in shadow git, not your repo. - Every model call becomes an `llm.call` event with model, tokens, duration and time to first token. - Subagent starts and ends are captured and linked to their child sessions. - No ingest step: reads, the viewer and MCP ingest new events first; a session end starts a locked background ingest. - `coding.sh collector` takes OTLP/HTTP traces and logs, protobuf or JSON, on 127.0.0.1:4318. [OTLP setup](https://coding.sh/docs/otel). - The receiver reads OTel GenAI semconv, OpenInference, the Vercel AI SDK, and Claude Code, Codex and Gemini CLI logs. ## Agents - Verified on real sessions: Claude Code, Codex and Antigravity. [Agent matrix](https://coding.sh/docs/agents). - Fixture-tested so far: opencode, cursor, gemini-cli, copilot-cli, amp, pi, droid, goose, t3code and aider. - Every adapter has fixture tests, and an `ingest --backfill` path where the agent keeps history. - T3 Code threads import from its local store; Codex and Claude Code runs in T3 count once, agent in `provider`. - `coding.sh usage --by agent-model` compares tokens per turn and cache-read share across harnesses. ## Cost - Calls with tokens and no cost get one from a dated price table (Claude, OpenAI, Gemini, open models), marked estimated; billed and reported costs win. [Where cost comes from](https://github.com/jackulau/coding.sh/blob/main/docs/how-to/expensive-sessions.md#where-cost-comes-from). - `[prices.""]` in config.toml sets your own rates; `usage --reprice` recomputes stored estimates. - A model call nested in another, or a repeated response id, counts once in session totals. ## Storage - One SQLite index (node:sqlite, no native deps) and a content-addressed, zstd-compressed blob store. [Storage](https://coding.sh/docs/storage). - OTLP and HTTP sinks send metadata only by default. ## Viewer - `coding.sh ui` serves a local viewer: sessions, search, a timeline of turns, tools and subagents, and outcome labels. ## MCP - `coding.sh mcp` lets agents query sessions, failures, latency, usage, subagents and experiments over stdio. [MCP](https://coding.sh/docs/mcp). - `init` registers it with Claude Code, Codex, Cursor, OpenCode and Gemini CLI. `--no-mcp` skips that. ## CLI - 27 commands, from `init` and `sessions` to `eval`, `exp` and `serve`, all with `--json` output. [CLI reference](https://coding.sh/docs/cli). ## Evals - Repo-replay tasks start from the snapshot taken before the agent ran. [Evals](https://coding.sh/docs/evals). - `eval validate` checks verifiers: a do-nothing run fails, the reference fix passes k of k, near-misses fail, nothing leaks. - Datasets with deterministic graders, command graders and LLM judges. - Judges are calibrated on human labels with Cohen's kappa on a held-out split. Reports flag uncalibrated judges. - Experiments run two arms on the same cases: a paired bootstrap CI of the difference and exact McNemar on flips. - Harness compare adds per-arm Wilson intervals and the first step where the traces diverge. - [Tool evals](https://coding.sh/docs/tool-evals) for CLI and MCP surfaces: description size, schema checks, per-task tool-call metrics in a sandbox. ## Production - `coding.sh serve` runs ingest (OTLP/HTTP and `POST /v1/ingest`), the viewer and its API on one port. [Production](https://coding.sh/docs/production). - Set the store with `CODING_SH_STORE_URL` or `DATABASE_URL`, and blobs with `CODING_SH_BLOB_URL` (file, postgres, s3). - Writes are acked after commit, a full queue answers 429 with `Retry-After`, and retries never duplicate. - Ingest keys pick the project and read keys open the viewer. `serve keygen` and `serve healthcheck` help. - `/healthz`, `/readyz`, Prometheus `/metrics`, and one writer lock with `--role viewer` replicas. - `CODING_SH_RETENTION_DAYS` sets retention and `CODING_SH_CAPTURE_MODE=metadata` drops message text and tool I/O. - `coding.sh-store-postgres` keeps the index, and blobs by default, in Postgres. Passes the store conformance suite. - Store URLs take `sqlite://`, `postgres://` or `postgresql://`. - `coding.sh-store-s3` keeps blobs in S3, R2 or MinIO. SigV4, no AWS SDK, credentials only from `AWS_*` env vars. - A Dockerfile with both store plugins, a Compose stack (serve, Postgres, MinIO), and examples in examples/production. - Deploy recipes for Fly, Railway and Render. [Deploy guides](https://github.com/jackulau/coding.sh/tree/main/docs/production). - `--store ` or `CODING_SH_STORE_URL` points `sessions`, `show`, `search`, `mcp`, `dataset` and `exp` at the serve database. - `sessions` and `search` take `--project`. ## SDKs - `@codingdotsh/sdk` for TypeScript: `observe(client)` for OpenAI and Anthropic, `observeFetch()`, `aiSdkTracer()`, `withSession`. - `coding-sh` for Python: `observe(client)` and OTel setup. - Both batch, retry on 429 and never throw into your app. Not on npm or PyPI yet. ## Integrations - Sessions sent through the Vercel AI Gateway are tagged `gateway:vercel`. [AI Gateway](https://coding.sh/docs/ai-gateway). - `coding.sh gateway status|enrich` adds billed cost, serving provider, client session join and the `vmc/` model per call. - The receiver maps gateway trace drains and AI SDK `gateway` spans. - `compare --gateway` and the `gateway` grader provider run arms across models with one key. - eve: span enrichment, a read-only importer for its local traces, and an experiment task that drives a local agent. ## Docs - README with a quickstart, the agent matrix, OTLP setup, storage, privacy and uninstall. - CONTRIBUTING, SECURITY and a release checklist in docs/RELEASE.md. [Newer release v0.1.1](https://coding.sh/changelog/v0-1-1) From [CHANGELOG.md](https://github.com/jackulau/coding.sh/blob/main/CHANGELOG.md) · [RSS](https://coding.sh/changelog.xml) · [All releases](https://coding.sh/changelog) --- # coding.sh v0.1.1 · Changelog URL: https://coding.sh/changelog/v0-1-1 [Changelog](https://coding.sh/changelog) Nov 7, 2026 · 38 changes # coding.sh v0.1.1 Daily benchmarks for any AI agent: `coding.sh bench` runs every contender on fresh cases each day, and experiments add cluster bootstrap, an A/A noise floor, interleaved runs, early stopping and `compare --without`. Seven more agents are verified on a real install, the viewer and the site work on phones, and the installer uses npm once the package is there. ## Benchmarks - Added `coding.sh bench` samples fresh cases each day from a command, failed sessions, GitHub issues, a sitemap or a dataset, freezes them as a dataset version and runs every contender on that set in random order. [Benchmarks](https://github.com/jackulau/coding.sh/blob/main/docs/guides/benchmarks.md). - Added Shared timeout and retries, a budget cap, quota and rate-limit cases dropped for every contender, and `bench run --resume` to finish a stopped run with the same seed and cases. - Added `bench report` writes a static HTML page and JSON: trend, gaps to a reference pooled over a fixed window, head-to-head stats, difficulty and failures. `bench regrade` re-scores stored outputs and `bench schedule` prints a cron line and a GitHub Actions workflow. - Added An `[anchor]` set of fixed cases runs every day next to the fresh ones and is scored apart, so its trend has no day-to-day difficulty in it. - Added `seed = "secret"` publishes the hash of each next day's salt and `bench show` checks the chain. Content hashes keep the same case under a new key from counting as fresh. - Added Input and output tokens are kept apart, and each cost says where it came from: reported, `costPerRun`, the shared price table, or unknown. ## MCP - Added A `benchmarks` tool lists benchmarks and reads results and history. ## Experiments - Added `exp compare --cluster ` resamples by repo or session and warns under 10 clusters. [Evals](https://coding.sh/docs/evals). - Added `exp aa` measures the run-to-run noise floor, and power planning uses it. - Added Compare flags uneven attrition between arms (Fisher exact) and splits infra errors from agent errors. - Added Under 100 paired cases, compare also reports P(B>A) from a paired posterior. - Added `exp run --arm a= --arm b=` interleaves both arms case by case in a seeded random order. - Added `--stop-early` checks at planned looks with O'Brien-Fleming alpha spending and reports a repeated confidence interval. - Added Pairwise judges see both outputs in both orders. `exp compare --pairwise` reports win rate, ties and position bias. - Added `exp sweep` races task configs on a train split and reports the winner's optimism gap on held-out cases. ## Compare - Added `compare --without ` measures what one skill, AGENTS.md or MCP config is worth. [Compare agents](https://coding.sh/docs/how-to/compare-agents). - Added With 2+ attempts per task, failed runs split into bad runs, arm limits and tasks no arm solves, with variance shares. - Added Cost rows show how many runs had a known cost, and "cheaper" needs a known cost for 80% of each arm. ## Agents - Added Verified on a real install against a local stub model: opencode, copilot-cli, droid, goose, pi, t3code and aider. [Agent matrix](https://coding.sh/docs/agents). - Added Antigravity backfill reads agy transcripts and conversation DBs for model, tokens, cache reads and latency per call. Tool steps missing from the transcript count as failures, and hooks reply `{}`. ## Viewer - Added Phone layouts: 44px targets, filters in a bottom sheet, a compact session header and safe-area insets. - Added The Benchmarks tab shows anchors, pooled gaps, difficulty, cost sources and tokens, and ranks follow the pooled gaps. ## Site - Added Mobile pass: touch targets, safe areas, stacked agent cards and 16px body text on phones. - Added A new landing page, a changelog page and docs pages. ## Install - Added `install.sh` installs from npm when the registry has the version, else from the GitHub release. `CODING_SH_SOURCE=npm|github` picks one. ## Release - Added A tag creates its GitHub release as soon as the checks pass, so the installer gets it right away. - Added The release workflow can publish all five packages to npm and PyPI with trusted publishing. Nothing is on npm or PyPI yet. ## Docs - Added A new README, reference pages, how-to guides and a code of conduct. ## Fixes - Fixed Safer installs: symlinked and JSONC agent configs survive, `init --dry-run` prints a short plan, and blockers show before the Y/n. - Fixed Each SQLite migration runs once when the hook and the CLI open a new index at the same time. - Fixed Experiments stop cleanly on Ctrl-C, kill their child processes and keep every finished run. - Fixed Capture, ingest and storage state bugs from an audit of each state machine. Sessions are priced locally at ingest. - Fixed Adapter fixes for opencode, goose, copilot-cli, droid, aider and t3code, found running each on a real install. - Fixed `coding.sh run --agent ` records sessions for names with no adapter. - Fixed `exp aa` refuses to compare a run with itself. - Fixed `coding.sh ui` prints the viewer URL once in a terminal. - Fixed Muted text in the viewer passes WCAG AA in both themes. - Fixed Timing tests compare against a baseline, so a loaded machine no longer fails them. [Newer release v0.1.2](https://coding.sh/changelog/v0-1-2) [Older release v0.1.0](https://coding.sh/changelog/v0-1-0) From [CHANGELOG.md](https://github.com/jackulau/coding.sh/blob/main/CHANGELOG.md) · [RSS](https://coding.sh/changelog.xml) · [All releases](https://coding.sh/changelog) --- # coding.sh v0.1.2 · Changelog URL: https://coding.sh/changelog/v0-1-2 [Changelog](https://coding.sh/changelog) Nov 8, 2026 · 177 changes · page 1 of 5 # coding.sh v0.1.2 coding.sh runs on Windows, and CI tests it there. ## Sessions - Added A new `text-tool-call` flag marks a session whose model wrote a tool call out as text, so the call never ran. In Claude Code this is usually a subagent whose `tools:` frontmatter is a map (`edit: false`), which leaves it with no tools; `improve` names the fix. [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md#recording). - Added A session whose main thread moved to another model mid-session, such as Claude Code's silent fallback to an older model after a refusal, gets the `model:switch` tag and `stats.modelSwitch` with both models, when, and the trigger (`refusal`, `error` or `/model`). `stats.models` counts calls per model (both show in a Models row in `show` and the MCP digest), a refused call adds the `refusal` flag, and a session's model is now the one most of its main-thread calls ran on instead of the last one seen. Helper models (Haiku titles) and subagents are not a switch. [Read API](https://github.com/jackulau/coding.sh/blob/main/docs/reference/api.md#get-apisessionsid). - Added Text that Claude Code hooks add to the context (a plugin's SessionStart notes, a UserPromptSubmit reminder, PostToolUse additionalContext) is stored as `hook.context` events with its hook event, source and size. `stats.hookContext` counts the tokens by source, each source tags the session (`hook:plugin:terse`, `hook:settings`), `sessions --where hook:plugin:` or `hook.tokens>500` filters on it, `--facets hook` counts sessions per source, and `show` lists it. [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md). - Added Headless sessions (`claude -p`, an Agent SDK, `codex exec`) are marked `stats.launch: "headless"`, and terminal or IDE sessions `"interactive"`, from Claude Code's entrypoint and Codex's originator. `sessions --where launch:headless` filters on it and `--facets launch` counts it. [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md). - Added `show` has a `Flags` row, and `sessions` ends with how many listed sessions are flagged and a pointer to `coding.sh improve`. - Added `show #n` (or a call id) prints one model call's system prompt, tools, settings, cache use and cost, and `diff-calls` compares two calls in prompt-cache order and names what broke the cache. The viewer shows both on a proxy call. [Capture with the proxy](https://coding.sh/docs/how-to/capture-with-proxy#4-why-did-the-cache-miss). - Added `replay --step N --fork` sends the model call behind step N again with a changed AGENTS.md, rule, skill, system prompt or model (`--with`, `--harness`, `--model`, `--system`, `-k`) and prints the recorded next action beside the forked one. `--run` carries on with Claude Code in a copy of the repo, the stored history spliced into its requests, and `--verify` runs a check there. It needs a session captured through the proxy. The viewer's Replay pane has a "Fork from here" tab, and the MCP `session_replay` tool takes `fork`. [Explore sessions](https://github.com/jackulau/coding.sh/blob/main/docs/guides/explore.md#fork-from-a-step). - Added Each skill a session used, through the Skill tool or by reading or running a file in its folder, adds a `skill:` tag, so `sessions --flag skill:pdf` finds those sessions. `show` lists them as `Skills pdf (project, invoked, first, before any edit) · pdf-tools (personal, read)`, and `show --json` and MCP `session_get` return `skills`. A new `skill-copy-conflict` flag marks a session that used two copies of one skill (arXiv 2610.11647). [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md). - Added New `test-tamper` flag: the agent added a skip marker to a test, deleted a test file while tests failed, or changed an assertion and that edit alone turned a failing run green. It reads Edit/Write/apply_patch changes and shell writes (`sed -i`, `>`, `tee`, `rm`, `mv`, `git checkout`), and covers test-runner configs and CI files. New tests and files the agent created itself are not flagged. On 3,957 local sessions it flags 13 (EPISTEME, arXiv 2611.06563). [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md). - Fixed Replay knows that a file a step created did not exist before that step, instead of calling it unknown, so `replay --fork --run` removes it from the fork's copy. [Replay a session](https://coding.sh/docs/use-cases/replay-agent-session#how-files-are-rebuilt). - Fixed `premature-completion` no longer flags an agent that stopped on a failing test run and said so ("2 tests still fail", "I couldn't get test_x to pass", or a question about the failing test). It still flags one that says it is done, fixed or passing, or says nothing about the failure. Stored sessions get their flags recomputed once. [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md). - Fixed Claude Code subagents get a title and a turn: the title is the Agent tool's `description`, else the first line of the prompt, and the prompt is recorded as the subagent's first message. Claude Code's internal forks (the compaction summarizer, prompt suggestions) no longer leave empty subagent sessions with no model, tools or title. Stored sessions are parsed again on the next ingest. [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md). - Fixed `skill:` tags and `skill-copy-conflict` count only installed skills: a `skills/` folder under an agent's dot folder or in a plugin. A `skills/` folder in source code (`skills/mod.rs`), a reference checkout, text inside a heredoc and a write to a skill file no longer tag a skill. Codex's `.system/` and `synced//` folders are skipped, so `skill:.system` and `skill:synced` become the skill's own name, and a plugin named for its one skill is tagged once (`skill:frontend-design`). [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md). - Fixed `test-tamper` no longer flags adding tests: a skip guard inside a new test, fixing a test the agent added earlier, updating an expectation its own source edit broke, or a green run of fewer tests than the failing one. A file a shell command's diff picked up without the command naming it (a test run, a `git rebase`, another agent in the checkout) is never flagged, and BSD `sed -i ''` no longer counts `''` as a file. A skip marker inside a string literal (a diff or `sed` command a test holds as data) is not a skip. Stored sessions are re-checked in the background after the upgrade. [Concepts](https://github.com/jackulau/coding.sh/blob/main/docs/concepts.md). - Fixed `coding.sh sessions` leaves out sessions with no turn, tool call or model call, such as a process that started and ended a session and did nothing; `--empty` lists them too, and `--limit` still fills with sessions that have work. [CLI reference](https://github.com/jackulau/coding.sh/blob/main/docs/reference/cli.md). ## Evals - Added `eval validate` takes several tasks, or `--tag ` for every task with that tag (`suite:` for a suite), prints one line per task and exits 1 if any did not validate. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#session-to-eval). - Added `eval init --sessions --repo ` drafts tasks from sessions in another checkout or worktree of the project, not only the current one. [Evals](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md). - Added `eval validate` checks verifiers that grade the agent's answer or trace: a task dir's `reference/` (`output.md`, `stdout.jsonl`) is a correct run that must pass, and `near-misses/