# coding.sh

> coding.sh is open-source, local-first observability and evals for AI agents. It records agent runs through coding-agent hooks and a local OpenTelemetry receiver, turns failures into evals with calibrated graders, and compares prompt, harness and model changes with paired statistics. MIT licensed, v0.1.4. Install: curl -fsSL https://coding.sh/install | sh

Also written coding shell, coding sh or codingsh (the `.sh` reads as "shell"); the CLI runs as `coding.sh` or `codingsh`. One tool.

Status: v0.1.4, early, MIT licensed. Source: https://github.com/jackulau/coding.sh

## Install

```sh
curl -fsSL https://coding.sh/install | sh      # checks Node 22.15+, installs the latest release, runs init, opens the viewer
coding.sh init --yes    # non-interactive: install hooks into every detected agent
coding.sh doctor --json # check the install
```

The installer is https://coding.sh/install (plain text; read it before piping it to sh). It runs `npm install -g`: from npm when
the registry has coding.sh, otherwise on the release tarball, e.g. `npm i -g https://github.com/jackulau/coding.sh/releases/download/v0.1.5/coding.sh-0.1.5.tgz`. Env: CODING_SH_YES=1 skips the prompt, CODING_SH_NO_INIT=1 installs only,
CODING_SH_NO_UI=1 skips the viewer, CODING_SH_VERSION=v0.1.0 pins a release,
CODING_SH_SOURCE=github skips npm.
From source: `git clone https://github.com/jackulau/coding.sh && cd coding.sh && npm i && npm run build`; the CLI builds to packages/coding.sh/dist/cli.mjs.

Full prose for every page is at https://coding.sh/llms-full.txt; each page also serves a Markdown copy at <page>.md (the home page at /index.md).
Screenshots and terminal output on the landing page come from the real CLI and viewer, run on a throwaway home where coding.sh works on its own repo; terminals in the docs marked as real output were captured from the CLI.

## Pages

- [Home](https://coding.sh/): What coding.sh is: a real view of the local viewer, what agents cost, and how to check that a change helped.
- [Changelog](https://coding.sh/changelog): Every coding.sh release, newest first, rendered from CHANGELOG.md.
- [Blog](https://coding.sh/blog): Notes on how coding.sh works and why, newest first.

## Docs

- [coding.sh docs](https://coding.sh/docs): Docs for coding.sh, open-source observability and evals for AI agents: what is built today, what is planned, and where to start.
- [Install](https://coding.sh/docs/install): Install coding.sh with one line, or with npm or from source; check it with doctor and remove it cleanly.
- [Quickstart](https://coding.sh/docs/quickstart): Record Claude Code, Codex and other coding agents with no code changes, then list, inspect and search their sessions.
- [Trace your own agent](https://coding.sh/docs/how-to/trace-your-agent): Record an agent you built with OpenTelemetry or the TypeScript and Python SDKs, then run evals on it and compare versions.
- [Concepts](https://coding.sh/docs/concepts): Sessions, events, turns, flags, datasets, experiments, graders, judges and benchmarks, each in a sentence or two.
- [Track Claude Code and Codex cost per session](https://coding.sh/docs/use-cases/claude-code-cost): Track Claude Code and Codex token usage and cost per session, repo and model, locally. Estimates from tokens × list price are marked ~.
- [Replay a Claude Code or Codex session step by step](https://coding.sh/docs/use-cases/replay-agent-session): Replay Claude Code and Codex sessions step by step: commands, output and diffs, from your local session history. Nothing is re-run.
- [Test an AGENTS.md or skill change](https://coding.sh/docs/use-cases/test-agent-instructions): Test an AGENTS.md or skill change on your own sessions and get a Ship it, Hold or Don't ship verdict.
- [Build evals from your agent's failures](https://coding.sh/docs/use-cases/evals-from-failures): Turn your coding agent's failed sessions into an eval suite: one task per session, checked by your verifier, run with eval run.
- [Open-source, self-hosted LLM observability](https://coding.sh/docs/use-cases/self-hosted-observability): Open-source, self-hosted LLM observability: one coding.sh serve takes OpenTelemetry traces and SDK events, on SQLite, Postgres or ClickHouse.
- [coding.sh with T3 Code](https://coding.sh/docs/use-cases/t3-code): See T3 Code threads next to your other agents' sessions, with cost by model, failure flags and evals. Read from T3 Code's local store.
- [coding.sh with Pi](https://coding.sh/docs/use-cases/pi): Record Pi coding agent sessions with a Pi extension: history next to your other agents, Pi's own cost per model call, failure flags and evals.
- [Log every model call your agent makes](https://coding.sh/docs/use-cases/proxy-model-calls): Log every LLM call Claude Code or Codex makes with a local proxy: full prompts, cache use and cost, and why the prompt cache missed.
- [Record any agent with OpenTelemetry](https://coding.sh/docs/otel): Point the Vercel AI SDK, Claude Agent SDK, OpenAI Agents SDK, LangGraph or any OTLP exporter at the local receiver.
- [Capture every model call with the proxy](https://coding.sh/docs/how-to/capture-with-proxy): Route Claude Code, Codex or any client through a local Anthropic/OpenAI proxy to log every call, see why the prompt cache missed, and replay a call.
- [Trace a durable workflow](https://coding.sh/docs/how-to/trace-durable-workflow): Record a Vercel Workflow or Temporal run as one session across crashes, deploys and days, with each step's attempts, waits and resume points.
- [LLM gateways](https://coding.sh/docs/ai-gateway): Agents behind Vercel AI Gateway, OpenRouter, LiteLLM, Portkey, Helicone and others: tagged sessions, billed cost per call, and A/B runs across models.
- [Production: API apps](https://coding.sh/docs/production): Run coding.sh serve on SQLite or Postgres and send traces from API apps with the SDKs or OpenTelemetry.
- [Coding agents](https://coding.sh/docs/agents): Which coding agents have adapters, how each is captured, and how subagents are linked to their parent session.
- [MCP server](https://coding.sh/docs/mcp): Let your agent query its own sessions, failures, latency and usage over MCP; init registers the server for you.
- [Evals](https://coding.sh/docs/evals): Your first eval suite in minutes with eval init and eval run, then datasets, the task under test, graders and calibrated LLM judges.
- [Experiments](https://coding.sh/docs/experiments): Compare experiments with paired statistics, sweep and matrix configs, stop a loop at its plateau, resume runs, A/B metrics and replay a model call.
- [Harness evals](https://coding.sh/docs/harness-evals): Test an AGENTS.md, rule or skill change before you keep it: A/B it, keep safety lines, refine in rounds, check skill triggers and run repo-replay tasks.
- [Benchmarks](https://coding.sh/docs/benchmarks): coding.sh bench: fresh cases every day from a live source, every contender on the same frozen set, and a history of who leads.
- [Workflow experiments](https://coding.sh/docs/workflows): Capture multi-agent workflow runs, score them against known answers, compare two versions with paired statistics, and replay with a control fork.
- [Tool evals](https://coding.sh/docs/tool-evals): coding.sh toolevals: measure how well agents use a CLI, MCP server or HTTP API, then keep the help, flag and description changes that raise the score.
- [Test an AGENTS.md change in five minutes](https://coding.sh/docs/how-to/test-agents-md-change): Replay the moments your agent went wrong with your committed AGENTS.md and with your edit, and get a ship-or-hold verdict before you commit.
- [Replay recorded model calls](https://coding.sh/docs/how-to/replay-llm-calls): Record a run's model calls, then re-run your change with the calls before it served from the recording and the rest live, against a control arm.
- [Compare two agents on your repo](https://coding.sh/docs/how-to/compare-agents): Turn a real task into a checked eval, snapshot two agent setups, and run both on it with a paired comparison.
- [Catch a regression in CI](https://coding.sh/docs/how-to/ci-regressions): Run an eval suite, or the same eval on the base branch and the pull request, with the coding.sh GitHub Action, and fail the job when it regresses.
- [Find your most expensive sessions](https://coding.sh/docs/how-to/expensive-sessions): Rank sessions by cost or tokens, break spend down by repo, agent and model, and see where cost data comes from.
- [Evaluate an Agent Skill](https://coding.sh/docs/how-to/evaluate-a-skill): Run the same behaviour tasks with and without a skill, grade each run with checks and a judge, and read a ship-or-hold answer.
- [Does this plugin help me?](https://coding.sh/docs/how-to/does-this-plugin-help): See what a Claude Code plugin's hooks add to your sessions and what it costs, then run your tasks with and without it.
- [Does a terse mode save money on your tasks?](https://coding.sh/docs/terse-mode): Run your tasks with and without a be-concise rule, skill or plugin, and compare pass rate, cost, output tokens and the input it adds.
- [Turn a real session into an eval](https://coding.sh/docs/how-to/session-to-eval): Keep a captured session as a validated repo task or a dataset case, so later changes are tested against it.
- [See which agent session wrote each commit](https://coding.sh/docs/how-to/attribute-commits): Match the lines each commit added to the session that wrote them, see it per line with coding.sh blame, and export Agent Trace records.
- [Self-host for a team](https://coding.sh/docs/how-to/self-host-team): Run one container, connect each machine with coding.sh init --remote, and open the team viewer.
- [Site speed](https://coding.sh/docs/perf): How we made coding.sh load faster on slow phones, measured with coding.sh exp metrics, and the command to measure your own site.
- [coding.sh compared with other tools](https://coding.sh/docs/compare): How coding.sh compares with Langfuse, LangSmith, Braintrust, AgentOps, Laminar and ccusage, and when each of them is the better choice.
- [coding.sh vs Langfuse](https://coding.sh/docs/compare/langfuse): Langfuse is an open-source LLM platform, hosted or self-hosted; coding.sh is a local CLI for coding agents and evals. A feature table and where each wins.
- [coding.sh vs LangSmith](https://coding.sh/docs/compare/langsmith): LangSmith is a hosted platform; coding.sh is MIT licensed and runs on your machine. A feature table, and where each one is the better choice.
- [coding.sh vs Braintrust](https://coding.sh/docs/compare/braintrust): Braintrust is a hosted eval platform; coding.sh runs evals locally with paired statistics. A feature table, and where each one is the better choice.
- [coding.sh vs AgentOps](https://coding.sh/docs/compare/agentops): AgentOps instruments Python agent frameworks; coding.sh captures coding agents and runs evals, locally. A feature table and where each is better.
- [coding.sh vs Laminar](https://coding.sh/docs/compare/laminar): Laminar is open-source agent observability with a cloud; coding.sh is a local CLI for coding agents and evals. A feature table and where each is better.
- [coding.sh vs ccusage](https://coding.sh/docs/compare/ccusage): ccusage reports coding-agent token cost from local logs; coding.sh adds each session's steps, replay, flags and evals. Where each is the better choice.
- [CLI reference](https://coding.sh/docs/cli): Every coding.sh command and option, generated from the CLI's own --help output at build time.
- [SDK reference](https://coding.sh/docs/sdk): Every function and option in the TypeScript and Python SDKs, generated from their source.
- [REST API reference](https://coding.sh/docs/api): Every route coding.sh serve answers, with auth and examples: ingest, OTLP, the model API proxy, health and metrics.
- [Read API: sessions and calls](https://coding.sh/docs/api/read): The /api routes the viewer and your scripts read, their auth and errors: projects, sessions, events, outlines, replays, blobs and model calls.
- [Read API: search, labels and evals](https://coding.sh/docs/api/read/search): The /api routes for search, stats, facets, events, labels, experiments, compare, benchmarks, tasks, runs and bundles, update and ingest status.
- [Configuration reference](https://coding.sh/docs/config): Every config.toml key and environment variable, their defaults, and which layer wins.
- [Storage](https://coding.sh/docs/storage): The SQLite index, the content-addressed blob store, sinks, and how to plug in a remote store.
- [Trace format](https://coding.sh/docs/trace-format): The coding.sh/trace v0 event format every adapter and the OTLP receiver normalize into.
- [Troubleshooting](https://coding.sh/docs/troubleshooting): What doctor checks, why sessions go missing, and every CLI error reason with its fix.
- [FAQ](https://coding.sh/docs/faq): Privacy, performance, status, and how coding.sh relates to hosted tracing tools.

## What it is for

Open-source, local-first observability and evals for AI agents: like LangSmith, Braintrust or Judgment Labs,
but on your machine. Pre-alpha: what is built is covered by the package's tests; items marked planned do not exist yet.

1. Record (built): a local OTLP receiver (`coding.sh collector`, http://127.0.0.1:4318) for Vercel AI SDK, Claude Agent SDK, OpenAI Agents SDK, LangGraph, any OTel GenAI exporter;
   model-call latency per call. Coding agents are first-class: hooks capture prompts, tool calls, file changes
   and subagents (as child sessions). `coding.sh run --` snapshots the working tree before and after;
   hook-captured sessions in a git repository start from the commit checked out at session start.
2. Evals (built): datasets with deterministic graders, command graders and LLM judges; judges are calibrated
   against human labels with Cohen's kappa on a held-out split, and uncalibrated judges are flagged in every
   report. Repo-replay tasks (partial): validation (do-nothing fails, reference passes k of k, near-misses
   fail, no leakage), harness bundles, compare and drafting a task from a captured session are built.
3. Compare (built): two experiments paired by case, with a paired bootstrap CI of the difference and McNemar's
   test on flipped cases; harness compare adds per-arm Wilson intervals and the first step where traces diverge.
4. Storage (built): SQLite index and a content-addressed, zstd-compressed blob store by default; OTLP and HTTP
   sinks, metadata-only by default; third-party stores and sinks load as npm plugins. The Postgres, ClickHouse and S3
   store plugins are on npm (coding.sh-store-postgres, coding.sh-store-clickhouse, coding.sh-store-s3) and used by
   `coding.sh serve`. The core is one npm package with zero runtime dependencies, Node 22.15 or newer.
5. Tool evals (built): `coding.sh toolevals` scores how well agents use a CLI, MCP server or HTTP API.
   Workflows (built): `coding.sh workflows` captures, scores and compares multi-agent runs.
   Trace format (built): TypeScript types (`coding.sh/schema`) and a JSON Schema (`coding.sh/schema.json`, https://coding.sh/schema.json).

## Compared with other tools

Langfuse, LangSmith, Braintrust, AgentOps, Laminar and ccusage overlap with coding.sh. Each comparison page has a feature
table, says where the other tool is the better choice, and cites its sources: https://coding.sh/docs/compare

## Coding agents (adapters in code)

- Claude Code (`claude-code`, verified on real sessions): capture: settings.json hooks; backfill: projects/<slug>/<session>.jsonl transcripts; subagents: SubagentStart/Stop hooks; subagents/agent-*.jsonl transcripts
- Codex CLI (`codex`, verified on real sessions): capture: hooks.json hooks (you approve them with /hooks); backfill: sessions/**/rollout-*.jsonl; subagents: SubagentStart/Stop hooks; subagent threads in rollouts
- OpenCode (`opencode`, verified on a real install): capture: plugin (2.x and 1.x); backfill: opencode.db (1.x and 2.x), JSON storage (older 1.x); subagents: Child sessions, linked to the parent session
- Cursor CLI (`cursor`, fixture-tested): capture: hooks.json, partial in -p mode; backfill: agent-transcripts/*.jsonl (undocumented format); subagents: subagentStop only (subagentStart is a permission gate, not registered)
- T3 Code (`t3code`, verified on a real install): capture: its Codex and Claude Code hooks, tagged as T3; backfill: ~/.t3/userdata/statev2.sqlite (and v1 state.sqlite); subagents: Subagent threads become child sessions
- Antigravity CLI (formerly Gemini CLI) (`antigravity`, verified on real sessions): capture: hooks.json; backfill: brain/<id>/…/transcript.jsonl; subagents: invoke_subagent tool call spans the subagent; its internals are not exposed
- Gemini CLI, older installs (`gemini-cli`, fixture-tested): capture: settings.json hooks; backfill: tmp/<slug>/chats/*.jsonl; subagents: Child chat files become child sessions
- GitHub Copilot CLI (`copilot-cli`, verified on a real install): capture: hooks/coding-sh.json; backfill: session-state/<id>/events.jsonl; subagents: subagentStart/Stop hooks; events inside carry the subagent id
- Amp (`amp`, fixture-tested): capture: plugin; backfill: none; subagents: Task, oracle, librarian, finder tool calls span the subagent
- Pi (`pi`, verified on a real install): capture: extension; backfill: sessions/--<cwd>--/*.jsonl; subagents: None (Pi has no subagents)
- Factory Droid (`droid`, verified on a real install): capture: hooks.json; backfill: sessions/-<cwd>/*.jsonl; subagents: Task tool call is the start, SubagentStop the end
- Goose (`goose`, verified on a real install): capture: plugin hooks.json; backfill: sessions.db; subagents: sub_agent sessions become child sessions
- Aider (`aider`, verified on a real install): capture: coding.sh run -- aider … (each reply's tokens and cost from Aider's analytics log); backfill: none; subagents: None
- Any other command (`wrapper`, tested): capture: coding.sh run -- <cmd>; backfill: none; subagents: None

Planned, no adapter yet: Cline CLI, Kiro CLI, Qwen Code, Crush.

## CLI

- `coding.sh init` (capture): Set up capture hooks and the MCP server for every detected agent
- `coding.sh doctor` (capture): Check agents, hooks, spool, store and the OTLP port
- `coding.sh status` (capture): Show capture totals and the latest session
- `coding.sh statusline` (capture): Print a Claude Code status line (model, cost, worst flag), or install it
- `coding.sh ingest` (capture): Move captured hook events from the spool into the index
- `coding.sh run` (capture): Run any agent command and capture the session
- `coding.sh collector` (capture): Start a local OTLP/HTTP receiver for agents that export traces
- `coding.sh proxy` (capture): Run a local Anthropic/OpenAI API proxy that logs every model call
- `coding.sh serve` (capture): Production server: OTLP + native ingest, viewer, health and metrics on one port
- `coding.sh uninstall` (capture): Remove coding.sh hooks and MCP entries from every agent
- `coding.sh sessions` (explore): List captured sessions
- `coding.sh show` (explore): Show a session timeline with subagents and latency
- `coding.sh tree` (explore): Show a whole multi-agent run as one tree, with rolled-up totals
- `coding.sh replay` (explore): Step through a session and see what it did to each file it touched
- `coding.sh attribution` (explore): Which agent session wrote the lines each commit added; export them as Agent Trace
- `coding.sh blame` (explore): git blame with the agent session that wrote each line
- `coding.sh diff-calls` (explore): Show what changed between two model calls, and why the second missed the prompt cache
- `coding.sh search` (explore): Full-text search over prompts, replies and tool calls
- `coding.sh usage` (explore): Tokens, cost, turns and cache hits by agent, model or day
- `coding.sh cache` (explore): Score prompt-cache health and name what breaks the cached prefix
- `coding.sh stats` (explore): Sessions, failures, cost, tokens and latency per day or week, split by agent, model, repo or flag
- `coding.sh sql` (explore): Run one read-only SQL query against the SQLite index
- `coding.sh export` (explore): Stream sessions, events or labels as JSONL or CSV, or write raw traces to a directory
- `coding.sh ui` (explore): Open the local session viewer
- `coding.sh mcp` (explore): Serve captured sessions to agents over MCP (stdio)
- `coding.sh eval` (evaluate): Build eval suites from sessions, skills or cases, and run them
- `coding.sh dataset` (evaluate): Manage versioned datasets of cases
- `coding.sh exp` (evaluate): Run a task over a dataset, compare experiments and sweep configs
- `coding.sh score` (evaluate): Add and list typed scores on sessions, events, cases and runs
- `coding.sh judge` (evaluate): Grade one output or the slop in captured diffs, or check an LLM judge or a test verifier against human labels
- `coding.sh toolevals` (evaluate): Measure how well agents use a CLI, MCP server or API
- `coding.sh workflows` (evaluate): Capture, measure and A/B-test multi-agent workflow runs
- `coding.sh bench` (evaluate): Dynamic benchmarks: fresh cases every run, same set for every contender
- `coding.sh improve` (harness): What to fix first in your sessions, the line that fixes it, and whether your last AGENTS.md changes helped
- `coding.sh harness` (harness): Snapshot and list agent harness bundles; audit which memory and instruction files agents use
- `coding.sh compare` (harness): A/B two harness bundles on eval tasks
- `coding.sh refine` (harness): Improve AGENTS.md and skills in tested rounds, and protect their safety rules
- `coding.sh gateway` (maintain): LLM gateways: detect routing, enrich sessions with billed cost
- `coding.sh gc` (maintain): Delete expired spool files, blobs and judge cache
- `coding.sh purge` (maintain): Remove a leaked secret from the store: events, search, blobs, spool, eval runs
- `coding.sh config` (maintain): Read and write ~/.coding.sh/config.toml
- `coding.sh update` (maintain): Install the latest coding.sh release
- `coding.sh upgrade-store` (maintain): Migrate the store for this install, or put back the index from before a migration
- `coding.sh version` (maintain): Print the version

Full reference: https://coding.sh/docs/cli

## Output and exit codes

- Every command takes `--json`: one JSON document on stdout. Errors go to stderr as one JSON line:
  `{"status":"error","reason":"<reason>","message":"<text>","next":["<what to do>:","<command>"]}`.
- Exit codes: 0 success, 1 error, 2 usage error (unknown option, bad value, missing `--yes`), 130 canceled.
- Commands that write config ask first; pass `--yes` to confirm without a prompt (`init --dry-run` prints every edit and writes nothing).