coding.sh docs

Docs for coding.sh, open-source observability and evals for AI agents: what is built today, what is planned, and where to start.

What it is

coding.sh is open-source, local-first observability and evals for AI agents. Think of a local LangSmith, Braintrust or Judgment Labs: traces and evals stay on your disk, in one npm package with zero runtime dependencies, on Node 22.15 or newer.

It works with any agent. Your own agent, built on the OpenAI Agents SDK, LangGraph, the Vercel AI SDK, the Claude Agent SDK or plain code, sends traces over OpenTelemetry or the TypeScript and Python SDKs. Coding agents are first-class: adapters capture prompts, tool calls, model calls, file changes and subagents from Claude Code, Codex and ten other agents with no code changes.

Start here

New to the terms? Read Concepts.

What is built

How it fits together

1. Capture. Your own agent sends OpenTelemetry spans to coding.sh collector, or SDK events to coding.sh serve. A coding agent's hook runs ~/.coding.sh/bin/hook.mjs, which appends one JSON line to a local spool file and exits 0. No network, no daemon, nothing parsed on the hot path.
2. Ingest. coding.sh ingest (and every read command, before it reads) normalizes spool lines into coding.sh/trace events, redacts secrets, and writes them to the index and blob store.
3. Explore. List, inspect and search sessions in the terminal or in the local viewer.
4. Evaluate. Turn cases into datasets, run any agent over them, grade the outputs, and compare two runs case by case.

Pages

Start

• Install: Install coding.sh with one line, or with npm or from source; check it with doctor and remove it cleanly.
• Quickstart: Record Claude Code, Codex and other coding agents with no code changes, then list, inspect and search their sessions.
• Your own agent: Record an agent you built with OpenTelemetry or the TypeScript and Python SDKs, then run evals on it and compare versions.
• Concepts: Sessions, events, turns, flags, datasets, experiments, graders, judges and benchmarks, each in a sentence or two.

Use cases

• Track agent cost: Track Claude Code and Codex token usage and cost per session, repo and model, locally. Estimates from tokens × list price are marked ~.
• Replay a session: Replay Claude Code and Codex sessions step by step: commands, output and diffs, from your local session history. Nothing is re-run.
• Test AGENTS.md and skills: Test an AGENTS.md or skill change on your own sessions and get a Ship it, Hold or Don't ship verdict.
• Evals from failures: Turn your coding agent's failed sessions into an eval suite: one task per session, checked by your verifier, run with eval run.
• Self-hosted observability: Open-source, self-hosted LLM observability: one coding.sh serve takes OpenTelemetry traces and SDK events, on SQLite, Postgres or ClickHouse.
• With T3 Code: See T3 Code threads next to your other agents' sessions, with cost by model, failure flags and evals. Read from T3 Code's local store.
• With Pi: Record Pi coding agent sessions with a Pi extension: history next to your other agents, Pi's own cost per model call, failure flags and evals.
• Log model calls: Log every LLM call Claude Code or Codex makes with a local proxy: full prompts, cache use and cost, and why the prompt cache missed.

Record

• OpenTelemetry: Point the Vercel AI SDK, Claude Agent SDK, OpenAI Agents SDK, LangGraph or any OTLP exporter at the local receiver.
• Model API proxy: Route Claude Code, Codex or any client through a local Anthropic/OpenAI proxy to log every call, see why the prompt cache missed, and replay a call.
• Durable workflows: Record a Vercel Workflow or Temporal run as one session across crashes, deploys and days, with each step's attempts, waits and resume points.
• Gateways: Agents behind Vercel AI Gateway, OpenRouter, LiteLLM, Portkey, Helicone and others: tagged sessions, billed cost per call, and A/B runs across models.
• Production: Run coding.sh serve on SQLite or Postgres and send traces from API apps with the SDKs or OpenTelemetry.
• Coding agents: Which coding agents have adapters, how each is captured, and how subagents are linked to their parent session.
• MCP server: Let your agent query its own sessions, failures, latency and usage over MCP; init registers the server for you.

Evaluate

• Evals: Your first eval suite in minutes with eval init and eval run, then datasets, the task under test, graders and calibrated LLM judges.
• Experiments: Compare experiments with paired statistics, sweep and matrix configs, stop a loop at its plateau, resume runs, A/B metrics and replay a model call.
• Harness evals: Test an AGENTS.md, rule or skill change before you keep it: A/B it, keep safety lines, refine in rounds, check skill triggers and run repo-replay tasks.
• Benchmarks: coding.sh bench: fresh cases every day from a live source, every contender on the same frozen set, and a history of who leads.
• Workflows: Capture multi-agent workflow runs, score them against known answers, compare two versions with paired statistics, and replay with a control fork.
• Tool evals: coding.sh toolevals: measure how well agents use a CLI, MCP server or HTTP API, then keep the help, flag and description changes that raise the score.

How-to

• Test an AGENTS.md change: Replay the moments your agent went wrong with your committed AGENTS.md and with your edit, and get a ship-or-hold verdict before you commit.
• Replay model calls: Record a run's model calls, then re-run your change with the calls before it served from the recording and the rest live, against a control arm.
• Compare two agents: Turn a real task into a checked eval, snapshot two agent setups, and run both on it with a paired comparison.
• Catch regressions in CI: Run an eval suite, or the same eval on the base branch and the pull request, with the coding.sh GitHub Action, and fail the job when it regresses.
• Find expensive sessions: Rank sessions by cost or tokens, break spend down by repo, agent and model, and see where cost data comes from.
• Evaluate a skill: Run the same behaviour tasks with and without a skill, grade each run with checks and a judge, and read a ship-or-hold answer.
• Does this plugin help?: See what a Claude Code plugin's hooks add to your sessions and what it costs, then run your tasks with and without it.
• Does a terse mode save money?: Run your tasks with and without a be-concise rule, skill or plugin, and compare pass rate, cost, output tokens and the input it adds.
• Session to eval: Keep a captured session as a validated repo task or a dataset case, so later changes are tested against it.
• Agent lines per commit: Match the lines each commit added to the session that wrote them, see it per line with coding.sh blame, and export Agent Trace records.
• Self-host for a team: Run one container, connect each machine with coding.sh init --remote, and open the team viewer.
• Measure site speed: How we made coding.sh load faster on slow phones, measured with coding.sh exp metrics, and the command to measure your own site.

Compare

• Overview: How coding.sh compares with Langfuse, LangSmith, Braintrust, AgentOps, Laminar and ccusage, and when each of them is the better choice.
• vs Langfuse: Langfuse is an open-source LLM platform, hosted or self-hosted; coding.sh is a local CLI for coding agents and evals. A feature table and where each wins.
• vs LangSmith: LangSmith is a hosted platform; coding.sh is MIT licensed and runs on your machine. A feature table, and where each one is the better choice.
• vs Braintrust: Braintrust is a hosted eval platform; coding.sh runs evals locally with paired statistics. A feature table, and where each one is the better choice.
• vs AgentOps: AgentOps instruments Python agent frameworks; coding.sh captures coding agents and runs evals, locally. A feature table and where each is better.
• vs Laminar: Laminar is open-source agent observability with a cloud; coding.sh is a local CLI for coding agents and evals. A feature table and where each is better.
• vs ccusage: ccusage reports coding-agent token cost from local logs; coding.sh adds each session's steps, replay, flags and evals. Where each is the better choice.

Reference

• CLI: Every coding.sh command and option, generated from the CLI's own --help output at build time.
• SDKs: Every function and option in the TypeScript and Python SDKs, generated from their source.
• REST API: Every route coding.sh serve answers, with auth and examples: ingest, OTLP, the model API proxy, health and metrics.
• Read API: The /api routes the viewer and your scripts read, their auth and errors: projects, sessions, events, outlines, replays, blobs and model calls.
• Search and evals API: The /api routes for search, stats, facets, events, labels, experiments, compare, benchmarks, tasks, runs and bundles, update and ingest status.
• Configuration: Every config.toml key and environment variable, their defaults, and which layer wins.
• Storage: The SQLite index, the content-addressed blob store, sinks, and how to plug in a remote store.
• Trace format: The coding.sh/trace v0 event format every adapter and the OTLP receiver normalize into.
• Troubleshooting: What doctor checks, why sessions go missing, and every CLI error reason with its fix.
• FAQ: Privacy, performance, status, and how coding.sh relates to hosted tracing tools.

Help and contributing

The source is on GitHub. Bugs and questions go to issues. The most useful contribution right now is a recorded session from an agent version the adapters haven't seen yet.