--- title: "coding.sh v0.1.6 · Changelog" description: "What changed in coding.sh v0.1.6. 149 changes, most to Capture, Experiments and Evals." canonical: https://coding.sh/changelog/v0-1-6 human_url: https://coding.sh/changelog/v0-1-6 markdown_url: https://coding.sh/changelog/v0-1-6.md agents_url: https://coding.sh/agents.md llms_url: https://coding.sh/llms.txt updated: 2026-11-11 --- [Changelog](https://coding.sh/changelog) Unreleased · 149 changes · page 1 of 4 # coding.sh v0.1.6 ## Privacy - Added `CODING_SH_PURGE_WAIT_MS` sets how long `purge` waits for another process's read of the index or the blob pack before it says which file still holds the old bytes. The defaults stay 10 s before the VACUUM and 5 s after it. [Configuration](https://github.com/jackulau/coding.sh/blob/main/docs/reference/config.md#paths-and-stores). - Added An opt-in PII pack: `capture.redact = ["secrets", "pii"]` also redacts email addresses, phone numbers, card numbers (issuer prefix and Luhn check), IBANs (country length and mod 97) and national ids (US SSN, UK NINO, Canadian SIN). Each kind (`phone`, `card`, `iban`, `national-id`, `email`) can be turned on alone. [Privacy](https://github.com/jackulau/coding.sh/blob/main/docs/guides/privacy.md#privacy). - Fixed A purge given a wait (`CODING_SH_PURGE_WAIT_MS`, or `waitMs` in code) now keeps to it while another process reads the index. The checkpoint before the VACUUM used to wait the full 10 s whatever wait was given. [Privacy](https://github.com/jackulau/coding.sh/blob/main/docs/guides/privacy.md). - Fixed The `env` redactor no longer replaces `PWD`, `OLDPWD` or variables named `*_PATH`, `*_DIR`, `*_FILE` or `*_SOCK`, or any value that is an existing path, so the working directory stopped turning into `[redacted:env]` in every path. [Privacy](https://github.com/jackulau/coding.sh/blob/main/docs/guides/privacy.md#privacy). - Fixed A source span or log record that `coding.sh serve` or `coding.sh collector --keep-raw` keeps as a blob, and an oversize structured value moved to a blob, is now redacted one string at a time (a string that holds JSON as JSON) before it is serialized. A `NAME=value` line after a newline and a JSON field such as `api_key` or `password` used to stay in clear in the stored blob and were served by `/api/blobs/`, while the event showed them as redacted. Blobs stored before 0.1.6 are not rewritten: remove such a value with `coding.sh purge`. [Privacy](https://github.com/jackulau/coding.sh/blob/main/docs/guides/privacy.md#privacy) ## Trace format - Added A JSON Schema (draft 2020-12) for `AgentEvent` and `Session`, generated from the TypeScript types, ships as `coding.sh/schema.json` and is served at https://coding.sh/schema.json. Every adapter and OTLP fixture validates against it. [Trace format](https://coding.sh/docs/trace-format#json-schema) ## Serve - Added `CODING_SH_FORWARD_URL` (or `[serve.forward]`) makes `coding.sh serve` forward every trace it commits to an OTLP/HTTP collector, with the same trace and span ids plus `coding_sh.project`. Native batches go out as `invoke_agent`, `execute_tool` and `chat` spans. Metadata mode sends no messages, tool arguments or results; a down collector never slows ingest, spans wait in a bounded buffer, and overflow is counted in `coding_sh_forward_spans_total`. [Forward and sinks](https://github.com/jackulau/coding.sh/blob/main/docs/PRODUCTION.md#19-forward-and-sinks) - Added `createFetchHandler()` from `coding.sh/server` runs serve's ingest routes (OTLP, `/v1/ingest`, `/readyz`) as a `Request` → `Response` function for Vercel Functions, Deno, Bun or Workers. With `coding.sh-store-postgres`, several instances can write one session at once: each session rebuild takes `pg_advisory_xact_lock` on the session id. [Serverless](https://github.com/jackulau/coding.sh/blob/main/docs/production/serverless.md) ## Sinks - Added `[[sinks]]` now mirror events from the OTLP receiver (`coding.sh collector`) and from `coding.sh serve`, so a team can set one sink on serve instead of on every laptop. [Config](https://github.com/jackulau/coding.sh/blob/main/docs/reference/config.md#sinks) - Changed The otlp sink sends `gen_ai.usage.input_tokens` as the whole prompt, cache reads and writes included, as the OpenTelemetry GenAI conventions ask; it used to send only the uncached part, so backends undercounted cached calls. Cache writes move to `gen_ai.usage.cache_write.input_tokens` (the old `cache_creation.input_tokens` is still sent through 0.1.5), the uncached part is `coding_sh.usage.uncached_input_tokens`, and reported reasoning tokens are `gen_ai.usage.reasoning.output_tokens`. The OTLP receiver reads the sink's spans back to the same token counts and cost. [Export to ClickHouse](https://github.com/jackulau/coding.sh/blob/main/docs/how-to/self-host-team.md#export-to-clickhouse-through-an-opentelemetry-collector). - Fixed Metadata sinks no longer receive a session's `stats.llmErrors.error` text, which could quote the prompt. [Config](https://github.com/jackulau/coding.sh/blob/main/docs/reference/config.md#sinks) - Fixed `coding.sh serve` and `coding.sh collector` now send the `otlp` sink a session's `invoke_agent` span while the session is still open: once it has been quiet for 10 seconds, and every 60 seconds while it stays busy. An OTLP session keyed by a conversation id (Claude Code, Codex, `gen_ai.conversation.id`, Agent Runs) never ends, so it used to reach the sink as `chat` and `execute_tool` spans with no parent and no session row. Each send repeats the span id with newer totals, and the ClickHouse queries now take the latest row per span id. [Export to ClickHouse](https://github.com/jackulau/coding.sh/blob/main/docs/how-to/self-host-team.md#export-to-clickhouse-through-an-opentelemetry-collector) - Fixed `coding.sh serve` and `coding.sh collector` send an event to `[[sinks]]` once. A request that was posted again, such as a client retry answered `"inserted":0`, used to send all its events again and doubled rows and cost downstream; now only the events a batch newly stored go on, and a batch that overlaps stored events sends the new ones. A commit that fails after its events were written sends them when the retry commits. [Forward and sinks](https://github.com/jackulau/coding.sh/blob/main/docs/PRODUCTION.md#19-forward-and-sinks) - Fixed Local ingest now sends the events it imports from agent transcripts to `[[sinks]]`, not only the events from the hook spool, so a sink has every model call the store has. When an import adds events to a session without ending it, the otlp sink gets the session's `invoke_agent` span again with the new totals. A sink that is down does not stop the import and catches up after its backoff, and a second ingest sends nothing new. [Config](https://github.com/jackulau/coding.sh/blob/main/docs/reference/config.md#sinks) ## Evals - Added Cases and tool-eval tasks take an optional `kind`: `bad` (fails before the change, passes after), `guard` or `control` (pass on both). `exp run`, `exp compare`, `toolevals run` and `toolevals compare` split results by kind and name broken controls as overcorrection, and `toolevals validate --against ` reports a bad task that passes on the old variant. See [Case kinds](docs/guides/evals.md#case-kinds-bad-guard-control) and [Task kinds](docs/TOOL-EVALS.md#31-task-kinds-bad-guard-control). - Added Typed scores: a number, true or false, a category or text on a session, event, dataset case or eval run, from a person, judge, code, the API or a monitor. Score configs fix a name's type and range, and a value outside it is refused. Add and list them with `coding.sh score`, `POST`/`GET /api/scores` and `/api/score-configs`, or the `score_add` and `score_list` MCP tools. Existing labels move into scores as human pass or fail scores, and `labels` stays readable as a view. [Scores](docs/guides/evals.md#scores) - Added `coding.sh eval candidates` (and the MCP tool `eval_candidates`) ranks captured sessions as eval tasks. Subagents, runs, and sessions with no git start, no prompt or no edits are left out; the rest are ordered by failure flags, pushback, a guessable verifier and recency. `eval init --sessions` drafts from that ranking and says how many it left out. [Your agent's failed sessions](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#your-agents-failed-sessions). - Added `eval init --where '' --validate` drafts tasks from the sessions a query matches and validates each one. A session diff that passes the verifier becomes the reference and the task is marked validated. Tasks from flagged sessions are tagged `captured-failure`, and `eval run` (and the GitHub Action's comment) says which of them now pass. [Your agent's failed sessions](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#your-agents-failed-sessions). - Added `eval new --statement ledger` writes the task instruction from every user turn and approved proposal, as cited obligations, instead of the first prompt only. With `--judge` a model merges them and drops the ones a later turn replaced, and code checks every citation and the leakage of the reference patch. [Session to eval](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#session-to-eval). - Added `eval trigger --draft` writes `triggers.toml`. It starts from the prompts of this repo's captured sessions that loaded the skill (should) or a sibling skill (should_not), then has a model fill each side to 8. Mined prompts are redacted, each prompt records its source, and the reply is cached. [Evaluate an Agent Skill](https://github.com/jackulau/coding.sh/blob/main/docs/how-to/evaluate-a-skill.md#does-the-skill-load). - Fixed `eval new --statement ledger` keeps a reply that approves and goes on ("yes, but call it goodbye instead") as its own obligation, after the proposal it changes. A short reply that opened with an approval word used to count as a bare approval, so the instruction asked for the assistant's proposal and dropped the user's change. [Session to eval](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#session-to-eval) - Fixed `eval init --validate` no longer marks a session that rewrote the test as the validated reference. A session diff that edits, deletes or skips the tests the verifier runs is kept as a near miss, with or without `--validate`, and a note names the test file. A diff that only adds test cases to a test file still counts as a reference. [Your agent's failed sessions](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#your-agents-failed-sessions) - Fixed `eval trigger --draft` reads only the sessions of the current git repo, so a `triggers.toml` you commit no longer holds prompts typed in other projects. `--repo DIR` reads the sessions under another directory and `--repo all` reads every repo's. [Does the skill load?](https://github.com/jackulau/coding.sh/blob/main/docs/how-to/evaluate-a-skill.md#does-the-skill-load) - Fixed `coding.sh score add event: --session ` (and `POST /api/scores` with a `sessionId`) no longer stores a score on an event that is not in the store, or under another session. An unknown event is `event not found` (exit 1, HTTP 404), and a session that is not the event's own is refused with both ids named (exit 2, HTTP 400). [Scores](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#scores) - Fixed A stored score keeps its type. A value that reads as another type (`1,5` over a number, `0.5` over a pass or fail label) is refused (exit 2, HTTP 400) and names `--type` and `score config`, instead of quietly turning the score into a category. `--type`, `dataType` or a config for the name still change the type, and `score add` now prints the type: `Scored n1 = -5 (numeric)`. [Scores](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#scores) ## Tool evals - Added `coding.sh toolevals traces --sessions 'since:30d'` (or `--from `) prints each MCP tool's calls, errors, repeats and retries after an error, and `--out ` writes the failing traces first (at most 3 passing), an `index.jsonl`, `tool-health.json` and `tools.json` for an improver to read. Held-out trials are never exported. [Tool evals](https://github.com/jackulau/coding.sh/blob/main/docs/TOOL-EVALS.md#121-toolevals-traces) - Added `toolevals run --shell-model ` has a model play the shell for a model driver on a CLI target, with a fixed date; the shell model is part of the arm name and `compare` refuses arms with different shells. [Tool evals](https://github.com/jackulau/coding.sh/blob/main/docs/TOOL-EVALS.md#42-model-driver-agent-free) - Added The model driver's MCP client reads resources and prompts, counts progress notifications, runs task-augmented tool calls, answers sampling with the driver's model, elicitation with a task's scripted `[mcp]` answers and roots with the sandbox, and runs OAuth (discovery, client registration, PKCE or client credentials) on a 401. `calls` checks can match `resources/read` by uri and `prompts/get` by name. [MCP client](https://github.com/jackulau/coding.sh/blob/main/docs/TOOL-EVALS.md#10-mcp-client) - Added The model driver caches the prompt on Anthropic (top-level `cache_control`; `promptCache = false` turns it off) and reads the final cache counts from the stream; the report shows cost with cache prices beside flat prices, the cache hit share, and says when the cheaper arm flips. [Tool evals](https://github.com/jackulau/coding.sh/blob/main/docs/TOOL-EVALS.md#42-model-driver-agent-free) - Changed The model driver's 16 KB tool-output cap is structure-aware by default: JSON stays valid and keeps every key, and logs keep whole lines; output under the cap is unchanged. `outputCap = { mode = "prefix" }` on a driver or variant brings back the old byte cut for an A/B, and the report counts cuts and broken-JSON cuts. [Tool evals](https://github.com/jackulau/coding.sh/blob/main/docs/TOOL-EVALS.md#42-model-driver-agent-free) - Fixed `toolevals validate --against --driver ` no longer counts a trial where the agent never worked on the task as a failing trial. A logged-out agent, a failed setup step, an agent that could not start (exit 127) or one that exited non-zero before its first turn used to make every bad task look like it holds; each such task is now reported as `couldn't run on ` with the reason (`infra: auth`, `agent never ran, exit 127`) and does not hold. `toolevals run` records the same never-ran trials with an error. [Task kinds](https://github.com/jackulau/coding.sh/blob/main/docs/TOOL-EVALS.md#31-task-kinds-bad-guard-control) ## Cache - Added `diff-calls` gives the line where a system prompt edit starts and the share of the prompt after it that the cache could no longer read (`system.firstChangeLine`, `system.wastedPrefixFrac` in `--json` and the viewer API). When over 20% is lost, `Why` says to move dynamic content such as timestamps to the end of the system prompt. [Capture with the proxy](https://github.com/jackulau/coding.sh/blob/main/docs/how-to/capture-with-proxy.md#4-why-did-the-cache-miss) ## Experiments - Added `compare --signals` checks whether each session flag and tool errors predict failure beyond task difficulty: pooled, within-task and within-task-and-bundle AUROC with a task-cluster CI, beside difficulty, bundle and length references, and a verdict (`adds-run-signal`, `tracks-difficulty`, `no-signal`, `too-few-pairs`). [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#do-the-flags-predict-failure) - Added `exp sweep` and a matrix spec in `exp run`, with three or more configs or cells, name the plausibly best set: the configs not significantly worse than the best, by a paired-bootstrap multiple comparison with the best, which holds the truly best config 95% of the time. [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#matrix) - Added `compare --pick N --pick-by irt` fits a Rasch (or 2PL) model on stored runs and runs first the tasks with the most information at the baseline's ability: mid difficulty, sharp discrimination. [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#picking-tasks-and-a-budget) - Added `exp run --record` saves each case-trial's Anthropic and OpenAI calls, and `exp run --replay ` serves them again up to the first changed call (`--replay-until N` stops after call N), so a late change re-runs only the calls it touches. System-prompt and `cache_control` edits still replay; API keys are never stored; with one task, the recorded config runs as a control arm and compare prints its noise floor. See [docs/how-to/replay-llm-calls.md](docs/how-to/replay-llm-calls.md). - Added `exp run --seq K --feedback bit|reasons|critic=` tries a failing case-trial again with feedback on its earlier attempts, up to K attempts, and reports Seq@1..K with intervals, EGS/LGS, unchanged revisions, cost to solve and, with 2+ trials, pass@k against seq@k by cost. The pass rate stays the first attempt's; feedback never shows the expected answer; `exp compare` adds a `seq@K` row and `--verdict seq` (arXiv 2610.37859). [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#sequential-attempts-with-feedback). - Added `exp run --failing `, `--errors-from`, `--case `, `--sample N --seed S` and `--first-n` run part of a dataset. When B ran only on A's failures, `exp compare` warns of regression to the mean, says inconclusive and `--gate` holds until A's config runs on the same cases. [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#running-a-subset-failures-samples-the-first-n). - Added `exp run` stores the experiment so far while it runs; after a crash `exp list` marks it `(partial)`, and `exp run … --resume ` runs only what is left (`--retry-errors` also re-runs its task errors). A resume with other graders, trials, task, dataset version or case selection is refused and names what changed. [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#long-runs-resume-retries-status). - Added `exp sweep --guard |prior` re-runs the start and winning configs on earlier suites and keeps the start config, exit 1, when the winner breaks more than `--guard-tau` of what the start solved there, even when the target suite ships. Shipped sweeps are remembered in `adoptions.jsonl` for `--guard prior`. [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#datasets-and-experiments). - Added `exp compare` takes `--pairwise` more than once: it reports each judge, Cohen's κ between judges, and `replicated-better`, `judges-disagree` or `unresolved`, screens out a judge whose calibration failed, and with `--gate` holds a ship when any judge prefers A or two judges do not both find B better (arXiv 2611.04090). [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#judges). - Added `exp compare --labels-a --labels-b` corrects each LLM judge grader's B − A with human labels of each arm's outputs (PPI++), and warns when the judge errs differently on the two arms. [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#correcting-a-judges-delta-with-human-labels). - Added `codingShReporter` (from `coding.sh/eve`) is an eve eval reporter: each `eve eval` run becomes a dataset version and an experiment with a grade per assertion, sent to `coding.sh collector` or to `coding.sh serve` with an ingest key. Skipped evals are listed, repeats become trials, the server redacts before storing, and a re-sent run is stored once. [eve guide](https://github.com/jackulau/coding.sh/blob/main/docs/integrations/eve.md#send-eve-eval-runs-to-codingsh). - Fixed `compare --signals` no longer gives a verdict from too few tasks. A signal needs task+arm pairs from 8 tasks or more, however many pairs there are; below that the verdict is `too-few-pairs` and the table says how many tasks it found. With one task the 95% CI column shows `–`, where it used to print a zero-width interval such as `[0.65, 0.65]`. A coin-flip signal on one task with `--k 10` used to get `adds-run-signal` in about 45% of tables. `--json` adds `withinTaskArm.tasks` to each row. [Evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#do-the-flags-predict-failure) [Next page v0.1.6, page 2](https://coding.sh/changelog/v0-1-6/2) From [CHANGELOG.md](https://github.com/jackulau/coding.sh/blob/main/CHANGELOG.md) · [RSS](https://coding.sh/changelog.xml) · [All releases](https://coding.sh/changelog) To set coding.sh up for a user, follow the agent workflow at [/agents.md](https://coding.sh/agents.md).