GitHub
Changelog

Unreleased · 146 changes · page 1 of 4

coding.sh v0.1.6

Privacy 5

  • # Added CODING_SH_PURGE_WAIT_MS sets how long purge waits for another process's read of the index or the blob pack before it says which file still holds the old bytes. The defaults stay 10 s before the VACUUM and 5 s after it. Configuration.
  • # Added An opt-in PII pack: capture.redact = ["secrets", "pii"] also redacts email addresses, phone numbers, card numbers (issuer prefix and Luhn check), IBANs (country length and mod 97) and national ids (US SSN, UK NINO, Canadian SIN). Each kind (phone, card, iban, national-id, email) can be turned on alone. Privacy.
  • # Fixed A purge given a wait (CODING_SH_PURGE_WAIT_MS, or waitMs in code) now keeps to it while another process reads the index. The checkpoint before the VACUUM used to wait the full 10 s whatever wait was given. Privacy.
  • # Fixed The env redactor no longer replaces PWD, OLDPWD or variables named *_PATH, *_DIR, *_FILE or *_SOCK, or any value that is an existing path, so the working directory stopped turning into [redacted:env] in every path. Privacy.
  • # Fixed A source span or log record that coding.sh serve or coding.sh collector --keep-raw keeps as a blob, and an oversize structured value moved to a blob, is now redacted one string at a time (a string that holds JSON as JSON) before it is serialized. A NAME=value line after a newline and a JSON field such as api_key or password used to stay in clear in the stored blob and were served by /api/blobs/<sha>, while the event showed them as redacted. Blobs stored before 0.1.6 are not rewritten: remove such a value with coding.sh purge. Privacy

Trace format 1

  • # Added A JSON Schema (draft 2020-12) for AgentEvent and Session, generated from the TypeScript types, ships as coding.sh/schema.json and is served at https://coding.sh/schema.json. Every adapter and OTLP fixture validates against it. Trace format

Serve 2

  • # Added CODING_SH_FORWARD_URL (or [serve.forward]) makes coding.sh serve forward every trace it commits to an OTLP/HTTP collector, with the same trace and span ids plus coding_sh.project. Native batches go out as invoke_agent, execute_tool and chat spans. Metadata mode sends no messages, tool arguments or results; a down collector never slows ingest, spans wait in a bounded buffer, and overflow is counted in coding_sh_forward_spans_total. Forward and sinks
  • # Added createFetchHandler() from coding.sh/server runs serve's ingest routes (OTLP, /v1/ingest, /readyz) as a Request → Response function for Vercel Functions, Deno, Bun or Workers. With coding.sh-store-postgres, several instances can write one session at once: each session rebuild takes pg_advisory_xact_lock on the session id. Serverless

Sinks 6

  • # Added [[sinks]] now mirror events from the OTLP receiver (coding.sh collector) and from coding.sh serve, so a team can set one sink on serve instead of on every laptop. Config
  • # Changed The otlp sink sends gen_ai.usage.input_tokens as the whole prompt, cache reads and writes included, as the OpenTelemetry GenAI conventions ask; it used to send only the uncached part, so backends undercounted cached calls. Cache writes move to gen_ai.usage.cache_write.input_tokens (the old cache_creation.input_tokens is still sent through 0.1.5), the uncached part is coding_sh.usage.uncached_input_tokens, and reported reasoning tokens are gen_ai.usage.reasoning.output_tokens. The OTLP receiver reads the sink's spans back to the same token counts and cost. Export to ClickHouse.
  • # Fixed Metadata sinks no longer receive a session's stats.llmErrors.error text, which could quote the prompt. Config
  • # Fixed coding.sh serve and coding.sh collector now send the otlp sink a session's invoke_agent span while the session is still open: once it has been quiet for 10 seconds, and every 60 seconds while it stays busy. An OTLP session keyed by a conversation id (Claude Code, Codex, gen_ai.conversation.id, Agent Runs) never ends, so it used to reach the sink as chat and execute_tool spans with no parent and no session row. Each send repeats the span id with newer totals, and the ClickHouse queries now take the latest row per span id. Export to ClickHouse
  • # Fixed coding.sh serve and coding.sh collector send an event to [[sinks]] once. A request that was posted again, such as a client retry answered "inserted":0, used to send all its events again and doubled rows and cost downstream; now only the events a batch newly stored go on, and a batch that overlaps stored events sends the new ones. A commit that fails after its events were written sends them when the retry commits. Forward and sinks
  • # Fixed Local ingest now sends the events it imports from agent transcripts to [[sinks]], not only the events from the hook spool, so a sink has every model call the store has. When an import adds events to a session without ending it, the otlp sink gets the session's invoke_agent span again with the new totals. A sink that is down does not stop the import and catches up after its backoff, and a second ingest sends nothing new. Config

Evals 11

  • # Added Cases and tool-eval tasks take an optional kind: bad (fails before the change, passes after), guard or control (pass on both). exp run, exp compare, toolevals run and toolevals compare split results by kind and name broken controls as overcorrection, and toolevals validate --against <variant> reports a bad task that passes on the old variant. See [Case kinds](docs/guides/evals.md#case-kinds-bad-guard-control) and [Task kinds](docs/TOOL-EVALS.md#31-task-kinds-bad-guard-control).
  • # Added Typed scores: a number, true or false, a category or text on a session, event, dataset case or eval run, from a person, judge, code, the API or a monitor. Score configs fix a name's type and range, and a value outside it is refused. Add and list them with coding.sh score, POST/GET /api/scores and /api/score-configs, or the score_add and score_list MCP tools. Existing labels move into scores as human pass or fail scores, and labels stays readable as a view. [Scores](docs/guides/evals.md#scores)
  • # Added coding.sh eval candidates (and the MCP tool eval_candidates) ranks captured sessions as eval tasks. Subagents, runs, and sessions with no git start, no prompt or no edits are left out; the rest are ordered by failure flags, pushback, a guessable verifier and recency. eval init --sessions drafts from that ranking and says how many it left out. Your agent's failed sessions.
  • # Added eval init --where '<query>' --validate drafts tasks from the sessions a query matches and validates each one. A session diff that passes the verifier becomes the reference and the task is marked validated. Tasks from flagged sessions are tagged captured-failure, and eval run (and the GitHub Action's comment) says which of them now pass. Your agent's failed sessions.
  • # Added eval new --statement ledger writes the task instruction from every user turn and approved proposal, as cited obligations, instead of the first prompt only. With --judge a model merges them and drops the ones a later turn replaced, and code checks every citation and the leakage of the reference patch. Session to eval.
  • # Added eval trigger <skill> --draft writes triggers.toml. It starts from the prompts of this repo's captured sessions that loaded the skill (should) or a sibling skill (should_not), then has a model fill each side to 8. Mined prompts are redacted, each prompt records its source, and the reply is cached. Evaluate an Agent Skill.
  • # Fixed eval new --statement ledger keeps a reply that approves and goes on ("yes, but call it goodbye instead") as its own obligation, after the proposal it changes. A short reply that opened with an approval word used to count as a bare approval, so the instruction asked for the assistant's proposal and dropped the user's change. Session to eval
  • # Fixed eval init --validate no longer marks a session that rewrote the test as the validated reference. A session diff that edits, deletes or skips the tests the verifier runs is kept as a near miss, with or without --validate, and a note names the test file. A diff that only adds test cases to a test file still counts as a reference. Your agent's failed sessions
  • # Fixed eval trigger <skill> --draft reads only the sessions of the current git repo, so a triggers.toml you commit no longer holds prompts typed in other projects. --repo DIR reads the sessions under another directory and --repo all reads every repo's. Does the skill load?
  • # Fixed coding.sh score add event:<id> --session <ID> (and POST /api/scores with a sessionId) no longer stores a score on an event that is not in the store, or under another session. An unknown event is event not found (exit 1, HTTP 404), and a session that is not the event's own is refused with both ids named (exit 2, HTTP 400). Scores
  • # Fixed A stored score keeps its type. A value that reads as another type (1,5 over a number, 0.5 over a pass or fail label) is refused (exit 2, HTTP 400) and names --type and score config, instead of quietly turning the score into a category. --type, dataType or a config for the name still change the type, and score add now prints the type: Scored n1 = -5 (numeric). Scores

Tool evals 6

  • # Added coding.sh toolevals traces <server> --sessions 'since:30d' (or --from <experiment>) prints each MCP tool's calls, errors, repeats and retries after an error, and --out <dir> writes the failing traces first (at most 3 passing), an index.jsonl, tool-health.json and tools.json for an improver to read. Held-out trials are never exported. Tool evals
  • # Added toolevals run --shell-model <provider/model> has a model play the shell for a model driver on a CLI target, with a fixed date; the shell model is part of the arm name and compare refuses arms with different shells. Tool evals
  • # Added The model driver's MCP client reads resources and prompts, counts progress notifications, runs task-augmented tool calls, answers sampling with the driver's model, elicitation with a task's scripted [mcp] answers and roots with the sandbox, and runs OAuth (discovery, client registration, PKCE or client credentials) on a 401. calls checks can match resources/read by uri and prompts/get by name. MCP client
  • # Added The model driver caches the prompt on Anthropic (top-level cache_control; promptCache = false turns it off) and reads the final cache counts from the stream; the report shows cost with cache prices beside flat prices, the cache hit share, and says when the cheaper arm flips. Tool evals
  • # Changed The model driver's 16 KB tool-output cap is structure-aware by default: JSON stays valid and keeps every key, and logs keep whole lines; output under the cap is unchanged. outputCap = { mode = "prefix" } on a driver or variant brings back the old byte cut for an A/B, and the report counts cuts and broken-JSON cuts. Tool evals
  • # Fixed toolevals validate --against <variant> --driver <d> no longer counts a trial where the agent never worked on the task as a failing trial. A logged-out agent, a failed setup step, an agent that could not start (exit 127) or one that exited non-zero before its first turn used to make every bad task look like it holds; each such task is now reported as couldn't run on <variant> with the reason (infra: auth, agent never ran, exit 127) and does not hold. toolevals run records the same never-ran trials with an error. Task kinds

Cache 1

  • # Added diff-calls gives the line where a system prompt edit starts and the share of the prompt after it that the cache could no longer read (system.firstChangeLine, system.wastedPrefixFrac in --json and the viewer API). When over 20% is lost, Why says to move dynamic content such as timestamps to the end of the system prompt. Capture with the proxy

Experiments 12

  • # Added compare --signals checks whether each session flag and tool errors predict failure beyond task difficulty: pooled, within-task and within-task-and-bundle AUROC with a task-cluster CI, beside difficulty, bundle and length references, and a verdict (adds-run-signal, tracks-difficulty, no-signal, too-few-pairs). Evals guide
  • # Added exp sweep and a matrix spec in exp run, with three or more configs or cells, name the plausibly best set: the configs not significantly worse than the best, by a paired-bootstrap multiple comparison with the best, which holds the truly best config 95% of the time. Evals guide
  • # Added compare --pick N --pick-by irt fits a Rasch (or 2PL) model on stored runs and runs first the tasks with the most information at the baseline's ability: mid difficulty, sharp discrimination. Evals guide
  • # Added exp run --record saves each case-trial's Anthropic and OpenAI calls, and exp run --replay <EXP> serves them again up to the first changed call (--replay-until N stops after call N), so a late change re-runs only the calls it touches. System-prompt and cache_control edits still replay; API keys are never stored; with one task, the recorded config runs as a control arm and compare prints its noise floor. See [docs/how-to/replay-llm-calls.md](docs/how-to/replay-llm-calls.md).
  • # Added exp run --seq K --feedback bit|reasons|critic=<grader> tries a failing case-trial again with feedback on its earlier attempts, up to K attempts, and reports Seq@1..K with intervals, EGS/LGS, unchanged revisions, cost to solve and, with 2+ trials, pass@k against seq@k by cost. The pass rate stays the first attempt's; feedback never shows the expected answer; exp compare adds a seq@K row and --verdict seq (arXiv 2610.37859). Evals guide.
  • # Added exp run --failing <EXP>, --errors-from, --case <PATTERN>, --sample N --seed S and --first-n run part of a dataset. When B ran only on A's failures, exp compare warns of regression to the mean, says inconclusive and --gate holds until A's config runs on the same cases. Evals guide.
  • # Added exp run stores the experiment so far while it runs; after a crash exp list marks it (partial), and exp run … --resume <experiment> runs only what is left (--retry-errors also re-runs its task errors). A resume with other graders, trials, task, dataset version or case selection is refused and names what changed. Evals guide.
  • # Added exp sweep --guard <dataset>|prior re-runs the start and winning configs on earlier suites and keeps the start config, exit 1, when the winner breaks more than --guard-tau of what the start solved there, even when the target suite ships. Shipped sweeps are remembered in adoptions.jsonl for --guard prior. Evals guide.
  • # Added exp compare takes --pairwise more than once: it reports each judge, Cohen's κ between judges, and replicated-better, judges-disagree or unresolved, screens out a judge whose calibration failed, and with --gate holds a ship when any judge prefers A or two judges do not both find B better (arXiv 2611.04090). Evals guide.
  • # Added exp compare --labels-a --labels-b corrects each LLM judge grader's B − A with human labels of each arm's outputs (PPI++), and warns when the judge errs differently on the two arms. Evals guide.
  • # Added codingShReporter (from coding.sh/eve) is an eve eval reporter: each eve eval run becomes a dataset version and an experiment with a grade per assertion, sent to coding.sh collector or to coding.sh serve with an ingest key. Skipped evals are listed, repeats become trials, the server redacts before storing, and a re-sent run is stored once. eve guide.
  • # Fixed compare --signals no longer gives a verdict from too few tasks. A signal needs task+arm pairs from 8 tasks or more, however many pairs there are; below that the verdict is too-few-pairs and the table says how many tasks it found. With one task the 95% CI column shows –, where it used to print a zero-width interval such as [0.65, 0.65]. A coin-flip signal on one task with --k 10 used to get adds-run-signal in about 45% of tables. --json adds withinTaskArm.tasks to each row. Evals guide

From CHANGELOG.md · RSS · All releases