coding.sh refine
Improve AGENTS.md and skills in tested rounds, and protect their safety rules. Every option of coding.sh refine and its subcommands, from its own --help.
coding.sh refine <action> Improve AGENTS.md and skills in tested rounds, and protect their safety rules Commands propose Ask a model for lines that address your sessions' most severe flags; stores them as a candidate test Test candidate edits against your current files; keep, reject or inconclusive per candidate, exit 0 when one is kept apply Write a kept candidate to your files, backing up what it replaces (never commits) rollback Put back what an apply replaced, from its backup log Every round in this repo: verdict, effect CI, guard result, cost and reason, and whether the loop plateaued; --suite climbs show pass rate, cost, time and size per round and the Pareto front run Loop: propose (or take --candidate), test, apply when kept; stop at --rounds, --max-cost or a plateau protected List the harness lines an edit may not delete, reword or relax check Check a harness edit for deleted, reworded or relaxed protected lines; exit 1 when it has any mine Rank failure patterns across sessions: repeated error lines, repeated instructions you typed, flags and failed eval runs, with counts and example sessions ($0) recheck Re-run a suite round's base and candidate under another model (a canary) and record the verdict; suggests a rollback when the edit regresses there distill Hand a coding agent the raw runs of an experiment (or this repo's recent sessions) to analyze with its own code and write one skill; the skill is checked by the critics and ledgered as a candidate for refine test diagnose Diagnose recent sessions with a model into Signals: cited problems grouped into issues, severity scored by code. Without an API key it prints a $0 plan signals List the Signals refine diagnose wrote, or show one with its cited evidence and fix score Score diagnoses against human labels: occurrence, Signal and session precision and recall, coverage, repeat agreement Notes A protected line is inside a <!-- coding.sh:protect --> … <!-- coding.sh:protect end --> block, matches a [refine] protect regex, or pairs a gate word (never, don't, must not, only, before, ask, confirm, verify, check, require) with a safety topic (credentials, secrets, access tokens, private keys, .env, prod, deploy, push, --force, force-push, destructive, --no-verify, rm -rf, delete, drop table, reset --hard, migrations, permissions, sudo, chmod, payments, PII, another user, owner, dry-run, backups). Set [refine] autoProtect = false in your user config to keep only markers and patterns. check rejects an edit that deletes or rewords a protected line (protected-line), or adds a line that waives one (skip, no need, only … when, unless, without asking, already verified …) and shares a backticked token, a word or a safety topic with it (relaxes-protected). exp ab runs the same check on its arms and warns; with --gate a protected-line problem fails the run. test decides each candidate in stages: the protected-line check, the harmless and duplicate critics on the added lines, an A/B against your current files on the decision points exp ab mines (the acceptance gate: ship goes on, hold is inconclusive, don't ship rejects), then the guard set: the decision points earlier kept edits won must not go from solved to unsolved. A run stopped at --max-cost is inconclusive. The ledger and every tested candidate's files live under the data dir (refine/<repo>-<hash>/). Rejected candidates and their reasons go into the next proposal prompt; protected lines go in as READ-ONLY. With --suite, each round runs the suite's repo tasks --trials times (default: the suite's trials, else 3) with the current files and with the candidate, interleaved, as one durable job; --model is the agent's model, --concurrency the parallel sessions, --max-cost caps agent and proposer spend. The candidate is kept when the pass rate is significantly better (one-sided paired permutation test over tasks, or the acceptance gate says ship), or when it holds within --margin (one-sided bootstrap lower bound) and a listed objective is significantly better by --min-gain (per-task log ratios, one-sided permutation test); a hard regression or a don't-ship gate rejects it, and so does a listed objective significantly worse, unless the pass rate is significantly better and the rise is within --max-cost-rise (default 50%; the verdict prints the trade). Below 5 tasks no test can reach significance. With --suite and a primary objective other than pass, the proposer cuts instead of adding: it sees the target file, its largest sections and what made the last runs slow or expensive (tokens, turns, repeated tool calls, long outputs), and replies with edits: find and replace an exact string, or replace a whole section by its heading. The protected-line check and the critics still run on the result. With --suite and --canary-model, a kept candidate that deletes or rewords existing lines (every kept candidate with --canary all) runs again with both arms under the canary model. It is rejected (canary-regression) when a task the base passed in every attempt passes in at most a third of the candidate's, when more of its passing runs tamper with tests, or when the paired bootstrap lower bound of the pass-rate change is at or below -10pp (--canary-delta). With fewer than 3 tasks differing on each side the decision stands, marked canary-uninformative. A subtractive candidate kept without a canary model gets a note that a success-only gate cannot see a lost safeguard the loop's model did not need. recheck runs the same check later for any suite round. distill writes the corpus as episodes.jsonl (one row per run or session: pass, verifier exit, tamper, turns, tool calls and errors, tokens, cost, flags), tool_calls.jsonl (one row per call with 500-character previews and an input hash), a redacted transcript per episode and a copy of the current harness files under prompt/. Cancelled, superseded and infrastructure-error runs, and runs of tasks tagged split:dev, split:test, split:heldout or refine:heldout, are left out. The distiller works in a temp directory and must write out/SKILL.md with name and description frontmatter, at most 160 lines and 10 KB; the harmless and duplicate critics then check it. Each skill is ledgered (op propose) as proposed or rejected; none is applied. diagnose sends each session (cut to the first prompt, the last 40 events and the events around failures when longer than 120) inside an untrusted <trace> block with the session's flags as rule candidates. An occurrence survives only when a quote it cites is in the cited event; ins.* types, tool.invalid-args, out.false-success and out.check-subverted also need the broken requirement cited. A second call groups new occurrences into Signals (existing occurrences never move) and a third writes one fix per changed Signal. Severity is computed from the cited factors and how many analyzed sessions show the issue. Replies are cached by model, prompt and catalog, so a rerun on unchanged sessions makes no call and writes the same signals.json. A Signal with no new occurrence in the next 20 analyzed sessions of the same agent and repo is quiet, which is not evidence a fix worked. score matches by code, never a model: a prediction matches a label in the same session with the same component (type, and name compared after Unicode, case, camelCase and separator normalization) whose top-3 type ids share one id, under a maximum one-to-one matching. Unmatched items count against precision or recall; a rate with nothing to divide by prints —. Without --pred, each diagnosed session flag is a prediction with no component, so it matches on session and type ids alone, and per-flag precision is reported. score writes nothing.
coding.sh refine propose
Ask a model for lines that address your sessions' most severe flags; stores them as a candidate.
coding.sh refine propose [options] Ask a model for lines that address your sessions' most severe flags; stores them as a candidate Options --since <DURATION|DATE> Sessions started since (default: 14d) --agent <AGENT> Only sessions of this agent (repeatable, or a comma list) --repo <DIR|all> Only sessions under this directory (default: this git repo) --provider <NAME> Who proposes, answers the next turn and judges it: an API, or claude-code / codex to run `claude -p` / `codex exec` on your subscription (default: anthropic; with no API key set, a logged-in Claude Code, else Codex) --base-url <URL> API base URL (default: ANTHROPIC_BASE_URL / OPENAI_BASE_URL, else the provider's) --proposer-model <MODEL> Model that writes the edit (default: --judge-model, else --model, else claude-sonnet-5-5 or gpt-6-sol) --proposer-command <CMD> Shell command that writes the edit instead of the proposer model: the prompt on stdin, the JSON reply on stdout (e.g. 'claude -p') --target <PATH> Harness file the proposal edits, or a skill directory for its SKILL.md (default: AGENTS.md, else the repo's Claude Code or Gemini instructions file) --mined <FILE> Also show the proposer the patterns in this file (the output of refine mine --json) --dry-run Print the findings and the prompt; make no model call Examples coding.sh refine propose coding.sh refine propose --target AGENTS.md --dry-run
coding.sh refine test
Test candidate edits against your current files; keep, reject or inconclusive per candidate, exit 0 when one is kept.
coding.sh refine test <CANDIDATE>... [options]
Test candidate edits against your current files; keep, reject or inconclusive per candidate, exit 0 when one is kept
Arguments
candidate Edited file, directory, git:REV, harness bundle or stored r_… id; a file replaces the harness file at its repo path, else the one with its name (./drafts/AGENTS.md replaces AGENTS.md)
Options
--behavior <ID[,ID]> Decision points to test: unverified-claim, done-while-failing, unchanged-retry (default: all)
--anchor <FILE> Module exporting your own anchor(s) (see the evals guide)
--since <DURATION|DATE> Sessions started since (default: 14d)
--agent <AGENT> Only sessions of this agent (repeatable, or a comma list)
--repo <DIR|all> Only sessions under this directory (default: this git repo)
--provider <NAME> Who proposes, answers the next turn and judges it: an API, or claude-code / codex to run `claude -p` / `codex exec` on your subscription (default: anthropic; with no API key set, a logged-in Claude Code, else Codex)
--base-url <URL> API base URL (default: ANTHROPIC_BASE_URL / OPENAI_BASE_URL, else the provider's)
--judge-model <MODEL> Model for the judge (default: --model)
-n, --limit <N> Maximum decision points (default: 40)
--model <MODEL> Model for the next turn (default: the one the decision points were recorded with, else claude-sonnet-5-5 or gpt-6-sol)
--trials <N> Next turns per case and arm (default: 1)
-c, --concurrency <N> Parallel model calls (default: 4)
--max-cost <USD> Refuse to start when the estimate is over this, and stop once the spend passes it
--max-guard <N> Most tasks in the guard set (the tasks earlier kept edits won), newest kept edits first (default: 24)
--cache Reuse stored model responses (--no-cache to ask the model again) (default: true)
--lambda <N> Acceptance gate: what breaking one solved decision point costs, in fixed ones (default: 2)
--gate-alpha <A> Acceptance gate: one-sided significance level (default: 0.05)
--tau <T> Acceptance gate: largest share of solved decision points B may break (default: 0.10)
--allow-protected-edit <PATH:LINE> Allow the edit to delete, reword or relax this protected line (repeatable; never implied by --force)
--suite <NAME> Judge candidates on this eval suite of repo tasks (evals/<NAME>) instead of mined decision points; a skill --target uses evals/<skill> when it exists
--objective <OBJ[,OBJ]> With --suite: what to improve, primary first: pass, cost, tokens, wall, llm, turns, size (default: pass). The pass rate must hold within --margin whatever the list
--margin <SHARE> With --suite: how far the pass rate may drop for a cheaper or faster candidate to be kept, e.g. 5pp (default: 2pp)
--min-gain <SHARE> With --suite: smallest relative gain in cost, tokens, time, turns or size that counts, e.g. 20% (default: 10%)
--max-cost-rise <RISE> With --suite: how far a listed cost, tokens, time, turns or size objective may rise for a significant pass-rate gain, e.g. 100%, or none (default: refine.maxCostRise, else 50%) (default: 50%)
--harness <BUNDLE> With --suite: stored harness bundle the suite runs, the repo's harness files laid over it (default: the suite's, else a snapshot of the repo)
--reuse <WHAT> With --suite: copy earlier rounds' finished runs of the unchanged base arm (baseline) or both arms (both) instead of running them again; the pair then did not run interleaved, so the round warns about drift (default: none) (default: none)
--reuse-max-age <AGE> With --reuse: only copy runs that finished this recently, e.g. 3d or 12h (default: 7d)
--canary-model <MODEL> With --suite: re-run a kept candidate's two arms under this cheaper model and reject it on a regression there (default: [refine] canary_model)
--canary <WHEN> With --suite: which kept candidates the canary re-tests: the ones that delete or reword existing lines (subtractive), every one (all), or none (off) (default: subtractive)
--canary-delta <SHARE> With --canary-model: how far the canary pass rate's lower bound may fall before the candidate is rejected, e.g. 10pp (default: 10pp)
--resume [<ID>] Continue an interrupted round: finished trials are kept, only the rest run (no ID: the newest unfinished one with the same config)
--retries <N> Retries per trial after a rate limit, 5xx or network error, with backoff; a failed setup step is retried once and a refused login never (failures are never retried) (default: 3)
--stop-after-infra <N> Stop once N trials in a row fail on infrastructure after their retries (a refused login, a failed setup, rate limits, 5xx, network): no new trial starts and the run is no quality verdict; 0 never stops (default: 5)
--trial-timeout <SEC> Time limit per trial, retries included; a trial over it fails
-y, --yes Answer yes to confirmation prompts. An agent passes it only once the user approves
--dry-run Show the checks, decision points, guard set size and cost estimate; make no model calls
Examples
coding.sh refine test ./drafts/AGENTS.md
coding.sh refine test r_3f9a1c02e1 --behavior unverified-claim --max-cost 2
coding.sh refine test ./a/AGENTS.md ./b/AGENTS.md --dry-run --json coding.sh refine apply
Write a kept candidate to your files, backing up what it replaces (never commits).
coding.sh refine apply [candidate] [options] Write a kept candidate to your files, backing up what it replaces (never commits) Arguments candidate Candidate id (default: the latest kept candidate not yet applied) Options --force Apply a candidate that was not kept, or whose files changed since its test (recorded in the ledger) Examples coding.sh refine apply coding.sh refine apply r_3f9a1c02e1
coding.sh refine rollback
Put back what an apply replaced, from its backup.
coding.sh refine rollback [candidate] [options] Put back what an apply replaced, from its backup Arguments candidate Candidate or apply id (default: the latest apply) Options --force Roll back though the file changed since the apply (those changes are lost) Examples coding.sh refine rollback coding.sh refine rollback r_3f9a1c02e1
coding.sh refine log
Every round in this repo: verdict, effect CI, guard result, cost and reason, and whether the loop plateaued; --suite climbs show pass rate, cost, time and size per round and the Pareto front.
coding.sh refine log Every round in this repo: verdict, effect CI, guard result, cost and reason, and whether the loop plateaued; --suite climbs show pass rate, cost, time and size per round and the Pareto front Examples coding.sh refine log coding.sh refine log --json
coding.sh refine run
Loop: propose (or take --candidate), test, apply when kept; stop at --rounds, --max-cost or a plateau.
coding.sh refine run [options]
Loop: propose (or take --candidate), test, apply when kept; stop at --rounds, --max-cost or a plateau
Options
--rounds <N> Most rounds (default: 3, none with --until-plateau)
--candidate <FILE|r_…> Test these in order instead of asking the proposer (repeatable)
--until-plateau With --suite: keep climbing until the e-detector says the climb stopped gaining; --resume continues an interrupted round
--proposer-model <MODEL> Model that writes the edit (default: --judge-model, else --model, else claude-sonnet-5-5 or gpt-6-sol)
--mined <FILE> Also show the proposer the patterns in this file (the output of refine mine --json)
--proposer-command <CMD> Shell command that writes the edit instead of the proposer model: the prompt on stdin, the JSON reply on stdout (e.g. 'claude -p')
--target <PATH> Harness file the proposal edits, or a skill directory for its SKILL.md (default: AGENTS.md, else the repo's Claude Code or Gemini instructions file)
--behavior <ID[,ID]> Decision points to test: unverified-claim, done-while-failing, unchanged-retry (default: all)
--anchor <FILE> Module exporting your own anchor(s) (see the evals guide)
--since <DURATION|DATE> Sessions started since (default: 14d)
--agent <AGENT> Only sessions of this agent (repeatable, or a comma list)
--repo <DIR|all> Only sessions under this directory (default: this git repo)
--provider <NAME> Who proposes, answers the next turn and judges it: an API, or claude-code / codex to run `claude -p` / `codex exec` on your subscription (default: anthropic; with no API key set, a logged-in Claude Code, else Codex)
--base-url <URL> API base URL (default: ANTHROPIC_BASE_URL / OPENAI_BASE_URL, else the provider's)
--judge-model <MODEL> Model for the judge (default: --model)
-n, --limit <N> Maximum decision points (default: 40)
--model <MODEL> Model for the next turn (default: the one the decision points were recorded with, else claude-sonnet-5-5 or gpt-6-sol)
--trials <N> Next turns per case and arm (default: 1)
-c, --concurrency <N> Parallel model calls (default: 4)
--max-cost <USD> Refuse to start when the estimate is over this, and stop once the spend passes it
--max-guard <N> Most tasks in the guard set (the tasks earlier kept edits won), newest kept edits first (default: 24)
--cache Reuse stored model responses (--no-cache to ask the model again) (default: true)
--lambda <N> Acceptance gate: what breaking one solved decision point costs, in fixed ones (default: 2)
--gate-alpha <A> Acceptance gate: one-sided significance level (default: 0.05)
--tau <T> Acceptance gate: largest share of solved decision points B may break (default: 0.10)
--allow-protected-edit <PATH:LINE> Allow the edit to delete, reword or relax this protected line (repeatable; never implied by --force)
--suite <NAME> Judge candidates on this eval suite of repo tasks (evals/<NAME>) instead of mined decision points; a skill --target uses evals/<skill> when it exists
--objective <OBJ[,OBJ]> With --suite: what to improve, primary first: pass, cost, tokens, wall, llm, turns, size (default: pass). The pass rate must hold within --margin whatever the list
--margin <SHARE> With --suite: how far the pass rate may drop for a cheaper or faster candidate to be kept, e.g. 5pp (default: 2pp)
--min-gain <SHARE> With --suite: smallest relative gain in cost, tokens, time, turns or size that counts, e.g. 20% (default: 10%)
--max-cost-rise <RISE> With --suite: how far a listed cost, tokens, time, turns or size objective may rise for a significant pass-rate gain, e.g. 100%, or none (default: refine.maxCostRise, else 50%) (default: 50%)
--harness <BUNDLE> With --suite: stored harness bundle the suite runs, the repo's harness files laid over it (default: the suite's, else a snapshot of the repo)
--reuse <WHAT> With --suite: copy earlier rounds' finished runs of the unchanged base arm (baseline) or both arms (both) instead of running them again; the pair then did not run interleaved, so the round warns about drift (default: none) (default: none)
--reuse-max-age <AGE> With --reuse: only copy runs that finished this recently, e.g. 3d or 12h (default: 7d)
--canary-model <MODEL> With --suite: re-run a kept candidate's two arms under this cheaper model and reject it on a regression there (default: [refine] canary_model)
--canary <WHEN> With --suite: which kept candidates the canary re-tests: the ones that delete or reword existing lines (subtractive), every one (all), or none (off) (default: subtractive)
--canary-delta <SHARE> With --canary-model: how far the canary pass rate's lower bound may fall before the candidate is rejected, e.g. 10pp (default: 10pp)
--resume [<ID>] Continue an interrupted round: finished trials are kept, only the rest run (no ID: the newest unfinished one with the same config)
--retries <N> Retries per trial after a rate limit, 5xx or network error, with backoff; a failed setup step is retried once and a refused login never (failures are never retried) (default: 3)
--stop-after-infra <N> Stop once N trials in a row fail on infrastructure after their retries (a refused login, a failed setup, rate limits, 5xx, network): no new trial starts and the run is no quality verdict; 0 never stops (default: 5)
--trial-timeout <SEC> Time limit per trial, retries included; a trial over it fails
-y, --yes Answer yes to confirmation prompts. An agent passes it only once the user approves
--dry-run Show the findings, decision points, guard set size and per-round estimate; make no model calls
Examples
coding.sh refine run --rounds 5 --max-cost 10
coding.sh refine run --candidate ./v1/AGENTS.md --candidate ./v2/AGENTS.md
coding.sh refine run --dry-run coding.sh refine protected
List the harness lines an edit may not delete, reword or relax.
coding.sh refine protected [options] List the harness lines an edit may not delete, reword or relax Options --a <FILE|DIR|git:REV|BUNDLE> Harness to read (default: the repo's current AGENTS.md and other instructions files, rules and skills) Examples coding.sh refine protected coding.sh refine protected --a git:main --json
coding.sh refine check
Check a harness edit for deleted, reworded or relaxed protected lines; exit 1 when it has any.
coding.sh refine check [options] Check a harness edit for deleted, reworded or relaxed protected lines; exit 1 when it has any Options --a <FILE|DIR|git:REV|BUNDLE|none> Before (default: git HEAD when B is omitted, else the repo's current files) --b <FILE|DIR|git:REV|BUNDLE|none> After; a file replaces the file of the same path in A (default: your uncommitted edits) --allow-protected-edit <PATH:LINE> Allow the edit to delete, reword or relax this protected line (repeatable; never implied by --force) Examples coding.sh refine check coding.sh refine check --b ./drafts/AGENTS.md coding.sh refine check --a git:main --b . --json
coding.sh refine mine
Rank failure patterns across sessions: repeated error lines, repeated instructions you typed, flags and failed eval runs, with counts and example sessions ($0).
coding.sh refine mine [options]
Rank failure patterns across sessions: repeated error lines, repeated instructions you typed, flags and failed eval runs, with counts and example sessions ($0)
Options
--since <DURATION|DATE> Sessions started since (default: 30d)
--agent <AGENT> Only sessions of this agent (repeatable, or a comma list)
--repo <DIR|all> Only sessions under this directory, and eval runs of tasks that start from it or of the suites under its evals/ (default: this git repo)
--min-support <SESSIONS[,DAYS]> Keep a pattern seen in at least this many sessions over this many days (default: 3,2)
-n, --limit <N> Most patterns shown (default: 20)
Examples
coding.sh refine mine
coding.sh refine mine --since 90d --repo all --json
coding.sh refine mine --min-support 2,1 --agent codex coding.sh refine recheck
Re-run a suite round's base and candidate under another model (a canary) and record the verdict; suggests a rollback when the edit regresses there.
coding.sh refine recheck <CANDIDATE> [options]
Re-run a suite round's base and candidate under another model (a canary) and record the verdict; suggests a rollback when the edit regresses there
Arguments
candidate The round's r_… id
Options
--model <MODEL> The model to re-run under (default: [refine] canary_model)
--trials <N> Runs per task and arm (default: the suite's trials, else 3)
--canary-delta <SHARE> How far the pass rate's lower bound may fall before it counts as a regression (default: 10pp)
--max-cost <USD> Stop starting runs once agent spend reaches this; the recheck is then inconclusive
--concurrency <N> Agent sessions at once (default: 4)
--dry-run Print what would run; start no agent
-y, --yes Answer yes to confirmation prompts. An agent passes it only once the user approves
Examples
coding.sh refine recheck r_3f9a1c02e1 --model claude-haiku-4-5
coding.sh refine recheck r_3f9a1c02e1 --model gpt-5-mini --max-cost 3 --yes --json coding.sh refine distill
Hand a coding agent the raw runs of an experiment (or this repo's recent sessions) to analyze with its own code and write one skill; the skill is checked by the critics and ledgered as a candidate for refine test.
coding.sh refine distill [options]
Hand a coding agent the raw runs of an experiment (or this repo's recent sessions) to analyze with its own code and write one skill; the skill is checked by the critics and ledgered as a candidate for refine test
Options
--experiment <ID> Distill from this experiment's runs (default: captured sessions)
--since <DURATION|DATE> Without --experiment: sessions started since (default: 30d)
--agent <AGENT> Without --experiment: only sessions of this agent (repeatable, or a comma list)
--repo <DIR|all> Without --experiment: only sessions under this directory (default: this git repo)
--distiller <AGENT> The coding agent that analyzes the corpus (default: claude-code)
--model <MODEL> The distiller's model (default: the agent's own)
--n <N> Independent skills to draw, each in its own directory (default: 1)
--max-cost <USD> Start no new draw once distiller spend reaches this (default: 5)
--min-episodes <N> Refuse a corpus with fewer episodes (default: 20)
--min-failures <N> Refuse a corpus with fewer failing or flagged episodes (default: 5)
--keep-corpus <DIR> Write the corpus here and keep it: an empty or new directory, or one that holds only an earlier corpus, which is replaced. Nothing is written there with --dry-run
--dry-run Write the corpus to a temp dir, print its counts and the plan; start no agent
-y, --yes Answer yes to confirmation prompts. An agent passes it only once the user approves
Examples
coding.sh refine distill --experiment exp_01J9Z --dry-run
coding.sh refine distill --experiment exp_01J9Z --n 3 --yes
coding.sh refine distill --since 60d --agent codex --distiller codex --keep-corpus ./corpus --yes --json coding.sh refine diagnose
Diagnose recent sessions with a model into Signals: cited problems grouped into issues, severity scored by code. Without an API key it prints a $0 plan.
coding.sh refine diagnose [options] Diagnose recent sessions with a model into Signals: cited problems grouped into issues, severity scored by code. Without an API key it prints a $0 plan Options --since <DURATION|DATE> Sessions started since (default: 7d) --agent <AGENT> Only sessions of this agent (repeatable, or a comma list) --repo <DIR|all> Only sessions under this directory (default: this git repo) --session <ID> Exactly these sessions: an id, a prefix or latest (repeatable; ignores --since, --repo and --agent) --max-sessions <N> Most sessions diagnosed, newest first (default: 50) --budget-tokens <N> Stop starting uncached sessions past this many estimated input tokens; the next run picks up the rest --provider <PROVIDER> anthropic, openai, claude-code or codex (default: the API whose key is set; with none, a $0 plan) --model <MODEL> The diagnosing model (default: the provider's current one) --concurrency <N> Sessions diagnosed at once (default: 4) --no-split One Signal per component and type, with no grouping call --include-runs Also diagnose eval-run sessions --dry-run Print the plan and make no call Examples coding.sh refine diagnose coding.sh refine diagnose --since 30d --budget-tokens 400000 coding.sh refine diagnose --session 7f3a9c --provider claude-code --json
coding.sh refine signals
List the Signals refine diagnose wrote, or show one with its cited evidence and fix.
coding.sh refine signals [show] [id] List the Signals refine diagnose wrote, or show one with its cited evidence and fix Arguments show show <id>: one Signal's occurrences, citations and fix id With show: the Signal id, or a prefix of it Examples coding.sh refine signals coding.sh refine signals show 3fa1c2 coding.sh refine signals --json
coding.sh refine score
Score diagnoses against human labels: occurrence, Signal and session precision and recall, coverage, repeat agreement.
coding.sh refine score <LABELS> [options]
Score diagnoses against human labels: occurrence, Signal and session precision and recall, coverage, repeat agreement
Arguments
labels Labels JSONL: one occurrence per line, {sessionId, signalId, component: {type, name}, typeIds: [1-3 ids], nature?}; a line with only sessionId marks a session with nothing found
Options
--pred <FILE> Predictions in the labels format (repeatable; 2 or more adds repeat agreement). Default: the labelled sessions' flags from the store
Examples
coding.sh refine score labels.jsonl --pred diagnoses.jsonl
coding.sh refine score labels.jsonl --pred run1.jsonl --pred run2.jsonl --pred run3.jsonl --json
coding.sh refine score labels.jsonl