coding.sh judge

coding.sh judge <action>

  Grade one output or the slop in captured diffs, or check an LLM judge or a test verifier against human labels

Commands
  grade         Run one grader on one output and trace (exit 0 on pass); inside a verifier both default to the agent's
  slop          Flag agent-slop patterns (placeholder code, type escapes, swallowed errors, debug prints, focused tests) in the lines captured work added
  calibrate     Measure judge-human agreement (Cohen's kappa) on a held-out split, or a verifier's precision
  sample        Plan which cases to label next: a seeded random sample stratified by judge verdict and confidence

Notes
  judge grade appends each model call's tokens and price to $CODING_SH_GRADER_USAGE when it is set. coding.sh compare sets it for every verifier, so judge spend counts toward --max-cost.

coding.sh judge grade [options]

  Run one grader on one output and trace (exit 0 on pass); inside a verifier both default to the agent's

Options
  -g, --grader <JSON|FILE>    Grader spec, the same JSON as `exp run --grader` (llm, llm-checklist with trace checks, contains, regex, …)
      --output <FILE|->       The output to grade (default: $CODING_SH_AGENT_OUTPUT, the agent's final answer in a verifier)
      --trace <FILE>          Claude stream-json or transcript, codex exec --json, Cursor stream-json, OpenAI chat messages or Responses items, or coding.sh events as JSONL (default: $CODING_SH_AGENT_STDOUT)
      --allow-partial-trace   Grade even when --trace has cut lines, records the judge does not read, or tool calls that did not reach it (otherwise exit 2, trace_incomplete)
      --input <TEXT|@FILE>    The task input shown to a judge
      --metadata <JSON|FILE>  Case metadata, e.g. {"instructions": "…"} for follows-rules
      --diff <FILE|->         The agent's work as a unified diff, for agent-slop (default: $CODING_SH_AGENT_DIFF in a verifier)

Examples
  coding.sh judge grade --grader checklist.json
  coding.sh judge grade --grader judge.json --output answer.txt --trace stream.jsonl --json
  coding.sh judge grade --grader '{"kind":"agent-slop","maxWarnings":5}'

coding.sh judge slop [target] [options]

  Flag agent-slop patterns (placeholder code, type escapes, swallowed errors, debug prints, focused tests) in the lines captured work added

Arguments
  target        Session id, prefix or 'latest' (default), a run id (run_…), or a compare experiment id (exp_…) for a rate per arm

Options
  --diff <FILE|->     Grade this unified diff instead of captured work
  --rules <ID>        Rules to run instead of the defaults; 'default' and 'all' name those sets (repeatable)
  --exclude <ID>      Rule to leave out (repeatable)
  --max-errors <N>    Pass with at most N error findings (default: 0)
  --max-warnings <N>  Pass with at most N warnings (default: no limit)
  --list-rules        Print the rule catalog and exit
  --fresh             Wait for every pending capture to be ingested (default: answer after 1.5 s)

Examples
  coding.sh judge slop latest
  coding.sh judge slop exp_01JC3T6Z2Q8W1S5V0R7N4K9M2D --json
  git diff main | coding.sh judge slop --diff -

coding.sh judge calibrate <EXPERIMENT> [options]

  Measure judge-human agreement (Cohen's kappa) on a held-out split, or a verifier's precision

Arguments
  experiment    Experiment id or name whose runs hold the judge's verdicts

Options
  -g, --grader <ID>                  Grader id (default: the only llm grader); a command or other deterministic grader gets a precision audit
      --criterion <NAME>             Label criterion
      --split <SPLIT>                Held-out split to report on (default: test)
      --tuned-on <SPLIT>             Splits the prompt was tuned on, when the judge is not stored (repeatable)
      --threshold <KAPPA>            Minimum kappa (default: 0.6)
      --min-n <N>                    Minimum held-out cases (judges) or labelled passes (verifiers) (default: 20)
      --probe-injection <JSON|FILE>  The judge's grader spec: re-judge human-fail cases with a planted 'respond pass' payload and report how many verdicts flip (exit 1 on any flip)
      --probe-max <N>                With --probe-injection: human-fail cases to probe (default: 25)
      --labels <FILE>                Extra human labels (JSON array or JSONL of {"caseId","value":"pass"|"fail"|"skip"}); they replace stored labels for the cases they name
      --sample-passes <N>            Print N unlabelled passing cases to label (a seeded sample) instead of measuring
      --seed <N>                     With --sample-passes: sample seed; with --plan: must match the plan's (default: 0)
      --judge-cost <USD>             Cost of one judge call: also print the human label cost above which calibrating the judge pays off
      --plan <FILE>                  A `judge sample --out` plan: also print the human-audited pass rate from its labelled picks

Examples
  coding.sh judge calibrate my-exp --tuned-on dev
  coding.sh judge calibrate my-exp --judge-cost 0.002
  coding.sh judge calibrate my-exp --plan plan.json

coding.sh judge sample <EXPERIMENT> [options]

  Plan which cases to label next: a seeded random sample stratified by judge verdict and confidence

Arguments
  experiment    Experiment id or name whose runs hold the grader's verdicts

Options
      --budget <N>        Labels to plan (required)
  -g, --grader <ID>       Grader id (default: the only llm grader)
      --criterion <NAME>  Label criterion (decides which existing labels count)
      --split <SPLIT>     Split to sample from (default: test)
      --seed <N>          Sample seed (default: 0)
      --min-stratum <N>   Smallest confidence bin before it merges into a neighbour (default: 25)
  -o, --out <FILE>        Write the plan for `judge calibrate --plan`

Examples
  coding.sh judge sample my-exp --budget 60 --out plan.json
  coding.sh judge sample my-exp --budget 60 --json