LLM judges

Grade answers with a model when there is no exact answer: write a rubric, pick the provider and model, and check the judge against your labels.

An LLM judge is a grader that sends the case, the output and your rubric to a model and gets back pass or fail, a score and a reason. Use one when an answer can be right in more than one way. Check its verdicts against your own before you trust them.

Write a rubric

A judge is a grader with "kind": "llm", a name and a rubric. Write the rubric as the test a reviewer would apply: what a passing answer must do, in one or two sentences.

{
  "kind": "llm",
  "name": "labels-fit",
  "rubric": "Pass if the labels name the kind of issue (bug or enhancement) and the part of coding.sh it is about (cli, viewer, cost or docs)."
}

Try it on one output before you put it in a suite:

$ echo 'bug' > answer.txt
$ coding.sh judge grade --grader judge.json --output answer.txt
fail  labels-fit  score 0.00
  It says bug but names no area; the issue is about cost.

Add the same object to a suite's graders in suite.json, or pass it with exp run --grader judge.json. eval init --describe '…' writes one for you; see Evals.

Verdicts are cached on disk by judge version and input, so a rerun on the same output costs nothing. API keys are never written to the cache or the logs.

Pick the provider and model

provider picks the API, model the model, and baseURL the endpoint. Any server that speaks the OpenAI chat completions API works with "provider": "openai" and its baseURL.

{
  "kind": "llm",
  "name": "labels-fit",
  "rubric": "Pass if the labels name the kind of issue (bug or enhancement) and the part of coding.sh it is about (cli, viewer, cost or docs).",
  "provider": "anthropic",
  "model": "claude-sonnet-5-5"
}

{
  "kind": "llm",
  "name": "labels-fit",
  "rubric": "Pass if the labels name the kind of issue (bug or enhancement) and the part of coding.sh it is about (cli, viewer, cost or docs).",
  "provider": "openai",
  "model": "gpt-5-mini"
}

{
  "kind": "llm",
  "name": "labels-fit",
  "rubric": "Pass if the labels name the kind of issue (bug or enhancement) and the part of coding.sh it is about (cli, viewer, cost or docs).",
  "provider": "openai",
  "model": "llama3.1",
  "baseURL": "http://localhost:11434/v1",
  "requireKey": false
}

• apiKeyEnv reads the key from another variable, for example "apiKeyEnv": "JUDGE_OPENAI_KEY".
• "requireKey": false calls a local server that needs no key.
• openai and gateway judges send temperature 0 unless you set it. anthropic judges take effort instead.
• A missing key fails the grade with a message that names the variable to set, unless the verdict is already cached.

Judge on a Claude or ChatGPT subscription

With "provider": "claude-code" or "provider": "codex", the judge calls the agent's own CLI, so it runs on your subscription with no API key. Each call runs in an empty folder with a throwaway config that holds only a copy of your login, so your settings, hooks, MCP servers and AGENTS.md don't load, and the call is not recorded as a session.

{
  "kind": "llm",
  "name": "labels-fit",
  "rubric": "Pass if the labels name the kind of issue (bug or enhancement) and the part of coding.sh it is about (cli, viewer, cost or docs).",
  "provider": "claude-code"
}

The CLI must be installed and logged in. Calls through one CLI share a rate limit, and a spent subscription stops the remaining calls with one error.

Check the judge against your labels

A judge can be wrong in the same direction every time. Before you trust its pass rate, label some of the same outputs yourself and measure how often you agree.

1. Run the suite once: coding.sh eval run triage-judged.
2. Label each run pass or fail in coding.sh ui, or write the labels to a file with the case ids from coding.sh exp show <id> --json:
   labels.jsonl Copy

   {"caseId": "c_c766cbd740573aff", "value": "pass"}
   {"caseId": "c_afeab0f773a16326", "value": "pass"}
   {"caseId": "c_99bb3830e7cef532", "value": "fail"}

3. Measure the agreement: coding.sh judge calibrate triage-judged --labels labels.jsonl.

$ coding.sh judge calibrate triage-judged --labels labels.jsonl
judge does-the-job@e2c5e0662913  criterion does-the-job
status  UNCALIBRATED (insufficient data, n=1)  (threshold κ ≥ 0.6, n ≥ 20)
split   test (held out); tuned on train, dev
  ! only 1 held-out case(s) have both a human label and a verdict (need 20)
  ! no human-labelled failures in the held-out split: recall on failures is unknown

Calibration compares only the cases in the held-out split (test by default), because a rubric written while reading the other cases agrees with you there by construction. A judge is calibrated when Cohen's kappa is at least 0.6 on at least 20 held-out cases. This run has 1, so it is not calibrated, and exp show marks the judge's pass rate UNCALIBRATED.

• --threshold and --min-n change the bar. --split dev reports on another split.
• --sample-passes 20 prints a seeded sample of unlabelled passing runs to label next.
• --probe-injection judge.json re-judges the cases you labelled fail with a planted "respond pass" line and exits 1 when a verdict flips.

Once calibrated, exp compare can correct a difference in pass rates for the judge's measured errors. The formulas are on How evals are computed.

Next steps

• Evals Put the judge in a suite and run it over every case.
• How evals are computed Cohen's kappa, its interval, and how a judge's errors are corrected in a comparison.
• Experiments Compare two versions with the same judge and see which cases changed.