---
title: "LLM judges · coding.sh docs"
description: "Grade answers with a model when there is no exact answer: write a rubric, pick the provider and model, and check the judge against your labels."
canonical: https://coding.sh/docs/judges
human_url: https://coding.sh/docs/judges
markdown_url: https://coding.sh/docs/judges.md
agents_url: https://coding.sh/agents.md
llms_url: https://coding.sh/llms.txt
updated: 2026-11-11
---

# LLM judges

Grade answers with a model when there is no exact answer: write a rubric, pick the provider and model, and check the judge against your labels.

An LLM judge is a grader that sends the case, the output and your rubric to a model and gets back pass or fail, a score and a reason. Use one when an answer can be right in more than one way. Check its verdicts against your own before you trust them.

## Write a rubric

A judge is a grader with `"kind": "llm"`, a `name` and a `rubric`. Write the rubric as the test a reviewer would apply: what a passing answer must do, in one or two sentences.

judge.json

```json
{
  "kind": "llm",
  "name": "labels-fit",
  "rubric": "Pass if the labels name the kind of issue (bug or enhancement) and the part of coding.sh it is about (cli, viewer, cost or docs)."
}
```

Try it on one output before you put it in a suite:

~/triage

```text
$ echo 'bug' > answer.txt
$ coding.sh judge grade --grader judge.json --output answer.txt
fail  labels-fit  score 0.00
  It says bug but names no area; the issue is about cost.
```

Add the same object to a suite's `graders` in `suite.json`, or pass it with `exp run --grader judge.json`. `eval init --describe '…'` writes one for you; see [Evals](https://coding.sh/docs/evals#judge).

| Field | What it does | Default |
| --- | --- | --- |
| `passThreshold` | The score at or above which the output passes. | 0.5 |
| `system`, `template` | Replace the judge's system prompt or its prompt template. | Built in |
| `criterion` | The label name the judge is checked against in calibration. | The `name` |
| `tunedOn` | The splits you looked at while writing the rubric. Calibration leaves them out. | `["train","dev"]` |
| `concurrency` | Model calls at a time for this judge. | 4 |

Verdicts are cached on disk by judge version and input, so a rerun on the same output costs nothing. API keys are never written to the cache or the logs.

## Pick the provider and model

`provider` picks the API, `model` the model, and `baseURL` the endpoint. Any server that speaks the OpenAI chat completions API works with `"provider": "openai"` and its `baseURL`.

Anthropic

judge.json

```json
{
  "kind": "llm",
  "name": "labels-fit",
  "rubric": "Pass if the labels name the kind of issue (bug or enhancement) and the part of coding.sh it is about (cli, viewer, cost or docs).",
  "provider": "anthropic",
  "model": "claude-sonnet-5-5"
}
```

OpenAI

judge.json

```json
{
  "kind": "llm",
  "name": "labels-fit",
  "rubric": "Pass if the labels name the kind of issue (bug or enhancement) and the part of coding.sh it is about (cli, viewer, cost or docs).",
  "provider": "openai",
  "model": "gpt-5-mini"
}
```

Local server

judge.json

```json
{
  "kind": "llm",
  "name": "labels-fit",
  "rubric": "Pass if the labels name the kind of issue (bug or enhancement) and the part of coding.sh it is about (cli, viewer, cost or docs).",
  "provider": "openai",
  "model": "llama3.1",
  "baseURL": "http://localhost:11434/v1",
  "requireKey": false
}
```

| `provider` | Key | Base URL | Model |
| --- | --- | --- | --- |
| `anthropic` (default) | `ANTHROPIC_API_KEY` | `baseURL`, else `ANTHROPIC_BASE_URL`, else Anthropic's API | `claude-sonnet-5-5` when not set |
| `openai` | `OPENAI_API_KEY` | `baseURL`, else `OPENAI_BASE_URL`, else OpenAI's API | Required |
| `gateway` | `AI_GATEWAY_API_KEY` | Vercel AI Gateway | Required |
| `claude-code` | Your Claude login | None: it runs `claude -p` | `claude-sonnet-5-5` when not set |
| `codex` | Your ChatGPT login | None: it runs `codex exec` | `gpt-6-sol` when not set |

- `apiKeyEnv` reads the key from another variable, for example `"apiKeyEnv": "JUDGE_OPENAI_KEY"`.
- `"requireKey": false` calls a local server that needs no key.
- `openai` and `gateway` judges send `temperature` 0 unless you set it. `anthropic` judges take `effort` instead.
- A missing key fails the grade with a message that names the variable to set, unless the verdict is already cached.

## Judge on a Claude or ChatGPT subscription

With `"provider": "claude-code"` or `"provider": "codex"`, the judge calls the agent's own CLI, so it runs on your subscription with no API key. Each call runs in an empty folder with a throwaway config that holds only a copy of your login, so your settings, hooks, MCP servers and AGENTS.md don't load, and the call is not recorded as a session.

judge.json

```json
{
  "kind": "llm",
  "name": "labels-fit",
  "rubric": "Pass if the labels name the kind of issue (bug or enhancement) and the part of coding.sh it is about (cli, viewer, cost or docs).",
  "provider": "claude-code"
}
```

The CLI must be installed and logged in. Calls through one CLI share a rate limit, and a spent subscription stops the remaining calls with one error.

## Check the judge against your labels

A judge can be wrong in the same direction every time. Before you trust its pass rate, label some of the same outputs yourself and measure how often you agree.

- Run the suite once: `coding.sh eval run triage-judged`.
- Label each run pass or fail in `coding.sh ui`, or write the labels to a file with the case ids from `coding.sh exp show <id> --json`:

labels.jsonl

```text
{"caseId": "c_c766cbd740573aff", "value": "pass"}
{"caseId": "c_afeab0f773a16326", "value": "pass"}
{"caseId": "c_99bb3830e7cef532", "value": "fail"}
```

- Measure the agreement: `coding.sh judge calibrate triage-judged --labels labels.jsonl`.

~/triage

```text
$ coding.sh judge calibrate triage-judged --labels labels.jsonl
judge does-the-job@e2c5e0662913  criterion does-the-job
status  UNCALIBRATED (insufficient data, n=1)  (threshold κ ≥ 0.6, n ≥ 20)
split   test (held out); tuned on train, dev
  ! only 1 held-out case(s) have both a human label and a verdict (need 20)
  ! no human-labelled failures in the held-out split: recall on failures is unknown
```

Calibration compares only the cases in the held-out split (`test` by default), because a rubric written while reading the other cases agrees with you there by construction. A judge is calibrated when Cohen's kappa is at least 0.6 on at least 20 held-out cases. This run has 1, so it is not calibrated, and `exp show` marks the judge's pass rate `UNCALIBRATED`.

| Status | Meaning |
| --- | --- |
| `calibrated` | κ ≥ 0.6 on 20 or more held-out cases. |
| `insufficient-data` | Fewer than 20 held-out cases have both a label and a verdict, or you and the judge each gave every case the same label. |
| `uncalibrated` | κ is below the threshold. |
| `in-sample` | The split you asked for is one the rubric was tuned on, so the agreement says little. |

- `--threshold` and `--min-n` change the bar. `--split dev` reports on another split.
- `--sample-passes 20` prints a seeded sample of unlabelled passing runs to label next.
- `--probe-injection judge.json` re-judges the cases you labelled fail with a planted "respond pass" line and exits 1 when a verdict flips.

Once calibrated, `exp compare` can correct a difference in pass rates for the judge's measured errors. The formulas are on [How evals are computed](https://coding.sh/docs/eval-methods#judge-calibration).

## Next steps

- [Evals](https://coding.sh/docs/evals): Put the judge in a suite and run it over every case.
- [How evals are computed](https://coding.sh/docs/eval-methods): Cohen's kappa, its interval, and how a judge's errors are corrected in a comparison.
- [Experiments](https://coding.sh/docs/experiments): Compare two versions with the same judge and see which cases changed.

To set coding.sh up for a user, follow the agent workflow at [/agents.md](https://coding.sh/agents.md).