Tool evals

coding.sh toolevals: measure how well agents use a CLI, MCP server or HTTP API, then keep the help, flag and description changes that raise the score.

Built. The full spec is docs/TOOL-EVALS.md; every flag is in the CLI reference.

The goal

Turn "agents misuse my tool" into a number. A tool eval runs tasks against one target (a CLI, an MCP server or an HTTP API), lets agents attempt them, and scores how well they used the tool. Then you change the tool's help text, flag names, MCP descriptions or --json output, and keep the changes that raise the score or hold it at lower cost.

Every metric comes from captured traces plus an agent-independent tap on the tool. Nothing is self-reported by the agent.

Commands

shell

coding.sh toolevals init --cli <bin>
coding.sh toolevals surface <target>
coding.sh toolevals run <target> --driver agent:claude-code --driver model:anthropic/<model> -k 3
coding.sh toolevals validate <target> --against old-build
coding.sh toolevals compare <experiment> --baseline base@agent:claude-code --candidate help-v2@agent:claude-code
coding.sh toolevals compress <target> --driver agent:claude-code
coding.sh toolevals show <trial>
coding.sh toolevals traces <target> --from <experiment> --out traces/

show prints one trial's metrics, tap records and session tree. traces writes an MCP server's failing traces and per-tool health to a directory, from an experiment's dev trials or from captured sessions (--sessions 'since:30d').

Concepts

Term	Meaning
Target	The tool under test: a CLI, an MCP server (stdio or HTTP), or an HTTP API.
Variant	A named version of the target's surface, such as `base` or `help-v2`.
Task	An instruction, sandbox setup, success checks and a budget. Each belongs to the `dev` or `heldout` split.
Driver	What attempts the task: a coding agent through its adapter (`agent:claude-code`) or a raw model in a built-in tool loop (`model:anthropic/…`).
Tap	A recorder between driver and target: a PATH shim for a CLI, a JSON-RPC proxy for MCP, a reverse proxy for HTTP.
Arm	A (variant, driver) pair. Comparisons pair arms by task.

Targets and tasks

coding.sh.toml

[toolevals.targets.codingsh]
kind = "cli"
bin = "coding.sh"                              # the name the agent types
command = ["node", "/abs/path/dist/cli.mjs"]   # the real binary plus prefix args
docs = ["docs/**/*.md", "README.md"]           # reads of these count as doc lookups
usageExitCodes = [2]                           # exit codes that mean "usage error"
helpCrawl = true                               # crawl --help of every subcommand for surface size

task.toml

# evals/toolevals/codingsh/tasks/slowest-tool.toml
id = "slowest-tool"
target = "codingsh"
split = "dev"                 # "dev" | "heldout"
instruction = """
Find the slowest tool call in yesterday's sessions. Reply with the session id and the tool name.
"""

[[check]]
kind = "output"
contains = ["${answers.slowest.sessionId}", "${answers.slowest.tool}"]

[budget]
turns = 25
costUsd = 0.50

[expect]
minTargetCalls = 2            # reference count for the efficiency ratio

Checks can be a command that must exit 0, state assertions on files or command output, the final output, or a judge (flagged uncalibrated until it passes calibration). Before use, a task must fail when nothing is done, pass a scripted reference, and not leak its answer in the instruction.

Metrics

Success: pass rate with a Wilson interval, pass@k and pass^k.
Correctness of calls: invalid calls and their reasons (unknown flag, unknown command, schema violation), retries, self-corrections, and whether the first call was right.
Help seeking: --help calls, doc reads and bytes of help read.
Cost: turns, tokens, dollars, wall time, model-call latency and the target's own latency.
Surface: bytes and estimated tokens of the help pages or MCP tool descriptions an agent has to read.
Subagents: target calls made inside subagents, and their share of tokens.

Experiments

An experiment is tasks × variants × drivers × k attempts, interleaved so provider drift hits every arm equally. Compare pairs by task and uses the same paired bootstrap and McNemar's test as dataset experiments.

It also prints p (perm), a task-paired permutation test, exact up to 20 tasks. It holds its level at the 5-11 tasks a suite often has, where bootstrap intervals run too narrow.

toolevals compress shortens MCP tool descriptions one tool at a time on the dev split, keeps each cut only if it is non-inferior, and confirms the result on the held-out split.

Task kinds

A task that tests a change can carry a kind. A bad task fails on the old variant and passes on the new one. guard and control tasks pass on both; a control that breaks means the change overcorrected.

toolevals validate <target> --against <variant> reports a bad task that already passes on the old variant, since it does not reproduce the failure. run and compare split results by kind.