coding.sh toolevals

coding.sh toolevals <action>

  Measure how well agents use a CLI, MCP server or API

Commands
  init          Add a target to coding.sh.toml and write starter tasks
  surface       Surface bytes, description bytes and tokens per tool or subcommand
  validate      Check no-op trials fail, references pass and nothing leaks
  run           Run tasks x variants x drivers x k and summarize each arm
  compare       Compare two arms of an experiment (paired by task)
  compress      Shorten MCP tool descriptions without losing success
  show          Show a trial's metrics, tap records and session tree
  traces        Export an MCP server's traces and per-tool health

Notes
  Arms are named <variant>@<driver>.
  Runs above [toolevals] confirmAboveUsd (default $5) need --yes when not on a TTY.

coding.sh toolevals init [options]

  Add a target to coding.sh.toml and write starter tasks

Options
  --cli <BIN>      CLI binary to evaluate
  --mcp <CMD>      MCP server command (stdio)
  --mcp-url <URL>  MCP server URL (Streamable HTTP)
  --http <URL>     HTTP API base URL
  --name <NAME>    Target name (default: from the binary or host)
  --dry-run        Show the exact changes without writing anything

Examples
  coding.sh toolevals init --cli gh
  coding.sh toolevals init --mcp "npx -y @modelcontextprotocol/server-everything" --name everything

coding.sh toolevals surface [target] [options]

  Surface bytes, description bytes and tokens per tool or subcommand

Arguments
  target        Target name (default: the only one)

Options
  --variant <VARIANT>  Variant to measure (default: base)

coding.sh toolevals validate [target] [options]

  Check no-op trials fail, references pass and nothing leaks

Arguments
  target        Target name (default: all; one target with --against)

Options
      --against <VARIANT>  Check kinded tasks on the old variant instead: bad tasks must fail there, guard and control tasks must pass
  -d, --driver <DRIVER>    With --against: run agent trials with this driver instead of the reference commands (needed when the variant changes only what the agent reads)
  -k, --k <N>              With --against --driver: trials per task (default: 1)
  -y, --yes                Answer yes to confirmation prompts. An agent passes it only once the user approves

Examples
  coding.sh toolevals validate gh
  coding.sh toolevals validate gh --against old-build
  coding.sh toolevals validate everything --against old-descriptions --driver model:anthropic/claude-sonnet-5-5 -k 3

coding.sh toolevals run [target] [options]

  Run tasks x variants x drivers x k and summarize each arm

Arguments
  target        Target name (default: the only one)

Options
  -d, --driver <DRIVER>               agent:<id>[:model] or model:<provider>/<model> (repeatable)
      --variant <VARIANT>             Variant (repeatable; default: base)
  -k, --k <N>                         Trials per task and arm (default: 3)
      --samples <MODE>                adaptive: 2 trials each, a 3rd when they disagree; it fails at half or more failing. Reports no CI (default: fixed)
      --stop-on-regression            Stop and exit 1 once a task that passed every trial of --baseline fails
      --baseline <EXPERIMENT>         With --stop-on-regression: the stored experiment to hold the run to
      --split <SPLIT>                 Task split (default: dev)
  -t, --task <ID>                     Only this task (repeatable)
  -c, --concurrency <N>               Parallel trials
      --keep                          Keep trial sandboxes
      --retries <N>                   Retries per trial after a rate limit, 5xx or network error, with backoff; a failed setup step is retried once and a refused login never (failures are never retried) (default: 3)
      --stop-after-infra <N>          Stop once N trials in a row fail on infrastructure after their retries (a refused login, a failed setup, rate limits, 5xx, network): no new trial starts and the run is no quality verdict; 0 never stops (default: 5)
      --shell-model <PROVIDER/MODEL>  Model drivers on a CLI target: a model plays the shell (fixed date); it becomes part of the arm name
  -y, --yes                           Answer yes to confirmation prompts. An agent passes it only once the user approves

Examples
  coding.sh toolevals run gh --driver agent:claude-code -k 3
  coding.sh toolevals run everything --driver model:anthropic/claude-sonnet-5-5 --variant base --variant terse
  coding.sh toolevals run gh --driver agent:claude-code --samples adaptive --stop-on-regression --baseline texp_01J…

coding.sh toolevals compare <EXPERIMENT> [options]

  Compare two arms of an experiment (paired by task)

Arguments
  experiment    Experiment id

Options
  --baseline <ARM>   Baseline arm, <variant>@<driver>
  --candidate <ARM>  Candidate arm, <variant>@<driver>

coding.sh toolevals compress [target] [options]

  Shorten MCP tool descriptions without losing success

Arguments
  target        MCP target name (default: the only one)

Options
  -d, --driver <DRIVER>  Driver to evaluate with
  -k, --k <N>            Trials per task (default: 3)
      --margin <DELTA>   Allowed success drop (CI lower bound) (default: 0.03)
      --max-evals <N>    Maximum candidate evaluations (default: 40)
      --batch            Compress every tool at one level per step
  -y, --yes              Answer yes to confirmation prompts. An agent passes it only once the user approves

coding.sh toolevals show <TRIAL>

  Show a trial's metrics, tap records and session tree

Arguments
  trial         Trial id

coding.sh toolevals traces <TARGET> [options]

  Export an MCP server's traces and per-tool health

Arguments
  target        Target or MCP server name

Options
      --from <EXPERIMENT>  Trials of this tool-eval experiment (dev tasks only)
      --arm <ARM>          Only this arm of --from, <variant>@<driver>
      --sessions <FILTER>  Captured sessions matching this query, e.g. 'since:30d'
      --server <NAME>      Server name in mcp__<server>__<tool> (default: the target name)
  -o, --out <DIR>          Write traces/, index.jsonl, tool-health.json and tools.json here
      --max <N>            Most traces to keep (default: 200)
      --no-tools           Don't connect to the server to list its tools

Examples
  coding.sh toolevals traces acme --sessions 'since:30d' --out traces-acme
  coding.sh toolevals traces acme --from texp_01J… --arm base@agent:claude-code --out traces-acme