Turn a real session into an eval

Keep a captured session as a validated repo task or a dataset case, so later changes are tested against it.

When an agent gets something wrong, or right after a struggle, keep it as a test so the next change to your prompt, harness or model is checked against it. There are two kinds:

• A repo task replays the session's starting repo and checks the result with a command. Use it for coding work.
• A dataset case keeps the session's first prompt as input. Use it for any agent, including your own agent traced over OpenTelemetry or the SDK, graded by a check, a judge or your own script.

Repo task: from a session to a validated task

A repo task needs the repo as it was when the session started. coding.sh run -- snapshots the whole working tree before and after the command, so it is the exact way to record one. Sessions captured by hooks work too when they ran inside a git repository: they start from the commit that was checked out, so uncommitted changes at that point are missing, and eval new says so.

cd coding.sh
coding.sh run -- claude -p 'Fix the failing test in packages/coding.sh/test/cli/output.test.ts'

Draft a task from it. The verifier must pass only when the work is done. Test files the session changed and that match --hidden-test are taken out of the reference diff and copied in only when the verifier runs, so the agent never sees the final tests:

coding.sh eval new latest --name output-test-fix \
  --verifier 'node --no-warnings --test packages/coding.sh/test/cli/output.test.ts' \
  --hidden-test 'packages/coding.sh/test/cli/output.test.ts' \
  --tag cli

The instruction defaults to the session's first prompt (for coding.sh run --, the command's prompt argument, such as -p or --message). Pass --instruction 'text' or --instruction @prompt.md to change it, and --setup 'npm ci' (repeatable) for setup commands.

Check that the task is fair:

coding.sh eval validate output-test-fix --k 5

A task that passes is marked validated; otherwise it stays a draft and the output says why. --keep keeps the work directories so you can look.

Use it:

coding.sh eval list --status validated
coding.sh compare <bundle-a> <bundle-b> --task output-test-fix --k 3   # see Compare two agents
coding.sh eval export output-test-fix --out ./tasks/output-test-fix               # instruction.md, task.json, reference.patch
coding.sh eval export output-test-fix --out ./tasks/output-test-fix --format harbor --force

Dataset case: from any session

Any session with a user prompt works, including hook-captured ones:

coding.sh sessions --since 7d                         # find the id
coding.sh dataset add regressions --session <session-id> --split test

The case's input is the session's first prompt. The agent's last reply is kept in the case's metadata as output, not as the expected answer, so grade these cases with a judge or a command grader rather than exact:

coding.sh exp run --dataset regressions --name base \
  --task '{"kind":"command","command":"node","args":["scripts/agent.mjs"]}' \
  --grader '{"kind":"llm","name":"resolved","rubric":"Pass if the reply resolves the request."}'

An LLM judge needs ANTHROPIC_API_KEY (or "provider":"openai" and OPENAI_API_KEY). Before you trust its verdicts, label some sessions pass or fail in coding.sh ui and run coding.sh judge calibrate base. See Evals.

From inside the agent

With the MCP server registered, an agent can call eval_from_session to draft a task from a session it has captured. It asks you first.