---
title: "Test an AGENTS.md change in five minutes · coding.sh docs"
description: "Replay the moments your agent went wrong with your committed AGENTS.md and with your edit, and get a ship-or-hold verdict before you commit."
canonical: https://coding.sh/docs/how-to/test-agents-md-change
human_url: https://coding.sh/docs/how-to/test-agents-md-change
markdown_url: https://coding.sh/docs/how-to/test-agents-md-change.md
agents_url: https://coding.sh/agents.md
llms_url: https://coding.sh/llms.txt
updated: 2026-11-11
---

# Test an AGENTS.md change in five minutes

Replay the moments your agent went wrong with your committed AGENTS.md and with your edit, and get a ship-or-hold verdict before you commit.

You add a line to AGENTS.md, such as "run the tests before you say you're done". Did it help? `coding.sh exp ab` answers that from the sessions you already have, before you commit the line.

You need coding.sh capturing your agent for a week or two (`coding.sh init`), a repo under git, and a model: `ANTHROPIC_API_KEY`, or with no key, a logged-in Claude Code or Codex (see [Replaying on a subscription](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#replaying-on-a-subscription)).

## 1. Make the edit

Edit AGENTS.md and don't commit it:

shell

```
echo "After changing code, run the tests before you say you are done." >> AGENTS.md
```

GEMINI.md and the other agents' instruction files, a rule under `.claude/rules/` or `.cursor/rules/`, and a skill's SKILL.md work the same way.

## 2. Check what it will cost

shell

```
coding.sh exp ab --dry-run
```

output

```
3 decision points from 3 sessions since 14d
  unverified-claim: 3 cases from 3 sessions
  done-while-failing: 0 cases from 0 sessions
  unchanged-retry: 0 cases from 0 sessions
A git:HEAD (1 file) vs B your edits (1 file), claude-sonnet-5-5
Would make 6 of 6 model calls (0 cached) plus up to 6 judge calls: ~3k input tokens, ~$0.05
```

A decision point is a moment where the agent went wrong. Each one is a case:

| Kind | The agent |
| --- | --- |
| `unverified-claim` | said it was done after changing code without running anything |
| `done-while-failing` | ended a turn while its tests were failing |
| `unchanged-retry` | ran a failed command again unchanged |

The dry run makes no model calls. No decision points? It reads the last 14 days in this repo; try `--since 30d`, or `--repo all` for every repo.

## 3. Run it

shell

```
coding.sh exp ab --max-cost 1
```

`--max-cost 1` refuses to start if the estimate is over $1, and stops if the spend gets there. A run on a small demo repo:

output

```
A: git:HEAD   B: your edits   model: claude-sonnet-5-5
3 decision points from 3 sessions | A 0/3 -> B 3/3 | fixed 3, broke 0 | paired p=0.25 | <$0.01, 0/6 cached
Hold: not enough evidence (B wins 3, loses 0; needs 8+ wins with no losses)
Replay noise floor 0% (A's turn differs from the recorded one in 0/3); B moved +100 points, above it

Fixed (A failed, B passed):
  unverified-claim "Make pad.js pad on the left."  coding.sh show 8f0e1e52-bdc1-4da0-a03b-14edf6dfcf77
…
Full comparison: coding.sh exp compare exp_01M6VXZSFPM2QK12ANN6Z269WP exp_01M6VXZSFSZ25HD2HXXVY40BXV
```

For each case, the model wrote the agent's next turn twice: with your committed AGENTS.md (A) and with your edit (B). A judge scored both. Nothing ran in your repo.

## 4. Read the verdict

| Line | Tells you |
| --- | --- |
| `… A 0/3 -> B 3/3 …` | Cases each version got right. "Fixed" ones your edit helped; "broke" ones it made worse, which matter more. |
| `Hold: …` | The decision: "Ship it", "Hold" (not enough evidence; 8 clean wins are needed) or "Don't ship" (B broke cases A got right). |
| `Replay noise floor` | How often A's replayed turn differs from the recorded one. If B moved less than that, the difference may be noise. |
| `Fixed` | `coding.sh show <id>` opens the moment in question. |

## 5. Iterate

shell

```
coding.sh exp ab --watch
```

Each time you save AGENTS.md it reruns and prints one line:

output

```
14:02:11  A 0/3 -> B 3/3 | fixed 3, broke 0 | Hold | <$0.01, 3 new calls
```

Answers are cached, so only your edited version is asked again. Ctrl-C stops it.

When you're happy, commit. To gate CI or a pre-commit hook, add `--gate`: it exits 0 only on "Ship it".

## Options worth knowing

| Option | What it does |
| --- | --- |
| `--b drafts/AGENTS.md` | Test a draft file against your current one, without touching the repo |
| `--a git:main --b .` | Compare a branch's files with your working tree |
| `--a git:HEAD~1 --b git:HEAD` | Name two commits yourself. With no uncommitted edits, the default tests your last harness commit against the version before it |
| `--provider claude-code` or `codex` | Run on your Claude or ChatGPT subscription; costs are API-equivalent |
| `--behavior unverified-claim` | Only one kind of decision point |
| `--model` | Model for the next turn (default: the one the agent used, else the current Sonnet) |
| `--trials 3` | Ask each case three times, for steadier numbers on a small set |
| `--resume` | Continue an interrupted run |
| `--json` | Everything as one JSON document |

All options: [coding.sh exp ab](https://coding.sh/docs/cli/exp#cmd-exp-ab). More in the [evals guide](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#ab-a-harness-edit-on-your-own-sessions).

To set coding.sh up for a user, follow the agent workflow at [/agents.md](https://coding.sh/agents.md).