--- title: "Test an AGENTS.md or skill change · coding.sh" description: "Test an AGENTS.md or skill change on your own sessions and get a Ship it, Hold or Don't ship verdict." canonical: https://coding.sh/docs/use-cases/test-agent-instructions human_url: https://coding.sh/docs/use-cases/test-agent-instructions markdown_url: https://coding.sh/docs/use-cases/test-agent-instructions.md agents_url: https://coding.sh/agents.md llms_url: https://coding.sh/llms.txt updated: 2026-11-11 --- # Test an AGENTS.md or skill change Test an AGENTS.md or skill change on your own sessions and get a Ship it, Hold or Don't ship verdict. You changed AGENTS.md, a rule or a skill, and want to know whether it helped. coding.sh answers from the sessions it has already recorded, or from tasks you give it. Some checks are free and wait for new sessions; others call a model now and cost a little. ## Pick a method | Question | Command | Needs | | --- | --- | --- | | What should I change first? | `coding.sh improve` | a week or so of sessions | | Did a committed change help on real work? | `coding.sh improve check` | sessions before and after the commit | | Does my uncommitted edit fix the moments the agent got wrong? | `coding.sh exp ab` | an API key, or a logged-in Claude Code or Codex | | Does a skill change what the agent does on whole tasks? | `coding.sh compare` | tasks, harness snapshots, agent and judge spend | | Does the agent load my skill when it should? | `coding.sh eval trigger` | a `triggers.toml` of prompts | ## Compare sessions before and after a change shell ``` coding.sh improve # issues ranked, each with a line to add coding.sh improve fix 1 # appends that line to AGENTS.md or the agent's own file, uncommitted git commit AGENTS.md -m "AGENTS.md: run the tests before saying done" coding.sh improve check # after a few sessions ``` coding.sh records the git HEAD each session starts at. `improve check` compares the sessions before a change with those after it: the share with a failure flag and, for a new line that names a command, path or identifier, how often the agent did what it says. Verdicts include Helped, Regressed, No clear change and Too early to tell. It makes no model calls, and it exits 1 on Regressed, so it can run in CI or a cron job. ## Replay the moments it went wrong shell ``` coding.sh exp ab --dry-run # the decision points it found, and the cost estimate coding.sh exp ab --max-cost 1 ``` `exp ab` finds moments where the agent said it was done without running anything, ended a turn while tests failed, or retried a failed command unchanged. The model writes the next turn twice, with your committed file and with your edit, and a judge scores both. Nothing runs in your repo. The answer is "Ship it", "Hold" or "Don't ship". `--gate` exits 0 only on "Ship it", and `--watch` reruns on every save. It uses `ANTHROPIC_API_KEY` or `OPENAI_API_KEY`, or with neither, a logged-in Claude Code or Codex. Walkthrough: [Test an AGENTS.md change in five minutes](https://coding.sh/docs/how-to/test-agents-md-change). ## Run whole tasks with and without a skill shell ``` coding.sh harness snapshot --agent claude-code --dir arms/no-skill --name no-skill coding.sh harness snapshot --agent claude-code --dir arms/with-skill --name with-skill coding.sh compare no-skill with-skill --tag skill-eval --gate --k 3 --max-cost 15 ``` `compare` runs the real agent on each task in both setups and grades every run. The pass-rate difference comes with a 95% paired bootstrap interval, and the gate reads "Ship it", "Hold" or "Don't ship". `--without ` removes one file from a setup and reads "Keep it" or "Drop it". ## Check that a skill loads The agent loads a skill based on its `description`. `coding.sh eval trigger ` runs prompts that should and should not load it and reports precision, recall and F1. `--description "…"` tests a new description against the current one on the same prompts. ## What it does not do - Small samples give wide answers. "Hold" and "No clear change" mean not proven, not no effect. - Flags are heuristics. Open the example session before you act on an issue. - `improve check` leaves out sessions with no recorded git HEAD. ## Questions ### Does this work for other agents' instruction files and Cursor rules? Yes. `exp ab` and `improve check` read AGENTS.md, GEMINI.md, Claude Code's instructions file, rules under `.claude/rules/` or `.cursor/rules/`, and a skill's SKILL.md. ### Can I test a draft without editing the repo? `coding.sh exp ab --b drafts/AGENTS.md` tests a draft file against your current one. `--a git:main --b .` compares a branch's files with your working tree. ### How many sessions do I need? The gate needs 8 or more wins with no losses to say "Ship it". `improve check` says Too early to tell with fewer than 5 sessions after a change. ## Next - [Test an AGENTS.md change in five minutes](https://coding.sh/docs/how-to/test-agents-md-change) - [Improve your agent every week](https://github.com/jackulau/coding.sh/blob/main/docs/guides/improve.md) - [Evaluate an Agent Skill](https://coding.sh/docs/how-to/evaluate-a-skill) - [Evals and comparisons](https://github.com/jackulau/coding.sh/blob/main/docs/guides/evals.md#ab-a-harness-edit-on-your-own-sessions) To set coding.sh up for a user, follow the agent workflow at [/agents.md](https://coding.sh/agents.md).