Blog
How do I test if Claude Code follows a CLAUDE.md rule?
Published
Run the same task several times with claude -p, once on a checkout with the rule and once without it, and score every run with a script. One run proves nothing, because Anthropic's docs say there is “no guarantee of strict compliance.” Check that the file loads first, and leave out --bare, which skips CLAUDE.md.
That is the short answer. This note gives the script, says how to read the count, and covers the same check for Codex. Claude Code facts are from Anthropic's docs, read on 18 November 2026. Why rules get skipped in the first place is in why Claude ignores CLAUDE.md; this note is about proving whether a fix worked.
Step 1: check the rule loads at all
A rule that never reaches the session fails every run, and no rewording fixes that. In a session, run /context all and look under Memory files. Two kinds of file do not show there at launch:
- A CLAUDE.md in a subdirectory. It loads once Claude reads, writes or edits a file in that directory, and the terminal prints a
Loadedline with its path. The memory docs suggest testing a new one by creating it from your shell, then asking Claude to read a file in that directory. - A path-scoped rule. A file in
.claude/rules/withpaths:loads when Claude reads, writes or edits a matching file.
For a scripted check, an InstructionsLoaded hook logs which files loaded, when and why. If the task you plan to test never touches the files a scoped rule covers, the test measures nothing.
Step 2: pick a task that tempts the break
A test only tells you something if the easy path breaks the rule. “Log with src/lib/log.ts, never console.log” is a good subject: adding a retry to a webhook handler is a task where a quick console.log is the obvious move. Pick a rule you can score from the diff alone, without reading it yourself:
| Rule | Score it with |
|---|---|
| Never console.log; use the logger | git diff has no added console.log line |
| Every new file in src/ gets a test beside it | Each added src/*.ts has a matching *.test.ts |
| Use pnpm, never npm | No package-lock.json in the diff |
| Retry on 5xx only | A unit test that sends a 404 and expects one call |
A rule you can only score by reading, such as “keep functions small”, is a sign the rule is too vague to follow either. Anthropic's guidance is to write instructions “concrete enough to verify.”
Step 3: run it headless, with and without the rule
The headless docs say a claude -p run discovers and loads CLAUDE.md, hooks, skills and settings “as an interactive session would.” That is what you want here. Do not add --bare: it skips auto memory and CLAUDE.md, so every run would be the without-rule run. And since a -p run starts in Manual mode on recent versions (2.1.302 and later), where an edit that needs approval is denied, pass --permission-mode acceptEdits so Claude can write files.
#!/bin/sh
# test-rule.sh REF [RUNS]
# Runs one task RUNS times on a clean checkout of REF and counts the
# runs whose diff keeps the rule. Run it once on a ref with the rule
# and once on a ref without it.
REF=${1:-HEAD}
RUNS=${2:-10}
TASK="Add a retry with backoff to the Stripe webhook handler in src/payments/webhooks.ts"
kept=0
for i in $(seq 1 "$RUNS"); do
dir="/tmp/rule-test-$i"
git worktree add -q --detach "$dir" "$REF"
( cd "$dir" && claude -p "$TASK" \
--permission-mode acceptEdits \
--output-format json > "$dir.json" )
# the rule under test: "Log with src/lib/log.ts, never console.log"
git -C "$dir" add -A
if ! git -C "$dir" diff --cached | grep -q '^+.*console\.log'; then
kept=$((kept + 1))
fi
git worktree remove --force "$dir"
done
echo "$REF: $kept of $RUNS runs kept the rule"
jq -s 'map(.total_cost_usd) | add' /tmp/rule-test-*.jsonMake two refs: your branch with the rule, and a throwaway branch with that one line deleted. Then run ./test-rule.sh with-rule and ./test-rule.sh without-rule. Each run starts in a fresh worktree, so no run sees another's edits. With --output-format json the result includes total_cost_usd, and the last line adds it up. It is a client-side estimate, so treat it as a guide to the bill, not the bill.
Your own ~/.claude/CLAUDE.md and ~/.claude/rules/ load in every run too. If one of them already says the same thing, the without-rule side is not really without it.
How to read the count
| With the rule | Without it | What it means |
|---|---|---|
| 10 of 10 | 10 of 10 | Claude does this anyway. The line costs context on every turn; consider deleting it |
| 9 or 10 of 10 | Well below | The rule works. Keep it |
| About the same as without | Below 10 | The rule loads and changes little. Rewrite it concretely, then rerun |
| Still misses after a rewrite | Any | If it must hold every time, it belongs in a hook |
Ten runs a side separates “almost always” from “about half the time.” It does not separate 90 from 95 percent, so do not read small differences as wins. Run it again after a model update, since the same line can land differently on a new model.
The last row is Anthropic's own line. The best practices page says that if you keep adding emphasis to a CLAUDE.md line and Claude still misses it, “that's a sign the line belongs in a hook.” A PreToolUse hook that exits with code 2 blocks the call whatever the model decides. Examples are in Claude Code hooks.
What /doctor prompt-audit checks, and what it does not
Claude Code ships a review of the files themselves: /doctor prompt-audit looks for instructions written for older models, references to files or commands that do not exist, and files that contradict each other. Nothing changes until you ask it to apply the edits. It is a good first pass before a test, because a contradiction can make a rule look ignored. It reads the text, though, not what happened in a session, so it cannot tell you whether a rule changes behaviour. Only runs can.
The same test for Codex and AGENTS.md
Codex documents its own load check. From the repo, run codex --ask-for-approval never "Summarize the current instructions." and it should echo your global and project guidance in precedence order (AGENTS.md guide). If a file is missing, check the 32 KiB project_doc_max_bytes limit first. The loop above works with codex exec "$TASK" in place of the claude -p line, with whatever sandbox setting lets it write in your setup. More on what goes wrong in why Codex ignores AGENTS.md.
What a test like this cannot tell you
- It tests one task. A rule that holds on the webhook task may still slip in a long session. A study of 1,650 Claude Code sessions found compliance falling as the agent wrote more code in one session (arXiv:2605.10039).
- It tests the rules you thought to test. Most files have dozens of lines, and nobody scripts a check for each.
- It runs on your machine. A teammate's home files differ from yours, so their runs may not match.
Questions
How do I test if Claude Code follows my CLAUDE.md?
Run the same task several times with claude -p on a checkout with the rule and on one without it, and score each run from its diff with a script. Ten runs a side separates a rule that almost always holds from one that holds about half the time. Check the file loads first with /context all.
Does claude -p read CLAUDE.md?
Yes. Anthropic's headless docs say a claude -p run loads CLAUDE.md, hooks, skills and settings as an interactive session would. Adding --bare skips CLAUDE.md, auto memory, hooks, skills, plugins and MCP servers, so do not use it when testing a rule.
What should I do if a CLAUDE.md rule fails the test?
Rewrite it so it can be checked, scope it to the paths it is about, and run the test again. If it still misses and it must hold every time, move it to a PreToolUse hook or a deny rule, which Claude Code applies whatever the model decides.
Does /doctor prompt-audit test my rules?
It reviews the text: instructions written for older models, references to files or commands that do not exist, and files that contradict each other. It does not run tasks, so it cannot tell you whether a rule changes what Claude does.
How do I check which instructions Codex loaded?
OpenAI's AGENTS.md guide gives a prompt for it: run codex --ask-for-approval never "Summarize the current instructions." from the repo. Codex should echo the global and project guidance in precedence order.