How to turn PR review comments into agent rules

Why are review comments where conventions get decided?

Most team conventions are never proposed. They are corrected. A reviewer leaves a comment, the author pushes a fix, the thread is resolved, the PR merges. The convention now exists, and its only written record is a collapsed thread on a closed PR.

A typical one, on a PR that touches the payments webhook handler:

reviewer: Don't retry on 4xx from the provider. A 4xx means we sent
          something wrong; retrying just repeats it. Retry 5xx and timeouts,
          max 3, and reuse the idempotency key on every attempt.
author:   Done in 3f2a91c.

The human author will remember this. The next coding agent will not. It will write the loop that retries on every non-2xx, and the same reviewer will leave the same comment again.

How do I turn review comments into agent rules by hand?

Start with the review comments on recent merged PRs. The GitHub CLI gets you there without a script:

# recent merged PRs
gh pr list --state merged --limit 50 --json number,title

# inline review comments on one of them
gh api repos/OWNER/REPO/pulls/412/comments --jq '.[].body'

Then filter hard. A comment becomes a rule only if:

• It names a specific code practice, not taste or a question.
• The author accepted it and changed the code.
• It has come up more than once, or it came out of an incident.

Write each survivor as one rule, the reason, and the source. The reason matters: a rule without its why is the first one somebody deletes.

## Webhooks
- Retry provider webhooks on 5xx and timeouts only, never on 4xx.
  Max 3 attempts, exponential backoff, same idempotency key each time.
  Why: a 4xx means the request is wrong; retrying repeats it. (PR #412)

Then choose where it goes:

• AGENTS.md at the repo root, if it applies everywhere. Codex, Cursor and Copilot's coding agent read it, and Claude Code reads it too when the project has no CLAUDE.md of its own (details in does Claude Code read AGENTS.md).
• A path-scoped rule, if it applies to one part of the code. In Claude Code, a file in .claude/rules/ with a paths: list loads when Claude reads a matching file. Cursor keeps its equivalent in .cursor/rules/.
• A nested AGENTS.md in the directory the rule is about, since agents read the nearest one.

This works. It also has a maintenance bill: every rule is added by hand, the file only grows, and nobody removes the rule when the webhook handler is rewritten.

How do I keep review rules from piling up?

A rules file fed by review only grows. A study of 7,310 rules from AI IDE projects found rule changes were mostly expansions and enrichments, and the developers surveyed said they changed rules mainly to correct AI errors, usually by adding a new negative constraint rather than editing an existing one. Three habits keep the list honest:

• Link every rule to its PR. When someone asks whether a rule still applies, the thread answers in one click, and a rule nobody can trace is the first candidate to cut.
• Delete with the code. When a PR removes the webhook handler, the same PR removes the rules about it. Make it a review checklist item, since no tool will do it for a flat file.
• Prefer one rule to three. If reviewers keep leaving variants of the same comment, the rule is badly worded. Rewrite it instead of adding a sibling.

Stale rules cost more than missing ones, because the agent follows them with confidence. That failure has its own note: context rot and stale rules.

Can AI code review tools learn rules from PR comments?

Yes, and the three best known do it differently. From each vendor's own docs, checked 2026-09-23:

Sources: Qodo Rule Miner, Qodo Get Rules, Greptile memory and learning, Greptile custom context over MCP, CodeRabbit Learnings.

If your team already runs one of these, turn the learning on. Qodo's acceptance filter (the author had to actually apply the fix) is the same filter you would use by hand, and it is the right one.

What do review bots still miss?

The fair version of the gap is narrower than “the bot keeps what it learns”, since two of the three now pass rules to coding agents. What remains:

• One source. A review bot learns from review. The webhook decision was argued in #eng-payments a week before the PR existed. The review comment is the second time the team said it, and the first time it cost a round trip.
• Two rulebooks. The rules the bot enforces and the rules in AGENTS.md are different lists, maintained by different people. They drift.
• Pull, not push. A skill or an MCP tool gives the agent rules when the agent asks. Vercel's evals found a skill went unused in 56% of cases. A rule the agent never fetched did not shape the code.
• No read on use. None of the docs above describe counting how often a rule reached an agent and how often the agent's answer relied on it. Without that, the rulebook only grows.

How Harbor turns review threads into rules

Harbor reads merged PRs on GitHub and GitLab, including their review threads. The accepted correction on PR #412 becomes a candidate fact, with the PR as its source. It lands in a review queue where you set the gate. Under Hybrid, the default, a routine convention like this one is added automatically; anything that contradicts an existing fact is held for a person. The Slack thread where the team first argued it can be a source too, so the decision has one home.

The approved rule is then served on turn one to Claude Code, Cursor and Codex when the task touches the webhook handler, through the hooks harbor init installs, and to any other agent over MCP. Each rule is counted: how often it was served, how often an answer cited it. When the handler it describes is deleted from the repo, the rule can be retired. Setup is in the quickstart, and what Harbor reads from GitHub is in the sources docs.

Related: why an agent keeps making the same mistake, Slack decisions for AI coding agents, and ADRs for AI coding agents.