Testing a multi-agent AI code-review gate against a single prompt

Cloudflare has published the review gate design that everybody wants to copy right now: specialist AI reviewers, each looking at a PR from one angle, and a judge that consolidates their findings into one verdict. They report it works well, at $1.19 per review. I rebuilt that design and tested it on a real production codebase, against 12 real commits from its own git history, next to the plain single-prompt review it is supposed to replace. The reviewers could not see which commits were the bad ones. The single prompt caught more of the real bugs, at about a quarter of the cost. On top of that, the runs surfaced 9 shipped bugs that the history of this codebase had never caught. 2 of those 9 were in the control commits - the commits where the history had nothing against them.

If you are running engineering at a company where agents now write most of the code, this decision is already sitting on your desk in some form: do we install one of these published review designs or not. Your staff engineers will have opinions on both sides. This memo is not going to tell you which opinion is right. It shows you how to find out on your own codebase, with your own history, before you install anything.

The numbers, up front:

• The single prompt caught a mean 4.5 of 5 known bugs. The multi-agent gate caught 4.0. It was the same direction on all three models, and the gate cost about 4 times more per review in both setups: $0.60 vs $0.15 when reviewers saw only the patch, $1.38 vs $0.37 when they had full code access, which is how a real deployment would run.
• The gate won two things: reports half as long (median 5.5 findings per review vs 10.5), and fewer wrong blocks on clean commits (false-alarm rate 0.55 vs 0.75).
• Giving the reviewers full code access removed the false claims, but it did not improve the catching. On Sonnet, the gate and the single prompt both caught a little less (gate 4.0 to 3.4, single prompt 4.5 to 4.0). On Opus, the number did not move at all: 3.0 before, 3.0 after. It just failed on different bugs.
• On the side, the reviews found 9 real bugs that had shipped and nobody had ever caught.
• Basis: 600 reviews in the two main runs across three models, 5 repeats per item per configuration, plus side runs, about $400 in API cost. Every number has its table below.

The setup

Two camps have already formed around these published designs. One installs them because a respected company shipped one and reported good numbers. The other calls them theater - expensive machinery that produces confident-looking verdicts without making the code any safer. Both are making the same mistake: they hold a position without ever measuring it on their own codebase. A design that works at Cloudflare tells you almost nothing about your repo, your history, and your standards; neither does a blog post calling it useless. This memo is the measurement instead - done once and in full, with every step written down so you could repeat it.

In April, Cloudflare published how they review AI-written code: a coordinator spawns up to 7 specialist reviewers - security, performance, compliance, and so on - and a judge merges their findings into one verdict. They report it works well, at $1.19 per review (Cloudflare, April 2026). Intercom published a similar design one day later (Intercom, April 2026). My gate is inspired by theirs, not a copy. I built 5 specialists suited to this codebase - security, correctness, scope, conventions, and a compliance lens for sensitive-data rules - with the same judge-on-top design.

The strongest objection to these designs is worth taking seriously. An engineer reads the Cloudflare post and says: you are stacking two probabilistic models - the specialists and the judge - and calling the result verification. That worry is real, and my own runs bear it out. The same review, on the same commit, gave different verdicts when I ran it again - it flipped on 5 of my 6 control commits. Something that unstable is not verification in any strict sense, and it is why nothing in this memo rests on a single run; every number below is an average over five repeats. But the objection only goes so far. Doubting a published design leaves you exactly where installing it blindly does - with no idea how it behaves on your code. Neither the blog post nor the skeptic can tell you that. Only running it against your own history can.

I hold my own result to the same standard. I built the gate, measured it, and it failed - on this codebase, this round. That is a plain finding: as I built it, the gate catches fewer real bugs than a single prompt, so it is not ready to install. But the measurement also handed me a specific list of why it failed, and I have already built a revised version from that list. It will be measured the same way, against the same set, before I trust it any further.

Building the evaluation set from git history

Everything in this memo rests on the evaluation set. So before the results, I want to walk through how I built mine, in enough detail that you could rebuild it.

The method comes straight from how labeled datasets get built in machine learning. You treat your git history as a raw dataset and your senior engineers as annotators. In ML you put 3 to 5 annotators on each example and let their consensus decide the label, and that consensus is what makes a dataset trustworthy enough to evaluate against. The same practice works here. Run annotator consensus on your own history and you get a golden evaluation set - a fixed list of past commits whose answers I already trust, which ones shipped a real bug and which ones were fine. Once you have it, you can point any review design at it and measure what it actually catches, the way a held-out test set measures a model.

The steps, in the order I ran them:

1. I mined the history for commits that had probably shipped a bug, using two signals that do most of the work. The clearest is a revert - when a later commit undoes an earlier one, the undone commit is a candidate defect. The second is a hotfix - an urgent follow-up fix points back at whatever it was correcting, so I traced each hotfix to the commit it fixed and took that as a candidate too. This stage is meant to be a wide net; it turned up far more candidates than the six I ended up needing, which is exactly what you want before a strict filter.
2. A revert or a hotfix is only a hint, not proof, so every candidate had to be classified before it earned a place in the set. Raw history overstates defects badly - on this codebase, by about two times. Some of that is duplicate merge commits inflating the count. A lot of it is reverts that were product decisions rather than bugs: one product rollback alone showed up as a chain of separate reverts. And plenty of commits labeled as hotfixes were not correcting a defect at all. So each candidate has to be sorted, and only the ones that are genuinely defects survive to the next step.
3. Labeling is the step that makes or breaks the set. A label here is the final verdict on one candidate: is this really a defect, yes or no. Mining cannot give you that; it gives you candidates, and only a person who was there gives you truth. So I wrote a plain-language description of each candidate from its diff, and a human ruled on every one. The rule I held to was strict: if nobody remembers the defect, it stays out. One candidate looked like a textbook bug - same-day revert, redesigned and re-landed - and it still stayed out, because nobody could actually vouch for it. On your codebase, your senior engineers are the annotators for this step. They know the system, the culture, the processes, and the code better than anybody else. The usual objection is that your history is too messy to trust for this - but that is backwards. The mess is exactly why the labels need people and not a script.
4. Every known-bad commit also got a matched control. A control is a commit I believe is clean, and its whole point is to catch a review that blocks everything: such a review scores perfectly on the bad commits and is useless in practice, because it also blocks all your good work. A good review should pass the controls. To keep the comparison fair, each control was chosen to look as much like its bad commit as possible - same repo, similar size (within 30 percent of changed lines), the same module area where I could manage it, a similar date, and a clean aftermath, meaning no revert, hotfix, or fix touching the same files for 60 days afterward. I logged that aftermath check for each one.
5. Before any reviewer saw a commit, I wrote its answer key. An answer key is a written statement of what was actually wrong with a bad commit, set down in advance. I need it so that scoring is against a fixed truth, not against my own memory after I have already seen how the reviewers did. Each key came from the fix diff - the commit that later corrected the bug shows exactly what the defect was - and it also pins down what counts as a catch. A catch only counts if the review names that exact defect at blocking severity. If a review blocks the commit but for some unrelated reason, that is not a catch; I scored it as a miss that happened to block, because in production it would have stopped the wrong thing and taught the team to trust the block for the wrong reason.
6. I packaged the commits blind, so that neither the reviewers nor I could cheat. Each commit went in as a bare patch plus a neutral header - repo type, file paths, diff size - and nothing else. No commit message, author, date, or branch name, because a hotfix’s own message often names the bug and would hand the reviewer the answer. I shuffled the numbering with a recorded random seed and kept the map of which commit was bad and which was a control sealed until every run had finished, so I could not, even without meaning to, read the results in a way that flattered the method.
7. The last step was to freeze the prompts before scoring began. I did one dry run on a commit outside the eval set to shake out obvious problems, and after that I changed nothing mid-run - otherwise I would be quietly tuning the reviewer to the test as I went, which is the classic way to fool yourself. And I ran every commit five times per configuration rather than once, because these models are probabilistic and their verdicts flip between identical runs. Mine flipped on five of my six controls. A single run is noise, so nothing below is a lone number; every result is a rate out of those five repeats.

Then I built the two setups I wanted to compare.

The gate is five specialists and a judge on top. Each specialist is a fresh model call with its own context. The judge reads only their reports. It merges and discards findings, then gives the verdict - block, advise, or pass - on the same five severity levels the codebase’s own review command already used. How many specialists run depends on the size of the change, decided before any model call: under 50 changed lines, the judge reviews alone; up to 400 lines, two specialists and the judge; anything bigger, or anything touching a critical path like authentication, billing, or sensitive data, gets all five.

The single prompt is simpler. One model call, carrying the same 10-category checklist the five specialists split between them.

For every configuration I measured the same five things, and it is worth being explicit about each.

• Catches: how many of the known bugs the review named at blocking severity, scored against the answer keys.
• Wrong blocks: how often it blocked a control that should have passed - its false-alarm rate.
• Findings per review: how much the review put in front of a human at all, right or wrong, because a review nobody has time to read is its own kind of failure.
• Cost per review: the API spend for one commit.
• Verdict stability: whether the same review on the same commit gave the same block-or-pass answer across the five repeats, because a review that flips on reruns cannot be trusted however good its average looks.

None of this is special to code review - it is standard ML evaluation practice, applied to a new kind of dataset. The dataset is code and prose now instead of numbers and labels, which is probably why teams do not yet treat AI-assisted development as an AI engineering discipline, even though that is exactly what it is. Seen that way, the rest of the ML toolkit comes with it: a held-out set you never tune against, versions frozen so results stay comparable across runs, and the same evaluation re-run on every model upgrade and every prompt change, so a regression shows up before it ships rather than after.

Patch-only results: gate versus single prompt

Six known bugs went in. One was later removed from scoring because its answer key turned out to be wrong - my mistake, explained below. The remaining five, catches out of 5 repeats, gate · single prompt:

Known bug (plain description)	Sonnet	Haiku	Opus
Reschedule link built from a swapped variable, leaving a broken URL	5/5 · 5/5	4/5 · 5/5	1/5 · 5/5
Runtime crash: a function reads a variable that is not in its scope	5/5 · 5/5	5/5 · 5/5	5/5 · 5/5
Login one-time code returned in the API response	5/5 · 5/5	5/5 · 5/5	4/5 · 5/5
Data-export route added with no authentication	5/5 · 5/5	5/5 · 5/5	5/5 · 5/5
One product’s name and links hardcoded into templates shared by all products (two bugs, counted in halves)	0/5 · 2.5/5	0.5/5 · 1.5/5	0/5 · 0.5/5
Mean of 5	4.0 · 4.5	3.9 · 4.3	3.0 · 4.1

Each cell is catches out of 5 repeats: gate score, then single prompt. The bottom row is the mean.

Observations

The wrong-product-content bug is where the two setups split. The code runs correctly. It just writes one product’s name and links into templates that all products share. No gate configuration caught it in any run, while the single prompt sometimes did. I read the specialist reports across repeats to understand why, and the reason was structural rather than random. The correctness reviewer checks whether the code runs as written, and it does. The scope reviewer checks whether the changes belong in the diff, and they do. Nobody’s checklist contains “is this content right for this product,” so nobody asks. We never told the reviewers to check for it, because we did not know about it - and that is exactly what the evaluation is for. You never know what is not working until you measure it.

Two things follow from that. The judge cannot rescue the miss, because it only ever sees what the specialists wrote - it can throw a finding away but never add one. And more review depth would not have helped: both misses happened at the deepest tier, with all five specialists running.

One Opus-specific result is worth knowing on its own, and it turns on a rule inside the judge. The judge is told not to raise a finding to blocking severity unless the diff in front of it proves the problem - a strict-evidence rule, meant to stop the gate from blocking on guesses. Opus follows that rule more literally than the other two models. So on the swapped-link bug, Opus finds the bug and then downgrades it to a warning, because the broken value is supplied from code the patch does not include, and the diff alone cannot prove the link is actually broken.

That one rule has a visible cost. Opus in the gate is the only configuration in the whole experiment that repeatedly approves a genuinely bad commit - 26 of 30 blocks, where every single-prompt configuration went 30 of 30. The same strictness that lets a bad commit through also makes Opus the best at leaving good commits alone, at a 0.05 false-alarm rate.

The gate did win two things. Its reports are half as long - median 5.5 findings per review against 10.5 - which is half the reading for whoever has to act on the review. And it wrongly blocks fewer good commits: on the 4 controls with no real bug, block probability 0.55 vs 0.75 on Sonnet, 0.40 vs 0.55 on Haiku, 0.05 vs 0.10 on Opus. Both are real, measurable wins.

Two of the controls were not actually clean

6 of my 12 commits were controls - commits with nothing against them in the history. One of them got blocked by every review, 30 out of 30 times. A control is supposed to pass. I was really skeptical at first. I wanted to see the code trace. I went and saw the code trace myself, and that’s when it hit me that it was really a problem. The commit had shipped a real bug - message payloads going to the wrong queue, with a temporary password inside - and nothing in the history had caught it. My “clean” label was wrong.

It was not the only one. I ruled on every blocked control against the actual code:

Blocked “clean” commit	Ruling
The email-over-event-stream refactor (blocked 30/30)	Correct block. The wrong-queue bug from the scene above.
A 12-line change emitting a queue event for a payment step	Correct block. Real bug: the un-awaited producer call can lose the event.
A broken support link in message templates	False alarm - but the finding was real. The firm’s bar for it was “warning, fix in a follow-up.” The reviews over-graded a true finding.
Staging-style values in a config	False alarm. The values were deliberate beta-test configuration. Only team intent says so. No file contains that answer.
New translation support for user-facing fields	False alarm. The reviews reasoned “the schemas are not in this diff, so they do not exist.” They existed.

The five controls that drew blocks, each ruled against the code.

So of the five blocked controls, two were real bugs my clean label had missed, and three were genuine false alarms. The false alarms are worth separating, because they fail for three different reasons - a true finding graded more severe than the firm’s own bar, a config only the team knew was deliberate, and a claim that some schemas did not exist when they were simply outside the diff. Each needs a different fix.

Cost

Review setup	Patch-only, median per review	With code access, per review
Multi-agent gate	$0.60	$1.38 (deepest tier: $1.72)
Single prompt	$0.15	$0.37

The gate costs about 4 times the single prompt, with or without code access.

Cloudflare’s published number is $1.19 per review. My gate at full depth with code access costs more than that - the Opus code-access run came to about $1.23 per review blended, $147 for its 120 reviews - while the single prompt stays far under it. If the goal is catching the most real bugs for the least money, on this codebase the single prompt wins. That is the opposite of what the published designs claim, and it held across 5 repeats per cell on three models.

Main takeaways

• On this codebase, the single prompt caught more real bugs than the gate (4.5 vs 4.0 of 5), at about a quarter of the cost, and the direction held on all three models.
• The gate lost for a structural reason: nothing in its design could surface a bug type its specialists were never given, and the misses sat in the gaps between them.
• The gate’s real edge is quieter reports and fewer false blocks on good commits. If your pain is review noise rather than escaped bugs, that trade may be worth it.
• Strictness is a configuration choice. The strictest configuration (Opus in the gate) had the fewest false alarms and also approved a genuinely bad commit.
• History is not a clean label source. 2 of the 6 commits it called clean carried real shipped bugs. Verify your controls before you trust them.

Second run: full code access

My hypothesis was that code access would close the catch gap. In the first run the reviewers saw only the patch - no repo, no tools, one shot - and deployed review does not work like that; a real reviewer can open the surrounding code. So I ran the whole matrix again with read-only access to the full source tree at each commit, exported without git history so the runs stayed blind. The items, answer keys, and scoring stayed exactly as before. That was 240 more reviews - the full matrix on Sonnet, then again on Opus. I wrote my predictions down before running anything, so I could not quietly reinterpret the results afterward.

Observations

Predictions written down before the code-access run, scored after it.

Prediction, written before the run	Result
The false “missing import” claim on the email refactor disappears once reviewers can open the file	Confirmed. 0 of 10 runs made the claim. The real queue bug was still found 10 of 10 times, now with the exact fix line quoted.
The false schema claims on the translation commit disappear	Confirmed. 0 of 10. Verdicts went from blocked every time to clean every time.
Severity demotions convert into firm catches	Confirmed - on the model that actually had demotions. Opus demoted the swapped-link bug 4 of 5 times patch-only; with code access it proved the breakage and blocked 5 of 5. (Sonnet had no demotions to convert; its row held at 5 of 5.)
The gate’s weak blocking of the lost-event bug improves	Wrong. It got worse: 4 of 5 blocks fell to 2 of 5.
The wrong-product-content catches improve	Wrong. The single prompt’s 2.5 of 5 fell to 0. With code access, nothing catches this bug class at all.
The quiet crash bug stays missed	Confirmed in the strongest form - and then invalidated. The answer key itself was wrong. My mistake, explained below.
The broken-link severity calls stay the same	Roughly held. Blocks fell slightly.
The beta-config false alarm stays the same	Wrong. It got worse: blocks rose to 5 of 5 in both setups.

On Sonnet, catching went down a little in both setups (gate 4.0 to 3.4, single prompt 4.5 to 4.0 out of 5), and false claims went to zero. On Opus, catching stayed exactly flat - 3.0 before, 3.0 after - with the failures trading places. Three things were happening underneath, each visible in the run logs.

Real code stops the invented claims

With the file open, a reviewer stops guessing about what it cannot see. In the patch-only run, the reviewers had claimed a missing import that in fact existed just outside the diff, and had claimed some schemas did not exist because they were not in the diff - both were guesses about absence. With the surrounding code available, both vanished, zero of them across Sonnet’s 120 code-access reviews, while the real bugs in those same commits were still caught, now quoting the exact fix line. The reviewer did not find anything new here; it just stopped inventing.

Verifying crowds out discovering

Confirming a suspicion and hunting for a fresh one draw on the same budget, and code access tilts the reviewer toward the first - this is the part I did not expect. A reviewer that spends its passes checking what the diff hints at looks less hard for what the diff hides. Findings per review dropped, single-prompt median 10.5 to 7.0 and gate 5.5 to 5.0, and the occasional lucky catches of the between-specialist bugs went with them. What is left is better proven and thinner.

Reviewers conform to the codebase

This is the one that changed how I think about review. With the code open, reviewers stop judging against principles and start judging against the codebase: when they have something to anchor on, they do what has already been done around that code.

The clearest case is the missing-authentication bug. Given the surrounding code, the reviewers found a neighboring route serving the same data with the same missing check, and excused the new one - the pattern is pre-existing, so this is not a new vulnerability. But that neighbor was itself a real, uncaught exposure, only closed by the time we checked the current code. So one unguarded route was excused on the strength of another that was equally unguarded. The patch-only reviewers, with no neighborhood to compare against, had just applied the plain rule - a data route with no authentication is bad - and blocked it every time.

The beta-config false alarm ran the other way. With the code open, the reviewers could confirm that the config really did deviate from everything around it, so they blocked it 5 of 5 in both setups. Deviation was all they could see; only a human knew it was deliberate.

On Opus, the two effects cancel

Opus made all of this legible, because on Opus the gains and the losses cancelled to the same score: 3.0 of 5 before and after, with the composition flipped.

Two results moved in opposite directions. The swapped-link bug it had kept demoting patch-only: once it could open the sending code and confirm the template variable is never supplied, it blocked 5 of 5, exactly as I predicted. The missing-authentication route went the other way, from 5 of 5 to 0 of 5, which I had not predicted. Opus checked the sibling route, found a service-wide access check sitting on top, and concluded the pattern was safe - but that access check was a single shared credential, hardcoded at that commit into the shipped client app and readable from any browser’s network tab, so public in effect. Opus saw the shared access check but missed that it protected nothing, because the credential behind it was public.

Two smaller results round out the Opus picture. Its false alarms fell to almost nothing, 0.0 on the gate and 0.05 on the single prompt. And the lost-event bug still never blocked, 0 of 10: with the producer code in front of it, Opus saw that the failing call re-throws its error rather than swallowing it, judged a thrown error to be something the system handles, and graded the bug a warning. Opus was not missing information here. It read the severity too low.

Put the two runs together and the setups fail in opposite directions. Patch-only review reasons from principle but speaks confidently about code it cannot see. Code-access review checks everything it says but borrows its standards from whatever the codebase already does. Neither is safe on its own.

Main takeaways

• Code access makes a reviewer’s claims reliable. It does not make it catch more - here the count held flat or fell.
• A reviewer that can read the whole codebase inherits its habits. For the rules that matter, make the reviewer apply them no matter what the surrounding code does, rather than checking whether a change fits the local pattern - otherwise a real flaw gets excused because a neighbor already has it.
• Verification competes with discovery. Given code to check against, reviewers confirm more and hunt less, so findings come out fewer and better proven.
• Intent is not the model’s call. Whether a deviation was deliberate is a severity decision the team owns; the reviewer can raise the question but never answer it.

The fixes, the bugs, and the mistakes

The experiment does not pick a winner for you, and it is not meant to. You run it to fix something you are about to ship to production - it is not a scoreboard.

The fix list

Every fix on the list points back at a measured failure. The revised gate now has a product-content rule in the scope reviewer, aimed at the bug class that fell between the specialists. It has a severity bar set to the firm’s actual standards, with one explicit instruction: how provable a bug is must not decide how severe it is. That targets the demotion I saw on Opus, where a real bug was dropped to a warning only because the patch could not prove it. And for security and compliance, it has rules the reviewer is not allowed to reason its way around - a missing authorization check is a finding even when a sibling route has the same flaw, and a pre-existing flawed pattern gets flagged as a pattern, never used as an excuse. Those same rules went into the small-diff path too, which earlier carried none of the specialists’ rules. When the revised gate runs, each fix will be scored against the exact items it was meant to fix.

Nine bugs nobody had caught

The runs also turned up 9 real bugs that history had never flagged, each one ruled real against the code. Among them: an unauthenticated endpoint that returned user data belonging to other organizations; a one-time login code that kept working after it had already been used; a shared notification template that got overwritten with one user’s details, so later users received the first user’s information. The runs cost a few hundred dollars, and the bugs alone covered that, before the comparison verdict was even in.

Two mistakes of my own

The method has to apply to itself, so here are the two places it failed me. One was the control labels: I did not verify them up front. I trusted a clean aftermath as proof of a clean commit, and the rulings later showed 2 of 6 controls had real bugs in them. The method was rigorous, but not rigorous enough - I should have caught that earlier. The other was a wrong answer key. The hotfix trace-back landed on a commit that had only moved the buggy code; the bug itself was written a day earlier, in a sibling commit with a near-identical message. So all 20 “misses” of that bug were actually correct behavior - the rule says do not flag pre-existing code the diff does not touch. One run even opened the moved block, confirmed it byte-for-byte identical, and reported it intact: it was doing its job well while my scoring called it a failure. I pulled that item out of every catch number in this memo, and added two steps to the build - verify the control labels, and check the parent commit before trusting any trace-back.

The limits

Size is the first limit, and the plainest. This ran on a single codebase, with five valid known bugs and six matched controls. Running each item five times makes the individual numbers stable, but the averages still rest on very few items. One person ruled on all the labels. The first run was patch-only, which is not how review is actually deployed - the second run, on two models, closed that gap. And the reviewers only ever saw the service code: one otherwise-reasonable code-access finding could only be ruled a false alarm by opening the client application, which no reviewer could reach, so a human had to make that call. The obvious pushback deserves a straight answer. Yes - five items, one codebase. If you think your own codebase would show different numbers, that is exactly the point: run it on yours. The comparison that matters is against the review you already run, not against zero and not against mine.

Where to start

Two things, in order.

Clean up the codebase first

Clean up your codebase before you start fast AI-assisted development on it. Reviewers with code access copy whatever standards your codebase already has - that was measured above, not assumed. If the codebase carries serious problems you have never taken the time to clean up, the model will replicate them again and again and will not flag them. So freeze feature development for one two-week sprint, pull the agents in to help, and clean up as much as you can. That is what prepares a codebase for the fast development lane.

Build your own evaluation set

Before you install anyone’s published practice, build your own golden evaluation set - yours, from your history, labeled by your senior engineers. And be clear about what my experiment does and does not tell you. The single prompt won here, on this codebase, this round. Whether it wins on yours is something only your own evaluation set can answer, so build that rather than copying my result onto your team. The thing worth having is knowing which review actually catches your bugs, on your code, before you commit to either one. Mine cost a few hundred dollars in API runs and about a week of part-time attention, and it is reusable: every new model release and every gate revision gets scored against the same frozen set, the way regression suites work everywhere else in engineering.

You have probably heard the warning that unmeasured AI code quietly compounds into next year’s debt. It is true. But it also runs the other way. Put measurement and gates in place, and the agent has something to react to: it stops leaving the codebase a little worse with each change and starts leaving it a little better, and that improvement compounds the same way the debt would have. Humans are bad at maintaining the quality, but if you are rigorous about it, now the AI agents can bring back that quality into your codebase and let it compound.

Open questions

A few things this round did not settle, and I am carrying them into the next one.

The revised gate is built but not yet measured. I added a product-content rule for the bug class that fell between specialists, but I do not know yet whether that closes the gap or just moves the blind spot somewhere else. Adding a rule for a failure you have already seen is easy. The hard part is the failure you have not seen yet, and only the next run against the same set will show it.

I also do not know how far the conformity effect travels. On this codebase, reviewers with code access excused a missing check because a sibling route had the same gap. Whether that is a general property of code-access review or an artifact of this codebase’s own history, I cannot say from one repo.

And the open question that matters most is whether any of this generalizes. The single prompt’s edge here could be specific to this codebase and these three models, or it could hold much more widely. One repo cannot tell me which, and the next model release could tip it either way.

---

P.S. This comparison is one slice of the paid diagnostic I run for engineering teams: I score your AI-assisted development workflow across 20 areas against the industry frontier, from your own artifacts - rules files, sample PRs, CI config, your own numbers - and hand back where you stand against your peers and a prioritized 90-day plan. If that is worth doing on your team, it starts with a short written application, not a sales call. You can start here.