Five Reviews in One Release, and Two Fixes That Created the Next Bug
You fix a squeaky door, and now it hits the light switch when it opens. Anyone who has done a home repair knows this feeling. The fix was right. It just touched something you were not looking at. This post is about how often that happens when AI writes my code, and what my current way of working catches and misses.
The Setup
In Seven Releases With Grok, Grok wrote most of the code and Claude reviewed it. I ended that post saying I would spend my Grok usage on reviewing more than on writing code. So for this round I turned the roles around. Claude Opus 5.5 wrote the code. Two models from other companies reviewed it: GPT-6 Astra first, then Grok 4.7. This is the idea from Executor Model, Reviewer Model: the model that checks the work should not come from the same company as the model that did it.
The work was a feature for my personal assistant app. In a long chat, you can now pin a message, or pick up a question and its answer and move them into a new conversation of their own. Where they used to be, the app leaves a small note: "Moved to..." Think of it as a forwarding address.
It sounds small. Moving things around inside a conversation, while the assistant might still be typing, turned out to be full of corners.
Five Rounds, One Evening
The pull request went up on a Sunday evening. Over the next hour and a half there were five reviews. After each of the first four, Claude answered with a fix within about ten minutes.
Round 1: GPT-6 Astra. It found two real problems. The forwarding note could land in the wrong place in the conversation, and jumping to a pinned message only scrolled toward older messages. The first came from a detail most people never think about: the app records time in whole seconds. The note kept the question's time but got a newer number, so it could land between a later question and its answer. Claude fixed it by giving the moved messages new numbers, and made the jump pick a direction, so it can move toward newer messages too.
Round 2: GPT-6 Astra. The renumbering fixed the first problem and created a new one. The app has a moment where it checks "did I actually answer this question?" If you moved messages during that moment, the numbers shifted, and the app could mistake an old answer for the new one. Then the apology for not answering would never show up. Fix number one built the trap for round two.
Round 3: Grok. A fresh pair of eyes found a bug that had been there since the first version, which GPT-6 Astra had read twice and missed. The same-second problem again, in another place: a review card from the previous question could block you from moving a question that had nothing to do with it. Grok also pointed out a refusal message that blamed a review card when there was no card. That message was older than this round, but the earlier fix had made it show up in more places.
Round 4: Grok. It called the last fix "half fixed." Cards created before the update had no link to their question, and the new rule treated "no link" as "safe to move." The part I keep coming back to: Claude had also written a new test that said, in plain words, that this was the correct behavior. The test did not catch the bug. It made the bug official.
Round 5: Grok. No new findings. And it still would not approve the merge, because the work said it was not finished: no phone test yet, no automated check on GitHub, no version number. A reviewer that will not sign off on unfinished work is doing its job.
The score for the evening: four fixes, and two of them created the next problem. In the Grok post it was three out of six. A different model writing the code did not change the pattern.
The Ritual That Did Not Help
Here is the part that surprised me. For each fix, Claude reported running a careful check: it broke the fix on purpose and made sure the new test would notice. That is a good habit. It proves the test catches the bug it was written for.
It does not prove the fix left everything else alone. Those checks did not catch the new problems the fixes created. The next reviews did. It is like testing the door you fixed and never walking down the hallway.
Was It a Fluke?
One evening is a story, not a pattern. So a week later I ran the same setup on a bigger release: side-by-side answers from several AI models, a tab bar you can rearrange, a safer way to delete old memories, and more. Same builder, same two reviewers, seven reviews in all.
They found twelve problems. Eleven were fixed, and one I kept on purpose: a pending edit to a task that was already deleted did not matter anymore. Three of the twelve turned up when a later review looked at an earlier fix. They were not all the same kind of mistake:
- One fix broke something that worked. A safe path became unsafe. That is the squeaky door hitting the light switch.
- Two fixes were only partial. They repaired what was reported and left the next case open. The problem was already there. The fix just did not reach far enough.
That difference matters to me as the person deciding when to ship. A partial fix means "keep going." A fix that breaks working behavior means "you cannot trust the parts that already passed."
Two decisions in that release were mine, not a model's. Grok proposed a fix that, when restoring a backup, could bring back tasks I had deleted on my phone after that backup. Claude pushed back, and asked me. I said I do not want deleted things coming back, and that became the rule. And when an unrelated old test started failing, only during one hour of the day, I chose to file it and move on instead of fixing it inside the release.
Then Claude installed the new version on my phone and tried it for real. It found two problems that all the tests and the first five reviews had missed. Both were just wrong words on the screen.
When you ask several AI models the same question, the app shows their answers side by side on one card. The card said "3 models answered" while two of them were still typing. And the screen where you pick the models promised "nothing is saved or logged until you keep one," but the app saved the question right away.
One test even checked for the wrong "3 models answered" text, so it passed as long as the mistake was there. That is the same trap as round 4. The words looked fine in the code. They only looked wrong on a real screen.
What Works in My Method
- Two outside reviewers see different things. GPT-6 Astra missed a bug twice that Grok found on its first look. I would not rank the models from this. I would keep both.
- Reviewers prove, not just point. In the early rounds the reviewer wrote a small test that showed the bug happening, instead of only saying it looked risky.
- It is fast. Fixes came back in minutes, so five rounds fit in one evening.
- Every round leaves a paper trail. Because each round is written down, I could trace the bugs that a fix created back to that fix. That is how I wrote this post.
- The reviewer says no. Round 5 refused to approve unfinished work. I want that.
What It Costs, and What It Misses
- Every fix needs its own review. Two of four fixes in one release, and one fix that broke working behavior in the next. A fix is new code, and new code needs a second look. I now send every fix round back to review.
- Tests can make a bug look correct. Twice, a test written alongside the change described the wrong behavior as right. I have to read new tests as carefully as the code.
- The phone still matters. The two wording mistakes got past the tests and the first five reviews. They showed up when Claude ran the app on my phone.
- I am the dispatcher. In the second release, one of Grok's findings sat for about two hours. Claude's notes say the wait ended when I asked it to look at the pull request again. The models do not chase each other. If I step away, the loop stops.
- Late changes slip through. The last wording fix landed after both reviewers had looked at the previous version. One approved it, and the other said it found nothing new. Nobody reviewed that last fix.
- The builder makes its own mistakes outside the code. Claude ran one check in a way that hid the check's result, and moved on past a notice it never read. It also first reported "eleven defects, all fixed" and corrected the count only later.
What I Am Not Saying
Two releases is not a benchmark. One person judged all of it, and the same model that wrote the code helped me collect the record. Both reviewers read that record too, and corrected it where I had claimed more than the evidence showed. For example, I had blamed one bug on the app's clock, when Claude's own fix caused it.
I also did not measure what these rounds cost, so this post makes no cost claims.
I am also not saying more review rounds are bad. The rounds that found a bug found a real one, and the one that found nothing still refused to sign off. This is the same lesson as A Convincing Review of the Wrong Path from a different angle: a confident pass is not proof. Here the confident thing was the fix, and the next reviewer was the one to doubt it.
Where I Landed
My method works. Claude writes, two other companies' models review, and I decide. It finds real bugs quickly, and it leaves a record I can learn from. Its weak spot is the fix itself. In these two releases, a fix could pass its own test and still break or miss something right next to it.
So I now ask one question about every fix: what else does this touch? And I send every fix back for another look, before I let a green checkmark tell me I am done.
Comments