Green Tests, Broken App: One Coordinator and Three AI Agents
A smoke detector with a green light is not proof that it works. The light only says the battery is in. To know, you press the button and listen. This post is about the week I stopped trusting the green light on my own app, pressed the button, and changed how the fixes get built.
In my last post, Five Reviews in One Release, I ended with one question to ask about every fix: what else does this touch? I also said I would stop letting a green checkmark tell me I am done. This post is the bigger version of that lesson. It was not one fix that fooled me. It was the whole app.
The Setup
My personal assistant app is the one I wrote about in Building a Personal Assistant: What I Got Wrong. Every change to it comes with tests: small programs that check the app does what it should.
By October 22, 905 issues were closed, 422 of them since August 1. Nearly all were closed because about 4,700 tests passed. That day I said it plainly: basic things did not work, and I had lost trust in the app. The morning summary card listed every event I had ever saved. A search key failed without any warning. Logging a meal went wrong.
So I set one rule. Proof means the app, not the tests. Proof is my own phone with my real data, or the real code checked against the exact words I typed. A passing test is not proof.
The Audit
I did it like a home inspection: room by room, the rooms I use most first. Food, then fasting, habits, tasks, the calendar and the morning summary. No repairs during the inspection: every problem gets written up, and the fixes come later, in their own update. I back up first, and I never press a button that destroys real data. One AI runs the checks, and another vendor's LLM checks its work.
The paper trail was worse than I expected. I went through all 422 issues closed since August by hand. Only 59 had specific proof from the phone. Five said they were tested on the phone when they were not. Four leaned on the same vague sentence.
Five areas are done so far, and each one was followed by an update with the fixes. Food had 13 problems. Chatting with the assistant had 5. Fasting had 6, habits 5, and tasks, reminders and shopping 5.
Two kinds of mistakes kept coming back. The assistant said it had done something it had not done: removed a food, ended a fast, completed a task. And one feature grabbed words meant for another. A shopping list got logged as a 1,740-calorie meal. "I did brunch" checked off my Run habit.
The Turning Point Came From Another App
Earlier in October, in my training app from Seven Releases With Grok, LLMs from two vendors wrote code for the same update for the first time. Grok built a Bluetooth bike sensor emulator and the first version of one feature, and then Claude Opus 5.5 took over. Every change went to two reviewers, Codex running GPT-6 Astra and Grok, and their numbered findings were disjoint: no item on one list was on the other. Codex found timing problems and proved each one with a small failing test. Grok found spots that skipped a rule the other spots follow, and a test that still passed with the fix taken out. That update is the evidence behind the rule this post ends with.
Before the Split: The First Two Fix Updates
Food (October 22). Claude Sonnet built all 13 fixes. Grok reviewed twice, and Codex once, with four problems found and fixed. Then every fix was tried on the phone. The phone found two problems that 4,769 passing tests had missed. One safety check waited for a reply the AI never actually sends, because Claude had guessed what that reply would look like. Another new check replaced a good recipe answer with its own canned correction.
Chat (the morning of October 26). Five fixes, such as a sentence about a price getting saved as an expense. Claude Sonnet built them, Codex reviewed four rounds with Grok alongside, and Claude Opus did the last round. One rule went around three times, and each fix opened the next case right beside it. That is the squeaky door from Five Reviews in One Release, again.
Two updates in a row: one builder, reviews one after another, and fixes that made the next problem. So for the next one I split the building itself.
One Coordinator, Three Agents
Think of a kitchen remodel. The general contractor does not hold the hammer. They hire the plumber, the electrician and the carpenter, and they check every job before the walls close.
Here, one Claude Opus 5.5 session is the contractor. I call it the coordinator. It plans the work and writes short instructions for each builder. When a reviewer reports a problem, it checks that the problem is real before passing it back. Then it puts the pieces together, tests them on my phone, and ships. It writes almost no code itself.
Three builders work at the same time, each on its own piece: Grok 4.7, Claude Sonnet 5.5, and Codex running GPT-6.1 Sol. Each piece is checked by the two other-vendor LLMs, never by the one that built it:
| Builder | Checked by |
|---|---|
| Grok | Claude Sonnet, Codex |
| Claude Sonnet | Codex, Grok |
| Codex | Claude Sonnet, Grok |
The first time, one piece did not follow this, and that is how the rule got written down. Since then, big fixes also go back to the same two reviewers.
Fasting and Habits, in Under Four Hours
The fasting and habits audits produced 11 issues that morning. The fasting audit started at 10:12 AM on October 26, and the update was ready at 2:03 PM. Each piece went through two rounds of review, and each reviewer caught something the others missed:
- Grok's piece. Claude and Codex both flagged the same blocker. A new check against false claims also blocked true ones, like "the start time has been updated."
- Claude's piece. Codex read the code of the tool the app uses for notifications. It found that a daily alarm ignores its start date, so the "start tomorrow" fix did nothing. Claude's own review and the tests could not see it.
- Codex's piece. Grok found that the start of a fast landed on the wrong day for fasts longer than a day. Claude's review missed it. Then the coordinator caught a new mistake in Codex's fix: a time corrected across midnight moved back a day.
All 4,972 tests passed. Then it ran on the emulator, went out to testers, and passed a check on my Pixel.
Tasks, Reminders and Shopping, the Same Afternoon
Same setup, five issues. The first round of review found:
- Undo on a finished task could delete a different task that Google added at the same moment. Codex said "do not merge."
- "Next day" was added as exactly 24 hours. That is wrong in the week the clocks change on November 1. Both reviewers of Grok's piece found it.
- "Log 2 eggs, then add eggs to my shopping list" logged nothing. Both reviewers of Codex's piece found it.
The fixes were big, so they went back to the same reviewers. Some of what they found next, the fixes had caused. At 8 AM, "remind me to call Bob at 10 tonight" now set the alarm for 10 that morning. And the shopping rule went from too broad to too narrow. After the last round, the coordinator's own test found one more: a curly apostrophe, the kind a phone keyboard types, slipped past the food-logging check. It fixed that itself. The update shipped at 4:35 PM, and it passed a check on my Pixel too.
In every piece, a round of fixes created or reopened a problem that only the next review by another vendor's LLM caught.
The Rule, Written Down
Looking back at the fasting and habits update, one piece had broken the pattern. Claude built the habits piece, and Claude was one of its reviewers, because I had picked the pairs each time. So I made it a rule: every piece is checked by both other-vendor LLMs, never by the one that wrote it.
The coordinator wrote the rule into AGENTS.md, the instruction file every AI reads when it works on the project. I wrote about that file in Claude Keeps Ignoring My AGENTS.md. Now no session has to be told.
Grok then reviewed the shipped habits piece, as the reviewer it should have had. It found that habit reminders stop after 7 days if I do not open the app.
Where the Hours Went
The models were not the slow part. The tools around them were:
- Claude's tool refused to start Codex or Grok with their safety turned off. Each had to run with limits instead.
- In its safe mode, Codex could not save its own work. The coordinator saved it for Codex, with Codex's name on it.
- Grok, working alone, stops at the first thing it is not allowed to do. Its instructions had to say "one step at a time."
What Works in This Method
- Other-vendor reviewers paid for themselves. The worst problem in each piece came from another vendor's LLM. Twice, only one reviewer found it.
- The coordinator checks first. Every finding was checked against the code before a builder saw it.
- It is fast. The fasting and habits update went from the start of the audit to ready in under four hours. The next one shipped the same afternoon.
What It Costs, and What It Misses
- Fixes still create bugs. Three vendors did not change the pattern from my last post. A fix is new work, and it needs a second look.
- I am still in the loop. I had to update Codex, Grok had to be signed in by hand, and I set the review rule only after one piece broke the pattern.
I did not measure what this cost, so this post makes no cost claims. Four updates and one person judging them is not a benchmark, and I would not rank the models from it.
Where I Landed
The green light is still useful. It is just not proof. Proof is the app on my phone doing the right thing with my real words.
And the way I build changed. One coordinator that checks instead of writes. Three builders from three vendors. Every piece checked by the two other-vendor LLMs, and every big fix checked again. It does not stop fixes from creating the next bug. It makes it more likely that someone else catches it before I do.
Comments