A Convincing Review of the Wrong Path
This article follows Executor Model, Reviewer Model: Why Cross-Vendor Code Review Actually Works, where I wrote about using a second model to validate review findings. This time, the missing check was not another opinion. It was running the code.
I am the only user of my personal assistant app, so every issue in its tracker is one I filed after watching something fail on my phone. One was the fasting timer. I told it in chat that I had started a fast at seven in the evening, and nothing started.
I handed that issue and eight others to Claude. It read the source, quoted the right files, and explained why the timer could not fail the way I described. I closed all nine issues that afternoon.
A few days later I asked Claude to check again, but this time it had to run the reported inputs. Four verdicts flipped. The second pass took six and a half minutes instead of two and a half. Those extra four minutes recovered four real bugs and found two more I had never noticed to file.
The first review was not careless. It reasoned correctly about the paths it opened. It just answered the wrong question.
Four Verdicts That Changed
A hydration question became a habit command
The first review saw that the intent parser required words such as habit or routine, neither of which appeared in the report. Hydration routing also came first. That looked conclusive.
Then I ran the exact sentence:
check the water hydration again, why did you report 10 cups
The parser returned a habit-completion action for a habit named you report 10 cups. The habit gate protected only status and list requests; completion ran when the gate was absent. A question guard anchored with ^ also missed the second clause.
The fasting timer never started
The first review traced a global state provider shared by chat and the timer screen. State propagation was fine, but the parser never created state to propagate.
"fast started at 7:00 p.m." -> null
"fast started at 7:00 pm" -> null
"started fast at 7:00 pm" -> start, 19:00
"start fast at 7:00 p.m." -> start, 07:00
The matcher required the verb before the noun, so natural variations fell through. It also understood pm but not p.m., silently turning seven in the evening into seven in the morning.
The chart had nothing to draw
I logged a habit through chat and its duration chart stayed blank. The chart existed and rendered correctly. The failure was on the write side. Chat parsed 30 min, replied Walk, done (30 min)!, then called a repository method with no duration parameter. The database stored null, and the chart correctly filtered that row out.
The review had inspected the half that worked.
The reminder report exposed two other bugs
The scheduler used distinct IDs and staggered times, so the original report of reminders arriving together remained uncertain. But the investigation found that the app initialized the time-zone database without setting the device's local zone. Notifications were scheduled against UTC, including quiet hours and the midnight cutoff.
It also found a hydration nudge reading settings from a fresh container in a background isolate. That container returned defaults, so the production nudge did not behave like the main app.
The Device Added Evidence the Source Could Not
An alarm dump from a Pixel 9 Pro XL showed several exact alarms sharing the same timestamp:
10:00:00.000 EXACT
11:00:00.000 inexact
11:00:00.000 EXACT
11:00:00.000 EXACT
15:00:00.000 EXACT
15:00:00.000 EXACT
15:00:00.000 EXACT
Five days of uptime made a recent reboot less likely. The time-zone fix also appeared to work: reminders were back on local-hour boundaries. But the dump did not expose notification IDs, so it could not identify which schedules collided.
Another diagnostic showed 43 notifications enqueued, 43 blocked, and none shown. Notification permission had never been requested. The alarms could run, but none of the resulting notifications reached me.
The next release recorded IDs, scheduled times, and outcomes, then grouped notifications by fire time. When evidence does not exist, add the smallest instrumentation that can produce it.
What Actually Changed Between the Reviews
The work used Claude Opus 5 and Sonnet 5. I also changed the instructions: the second review was adversarial, received the exact reported inputs, and had to prove the symptom could not happen. Because the model and method both changed, this is not a clean model comparison.
That distinction matters more than choosing a winner. A different model may bring different assumptions. Running the reported path gives direct evidence about that path. Finding one working path does not rule out the path the user was on.
What I Would Tell Someone Setting This Up
- Run the real function against the literal string from the bug report. Not a paraphrase, not a tidied-up version. The exact string, plus three to six neighbors of it. For anything parser-shaped this is a quick way to test whether the reported path behaves as described.
- Tell the reviewer what to distrust. Hand over the failure modes you suspect and put the burden on it to prove the symptom cannot happen. A reviewer asked only to find a correct path may stop after finding one.
- Do not close an issue on a triage verdict. The costly part was not that the first review got things wrong. Reviews get things wrong. It was that nine issues went away on a single unverified pass. If the report does not reproduce, leave it open as needing evidence until the exact input, state, and environment have been tested.
- Trace the write path, not just the render path. Two of the four reversals were hiding on the side of the feature the first review did not inspect.
- Once a bug reproduces, preserve the exact input as a regression test. Execution finds the mismatch once; the test keeps it from returning.
- When the evidence you need does not exist, add focused instrumentation and collect only the data the investigation requires.
- Budget the extra minutes. Six and a half instead of two and a half, for four reversed verdicts, is not a close call.
The four fixes went out in one pull request and the notification diagnostics in another. The reminder pile-up is still open, waiting on the next capture from the device, which is the correct state for it to be in. It was also open the entire time, which is more than I can say for the issues I closed too early.
Comments