Stories, user journeys, and AI code review

I wondered whether giving an AI coding agent an unrelated story might help it notice something it would otherwise miss. The more useful signal in a small experiment came from asking it to follow concrete user journeys through the code.

The original idea

John Cleese’s lecture on creativity was the starting point. He distinguishes an open mode for exploring possibilities from a closed mode for carrying out decisions. I wondered whether a similar separation could help an agent avoid committing too quickly to its first interpretation of a problem.

My initial hypothesis was that an unrelated story might suggest associations or relationships that an ordinary technical review would overlook. It was a question about model behaviour inspired by a discussion of human creativity. The connection needed testing.

I also had a more concrete observation. Looking at a diagram of a workflow in Megamind, my assistant project, I had spotted a problem without reading the underlying implementation. Following a request from the user’s perspective made the issue visible. Could an agent do that deliberately?

Five approaches, fifteen reviews

I compared five procedures, with three independent reviews each: ordinary review, structured exploration, a story alone, a story with explicit analogy-based exploration, and exploration through user journeys.

All fifteen used the same frozen repository snapshot, reviewer model and high reasoning setting. The ordinary review already required source inspection, tracing interactions and concrete failure scenarios. The journey condition added a procedure for discovering questions: construct requests and follow-ups, then trace their consequences in the implementation.

A separate model evaluator, given reports under neutral filenames, checked their claims against the source. Its assessment distinguished nine source-established defects from ten supported risks and one speculative opportunity. The chart counts only full defect diagnoses.

Journey reviews averaged 2.67 fully diagnosed defects, compared with 2.00 for ordinary review. They did not find more in every block: the last two blocks were tied. Across all three journey reports, however, they fully diagnosed six distinct defects, compared with three across the ordinary reviews.

That six is a combined result from three reviews, not what one reviewer reliably finds. The per-run data and method notes make the counting explicit.

What a journey revealed

One finding concerned a background initiative task. It could read a profile with initiative enabled, begin model work, and later save its old copy. Meanwhile, the user could successfully disable initiative. When the background task finished, its stale write could restore the enabled setting.

The disable request itself was only part of the story. The user’s decision also had to survive work already in progress.

Other journey reviews found a saved preference omitted when an interrupted task resumed, and a future-effective replacement that could hide the arrangement still applicable today. These were findings about transitions: what happens after a seemingly successful operation, or when two otherwise reasonable paths overlap.

The agent was enacting requests while remaining a technical reviewer. We did not test replacing its identity with a child, a beginner or another personality.

What the stories did—and did not—show

The story-plus-analogy procedure contributed complementary defects, including pending work stranded by configuration changes and cleanup code supplying diagnostic values that its receiving component rejected. Both came from one report, so this is useful evidence of complementarity, not reliable specialisation.

The story-alone condition showed no clear standalone advantage. And the assisted condition bundled the story with explicit instructions to examine analogies. We still cannot tell how much the story itself contributed.

My practical takeaway is to give an agent a concrete way to change how it investigates, then require source verification of whatever that reveals.

Can one reviewer do this in stages?

The next experiment is planned, not completed. I want to compare three workflows after the same initial methodical review:

A continuation could reuse its understanding and avoid repeated investigation. It could also inherit the first pass’s blind spots. Independent reviewers can preserve their findings in a shared inventory without sharing the assumptions that guided discovery.

I would measure additional verified findings, retention of earlier findings, serious misses and actual resource use. A single agent is not automatically cheaper: retained context has a cost too. Previously established defects give us a reference set to validate and withhold from reviewers, while genuinely new findings still count separately.

What I am still unsure about

This was one repository, one reviewer model, three repetitions per method and one evaluator. Every report was capped at five findings, so discovery and selection were mixed together. That cap should go in the next comparison. Equal reasoning settings also do not establish equal computation.

The evaluator inspected source; it did not reproduce every defect by running tests. Earlier reviews had found issues that none of these fifteen reported. Our known-defect inventory is incomplete, and finding all its entries would not establish that the code is sound.

For now, the result is enough to try journey-driven discovery alongside methodical review. Whether one agent can retain the useful coverage of independent reviews at lower cost is the question I want to answer next.

Sources and experiment notes

• John Cleese on Creativity in Management, published by Video Arts.
• Per-run defect counts and recorded resource use, recalculated from the completed round-two assessment. Only full defect diagnoses count.
• Method notes, definitions and limitations. The experiment used Pi 1.4.0 and gpt-6.1-sol through openai-codex; model identifiers and settings are recorded, not a guarantee of immutable weights.