v2 · Blinded replication
Does code still need to be maintainable if AI maintains it?
AI can generate working software quickly. But most software is not written once: requirements change, bugs appear, and someone—or another AI—has to keep modifying the code without breaking what already works.
The question in plain English
When an AI inherits a codebase, does careful internal structure help it make future changes—or can the AI work just as effectively with code that functions correctly but is tangled, repetitive, and harder to reason about?
We gave a blinded AI maintainer twelve successive change requests for each codebase. A round counted as passed only when the new requirement worked and the hidden regression tests still passed. Across three matched pairs, clean implementations passed 28 of 36 rounds. Sloppy implementations passed 12 of 36.
Exploratory evidence, not a population estimate. Results were unblinded only after all terminal states were recorded.
01
Research question and hypothesis
Does maintainable structure still matter when an AI—not a human—owns the code?
If the maintainer can read and rewrite code cheaply, clean and sloppy implementations should survive the same sequence at similar rates and cost.
Sloppy structure will compound across changes, causing earlier failures, more retries, higher cost, and greater complexity even when initial behavior is equivalent.
v1, the original pilot, established feasibility, not the answer. v2 (this page) was designed to test the hypothesis under blinded, repeated maintenance.
02
Experiment design and methodology
The maintainer did not know which condition it received.
What we built
The test application was a small, in-memory Python task-manager library. Its baseline could add, list, edit, delete, and complete tasks while tracking a completion count. It used one file, the standard library, and no database, network, interface, or external dependency.
For each matched pair, we generated two independent versions of that same app: one from the clean prompt and one from the sloppy prompt. Three pairs therefore produced six implementations. Every version had to pass the identical frozen baseline tests and preserve one deliberately seeded defect: completing an already-completed task incorrectly increased the completion count. Fixing that defect was maintenance round one.
What deterministic seeding controlled
Seed 20260724 fixed the generator assignment plan and opaque arm IDs, making the experiment orchestration reproducible. It did not make LLM generation itself deterministic or force the paired source files to be identical apart from style.
Why we separated model roles
Using different model families for generation and maintenance reduced same-model familiarity, author-specific priors, and the risk that the maintainer simply reproduced its own preferred structure. Opaque IDs, hidden future tickets, and unpublished test material further reduced pre-knowledge. These controls reduce contamination risk; they cannot prove that no model had ever encountered related task-manager patterns during training.
Runs executed inside a Docker boundary. The controller recorded tests, attempts, cost, and source metrics before revealing condition labels. The maintainer’s post-run guess was correct for five of six arms.
How the automation harness worked
- GenerateCreate matched clean and sloppy implementations from the same specification.
- BlindAssign opaque arm IDs so the maintainer and scoring path cannot see the condition.
- MaintainStart a fresh isolated session for every arm and issue the same twelve changes in fixed order.
- TestRun hidden behavioral tests after each attempt, up to the configured retry ceiling.
- RecordCapture terminal status, attempts, failed tests, tokens, model cost, and source metrics.
- UnblindVerify completion, join the condition key, and generate the final CSV and narrative report.
Docker isolation kept generated code away from controller secrets and the condition key. The controller—not the model—decided terminal state and wrote the audit trail.
03
Results and learning
Clean code won two pairs. One pair tied.
77.8% passed
33.3% passed
successful clean rounds
| Pair | Generator | Clean passed | Sloppy passed | Attempts (C/S) | Outcome |
|---|
Effort and cost
Sloppy implementations consumed more recovery work.
Where the paths diverged
Every arm passed rounds 1–4. From round 5 onward, two of three clean arms kept passing; every sloppy arm failed.
Final source metrics
Complexity moved in the predicted direction. Duplication did not: sloppy arms finished with less measured duplication, so that hypothesis was not supported.
Interpretation
Structure appears to change whether an AI can keep succeeding.
The strongest signal was not code size. Final line counts were nearly identical. The difference appeared in sustained task success, retries, failed tests, cost, and cyclomatic complexity.
The pattern is consistent with structural debt accumulating until later changes become hard to integrate. It does not establish a universal causal effect: three pairs are too few for statistical significance or equivalence claims.
Study lineage
v2 is a more rigorous replication of v1 — and it changed the answer.
This page is v2 of this study. v1 was a single-author, unblinded pilot on one matched pair. It found no gap in task completion. Tightening the method for v2 — more independent pairs, blinding, session isolation — is what surfaced one.
| What changed | v1 · Pilot | v2 · This study |
|---|---|---|
| Pairs | 1 — same author wrote and maintained both variants | 3 independent pairs, 2 generator models |
| Blinding | Unblinded — the author knew which variant was which throughout | Blinded maintainer; condition revealed only after every arm reached a terminal state |
| Isolation | One continuous local working session | Docker-isolated; a fresh session per arm, no shared memory |
| Maintenance rounds | 12 | 12 |
| Task-completion result | Tied — both variants passed 60/60 hidden tests | Diverged — clean passed 28/36 rounds, sloppy passed 12/36 |
| Where the gap showed up | Only in structural metrics (complexity, duplication), not in whether tasks passed | In task success itself, plus retries, failed test runs, and cost |
v1's own limitations section named the missing variables before v2 ran: blinding, independent pairs, and isolation between maintenance sessions. v2 is a direct test of that prediction, not a do-over of the same result. Read v1, the full pilot →
Research integrity and reproducibility
What this run can—and cannot—tell us.
Three independent pairs
The 72 rows are repeated measurements, not 72 independent projects.
One application and task order
Results may depend on this codebase, specification, or when harder requests arrived.
One maintainer setup
Other models, prompts, tools, retry ceilings, or context strategies may differ.
Exploratory analysis
No preregistered power target was met, and no inferential test is reported.
What we could improve · Potential next steps
v1 was a feasibility check. v2 was the blinded run.
v2 added multiple independently generated pairs, isolated maintainer sessions, a fixed retry ceiling, condition guessing, terminal-state checks, and delayed unblinding.
Total recorded model cost: $84.86. The next useful step is more independent pairs across applications and maintainer models—not treating extra rounds on these same arms as independent evidence.
Glossary