Split the finding from the fix
Agents are good at finding holes
and even better at sounding sure
they closed them. Datasette’s
week-long audit is the
counter-move: one human writes
the failing tests, another human
implements, and the models on
the case are not the same model
that will “confirm” the patch.
Cursor’s coordinator is the same
shape at product scale. The
planner does not author the
diff. You still sign it.
The failure mode is a single
loop that reports a bug, writes
a fix, and then runs a check it
also wrote. That loop will
converge. It will not necessarily
converge on the bug. Shared
misunderstanding survives
because the finder and the fixer
share a story. Complexity
ratchets because “one more
guard” is cheaper than proving
the original hole is gone.
Nerd-sniping survives because
the interesting patch is more
fun than the boring regression.
Two sets of eyes do not mean
two chat windows. They mean two
jobs with a hard handoff: a
red test that fails on main,
then a patch from someone who
did not invent the test.
Put the same split in the
harness. A coordinator may
plan, spawn, and collect. It
may not merge. Tool calls get a
server-side
auto: run, deny, or
pause — and the evaluation lands
in the log next to the call.
Humans keep the jobs models
fake well: is this actually a
bug, is the test attacking the
right invariant, and is this
patch allowed to add surface.
Stop condition for reviewers:
if the new tests fail on the
old code and pass on the patch,
and the patch does not grow
the attack surface you were not
asked to touch, you are done.
“Find every possible problem”
is how a one-week audit becomes
a rewrite.
Checklist
-
One person (or model) writes
the failing test. A different
one lands the fix.
-
Run the audit on at least two
models. Do not let the finder
rubber-stamp the patch.
-
Coordinator plans and
delegates. It does not edit
the repo.
-
Record how each tool call was
evaluated: run, deny, or
pause — not only that it
fired.
-
Reviewer stop: tests fail on
main, pass on the patch, no
extra surface. Then ship.
-
Keep “is this a real bug”
as a human job. Taste is not
a unit test, but the hole
should be.
This morning: pick one open
finding. Write the red test
yourself. Hand the patch to
another loop. Merge only if
the evaluation log is boring.