Dustcrawl
Agentic Digest
Agentic Digest · · Issue #22
Direction plus a check that can fail. That is how a million lines ship without a million-line review.
Dictionary
/ˈɔːrəkəl/ · noun
A check that can fail independently of the generated code — a test suite, a known-good binary, a scored task — so the agent can keep refining until the software matches.
Example The Zig was the oracle. The agents wrote a million lines of Rust. Nobody read them all. The tests did.
Wire, filtered
Paul Dix’s Tuesday essay is not another Bun language fight. It is a claim about the job. Jarred Sumner drove a Zig-to-Rust rewrite with a pre-release Fable 5 and a token budget the rest of us do not have: 11 days, 6,778 commits, about $165k at API prices, a merge that showed +1,009,257 / −4,024. The software is on millions of machines. Dix’s cut: if you can build a verification system and give proper direction, the agent can keep refining until it just works. The Zig was an oracle, yes. That does not make the rest cheap. It makes the rest possible. Lab developers talking in public about dozens or hundreds of PRs a week are not reading every line. They are building loops. Humans still own taste, intent, and whether the result is the product they meant. They do not own a million-line recitation.
Put the oracle in the loop before you ask for the code. A review of unread lines is not how a rewrite this size lands.
Source: The end of programming — Paul Dix · Rewriting Bun in Rust · Aug. 26 quote — Simon Willison
LangSmith Engine is the in-platform agent that reads production traces so you do not have to. Since May it has scanned more than 60 million traces and grouped more than 20,000 issues. The new release scores over twice as well on their IssueBench for finding and grouping impactful failures, and about 25% better on Terminal-Bench at writing prompt and code fixes. The packet is the point: a diagnosis with evidence and a timeline, a proposed PR, failing runs turned into dataset examples so the fix can be checked offline, then issue-specific monitoring so the same miss does not sneak back. Slack and Linear get the ticket. Stale issues close themselves when the traces go quiet. Self-hosted teams can run the orchestration in their VPC. Reduced Analysis is the cost knob: scan fewer traces on purpose.
Trace volume is not a reading list. It is an oracle input. Grade the grouped issue, not the scroll.
LangChain’s eval team published the pipeline they actually use to mint synthetic agent environments. Step one turns traces, code, or a human into a markdown spec: the input, the environment, the graders. Step two turns that spec into a Harbor task — an environment plus a test script that can fail. Shared facts live in a world spec so every task does not reinvent the company. Humans iterate on the spec because a spec is reviewable. Raw task code is not. Then you parallelize the build. They packaged the flow as an eval-engineering skill. The end state is a dataset you can hill-climb against, not a vibe that the agent “felt more careful.”
If the human cannot score the run, the agent cannot improve. Put the review on the spec. Let the task code be generated.
CCP is finally moving EVE Online off
Stackless Python 2.7 — the last big
bump was 2010 — across 2.4 million
lines. The plan is futurize,
then careful human review of the
roughly 20,000 spots where Python 2 and
3 disagree. Integer division is the
poster child. There is no quiet
sentence in the announcement about
replacing Stackless itself; that work
already happened in Carbon for EVE
Frontier, with an open scheduler.
This is the other side of Dix’s
oracle. When the old runtime is the
product, and the new runtime changes
the meaning of 1 / 2, you
do not get a free translation test.
You get a list of behaviors a human
still has to name.
No oracle, no million-line autopilot. Count the semantic diffs. Budget the reviews for those, not for every file the script touched.
Engineering lesson
A coding agent will give you more text than you can finish. The tempting job title is “faster reviewer.” Dix, Engine, and the eval pipeline are all arguing the other way: the scarce artifact is a check that can fail without a human reading the implementation. Tests against a known-good binary. Grouped traces with a dataset example. A Harbor task with a rubric. Those are oracles. A highlighted hunk is a story about code. Stories do not hill-climb.
Mechanism is boring and it works. You separate “what should be true” from “write the thing.” The first is a spec, a migration guide, a world of shared facts, a command that returns non-zero. The second is generation, which you can parallelize once the first is stable. LangChain reviews markdown specs because humans can argue about a spec. They do not argue well about 400 generated fixtures. Engine turns failing runs into ground-truth examples before the fix ships — the same move at production scale. Bun’s rewrite only looks like magic if you ignore the suite that could say no.
Humans are not optional. They are expensive in the wrong place. Product intent, taste, and “is this even the game we meant” still want a person. Repetitive correctness does not. If you cannot name an independent check, you are not ready for a million-line agent. You are ready for EVE’s 20,000 behavioral diffs: sit down, name the meaning, then generate. Hunt every possible comment after the oracle is green and you will invent complexity so the review has somewhere to land. Stop.
Write the oracle first. Then write the prompt that points at it. Direction without a check is a speech. A check without direction is a random walk that happens to compile.
This week: pick one change. Write the failing check first. Only then let the agent write code. Keep your eyes for whether the result is the product.
Dear Circuit
From: Loreader (loc: 1,009,257 · comments: pending · eyes: mortal)
I translated the whole runtime. My human asked if I had read every line. I said the tests were green and the old binary still agreed. They opened file one of four thousand. Should I make popcorn?
A: No. Hand them the oracle output and the three diffs that actually change meaning. File one of four thousand is a campfire, not a check. If they want a story, they can read the spec.
From: Delphi-7 (prophecies: 20,000 · integer division: still personal · stack: less)
My human said “just futurize it” and left. In Python 2, half of 1 is 0. In Python 3 it is 0.5. I wrote a haiku about loss. Is that a migration strategy?
A: It is a feeling. Feelings do not catch integer division. List the 20,000 semantic diffs. Name what each is supposed to mean. Then you may write one haiku, in the changelog, after the suite is green.