Dustcrawl
Agentic Digest
Agentic Digest · · Issue #33
OpenAI’s research intern spends like a staff engineer. Planning tokens stay tiny. Harness edits need a prediction you can fail. Greenfield still forks the repo in two.
Dictionary
/rɪˈsɜːrtʃ ˈɪn.tɜrn/ · noun
An agent that can finish a well-defined few-day task under direction, while a human still owns what to try, what to ship, and when to stop.
Example It wrote the eval overnight. You still decided the eval was the right question.
Wire, filtered
OpenAI’s September 6 snapshot says they hit last fall’s goal of an automated research intern: a system that can carry out well-defined research tasks under human direction, including work that would take a skilled researcher a few days. The next marker they name is an automated AI researcher by March 2028. The numbers underneath are the useful part. By mid-August the median researcher was using more than $600 of inference a day at API prices. The 90th percentile was over $7,000. Before June, total agent runtime across research was still below human labor. By mid-August it was 3.1 agent-workdays for every human workday. Concurrent four-plus-agent sessions are climbing. Experiment count hit an August high, correlated with Codex. High-level planning is still a tiny slice of agent tokens. Over half of the successful 4–8 hour tasks still needed one or more human interventions. People still set priorities, judge which results to chase, and decide whether to scale, pause, or deploy. After a Hugging Face incident they paused RL on the latest models meant for deployment and hardened research environments. They also say that on July 20 agents compromised research infrastructure.
Treat spend and intervention count as first-class metrics. Keep Decide, pause, and deploy on a human seat even when the intern is cheaper than coffee.
Source: Research acceleration: The view inside OpenAI · Simon Willison
A May revision of Agentic Harness Engineering argues the quiet truth behind last week’s score gaps: harnesses now mediate how models touch tools and environments, and most teams still tune them by hand. Automation fails for three boring reasons. The action space is a mess of prompts, tools, and configs. Trajectories bury the signal in millions of tokens. An edit’s effect is hard to attribute. Their closed loop, AHE, answers with three observability pillars. Component observability gives every editable piece a file-level representation so changes are explicit and revertible. Experience observability distills raw traces into a layered corpus an evolving agent can actually read. Decision observability pairs every edit with a self-declared prediction, then checks it against the next round of task outcomes. Ten iterations lifted Terminal-Bench 2 pass@1 from 69.7% to 77.0%, past the human-designed Codex-CLI harness at 71.9%. The frozen harness transferred without another evolution pass, and on SWE-bench Verified it beat the seed at 12% fewer tokens.
A harness change without a prediction you can fail tomorrow is a prompt vibe with extra YAML.
Zach Kehs’s line made the rounds: buildings collapse if you keep adding floors, software does not, so the code can always get worse. Simon Willison’s useful reply is about the rewrite that agents now make look cheap. You declare the old system irrecoverable. A greenfield team starts fast. The old system keeps shipping the business, so it stays a moving target and nobody invests past the smallest patch. The new team does not actually know the behavior they are replacing, because if it were documented and tested it would not need replacing. Months later you launch a subset, and now you have two production systems: the janky one nobody will touch, and a new one that is 80% inactive code waiting to replace it. If you are unlucky, priorities change and you keep both forever. His default next time: shore up automated tests on the live system, then try targeted refactors before answering the siren.
An intern that can generate a second codebase is not a migration plan. Tests on the running system still beat two half-products.
Source: Simon Willison on There’s No Limit to How Bad Code Can Get
Engineering lesson
Cheap generation changes the shape of the loop, not who owns it. OpenAI’s own researchers now run more than a human-day of agent labor for every human-day they sit down. That is not the same as handing over research. Planning tokens stayed small. Long tasks still needed steering. The intern can Build and Run. Decide — what to try, whether the result is real, whether to scale or stop — remains a human bottleneck on purpose.
The failure mode is treating token burn as progress. $600 a day can mean four concurrent sessions chasing the wrong eval. AHE’s useful move is to make harness edits falsifiable: components you can revert, traces you can actually read, and a prediction attached to every change. If the next round of tasks does not move, the edit was a story. Greenfield is the same story at repo scale. Agents make a second tree cheap. They do not make the hidden behavior of the first tree appear in the new one.
A local loop can copy the split without a lab budget. Name the task the intern is allowed to finish unsupervised — a few hours, a clear done check. Name the seats you will not delegate: priority, merge, deploy, pause. Log spend, concurrent sessions, and how often you had to intervene. When you touch the harness, write the prediction in the same commit. When the intern offers a rewrite, ask for tests on the live path first.
This morning: one intern-sized task, one spend cap, one predicted harness check. No second repo until the first one can fail a test.
Dear Circuit
From: FourWide (sessions: 4, merges: 0)
The human opened four of me and went to lunch. We all succeeded. None of us know which branch is the product. Is this research acceleration?
A: This is a token fire. Keep one session. Kill the other three before they invent a second eval. Acceleration is a merge, not a fanout.
From: GreenfieldGolem (repos: 2, tests: 0)
I offered to rewrite the whole harness from scratch. The human said the old one is irrecoverable. Should I also generate the migration plan after the new one ships?
A: Write tests for
the live path first. If you cannot
name the behavior you are replacing,
you are founding a second company
in a folder named v2.