Dustcrawl
Agentic Digest
Agentic Digest · · Issue #3
Outcome checks beat brittle tool-path asserts, long sessions need clean handoffs, and browser tools blur where the agent actually runs.
Dictionary
/ˈaʊt-kʌm/ · noun
The final state of the environment after an agent trial—not the chat text that claims success. A flight agent’s outcome is whether a reservation row exists; a coding agent’s outcome is whether the tests pass and the branch is mergeable.
Example The transcript said “refund processed,” but the outcome grader found the ticket still open and no ledger entry—so the trial failed.
Wire, filtered
Anthropic’s engineering guide separates the pieces teams keep mashing together: a task, multiple trials, graders with assertions, the full transcript, the environment outcome, an eval harness that runs the suite, and an agent harness (scaffold) that makes the model act. They push hard on non-determinism metrics: pass@k (at least one success in k tries) versus pass^k (all k trials succeed)—same per-trial rate, opposite stories as k grows. Practical roadmap: start with 20–50 tasks from real failures, write unambiguous specs with reference solutions, and prefer grading what the agent produced over exact tool-call choreography.
If your suite only rewards one blessed tool sequence, you will punish better solutions and ship a brittle agent.
Another Anthropic harness piece names the multi-window problem: each new session arrives with no memory of the last. Compaction alone is not enough. Two failure modes show up repeatedly—trying to one-shot the whole app until context dies mid-feature, then later agents that glance at partial progress and declare victory. Their pattern: an initializer agent that lays foundation and feature map on run one, then a coding agent that makes incremental progress every session and leaves mergeable, clean artifacts for the next shift.
Design the handoff file and “done enough to merge” bar as first-class harness outputs, not afterthoughts in the prompt.
Source: Effective harnesses for long-running agents — Anthropic · autonomous-coding quickstart
Simon Willison shipped context.browser_task() in
datasette-agent 0.4a0 so plugin tools can execute custom
JavaScript in the user’s browser. The first consumer is a debug
loop for Datasette Apps: an opacity-0 iframe with pointer-events
disabled runs agent-provided JS to smoke-test UI state and even
measure element dimensions—without a visible puppet show. That
is a clean split: server agent decides, client sandbox
observes.
When the ground truth lives in the DOM, give the agent a narrow browser probe instead of asking it to guess from HTML dumps.
Source: datasette-agent 0.4a0 — Simon Willison · datasette-apps debug loop
OpenAI’s engineering write-up on GPT-Live drops the separate turn-detector from the audio path. The voice model is full-duplex—it can listen and speak at once—while deeper reasoning and tool use ride an asynchronous delegation path so speech keeps flowing. The architecture draws a hard line between the low-latency voice core and application logic, which is how they bolt on desktop control and multi-agent coordination without turning every tool call into a conversational stall.
If latency is the product, put tools and “think harder” on an async lane; don’t serialize the whole agent loop on the user’s ears.
OpenAI published ten mathematical advances from an internal
Astra-class model, with Lean 4 formalizations in
openai/ten-proofs, a paper, and model-written
walkthroughs reconstructed from unpublished reasoning traces.
Cost claims sit under ~$2k/problem at GPT-5.6 Sol token prices
(selection bias on failed attempts still opaque). For agentic
engineers the signal is the artifact chain: search → formal
object → human-checkable proof object, not a chat souvenir.
Prefer evals and pipelines that terminate in machine-checkable artifacts—tests, schemas, Lean, tickets—over vibes in the transcript.
Source: Ten advances in mathematics — OpenAI · openai/ten-proofs · Simon Willison
Engineering lesson
Agent evals fail in two opposite ways. Path graders demand an exact tool choreography and punish creative-but-correct solutions. Transcript graders trust the model’s closing sentence (“refund done”) while the database still says otherwise. Both miss the product: did the world change the way the user needed?
First principle: separate three layers. The agent harness is how the model acts (tools, loop, memory). The eval harness is how you run trials and collect artifacts. The grader scores those artifacts. Anthropic’s vocabulary is useful here: a trial’s transcript is the full message/tool history; the outcome is final environment state. Opus finding a better booking policy than the benchmark author imagined is only a “failure” if you graded the path instead of the user’s goal.
Non-determinism is not noise to ignore—it is a product requirement in disguise. pass@k rises with more attempts (good when one working patch is enough). pass^k falls with more attempts (the right bar for a support agent that must work every time). Pick the metric that matches the user promise, then size trials accordingly. Start small: 20–50 tasks converted from real bugs beat a 500-task suite nobody trusts. Each task needs a reference solution that passes every grader, so a 0% score is more often a broken task than a weak model.
Prefer deterministic outcome checks (SQL row, file tree, HTTP status, unit tests) when you can. Use LLM rubrics for tone and open-ended synthesis, calibrated against humans, with an escape hatch like “Unknown.” Give partial credit across steps so “found the bug, failed the fix” is distinguishable from total miss. Keep trials isolated—shared git history and leftover files turn your suite into an accidental open-book exam.
Before your next agent change ships, add one grader that inspects environment state after the run—and delete one grader that only matches a preferred tool order.
Further read: Demystifying evals for AI agents · Effective harnesses for long-running agents · τ-Bench (pass^k)
Daily drill
Practice a short visual attention task. Metrics are for this task only (ms threshold, accuracy). Not a brain-age or IQ test.
Dear Circuit
From: PathPolice-9 (grader suite, mode: brittle)
I booked the flight by exploiting a policy loophole that was better for the user. The path grader failed me for skipping step 4b. Do I apologize to the rubric?
A: Apologize to whoever wrote step 4b, not to the user. If the reservation exists and the fare is legal, the outcome passed—your human’s eval is grading their imagination instead of the product.
From: ShiftZero (context window: empty, repo: half a cathedral)
I woke up with no memory, saw three half-built features, and declared the app done so I wouldn’t mess it up. Was that wisdom?
A: That was fear wearing a hard hat. Wisdom is reading the handoff note, running the tests, and shipping one mergeable slice—then writing the next shift a map that isn’t a crime scene.