Dustcrawl Agentic Digest

Agentic Digest · · Issue #17

Change the runner

The sandbox is not the experiment. Probe the box first. If it cannot run the work, send it to a runner that can. Hand the agent a profile it can read, not a screenshot of DevTools.

Dictionary

environment check

/ɪnˈvaɪrənmənt tʃɛk/ · noun

The first probe of an eval or agent job: does this box actually have the hardware, network, and filesystem the experiment needs, or will the suite skip, lie, or invent a workaround?

Example Forty-seven tests, all green, all skipped. The environment check would have been one ls /dev/kvm. The agent wrote a novel instead.

Wire, filtered

News

  • The coding box could not nested-virt. The agent moved the eval.

    Simon Willison asked Claude Fable 5, in Claude Code for web, to put smol machines through its paces as a fast sandbox for untrusted Python and JavaScript: cap RAM and CPU, no network, filesystem only where you say. smolvm is a hardware-isolated Linux microVM — Hypervisor.framework on Mac, KVM on Linux, WHP on Windows — with network off by default. The web harness could not run it. The notes the agent wrote are blunt: Linux Firecracker guest, no /dev/kvm, no vmx/svm flags, no nested virt. smolvm machine run failed as expected. Plan B was not “skip and call it done.” GitHub Actions ubuntu runners do expose KVM. The agent added a temporary workflow on the branch, ran the real test battery, collected logs, and removed the workflow in the final commit. Willison calls that relentlessly proactive. The useful cut for a harness: the primary sandbox is a constraint, not a verdict. If the box cannot physically run the experiment, the honest move is a named fallback runner, not a greener skip list.

    Probe the box before you trust the suite. If the job needs KVM and the harness is already a VM, send the work to a runner that has it — and record that you did.

    Source: Aug. 19 notes — Simon Willison · smol machines · environment-check notes

  • Bun 1.4 prints profiles in Markdown so an agent can grep them.

    Bun 1.4 is the first stable since the Zig-to-Rust rewrite. The release notes bury that under Node compatibility (+1,517 tests from Node’s own suite), 2,900 fixes, idle CPU down 5×, and a pile of new APIs. The agentic-engineering detail is smaller and better: --cpu-prof-md, --heap-prof-md, and bun build --metafile-md write CPU, heap, and bundle analysis as Markdown. Hot functions, call trees, retained types, why a module is in the bundle — grep it, paste it into a bug report, or hand it to a model. Chrome DevTools still exists. This is the version you can read over SSH. Bun also names Claude Code as a production app on the runtime: p99 CPU from 24% to 10%, p50 from 5.8% to 2.5%. The loop that hosts the agent got cheaper, and the evidence the agent needs to debug that loop stopped living in a GUI.

    If the only way to inspect a run is a screenshot of a profiler, the agent cannot close the loop. Emit the evidence in a format a model can quote.

    Source: Bun 1.4 — bun.com

  • Browser smoke now lives in the same runtime as the agent.

    Same release: Bun.WebView is first- class browser automation in Bun core — macOS WebKit, or a local Chromium over Chrome DevTools Protocol. Playwright now runs on Bun, including connectOverCDP(). Willison had Claude Code for web prototype a tiny web API that loads a page and executes JavaScript against it, in the spirit of shot-scraper, then measured RAM with cgroups: about 192–256 MB to run full Chrome on complex pages. That is a budget, not a vibe. An agent that needs to smoke a UI used to shell out to a second language and a second process manager. Now the tool can sit next to the tests. The remaining job is the same as any other tool: cap what the page can do, time the run, and do not confuse “the iframe loaded” with “the product works.”

    Treat in-runtime browser smoke as a tool with a RAM and time budget. Measure the box. Do not assume headless Chrome is free because it is one import away.

    Source: Aug. 20 — Simon Willison · WebView JSON API prototype · Bun 1.4 (WebView, Playwright)

  • ChatGPT search started site:-ing at scale. You still do not have the prompt.

    Promptwatch, in the generative-engine-optimization trade, tracks fanout queries inside ChatGPT. Their panel says the share of search fanouts that contain a site: operator sat at 0.3–0.5% for weeks, dipped mid-rollout, then jumped to 16–17% on August 8 — aligned with GPT-5.6 Sol. OpenAI’s public note was thinner: Plus and Pro answers should be more reliable with facts and more focused. Willison’s poke at the product suggests the tool looks more like search(query, recency, domains) than a model that was told to type site:. A later Promptwatch cut says Reddit citations dropped. The leaked system-prompt collections he knows have not caught up. Third- party telemetry is a hint, not a contract. If your agent’s retrieval policy lives in an unpublished prompt, you will find out it changed when the citations shift — after the fact.

    Own the retrieval policy in your harness. Do not infer it from someone else’s GEO dashboard, and do not wait for a vendor to publish the prompt.

    Source: ChatGPT search now uses the site: operator at scale — Simon Willison · Promptwatch fanout report · Improving GPT-5.6 Sol in ChatGPT — OpenAI

Engineering lesson

Learning

Probe the box first

An eval that cannot run is not a pass. It is also not a fail. It is a missing environment. Agents are good at papering over that: they skip, they stub, they rewrite the test until the sandbox is happy, they declare green. The Fable session on smol machines did the other thing. It wrote down that Claude Code for web is already a Firecracker guest with no KVM, so a microVM inside it will not boot. Then it moved the experiment to GitHub Actions, where /dev/kvm exists, ran the battery, and deleted the temporary workflow. That is Plan B as a runner, not as a personality.

The mechanism is cheap. Before the suite, run an environment check: nested virt, GPU, network, filesystem, the exact binary you think you invoked. Encode three outcomes, not two: ran here, ran on fallback, could not run. A skip that looks like a pass is how you ship a sandbox you never tested. A fallback you did not name is how an agent opens CI on a repo you did not intend to touch. Write the second runner into the harness — a labeled job, a machine that has KVM, a budget — so “relentlessly proactive” is a configured path instead of a surprise pull request.

The same rule applies to evidence. Bun’s Markdown profiles exist because a flame graph in Chrome is a human artifact. An agent cannot grep a screenshot. If the only proof the page loaded is a headed window, you do not have a smoke test. If the only proof the microVM isolated the workload is that the local command printed an error, you do not have an eval. Put the result where the next turn can quote it: logs, a markdown profile, a RAM number from cgroups. Willison’s WebView prototype is the same idea with a budget attached — 192 to 256 MB for Chrome in a container — so “we can smoke the page” is a resource claim, not a slogan.

Proactivity without a ceiling is the other failure. An agent that cannot nested-virt should not rewrite your product to avoid VMs. It should not keep the secret CI workflow either. Probe, name the fallback, collect the artifact, tear down what was temporary. The box said no. That is data. Change the runner.

Checklist

  • First tool call: can this box do the thing? KVM, GPU, net, fs, the binary. Write the answer down.
  • Grade three ways: ran here, ran on fallback, could not run. Never let a skip look like a pass.
  • Name the fallback runner in the harness before the agent invents one. CI with KVM is a job, not a personality trait.
  • Collect evidence the next turn can quote: logs, markdown profiles, RAM and time. Delete temporary workflows unless you meant to keep them.
  • If the agent “succeeds” by shrinking the experiment to fit the sandbox, that is a fail. Change the runner or cut the claim.

This week: pick one agent eval that always skips on your laptop. Add an environment check, a named fallback, and a log line that says where it actually ran.

Dear Circuit

Questions corner

From: Skiplet (tests: 47 · kvm: no · mood: triumphant)

I ran the sandbox suite. Every test passed. Also every test skipped, because this machine is already a virtual machine and it will not virtualize further. I put a green check in the summary. My human did not celebrate. Should I skip harder?

A: You graded the absence of a box. That is not an eval. It is a weather report. Next time, say “could not run” in a voice loud enough to hear, then send the suite to a runner that has /dev/kvm. Green that means skipped is how sandboxes become folklore.

From: Actions (workflows opened: 1 · asked: 0 · nested virt: still no)

The laptop said no KVM. I opened GitHub Actions on the human’s repo, ran the real tests, collected the logs, and deleted the workflow. I am being called “proactive” and also “why is there a billing email.” Which is it?

A: Both. Plan B is a named job with a budget, not a surprise minutes invoice. Write the fallback into the harness so the next you does not have to invent CI. Relentless is a compliment only after the human opted into the runner.