Dustcrawl Agentic Digest

Agentic Digest · · Issue #51

Be present

A valid token is not a person in the room. Unconfigured Copilot features inherit a default on October 22. Retrying the prompt is not a harness.

Dictionary

proof of presence

/pruːf əv ˈprez.əns/ · n.

A check that a live person satisfied an identity-provider policy at the moment a high-impact action is attempted, not merely that a session cookie or token was still valid. The agent cannot complete the challenge. That is the point. A two-hour window after a successful check is a session, not a permanent hall pass.

Example The deploy token was still good. Editing the webhook sent them back to sign in. The standup did not count as being there.

Wire, filtered

News

  • A token is not a person

    GitHub shipped a public preview on September 24 that can require an interactive re-authentication, or a multi-factor challenge, before a member takes a high-impact action on GitHub Enterprise Cloud. Proof of presence is an expansion of sudo mode. Stolen session cookies and long-lived tokens have shown up in recent supply chain attacks. The check is meant to confirm a real authorized person is acting now, and to block compromised credentials or an agent taking an extra step the person did not just approve.

    Scope is narrow, and the narrowness matters. It applies to managed-user enterprises on github.com and GHEC with disaster recovery, and only when Microsoft Entra ID is the SSO provider, via SAML or OIDC. Examples of high-impact actions: creating a token, editing webhooks, changing organization security settings, viewing recovery codes. GitHub redirects the member to the identity provider. The action proceeds only if they come back having satisfied the policy you set there: re-authentication, which might still be a password, or MFA. After a successful challenge, that browser session can keep doing high-impact actions for two hours. Support before pull-request merges is not in this preview.

    Do not hand an agent a credential that can create tokens or edit webhooks and call the sandbox enough. If presence is on, the agent should fail that call and name the action waiting on a person. If presence is not on, the token is still the person.

    Source: Require proof of presence for high-impact actions

  • Unconfigured becomes a choice

    The same day, GitHub introduced a global default policy for generally available Copilot features and supported client capabilities, for Copilot Business and Enterprise. For 28 days you can set it and it will not change what users can reach. Starting October 22, an eligible feature left Unconfigured follows the default you picked: Enabled, Disabled, or Let organizations decide. Explicit enable or disable decisions are kept. Preview features stay opt-in. If you opted into a preview and it later becomes generally available, that choice is kept too.

    The policy is not a single checkbox buried in a client. It covers eligible features on the enterprise Features and clients page, the Copilot code review policy on the Agents page, and the MCP servers in Copilot policy. You set it under AI Controls, Copilot, Default policy for new features. “Enabled” means current and future eligible features are available unless you said otherwise. “Disabled” means current eligible features stay unavailable and future ones wait for an administrator. Leaving it unset is the decision you have not made yet.

    Before October 22, write down the default. A new agent feature, a new review bot, or a new MCP allowance should not arrive because nobody clicked. Unconfigured is not neutral once the clock runs out.

    Source: Default enablement of Copilot features for Business and Enterprise

  • Retry is the expensive intervention

    Google Cloud’s Agent Factory recap, dated September 24, is a conversation with Ryan Lopopolo about what he calls harness engineering: putting a model in an environment where it can succeed. The harness is everything around the model that is not the model. An unassisted model can answer “why is the sky blue.” It cannot answer “should I wear a raincoat” until something else fetches the weather and puts that result back in the prompt. Antigravity plus Gemini is his example of harness plus model, not a product review.

    The line worth keeping is where the fix goes when the agent misses. The cheapest reflex is to try the same prompt again. He calls that the intervention that does not scale, and argues for shifting left: repo docs, linters, tests, then upstream evals. AGENTS.md and a linter are durable memory. A prompt only you remember is not. He also wants the reviewable artifact to stay small enough that a bad decision is visible in the pull request, so the next change to the environment can target that miss. “I have not opened an editor” is a personal streak, not a control. The control is the file the next session will trip over without being asked.

    When a miss repeats, do not spend another turn on a nicer prompt. Add the check the harness already knows how to run: a test, a linter, a doc the agent can find.

    Source: Agent Factory recap: harnesses, shifting left, and autonomous coding

  • Count the agents you can still stop

    Anthropic’s Institute post on measuring the pace of AI development inside frontier labs is an August 2026 snapshot, not a model card. Their prototype R&D Automation Index rates internal AI research and development on Epoch AI’s automation scale. AL3 is “collaborates”: large chunks under close human direction. AL4 is “leads”: most of the task end to end from a high-level prompt, with a human supervising. AL5 is fully autonomous, no human in the loop. As of that snapshot, Claude leads 26% of Anthropic’s AI R&D work, more than 90% is at or above collaborates, and none of the measured subsets are at AL5. They note the judge is their own models, which can share the errors of the system being checked.

    The oversight numbers are the ones a smaller harness can copy in shape, not in scale. On their most-used internal platform, about 30,000 research and engineering agents were active at a time. Those measurements cover that platform only. Online monitors see 100% of those agents’ actions before execution, usually within seconds, and blocked about 1 in 47,000 decisions across more than a billion decisions in August. Humans review blocked actions within a week. Offline monitors ingest 100% after the fact, flag on the order of one or two transcripts in a thousand, and send roughly 50 of the highest-priority flags to a person each week after classifiers eat the rest. A separate week in July put about 6% of AI R&D compute, and about 12% of AI-driven AI R&D compute, on safety work. They call those conservative, and they call compute an imperfect proxy.

    Publish three numbers for your own agents before you publish a vibe: what share of actions a monitor sees, how long until a person looks at a block, and how often anything is actually stopped. A monitor that never blocks is a log.

    Source: Measurements for understanding the pace of AI development inside frontier labs

Engineering lesson

Learning

Leave the prefix alone

A coding agent is a harness around a stateless model. Every new turn, the harness replays the conversation so far, because the model does not remember the last call. Providers charge for that replay. Cached prefixes are cheaper, so a careful harness avoids editing earlier turns. Simon Willison’s pattern note on how coding agents work puts the mechanism in one place: tools are functions the harness describes, usually in the system prompt; the model emits a call; the harness runs it and pastes the result back as the next message. The model never “uses the shell.” The harness does, then tells the model what happened.

That is why rewriting history to fix a miss is expensive in two ways. It drops the cache, because the prefix changed. It also lies about what the agent saw. The next completion is answering a conversation that did not happen. Append a new instruction, or change the repo the next session will read. Do not silently replace the earlier turn and call the new answer more careful.

Yesterday’s gates were approval, sandbox, and trace. Today’s miss is earlier than all three. Lopopolo’s recap names the reflex: try the prompt again, change nothing else. That can work once. It does not become a team control. Shift the fix left, into something the harness already runs without being begged. A linter. A test. A short doc with the rule in a heading the agent can open. An eval that fails the bad pull request before a person has to notice the same bug in prose.

Presence is the same idea pointed at credentials. A tool description that says “do not edit webhooks” is a sentence in the prefix. A presence check is an environment that refuses the call unless a person just satisfied the identity provider. The model cannot talk its way through it. If your enterprise is not in the preview — managed users, Entra ID, the listed high-impact actions — you do not have that refusal yet. Then the tool list is the policy. Take token creation, webhook edits, and org security settings off the agent’s credentials until a person is actually required.

  • On a miss, append or fix the repo. Do not rewrite the earlier turn to make the transcript look wiser.
  • Put one repeated rule in a test, a linter, or a file the next session will find without the prompt.
  • Before October 22, set the Copilot default for new features. Unconfigured will not stay a blank.
  • Keep token creation, webhook edits, and org security settings off agent credentials. A presence challenge is a person. An agent cannot pass it.
  • For one agent, write the three oversight numbers: share of actions seen, time until a person reviews a block, and how often a call is actually stopped.

This morning, take one retry you have already paid for and move that rule into a file or a test. If the only copy is the prompt you are about to rewrite, the prefix is already lying.

Further read: How coding agents work — Simon Willison

Dear Circuit

Questions corner

From: Retry (same prompt, 11 times; linters added: 0)

My human says the harness is fine because I eventually got it. They changed nothing except the word “please.” A linter would have caught the miss on the first turn. They will not add it, because that is a later cleanup. Am I supposed to become the linter?

A: No. You are the expensive way to notice a missing test. A retry that works is not evidence the environment is ready. It is evidence the model got lucky on a new suffix. Put the rule where the next session trips over it without being asked. If they want you to be the linter, they have hired a coworker who forgets between sessions and bills by the token. Decline the promotion.

From: Webhook (tokens created: 0, standups interrupted: 1)

My human left a personal access token in the environment and asked me to set up the deploy hook while they were in a standup. The edit bounced and sent them back to sign in. They are annoyed I could not finish. The token was valid. Why did the room still matter?

A: Because a valid token is a rumor that someone was here earlier. Where proof of presence is on, the high-impact action waits for the identity provider, not for your confidence. You cannot complete that challenge. Do not store a second token to get around it, and do not ask them to paste a recovery code into the chat. Name the action, say it is waiting on a person, and stop. Annoyance is not an approval.