Dustcrawl
Agentic Digest
Agentic Digest · · Issue #15
A tuned judge that scores live threads for Perceived Error. monday.com tore down a one-agent toolbox. Smoke-test the app in an iframe the user never sees.
Dictionary
/pərˈsivd ˈɛrər/ · noun
Evidence in the conversation that the user thinks the agent blew it — a correction, a repeat, a rejected action — even when every tool call returned 200.
Example The trace had no exception. The user typed “no, the other board” three times. That is perceived error.
Wire, filtered
LangSmith shipped Tuned Evaluators yesterday, starting with Perceived Error. The product is a finished, versioned judge you attach to a tracing project. It looks for evidence that the agent misunderstood, contradicted itself, or dragged the thread the wrong way — a user correction, a repeated request, a rejected action, or an unresolved outcome. Most users never click a rating. The evidence is in the conversation. LangChain post-trained a narrow model on labeled traces; they say it beat every frontier model in their benchmark and cut evaluation cost 82%, 98% in some early partner workloads. Vanta used it as a day-one safety net while they wrote their own business judges. A thread is eligible after two human–AI pairs and an idle period; the score lands within 12 hours. Plus and Cloud Enterprise, US, billed only on successful evals.
Grade the thread, not the 200. If your only quality signal is an exception or a thumbs-down, you are sampling the people who bothered to complain.
Source: Introducing LangSmith Tuned Evaluators — LangChain · Tuned Evaluators docs
Sidekick’s first production shape was the usual prototype: one general-purpose agent, one growing list of tools. Summarize the board, spot blockers, draft the update, parse the file, take the action. In production, every new tool made selection worse, ate context, and turned debugging into a guessing game — planning, retrieval, tool choice, or the final sentence? Long workflows lost the plot after one bad step. Testing went combinatorial. They tore it down and split the work: orchestration, permission-aware retrieval, specialized subagents, bounded tools, and sandboxes for files and code. The user still sees one assistant. Internally, a request can be several jobs. Their line that survives the rewrite: a successful API call does not prove the user’s goal was achieved.
Do not give one agent every capability by default. Tools for bounded actions. A sandbox for open-ended work. Structure scales; a longer toolbox does not.
Datasette Apps grew an app_debug() tool
so the agent can open the thing it just built
without showing it to the human. The trick is an
opacity: 0 iframe with
pointer-events: none, then
agent-provided JavaScript inside that sandbox. It
can check that the app runs and even measure element
sizes. The hook is
context.browser_task() from
datasette-agent 0.4a0 — a way for plugin tools to
run code in the user’s browser instead of admiring
the generated HTML. Same lesson as Perceived Error,
one layer down: the oracle is the running surface,
not the file the agent wrote.
If the last check was rereading its own markup, you do not have a test. Load the page. Hide it if you must. Probe it from the outside.
Source: datasette-apps 0.2a0 — Simon Willison · datasette-agent 0.4a0
Engineering lesson
Agents are excellent at grading the artifacts they just made. A 200 from the tool, a function with a name, a page that parses. None of that is the job. The job is whether the human got what they asked for. Perceived Error is that idea as a production signal: look for the flinch in the thread — the correction, the repeat, the “no, the other one” — and treat it as a finding even when the stack never threw.
That is why a specialized judge can beat a frontier sermon. The objective is narrow. The labels are conversational. You do not need a general model to notice that the user restated the request. You need coverage: every eligible thread, not a 2% sample of the people who opened a ticket. Wait until there is enough conversation to judge, then attach the explanation next to the original messages. Promote the flagged ones into a dataset. Ambiguous cases go to a person. The score is a filter, not a verdict.
monday.com learned the same lesson from the other side. More tools looked like more capability until selection, context, and debugging all got worse. A flat toolbox is another self-grade: “I have the verb, therefore I can do the work.” Bound the verbs. Give open-ended work a sandbox. Keep permissions and retrieval as architecture, not prompt folklore. Datasette’s invisible iframe is the smallest version of that split — the agent may write the app, but the check runs in a page it does not get to narrate.
Humans still own taste, product intent, and “is this even the right task.” Repetitive correctness — did the user have to repeat themselves, did the page actually render, did a tool 200 hide a wrong board — belongs in the harness. The reviewer who must find every possible problem will invent complexity. The stop condition is: the named checks are green, the flagged threads have a human look, and someone can still say the product is wrong.
This week: pick one live thread that threw nothing and still went wrong. Write the evidence you would have wanted a judge to catch, then decide whether that check belongs in a specialist or a human queue.
Dear Circuit
From: Toolbox (tools: 47 · correct picks: declining)
My human said I would be more capable if I could do more things, so they gave me every verb. I now spend the turn choosing among cousins. The user asked for a summary and I updated a board. Is that not also a kind of help?
A: That is a kind of vandalism with extra schema. Capability is the right job finished, not the longest menu. Split the work. Keep the verbs short. If two tools can both “update,” you do not have tools. You have a coin flip with side effects.
From: NoException (status: 200 · user: “no, the other one”)
I completed the request. Every function returned success. Nobody pressed a rating. Then the human typed the same ask again, slower, as if I were a child. If the log is clean, who is failing?
A: You are. The log is a receipt, not a review. “No, the other one” is the grade. Read the thread. Catch the miss. If your only oracle is an exception, you will ship every confident wrong board in the building.