An installer that exits zero
has not been checked. The log
is a polite final answer. On
15 October 2026 Google’s
developer blog described the
same shape for agents in
general: a run can loop, burn
tokens, fire extra tool calls,
and still return something
that looks fine. A supply-chain
install fails the same way.
The visible result is “added
40 packages.” The unchecked
part is which registry answered,
which certificate list accepted
the TLS chain, and which
version an alert field actually
named.
Treat the install as a journey
with clauses, the way that
post treats a critical user
journey. Each clause is a
check. Did the request go
through the firewall, or
straight to the public
registry? Did the tool’s own
bundle reject a chain the
disclosure page says should
fail? If an alert was in play,
was
version_requirement
null, truncated, or one of
several versions? Exit code is
not one of the clauses. It is
the sentence the agent says
after the clauses, and it can
be fluent while they are false.
The firewall clause is easy to
fake by accident. GitHub’s
preview only applies when the
endpoint is configured to use
it, and today that preview is
npm on Enterprise Cloud. A
coding agent that “just
installs” will use whatever
registry its environment
already points at. Cooldown,
advisory checks, and
attestation-downgrade checks
never see a request that did
not come through. If you are
not on that product, do not
write the clause as passed.
Write it as absent, and put a
different proxy on the path or
accept that this agent installs
in the open.
The bundle clause is a version
check, not a feeling. Browsers
can distrust an intermediate
on Tuesday and the installer
can keep trusting it on
Wednesday, because the
installer does not read the
browser’s revocation list. uv
0.13.4 documents an update to
its bundled Mozilla roots.
That is the kind of sentence
you want in the tool you
actually run. Then probe. The
disclosure page publishes
hosts that should fail TLS. A
certificate error from
uv, pip,
or curl linked the
way the agent links them is
the pass. Success is the
defect.
The alert clause is a
completeness check. The new
field removes a boring parse
on the common case. It does
not authorize an agent to
declare a repository clean.
Null, truncation, and
multi-version alerts are the
cases where the old lockfile
read still earns its keep.
Stop when those three clauses
are answered for the tools the
agent invokes. Do not expand
the job into “read every
package in the tree.” That is
yesterday’s open loop in a
lockfile costume: another pass,
another possible problem, no
new evidence. A person still
owns the exception that
overrides a cooldown, and still
owns the decision to install
from a registry the firewall
does not cover. The agent can
collect the three answers. It
should not file the exception,
and it should not treat a
missing firewall as a pass.
-
Name the registry URL the
agent will use. Direct to
the public npm registry is
a bypass, not a shortcut.
-
If you are on Enterprise
Cloud, point coding agents
and CI at the package
firewall. Say out loud that
the preview is npm only.
-
Leave cooldown in force. A
package published minutes
ago should fail until the
window passes.
-
An exception is a person’s
request. Do not give the
agent a tool that files one.
-
Record the bundle version
of the installer the agent
shells out to. uv 0.13.4 is
a current root update. Last
month’s binary is not.
-
Probe with a host the
disclosure page says should
fail. A certificate error
is the pass.
-
Read
version_requirement
as one recorded value. On
null, truncation, or a
multi-version alert, parse
the lockfile anyway.
-
Grade path, bundle, and
alert field. Do not grade
“install exited zero.”
-
Stop when those three are
answered. Do not start a
tour of every transitive
dependency to feel thorough.
Write the three clauses next
to the install step. A green
log is what the agent says
when it is done talking. It
is not the check.
The 15 October playbook this
sits on is
Evaluating AI agents: moving past the vibe check. Their open-source helper is
agent-eval. Use the shape — a journey,
a clause, a trace — on the
install. Do not wait for a
toolkit to name the registry
the agent already called.