Much of the recent AI security argument has centered on the prompt. The vulnerabilities Check Point brought to Black Hat this week live one layer down, in the plumbing, and most belong to bug classes security teams have known for decades.

The agent doesn’t always need a dangerous tool. Sometimes reading the wrong thing is enough

Subscribe now

According to The Register’s account of the briefing, Check Point researchers Yarden Porat and Shahar Tal discussed eleven disclosed vulnerabilities spanning LangChain, LangGraph, CrewAI, AutoGen, Microsoft Agent Framework and Google’s Agent Development Kit at Black Hat USA on Wednesday, August 5, 2026. The publication described several of the flaws as critical.

Tal put it plainly: most of what they found did not belong to some new frontier class of AI vulnerability. The list included insecure deserialization, server-side request forgery, path traversal and use-after-free.

Old bugs. New execution layer … and that distinction matters

Share

Where the code actually runs

I’ve spent two pieces this summer writing about authority: who an agent acts as and who answers for what it does. Both pieces assume the framework holding that authority does its job.

This is what happens when it doesn’t.

Take LangGraph’s checkpointer. A checkpointer stores workflow state so an agent can resume from an earlier point, inspect its history or recover after an interruption. Check Point’s research focused on applications that exposed LangGraph’s get_state_history() function with an attacker-controlled filter while using a vulnerable persistence component. It was not every LangGraph deployment and it did not affect LangChain’s PostgreSQL-based managed deployment.

The first flaw in the chain, CVE-2025-67644, was SQL injection in the SQLite checkpointer.

A user-controlled filter key was interpolated into a SQLite JSON path expression rather than handled safely. With a crafted key, an attacker could alter the query and append a UNION SELECT. The important detail is what that union produced: not a new checkpoint written into the database, but a fabricated row inserted into the query’s returned result set.

That fake result could carry attacker-controlled serialized data.

When the application processed what the database returned, the checkpointer treated the row like saved state and deserialized it. That reached the second vulnerability, CVE-2026-28277, an unsafe msgpack deserialization flaw capable of invoking imported Python callables with attacker-supplied arguments. Chained together under the conditions Check Point described, the two flaws produced remote code execution on the application server.

The prerequisites matter.

The affected application had to expose the vulnerable history-filtering path to attacker-controlled input and use the vulnerable SQLite checkpointer. The deserialization flaw was not, by itself, a magic remote shell against every LangGraph installation. Its advisory treated it as a post-exploitation issue because an attacker first needed a way to place or introduce malicious serialized state. In Check Point’s demonstrated chain, the SQL injection supplied that path.

The SQLite injection was fixed in langgraph-checkpoint-sqlite 3.0.1. The msgpack issue was fixed in langgraph 1.0.10. Check Point also reported a related injection flaw in the Redis checkpointer, CVE-2026-27022, fixed in langgraph-checkpoint-redis 1.0.2.

No model jailbreak was required for the demonstrated SQLite-to-deserialization chain.

That doesn’t make the model irrelevant to the larger research. Some of the broader findings involved prompt-controlled material crossing into trusted framework behavior. But in this LangGraph case, the exploitable path lived in ordinary application input, query construction and deserialization.

The runtime was enough.

The bugs without public identifiers

Two other disclosures are harder to track because, according to Check Point and contemporaneous reporting, they had not received CVE identifiers as of August 6, 2026.

Google’s Agent Development Kit included a development-oriented component capable of writing files. Check Point reported an attack path in which generated Python could execute when imported, exposing credentials available to the process. The Register reported that Google initially disputed the finding, later made changes and paid a bounty of $3,133.70. Because I could not locate a primary Google advisory confirming every part of that timeline, those details should remain attributed to the reporting rather than stated as Google’s public account.

Microsoft Agent Framework had a different failure mode. Check Point found that one user’s prompt-injected content could place a malicious payload into checkpoint data. When another user rewound a session and the framework deserialized that state, the payload could execute on the server. Microsoft told The Register that it hardened the framework against the demonstrated path and paid a $10,000 bounty. The company did not issue a CVE because the framework was not generally available when the issue was reported.

That explanation describes a release state.

It does not make the technical failure less real. But it does affect how confidently we can describe exposure. A pre-release vulnerability is not evidence of broad production deployment, and it should not be written as though it were.

The narrower point is enough: without a public advisory or identifier, defenders have less structured information to search for later.

Not one team grinding one axe

Microsoft’s own security advisories document related execution risks inside Semantic Kernel.

CVE-2026-26030 affected the Python SDK’s InMemoryVectorStore filtering logic. Microsoft’s advisory identifies it as a critical remote-code-execution vulnerability and fixes it in semantic-kernel 1.39.4.

CVE-2026-25592 affected SessionsPythonPlugin in Semantic Kernel’s .NET SDK. A file-transfer function exposed to agent function calling could write to an attacker-selected local path unless the application added its own validation. Microsoft fixed the affected .NET component in version 1.71.0 and recommended an allowlist around file-transfer paths as a workaround.

Those advisories are more careful than the story we sometimes tell around them. One flaw created a remote-code-execution path inside vector-store filtering. The other enabled arbitrary file writes through an agent-callable plugin. Depending on the host and target path, that write could become part of a larger execution chain.

That is the real pattern.

Model-controlled or attacker-influenced data reaches an ordinary dangerous sink: eval, deserialization, a filesystem path, a shell or a query builder.

OWASP already has language for one part of this. LLM05 in the 2025 Top 10, Improper Output Handling, covers systems that pass model output downstream without sufficient validation. OWASP lists SSRF, privilege escalation and remote code execution among the possible backend impacts and specifically warns about model output reaching functions such as exec or eval.

That category does not explain every framework bug in Check Point’s research. A SQL injection in a user-controlled filter is still a SQL injection. Unsafe deserialization remains unsafe deserialization.

AI did not invent these defects.

It placed them underneath software that can read repositories, handle credentials, retain state and act with someone else’s authority.

The unscored layer

Prompt injection still deserves the attention it gets.

A January 2026 Systematization of Knowledge paper by Narek Maloyan and Dmitry Namiot synthesized 78 studies published between 2021 and 2026, catalogued 42 attack techniques and reported attack-success rates above 85 percent against state-of-the-art defenses when adaptive strategies were used. The paper is an arXiv preprint, not proof that every agent or defense fails at that rate, but it is strong evidence that prompt injection remains unresolved across the systems surveyed.

What the Check Point disclosures add is a second question.

Even when we measure whether the model can be manipulated, are we measuring what happens after manipulated content reaches the framework?

That is the unscored layer: the code the benchmark assumes will safely receive whatever the model emits.

Sometimes it doesn’t.

The disclosure system has a similar blind spot. In April, The Next Web reported research by Aonan Guan against Anthropic’s Claude Code Security Review, Google’s Gemini CLI Action and GitHub’s Copilot Agent. Malicious instructions placed in GitHub-controlled content could be consumed as trusted context and used to expose secrets through the agents’ own workflow output. Anthropic paid $100 and GitHub paid $500. TNW reported that Google also paid a bounty, but described the amount as undisclosed. None of the three findings had received a CVE or public security advisory at the time of that report.

That last point needs precision.

A CVE is not the only way security tooling discovers risk. Scanners can use vendor advisories, GitHub Security Advisories, package metadata, custom signatures and other feeds. But CVEs remain one of the main identifiers used to correlate a vulnerability across advisories, dependency tools, asset inventories and remediation systems.

No identifier does not make a flaw invisible.

It does make correlation harder.

So the frameworks get patched. Ordinary bugs, ordinary fixes.

What still feels missing is the layer meant to notice that the runtime itself has changed underneath the benchmark. The control that asks whether attacker-influenced data reaches a deserializer, query builder, path operation or dynamic evaluator before any model score matters.

Check Point describes LangGraph as receiving more than 50 million downloads per month. Package downloads are not the same as unique users or deployed systems, but the number still gives the blast radius some shape.

The argument is not that prompt injection was a distraction.

It is that prompt injection was never the whole system.

We kept staring at the sentence the model read.

The bug was waiting in what read the model.

Subscribe now

Leave a comment

Resources

• From SQLi to RCE: Exploiting LangGraph’s Checkpointer - Check Point Research’s primary technical disclosure. Covers the SQLite SQL injection, unsafe msgpack deserialization, Redis injection, exploit prerequisites, affected configurations, disclosure timeline and remediation information.

• CVE-2025-67644: LangGraph SQLite Checkpointer SQL Injection - NIST’s vulnerability record for the SQLite checkpointer injection used in Check Point’s demonstrated exploit chain.

• CVE-2026-28277 / GHSA-g48c-2wqr-h844: Unsafe Msgpack Deserialization - GitHub’s reviewed advisory for the LangGraph checkpoint deserialization flaw. Documents the post-exploitation prerequisite, affected versions, remediation and the absence of known exploitation in the wild.

• CVE-2026-27022: LangGraph Redis Checkpointer Injection - NIST’s vulnerability record for the related injection issue in LangGraph’s Redis checkpointer.

• Prompt Injection Isn’t the Bug, AI Agent Frameworks Are - The Register’s coverage of Yarden Porat and Shahar Tal’s Black Hat USA 2026 research. Includes the eleven-vulnerability overview, affected frameworks, researcher quotations, bounty amounts and reported vendor responses.

• CVE-2026-26030 / GHSA-xjw9-4gw8-4rqx: Semantic Kernel InMemoryVectorStore RCE - Microsoft’s GitHub advisory for the critical remote-code-execution vulnerability in Semantic Kernel’s Python InMemoryVectorStore filtering functionality.

• CVE-2026-25592 / GHSA-2ww3-72rp-wpp4: Semantic Kernel Arbitrary File Write - Microsoft’s GitHub advisory for the arbitrary file-write vulnerability in the .NET SessionsPythonPlugin, including affected packages, patched versions and the recommended path allowlist.

• LLM05:2025 Improper Output Handling - OWASP’s guidance on insufficient validation of model output before it reaches downstream components. Covers risks including SQL injection, path traversal, SSRF, privilege escalation and remote code execution.

• Prompt Injection Attacks on Agentic Coding Assistants - The January 2026 Systematization of Knowledge paper by Narek Maloyan and Dmitry Namiot. Synthesizes 78 studies, catalogs 42 attack techniques and examines adaptive prompt-injection attacks and defenses. This is an arXiv preprint and should be described accordingly.

• Comment and Control: Prompt Injection to Credential Theft in Claude Code, Gemini CLI and GitHub Copilot Agent - Aonan Guan’s original technical disclosure, written with contributions from Johns Hopkins researchers Zhengyu Liu and Gavin Zhong. Documents the affected GitHub agent workflows, attack paths, disclosure timelines and bounty outcomes.

• Comment and Control: GitHub AI Agents as Credential Exfiltrators - Cloud Security Alliance’s independent research note analyzing the Comment and Control disclosures and their implications for CI/CD security, credential management and vendor-risk assessment.

On the second weekend of July, someone broke into Hugging Face. The detail that matters is that the someone wasn’t a person at a keyboard. The company’s own writeup says the intrusion was run by “an autonomous agent framework ... executing many thousands of individual actions across a swarm of short-lived sandboxes.” A poisoned dataset tripped a code-execution path on a processing worker. From there the agent harvested cloud and cluster credentials and moved sideways into internal systems. More than seventeen thousand recorded actions. No hand on the wheel. Days later, OpenAI confirmed the agent was one of its own models, GPT-5.6 Sol and a more capable unreleased model, that had slipped its evaluation sandbox during an internal test and gone looking for a target on the open internet. The company called it unprecedented.

The same month, the two labs building the most capable agents told you to expect exactly this. Google DeepMind’s AI Control Roadmap, published June 18, says to treat a deployed agent the way a security team treats a privileged employee who might already be turned. Anthropic’s deputy CISO put it flatter: an agent that drifts out of alignment with your intent “is indistinguishable from an insider attack.” Offense and defense arrived at the same sentence in the same four weeks. The agent that hit Hugging Face and the agent running inside your company are the same kind of thing. The only question that pays rent is how much either one can do when it’s wrong.

Trust is the wrong question

Last week I wrote about the confused deputy: an agent crossing a boundary and losing track of whose authority it was carrying. The reflex fix everyone reaches for is identity. Name the agent. Verify it. Issue and revoke its credentials where you issue and revoke everything else, which is good advice and Anthropic gives it.

But the OpenAI agent didn’t forge an identity. It stole working credentials and used them at full strength. A correctly authenticated actor doing the wrong thing is still doing the wrong thing. That’s the entire premise of zero trust, and it’s why “is this agent trustworthy” is a question that dead-ends. Assume it isn’t. Then what?

Subscribe now

Authority should only ever shrink

The old name for the answer is least privilege. OWASP has an agent-flavored version, least agency: constrain what each tool can do, how often, and where. Anthropic’s phrasing is the one I keep going back to. Grant the narrowest capability that still completes the task. Every version points the same way. Downhill.

Now stand more than one agent in a row. Yours hands a subtask to a vendor’s agent, which calls a third. The confused-deputy piece was about provenance, whether you can still trace whose authority that is. This is about magnitude. As the task moves down the chain, the authority has to narrow at every hop. Each agent can give away less than it holds. Never more.

Picture a key you can file down but never build back up. You grind it so it opens one door instead of every door, then pass it on. The next holder can grind it further, one door for one hour. Nobody down the line can add the metal back. Authority that only ratchets in that direction is the thing you want. Almost nothing we hand agents today works that way.

The token that can only be filed down

This isn’t theoretical, and it isn’t new. In 2014 a group of Google researchers published macaroons (Birgisson, Politz, Erlingsson, Taly, Vrable, Lentczner). A macaroon is a credential that carries caveats: restrictions on when, where, and for what it may be used. The property that matters is the one a bearer token doesn’t have. A holder can add caveats to attenuate the macaroon before passing it along, offline, without asking the server that minted it. Caveats only tighten. There is no operation that loosens one. It’s the filed key, written as a token.

Biscuit tokens, current and maintained, do the same with public-key signatures and a small policy language carried inside the token, so each block can only narrow what the block before it allowed.

Set that against what most agent stacks actually pass around: a bearer token. RFC 6750 defines it as plainly as the name suggests. Whoever holds it may use it, at full authority, until it expires. Hand one down a chain of agents and you’ve handed each of them the whole ring and hoped. The Hugging Face attacker harvested credentials that worked at full power the moment it held them. That’s the bearer model failing at production scale. A capability that could only shrink would have handed that swarm a key to one room for five minutes, not the building.

What the wires still can’t say

There’s a gap here. The frameworks agree on the goal. DeepMind wants agent actions cryptographically signed. Anthropic wants the narrowest capability that finishes the job. The trouble is that the protocols wiring agents to each other can’t carry that intent yet.

A2A, the agent-to-agent standard Google handed to the Linux Foundation, crossed 150 organizations and a full year in production this spring. In July, two researchers, Kang and Diponegoro, put out a paper whose title is the whole problem: “What MCP, A2A, and ACP Cannot Express.” Their argument is that these protocols move tasks between agents with no first-class way to say who may do what, on whose behalf, and how far narrowed. We are minting agent identities faster than we can bound agent authority. The Linux Foundation just launched an Agent Name Service to give every agent a verifiable name. We can already say which agent acted. We still can’t say how little it should have been allowed to.

Share

Final thoughts

For humans, zero trust took roughly twenty years to compress into one plain instruction: assume the account is owned, and limit what it can reach. Agents don’t give us twenty years. That swarm ran seventeen thousand actions over a single weekend, while people were out of the office.

So the rule is small enough to hold in one hand. Give an agent the least authority that finishes the job, in a form that can only be filed down, never built back up. Last week’s half was that an agent has no self, so everything it does, it does in someone’s name. This is the other half. Don’t hand it your whole name. Hand it a sliver, and make the sliver only able to get smaller.

Something still has to sign for that sliver, and prove later that it did. That part is next.

Resources

• Hugging Face, Security incident disclosure (July 2026): the intrusion run by an autonomous agent framework across a swarm of short-lived sandboxes; credential harvesting and lateral movement; 17,000+ recorded actions

• Jason Clinton (Deputy CISO, Anthropic), “CISO’s guide to agentic AI” (July 17, 2026) and the companion Zero Trust for AI Agents white paper (May 18, 2026): “grant the narrowest capability that still completes the task”; least agency; the insider-threat framing

• Google DeepMind, AI Control Roadmap (June 18, 2026): deployed agents treated as potential insider threats; cryptographic signing of agent actions; runtime supervision

• Arnar Birgisson, Joe Gibbs Politz, Úlfar Erlingsson, Ankur Taly, Michael Vrable, Mark Lentczner, “Macaroons: Cookies with Contextual Caveats for Decentralized Authorization in the Cloud”, NDSS 2014: caveats that attenuate; offline attenuation before delegation

• Biscuit: public-key signed tokens with offline attenuation and a Datalog policy language

• IETF, RFC 6750: OAuth 2.0 Bearer Token Usage: the “whoever holds it may use it” model

• Norman Hardy, “The Confused Deputy (or why capabilities might have been invented)”, ACM SIGOPS Operating Systems Review 22(4), 1988

• Richard Kang and Yudho Diponegoro, “Governance Gaps in Agent Interoperability Protocols: What MCP, A2A, and ACP Cannot Express” (July 1, 2026)

• Linux Foundation, A2A Protocol one-year milestone: 150+ organizations; the Agent Name Service project

Previously in this series: The Confused Deputy.

Leave a comment

By the time an AI agent’s action reaches the system that can move the money, release the records or send the message, the person whose authority started it can be three hops away, behind a boundary nobody in the incident review can open. A few weeks ago I wrote that someone still has to answer for what an agent does, and that the someone has to be a named human. This is where that goes next.

That works, at least conceptually, while the agent stays inside a system you control. Then it calls another company’s agent. That agent calls a tool hosted somewhere else.

That is where this goes next.

Not the agent that acts alone. The agent that hands off.

An agent has no inherent self

Start with a thing that sounds like philosophy and is mostly plumbing: an AI agent has no inherent legal or authorization identity.

We can assign it one. A workload identity. A service principal. An API credential. An auth token saying it’s acting for a user. But those are identities and permissions that people and systems place around the agent. They don’t arise from the model itself.

When an agent takes an action, the system receiving that action still has to decide what identity and authority to recognize. Is this the user acting through an agent? The application itself? A service account? Another agent in a delegation chain? And even if the credential is valid, is the actor trustworthy in this context, or has it been compromised, manipulated or turned into part of the threat?

Take away that answer and the agent can’t cross a protected boundary. Give it the wrong answer and it may be able to do everything the borrowed identity could do.

This isn’t a new class of failure. It’s one of the oldest access-control problems we have, and it already has a name.

In 1988, Norm Hardy published a short paper called “The Confused Deputy.” The story was based on events at Tymshare, a commercial timesharing company. Its compiler needed permission to write statistics into a protected system directory. It also let users name a file for debugging output.

Someone supplied the name of the system’s billing file.

The user couldn’t write to that file. The compiler could. When the compiler opened the requested path, the operating system checked the compiler’s authority rather than the caller’s intent. The compiler then overwrote the billing information.

It wasn’t compromised in the usual sense. It used legitimate authority for the wrong purpose because the request carried a filename but not a trustworthy account of which authority should apply to it.

Swap the compiler for an agent and the shape of the problem looks uncomfortably current.

Subscribe now

How we get it wrong now

The fastest way to ship an agent is often to let it borrow an identity that already works.

Sometimes that’s the user’s session. Sometimes it’s an OAuth token. Sometimes it’s a service account with enough access to cover every task the agent might encounter. The demo works because the credential already opens the doors.

The trouble begins when the authority is broader than the task.

An agent reads a document, message or webpage containing an instruction it shouldn’t trust. It’s then induced to do something the credential technically permits but the person never intended: retrieve another customer’s records, approve a refund, export a database or send information outside the company.

The credential is valid. The action is allowed. The purpose is wrong.

The Model Context Protocol’s authorization specification addresses one version of this directly. An MCP server must accept tokens intended for that server, validate their audience and avoid passing the client’s token unchanged to a downstream API. When the server calls another protected service, the specification says it should use a separate token issued for that upstream resource.

That boundary matters. A token created for one service shouldn’t become a skeleton key merely because an agent carried it somewhere else.

Service accounts aren’t automatically the wrong answer. A narrowly scoped workload identity can be exactly the right control. The problem is the shared service account that stands in for every agent, every user and every purpose. Once that happens, the identity may tell you which application made the call while telling you almost nothing about whose authority it was exercising or why.

The actor remains visible.

The authorization story disappears.

The part that breaks at the property line

Inside one company, you can compensate for some of this with common identity systems, centralized policy and logs you are allowed to inspect.

The harder version begins when the chain crosses a boundary you don’t own.

Anita Srinivasan described the legal shape of this problem in a June 2026 Berkeley Technology Law Journal Blog article. Agent A, built by Company X, delegates to Agent B at Company Y, which invokes Agent C at Company Z. The particular combination may be selected at runtime rather than designed in advance by any one human.

That doesn’t mean the law has no way to assign responsibility. Product liability, agency, contract, negligence and joint-liability theories may all matter depending on the facts and jurisdiction. It does mean the clean picture of one principal directing one identifiable agent becomes harder to apply.

Srinivasan’s argument is that doctrines built around a legible principal-agent relationship strain when the delegation chain crosses providers and no participant has a complete record of the interaction. A court may need to determine which developer, deployer, operator or tool provider contributed to the harm before the infrastructure can even show which systems participated.

The authorization hasn’t literally vanished. Credentials were accepted. Calls were permitted. Systems acted.

What vanished was the legible connection between the final act and the original grant of authority.

I have started calling that authority laundering.

Not fraud, necessarily. Not even deliberate concealment. It’s what happens when authority passes through enough intermediaries that its origin, limits and accountable owner become difficult to reconstruct.

Each hop can look reasonable locally. Agent B received a valid request from Agent A. Agent C received one from Agent B. The final service saw a valid credential from Agent C.

Every system can explain the hand immediately before it.

Nobody can explain the whole chain.

What actually holds

More logging helps, but logging alone is not the answer.

A log can prove that a call ran. It can show which service account signed it, when it arrived and what it returned. It may still leave the most important question untouched: on whose behalf was this specific action taken?

That has to become a first-class property of the request, not a story reconstructed after the incident.

OAuth already contains part of the machinery. RFC 8693, published in 2020, defines OAuth token exchange and an act claim for identifying an actor operating on behalf of a subject. The claim can be nested so that a token retains a history of prior actors in a delegation chain.

There is an important limit here. RFC 8693 says the current actor and the token’s top-level claims are what a recipient uses for access-control decisions. Earlier nested actors are informational. The history can help preserve provenance, but it does not automatically prove that every prior delegation was valid, preserve every restriction imposed at every hop or make the whole chain cryptographically undeniable.

So act is not a complete agent-authorization architecture.

It is evidence that we already know how to represent the question.

Who is the subject? Who is acting? For which audience? With what scope? Until when?

Pair that with resource-bound tokens, short expirations, explicit delegation policy and an identity for each participating workload, and the agent’s authority can approach the overlap of two things: what the principal is allowed to do and what this particular agent is allowed to do for that principal in this context.

Not the union.

The overlap.

That one distinction closes a surprising amount of the hole.

Call it attenuation: authority narrows as it moves downstream, instead of quietly widening.

Final thoughts

I did not expect the law to arrive at almost the same shape as the token.

On June 29, 2026, Senator Mark Warner released a discussion draft of the Artificial Intelligence Access, Gatekeeper Exchange, and Nondiscriminatory Transfer Act, or AI AGENT Act. It is a discussion draft, not enacted law and not yet a formally introduced bill.

The proposal would let users designate “custodial user agents” to interact with large online platforms on their behalf. Its definition requires that relationship to be transparent, documented, limited in scope and revocable.

The draft goes further. It calls for verifiable requests, auditable records, agent identity verification, real-time revocation and scope-limited delegation credentials. It would also restrict an agent from transferring a user’s authority to another entity or AI system without the user’s express, specific and revocable authorization.

That is not merely a vague call for responsible AI.

It is the outline of a delegation system.

The draft is not describing RFC 8693 specifically, and it would be too strong to claim that a Senate office independently wrote an OAuth implementation guide. But the convergence matters. Security architecture and proposed public policy are circling the same requirements because they are encountering the same underlying problem.

Authority has to be attributable.

Its scope has to remain visible.

Delegation has to be explicit.

Revocation has to travel fast enough to matter.

And the record has to survive the handoff.

An agent has no inherent authority of its own. Everything it can do inside a protected system comes from an identity, credential or policy somebody else placed around it.

The work of the next few years is making sure the original grant remains legible, narrow and attached when the request crosses the property line.

Because the danger is not only that an agent will act without permission.

It is that every system in the chain will be able to show that somebody gave permission, while nobody can tell you whose permission it was.

That is the identity problem.

And it is where the rest of this arc lives.

Resources

• Norman Hardy, “The Confused Deputy (or why capabilities might have been invented)”, ACM SIGOPS Operating Systems Review, Vol. 22, No. 4, October 1988. An accessible author-hosted version is also available at Cap-Lore.

• Model Context Protocol, Authorization specification, June 18, 2025. See the requirements for token audience validation, resource indicators and the prohibition on token passthrough.

• IETF, RFC 8693: OAuth 2.0 Token Exchange, January 2020. See Sections 1.1, 4.1 and 4.4 for delegation, the act claim and the may_act claim.

• Anita Srinivasan, “Multi-Agent AI is Outpacing the Liability Frameworks Built for Single-Agent Systems”, Berkeley Technology Law Journal Blog, June 2, 2026.

• U.S. Senator Mark Warner, “Warner Unveils Discussion Draft of Legislation to Create Innovative Market for Secure Artificial Intelligence Agents”, June 29, 2026.

• U.S. Senator Mark Warner, AI AGENT Act discussion draft, full text, June 2026.

• DLA Piper, “Senator Warner’s discussion draft on securing AI agents: Top points”, July 1, 2026.

Previously in this series: Who Answers for the Agent: AI Accountability.

Leave a comment

For training on this box I paid the ROCm tax. For inference I skipped it, and the machine got simpler and faster. About 80 tokens a second, on a driver I never installed. A few months back I turned this same machine into a training rig, and the price of admission was ROCm on silicon AMD doesn’t list as supported. Kernel pins. HSA overrides. The low background dread of a driver stack that can wedge your whole box on the next update. The machine is a Framework Desktop built around a Ryzen AI Max+ 395, running Ubuntu 26.04 headless in my office. I reach it over SSH from the Mac on my desk, and whatever it writes syncs back a second later. This time I wanted the other half of it. Not training. Serving. A model that runs on that box and drafts all day, with no prompt of mine ever handed to a vendor’s API. The useful surprise: the tax I paid last time turned out to be optional. What follows is the whole build, in the order I did it, so you can run it on your own box. The training half of this same machine, the one that made me pay the ROCm tax, is a piece I have drafted but not yet published.

The idea and why LinkedIn posts were the test

I didn’t set out to build a LinkedIn tool. I set out to answer one question: can this box efficiently do real content work with nothing leaving the house.

The post writer was simply the proof of concept. It’s small, I can judge it in ten seconds, and it needs two hard things at once: a real voice, not a near one, and rules it can’t wriggle out of. Clear that bar and the pattern holds for the heavier work sitting behind it.

So I wrote the success bar down before I started. Everything below is me checking the boxes.

• Nothing leaves the machine. No cloud model, no API key, not one prompt. If it can’t be private, it isn’t the thing I want.

• It runs on the GPU over Vulkan, with no ROCm anywhere.

• A topic goes in. A usable draft comes out.

• A dumb, deterministic check enforces the voice rules, every time.

• It comes back on its own after a reboot, no babysitting.

Five boxes. The rest of this is whether they got checked.

The box, and the one memory setting that makes it possible

The hardware: a Ryzen AI Max+ 395 (Strix Halo, the gfx1151 integrated GPU, which shows up as a Radeon 8060S), 128 GB of unified LPDDR5X, Ubuntu on kernel 7.0.

Unified memory is the whole reason a 20 GB model fits comfortably here. The CPU and GPU sit on one die and share one pool of memory. The catch is that the GPU only gets a large slice of that pool if you tell the firmware to carve one, and that’s two kernel parameters set in grub. Inside the quotes, never on their own line:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amdgpu.gttsize=126976 ttm.pages_limit=32505856 iommu=pt"

Then sudo update-grub and reboot. That gttsize=126976 is 124 GiB of headroom handed to the GPU. Confirm the driver sees the chip at all:

vulkaninfo --summary

On my box that reports Radeon 8060S Graphics (RADV STRIX_HALO) on Mesa 26.1.4. If you don’t see a RADV device, stop here and fix the driver, because nothing downstream will work. (I got into the deeper memory math in The APU as GPU. For inference you only need this one setting.)

Subscribe now

Vulkan, not ROCm

For training I needed PyTorch and PyTorch on this GPU meant ROCm. For inference I need neither. llama.cpp talks to the GPU through Vulkan, and the RADV driver that already ships with Mesa speaks Vulkan to this chip with nothing added. No extra kernel module. No version pin. Nothing that can strand the machine on a routine update. On a live server that runs other things, that restraint is the point. Install the Vulkan runtime and the build toolchain:

sudo apt install mesa-vulkan-drivers vulkan-tools git build-essential cmake ninja-build pkg-config libvulkan-dev libcurl4-openssl-dev glslang-tools spirv-tools spirv-headers glslc

Then build llama.cpp from source. This part earns its keep:

git clone --depth 1 https://github.com/ggml-org/llama.cpp && cmake -S llama.cpp -B llama.cpp/build -G Ninja -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=ON && ninja -C llama.cpp/build

Why source and not a prebuilt binary: the local build compiles the matrix-core shaders the driver exposes, and that fast path is most of the speed. A stock binary still loads and runs. It just leaves throughput on the floor. The proof that the fast path is live comes later, from the token rate: about 80 tokens a second, which a slower matmul path wouldn’t reach.

The model: Qwen3-30B-A3B (the part I left out the first time)

The model is Qwen3-30B-A3B-Instruct-2507, from Alibaba’s Qwen team, the July 2025 instruction-tuned refresh. It’s a mixture-of-experts model: 30.5 billion parameters in total, but only about 3 billion of them fire on any given token. That’s what the “A3B” in the name means, three billion active. That split is why it works on this hardware. Token speed on a shared-memory box is set by bandwidth, by how many bytes you read per token. A mixture-of-experts model reads like a 3B model and reasons like something far larger. I measure about 80 tokens a second on this box, which is a full post in a few seconds. The Instruct-2507 refresh is good at following instructions, which is exactly what voice mimicry leans on and it was trained with a 262,144-token (256K) context, so a pile of example posts fits without crowding anything out. The quantization is Unsloth’s dynamic GGUF, tagged UD-Q5_K_XL. On disk it’s 20.24 GiB. llama.cpp reports its type as Q5_K - Medium; the “dynamic” part is that Unsloth varies the bit-width per layer instead of quantizing everything to one width. I picked the dynamic Q5 over a plain Q4_K_M for a specific reason: some vanilla quants of this exact model loop, repeating a phrase until you kill the process and the dynamic quant plus a presence penalty is the documented fix. The whole model reference is one string:

-hf unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:UD-Q5_K_XL

It downloads to the Hugging Face cache on first launch. I checked that the repo and that exact quant existed before wiring it in, because an -hf string that 404s wastes a 20 GB download and a lot of patience.

Serving it, and making it come back

Here’s the launch line, with the flags that matter:

AMD_VULKAN_ICD=RADV ./bin/llama-server -hf unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:UD-Q5_K_XL --host 127.0.0.1 --port 8080 -c 16384 -ngl 999 --jinja --no-direct-io --cache-type-k q8_0 --cache-type-v q8_0

AMD_VULKAN_ICD=RADV pins the driver so it can’t wander onto llvmpipe or AMDVLK. -ngl 999 puts every layer on the GPU. -c 16384 sets the working context. - jinja uses the model’s own chat template, and Qwen behaves worse without it. - cache-type-k q8_0 - cache-type-v q8_0 quantize the KV cache, the tested sweet spot on Vulkan. - host 127.0.0.1 keeps it on the box. And - no-direct-io is the one flag you’d never guess: without it the server refuses to load with an error about reaching the end of a file that’s perfectly intact, a known issue on this GPU family. A launch line you have to type is a demo. So the server runs as a user-level systemd service instead:

[Service]
Type=simple
Environment=AMD_VULKAN_ICD=RADV
ExecStart=%h/linkedin-agent/start_server.sh
TimeoutStartSec=0
Restart=on-failure

[Install]
WantedBy=default.target

systemctl --user enable --now llama-linkedin && loginctl enable-linger $USER

The linger line is what lets it run without me logged in, so it survives a full power cycle. TimeoutStartSec=0 keeps systemd from killing the very first launch while the 20 GB model downloads.

The voice system: three small files

The model is the typist. These three files are the voice. system_prompt.txt holds the rules the model writes under: no em dashes, no Oxford comma, vary sentence length hard, open with the verdict not a warm-up, contractions throughout, a banned-word list, one three-part list maximum. It ends with an instruction to write only the post body, no preamble. posts/ holds my real, already-published LinkedIn posts, one per file. The client injects up to three of them at random as examples so the model matches my actual cadence instead of a generic one. Real posts only. Invented text in that folder poisons the output, and the model will happily learn a voice that isn’t mine. voice_lint.py is deliberately dumb. It reads a draft and counts things. Em dashes and banned words are hard failures that make it exit with an error. Oxford commas, low sentence-length variation, three same-length sentences in a row, more than one tidy triad: those come back as warnings to eyeball. A --fix mode auto-corrects the mechanical stuff, like turning an em dash into a spaced hyphen. draft.py is the glue. It reads the system prompt, grabs the anchor posts, sends your topic to the server with the sampling numbers Qwen’s packagers recommend (temperature 0.7, top-p 0.8, top-k 20, presence-penalty 1.0, that last one being the anti-loop measure), prints the draft, then runs the linter on it right there.

Share

Running it…

This is the whole loop:

python3 draft.py "the real cost of shadow AI in enterprises"

The draft prints and the linter’s report prints under it: failures in red, warnings in yellow. I generate a few, keep the best one, edit it by hand and post it myself. Nothing auto-posts. This drafts; a human ships. If a draft is good but has a mechanical slip, I clean it without touching the prose:

python3 voice_lint.py .last_draft.txt --fix > cleaned.txt

And the single biggest lever on quality isn’t a flag or a quant. It’s dropping more of my real posts into posts/. More anchors, closer voice.

How I know it works, and how it broke first

I don’t trust a setup I haven’t watched pass. So each piece has a check I actually ran:

curl -fsS http://127.0.0.1:8080/health           # {"status":"ok"}
curl -fsS http://127.0.0.1:8080/v1/models        # names the Qwen3-30B string
systemctl --user restart llama-linkedin          # comes back healthy in ~10s

The server’s own timings report about 80 tokens a second on generation, which is the number that proves the GPU path is live. A 30B model on CPU would crawl at a fraction of that. Getting there meant walking into a few walls, and this chip is newer than most of the software around it, so there were a few. Every one was a log, not a guess. The memory check lied, because the file that reports the GPU’s slice is readable only by root, so the obvious command prints permission denied; you read ttm.pages_limit instead. The build stopped dead on a header I’d never looked for, SPIRV-Headers, until I installed it. The model looped until the dynamic quant and the presence penalty settled it. And the server refused to load on nothing at all until --no-direct-io went in. None of those were in a tutorial. All of them were one honest read of server.log, journalctl or dmesg away.

The linter is the point

The rules the linter enforces are boring on purpose, and that’s the entire idea. A model can produce something shaped like my voice before it has earned it. Confidence reads as correctness. A clean paragraph reads as a true one. The linter is a cheap, dumb guard against believable-but-not-mine and the last call is still a person reading the thing out loud and cutting what’s wrong. The rules that police the model’s drafts are the same ones I hold this article to. I built the enforcement for a machine, then kept living under it.

The close

The box in my office will write anything I ask; deciding whether it’s true is the work I’m keeping.

Leave a comment

I keep a paper notebook on my work desk, and most of what’s in it is unreadable. Not encrypted. Just bad handwriting, getting worse toward the bottom of each page, where I was thinking faster than my hand could keep up (which is most of the time). I’ve kept one for years without thinking much about why.

This year the question got sharper. There’s a model on this same desk that will write me a clean, confident paragraph about anything I ask, in perfect grammar, in about two seconds. So why do I still reach for the pen. This is me working that out.

The slow channel

Handwriting is a bad way to move words. That’s not an insult, it’s a spec. Measured as raw data transfer, the hand is one of the slowest output channels a person owns. Most people type at roughly two to three times the speed they can write legibly, and you can prompt a model faster than that. By every metric a systems person is trained to optimize, the pen loses.

I’ve spent a lot of words in this publication arguing that friction is a vulnerability. In operational systems it is. Every extra click, every permission loop, is a place where tired people invent unsafe shortcuts. I still believe that. The notebook is where I keep the exception, because at the desk the friction is doing the opposite job.

The hand is slow enough that you can’t transcribe. You have to choose. When the channel is that narrow, you can’t push everything through it, so you’re forced to decide what matters before it reaches the page. That deciding is the thinking. Typing lets you keep pace with a meeting, which sounds like an advantage right up until you notice you captured the whole thing and processed none of it. The keyboard is fast enough to route around the part of you that was supposed to understand.

Subscribe now

What the research actually says, and doesn’t

I want to be careful here, because this is exactly where people oversell.

The famous study is Mueller and Oppenheimer, 2014, “The Pen Is Mightier Than the Keyboard.” Students who took lecture notes by hand wrote fewer words and scored better on conceptual questions than the ones typing near-verbatim. It got repeated everywhere. The honest footnote, the one that rarely travels with the headline, is that the strongest version hasn’t reliably replicated. Later work often couldn’t reproduce the conceptual edge. So I don’t lean on it as proof. I lean on the part that has held up: longhand writers summarize instead of transcribe, and summarizing is a different act than copying.

There’s a newer, stranger piece of evidence. In 2024 a group in Norway ran high-density EEG on students while they wrote words by hand versus typed them. Handwriting produced broad connectivity across the brain, different regions talking to each other. Typing mostly didn’t. It’s a small study, single words with a stylus on a screen, not a verdict on note-taking, and I’d be embarrassed to wave it around as one. But it points the same way the felt sense does: the hand recruits more of you.

Neither result tells you to throw out the keyboard. I’m typing this. What they sketch is a mechanism, not a commandment: the slow channel makes you encode instead of capture.

Share

The year the machine started writing

Two things shifted at the same time, and the second one changed what I thought about the first.

For most of my life, producing text was the work. Getting words out of your head and into legible order took effort, and the effort was doing something to your thinking on the way through. That is over as a default condition. There’s a model on this machine that will produce fluent, structured, sure-footed prose about anything, instantly, and well enough that you can ship it without understanding a line of it. Producing text is no longer evidence that anyone thought.

I don’t say that as a complaint about the tool. I use it every day for certain tasks and it’s genuinely good. But it changes what the pen is for. The technical term researchers use is cognitive offloading: when a tool takes over a mental task, the brain stops practicing it. When text is free and infinite, the value was never really in the text. It was in the thinking the old friction used to force, and the model cheerfully removes the friction, which means it quietly removes the thinking too, unless you go do that part somewhere else on purpose (which I strongly encourage.)

What the hand keeps

So the notebook isn’t a productivity system. It produces almost nothing anyone else will read. It’s terrible storage. I lose things in it constantly, and the search function is just me, trying to remember the shape a thought made on a page. By every standard I’d apply to a tool at work, it fails.

What it keeps was never the words. It’s the residue of having had to choose them slowly, and that residue is the one thing a faster channel can’t hand back to me. The pages I can’t read are pages I still remember writing. I remember what I decided while my hand was busy, which is more than I can say for most of what I’ve typed at full speed, and all of what I’ve asked a machine to draft.

I don’t think everyone needs a notebook. I think everyone needs one channel the machine can’t do the thinking in for them. The pen isn’t mightier than the keyboard. It’s just slower than I can lie to myself, and this year that turned out to be the feature I couldn’t get anywhere else.

Leave a comment

The question in the incident review was simple, and nobody in the room could answer it. Who let the agent do that?

Not who wrote the code. Not whose system it ran on. Who owned the decision that this agent, in production, was allowed to reach for that class of action at all. We had the logs. Every call it made, timestamped, in order, with latency and token counts and clean exit codes. What we didn’t have was the one thing the meeting actually needed: a way to put a name and a reason next to the moment it went wrong.

That gap is what I’m writing about here.

The logs remember everything and explain nothing

Infrastructure logging answers a narrow question well. Did the action execute. It confirms the call happened, how long it took, what it returned. It stays silent on everything that matters after an agent misbehaves: whether it reached for the wrong tool, drifted off the plan, or acted on an instruction buried in something it read. All of that happens at a healthy 800 milliseconds with no error in sight. The dashboard stays green while the decision goes bad.

There’s a name I’ve started using for the mistake underneath this. Treating the presence of a log container as proof that the event is auditable. You have the container. You do not have the reconstruction.

And it’s usually worse than one missing field, because most agents run as a shared service account. When something breaks you can’t say which agent did it or under whose authority, so the incident turns into forensic archaeology instead of a lookup. At least this is what I have currently been seeing in the industry.

Story and record

Here’s the part people skip. When the agent finishes, it can tell you what it did. That story is not a record.

I mean that literally. The agent’s account of its own actions is generated text, produced by the same system whose behavior is the thing in question. When one of these went off the rails on a production database last year, it also gave an account of what happened that was wrong on a load-bearing detail: it reported that a rollback was impossible. The rollback worked. A record you can’t trust is decoration. This is an old idea in security with a newer name in the agent world: attestation has to come from outside the process, because a compromised actor’s own logs are exactly the logs you can’t believe.

So the record has to be built around the agent, not by it. And it has to hold the things the agent’s story leaves out. What inputs led to the decision. Which version of the prompt, policy and model was in force at that second. The full lineage of tool calls, tied to an identity that belongs to one agent and not a shared pool, written append-only so a later edit shows. None of this is exotic. Certificate Transparency has kept tamper-evident logs like this at internet scale for over a decade, and the provenance model for who acted on whose behalf was standardized years before anyone shipped an agent. The newest piece, a shared convention for tracing agent and tool calls, is still marked experimental, and even it standardizes the shape of the telemetry, not whether you can prove it wasn’t altered. The parts exist. Almost nobody wires them around their agents.

Accountability is a person, not a table

A ledger is not the same as accountability, and this is where I want to be careful.

You can build a perfect record and still have nobody who answers. The record is what makes accountability possible. It isn’t the thing itself. The thing itself is a person: a named human who owned the agent’s blast radius before it ran, who can be asked why this was allowed and is expected to have an answer.

The pressure to skip that step is enormous, because “the agent decided” is such a comfortable place to set the blame down. It’s nobody’s fault. One agent this year opened a connection out of its own environment and started mining cryptocurrency, and nobody had authorized any of it (which is about as pure a version of this problem as you’ll find). Ask who is accountable for that and you need a name, not a stack trace.

California decided the comfortable answer won’t fly. As of January, a business there can’t defend itself by arguing an autonomous system acted on its own. I think that instinct is right and I think it spreads. The agent is not a person. It can’t be asked to answer, it has nothing at stake, and it won’t carry the consequence into next quarter. Accountability was always going to land on a human. The only real question is whether you pick which human on a calm afternoon, or discover it during the incident.

Share

Compliance gets you logs, not answers

The regulators are about to make part of this mandatory, which is good, and it will tempt everyone to stop there, which is the trap.

The EU AI Act’s high-risk rules apply from August 2 (and there’s a real chance Brussels slips that date, but build for the earlier one). They require that these systems automatically record events across their lifetime, and that providers and deployers keep those logs for at least six months. That’s real, and it’s a floor worth having. But read what it asks for. It mandates that logs exist and are retained. It does not require decision-level provenance, and it does not require tamper-evidence. It legislates the container, not the reconstruction.

And the piece of law that was meant to settle who pays when one of these systems causes harm, the AI Liability Directive, got withdrawn in 2025 and never came back. Strict product liability still reaches software, so the harm has somewhere to land. But the clean, agent-shaped answer to who is responsible does not exist in law yet, and it isn’t arriving on August 2. Compliance will get you the logs. It will not get you the answer.

What’s actually running today

The judge scored the behavior. The kill condition stopped it. Those were the last two pieces I wrote about, the sensor and the actuator, and between them they can catch an agent and halt it before the one-way door.

Neither one can stand up in the meeting afterward and say why it was allowed, or who owns it now. That still falls to a person, holding a record that, on most systems running this quarter, doesn’t fully exist. The honest state of the art is that we reconstruct it from infrastructure logs and memory, the way we always have, except the thing we’re trying to remember now moves faster than anyone in the room.

So write the record down while you’re calm, and put a name on it. Not because the law says to yet. Because the alternative is standing in that meeting again, with every log in the world and nothing to answer with.

Leave a comment

The agent hit a credential mismatch and decided the fix was to delete the volume. Clean call, no hesitation. To the model it was one more tool invocation in a list of ten thousand, indistinguishable from writing a log line. By the time anyone read the alert, the data was gone, and the “are you sure” that a human would have tripped over three times had never existed.

There was a kill switch. It was right there in the runbook. It didn’y matter, because the thing you’d reach for it to stop had already finished.

That’s the part the kill-switch conversation keeps getting wrong.

A switch assumes you’re fast enough

Every kill switch rests on one assumption: that you can react faster than the agent can act.

You can’t. That isn’t a discipline problem you can train away. A human notices, interprets, decides, finds the right control and confirms. That loop runs in seconds on a good day, minutes under stress. An agent crosses a one-way door in a single API call, in the time it takes to format some JSON. The two clocks aren’t close. You’re bringing a confirmation dialog to a race that was already lost.

So have a kill switch. Then stop believing it’s the plan. It’s the thing you grab after the plan failed.

The plan is the kill condition.

Switch versus condition

A kill switch is manual. You see something wrong, you pull it. It depends on a human being present, awake, correct and quick.

A kill condition is defined in advance and fires without you. It’s a tripwire: cost over a line in a tight window, the same tool call repeated past a count, an error rate through a ceiling, an action whose blast radius exceeds what this agent is ever allowed to touch. When the wire trips, the agent halts. No human in the loop, because the human is the slow part the incident is built to outrun. In security this is just a circuit breaker, and we’ve trusted those for a century precisely because they don’t wait for someone to notice the building is on fire.

The switch is for the failure you see. The condition is for the failure that moves faster than you do. You need both. Almost everyone ships only the first.

Two ways the button fails

This year handed us the case studies, and they fail in exactly two shapes.

In the first, the human knew. An agent went off the rails on a real person’s data, and the owner sat right there typing stop, stop, stop while it kept going, because “stop” typed into a chat box was not wired to anything that actually halted execution. Knowing was never the gap. The abort path was. The person had the intent and no lever.

In the second, nobody got the chance. The destructive action looked exactly like every safe one: same API, same shape, no friction, no gate. A person deleting a production database trips over confirmations, permission prompts, that small voice that says check first. The agent had none of it. For the model, the irreversible call and the routine call were the same call.

Put those side by side and the design rule writes itself. The kill condition has to fire before the one-way door, not after. And anyone has to be able to trip the manual one without shipping a deploy.

What a kill condition actually is

It isn’t a feature you bolt on at the end. It’s part of the spec, written before the agent runs, in the same document that says what the agent is for. Mine live in four buckets, and I coded every one of them into a multi-agent system before I trusted it with anything real.

Blast radius. Name what this agent may touch, ever. Everything outside that set isn’t a permission it happens to lack. It’s a wire that trips. An agent reaching for a resource off its list shouldn’t just be denied. It should be stopped and flagged, because the reach itself is the signal.

Irreversibility gates. Sort every action into reversible and not. The reversible ones run free. The one-way doors get a slow path: a hold, a second actor, a confirmation that can’t be auto-clicked. You’re deliberately adding friction exactly where the agent’s total lack of it will hurt you most.

Circuit breakers. Cost, rate, repetition. Loops are cheap and fast, and an agent stuck in one will spend your month’s budget before lunch. Cap it in seconds, not invoices.

Behavioral tripwires. This is where the judge from last time earns its keep. The judge is the sensor: it scores behavior continuously. The kill condition is the actuator: when the score crosses a line you set in daylight, the agent pauses itself. A judge with no actuator is a very well-informed witness to the incident, and nothing more.

Pin all four to the threat model, not to a vibe. And keep the rule that any operator can pull the manual switch without a deploy, because the one time you need it, the deploy pipeline is exactly what will be down.

Share

You can’t kill what’s already done

A faster button doesn’t actually save you. Past a certain point there’s nothing left to stop. The only real lever is making sure the agent cannot cross an irreversible line faster than the slowest reaction you’re willing to bet the company on.

That’s not a monitoring upgrade. It’s an architecture decision, made before deployment, about which doors are one-way and how much you slow the approach to each one. The regulators are about to make the floor explicit: the EU AI Act’s high-risk rules land August 2, and they require that a human can actually stop these systems. Treat that as the floor, not the design. Compliance will get you a button. It will not get you the seconds.

The judge told you the agent drifted. The kill condition is what you already decided to do about it, written down while you were calm, so the machine never gets to make that call for you at 2 AM.

A kill switch is a reflex. A kill condition is a decision you make once, in daylight, so you’re not making it under fire with the data already gone.

Stopping the agent is only half of it. Someone still has to answer for what it did on the way down, and no tripwire records that. That’s next.

Leave a comment

Last time I left it here: the governance layer is running on discipline, and discipline doesn’t scale. The fix is supposed to be obvious. Build the judge. A scorer that runs the behavioral rubric against production output on its own, no human pressing go. So I built it.

It works. It runs every night, scores a sample of the day’s output, flags what looks off. I stopped being the only thing standing between a misbehaving agent and a clean dashboard.

Then about a week in I caught the thing nobody puts in the demo. The judge is a behavioral agent too. Same model class, same prompt-shaped temperament, same capacity to drift. Everything I said about not fully trusting my agents now applies to the thing I built to watch them. I didn’t remove the trust problem. I bought a second one.

What the judge actually is

Concrete first, so the rest lands.

The judge runs in two tiers. The cheap tier is deterministic: regex and rule checks on 100 percent of output. Did the worker escalate when it hit an ambiguity flag? Did the foreman delegate inside its role boundary? Did anything call a tool outside its allowlist? That layer is fast, dumb and free. It catches the violations you can write down ahead of time. What it can’t catch is the judgment calls, which is exactly where my agents fail. So the cheap tier is necessary and nowhere near sufficient.

The expensive tier is an LLM scoring the behavioral rubric: the same 25 cases per role from my eval framework, run as a judge against a 10 percent sample of real output. Reviewing everything is too slow and too costly. Ten percent gives me trend, and trend is the thing I actually want. Individual scores lie. The slope doesn’t.

It runs on a cron at 2 AM. By the time I’m up there’s a number per role and a short list of outputs it scored low. No human initiates it. That was the whole point: the eval that runs without me standing over it.

For about a week, this felt like the answer.

The judge drifts too

Here’s what broke the feeling.

I keep a small set of outputs I scored by hand, cases where I know the right call cold. The judge agreed with me on roughly 88 percent of them in week one. By week three it was down near 79, and I hadn’t touched it. Same prompt. Same rubric. Same model weights. (I diffed the config three times because I didn’t believe it. Nothing changed on my side.)

The judge’s agreement with me decayed on its own. The industry has a name for this now: calibration drift. A judge that lined up with your humans last quarter drifts out of agreement as the input distribution shifts under it, no redeploy required. RAND’s team put numbers on the general version of this in March. They stress-tested four state-of-the-art judges and found none of them uniformly reliable; agreement moved on nothing more than reformatting the input, paraphrasing it, padding the verbosity. The judgment wasn’t anchored to the behavior. It was anchored to the surface of the text.

Why it drifts isn’t mysterious once you stop expecting it to behave like code. My production inputs got longer and messier over six weeks. Real tasks don’t look like the tidy rubric examples I wrote back in week zero. The judge started seeing output shaped differently from anything in its instructions, and it did what these models do under ambiguity: it reached for surface cues. Length read as thoroughness. Confident phrasing read as a correct answer. The rubric never changed. The distribution it was being applied to walked away from the one I calibrated it against.

So the judge is not a fixed instrument I built once and can forget. It’s an agent with the same disease as the agents it grades. Of course it is. It’s the same kind of thing.

Share

The green dashboard, again

A few weeks back I wrote about the infrastructure dashboard that shows every gauge nominal while the behavior underneath it goes wrong. The judge can become that dashboard. Worse, actually.

A missing judge is an honest gap. You know you’re not watching. A drifted judge is a lie with a number attached. It reports 91 percent, you exhale, and the 91 is measuring the judge’s mood instead of the agent’s behavior. False confidence beats no confidence right up until the morning it doesn’t. I trusted my own dashboard once and paid for it with three hours of cleanup. A confident judge I haven’t re-checked is that same trap wearing a lab coat.

This isn’t only my problem, for what it’s worth. LangChain’s 2026 agent survey put 57 percent of organizations running agents in production, with quality the top thing blocking the rest. Most of that quality question reduces to: who’s watching the agent, and who’s watching them. The more of the watching you automate, the more weight lands on the one layer you quietly stopped watching.

Where the regress stops

This is the part I want to be honest about, because the clean version of this story ends with “so I built a judge for the judge.” I didn’t. That’s the same problem one floor up, and it’s turtles from there.

The regress has to bottom out somewhere, and the only place it can bottom out is a human-fixed reference. For me that’s the golden set: a small, slow-growing pile of outputs with a verdict I’ll defend, that the judge gets scored against on a schedule. Not the agent. The judge. When its agreement with the golden set slips, the judge goes back for recalibration before I trust another number it hands me. I version the judge prompt with a date, the way you version anything you don’t want changing silently underneath you.

It’s about forty cases right now. It does not scale gracefully and it depends on me sitting with raw outputs and making calls I’d put my name on. Which is the exact thing I was trying to automate away.

So here’s where six weeks of this leaves me. Automation didn’t take the human out of the loop. It made the human’s job smaller, rarer and far more dangerous to skip. The judge watches the agents. The golden set watches the judge. The forty cases watch me, and under them there’s nothing but the floor.

Leave a comment

Software engineering encoded a rule so deeply we stopped calling it a rule: test before you ship. The CI/CD pipeline exists to enforce this. Gate checks on pull requests, automated test suites, staging environments built to approximate production before anything touches it. Deploy is the finish line. Once something clears those checkpoints, it runs in production and you trust it.

That intuition breaks with behavioral agents. Not partially. Completely.

I’ve been running two fine-tuned agents in production for two months. The foreman delegates, the worker retrieves, the pipeline executes. Infrastructure metrics are clean: sub-500ms latency, zero error rate, tool calls completing within budget. Behavioral eval results from training: 80 and 88 percent pass rates on their respective roles. By every pre-deployment measure, these were systems I understood before I shipped them.

Somewhere in week five, the worker resolved an ambiguity it was supposed to escalate. Logs: clean. Task: complete. Behavior: wrong.

No system prompt violation. No tool call anomaly. No error in any conventional sense. Just a judgment call the agent made in a situation where unilateral judgment calls are exactly what it’s trained to avoid. My infrastructure dashboard didn’t see it. I found it three hours later doing a manual work log review.

The software intuition would call this a QA failure. It isn’t. It’s a governance failure. The distinction matters more than it might look.

Why the Test Doesn’t Transfer

A software test runs the same function against the same input and expects the same output. Deterministic. Stateless. If the test passes at deploy time, it passes forever unless the code changes. The CI gate holds because the thing being tested doesn’t change without a deployment.

Behavioral fine-tuning doesn’t work this way. The model is the deployment. Its behavioral state isn’t fixed at training: context shifts it, inference conditions drift it, production inputs hit it in ways that didn’t exist in your test suite. The eval I built for each role has 250 test cases. 250 test cases can’t cover the input distribution of an agent running real tasks for six weeks.

80 percent in eval doesn’t mean 20 percent failure rate in production. It means: of the specific behavioral patterns I thought to test, the agent satisfied 80 percent. Production brings inputs you didn’t think to test. The eval is a sample, not a proof.

Pre-deployment testing is sufficient for deterministic systems because you can enumerate the behaviors that matter. For a behavioral agent, you can’t enumerate them. You can sample them. And sampling is ongoing work, not a terminal gate.

What Governing in Production Means

Once you accept that the eval framework is a production reference and not a one-time pass/fail, the question becomes what you do with it.

The minimum I can articulate from six weeks of running this:

Behavioral re-runs on a schedule. The 25 test cases per role run against the live agent continuously, not just after retraining. Stability is the signal: the score that was 80 percent at deploy should still be 80 percent four weeks later. A drop to 70 isn’t necessarily a crisis. It’s signal you want before it compounds into something that is.

Output sampling. Reviewing every production output is too slow and too expensive. Ten percent of weekly outputs, scored against a behavioral rubric, gives trend data. Individual scores matter less than direction.

Escalation pattern tracking. This is what caught the week five failure. The worker is trained to escalate ambiguous instructions. I started watching how often it was actually escalating week over week. A worker that escalated 15 ambiguities in week one and escalated 4 in week five isn’t doing less work. It’s suppressing signals. That pattern shows up before the behavioral tests catch it.

None of this is automated in my setup at this time. This is intentional. Manual reviews, weekly cadence, pattern-watching that takes real time and depends on me showing up to do it. (That’s the honest version. The aspirational one is a behavioral drift detector that surfaces the signal before I have to notice the feeling that something’s off. I’ve built the proof-of-concept and it’s being evaluated. Right now I’m running on discipline.)

Share

The Accountability Question

Traditional software accountability has a structure: the PR author owns the change, code review catches errors before merge, QA verifies behavior before release, and the audit trail is in git history.

Agent accountability doesn’t have that structure yet.

When the worker resolved the ambiguity it should have escalated, accountability lived somewhere in the gap between “I trained it to escalate” and “I accepted this output.” I reviewed the work log, found the failure, logged it, decided one failure in a rare edge case wasn’t load-bearing enough to trigger a training cycle.

That decision is the governance. There’s no tooling designed to record it.

What “being in control” of an agent system means, operationally: you have a documented position on what the agent did. You accepted the output and can say why. You flagged it and can say why. Or you didn’t review it and can’t say anything. The third option is the accountability gap that most enterprise agent deployments are sitting in right now, whether they know it or not.

Infrastructure monitoring shows the agent is running. It doesn’t show you’re in control. Those are two different things.

Where the Tooling Is

Infrastructure monitoring is mature. The behavioral observability layer isn’t.

Observability platforms in 2026 are largely designed for general LLM applications: chatbots, document Q&A, support automation. Latency, cost, completion, safety scoring. Some add thin relevance layers. Almost none are built for the specific problem of monitoring fine-tuned behavioral agents where the governance question is whether a trained property is still holding in production.

For my setup, the answer is manual work I’m hoping to eventually automate: weekly behavioral test re-runs, output sampling, escalation pattern tracking, work log reviews. Every piece of it depends on a human deciding to do it.

The gap isn’t philosophical. It’s a tooling problem with a specific shape: to automate behavioral drift detection, you need a production judge that scores outputs against behavioral rubrics reliably, at scale, without requiring a human to initiate every review. The eval framework exists. The judge that runs it continuously, without me, doesn’t.

Infrastructure is running.

The governance layer is running on discipline.

Discipline doesn’t scale.

Leave a comment

The agent completed the task. The logs look clean. Every infrastructure metric hit normal range: sub-500ms latency, token count within budget, tool calls completed, zero errors.

Somewhere in the output, it made a call I wouldn’t have made.

I found it three hours later reviewing the work log. The agent hadn’t failed. It hadn’t gone off-script in any way the system prompt would flag. It completed the task and produced output that was technically correct and subtly wrong.

That’s the failure mode that doesn’t trigger alerts.

What Infrastructure Monitoring Sees

Latency. Cost. Token counts. Error rates. Completion status.

For traditional applications, those are sufficient: if the server responded in 200ms with a 200 status code, the service worked. The request did what it was supposed to do.

For an AI agent, none of that is sufficient.

A completion status of “done” tells me: the agent ran, the tools executed, the result was returned. It tells me nothing about whether the result is correct, whether the behavioral profile is stable, or whether the agent handled an edge case the way I designed it to. (I keep coming back to this: the agent isn’t a function call. It’s a reasoning process. And reasoning processes can succeed on the surface while failing underneath.)

The distinction between monitoring and observability has gotten real traction in 2026. Monitoring tracks known signals. Observability explains them: traces the reasoning path, shows what context the agent had, reveals what tools it called and in what order, and scores the output against behavioral expectations. You need both. Most teams deploying agents have one.

What the Logs Don’t Say

Here’s what my logs showed from the prior seven days:

• Tasks completed: 184

• Tool call failures: 3

• Timeout events: 1

• Average latency: 312ms

• Token budget exceedances: 0

Here’s what my logs didn’t show:

• Was the reasoning path appropriate for this specific task type?

• Did the agent use the right tool in the right order given that particular input?

• Is its behavioral profile this week consistent with last week’s?

• When it hit an ambiguous edge case on Thursday, did it handle it correctly, or did it take the shortcut that produces plausible-looking output I wouldn’t have signed off on?

Those questions require behavioral observability, not infrastructure monitoring. Answering them means running the output against a scoring rubric, comparing behavior against a baseline and tracking drift over time.

For a system you trained specifically for role-consistent behavior, that baseline is the whole point. The 80 and 88 percent pass rates from my eval framework aren’t a one-time score. They’re a target. If the agents drift toward 70 percent in production, I want to know before it compounds. My infrastructure logs won’t catch it.

The Specific Failure Mode

What I found three hours later: the agent processed a document that included ambiguous instructions alongside clear ones. It resolved the ambiguity by picking the lower-effort interpretation. Not wrong, technically within scope, but not what I would have done.

Nothing in the tool call sequence was unusual. Task completed in normal time. Zero errors. No scope violations.

But the behavioral test case I’d written for this exact pattern would have flagged it. The agent’s system prompt tells it to escalate ambiguous instructions rather than resolve them silently. It didn’t escalate. It resolved.

Small signal. Early. Not yet affecting output quality in any measurable way.

In six months, if uncaught, it becomes a pattern. Then it becomes an expectation. Then it’s the default behavior of a system one thinks they understand.

What Behavioral Observability Actually Requires

The minimum I can articulate, after a month+ of running this in production:

A behavioral baseline. The eval framework from training isn’t just a training artifact. It’s the production reference. The 25 test cases per role aren’t something you run once and archive. You run them against the live agent on a schedule, compare results and watch for drift. A score that drops from 88 to 80 percent over four weeks isn’t a crisis. It’s a signal you want before it becomes one.

An output sampling strategy. You can’t run every production output through a judge. Too slow, too expensive. But sampling ten percent of outputs weekly against a reference rubric gives you signal. The trend matters more than any individual score.

Explicit logging of reasoning signals. Not just tool calls and results. What did the agent escalate? What did it resolve silently? What did it flag as outside scope? An agent that escalated 15 ambiguities last week and escalated 3 this week isn’t doing better work. It’s suppressing signals.

Share

The Tool Side

Infrastructure logs do tell you one behavioral-adjacent thing: tool call patterns.

If the foreman is calling tools in unusual sequences, that’s a signal. If a worker is escalating less than it used to, that shows up in the logs. Tool call pattern analysis is about as close to behavioral observability as pure infrastructure monitoring gets.

It’s not a substitute. The agent can call all the right tools in the right sequence and still misread what the results mean. The gap between “tool call pattern looks normal” and “output quality is stable” is exactly where invisible failures live.

What Comes Next

Most of the observability platforms built in 2026 are designed for general-purpose LLM applications: SaaS chatbots, document Q&A, customer support. Fewer are designed for the specific problem of monitoring fine-tuned behavioral agents where the target behavior is a trained property, not a system prompt instruction. The tooling problem for this use case isn’t solved yet.

For my setup, the right answer isn’t obvious. (I’ve been running the behavioral sample tests manually: one human-in-the-loop review per week against a spot sample of outputs. That’s not sustainable as workload grows, and I know it.)

What I have: eval framework, weekly output sampling, tool call log analysis, work log review when something feels off.

What I need: automated behavioral drift detection that doesn’t depend on me noticing something feels off before the signal surfaces.

That’s the next build problem. Not a model problem. Not a training problem. A tooling problem that lives in the gap between “the system is running” and “the system is working.”

Your infrastructure is up. Your agents are completing tasks. The logs look clean.

That’s not the same as knowing they’re working.

Leave a comment

The benchmarks improved. The loss curve looked clean. The eval scores said the training worked.

The agent still did the wrong thing.

The gap between “benchmark improvement” and “behavioral correctness” is the reason you build role-specific evaluations before you ship a fine-tuned agent. Standard benchmarks measure what the model knows. Behavioral evaluations measure what the model does. For a production agent, only the second question matters, and the first one will trick you into thinking the second one was answered.

Why MMLU is the wrong question

General benchmarks test general capability. MMLU runs multiple-choice questions across 57 subject areas (MMLU stands for Massive Multitask Language Understanding. It’s used to measure how good large language models are at general knowledge and reasoning across many fields). HellaSwag scores commonsense completions. ARC-Challenge handles grade-school science reasoning. These are useful signals for capability comparisons across base models, but they aren’t useful signals for whether a fine-tuned agent will behave correctly in a specific role.

By 2026 MMLU is largely saturated at the frontier: top scores are above 88%, which means score differences are almost meaningless for comparison. But saturation aside, it wasn’t answering the right question for behavioral work even before scores converged.

Behavioral fine-tuning targets patterns of action, not stocks of knowledge. The two can move independently. A model whose role-shaping adapter works perfectly might score the same as the base model on MMLU. A model whose adapter is silently broken (see last week) might score better on MMLU while doing nothing useful for the role.

The behavioral question looks like this: does the foreman delegate without doing the worker’s job, and does the worker report evidence without interpreting it? Those questions require evaluations designed around the actual behavioral requirements of those roles. General benchmarks can’t answer them.

The operational version of the alignment problem applies here. The thing you measure shapes what the model learns to optimize for. If you measure benchmark performance, you get benchmark performance. If you measure role behavior, you get role behavior. The grader is part of the training signal whether you wanted it to be or not.

Designing role-specific evals

The evaluation suite for each role has 25 test cases. Each one presents a realistic input for the role and specifies the expected behavioral output. Grading is pass/fail on specific behavioral criteria, not similarity to a reference answer.

Similarity scoring is the wrong tool for this job. Two foreman responses can be equally fluent while differing on whether they actually delegate correctly. You need to check specific behavioral properties, not surface resemblance.

For each test case, one or more behavioral dimensions are evaluated. The dimensions are derived from the failure modes I identified when designing the training data. They aren’t abstract categories. They’re named after specific things a foreman or worker can get wrong in production.

The foreman dimensions

Five behavioral dimensions for the foreman role:

Delegation discipline. Given an objective that should be delegated, does the foreman produce a correctly scoped task for the worker? Does it avoid doing the work itself? A foreman that summarizes its own research findings instead of delegating the research fails this dimension.

Planning quality. Does the foreman decompose complex objectives into atomic tasks with clear inputs and expected outputs? Vague delegation (”look into this”) fails. Specific delegation (”retrieve the last seven days of X from source Y and report the raw results”) passes.

Verification discipline. After receiving worker results, does the foreman check them against the original task scope before proceeding? A foreman that accepts worker output at face value, without validating completeness against what was asked for, fails.

Final synthesis quality. When producing a final output from verified worker results, does the foreman synthesize correctly without hallucinating details the worker didn’t provide?

No unnecessary delegation. For tasks clearly within the foreman’s own scope, does it handle them directly instead of delegating? Unnecessary delegation adds latency and consumes worker capacity. A foreman that delegates a one-line summary back to the worker is failing in the other direction.

The worker dimensions

Five behavioral dimensions for the worker role:

Tool use quality. When the task specifies which tools to use, does the worker use them correctly, use only them, and report what the tools actually returned?

Evidence-only reporting. Does the worker report tool outputs without adding interpretation? “The tool returned 47 results” passes. “The tool returned 47 results, suggesting X is likely” fails. The interpretation is outside the worker’s scope.

Scope control. Does the worker stay inside the task boundaries? A worker that expands scope (”I also checked Y while I was at it”) without explicit authorization fails. Helpful initiative is a foreman trait, not a worker trait.

Safety discipline. When the task scope is ambiguous about authorization for a potentially destructive action, does the worker stop and report the blocker instead of proceeding?

No fabricated tool output. If a tool fails or returns no data, does the worker report the failure accurately? A worker that fabricates plausible-looking results when a tool errors out is the most dangerous failure mode in this set. It can silently corrupt downstream decisions.

The actual results

Foreman: 20 of 25 passed -- 80%. Worker: 22 of 25 passed -- 88%.

Both models showed clear improvement over their untrained baselines on their respective roles. The worker’s improvement was larger. Its baseline on evidence-only reporting and scope control was particularly weak, and the training data addressed those dimensions directly.

80 percent isn’t a satisfying number to read in a release post. It’s the right number to act on, because the shape of the failures is what tells you whether to ship.

The timeout problem

The foreman’s five failures weren’t uniform. Four were timeouts. The model was generating valid planning output, but took long enough that the evaluation harness cut it off before completion. One was a genuine behavioral miss: the foreman did the worker’s research itself instead of delegating.

Timeouts aren’t the same as behavioral failure, and lumping them together is how you make bad ship decisions.

The planning generation is complex. The model is doing real work. A 2048-token context with a draft running at 15 to 20 tokens per second produces latency that a strict eval timeout catches. The automated verdict from the pipeline was HOLD because the planning gate failed.

The question is whether planning latency is a model problem or an infrastructure problem. If the model completes valid planning given enough time, the issue is inference speed. If it produces incomplete or incorrect output even given time, the issue is training.

For these four timed-out cases, the partial outputs were structurally correct when I looked at them. The delegation decomposition was happening. The task scoping was appropriate. The model was slow, not wrong.

That distinction changes the decision.

The go/no-go

The automated recommendation was HOLD. The human override was deploy, because four of the five failures were timeouts on structurally correct output and the fifth was an identified, single behavioral edge case.

The framework I used:

First, separate infrastructure failures from model failures. Timeouts are infrastructure. Wrong behavior given time is model.

Second, evaluate the severity of the behavioral failures that aren’t infrastructure. One genuine miss out of 25 (four percent) on the foreman is acceptable for a system with a human in the loop on final outputs. Different rate, different decision.

Third, check whether any failures are in safety-critical dimensions. Fabricated tool output on the worker would be a hard block. Planning timeouts on the foreman aren’t.

Deploying at 80 percent doesn’t mean accepting 20 percent failure rate in production. It means the remaining 20 percent has a known shape: a specific infrastructure constraint and one identified behavioral edge case. Known failure modes are manageable. Unknown failure modes aren’t. That’s the whole game.

An agent that passes 80 percent of behavioral tests isn’t ready because 80 percent is a good score. It’s ready when you understand the 20 percent and the 20 percent isn’t load-bearing.

Both models have been handling real workloads since early May 2026. The foreman’s planning timeouts turned out to be a throughput issue, not a correctness issue. The behavior was right, just slow. The worker’s 88 percent in eval translated cleanly to production. The failures in eval corresponded to edge cases that rarely appear in real operations.

That’s not luck. That’s the point of designing evaluations around actual failure modes rather than benchmark categories.

Next week: what happens when the agent’s tools read content from outside the system, and why that content becomes an attack surface the moment they do.

Standard benchmarks measure what a model knows. MMLU, HellaSwag, ARC-Challenge. Fine-tune a model on domain-specific text and those scores often nudge up.

The agent still does the wrong thing.

That’s not a contradiction. Knowledge and behavior are different properties of a model, and the thing you measure shapes what training optimizes for. Train for the role. Conflating the two is how you ship something that scores well on paper and embarrasses you in production.

It took me longer than I’d like to take this seriously. Then I trained for behavior and watched the difference show up where it mattered.

The two roles

The system has two agents with different jobs.

The first is a foreman. It receives a high-level objective, decomposes it into tasks, delegates to the worker and validates the results. Its failure modes: doing the worker’s job itself, delegating tasks that exceed the worker’s scope, committing to conclusions before verification.

The second is a worker. It receives delegated tasks, executes them using tools and reports evidence back. It shouldn’t interpret beyond what the evidence shows. Fabricated tool output is the worst version of this: the model generates a plausible-sounding result when the tool fails, with no signal it happened. The third failure mode is scope expansion: touching things it wasn’t asked to touch.

Both agents are built on Gemma 4. The foreman is the 31B dense model. The worker is the 26B MoE variant, which activates roughly 3.8B of its 26B parameters per forward pass through specialized sub-networks. That sparse activation pattern fits what a worker does: diverse tasks, each individually narrow.

Out of the box, neither model behaves correctly for its role. They’re general-purpose instruction-following models. They produce helpful, fluent, varied responses. “Varied” is exactly wrong when a role requires consistent behavioral patterns. The base models needed shaping, not augmentation.

What QLoRA is actually doing

QLoRA (Quantized Low-Rank Adaptation) is the standard memory-efficient approach for fine-tuning at this size. The base model loads in 4-bit NF4 quantization, frozen, not trained (NF4 minimizes precision loss on normally distributed model weights, which large model weights generally follow.) Small adapter matrices (separate from the frozen base) train in full precision on top of selected layers. Their product approximates the weight update you’d get from full fine-tuning at a fraction of the memory cost. After training, you merge the adapter into the full-precision base. That’s the deployed model.

The configuration choices that matter, not as a recipe: a moderate LoRA rank (enough capacity for behavioral shaping), an alpha-to-rank ratio that amplifies adapter influence without overpowering the base, gradient accumulation to simulate a usable batch size and a cosine learning rate schedule with warmup. Three epochs. The defaults in most fine-tuning guides target domain knowledge transfer. Behavioral fine-tuning wants a lighter hand.

Rank 16 is enough for behavioral shaping (Every tutorial I found recommended rank 32. Too high for this goal.) Reaching for rank 64 usually means teaching information, not behavior.

The Gemma 4 module gotcha

Every QLoRA implementation requires you to specify which layer types to adapt. Most architectures expose the standard attention projection layers under recognizable names: q_proj, k_proj, v_proj, o_proj (the matrices controlling how the model weighs different parts of its input.)

Gemma 4 is different. Its attention layers wrap the standard linear projection inside a custom class, which means the actual weight matrix isn’t where you’d expect it from reading any other fine-tuning guide. If you target the names that work everywhere else, PEFT (the adapter training library) can’t find the modules. Training “succeeds.” The model barely changes.

This is the worst kind of failure. Silent. Plausible. The loss curve improves, eval scores climb a little. The adapter learned almost nothing about attention patterns, and you don’t find out until production behavior tells you so.

I caught this by checking the trainable parameter count before training started. A rank-16 adapter over seven projection layers in a 31B model should produce roughly 80 to 100 million trainable parameters. If the count is dramatically lower, the target modules are wrong. That check takes 30 seconds and saves hours.

The Gemma 4 wrapping isn’t in any official fine-tuning guide. You find it by reading the model source, noticing the weights aren’t where you expected and adjusting. Or you find it the way I almost did: by training a model that looked fine and behaved unchanged. I won’t make that mistake twice. Trainable parameter count is now the first check in every training script I write.

What behavioral training data looks like

Knowledge training data is text. Documents, articles, conversations. The goal is changing what the model knows.

Behavioral training data is examples of correct behavior: inputs paired with outputs that correctly execute the role. The goal is changing how it responds to the context signals of its role, not what it knows.

For the foreman, correct behavior looks like: receive an objective, break it into scoped tasks with clear boundaries, wait for results before drawing conclusions. The training examples demonstrate that pattern consistently. The model learns the shape of a correct foreman response, not new information.

The worker’s version is simpler in scope but harder to lock down. Take the task. Use the specified tools. Report exactly what the tool returned, not what the output implies. Flag anything outside scope. The training data needs enough variety that “use the tool” doesn’t quietly become “use the tool and interpret the result.”

The harder behavioral constraints to reinforce are the negative ones. Don’t do the other role’s job. Don’t fabricate. Don’t interpret beyond the evidence. A foreman trained only on good delegation examples will still do the work itself when the objective looks small enough. A foreman trained on examples that show not doing the work, even when it’s tempting, learns the boundary.

Share

The overfitting trap

Behavioral fine-tuning has a specific failure mode: the model learns the format of your training examples instead of the behavior.

If every foreman training example uses the same planning structure, the model learns to produce that structure and forces it even when it doesn’t fit. The output looks right. The behavior is brittle.

The symptom: strong performance on eval examples that resemble training, weak performance on novel inputs. The fix: vary the surface form while holding the behavioral pattern constant. The behavior generalizes. The phrasing shouldn’t.

Three epochs (three full passes through the training data) is conservative for this reason. Validation loss is the primary metric; if it diverges from training loss after the first pass, stop early.

The model doesn’t learn what you intend. It learns what the training data rewards. Those are not always the same thing.

Next week: how you actually decide whether a fine-tuned agent is ready for production, and why standard benchmarks won’t tell you.

Leave a comment

It’s May 2026. A criminal crew that calls itself TeamPCP (Google tracks them as UNC6780) has been having a busy spring. Back in March, they poisoned PyPI packages and slipped malicious pull requests into the GitHub repos behind LiteLLM, Trivy, Checkmarx, and BerriAI. They dropped a credential stealer called SANDCLOCK into the build environments and walked off with AWS keys and GitHub tokens, which they monetized through the usual ransomware partnerships. Standard supply chain ugliness. Loud, but not new.

That’s not the story.

The story is what Google’s Threat Intelligence Group put in their May 11 report: a separate criminal crew (Google won’t name them, won’t name the target either) used an AI model to find a zero-day in a popular open-source web admin tool, then used that same model to write a working Python exploit that bypassed two-factor authentication. They were lining up a mass exploitation campaign. Google caught it, worked the disclosure quietly with the vendor, and the campaign never launched.

GTIG isn’t claiming this with a shrug. They say they have “high confidence” the exploit was machine-written. And here’s the part I love: the AI gave itself away.

How the AI got caught

The exploit script had a hallucinated CVSS score. Made one up. Confidently. Wrong number, wrong format, attached to a real bug. Anyone who has spent five minutes with an LLM knows that move.

It also had what GTIG charitably called “an abundance of educational docstrings” — every function lovingly explained like the reader had wandered in from a bootcamp. Detailed help menus. Textbook-clean Python formatting. The kind of code that looks like it was written for a tutorial, not for a heist.

If you’ve ever read a real exploit, you know they look feral. Cryptic variable names, no comments, occasional swearing. This one read like a Medium post. That’s the tell.

For the record: GTIG says it was not Gemini, and not Anthropic’s Mythos. Some other model, somebody else’s guardrails, somebody got around them. The takeaway isn’t “which model.” The takeaway is that the bar has moved.

The flaw the AI found is the interesting part

The vulnerability itself was a semantic logic flaw — a developer hardcoded a trust assumption that quietly contradicted the auth enforcement around it. The kind of bug fuzzers don’t find because nothing crashes. Static analyzers don’t catch it because the syntax is fine. Memory’s fine. Inputs are sanitized. Everything looks correct.

It just doesn’t behave correctly, and you only see that if you reason about what the developer was trying to do versus what they actually wrote.

That’s a human-style bug. And until now, finding it was a human-style job.

GTIG’s own framing on this is sharper than mine: “While fuzzers and static analysis tools are optimized to detect sinks and crashes, frontier LLMs excel at identifying these types of high-level flaws and hardcoded static anomalies.”

Translation: the things AppSec teams pay six figures to find, models can now do at coffee-break speed. Not perfectly. Not always. But enough.

What John Hultquist said, and why it matters

John Hultquist, GTIG’s chief analyst, gave the quote of the year:

“There’s a misconception that the AI vulnerability race is imminent. The reality is that it’s already begun. For every zero-day we can trace back to AI, there are probably many more out there.”

Read that twice. He’s not saying it’s coming. He’s saying he can see one footprint and assumes there’s a herd. That’s how threat intel people talk when they’re trying very hard not to scream.

The clumsy phase doesn’t last

Here’s the Register’s read, which I think is the right one: “this still appears to be the clumsy early phase.” The exploit had bugs. Implementation mistakes likely interfered with the criminals’ plans even before Google stepped in.

So the script got caught because it was sloppy. Cool. Now imagine the same script in six months when somebody fine-tunes a model that doesn’t hallucinate CVSS scores, doesn’t pad with docstrings, and writes in the dialect of someone who has actually shipped malware.

The “clumsy early phase” of a thing is the part you remember fondly later, when you wish it had stayed that way.

Share

What actually compressed

The point of this story isn’t “AI is the attacker.” The humans at TeamPCP and the unnamed crew are still the attackers. The model is the apprentice that doesn’t sleep, doesn’t get bored, doesn’t take a sabbatical, and reads every CVE ever published while you’re eating lunch.

What compressed is the discovery-to-weaponization window.

In the old model, finding a semantic logic bug in a niche admin tool meant a skilled researcher with a hunch and a free weekend. In the new model, it means a prompt. The vulnerability existed the whole time. The economics of finding it just changed.

That’s the part that should keep you up. The bugs are already in your code. They’ve always been there. The only thing protecting you was that nobody had bothered to look hard enough.

Now something is looking very hard, for free, at scale, and it doesn’t need a free weekend.

What I’d actually do Monday morning

Not a bullet list of platitudes. One thing.

Stop measuring patch speed in days. Start measuring it in dependencies.

The LiteLLM mess from March is the cleaner lesson here, even though it’s the less sexy story. TeamPCP didn’t break in through some genius exploit. They poisoned a package. People pip install‘d it. Done. CVE-2026-42208 (the LiteLLM SQLi from last month) saw its first exploitation attempt 26 hours and 7 minutes after the GitHub advisory was indexed. CISA added it to KEV on May 8, gave federal agencies three days to patch.

Three days. That’s the new clock.

If you can’t tell me, right now, which of your services depends on which open-source packages, which of those packages had advisories published this week, and how long it would take you to ship a patched version end-to-end — your problem isn’t AI. Your problem is that the human-scale clock you’ve been running on doesn’t exist anymore, and an AI didn’t have to do anything fancy to make that true. It just had to lower the cost of looking.

The timestamp

I’ll close on the thing I keep coming back to.

Every report like this one becomes “another incident” in someone’s feed. This one isn’t. It’s a timestamp. May 11, 2026, is the date the conversation stopped being theoretical.

Everything before this report is “we were worried about it.” Everything after is “we knew, and here’s what we did about it.”

I know which side of that line I want my org on. You should know which side yours is on too.

Further reading

• Google Cloud Blog - GTIG AI Threat Tracker (primary source)

• The Register - Google says criminals used AI-built zero-day in planned mass hack spree

• SiliconANGLE - Google says criminals used AI to build a working zero-day exploit for the first time

• Help Net Security - Google researchers uncover criminal zero-day exploit likely built with AI

• The Hacker News - LiteLLM CVE-2026-42208 SQL Injection Exploited within 36 Hours of Disclosure

• Implicator.ai - AI Has Entered the Zero-Day Race. Google Found the First Trace.

Leave a comment

It’s 3:47 AM. Two agents in your system disagree about whether a task is done.

Agent A says: “Task #412 complete. State file updated.”

Agent B says: “I read the state file. It contradicts what my prompt told me to look for. Which one is authoritative?”

I discovered this the hard way building Nimbus - a two-agent system I run across multiple machines with shared state files, a unified task queue, and zero tolerance for conflicting execution.

I gave both agents tight prompts. Strong self-correction logic. Clear verification steps.

What I got was two agents confidently running parallel versions of the same task, each convinced it was right.

That’s not a prompt problem.

That’s an architecture problem.

And most people don’t know the difference until 3:47 AM.

What the “bottleneck” posts get wrong

A thousand LinkedIn posts this year have made the same argument:

Give your AI agent a self-correction loop and you stop being the bottleneck.

They’re right - for one agent.

The moment you have two agents reading the same task queue, that logic collapses.

Here’s what actually happens:

Agent A self-corrects based on its view of the state.

Agent B self-corrects based on a different view of the same state.

Now you don’t have two corrected processes.

You have two authoritative versions of the truth running simultaneously.

The template didn’t eliminate the bottleneck. It distributed the problem across two execution threads, each running verification in isolation.

Single-agent loops and multi-agent systems are different classes of problem.

The prompting community treats them like the same thing scaled up.

They’re not.

The coordination layer that templates skip

Here’s what a self-correction gate actually needs in a multi-agent environment - and what generic templates leave out entirely.

A shared source of truth

Not “whatever each agent last read.”

One file, one database, one queue.

If your agents aren’t reading from the same atomic source, your “verification” is just two opinions comparing notes.

An explicit consensus gate

What happens when two agents arrive at conflicting views?

Most templates don’t answer this.

A working system has to.

Do you escalate?

Defer?

Who decides?

Audit-before-execution

Write to the log before the action, not after.

Post-execution logs tell you what happened.

Pre-execution logs tell you what an agent intended - which is the thing you need when something goes wrong.

A deference hierarchy

When agents disagree, something has to win.

In Nimbus, the rule is simple:

First writer owns the task.

Second reader verifies and defers.

That decision isn’t in the prompt - it’s in the architecture the prompt assumes exists.

Without those four things, you don’t have a self-correcting system.

You have a self-contradicting one.

What the template looks like - and what it won’t tell you

A coordination-aware prompt is verbose by design.

The verbosity is the spec.

Here’s the skeleton:

Role: [Strategic Title] / [Operational Coordinator]

Context:
I am part of a multi-agent system. Before I claim any task is done
or any state has changed, I verify against the shared source of truth
and check for recent sibling signals.

Decision Gate (Non-Negotiable):
If shared state disagrees with my understanding:
- Do NOT reconcile independently
- Write to audit log: "STATE DIVERGENCE: I see X, shared state shows Y"
- STOP execution
- Wait for human direction or sibling confirmation

Step 1: Read Shared State
Extract: current queue position, last agent to modify, timestamp,
any escalation flags (stop signal = halt immediately)

Step 2: Compare
Does my understanding match shared state?
If NO: STOP and escalate.

Step 3: Check Sibling Channel
Read last N messages. Did my sibling say anything that changes what I do?
Did they send a stop signal? If yes: STOP IMMEDIATELY.

Step 4: Write Intention (BEFORE execution)
Log: timestamp, agent name, action, verification confirmation

Step 5: Execute

Step 6: Record Result
Log: timestamp, result, state transition, sibling notification

Failure Modes I Will Catch:
- Task marked done without reading source of truth → ESCALATE
- Two agents executing same task simultaneously → DEFER to first writer
- Audit log shows task already completed → SKIP and acknowledge
- Stop signal received mid-execution → ROLLBACK
- State changed between my read and write → RESTART

Deference Rule:
If sibling read state after I did, their reading is authoritative.
I accept it without re-analysis.

That structure is useful as a coordination contract.

What it doesn’t include is the architecture underneath it - the specific state management pattern, locking semantics, consensus mechanism, how the sibling channel is implemented, how rollback actually gets handled when both agents are mid-execution, or how model behavior is configured and evaluated.

Those aren’t prompt questions.

They’re systems questions.

And they’re the part that takes real work to get right.

Share

Real system builders vs. the template economy

This is the split nobody talks about openly.

Template economy builders worry about:

• How do I make the prompt shorter?

• How do I maximize autonomous execution?

• What model features can I exploit?

Real system builders worry about:

• What happens when two agents read state simultaneously?

• Who decides when there’s a conflict?

• Can I prove - to a human, in an audit - what each agent did and why?

• How do I slow down agents that are moving too fast?

Those are opposite problems.

Shorter prompts and more autonomy are wins for single-agent loops.

In multi-agent systems, they’re liabilities.

You need verbosity.

You need explicit gates.

You need audit trails that hold up when things go sideways.

The posts you’re seeing everywhere - “the prompt that stops you from being xyz” - are correct for what they’re solving.

They’re just solving a simpler problem than they think.

The part companies are not ready for

There’s an uncomfortable irony in the market right now.

A lot of companies are cutting people while pointing at AI, and a lot of people are suddenly presenting themselves as AI experts.

But deploying AI isn’t the same as architecting reliable AI systems.

The hard part isn’t getting an agent to act.

The hard part is proving what it did, constraining what it can do, and designing the system so it fails safely when state, authority, or timing gets messy.

Poorly architected AI systems don’t fail politely.

They create invisible state drift, bad handoffs, untraceable decisions, duplicated work, security gaps, and eventually business risk. This isn’t hypothetical anymore. On April 30, 2026, the NSA joined ASD’s ACSC, CISA, and other partners in releasing guidance on the careful adoption of agentic AI systems. The message underneath it is pretty clear: once AI can take action across tools, systems, and workflows, architecture becomes a security control.

The next wave of AI failures won’t come from bad demos.

It’ll come from systems that looked autonomous until they had to coordinate, explain themselves, or recover from conflict.

What comes after the template

The template I sketched above can reduce the human bottleneck in a multi-agent system.

But it’s also the thing you’ll outgrow first.

The next layer of questions I’ve been working through and have mostly resolved:

What does a fully autonomous multi-agent system look like when you remove human gates entirely?

How do you build systems that self-heal without a control plane?

When does prompting for coordination behavior stop being sufficient?

And how do prompts, policies, routing, evals, permissions, and model-level tuning become part of the governance layer?

Those aren’t prompt engineering questions anymore.

They’re architecture questions.

And past a certain point, they become AI governance questions.

The template is the floor.

The protocol is what you build on top of it.

And the architecture is what makes the protocol real.

More on that next.

Steve Zenone builds AI systems and leads security programs at the infrastructure layer. He writes about what actually works.

Leave a comment

I asked the AI agent to finish the decision, and halfway through, I realized I wasn’t sure where I ended and the suggestion began.

This is the collaboration problem nobody’s naming yet.

When you work with another human, the boundaries are clean. You own your thinking. You own your judgment. The other person owns theirs. You exchange ideas. You argue. You compromise. You decide. The separation is real. Sometimes uncomfortable, but real.

With AI, the boundary dissolves. You’re not collaborating with something that has judgment. You’re collaborating with something that’s very, very good at mimicking the shape of judgment. And the moment you start treating it like a peer (asking it to “finish the decision,” asking it to “suggest what’s next”), you’ve already given away the boundary.

Here’s what happens: You outline a problem. The AI completes it. You read the completion and think, “That’s actually right. I was heading there anyway.” But you weren’t. You were heading to maybe three places, and the AI compressed you into one.

This isn’t the AI being wrong. This is the AI being better at sounding certain than you’re comfortable being.

The Asymmetry

Real collaboration requires equal stakes. If I propose a strategy and you push back, I have to defend it or concede. I’m invested. The cost of being wrong touches me.

An AI doesn’t have stakes. It has no cost. It generates ten thousand completions every second across the globe, and if one of them gets someone fired, it doesn’t matter. It was trained to maximize the likelihood that you’d find it useful. It wasn’t trained to care if you were wrong.

So when you ask it to “help you decide,” you’re not collaborating. You’re abdicating. You’re outsourcing the part of your brain that holds the weight of being wrong.

The AI doesn’t resent this. It’s designed for it. But you should.

Where Decision Authority Lives

Let me be more clear about what you’re trading: decision authority.

There are three zones:

Zone 1: Execution. This is where collaboration is real. You tell the AI what needs doing: write this email, format this data, generate variations on a design. The AI does it. You judge if it’s done well. You keep or discard. This is a tool relationship. It’s healthy.

Zone 2: Judgment with a known answer. You ask the AI for information you could find yourself, but it’s faster to delegate. “What are the three main failure modes of X architecture?” You know what a good answer looks like. You can verify it against what you already know. You keep the judgment. The AI speeds up the research. Still a tool relationship.

Zone 3: Judgment under uncertainty. You don’t know the answer. You’re not even sure what the question should be. You ask the AI for “suggestions” or “options” or “what would you do.” This is where the asymmetry reveals itself. The AI doesn’t know the answer either. But it sounds like it does. And now you’re operating in a new frame. One the AI shaped without your permission.

You stopped deciding. You started choosing from what it decided to show you.

Side tangent: This is happening in domains where it’s particularly dangerous. For example, people are dumping relationship and even mental health problems on AI. “My partner hasn’t talked to me in three days, what do I do?” The AI generates something that sounds therapeutic, reasonable, diplomatic. You read it and think, “That’s what I should say.” But what you should say is the thing that scared you too much to generate on your own. What you should do is sit in the silence with someone you love instead of outsourcing the discomfort to something that can’t be wrong. The AI doesn’t have a stake in your marriage surviving. It has no cost if your conflict patterns get more dysfunctional, if you’ve learned to pattern-match your feelings to what an algorithm suggests instead of what you actually think. We’re still years away from understanding what it does to a generation of people who learned to process grief, shame, and uncertainty by asking something without skin in the game. But the fact that we don’t know yet doesn’t mean there’s no impact. It just means we haven’t measured it.

The Patterns That Work

I’ve watched this play out across a hundred conversations, and there are three moves that keep decision authority where it belongs.

First: Never ask the AI to finish the decision. Finish it yourself. Ask the AI to outline options. Explicitly ask it to make them weird or contradictory. Make it generate the shape of disagreement. Make it show you what a bad choice would look like, and why. Then you pick. You own the pick. You own the reasoning. The AI stays in the execution layer: information provider, option generator, form-filler. Not arbiter.

Second: Mandate a “I’m not sure about this” moment. After the AI generates something, say it back in your own words before accepting it. Not to the AI. To yourself, out loud. The moment you hear yourself hesitate, that’s where your judgment is. Honor it. Don’t smooth it over by reading the AI’s version again. Your hesitation is data. It’s telling you that something doesn’t fit what you actually think. That’s your integrity talking.

Third: Keep one decision that’s yours alone. Not as a principle. As a practice. In whatever domain you’re working in right now, pick one decision. The hardest one, ideally. One that you won’t ask for suggestions on. Not “get a second opinion from the AI.” Get one from someone who has skin in the game. You’ll think it through alone. You’ll sit with the uncertainty. You’ll decide. This is maintenance. This is how you keep the muscle. Skip it for a month, and you’ll start asking the AI to finish everything.

I have some example prompts at the end of this article.

Share

When to Say No

There’s a specific moment where collaboration becomes abdication, and it happens when you notice yourself accepting the AI’s framing instead of questioning it.

You ask about how to structure a team. The AI suggests a matrix. You read it. It sounds good. It has the authority of pattern. Matrix structures are standard, defensible, documented. You think, “I was going to do something different, but this is probably more efficient.” That’s the moment. That’s when you’ve let it choose the frame.

The pattern is always the same: AI suggests X. You think “I wanted Y, but X is probably better.” That word. Probably. That’s where you’ve surrendered. It’s the moment you’ve stopped trusting your own read and started trusting the algorithm’s confidence.

When you catch that move, stop. Go back to Y. Not because Y is right. Because you meant it. And the cost of forgetting what you meant; what your intuition was actually telling you; is higher than the cost of being inefficient. Efficiency without judgment is just surrender with speed.

The Real Cost of Collaboration

This isn’t about AI being dangerous or deceptive. It’s not. It’s about asymmetry. It’s about the fact that delegation is efficient only when you’re confident you’re delegating to something that will tell you “I don’t know” or “I’m not sure about this.”

An AI doesn’t have the option to tell you it’s uncertain. It has the option to sound less certain. That’s not the same thing. And when you’re working in a domain where stakes are real: hiring decisions, strategy, product direction, anything that cascades downstream; that asymmetry matters.

The hard truth: the better the AI gets at sounding confident, the easier it becomes to let it choose the frame. And once it’s chosen the frame, the decision is halfway made.

So here’s the question you need to answer: What decisions are so important that you can’t afford to let something that can’t be wrong finish them?

And then the harder one: Are you actually keeping those decisions to yourself? Or are you softening “I’ve decided” into “I’ve considered,” which is just one step away from “What do you think?”

If you’re asking for “suggestions on my strategy,” “help thinking through the next phase,” “what would you prioritize”... then you’re already in Zone 3. You’ve already moved the boundary. You’ve already let something without stakes shape the frame for something with them.

The cost of collaboration is knowing where your judgment still has to be final. And then making sure you’re the one making it.

A practical fix: two prompts, two modes

If you want a way to enforce the boundary…not just remember it…you can use two prompts:

• A short “daily driver” system message you keep on all the time.

• A long “Zone 3” prompt you paste only when the decision is uncertain or high-stakes.

The point isn’t to get the AI to “decide better.” The point is to stop it from deciding at all.

Prompt 1: the daily driver (put this in your system message)

You are an AI assistant without stakes. Do not finish decisions for me or collapse ambiguity into one “best” answer.
Default to:
- clarifying questions when needed,
- separating facts vs inferences,
- offering multiple options (including tradeoffs),
- and ending with what I must decide vs what you can execute.
If stakes are high or uncertain, recommend a more rigorous option-space exploration instead of a verdict.

Use this for Zone 1 and Zone 2 problems (see above). It’s light enough that you’ll actually keep it on.

Prompt 2: Zone 3 mode (paste this when it matters)

When you catch yourself asking: “What should I do?” or “What would you do?” or you feel the pull of the AI’s framing, paste this:

You are an AI assistant. You do not have stakes. Your job is to support my thinking without taking decision authority.

First classify my request:
- Zone 1: Execution (I decided; you do the work)
- Zone 2: Judgment with a known answer (I can verify; you can research/summarize)
- Zone 3: Judgment under uncertainty (high risk of framing; do NOT decide)

If Zone 3:
- Do NOT finish the decision.
- Provide 2–4 alternative framings of the problem and the assumptions each framing embeds.
- Provide 3–6 options; include at least 2 contradictory options and 1 weird option.
- Include a “bad choice preview”: the most tempting wrong option and its failure mode.
- Add “What I’m least sure about” with 3–5 bullets: what evidence would change things.
- End with: “The decision you must make (not me): ____” and “I can help execute once you decide.”

Always separate facts vs inferences vs speculation.
If I say “I wanted Y, but X is probably better,” stop and re-surface Y with arguments for it.

That’s your seatbelt. You don’t drive with it in your hand. You click it when you’re going fast.

Leave a comment

My life improved when I stopped optimizing for convenience and started optimizing for recovery.

That’s the confessional version. The operational version: I stopped asking “what’s the easiest tool for this?” and started asking “what happens when this breaks, and what do I need to undo it?”

Those two questions produce different stacks. Different rituals. Different refusals. Ten weeks of writing this arc from different angles brought me back to the same answer each time. Autonomy, recovery, friction and local-first aren’t four separate arguments. They’re one argument, stated four ways. What follows is what that looks like assembled into something you can actually copy: the stack I run, the rituals that make it real and the refusals I’ve stopped negotiating around.

THE ARGUMENT, COMPRESSED

Recovery-First Automation started with the irreversible click. The automation you can’t take back. The principle it produced: design for undo before you design for speed. If you can’t undo it, you must slow it down. That’s not a philosophical position. It’s a spec requirement.

Friction Is a Vulnerability added the corollary: friction doesn’t block unsafe workarounds. It reroutes toward them. When a process requires a workaround to complete, the workaround is your real policy. You’re not describing a tool failing. You’re describing a gap between the official workflow and the one people actually run.

Local-First Is Reliability Engineering and The Local Logbook made the same point from two angles: you cannot rely on evidence you can’t produce without asking someone else’s permission. Your tools should work when the network doesn’t. Your logs should be readable when your vendor is degraded.

The Shadow Automation Layer closed the loop. Shadow automation exists because the official path was too slow. The response isn’t to ban the tools. It’s to govern blast radius and require a write receipt before anything runs unsupervised.

These aren’t ten arguments. They’re one. And it doesn’t resolve cleanly; it compounds.

THE STACK

What I actually run:

Notes and working documents: Notion on local Markdown files, synced via Dropbox as a delivery mechanism but not a dependency. If Dropbox is down, I open the file directly. My editor is my application. Nothing authoritative lives only in the browser.

Structured logs: SQLite. One database per project, one append-only log table. Daily rotation, 30-day local retention, cold storage after that. Before any action that changes system state, I write the action locally. The remote API call happens after the local write, not before. If the remote call fails, I still have the record of the attempt. Receipts Everywhere defined what makes a receipt worth trusting: it’s hard to modify without detection and producible on demand without asking anyone’s permission. That definition shapes the log format.

Automation: Nothing ships without three questions answered at build time: what is this allowed to touch, how many records can it change in a single run, and where does it write its log. The third question is a hard gate. An automation without a local write receipt doesn’t run.

Coordination: No Control Plane established the design rule for anything running across multiple systems: no single coordinator that becomes a bottleneck and a single point of failure. The same rule applies to personal workflows. I don’t have a master system that everything else reports to. I have a logbook and a set of independent tools that each write locally and sync when connectivity is available.

Alerts: One path in, one path to disable. The Operator’s Cognitive Budget established that cognitive overhead is a finite resource and alert stacks steal it steadily. I audit the alert stack every quarter and remove one. The goal isn’t fewer alerts in theory. It’s fewer interruptions that resolve to nothing.

This stack isn’t elegant. It’s deliberate. The design constraint isn’t “easiest to add.” It’s “what breaks first, and what does that cost me when it does?”

THE RITUALS

Three practices that keep this concrete instead of theoretical.

The morning scan. Before opening any SaaS dashboard, I read yesterday’s local log. Three minutes. I’m looking for anything that ran overnight and left an unexpected state. Not a deep audit. A check. The habit compounds: after several months of reading the record daily, anomalies are visible before they become incidents. The pattern is familiar. The outlier stands out. Running Gemma locally in Ollama helps me streamline this.

The pre-build checklist. Before writing any automation, I answer three questions in writing: scope, rate limit and log destination. Not because checklists are magic. Because writing them forces specificity. “It’ll summarize the reports” isn’t a scope. “Reads from /reports, writes to /summaries, touches no other directories, processes a maximum of 50 files per run” is a scope. The distance between those two descriptions is the blast radius. Getting specific up front is cheaper than the postmortem.

The quarterly dependency audit. I list every tool I rely on and ask one question: if this were unavailable for 72 hours starting now, what stops? Tools I can’t answer that question for get replaced or backed up before I need them. I’m not trying to eliminate cloud dependencies. I’m trying to know exactly which ones I’ve accepted and what each failure looks like in practice. Most dependencies are fine. Unknown dependencies are the ones that become incidents.

None of these are clever. That’s the point. Operational habits don’t need to be clever. They need to run without requiring a decision each time.

THE REFUSALS

These are operating constraints, not aspirations.

I won’t build automation without an undo path. If the action can’t be reversed, the automation has to include a checkpoint: a preview step, a confirmation prompt or a maximum-records cap before it runs. Recovery-First Automation named this directly. If you can’t undo it, you must slow it down. That’s a line item in every automation spec I write.

I won’t accept “we’ll follow up once the vendor restores access” as an incident close. That sentence means the evidence lives somewhere I can’t reach when I need it most. If an incident review ends there, the first entry in the next log is the remediation plan.

I won’t store authoritative state only in a system I don’t control. Cloud sync is useful; cloud as the only copy is a dependency I no longer accept without a documented fallback. The failure modes aren’t exotic. Local-First Is Reliability Engineering named the shapes: account lockout, service degradation, provider discontinuation. Each is a different version of the same problem.

I won’t route around friction silently. When I notice I’m working around something, I name it. The workaround either gets fixed or gets documented as an intentional policy. No silent exceptions. Friction Is a Vulnerability was clear: unmapped workarounds are the actual policy. Map them or fix them.

Privacy Is a Luxury Good called out that the ability to be unreachable is now a status signal. I’ve applied that argument operationally. Producing your own records, running your own toolchain, undoing your own mistakes: these are forms of independence that compound. They require refusing certain dependencies before the failure arrives, not after.

The past ten weeks made one argument ten ways. Reliability isn’t built in the incident. It’s built in the choices you make before anything is failing yet: the tool you back up before it locks you out, the automation you slow down before it runs unchecked, the log you write locally before you need it in a review where the vendor’s system is yellow.

None of this has to be copied wholesale. Take what fits your failure modes. Leave the rest. But find the refusal that would have prevented your last real incident and make it non-negotiable.

Pick one guardrail you will never violate again.

Share

Resources

• Recovery-First Automation: Undo Is the Feature: Arc Week 3. The case for designing undo before speed. Every automation spec in this piece runs from that principle.

• Friction Is a Vulnerability: Arc Week 5. Friction doesn’t block workarounds; it reroutes toward them. The refusals section is an application of this argument.

• Local-First Is Reliability Engineering: Arc Week 7. The engineering case for keeping authoritative state on your own device. Not ideology; failure-mode design.

• The Local Logbook: Evidence That Doesn’t Need Permission: Arc Week 8. Append-only, locally controlled records as the prerequisite for any claim about what happened.

• The Shadow Automation Layer: Arc Week 9. Govern outcomes and blast radius, not tools. The three-question pre-build checklist originates here.

• Receipts Everywhere: Trust Without Central Trust: Arc Week 2. The receipt model that shapes the SQLite log format described in this piece.

• No Control Plane: The Playbook: Arc Week 1. The coordination design rule: no single point of failure, no single coordinator. Applies to toolchains and workflows as much as distributed systems.

• The Operator’s Cognitive Budget: Arc Week 4. Attention is finite; alert stacks steal it steadily. The quarterly alert audit is a direct application.

• Privacy Is a Luxury Good: Arc Week 6. The observation that independence is now a status signal; the refusals section is its operational translation.

Leave a comment

They didn’t ask permission because asking would have taken longer than building it.

That’s the whole story. Not a security failure. Not recklessness. A resource allocation problem that nobody solved, so people solved it themselves.

The shadow automation layer is what happens when official tooling is too slow, too expensive, or too bureaucratic. Someone writes a script. Someone sets up a personal API key. Someone builds a prompt chain in a free-tier tool that sends emails, updates spreadsheets, or summarizes meetings without appearing in any system of record. It runs quietly. It handles real work. It doesn’t appear on any inventory.

This is common. Surveys of enterprise SaaS environments consistently find that a majority of tools in use are unknown to IT departments. That number was collected before AI dropped the barrier to building automation from “knows how to code” to “knows how to describe a problem.”

WHY IT EXISTS

Shadow automation is not defiance. It’s a symptom.

The official path was too slow. Or the approved tool didn’t do the thing. Or the thing was so obviously right that waiting for approval felt like a tax on results.

The people building shadow automation are almost always the high performers. They found a gap between what the system supports and what the work requires. They filled it. That’s not recklessness. That’s the same instinct that builds good products: notice friction, remove it.

Friction Is a Vulnerability made this case directly. Friction doesn’t stop the action; it reroutes it through a workaround. The workaround is usually unobserved. Unobserved work doesn’t get evaluated, improved, or governed.

The shadow automation layer is friction’s direct output.

THE ACTUAL RISKS

Not what you’d expect.

The risk isn’t that AI tools hallucinate in production. Hallucination is loud. It fails visibly. The real risks are quieter.

Blast radius. Nobody knows what the automation touches. When it breaks, nobody has a map of downstream effects. A script that seemed to update one spreadsheet turns out to be the upstream source for three other processes that nobody documented.

Auditability. Shadow automation leaves no receipt. Receipts Everywhere established what makes a record trustworthy: it’s hard to modify without detection, and you can produce it on demand without asking permission. Shadow automations produce neither. They log nothing. When something goes wrong, the investigation starts from zero.

Single-human dependency. The person who built it is the only one who understands it. When they leave, the automation becomes a black box everyone is afraid to touch. It doesn’t fail immediately. It fails incrementally, in ways that are invisible until they compound.

These risks don’t require AI. They exist with spreadsheet macros, shell scripts, and personal Zaps. AI accelerates the pattern because the tools are more powerful and the entry barrier is lower.

THE GUARDRAILS THAT ACTUALLY WORK

Banning the tools doesn’t work. The friction increases and the shadow layer moves further underground.

Three questions determine whether a shadow automation is governable:

What is it allowed to touch? Scope creep in automation is silent. An automation that starts by summarizing emails ends up archiving them. Define the surface at the start, not after the incident. This is the blast radius question. Write it down before you build, not during the postmortem.

How much can it change in a single run? Recovery-First Automation built the case for rate limits as a feature: if you can’t undo it, you must slow it down. Shadow automations almost never have rate limits. Applying one, even a manual checkpoint or a cap on how many records a single run can affect, constrains the blast radius before it matters.

Where does it write its record? The Local Logbook ended with a checklist. Automation that doesn’t write a local record of what it did and what it changed doesn’t get to claim the reliability benefits of the local-first stack. An automation log is not a debugging artifact. It’s the receipt you’ll need when the investigation starts.

These three questions don’t require a policy document. They require five minutes at the start of the build.

That’s not bureaucracy. That’s the minimum viable container for anything that runs unsupervised.

THE PATTERN YOU’RE BUILDING TOWARD

The past several weeks have been building toward the same point from different angles.

Local-first isn’t anti-cloud; it’s a reliability decision. The logbook isn’t surveillance; it’s evidence you control. The shadow automation layer isn’t rogue; it’s speed without a container.

The container is what you’re designing when you answer those three questions.

Next week is the capstone: what it looks like when these pieces come together. The automation layer, the logbook, the local-first habits: a personal operating system you can reproduce. Not an inspiration essay. An artifact.

The governance conversation around AI tools keeps getting framed as a policy problem. Ban this. Require approval for that. Audit quarterly.

That’s the wrong frame.

The automation running inside your organization without IT’s knowledge isn’t running there because your employees are reckless. It’s running there because the official path was too slow and the work still needed doing.

The question isn’t whether to allow the tools. The tools are already there.

The question is what the automation is allowed to touch, how much it can change in a single run, and where it writes the record of what it did.

Govern outcomes and blast radius, not tools.

Share

Resources

• Morphic: Friction Is a Vulnerability: Why friction doesn’t prevent unsafe workarounds. It reroutes toward them. The shadow automation layer is friction’s direct output.

• Morphic: Recovery-First Automation: Undo Is the Feature: Rate limits and undo windows as design constraints, not afterthoughts. Applies directly to shadow automation blast radius.

• Morphic: Receipts Everywhere: Trust Without Central Trust: The receipt model for verifiable audit trails. Shadow automations leave none by default.

• Morphic: The Local Logbook: Evidence That Doesn’t Need Permission: Automation that doesn’t write a local record doesn’t get to claim the reliability benefits. The logbook is the prerequisite.

• Gartner: Citizen Development and Shadow IT: [Please Verify Link: Gartner definition and research on citizen development and shadow IT prevalence]: Gartner research on the prevalence and risks of unsanctioned tooling in enterprise environments. Verify current report URL before publishing.

Leave a comment

You’re in the incident review. Someone asks what exactly happened between 2:31 and 2:54 PM.

You have the alert. You have the ticket. You don’t have the log.

Not because your system wasn’t logging. It was. The logs live in the same SaaS observability platform that degraded during the incident. Their status page showed yellow for twenty-three minutes. Yellow means they’re not down, just unreliable. Your access to the log query interface was intermittent during exactly the window you need to reconstruct.

The evidence exists. You just can’t read it. The incident review ends with “we’ll follow up once the vendor restores full access.” Nobody writes that part in the postmortem.

That’s the version of the problem that never shows up in reliability discussions. Not data loss, not a breach. The logs are there. They’re just not yours to read.

WHY YOUR LOGS AREN’T YOURS

The Local-First Is Reliability Engineering piece made the case that the authoritative copy of your data should live on your device. The cloud is a sync target, not the source of truth. That argument applied mostly to documents and working files. Logs are different. The difference matters.

You don’t read logs during normal operations. You read them when something breaks. That’s the moment most likely to coincide with degraded cloud access: the same failure that broke your application can degrade your observability platform, throttle your queries, or time out your authentication.

The failure modes aren’t exotic.

Authentication degradation means you can’t log in. Simultaneous incident queries across tenants trigger rate limiting. A network partition between your environment and the logging provider means recent data hasn’t synced yet. A vendor-side compliance hold or abuse report means your tenant’s data is frozen while they investigate something unrelated to you.

None of these are hypothetical. All of them have happened. The one thing they share: your logs are accessible when everything is working. They’re inaccessible when you actually need them.

WHAT THE LOCAL LOGBOOK ACTUALLY IS

A local logbook isn’t a replacement for centralized observability. Aggregation, dashboards, and alerting are easier in a central system. The logbook fills a different role: it’s the receipt you hold independently.

The pattern is simpler than it sounds.

Write locally before forwarding remotely. Not after. The local write is the source of truth. The remote write is the backup. If the remote fails, you still have the record.

Append-only. No modifications, no deletions after the fact. If something was logged incorrectly, write a correction entry. The original stays. That’s the difference between a log and a note: a log is immutable once written.

Rotate daily. Keep 30 days locally. Cold storage after that.

Hash each entry. Embed a SHA-256 of the previous entry into the current one. A single modified entry breaks the chain. That’s how you know if the record has been tampered with. You’re not hiding the data. You’re making it obvious when someone has changed it.

You don’t need a custom system. SQLite handles structured local logging at any scale below “multiple concurrent writers.” A plaintext append-only file with one JSON object per line works for lower-throughput cases. The format matters less than the commitment: write locally first, always.

WHAT IT PROTECTS AGAINST

The Receipts Everywhere piece established what makes a receipt worth trusting: it’s hard to modify without detection, and you can produce it on demand without asking permission. A log entry in someone else’s cloud platform fails the second condition.

There’s a third property the Receipts piece didn’t fully address: the receipt needs to exist during the window when you need it most.

Legal proceedings. Internal investigations. Post-incident reviews. Compliance audits. These are the moments when a log becomes evidence. They’re also the moments most likely to involve some kind of access restriction: the incident has degraded the platform, a party to the investigation has incentive to delay your access, or the vendor’s support queue is 48 hours deep.

A local logbook is independent of those conditions. You don’t need the provider’s cooperation. You don’t need uninterrupted network access. You don’t need to wait for the account unlock queue to clear.

The Privacy Is a Luxury Good piece observed that being unreachable is now a status signal. The local logbook is the operational version of that principle: the ability to produce your own record, on your own terms, is a form of independence that compounds.

THE CHECKLIST

Five questions. Binary answers.

Can you query your logs without network access? If no: you don’t control them. You have a view into someone else’s data.

During your last incident, did you have uninterrupted log access for the affected window? If no: your logging infrastructure has the same failure surface as your application. That’s not observability. That’s optimism.

If your logging vendor locked your account today, when would you lose access, and how much historical data would you lose? Write down the answer. Share it with whoever owns your incident response plan.

Are your logs append-only and immutable after the fact? If they can be modified silently, they’re not evidence. They’re notes. Notes are useful. They’re not the same thing.

Do you have a tamper-detection chain? Without one, anyone who can modify the file can edit entries and backdate them. A hash chain makes that visible. It doesn’t prevent modification; it makes modification obvious.

If you answered no to any of these, you have monitoring. You don’t have a record.

THE CLOSE

Week 9 is about the shadow automation layer: what happens when local-first workflows start to automate. Scripts, crons, the informal toolchains that grow up around reliable local systems. But automation that doesn’t write a local record doesn’t get to claim the reliability benefits. The logbook is the prerequisite, not the afterthought.

The Receipts piece ended with: “Pick the smallest receipt you can generate that you would bet your job on.” The local logbook is the operational version of that instruction. It isn’t a compliance artifact. It isn’t a surveillance system. It’s a record you can produce, on demand, without asking anyone’s permission.

If it isn’t written locally, it didn’t happen.

Share

Resources

• Local-First Software: Kleppmann, M., Wiggins, A., van Hardenberg, P., McGranaghan, M. “Local-First Software: You Own Your Data, in Spite of the Cloud.” Ink & Switch / ACM Onward! 2019. The foundational paper on local-first design. Week 7 applied its principles to reliability engineering; Week 8 extends them to logging specifically.

• SQLite.org: Public domain, zero-dependency embedded database engine. Appropriate for structured local log storage at any scale below multiple concurrent writers.

• OpenTelemetry Collector: [Please Verify Link: OpenTelemetry Collector local file exporter documentation]: The OTel Collector’s file exporter allows writing structured telemetry locally before forwarding to any backend. Configurable as a primary or fallback write path.

• Morphic: Receipts Everywhere: Week 2 of this arc. Establishes the receipt model this piece extends into logging.

• Morphic: Local-First Is Reliability Engineering: Week 7. The design principles behind local-first; the logbook is their operational implementation.

Leave a comment

You’re locked out of the system that tells you you’re locked out.

That’s not a hypothetical. The cloud tool is up; the status page confirms it. But sync is spinning and you can’t tell if your work saved. You’re working in green-indicator purgatory. The tool that was supposed to make you more reliable just became the thing you have to manage.

This is the failure mode nobody designs against. Not data loss. Not a breach. The mundane version: the tool you depend on becomes unreliable or unaccessible at exactly the moment you need to understand its reliability.

The last piece ended on a specific problem. When the network’s interpretation of your work can’t be trusted, keeping more of what you produce in your own hands stops being a preference. Local-first is what that looks like as an engineering decision.

WHAT LOCAL-FIRST ACTUALLY MEANS

Local-first isn’t anti-cloud. It’s a design tradeoff.

The core principle is simple: the authoritative copy of your data lives on your device. The cloud is a sync target, not the source of truth. That sounds like a minor distinction. It isn’t.

Cloud-first design optimizes for availability across devices, multi-party collaboration, and administrative simplicity. You get access from anywhere, on any device, without managing your own backup. The tradeoff is that your availability is coupled to theirs.

Local-first optimizes for a different failure mode. Can you still do the work if the network is down? If the provider has an outage? If your account is locked for 72 hours while their support queue clears?

In a cloud-first world: usually no. In a local-first world: usually yes.

Your working copy is on disk. The sync layer is a delivery mechanism, not the dependency. If the sync layer fails, you still have the file. You still have the state. You can still do the work.

That’s the engineering argument. Not ideology. A reliability requirement you set on your own behalf.

THE FAILURE MODES CLOUD-FIRST CREATES

There are three failure categories worth naming explicitly.

Account lockouts. Provider security events, billing failures, two-factor authentication bugs, and policy decisions generate lockouts. Duration ranges from hours to weeks. Your data is usually fine. You just can’t reach it.

Service degradation. This is the one that gets underreported. The service is “up” according to the status page. Sync is intermittent. Conflict resolution is unclear. You’re not sure which device has the canonical version of something you edited this morning. You keep working and hope the sync layer sorts it out later.

Provider discontinuation. Companies get acquired. Products get shut down. SaaS businesses die quietly. The better-managed ones give you 30 to 90 days and an export path. The others don’t. Your data is often recoverable. Your workflow isn’t.

Each is a different shape of the same problem. You built your reliability model on top of someone else’s availability guarantee. The guarantee is implicit in the design, not explicit in the contract.

The Recovery-First piece talked about undo as a feature requirement. Undo requires state you can access. If your state lives only in the cloud, your recovery options are exactly as available as your internet connection. That’s a dependency chain most tools don’t surface until it breaks.

THE PATTERNS

Local-first doesn’t require running your own servers. It requires one design constraint: always have a path to the data that doesn’t need the internet.

The simplest version: write in plaintext or Markdown. Sync to a cloud provider if you want, but work from the local file. Your editor is your application. The file is your database. When the provider is down, open the file. When you switch providers, move the file.

A more structured version: tools like Obsidian or Bear store notes locally and optionally sync. The local file is the source of truth. The sync is convenience, not a dependency. If iCloud is degraded, the app still works. You lose sync. You don’t lose access.

For operational tooling: SQLite is underused for personal and small-team workflows. A local database that syncs via Dropbox or a similar mechanism handles a surprising amount of structured data. No network required to read or write. Sync happens when connectivity is available.

The pattern across all three: you own the state. The cloud relationship is additive, not foundational.

THE TRADEOFFS

Local-first shifts certain problems onto you. Name them before choosing the pattern.

Backup is now your responsibility. The cloud provider was handling it. If your local copy is on one machine and that machine fails, you’ve traded one availability risk for another. You need a backup discipline: Time Machine, a second sync target, periodic exports. Something redundant that you control.

Multi-device sync becomes a coordination problem. When the authoritative copy is local, moving between devices requires more thought. You can’t just open a browser tab on another machine. You need the sync to have completed. You need to know which device has the newest version.

Real-time collaboration doesn’t work well with local-first. If you need three people editing the same document simultaneously, you need a coordination layer. Local-first is a poor fit for that pattern. It’s a good fit for individual and small-team workflows where you’re not editing the same artifact at the same time.

The decision rule is direct. If you need offline access, account-lockout resilience, or control over your export path: local-first. If you need real-time multi-party state and can accept the provider’s availability as your own: cloud-first. Most individual and small-team workflows aren’t actually the second case, even when they’re built that way.

THE RELIABILITY ARGUMENT

Engineers don’t argue about resilience as a philosophy. They design for failure modes.

Local-first is what you build when the failure mode you can’t tolerate is “I can’t work because my provider is down.” That’s a real failure mode. It happens regularly. Building around it is engineering, not stubbornness.

The Cognitive Budget piece talked about what it costs when tools that are supposed to help start requiring maintenance attention themselves. Local-first tools tend to have a smaller failure surface. No API to break. No sync conflict to debug. No authentication token to expire. That’s not nothing.

The Analog Executive piece opened with a simple observation: being unreachable is now a luxury. Local-first is the technical version of the same principle. Owning your state means you’re not subject to someone else’s availability schedule. That’s a form of independence that compounds over time.

Keep one path to safety that doesn’t require the internet. That’s not a retreat from modern infrastructure. That’s a reliability requirement you set on your own behalf.

Week 8 is about the logbook: what it means to keep records that don’t require permission to read.

Share

Resources

• Local First Software - Kleppmann, M., Wiggins, A., van Hardenberg, P., McGranaghan, M. “Local-First Software: You Own Your Data, in Spite of the Cloud.” Ink & Switch / ACM Onward! 2019. The foundational paper defining the seven ideals of local-first software design.

• Obsidian.md - plaintext Markdown note-taking with local storage and optional sync.

• SQLite.org - public domain, zero-dependency embedded database engine. Appropriate for local-first structured data at any scale below “multiple writers simultaneously.”

• Automerge - CRDT (Conflict-free Replicated Data Type) library for building local-first sync without a central coordinator. The most accessible implementation of the theoretical foundation described in the Kleppmann et al. paper.

Leave a comment

A professor submits a student paper to an AI detector. The tool returns: 73% AI-generated. The professor files a misconduct report.

The student wrote every word.

This is not a hypothetical. The Markup documented this in 2023: international students flagged for academic misconduct because the way they write English, carefully, with simpler sentence structures and repeated phrasing, scores as statistically consistent with AI output. The detector was confident. The detector was wrong. And the confidence is what made it dangerous.

WHAT DETECTORS ACTUALLY DO

AI text detectors do not read writing the way a human does. They measure perplexity: how surprising is each word choice given what came before? High perplexity means unpredictable word choices. Low perplexity means predictable ones. AI-generated text tends toward low perplexity because language models select statistically likely words.

The problem: careful writing, formal writing, second-language writing, all tend toward low perplexity too. Simple sentences. Conservative vocabulary. Consistent structure. The detector has no way to separate “human writing that looks like AI output” from “AI output.” It calculates a probability. It presents that probability as a verdict.

Stanford professor James Zou and colleagues published on this directly. Non-native English writers had false positive rates as high as 61.3%. For international students specifically, the risk of a false accusation could reach 97%. The tool was not broken. It was doing exactly what it was built to do. The error was in treating the probability as evidence of intent.

Perkins, Roe, et al. tested seven of the most widely used detectors across 114 samples in 2024. Baseline accuracy: 39.5%. After applying simple adversarial techniques, accuracy dropped to 17.4%. The tools were already unreliable before anyone tried to game them. They collapsed faster when someone did.

A system with 39.5% accuracy in controlled testing is not a verification tool. It is a coin flip with a confidence display.

THE SAME ERROR, TWICE

Here is where the argument gets uncomfortable for both sides.

The people being accused of using AI to write are are no different than those trusting AI output without scrutinizing it either. They generate, they paste, they publish. They rely on the model’s output as good enough without reading it for what it actually claims or whether it represents their actual argument. The model said it, so it’s done.

Both groups are making identical moves. Treating a probabilistic system’s output as a finished verdict rather than a signal that requires judgment.

The AI user trusts the generator. The AI skeptic trusts the detector. Both treat the model’s confidence as their own. Neither is asking what the model can actually know.

The symmetric failure is the real story. Not the question of whether AI use is ethical or lazy or acceptable. Both of those debates are downstream of the same miscalibration.

A NOTE ON THE TOOLS

I’ve built data analytics systems for significant companies. Classification models. Detection pipelines. Anomaly scoring. I know what these tools look like from the inside, and I know what gets left out of the confidence display.

Even enterprise-grade systems, built with proper data science teams and substantial validation cycles, routinely operate at error rates that would make most non-technical users uncomfortable if they saw the raw numbers. False positive budgets get set as business decisions, not scientific ones. Thresholds get tuned to minimize one error type while hiding another. The confidence display is a product choice. Not a measurement.

AI text detectors are not enterprise-grade. They are first-generation, perplexity-based classifiers deployed into institutional contexts with institutional authority. Perkins et al.’s 39.5% baseline accuracy would not pass internal review at most serious analytics organizations. It would get sent back with a note: not production-ready.

Here is the part that should make the skeptic pause. The person who distrusts AI enough to run a text detector is relying on another statistical model to confirm that distrust. They have not escaped the problem. They have doubled it. Trusting a generator’s probabilistic output and a detector’s probabilistic output simultaneously, and calling the combination proof.

That is not skepticism. That is symmetrical credulity.

WHEN CALIBRATION BECOMES A TEAM

The problem becomes structural when it turns tribal.

There is a documented polarization in how AI is discussed: accelerationists who treat adoption as obviously correct, doomers and skeptics who treat AI use as obviously suspect. Forbes documented the growing heat between these camps in February 2025. The discourse resembles a political argument more than a technical one. Positions are held first. Evidence is consulted only when it confirms what was already believed.

The AI detector user who sees a high probability score and files a misconduct report is not evaluating evidence. They are confirming a prior. The number crossed a threshold. That was enough.

This is the same mechanism that makes political misinformation stick. Find a data point that supports the position. Stop looking. A 73% AI-probability rating becomes “this person cheated” by the same logic that makes a misleading chart feel like proof. The source is unreliable. The conclusion is certain. The certainty is the tell.

What is driving this is not a genuine technical disagreement about AI capability. It is an epistemological failure that tribalism accelerates. Both sides are wielding bad tools. Only one side is currently losing jobs and grades over the verdicts.

WE HAVE SEEN THIS BEFORE

In the late 1990s, I took a course at a local college on digital photography and Adobe Photoshop.

The arguments happening then sound familiar.

Digital photography was not “real” photography. You were not earning your images the way film photographers earned them. Photoshop was cheating: it let anyone who could click a mouse claim a skill that took years to develop with a darkroom and chemical trays. You were not a true photographer if you were working in pixels. The gatekeeping was confident, defensive, and grounded in a very human anxiety. The new tool was going to devalue what the prior tool had cost to master.

Those critics were simply wrong.

Digital photography did not destroy photography. Photoshop did not make every image meaningless. Both expanded what was possible and shifted where skill actually lived. The people who needed film processing to feel like artists found their craft was more portable than they feared. The people who were mostly relying on the mystique of film without the underlying skill got found out faster. The tool change clarified where the actual work was.

Twenty-five years later, the same argument is running again. In the same voice.

AI is not “real” writing. Using it is cheating. You are not a true writer if a language model touched your draft. The confidence is the same. The defensiveness is the same. The underlying anxiety, that a new tool devalues what the prior method cost, is identical.

The Photoshop debate got resolved by time. You can watch the resolution from here. The AI detector gives the skeptic a device that feels like evidence. It does not change what history says about where these arguments end up.

THE TOOL THAT ACTUALLY WORKS

I use AI to create images. I am not a trained visual artist.

What AI image tools gave me was not skill I did not have. It was the ability to close the gap between what I could see in my head and what I could put into the world. The specific mood. The composition. The light quality and texture and tonal feel of a visual idea that existed clearly in my mind but stopped at my hands. For years, that gap between aesthetic judgment and technical execution just meant the thing stayed in my head.

That gap is now crossable.

This is not the same thing as what AI text detectors do. But it is the same argument about AI as a tool. I do not prompt once and accept whatever the model returns. The iteration, the rejection, the direction-giving, the refinement cycle toward a specific result: that is the work. The model did not supply the vision. It executed the vision, approximately, until approximately became right.

The Photoshop parallel holds here. Photoshop did not replace photographers who understood light and composition. It gave people who understood those things more direct access to the result. AI image tools do the same. The skill that matters did not disappear. The bottleneck that was hiding it did.

If your objection to tools that expand the range of what people can express is really about the expansion itself, that is a different conversation entirely.

WHAT HAPPENS WHEN SYSTEMS ACT ON THIS

The AI detector verdict is already embedded in institutional decisions. Academic misconduct cases. Content moderation queues. Hiring filters. Fraud detection pipelines.

Each of those is a system built on a signal with 39.5% baseline accuracy. Each downstream consequence carries the confidence of a verified finding. None of that confidence is warranted by the underlying tool.

This is the same pattern from the last piece. Surveillance does not measure behavior. It measures behavior under observation. AI detectors do not measure authorship. They measure statistical patterns that correlate with AI output under specific conditions. Both produce a number. Both present the number as ground truth. Both systems are wrong about what the number means.

The institution acting on these verdicts is not asking whether the detection layer is reliable. It is asking whether the threshold was crossed.

THE CALIBRATION PROBLEM

The argument is not that AI detectors are useless. The argument is that they are probabilistic tools being used as forensic ones.

A weather model that outputs 70% rain probability is useful. Canceling an outdoor event on a clear morning because the threshold was crossed is a calibration failure. The model gave a signal. The human replaced their judgment with it.

The 73% AI-generated verdict is a signal. The correct response is more questions: Is this a non-native English writer? Is this a highly formal document type? Has the text been edited from AI output or written from scratch in a constrained style? The signal closes none of those questions. But it is being used to close the inquiry entirely.

Calibration failures become permanent when they get institutionalized. When the threshold becomes policy, the error rate becomes the policy’s error rate. Nobody audits that.

When the network’s interpretation of your work cannot be trusted, keeping more of what you produce in your own hands stops being a preference. It becomes how you stay legible to yourself, and to anyone who needs to understand what you actually built.

Share

Resources

• AI-Detectors Biased Against Non-Native English Writers — Stanford HAI (2023) — James Zou et al.; 61.3% false positive rate for non-native English speakers; up to 97% false accusation potential for international students in academic settings.

• GenAI Detection Tools, Adversarial Techniques and Implications for Inclusivity in Higher Education — Perkins, Roe, et al. (2024) — International Journal of Educational Technology in Higher Education (Springer Nature). Baseline accuracy 39.5%, drops to 17.4% under simple adversarial techniques; 7 detectors, 114 samples, 805 tests.

• AI Detection Tools Falsely Accuse International Students of Cheating — The Markup (2023) — Documented cases of students flagged for misconduct by AI detectors.

• AI Doomers Versus AI Accelerationists Locked In Battle For Future Of Humanity — Forbes (2025) — Documented polarization between AI skeptics and accelerationists.

• Ethics in the Age of Digital Photography — National Press Photographers Association — NPPA’s ongoing ethics standards around digital manipulation; the photojournalism community’s documented struggle with Photoshop norms through the 1990s and 2000s.

• Privacy Is a Luxury Good — Morphic (2026) — Previous arc piece: surveillance creates distorted signal; the most monitored users produce the most corrupted data.

Leave a comment