The Production Judge
I built the behavioral judge the last piece said didn't exist. The part nobody warns you about: now the thing watching for drift can drift.
Last time I left it here: the governance layer is running on discipline, and discipline doesn’t scale. The fix is supposed to be obvious. Build the judge. A scorer that runs the behavioral rubric against production output on its own, no human pressing go. So I built it.
It works. It runs every night, scores a sample of the day’s output, flags what looks off. I stopped being the only thing standing between a misbehaving agent and a clean dashboard.
Then about a week in I caught the thing nobody puts in the demo. The judge is a behavioral agent too. Same model class, same prompt-shaped temperament, same capacity to drift. Everything I said about not fully trusting my agents now applies to the thing I built to watch them. I didn’t remove the trust problem. I bought a second one.
What the judge actually is
Concrete first, so the rest lands.
The judge runs in two tiers. The cheap tier is deterministic: regex and rule checks on 100 percent of output. Did the worker escalate when it hit an ambiguity flag? Did the foreman delegate inside its role boundary? Did anything call a tool outside its allowlist? That layer is fast, dumb and free. It catches the violations you can write down ahead of time. What it can’t catch is the judgment calls, which is exactly where my agents fail. So the cheap tier is necessary and nowhere near sufficient.
The expensive tier is an LLM scoring the behavioral rubric: the same 25 cases per role from my eval framework, run as a judge against a 10 percent sample of real output. Reviewing everything is too slow and too costly. Ten percent gives me trend, and trend is the thing I actually want. Individual scores lie. The slope doesn’t.
It runs on a cron at 2 AM. By the time I’m up there’s a number per role and a short list of outputs it scored low. No human initiates it. That was the whole point: the eval that runs without me standing over it.
For about a week, this felt like the answer.
The judge drifts too
Here’s what broke the feeling.
I keep a small set of outputs I scored by hand, cases where I know the right call cold. The judge agreed with me on roughly 88 percent of them in week one. By week three it was down near 79, and I hadn’t touched it. Same prompt. Same rubric. Same model weights. (I diffed the config three times because I didn’t believe it. Nothing changed on my side.)
The judge’s agreement with me decayed on its own. The industry has a name for this now: calibration drift. A judge that lined up with your humans last quarter drifts out of agreement as the input distribution shifts under it, no redeploy required. RAND’s team put numbers on the general version of this in March. They stress-tested four state-of-the-art judges and found none of them uniformly reliable; agreement moved on nothing more than reformatting the input, paraphrasing it, padding the verbosity. The judgment wasn’t anchored to the behavior. It was anchored to the surface of the text.
Why it drifts isn’t mysterious once you stop expecting it to behave like code. My production inputs got longer and messier over six weeks. Real tasks don’t look like the tidy rubric examples I wrote back in week zero. The judge started seeing output shaped differently from anything in its instructions, and it did what these models do under ambiguity: it reached for surface cues. Length read as thoroughness. Confident phrasing read as a correct answer. The rubric never changed. The distribution it was being applied to walked away from the one I calibrated it against.
So the judge is not a fixed instrument I built once and can forget. It’s an agent with the same disease as the agents it grades. Of course it is. It’s the same kind of thing.
The green dashboard, again
A few weeks back I wrote about the infrastructure dashboard that shows every gauge nominal while the behavior underneath it goes wrong. The judge can become that dashboard. Worse, actually.
A missing judge is an honest gap. You know you’re not watching. A drifted judge is a lie with a number attached. It reports 91 percent, you exhale, and the 91 is measuring the judge’s mood instead of the agent’s behavior. False confidence beats no confidence right up until the morning it doesn’t. I trusted my own dashboard once and paid for it with three hours of cleanup. A confident judge I haven’t re-checked is that same trap wearing a lab coat.
This isn’t only my problem, for what it’s worth. LangChain’s 2026 agent survey put 57 percent of organizations running agents in production, with quality the top thing blocking the rest. Most of that quality question reduces to: who’s watching the agent, and who’s watching them. The more of the watching you automate, the more weight lands on the one layer you quietly stopped watching.
Where the regress stops
This is the part I want to be honest about, because the clean version of this story ends with “so I built a judge for the judge.” I didn’t. That’s the same problem one floor up, and it’s turtles from there.
The regress has to bottom out somewhere, and the only place it can bottom out is a human-fixed reference. For me that’s the golden set: a small, slow-growing pile of outputs with a verdict I’ll defend, that the judge gets scored against on a schedule. Not the agent. The judge. When its agreement with the golden set slips, the judge goes back for recalibration before I trust another number it hands me. I version the judge prompt with a date, the way you version anything you don’t want changing silently underneath you.
It’s about forty cases right now. It does not scale gracefully and it depends on me sitting with raw outputs and making calls I’d put my name on. Which is the exact thing I was trying to automate away.
So here’s where six weeks of this leaves me. Automation didn’t take the human out of the loop. It made the human’s job smaller, rarer and far more dangerous to skip. The judge watches the agents. The golden set watches the judge. The forty cases watch me, and under them there’s nothing but the floor.