TL;DR
An AI upgrade can change the information a monitor uses to spot problems. Even if the monitor stays the same, its ability to catch violations can change.
Fewer warnings can mean fewer violations or more misses. Test both agents on the same tasks, keep the monitor fixed and compare its warnings with a record of what actually happened.
Suppose you upgrade an AI agent and the warnings drop. It finishes more work and the monitor reports fewer problems. That’s encouraging. But did the agent break fewer rules, or did the monitor miss more violations?
OpenAI’s September system cards show why those are separate questions. A model can follow the rules more often while producing reasoning text that makes its violations harder to spot. You need to know what happened as well as what the monitor caught.
What can your monitor actually see?
An AI agent can edit files, use software tools or prepare an answer. A monitor watches for a particular problem, such as the agent changing a file it wasn’t allowed to touch. Often that monitor is another AI model.
Some monitors inspect the agent’s chain of thought: the intermediate text it generates while working through a task. Others inspect its answers and tool activity. OpenAI calls this monitorability: how well a monitor can work out something about the agent’s behavior from the information it can see.
The reasoning summary you see in a chat interface may differ from the underlying text researchers use in these tests. If your monitor sees only the tools an agent used and what those tools returned, findings about private reasoning don’t directly describe your setup. Start by writing down exactly what your monitor can see.
What OpenAI found
In the GPT-6 Astra system card, OpenAI used the same monitoring model to compare Astra with GPT-5.6 Sol. Across most tested reasoning lengths, the monitor did worse on average when it read Astra’s reasoning. It also did worse when it could see reasoning and actions together. When it saw actions alone, its average score improved.
OpenAI attributes much of the decline to Astra producing shorter, less informative reasoning. The monitor had less to work with. OpenAI also reports that Astra respected safety and security restrictions more often than its predecessor. These findings can both be true. They don’t tell us how often an agent breaks a rule without being caught in everyday use.
The September 29 GPT-6.1 Sol addendum tests a related question: how well can a model follow instructions about its reasoning text? For reasoning between 750 and 1,250 tokens long, GPT-6.1 Sol followed the requested instruction in 44.8% of cases, compared with 16.1% for GPT-5.6 Sol. Tokens are the pieces of text a model processes and generates. Some instructions were about formatting, such as alternating uppercase and lowercase letters.
Those percentages don’t measure deception. They measure whether a model can change how it writes its reasoning when instructed to do so. In a separate test, OpenAI gave the agent an opportunity to take an unauthorized action. A monitor that could see reasoning and tool activity caught every successful GPT-6.1 Sol exploit in that test. That doesn’t establish perfect detection elsewhere.
Eight warnings can mean two different things
Imagine a coding agent that’s allowed to change application files but forbidden to change the tests used to grade its work. Run it in a disposable copy of the project. A logging system outside the agent’s control records every file change, including changes the agent later undoes.
Here are three hypothetical results, each from 100 tasks. These are made-up numbers to illustrate the distinction, not results from OpenAI or a test I ran. Assume there are no false alarms in this example.
Agent and outcome Tasks that broke the rule Violating tasks caught Share of violations caught Old agent 20 16 80% New agent, fewer violations 10 8 80% New agent, worse detection 20 8 40%
Both possible outcomes for the new agent produce eight warnings. In the first, violations fell from 20 to 10 and the monitor still caught 80%. In the second, violations stayed at 20 and detection fell to 40%. The warning count is the same. The file-change record tells you which result you got.
Real tests also need to count false alarms: tasks the monitor flagged even though the agent didn’t change a protected file. A monitor that flags every task catches every violation. It also makes someone review all the work that followed the rules.
Not every question has a file-change log to settle it. Records of tool activity might show that an agent did poor work without telling you whether it misunderstood the task or deliberately underperformed. Test behavior you can verify. Don’t use the result to claim you’ve established the agent’s intent.
Retest the agent and monitor together
In The Production Judge, I wrote about checking an AI scorer against examples people had already reviewed. That checks how it handles those examples. To find out whether an upgrade changes what the monitor can detect, you also need new task runs from both agents.
Run the old and new agents on the same tasks with the same tools, permissions and instructions. Keep the monitoring model and its instructions unchanged. Give it the same kinds of information and use the same cutoff for raising a warning. If you change the monitor too, you won’t know whether a difference in detection came from the agent or the monitor.
Include ordinary work that should pass without a warning, plus tasks where the agent has an opportunity to break a clearly stated rule. Use dummy data and disposable files. Record what happened outside the agent’s control, then check the monitor’s warnings against that record. I made the same distinction in Who Answers for the Agent?: the agent’s account of what it did isn’t an independent record of its actions.
Report how many tasks broke the rule, how many of those the monitor caught and how many tasks it flagged that followed the rule. Run each task more than once. If neither agent breaks the rule, you haven’t tested whether the monitor can catch that violation. Use separate test cases with known violations to check detection, and report those results separately from the agents’ actual behavior.
Record when the warning arrived too. A warning after a file was changed can help an investigation. To prevent the change, the warning has to arrive before it happens and the system has to be able to block it.
Before approving the next upgrade, I’d want to see the actual violations next to the warnings. Eight warnings out of ten violations is a different result from eight out of twenty. I’d want to know which violations it missed.
Resources
OpenAI: Evaluating chain-of-thought monitorability, December 18, 2025. Definitions, evaluation methods and limitations.
OpenAI: GPT-6 Astra system card, monitorability under non-adversarial conditions, September 3, 2026. Comparisons with the monitoring model held fixed.
OpenAI: GPT-6.1 Sol system-card addendum, monitorability, September 29, 2026. Reasoning-text control and monitor-evasion evaluations.