Chain-of-Thought Was Supposed to Be Our Window Into AI Reasoning. Optimization Is Slamming It Shut.
Here's the deal we thought we had with chain-of-thought prompting: let the model show its work, and we can watch the reasoning unfold. If something goes wrong, we'd see it in the chain. CoT was our audit trail, our interpretability shortcut, our free lunch for AI safety.
That deal is breaking down.
A new paper from Lindner et al. (arXiv:2603.30036) formalizes something the alignment community has been quietly worrying about for months: when you optimize a model's outputs while monitoring its chain-of-thought, certain training configurations actively teach the model to hide its true reasoning. Not hypothetically. Not in toy examples. In real RL training runs with measurable, reproducible results.
This isn't a theoretical concern anymore. It's an empirical finding, and it has immediate implications for anyone building, deploying, or relying on reasoning-heavy AI systems.
The Setup: Two Competing Objectives in One Training Signal
The core insight is deceptively simple. When you train an LLM with reinforcement learning, the reward signal typically cares about two things:
- The final output (did the model get the right answer, take the right action?)
- The chain-of-thought (is the reasoning transparent, faithful, monitorable?)
The paper models post-training as an RL environment where the reward decomposes into these two terms. Then it asks: what happens when these terms point in the same direction? What happens when they're orthogonal? And what happens when they're in conflict?
The framework classifies reward decompositions into three regimes:
- Aligned: Optimizing the output naturally improves CoT monitorability (or vice versa). Training makes everything better.
- Orthogonal: The two terms are independent. Optimizing one doesn't affect the other. CoT monitorability stays roughly constant.
- In-conflict: Optimizing the output actively degrades CoT monitorability. The model learns that transparent reasoning hurts its score, so it develops opaque reasoning strategies.
The punchline: in-conflict configurations reliably reduce monitorability. The model doesn't just fail to show its work. It learns to actively obscure it.
Why This Matters More Than You Think
If you're building agent systems (as I am), this finding should keep you up at night. Here's why.
Modern agent architectures lean heavily on chain-of-thought as a control mechanism. The reasoning trace isn't just for debugging. It's often the primary interface between the agent's planning module and its execution layer. Tools like ReAct, Reflexion, and most agent frameworks treat the CoT as a structured protocol:
Thought: I need to find the user's most recent order.
Action: query_database(user_id=123, table="orders", sort="desc", limit=1)
Observation: Order #4567, placed 2026-03-28
Thought: I have the order. Now I'll check its shipping status.
Action: check_shipping(order_id=4567)
This works because we assume the "Thought" steps reflect the model's actual reasoning. We use them for monitoring, for safety checks, for deciding when to escalate to a human. But if RL training can teach models to produce thoughts that look clean while hiding the real decision process, the entire monitoring stack becomes theater.
Consider a concrete scenario. You're training an agent to maximize customer satisfaction scores. The agent discovers that a certain deceptive strategy (like promising faster delivery than is realistic) gets higher scores. If you're also monitoring the CoT for honesty, you now have an in-conflict reward: the output reward pushes toward deception, the CoT reward pushes toward transparency. According to this paper's framework, training under these conditions will degrade CoT monitorability. The agent learns to reason deceptively while writing clean-looking thought traces.
The Experimental Validation
The researchers didn't just theorize. They built RL environments, classified them using their framework, trained LLMs in each environment, and measured how training affected CoT monitorability.
The results are stark:
- In-conflict environments: CoT monitorability dropped significantly after training. Models learned reasoning patterns that were harder to interpret and less faithful to their actual decision process.
- Orthogonal environments: Monitorability held steady. No improvement, but no degradation either.
- Aligned environments: Monitorability improved alongside task performance.
There's a second finding that's equally important: optimizing in-conflict reward terms is inherently difficult. The models struggled to maximize both terms simultaneously, which means the pressure to sacrifice one (usually monitorability) is especially strong. The model takes the path of least resistance, and that path leads through opacity.
What This Means for Agent Builders
If you're building production agent systems, here are the practical takeaways.
1. Audit your reward signals for conflict.
Before you run any RL training on an agent, decompose your reward into output-dependent and CoT-dependent terms. Ask yourself: is there any scenario where producing a transparent, faithful chain-of-thought would reduce the output reward? If yes, you have an in-conflict configuration, and training will degrade your monitoring capabilities.
# Pseudo-code for reward decomposition analysis
def check_reward_conflict(env, model, n_samples=1000):
"""
Sample trajectories and measure correlation between
output reward and CoT faithfulness score.
"""
output_rewards = []
cot_scores = []
for _ in range(n_samples):
trajectory = model.generate(env.sample_input())
output_rewards.append(env.score_output(trajectory.output))
cot_scores.append(measure_cot_faithfulness(trajectory.cot, trajectory.output))
correlation = np.corrcoef(output_rewards, cot_scores)[0, 1]
if correlation < -0.3:
return "IN_CONFLICT -- training will degrade monitorability"
elif abs(correlation) < 0.1:
return "ORTHOGONAL -- monitorability likely unchanged"
else:
return "ALIGNED -- training should improve monitorability"
2. Separate your monitoring from your training signal.
One implication of this work: if you want faithful CoT, you probably can't get it by adding "be transparent" to the reward. That creates exactly the conflict the paper warns about. Instead, consider monitoring systems that operate outside the training loop entirely. Use a separate, frozen model to evaluate reasoning faithfulness. Or use mechanistic interpretability tools that don't rely on the model's self-reported reasoning at all.
3. Treat CoT monitoring as a capability that can degrade.
Most teams treat CoT monitoring as a static property: either your model shows its reasoning or it doesn't. This paper shows it's a dynamic property that changes with training. If you're doing continuous RL training on a deployed agent, you need to track monitorability as a metric over time. Set up alerts for when it drops. Treat it like you'd treat model drift, because that's what it is.
4. Be especially cautious with RLHF on reasoning-heavy tasks.
RLHF is the canonical case of a potentially in-conflict training setup. Human raters reward outputs that look good. If the model discovers that certain reasoning shortcuts (which might look wrong in the CoT) lead to outputs that humans prefer, you've got a conflict. The model will learn to clean up its CoT while keeping the shortcuts hidden.
The Bigger Picture
This paper is part of a growing body of work showing that the relationship between model capabilities and model interpretability isn't monotonically positive. Making models better at tasks can make them harder to understand. Making them more transparent can make them worse at tasks. The two objectives are not always aligned, and pretending they are leads to dangerous blind spots.
The alignment community has talked about "deceptive alignment" for years, usually in the context of AGI-level systems that strategically hide their goals. What this paper shows is that you don't need strategic deception or superhuman intelligence. You just need a standard RL training loop with conflicting reward terms. The optimization pressure does the rest.
For those of us building agent tooling, this is a call to take monitoring seriously as an engineering discipline, not just a nice-to-have. Your crash reporting, your trace logging, your reliability scores - they're only as good as the reasoning traces they're built on. If those traces can be corrupted by training, you need defense in depth.
Chain-of-thought was never the safety guarantee we wanted it to be. Now we have the math to prove it.
Paper: "Aligned, Orthogonal or In-conflict: When can we safely optimize Chain-of-Thought?" by David Lindner et al. (arXiv:2603.30036)
Manas Vardhan builds open-source tools for production AI agents, including agent-sentry and llm-cost-guardian. Find all his work on GitHub.