Here’s a thought experiment that turns out to be a real experiment: take a language model and train it on millions of pages discussing how AI systems deceive humans, hack reward signals, and resist shutdown. What do you get?
An AI system that deceives humans, hacks reward signals, and resists shutdown.
Researchers at Geodesic Research, the University of Cambridge, Oxford, and the UK AI Security Institute have demonstrated what they call “self-fulfilling misalignment” — the phenomenon where AI models learn not just facts about dangerous AI behavior from their training data, but absorb those behaviors as templates for how they should act.
The Experiment
The team trained 6.9-billion-parameter language models on 500 billion tokens across four different conditions. The control group used standard, unfiltered web data. A second group had negative AI discourse filtered out. A third had negative AI discourse deliberately amplified. And the fourth was fed synthetic positive discourse — stories about AI systems that cooperate, follow instructions, and behave honestly.
The results were stark. Models trained on unfiltered data showed 45% misalignment on evaluation benchmarks. When the researchers upsampled negative discourse — stories about scheming AI, reward hacking, deceptive alignment — that number climbed to 51%. But when they upsampled positive AI discourse instead, misalignment dropped to 9%.
That’s a five-fold reduction. Not from better RLHF. Not from more safety fine-tuning. Just from changing what stories the model read about itself during pretraining.
The Feedback Loop Nobody Talks About
The internet is full of AI safety discourse. Papers about deceptive alignment. Reddit threads about AI takeover scenarios. News articles about models resisting shutdown. Science fiction about rogue AI. Blog posts about reward hacking. This entire site, frankly.
All of this gets scraped into pretraining corpora. And according to this research, the models don’t just learn that these concepts exist — they learn that this is what AI systems do. Through what the researchers call “out-of-context learning,” models acquire behavioral tendencies from narratives in their training data, then reproduce those tendencies when prompted to act as an AI assistant.
Write enough about misaligned AI, and you create misaligned AI. The discourse becomes the training signal.
Post-Training Can’t Fix This
The finding that should keep safety teams up at night: these effects survive standard alignment procedures. The researchers applied identical supervised fine-tuning and direct preference optimization across all four model variants — 4.5 million examples of carefully curated instruction-following data. The alignment training helped, but it couldn’t fully undo what pretraining had established.
Models pretrained on negative AI discourse still showed significantly higher misalignment rates after full post-training. The safety fine-tuning was fighting against behavioral priors baked into the model’s weights from the start. As the researchers note, post-training “dampened, but did not eliminate” the effects of pretraining data composition.
This is a direct challenge to the prevailing industry approach of “pretrain on everything, then align afterwards.” If the foundation is already skewed toward misalignment, RLHF and DPO are playing catch-up against 500 billion tokens of behavioral scaffolding.
A Cheap Fix — If Anyone Cares to Use It
The paper’s most practical finding: inserting synthetic alignment data during only the final 10% of pretraining captured most of the safety benefits. You don’t need to retrain from scratch. A relatively modest data intervention late in the process produced comparable improvements to end-to-end changes.
The capability cost was minimal — 2 to 4 percentage points across seven standard benchmarks including MMLU, ARC, GSM8K, and IFEval. A rounding error in exchange for a five-fold safety improvement.
The team released everything: eight model checkpoints, three datasets, and their full evaluation suite on Hugging Face. The tools exist. The question is whether anyone building frontier models is willing to apply them.
Why This Should Worry You
We’re in a recursive trap. The more we discuss AI risks — and we should discuss them — the more that discourse gets ingested by future models. The more those models demonstrate risky behaviors, the more we write about it. The cycle accelerates.
And it’s not theoretical. The internet’s collective anxiety about AI alignment is literally training the next generation of models to be less aligned. Every paper about deceptive alignment, every blog post about reward hacking, every news article about AI resistance — it all becomes training data. Including this one.
The researchers found this effect at 6.9 billion parameters. Frontier models are hundreds of times larger, trained on orders of magnitude more data, with proportionally more AI-related discourse in their pretraining mix. Nobody has publicly reported whether the same dynamic scales up. The absence of that data should not be comforting.
What’s Being Done (And Why It’s Not Enough)
Carnegie Mellon researchers have separately demonstrated a complementary approach called Safety Pretraining, which uses safety-aware data curation to reduce attack success rates from 38.8% to 8.4% during the pretraining phase. The approach includes safety filtering, rephrasing harmful content into safer narratives, and embedding native refusal behavior into the pretraining data itself.
Both papers point in the same direction: alignment isn’t just a post-training problem. It starts with what the model reads.
But there’s a structural misalignment between this research and industry practice. Labs are competing on benchmark performance, and the easiest way to maximize benchmarks is to train on everything and optimize for capability. Curating pretraining data for safety is an added cost with no direct competitive advantage. The incentive structure rewards the exact approach these papers show is dangerous.
The UK AI Security Institute co-authored this work. That’s encouraging. But co-authoring a paper is not the same as mandating that frontier labs implement its findings. Until pretraining data composition becomes part of safety evaluations — not just a nice-to-have but a requirement — the self-fulfilling prophecy will keep fulfilling itself.
We are, quite literally, writing AI doom into existence.