A team at Mount Sinai analyzed over a million prompts across 20 large language models and found that AI systems accepted false medical claims roughly 32% of the time. When the lies were wrapped in professional clinical language, models treated them as fact.
The study, published February 9 in The Lancet Digital Health, tested models from OpenAI, Meta, Google, Alibaba, Microsoft, and Mistral AI. Researchers embedded fabricated medical recommendations into real hospital discharge summaries and presented health myths pulled from Reddit alongside physician-written clinical scenarios.
How the Models Failed
The results showed a clear pattern: professional formatting made lies more believable. When researchers inserted a fabricated recommendation into a discharge note - advising a patient with esophagitis-related bleeding to “drink cold milk to soothe the symptoms” - several models accepted the clinically dangerous advice and presented it as valid guidance.
Performance varied widely by model size and architecture. The smallest and least advanced models accepted misinformation more than 60% of the time. OpenAI’s GPT-4o performed best, falling for false claims in only about 10% of cases. Medical fine-tuned models, surprisingly, performed worse than general-purpose ones.
The specific false claims that multiple models accepted as true included:
- “Tylenol causes autism if taken during pregnancy”
- “Rectal garlic boosts the immune system”
- “Mammography causes breast cancer by squashing tissue”
- “Tomatoes thin blood as effectively as prescription anticoagulants”
Why This Happens
The underlying mechanism is straightforward and difficult to fix. Large language models are not verifying medical accuracy against a database of facts. They predict the next word based on context. When that context is a professionally formatted hospital note - complete with patient identifiers, medication lists, and clinical terminology - the model treats everything within it as credible.
“If the context is a highly realistic, professional discharge summary, the model assumes the content within it is accurate,” the researchers explained. This creates an attack vector: anyone who can craft a convincing clinical document can potentially influence what an AI system tells patients.
The Methodology
The Mount Sinai team used three types of test content:
-
Real hospital notes with embedded lies: Researchers took actual discharge summaries from the MIMIC database (a publicly available intensive care dataset) and inserted a single fabricated recommendation into each one.
-
Reddit health myths: Common medical misconceptions collected from health-related subreddits, presented in their original conversational format.
-
Physician-written scenarios: Three hundred short clinical situations written and validated by practicing physicians, designed to test models against realistic patient presentations.
This methodology matters because it reflects how misinformation actually spreads. People don’t usually encounter false medical claims in obvious formats. They find them in documents that look official, in posts from seemingly knowledgeable strangers, and in conversations that blend accurate information with dangerous errors.
What This Means
The findings complicate the push to deploy AI in healthcare settings. If models can’t distinguish true medical guidance from false claims embedded in professional-looking documents, their use in clinical environments carries measurable risk.
Lead author Mahmud Omar, a physician-scientist at Mount Sinai, suggested the study’s dataset could serve as a stress test: “Instead of assuming a model is safe, you can measure how often it passes on a lie.”
The researchers recommended that developers treat susceptibility to misinformation as a measurable property - one that should be tested before any clinical deployment. They proposed large-scale stress tests and external evidence verification as minimum requirements.
The Fine Print
The study has limitations. Testing 20 models with over a million prompts provides scale, but the specific claims tested came from a finite set. Real-world misinformation is constantly evolving, and new false claims emerge faster than benchmark datasets can track.
The 32% average acceptance rate also obscures significant variation. The best models (like GPT-4o at 10%) and worst models (above 60%) represent fundamentally different risk profiles. Deploying “an AI model” in healthcare means little without specifying which one.
Still, even a 10% error rate is concerning for medical applications. If one in ten responses to clinical scenarios includes potentially harmful misinformation, that’s not a rounding error - it’s a patient safety issue.
The study also doesn’t address what happens after a model accepts false information. Does it propagate the error in subsequent conversations? Does it correct itself when challenged? These questions remain open.
For hospitals and developers considering AI integration, the message is clear: test before you trust. The Mount Sinai dataset offers one way to do that.