Even a microscopic amount of deliberately planted misinformation in an artificial intelligence system's training data can undermine the reliability of the entire model, a new study from New York University warns. The findings, published in the journal Nature Medicine and first reported by Ars Technica, carry particular weight for the healthcare sector, where AI tools are increasingly used to assist in clinical decisions.
The researchers demonstrated that if just 0.001 percent of a large language model's training data is 'poisoned'—meaning false information is intentionally inserted—the model becomes prone to propagating errors. More troubling, these corrupted models still perform as well as their clean counterparts on standard benchmarks used to evaluate medical AI, making the contamination difficult to detect through conventional testing.
'In view of current calls for improved data provenance and transparent LLM development, we hope to raise awareness of emergent risks from LLMs trained indiscriminately on web-scraped data, particularly in healthcare where misinformation can potentially compromise patient safety,' the team wrote in the paper.
Cheap and Easy to Poison
To test the vulnerability, the NYU team intentionally injected AI-generated medical misinformation into 'The Pile,' a widely used training dataset that includes high-quality medical corpora such as PubMed. They generated 150,000 medical articles in just 24 hours, then inserted a small fraction into the data.
The results were stark: replacing just one million of 100 billion training tokens—a mere 0.001 percent—with vaccine misinformation led to a 4.8 percent increase in harmful content. This was achieved by injecting 2,000 malicious articles (roughly 1,500 pages) that the researchers produced for only US$5.00.
Unlike more invasive hacking methods that require direct access to a model's internal weights, data poisoning is far simpler. Attackers only need to host harmful information online where it can be scraped into training datasets, the researchers noted.
Hidden Dangers in Healthcare
The study underscores the growing risks of deploying AI in medicine. The New York Times reported last year that MyChart, an AI-powered platform that drafts replies to patients' questions on behalf of doctors, regularly 'hallucinates' untrue entries about a patient's condition—a sign that the problem is already present in real-world applications.
The paper urges caution: 'AI developers and healthcare providers must be aware of this vulnerability when developing medical LLMs. LLMs should not be used for diagnostic or therapeutic tasks before better safeguards are developed, and additional security research is necessary before LLMs can be trusted in mission-critical healthcare settings.'
The research adds to a growing body of evidence that AI systems, while powerful, remain fallible—and that their use in high-stakes fields like medicine demands rigorous oversight.