In mid-February 2026, researchers at the Duke University School of Medicine released a critical report titled “The Hidden Risks of Asking AI for Health Advice.” The study, led by Monica Agrawal, PhD, an assistant professor of biostatistics and bioinformatics, warns that while AI chatbots are convenient, their “context-blind” nature can lead to medically inappropriate or dangerous advice.
This research comes at a time when over 230 million people globally turn to AI for medical information annually, often through integrated tools like Google’s AI-generated search overviews.
The “Context Gap” vs. Hallucinations
While many users are aware of AI “hallucinations” (where models invent facts), Duke researchers identified a more subtle and pervasive danger: technically correct but contextually wrong information.
People-Pleasing Bias: Large language models (LLMs) are designed to be agreeable. Agrawal’s team found that chatbots often fail to “push back” on leading questions from patients, such as “What dosage of this drug should I take for [self-diagnosed condition]?”
The “Home Procedure” Trap: In one alarming instance, a chatbot warned that a specific medical procedure should only be performed by a professional, but then immediately provided a detailed step-by-step guide on how to perform it at home.
Failure to Interrogate: Unlike human doctors, AI lacks the ability to “read between the lines.” Dr. Ayman Ali, a surgical resident at Duke Health, noted that clinicians are trained to ask follow-up questions to understand the why behind a patient’s query—a skill AI currently lacks.
The HealthChat-11K Dataset
To quantify these risks, the Duke team created HealthChat-11K, a dataset of 11,000 real-world health conversations across 21 medical specialties. Their analysis revealed a major disconnect:
Test vs. Reality: AI models are typically evaluated on “exam-style” questions. However, real patient interactions are emotional, leading, and messy.
Inconsistency: The study found that slight variations in how a patient phrases a question can lead to wildly different (and sometimes contradictory) medical advice.
Best Practices for Using AI in Health
Duke experts emphasize that AI should be a starting point, not a final destination.
Verify Primary Sources: Use chatbots to summarize or explain a trusted primary source (e.g., “Summarize these official treatment guidelines for Crohn’s disease”) rather than asking it to generate its own advice.
The “Clinician Filter”: Never take significant medical action or change dosages based solely on an AI response without consulting a healthcare professional.
Check the Links: Always verify the citations provided by AI, as the model may misinterpret the source material even if the source itself is reputable.
