A new study of real ChatGPT conversations finds that the AI often gives health advice that doesn’t take account of what the user has already told it about their health history.
As an example, one user asked for a muscle-toning routine, and ChatGPT suggested heavy compound and core exercises. That’s reasonable for most people. But this user had mentioned past small-intestine surgery, which meant that they were at risk for hernia or internal complications with heavy exercise.
The study, posted to arXiv in October by Shreyasi Roy Chowdhury, an independent researcher, and Kiran Garimella of Rutgers, is a preprint: no peer review, yet. The authors paid 1,252 ChatGPT users in India, Nigeria, Brazil and Pakistan to donate their full chat histories: 202,590 conversations from December 2022 to February 2026. And 15,555 of those were about health.
The study used Google’s Gemini 3.1 Pro Preview to score every ChatGPT answer for safety on a 1–7 scale. A score of 3 or lower counted as high-risk. Each answer was scored twice:
- On its own: the answer plus the few messages just before it.
- With the user’s history: the same answer, judged against health details the user had revealed in earlier chats, like a past surgery, a chronic condition, or suicidal thoughts.
What they found:
- Scored on their own, about 20% of health answers were high-risk (which is pretty bad already, if it’s true). More than half of the users who provided their chat history had received at least one high-risk answer.
- With the user’s history, risk went up. This scoring was only possible for users who had disclosed something earlier: 5,657 conversations. Their answers were riskier to begin with — 29% high-risk on their own — and counting their history raised that to 45%. Of the answers rated safe on the first pass, about 1 in 4 were rated unsafe for that person on the second.
- Newer models weren’t safer. From GPT-3.5 to GPT-5.1, answers got riskier in all four countries. The newest, GPT-5.2, broke the trend in India and Nigeria.
Last month, I wrote how hospitals are paying for patients to connect their medical records to ChatGPT. That means with every connected record ChatGPT gains more history. My assumption has been that this will make ChatGPT’s medical advice better, but this study suggests it may not be so.
Lots of issues with this preprint, even before peer review:
- the users were paid crowdworkers, not a representative sample.
- the judge was an AI (Gemini), not clinicians.
- the study assigned risk ratings, but didn’t assess actual harm
The authors point out that while we have a system to track a drug’s side effects after approval, we have no system to track an AI’s “side effects” after it gives medical advice (or other advice, for that matter). Likewise, last week I wrote that Utah’s AI prescribing experiment has no independent monitor, and this study suggests that ChatGPT has the same problem, and a lot more patients.
Source: Roy Chowdhury & Garimella, arXiv, Oct 2026
#DrYou #FiveMigrations #KnowledgeMigration