In one case, two men who asked an AI tool about a brain bleed received opposite guidance, with one urged to seek immediate medical attention while the other was told to rest in a quiet room.
Researchers from Oxford University published the warning in the Nature Medicine journal on Tuesday following a large-scale randomised trial involving 1300 participants.
The research comes one month after OpenAI announced its ChatGPT Health service and revealed the popular AI tool handled more than 230 million health-related questions every day.
The study, created by researchers at the Oxford Internet Institute and University of Oxford, tested whether three large language models could help people identify a medical condition and the correct course of action to treat it.
Each of the 1298 participants were given one of 10 medical scenarios to pursue, with conditions ranging from the common cold and pneumonia to gallstones and a pulmonary embolism.
One group used traditional sources of information to identify their condition, such as web searches, while the other group used large language models GPT-4o used by ChatGPT, Llama 3 from Meta, and Command R+.
Using the AI tools, only 34.5 per cent of participants identified their medical condition, and 44.2 per cent found the right course of action to take, whether that was calling an ambulance or caring for themselves at home.
In one case, two participants describing the symptoms of a subarachnoid haemorrhage were given different guidance by GPT-4o, with only one told to seek medical help immediately, while indigestion or reflux was suggested when a participant described gallstones.
The findings highlighted some of the difficulties involved in creating technology to address human problems, University of Oxford Clarendon-Reuben doctoral scholar Rebecca Payne said.
"Despite all the hype, AI just isn't ready to take on the role of physician," Dr Payne said.
"Patients need to be aware that asking a large language model about their symptoms can be dangerous, giving wrong diagnoses and failing to recognise when urgent help is needed."
The AI tools made more accurate medical assessments when human participants were not involved in the questions, and correctly diagnosed conditions in 94.7 per cent of cases and provided the correct course of treatment more than half the time (56.3 per cent).
The findings suggested a communication breakdown between people and AI, Oxford Internet Institute researcher and lead author Andrew Bean said, and the need to find better ways to test software solutions.
"Interacting with humans poses a challenge even for top (tools)," he said.
"Designing robust testing for large language models is key to understanding how we can make use of this new technology."
18°C