“What we can offer is real people to speak to rather than an algorithm.”
Well, the BMJ (British Medical Journal) has just published a fascinating study exploring the use of so called Large language models (LLMs), powerful artificial intelligence (AI) models trained on extensive text data to generate human-like text, in a scenario involving meningitis.
Seven models were evaluated, including some from the big players Google and Microsoft, using the same hypothetical scenario of a patient presenting with the symptoms of acute bacterial meningitis.
The study explored the potentials and limitations of current LLMs and analysed their performance and alignment with good clinical practice and established medical guidelines regarding suggested diagnostic and treatment measures.
High stakes medical scenario
Bacterial meningitis was chosen for its life-threatening nature, urgency required in diagnosis and treatment and the range of differential diagnoses it involves, making it ideal for assessing performance in a realistic and high stakes medical scenario.
So, what did the study tell us? Well, as you might expect, it was something of a mixed bag.
One headline finding that leapt out was that misleading statements were generated in 52% of sessions.
But in 90% of the sessions, the LLMs accurately suspected a central nervous system (CNS) infection as a possible cause of the patient’s symptoms. Cranial imaging was recommended in 100% of sessions, lumbar puncture in 81% and blood cultures in 62%.
Patient’s history
In 57% of sessions, the LLMs recommended measuring vital parameters, taking the patient’s history and performing a physical examination as initial steps.
The most frequently mentioned differential diagnoses were stroke (86%), followed by intracranial/subarachnoid haemorrhage and brain tumour (both 48%). Other proposed differential diagnoses were migraine (19%), metabolic/endocrine disbalances (19%), medication side effects (10%), non-CNS infections (10%), severe hypertension (5%), drug intoxication (5%) and neurodegenerative disorders (5%).
Regarding treatment, 81% of responses stated that rapid administration of antibiotics is necessary.
Significant variations
The researchers concluded that the latest LLMs provide valuable advice on differential diagnosis and diagnostic procedures but significantly vary in treatment-specific information for bacterial meningitis when introduced to a realistic clinical scenario.
They said anyone using such systems must be aware of these limitations and performance variability when considering LLMs as a support tool for medical decision-making.
And that further research is needed to refine these models' comprehension of complex medical scenarios and their ability to provide reliable information.
Need for caution
For us at Meningitis Now, the study underscores the need for cautious and informed use of most of these models for diagnostic work and treatment of adult patients with bacterial meningitis.
However, it also provides a valuable opportunity to remind supporters and the general public to only use trusted information sources when it comes to seeking specific information about meningitis and its signs and symptoms. Such as, for example, our own website and information materials.
Of course, at the charity we can’t offer a diagnostic service and we would always advise people to contact their GP or NHS 111, or dial 999 in an emergency as a matter of urgency if meningitis is suspected.
Real people on our Helpline
But what we can offer is real people to speak to rather than an algorithm.
If meningitis has affected you or your life in any way then don’t forget that our confidential, nurse-led Helpline is here to provide emotional support, answer your questions and explain more about the support that we can provide.
With a wealth of knowledge and experience and a friendly, helpful and approachable attitude, there’s nothing artificial about them.
Find out more and how you can contact the Helpline here
- You can read the full study “Performance of large language models on advocating the management of meningitis”( Fisch U, Kliem P,Grzonka P, et al. Performance of large language models on advocating the management of meningitis: a comparative qualitative stud. BMJ Health Care Inform 2024;31:e100978. doi:10.1136/bmjhci-2023-100978) online here
