A new study shows that radiologists’ diagnostic accuracy outpaces commercial artificial intelligence (AI) tools.
In research published in Radiology, radiologists demonstrated superior diagnostic capabilities compared to commercially available AI tools when identifying common lung diseases through chest X-rays.
The research, which analysed more than 2,000 chest X-rays, raises questions about the current readiness of deep-learning-based AI for clinical diagnosis.
While AI has shown promise in aiding radiologists in interpreting chest radiographs, its real-world diagnostic accuracy has yet to be determined.
The retrospective study aimed to evaluate the performance of four commercially available AI tools in detecting airspace disease, pneumothorax and pleural effusion on chest radiographs.
It included adult patients who underwent chest radiography at four Danish hospitals in January 2020.
Two thoracic radiologists, and in cases of disagreement, three radiologists independently assessed chest radiographs to establish a reference standard.
The researchers calculated the area under the receiver operating characteristic curve, sensitivity and specificity. They also stratified these metrics based on the severity of findings, the number of results on chest radiographs and radiographic projection.
Statistical tests were employed for comparisons.
Four commercially available AI tools evaluated 2040 chest radiographs, achieving sensitivities ranging from 72-91%, 63-90%, and 62-95% for airspace disease, pneumothorax and pleural effusion, respectively.
AI tool specificity was high for radiographs with normal or single findings (range for airspace disease, 85-96%; pneumothorax, 99-100%; pleural effusion, 95-100%) but lower in radiographs with multiple findings (range, 27-69%, 96-99%, 65-92%, respectively) (P < .001).
False-positive rates were higher for AI tools than for radiology reports, whereas false-negative rates were similar.
The study’s findings indicate that current-generation AI tools exhibit moderate to high sensitivity in detecting airspace disease, pneumothorax and pleural effusion on chest radiographs.
They also produced more false-positive findings than radiology reports, particularly for smaller-sized target findings and when multiple findings are present.
Lead researcher Louis L Plesner, MD, resident radiologist and PhD fellow in the Department of Radiology at Herlev and Gentofte Hospital in Copenhagen, Denmark, commented: ‘Chest radiography is a common diagnostic tool, but significant training and experience is required to interpret exams correctly. While AI tools are increasingly being approved for use in radiological departments, there is an unmet need to further test them in real-life clinical scenarios. AI tools can assist radiologists in interpreting chest X-rays, but their real-life diagnostic accuracy remains unclear.’
The results highlight the ongoing challenges and limitations in integrating AI into clinical practice, emphasising the continued importance of radiologists in providing accurate diagnoses for patients with lung diseases.
While AI holds promise for enhancing healthcare, radiologists remain essential for ensuring the highest diagnostic accuracy in chest radiography.
Dr Plesner noted that these AI tools could boost radiologists’ confidence in their diagnoses by providing a second look at chest X-rays.
You can watch him discuss his research on how radiologists outperformed AI in identifying lung diseases on chest X-ray here.
Photo caption - Representative chest radiographs in six patients show (A, C, E) false-positive findings and (B, D, F) false-negative findings as identified by the artificial intelligence (AI) tools.
Photo credit: Radiological Society of North America
