- An artificial intelligence (AI) algorithm increased aneurysm detection, identifying 55 true-positive brain aneurysms that radiologists had missed, representing a 39% relative increase in detection.
- AI and radiologists had different strengths: AI was more sensitive (84.6% vs 71.8%), whereas radiologists were more accurate at identifying an aneurysm (92.7% vs 78.2%).
- AI was particularly useful in higher-acuity settings, with the greatest benefit seen among inpatients and emergency department patients, while the benefit was more modest in outpatient settings.
- The findings suggest combining AI with physician interpretation could improve aneurysm detection. However, further research is necessary to determine whether increased detection improves patient outcomes.
A brain aneurysm is a weakened area in the wall of a blood vessel in the brain that can bulge outward. Unruptured brain aneurysms are relatively common, although many cause no symptoms and are discovered incidentally.
Radiologists can often detect brain aneurysms on CT imaging, particularly with CT angiography (CTA), which allows assessment of the aneurysm’s location, size, shape, and relationship to nearby blood vessels, helping determine appropriate clinical management.
Accurate detection of brain aneurysms on CT scans is crucial because early identification can help guide timely treatment and reduce the risk of potentially life-threatening complications.
Now, a new real-world study suggests that a Food and Drug Administration (FDA)-cleared AI algorithm could help detect more brain aneurysms on CT scans.
The prospective study involved 3,856 CTA examinations performed across the Northwell Health system. Researchers evaluated an AI algorithm developed by Aidoc, called aiOS, which is designed to detect intracranial aneurysms.
Importantly, the AI analyzed scans in parallel with routine clinical care, but radiologists did not have access to its results when initially interpreting the examinations.
This allowed researchers to assess how the technology performed in a real-world clinical environment, gauge how many additional aneurysms it could potentially identify, and determine the added value of combining AI with physician interpretation.
Overall, radiologists and the AI algorithm agreed in more than 96% of examinations.
“AI performance in clinical practice requires pre-deployment validation to understand not only the accuracy of the tool itself, but to test the expected operational utility in clinical practice,” explained lead study author Shlomit Stein, MD, FACR, Professor of Radiology at the Zucker School of Medicine at Hofstra/Northwell and Director of Artificial Intelligence in the Department of Radiology at Northwell Health.
“In our study, we found that AI identified 55 aneurysms not detected by initial radiologists, and that the operational metrics of the AI tool were overall favorable. AI tools can enhance the detection of findings by radiologists. In our study, the AI tool achieved a 39% relative enhanced detection rate, suggesting radiologists using AI will outperform those who do not.”
– Shlomit Stein, MD, FACR
The AI was more sensitive than radiologists alone, detecting 84.6% of aneurysms compared with 71.8% for radiologists, meaning it found more aneurysms that were truly present.
However, radiologists were more likely to be correct when they reported that an aneurysm was present. Their positive predictive value was 92.7%, compared with 78.2% for the AI.
Both radiologists and AI had similar strengths in ruling out aneurysms when none were present and in avoiding false alarms.
Notably, some of the additional aneurysms identified by the AI were among the smallest lesions. As such, this could provide clinicians with an opportunity to assess the patient’s risk and determine whether monitoring or treatment is appropriate, before a life threatening hemorrhage occurs.
“The AI tool demonstrated high sensitivity for aneurysm detection which in fact surpassed that of radiologists, implying a potentially high Radiologist-AI collaborative detection rate,” Stein told Medical News Today.
“AI was able to identify mostly small aneurysms missed by radiologists. Despite their small size, they may nevertheless be clinically important since risk is not only a function of aneurysm size, but also shape, location, and other patient-related risk factors,” she noted.
The researchers emphasized that the findings do not suggest that AI should replace radiologists. Instead, the results indicate that the two approaches may identify different aneurysms.
The AI detected 55 aneurysms that radiologists missed, while radiologists identified 30 that the AI did not detect.
Of 101 findings identified only by the AI, 46 were ultimately determined to be false positives. Despite this, the researchers found that the number of additional true aneurysm detections enhanced overall detection performance and outweighed the false-positive findings. Additionally, radiologists identified important aneurysms that the algorithm missed.
They suggest that this illustrates the complementary strengths of combining radiologists and AI.
The researchers also found that the usefulness of the AI differed depending on where patients were receiving care.
The strongest performance was observed among inpatient examinations. In this setting, the AI identified 18 additional aneurysms while producing seven false-positive alerts. The technology also performed favorably in the emergency department.
However, its benefit was more limited in outpatient care, where the AI identified four additional aneurysms but generated more false-positive than true-positive findings.
The researchers suggest that differences in patient populations and examination complexity could help explain these findings. Higher-acuity inpatient and emergency settings may involve more clinically complex cases, offering more opportunities for AI to serve as a complementary detection tool.
The study provides evidence that evaluating medical AI in routine clinical settings may reveal strengths and weaknesses that are not apparent during initial testing.
An algorithm can perform well in a controlled validation study but behave differently when exposed to the variety of patients, imaging equipment, clinical indications, and workflows found in everyday healthcare. As such, it is important to assess AI according to whether it improves physician performance and patient care in real-world settings.
The researchers also emphasized the importance of continued monitoring after AI systems are introduced into clinical practice.
The findings suggest that AI could serve as an additional layer of review when radiologists interpret brain CTA examinations.
While the technology was not perfect, radiologists also missed aneurysms that the AI identified. Rather than indicating that one approach is superior, the results point toward a potential benefit from combining the two.
For patients, the potential advantage is that an aneurysm that might otherwise go unnoticed could receive further clinical attention.
“Aneurysms carry the risk of rupture and potentially catastrophic brain hemorrhage. Some of these detected aneurysms will therefore require surveillance or preventative intervention. The first step is aneurysm detection, which we have shown can be aided by the use of AI,” Stein concluded.
However, the study does not establish that AI-assisted detection ultimately leads to better patient outcomes. Further research is still necessary to determine whether increased detection translates into meaningful reductions in aneurysm rupture or other complications.
For now, the research adds to evidence that AI may be most useful in radiology when it works alongside clinicians, helping identify findings that might otherwise be overlooked while leaving final interpretation and clinical decision making to physicians.
