Conformal Triage for Medical Imaging AI Deployment
Preprint 2024 en
Authors
AA
Anastasios N. Angelopoulos
SP
Stuart R. Pomerantz
SD
Synho Do
Abstract
2 min read
Abstract Background The deployment of black-box AI models in medical imaging presents significant challenges, especially in maintaining reliability across different clinical settings. These challenges are compounded by distribution shifts that can lead to failures in reproducing the accuracy attained during the AI model’s original validations. Method We introduce the conformal triage algorithm, designed to categorize patients into low-risk, high-risk, and uncertain groups within a clinical deployment setting. This method leverages a combination of a black-box AI model and conformal prediction techniques to offer statistical guarantees of predictive power for each group. The high-risk group is guaranteed to have a high positive predictive value, while the low-risk group is assured a high negative predictive value. Prediction sets are never constructed; instead, conformal techniques directly assure high accuracy in both groups, even in clinical environments different from those in which the AI model was originally trained, thereby ameliorating the challenges posed by distribution shifts. Importantly, a representative data set of exams from the testing environment is required to ensure statistical validity. Results The algorithm was tested using a head CT model previously developed by Do and col-leagues [9] and a data set from Massachusetts General Hospital. The results demonstrate that the conformal triage algorithm provides reliable predictive value guarantees to a clinically significant extent, reducing the number of false positives from 233 (45%) to 8 (5%) while only abstaining from prediction on 14% of data points, even in a setting different from the training environment of the original AI model. Conclusions The conformal triage algorithm offers a promising solution to the challenge of deploying black-box AI models in medical imaging across varying clinical settings. By providing statistical guarantees of predictive value for categorized patient groups, this approach significantly enhances the reliability and utility of AI in optimizing medical imaging workflows, particularly in neuroradiology.
Shandong Wu, David Vorp, Ashok Panigrahy, John R. Gilbertson, Wendie A. Berg, Seong Je Hwang, Kayhan Batmanghelich, Rivka R. Colen, Lucas Peter, Robert M. Nishikawa
Josh Hanson, Sue J. Lee, Sanjib Mohanty, Maryam Faiz, Nicholas M. Anstey, Ric N. Price, Prakaykaew Charunwatthana, Emran Bin Yunus, Saroj K. Mishra, Emiliana Tjitra, Ridwanur Rahman, François Nosten, Ye Htut, Richard J. Maude, Tran Thi Hong Chau, Nguyen Hoan Phu, Tran Tinh Hien, Sir Nicholas White, Nicholas Day, Arjen M. Dondorp
Discussion(0)
No comments yet. Be the first to comment.