DeepCOVID-XR: An Artificial Intelligence Algorithm to Detect COVID-19 on Chest Radiographs Trained and Tested on a Large US Clinical Dataset.
Researchers
Journal
Modalities
Models
Abstract
Background There are characteristic findings of Coronavirus Disease 2019 (COVID-19) on chest imaging. An artificial intelligence (AI) algorithm to detect COVID-19 on chest radiographs might be useful for triage or infection control within a hospital setting, but prior reports have been limited by small datasets and/or poor data quality. Purpose To present DeepCOVID-XR, a deep learning AI algorithm for detecting COVID-19 on chest radiographs, trained and tested on a large clinical dataset. Materials and Methods DeepCOVID-XR is an ensemble of convolutional neural networks to detect COVID-19 on frontal chest radiographs using real-time polymerase chain reaction (RT-PCR) as a reference standard. The algorithm was trained and validated on 14,788 images (4,253 COVID-19 positive) from sites across the Northwestern Memorial Healthcare System from February 2020 to April 2020, then tested on 2,214 images (1,192 COVID-19 positive) from a single hold-out institution. Performance of the algorithm was compared with interpretations from 5 experienced thoracic radiologists on 300 random test images using the McNemar test for sensitivity/specificity and DeLong’s test for the area under the receiver operating characteristic curve (AUC). Results A total of 5,853 patients (58±19 years, 3,101 women) were evaluated across datasets. On the entire test set, DeepCOVID-XR’s accuracy was 83% with an AUC of 0.90. On 300 random test images (134 COVID-19 positive), DeepCOVID-XR’s accuracy was 82% compared to individual radiologists (76%-81%) and the consensus of all 5 radiologists (81%). DeepCOVID-XR had a significantly higher sensitivity (71%) than 1 radiologist (60%, p<0.001) and higher specificity (92%) than 2 radiologists (75%, p<0.001; 84% p=0.009). DeepCOVID-XR’s AUC was 0.88 compared to the consensus AUC of 0.85 (p=0.13 for comparison). Using the consensus interpretation as the reference standard, DeepCOVID-XR’s AUC was 0.95 (0.92-0.98 95%CI). Conclusion DeepCOVID-XR, an AI algorithm, detected COVID-19 on chest radiographs with performance similar to a consensus of experienced thoracic radiologists. See also the editorial by van Ginneken.