Introduction

Chest X-rays are the most commonly performed medical imaging exam, and accurate and timely interpretation of chest X-rays is critical for providing high quality patient care1. However, it has been reported that physicians miss findings in approximately 30% of abnormal exams2,3, which may lead to delayed treatments, unnecessary costs, malpractice lawsuits, and preventable morbidity1,4,5,6,7,8,9,10,11. Physicians, regardless of specialty12, can make interpretational errors on radiographs and misinterpretations can arise from many factors including, but not limited to, the number of findings on a radiograph, amount of experience or diagnostic skill of the physician, fatigue, and cognitive or attentional constraints3,13,14,15,16,17,18,19,20,21. Further, the substantially growing workload for radiologists and the need for treating physicians to make time-sensitive decisions results in non-radiologist physicians being tasked with interpreting medical images despite lacking extensive radiology education and training16,22,23,24,25. Physicians without extensive radiology training are less accurate at interpreting chest X-rays26,27,28 and pose an even larger risk of misdiagnosis29,30.

Artificial intelligence (AI) systems using a subclass of AI called deep learning can accurately interpret and classify abnormal findings on chest X-rays31,32. These AI models are often highly successful, with performance matching, or even exceeding, that of radiologists33,34,35,36,37,38,39,40. However, many of these models have shortcomings, which limit the scope of their clinical applications. Some models only detect select findings on chest X-rays37,38,39,40,41,42,43,44,45,46,47,48,49,50,51 and others have large variations in the amount and quality of data used during training and testing38. Additionally, the majority of clinical AI models have not been assessed and cleared through FDAā€™s rigorous review process52. Finally, most AI models have only been shown to improve radiologistsā€™ accuracy41,53, but it is important to understand how other physician specialties may benefit from these AI systems because non-radiologist physicians often interpret chest X-rays22. For an AI model to be the most beneficial in reducing errors in chest X-ray interpretation, it should be able to detect multiple abnormalities, generalize to new patient populations, and be cleared by FDA to enable clinical adoption across physician specialties54,55.

In the current work, we describe our development of an FDA-cleared, computer assisted detection (CAD) system (device name: Chest-CAD, 510(k) number: K210666), which uses deep learning to assist physicians during their interpretation of chest X-rays. The AI system56 identifies suspicious regions of interest (ROIs) and assigns each ROI to one of eight clinical categories consistent with the reporting guidelines from the Radiological Society of North America (Cardiac, Mediastinum/Hila, Lungs, Pleura, Bones, Soft Tissues, Hardware, or Other)57. The eight categories exhaustively encompass any suspicious ROIs present in a chest X-ray and the AI system produces boxes around the ROIs. Figure 1 shows an example of the AI system output and how the AI system identified three ROIs, and classified them into two categories, Lungs and Mediastinum/Hila. See the Supplementary Information for more details about how the AI algorithm was trained for each category.

Figure. 1
figure 1

Chest X-ray unaided and aided by the AI system. (A) Chest X-ray unaided by the AI system (B) Chest X-ray aided by the AI system shows two Lung ROIs (green boxes) and a Mediastinum/Hila ROI (yellow box). The AI system identified abnormalities that were characterized as regions of interest in Lungs and Mediastinum/Hila. The abnormalities were bilateral upper lobe pulmonary fibrosis (categorized as ā€˜Lungsā€™), and pulmonary artery hypertension along with bilateral hilar retraction (categorized as ā€˜Mediastinum/Hilaā€™). The ROIs for each category are illustrated in different colors for readability.

Full size image

To test the efficacy of the AI system, we first assessed performance overall and per category in the absence of any interaction with a physician (referred to as standalone testing) on a dataset of 20,000 chest X-ray cases from 12 healthcare centers in the U.S, which was independent from the dataset used to train the AI system. We also tested the generalizability of our findings on a publicly available chest X-ray dataset. Next, we evaluated radiologist and non-radiologist physician performance with and without the assistance of the AI system (referred to as clinical testing). In this study we aim to find (1) the accuracy of the AI system, trained on a large, diverse dataset (see Table 1 for dataset characteristics), in identifying and categorizing abnormalities on chest X-rays, (2) the consistency of the AI systemā€™s standalone performance across multiple datasets, and (3) the performance of the radiologist and non-radiologist physicians in detecting and categorizing abnormalities on chest X-rays when aided by the AI system compared to when unaided.

Results

Evaluation of the AI system

We first evaluated the standalone performance of the AI system to determine its accuracy in detecting chest X-ray abnormalities relative to a reference standard defined by a panel of expert radiologists. Details about the reference standard are described in Methods and in Table 1. Standalone performance was assessed on 20,000 adult cases that met the inclusion criteria for the AI systemā€™s indications for use (patient ageā€‰>ā€‰=ā€‰22 and 1 or 2 image chest X-ray cases). Sampling procedures for the standalone testing dataset are described in the Methods. The AI system had a high level of agreement with expert radiologists and also generalized across patient and image characteristics. The AI system demonstrated high overall AUC (0.976, 95% bootstrap CI: 0.975, 0.976), sensitivity (0.908, 95% bootstrap CI: 0.905, 0.911), and specificity (0.887, 95% bootstrap CI: 0.885, 0.889) in identifying chest abnormalities in the standalone testing dataset. Figure 2 shows the number of positive cases in each of the eight categories, along with high sensitivity, specificity, and AUCs per category. The AI systemā€™s performance was also evaluated across patient and image characteristics in the standalone testing dataset. Supplementary Table 1 shows the AUC by patient sex, patient age group, number of images per case, brightness level, contrast level, resolution level, and X-ray device manufacturer. All AUCs were over 0.950, indicating that the AI system had similarly high performance across subgroups. The accuracy of the ROI location was also compared between the reference standard ROI and the AI systemā€™s ROI by measuring the intersection-over-union (IoU) per category and showed a high degree of overlap58 (see Supplementary Table 2).

Figure. 2
figure 2

Standalone performance of the AI system per category. Number of positive cases, sensitivity, specificity, and AUC per category in the standalone testing dataset ( n ā€‰=ā€‰20,000 patient cases). Error bars represent 95% confidence intervals calculated using bootstrap sampling (mā€‰=ā€‰1000).

Full size image

Performance of the AI system on the NIH ChestX-ray8 dataset

To test the accuracy of the AI system on an independent chest X-ray dataset, standalone performance was also assessed on a subset of the publicly available National Institutes of Health (NIH) data (ChestX-ray8)59. One thousand cases were randomly selected, and analyses were conducted on 922 cases that met the inclusion criteria for the AI systemā€™s indications for use. Details about the reference standard labels are in Table 1 and are described in Methods. The AI system demonstrated high overall AUC (0.975, 95% bootstrap CI: 0.971, 0.978), sensitivity (0.907, 95% bootstrap CI: 0.894, 0.919), and specificity (0.887, 95% bootstrap CI: 0.878, 0.896) in identifying chest abnormalities in the subset of data from the ChestX-ray8 dataset. The AI system also demonstrated high AUCs for each category, consistent with the AUCs in the standalone testing described above, suggesting that the performance of the AI system generalizes to distinct datasets.

Evaluation of physician performance

We tested physiciansā€™ accuracy in detecting abnormalities on chest X-rays when unaided and aided by the AI system through a multi-reader, multi-case (MRMC) study described in Methods. The AI system significantly improved physiciansā€™ accuracy at detecting chest abnormalities across all categories (pā€‰<ā€‰0.001; unaided AUC: 0.773; aided AUC: 0.874; difference in least squares mean AUCā€‰=ā€‰0.101, 95% CI: 0.101, 0.102). Figure 3a shows the Receiver Operating Characteristic (ROC) curves when physicians were unaided and aided by the AI system. Physician sensitivity increased from 0.757 (95% CI: 0.750, 0.764) when unaided to 0.856 (95% CI: 0.850, 0.862) when aided, demonstrating that physicians had a relative reduction in missed abnormalities of 40.74%. Physician specificity increased from 0.843 (95% CI: 0.839, 0.847) when unaided to 0.870 (95% CI: 0.866, 0.873) when aided, showing that the use of the AI system did not result in physicians overcalling abnormalities, but instead assisted in correctly identifying cases with no suspicious ROIs. Further, AUC values for every individual physician increased when aided by the AI system compared to when unaided (Fig. 3b; see Supplementary Table 5 for the 24 physiciansā€™ unaided and aided performance).

Figure. 3
figure 3

Physician performance unaided and aided by the AI system. A. ROC curve for all physicians when unaided (dashed red line, red circle) and aided (solid blue line, blue triangle) by the AI system. B . AUCs for all physicians when unaided (red bar and circles) and aided (blue bar and triangles) by the AI system. Solid lines represent an individual physicianā€™s improvement when unaided and aided.

Full size image

As expected, radiologists were more accurate than non-radiologist physicians (emergency medicine, family medicine, and internal medicine physicians) at identifying abnormalities in chest X-rays when unaided by the AI system (Fig. 4). Despite high unaided accuracy, radiologists still showed an improvement when assisted by the AI system with an unaided AUC of 0.865 (95% CI: 0.858, 0.872) and an aided AUC of 0.900 (95% CI: 0.895, 0.906). Internal medicine physicians showed the largest improvement when assisted by the AI system with an unaided AUC of 0.800 (95% CI: 0.793, 0.808) and an aided AUC of 0.895 (95% CI: 0.889, 0.900; see Fig. 4). There was a significant difference in AUC between radiologists and non-radiologist physicians in the unaided condition (pā€‰<ā€‰0.001), but no significant difference in the aided condition (pā€‰=ā€‰0.092), suggesting that the AI system aids non-radiologist physicians to detect abnormalities in chest X-rays with similar accuracy to radiologists. Radiologists and non-radiologist physicians also experienced a relative reduction in missed abnormalities of 29.74% and 44.53%, respectively. The AUC values overall and per category for radiologists and non-radiologist physicians when unaided and when aided are reported in Supplementary Table 3.

Figure. 4
figure 4

Performance by physician specialty. Average AUC for radiologists, emergency medicine physicians, family medicine physicians, and internal medicine physicians when unaided (red) and aided (blue) by the AI system. Error bars represent 95% confidence intervals calculated using bootstrap sampling (mā€‰=ā€‰1000).

Full size image

In addition to physicians demonstrating improved detection accuracy, assistance by the AI system reduced physiciansā€™ case read times. Non-radiologist physicians detected abnormalities in chest X-ray cases significantly faster when aided by the AI system versus when unaided, with an average improvement of 10 s (t (1,17)ā€‰=ā€‰2.281, pā€‰=ā€‰0.036 (uncorrected), 7.94% faster, aided median read timeā€‰=ā€‰98.5 s, unaided median read timeā€‰=ā€‰107 s). There was no difference in radiologist read times when aided by the AI system versus when unaided (t (1,5)ā€‰=ā€‰0.267, pā€‰=ā€‰0.800, aided median read timeā€‰=ā€‰67.5 s, unaided median read timeā€‰=ā€‰69.5 s; see Supplementary Table 4).

Discussion

The FDA-cleared AI system demonstrated strong standalone performance, detecting chest abnormalities on X-ray on par with radiologists, and the AI systemā€™s performance generalized to a separate publicly available chest X-ray dataset. In the clinical testing, we showed that overall physician accuracy improved when aided by the AI system, and non-radiologist physicians were as accurate as radiologists in evaluating chest X-rays when aided by the AI system. Taken together, our findings show that the AI system supports different physician specialties in accurate chest X-ray interpretation.

The AI system was trained using deep learning and performed with high accuracy in the standalone testing dataset, with an overall AUC of 0.976. Accuracy remained high across the eight categories of the AI system (all AUCsā€‰>ā€‰0.92) and across different patient and image characteristics (all AUCsā€‰>ā€‰0.95). Additionally, we demonstrated the generalizability of the AI system by evaluating performance on the NIHā€™s publicly available chest X-ray dataset. The system generalized well, with similarly high performance on both datasets (NIH dataset AUC: 0.975; standalone testing dataset AUC: 0.976). The high performance of the AI system can be partly attributed to the large and diverse training dataset and quality of the labels. We manually created over six million labels from nearly 500,000 radiographs. Consistent with our previous work, the labels were from expert U.S. board-certified radiologists to ensure high clinical accuracy and generalizability of the machine learning algorithms60,61. Prior work with AI systems for radiography have relied on labels extracted retrospectively from radiology reports using NLP techniques36,62,63,64, which can result in inaccuracies65,66,67. As such, there is a growing consensus that high-quality clinical labels are essential to high-performing AI systems68,69,70.

The AI system has the potential to positively impact patients and the healthcare system as demonstrated by physiciansā€™ consistently higher accuracy at detecting abnormalities on chest X-rays when assisted by the AI system. Every physician in the study improved when aided compared to unaided, suggesting each physician, even those that are highly skilled at chest X-ray interpretation, benefited from using the AI system. There was no difference in accuracy between radiologists and non-radiologist physicians when aided by the AI system. Further, non-radiologist physicians were faster to detect abnormalities on chest X-rays when aided compared to when unaided. This suggests that the AI system increased non-radiologist physiciansā€™ accuracy and efficiency, such that they performed more similarly to radiologists. Improving the accuracy of chest X-ray interpretation with the AI system has the potential to reduce misdiagnosis, leading to improved quality of care and decreased costs, and should be directly measured in future prospective trials.

Improving non-radiologist physiciansā€™ ability to interpret chest X-rays is critical for improving patient outcomes. Non-radiologist physiciansā€™ are often the first to evaluate patients in many care settings and routinely interpret chest X-rays when a radiologist is not readily available27. Additionally, the share of practicing radiologists relative to other physician specialties is declining in the U.S.71 , particularly in rural counties71,72,73. As a result, practicing radiologists are overburdened and even large institutions, such as the Veterans Affairs hospitals, have backlogs of hundreds of thousands of unread X-ray cases74. Therefore, non-radiologist physicians are being tasked with interpreting chest X-rays, despite lack of training and poorer accuracy22,23,24,25. The majority of prior studies have conducted clinical testing on AI systems with radiologists only48,50,53. Here, we provide evidence that the AI system aids non-radiologist physicians, which can lead to greater access to high quality medical imaging interpretation and may reduce misdiagnoses on chest X-rays when radiologists are unavailable.

There were limitations in this study. First, in the clinical study physicians were not given access to additional patient information for the chest X-rays in the study. This was an intentional study design decision to keep the information provided to the physician and the AI system the same. Second, future prospective studies will be necessary to measure the real-world impact of the AI system on directly reducing physiciansā€™ misdiagnoses and subsequently improving patient outcomes. The AI system is well positioned to be implemented and integrated into clinical settings due to its FDA-clearance56 and category outputs that align with standard radiology reporting guidelines57. Third, a subset of positive cases were labeled for the specific abnormality detected by the expert radiologistsā€™ and the performance of the AI system was evaluated in a more granular manner, however, future analyses with a larger sample size is required to investigate the performance for lower prevalence conditions (see Supplementary Table 6). Finally, while the AI systemā€™s categories were designed to be mutually exclusive, there is a possibility that the expert radiologists interpreted the same abnormality on an X-ray differently and selected two different categories as abnormal when providing ground truth labels. Since the reference standard is determined by the majority opinion, this situation could result in no suspicious ROI being found, which may cause the reported sensitivity values to be higher than the true performance. The expert radiologists received extensive training before providing labels as to the types of abnormalities associated with each category to minimize any confusion about the definitions for each category (see Supplementary Information AI System Categories).

Overall, the FDA-cleared AI system, trained using deep learning, demonstrated strong standalone performance and increased physician accuracy in detecting abnormalities on chest X-rays. It eliminated the gap in accuracy between radiologists and non-radiologist physicians when detecting abnormalities on chest X-rays and facilitated non-radiologist physicians to read cases more efficiently. Thus, the AI system has the potential to increase timely access to high-quality care across the U.S. healthcare system and improve patient outcomes.

Methods

AI system development

To build the AI system, 17 U.S. board-certified radiologists with a median of 14 years of experience manually annotated a development dataset of 341,355 chest X-ray cases, generating a total of 6,202,776 labels. The development dataset consisted of 492,996 de-identified radiographs from 185,114 patients collected from 15 hospitals, outpatient care centers, and specialty centers in the United States. The development dataset was then randomly split into a training set (326,493 cases; 471,358 radiographs) and a tuning set (14,862 cases; 21,638 radiographs). See Table 1 for details about the labels, as well as the patient and image characteristics of the development dataset.

Table 1 Dataset characteristics for the development, standalone testing, clinical testing, and subset of NIH ChestX-ray8 datasets.
Full size table

To ensure that the radiologists produced high-quality labels for the AI system, each radiologist completed rigorous, multi-step training programs. The expert radiologists were responsible for jointly localizing abnormalities on all radiographs within the case (the localized area was considered the ā€œreference standardā€ area) and producing case-level judgments as to the presence or absence of abnormalities for each of the eight categories. Each case was interpreted by one to six radiologists. Categories were based on standard radiology reporting guidelines as defined by Radiological Society of North America57. The expert radiologists were instructed to look for abnormalities when labeling a suspicious ROI in each category. For example, in the Cardiac category, specific abnormalities the expert radiologists were instructed to look for included cardiomegaly, suspected pericardial effusion, chamber enlargement, valvular calcification, dextrocardia, dextroposition, constrictive pericarditis, coronary artery calcifications, suspected pericardial cyst, and obscured cardiac silhouette. The AI system categories are explained in more depth in Supplementary Information.

The AI system was trained using deep learning to recognize visual patterns that correspond to abnormalities within each category through a supervised training process. The system takes a chest X-ray case as input and produces a binary output indicating the presence or absence of an abnormality for each category. It also produces category-specific bounding box(es) when an abnormal region is identified. The binary outputs for each of the eight categories were defined to be mutually exclusive and collectively exhaustive, so that any clinically relevant abnormality detected by any of the expert radiologists was included in one of the categories.

Algorithm design

The AI systemā€™s processing of a chest X-ray case consisted of three stages: pre-processing, analysis, and post-processing.

Pre-processing stage

Input radiographs from a given chest X-ray case were automatically pre-processed in order to standardize their visual characteristics. Each radiograph, which was required to be a high resolution input (over 1440 pixels), was first cropped to remove excess black padding around the edges. Next, resizing operations that preserved the aspect ratio were applied to standardize image resolution to a height of 800 pixels. The resizing operations downscaled the image with anti-aliasing filters on and with bilinear interpolation. If necessary, in order to reach a target width of 1,024 pixels, the resize operation added padding to the edges of the image. The image was then cropped, if necessary, to a target width of 1,024 pixels.

Analysis stage

The analysis stage took the pre-processed radiographs of a chest X-ray case and used a convolutional neural network to create two outputs per category. One was a Bayesian posterior probability distribution characterizing the modelā€™s belief about the presence of an abnormality within the category. The other was a pixel-wise probability map representing an estimate of where any such abnormality would be within the case. A high-level schematic of the architecture is shown in Fig. 5.

Figure. 5
figure 5

Neural network architecture. The deep convolutional neural network architecture for the AI system is made up of the feature encoder (green), the classification branch (yellow), and the localization branch (red). The feature encoder generates a latent representation of each radiograph. The classification and localization branches use the latent representations of radiographs to produce a probability model of expert radiologists identifying abnormalities for a category and a probability map of where the expert radiologists may identify abnormality for a category, respectively.

Full size image

The model produced case-level decisions by jointly analyzing one or more input radiographs from a given case. First, a single case-level embedding was created by separately running each radiograph through an image encoder, producing one 512-dimensional feature vector per radiograph. Each of the 512-dimensional vectors was then passed through a fully connected layer to create 1024-dimensional image feature vectors. Next, the image encodings were collapsed across case-specific radiographs into a single 1024-dimensional feature vector through max-pooling. The image encoder used for this architecture was Resnet-3475, which was chosen for parsimony from several options that all yielded similar results (e.g., Densenet). It is a widely used backbone for many computer vision tasks and has a low computational cost compared to many other common architectures. Probability maps for localization were produced by the model through a convolutional layer applied to feature maps from the image encoderā€™s penultimate layer.

The case embedding was run through a fully connected layer to produce the output parameters ā€œalphaā€ and ā€œbetaā€ for each category, which parameterized a beta-binomial distribution. From a Bayesian perspective, this distribution represents the modelā€™s prediction for how many k expert radiologists out of a group of n expert radiologists are likely to identify an abnormality when asked to interpret the chest X-ray. This likelihood model accounts for the observed spread of the available k-of-n binomial labels, where, as previously described, the binomial labels arose by having multiple expert radiologists review many of the cases within the development dataset. At inference time, a point estimate defined by the mean of the distribution was used by the post-processing stage of the software to determine the presence of a potential abnormality.

Post-processing stage

The system took the output from the model and applied post-processing operations to create two outputs per category: (1) a binary determination representing the systemā€™s prediction of whether any abnormalities for that category were present within the X-ray case and (2) a set of bounding boxes surrounding any such abnormalities.

The binary determination was calculated from the point estimates produced by the model using a category-specific threshold pre-computed on the tuning dataset. Any score lying on or above the threshold resulted in an abnormality-present determination, and any score below the threshold resulted in an abnormality-absent determination. The thresholds were optimized to yield equal sensitivity and specificity per category on the tuning set. The set of bounding boxes were created from the categoryā€™s pixel-wise probability map output. This was done using a heuristic that placed boxes around the site of high-probability regions using pre-computed, category-specific thresholds that binarized the probability map.

Model training

The model was trained by minimizing a joint loss function that assessed the modelā€™s ability to correctly predict the case-level classifications (ā€œROI presentā€ or ā€œROI absentā€ for each output category) and the ability to correctly predict the location of the abnormality within individual radiographs. The joint-loss function was defined as a weighted sum of two terms for each category and then summed across categories. The first term was the across-radiograph average per-pixel binary cross-entropy loss between a given radiographā€™s predicted probability map and the ground truth map for that radiograph. The second term was the negative log-likelihood of the beta-binomial distribution, where the distributionā€™s two free parameters (denoted as ā€œalphaā€ and ā€œbetaā€ in Fig. 5) were outputs of the model. The binomial observations used in the beta-binomial negative log-likelihood correspond to k out of n labeling radiologists indicating the presence of an abnormality for the given case and category; note that n and k vary per case, where some cases were labeled by only a single expert radiologist and others were labeled by multiple independent expert radiologists. To increase the robustness of the model, data augmentation was used during training. Radiographs were randomly rotated, vertically or horizontally flipped, gamma-corrected, contrast-adjusted, and cropped.

The training algorithm repeatedly iterated through the training set in randomized batches of 64 cases. The parameters of the model were updated after processing each batch to minimize the aforementioned loss function. This minimization was achieved using a variant of the stochastic gradient descent algorithm called AdamW76. After each epoch, the model was evaluated on the tuning set. An early stopping criterion (across-category mean AUC on the tuning set) was used to determine the continuation of model training based on whether an improvement occurred in the last 10 epochs.

After training finished, the chest X-ray cases in the tuning set were run through the trained model. Prediction scores on cases in the tuning set were used to compute the operating point for each category in the final model. The resulting decision thresholds (one per category) were then fixed and held constant prior to testing.

Standalone evaluation

Standalone testing dataset

The AI systemā€™s performance was evaluated on a standalone testing dataset consistent with FDA guidelines77. The standalone testing dataset consisted of 20,000 chest X-ray cases that were retrospectively randomly sampled from 12 hospitals and healthcare centers between July and December of 2017 to create a set of patient cases that was representative of the AI systemā€™s intended use (see Fig. 6). The 12 hospitals and healthcare centers were a subset of the 15 hospitals and healthcare centers that were used for the training dataset. There was a diligent segmentation process to minimize patient overlap between the testing and training sets. There was an overlap of 0.33% of the patients in the training dataset with the patients in the standalone testing dataset, however, the impact of these overlapping patients was negligible on the study results as the performance remained unchanged when these patients were removed from the analysis (overall AUC remained 0.976). Reference standard labels for the standalone dataset were provided by 17 U.S. board-certified radiologists with a median of 14 years of experience post-residency. Each case was independently reviewed by three expert radiologists, and if any findings associated with the category (see Supplementary Information AI System Categories) was identified then the expert radiologist would determine a positive ROI for that category. The reference standard for each category was determined by the majority opinion (at least two of three) of the reviewing expert radiologists. During labeling, each expert radiologist was instructed to make a binary decision on the presence or absence of an abnormality for each of the categories, for each chest X-ray case they were assigned. Twenty thousand cases ensured a large natural positive incidence rate for each category with at least 600 reference standard positive cases per category. See Table 1 for the number of positive cases per category, details about the labels used to create the reference standard, and the patient and image characteristics for the standalone testing dataset. See Fig. 6 for data sampling procedures for the testing datasets. There was no case overlap between the development dataset and the standalone testing dataset.

Figure. 6
figure 6

Chest X-ray sampling procedures. Flow diagram illustrating chest X-ray sampling procedures for the standalone and clinical testing datasets.

Full size image

Performance of the AI system was evaluated using the area under the curve (AUC) of the receiving operator characteristic (ROC) curve per category and overall. The ā€œoverallā€ calculations were performed by aggregating the data from all categories and all cases through a technique called micro-averaging78. The sensitivity and specificity were also calculated per category and overall. The 95% confidence intervals were reported for each metric using bootstrap resampling (mā€‰=ā€‰1000).

To measure the localization accuracy of the AI systemā€™s ROIs, the Intersection-over-Union (IoU) was used to calculate the overlap of the predicted bounding box(es) and the reference standard bounding box(es) for each category. When multiple predicted bounding boxes were present for a category, the areas were summed to find the total predicted area. Similarly, when multiple reference standard bounding boxes were found for a category, the areas were also summed to find the total reference standard area. The IoU is determined by finding the area of the intersection of the predicted area with the reference standard area and dividing by the union of the predicted area with the reference standard area. The reported aggregate IoU per category is the average IoU of the true positive cases.

To measure the performance of the AI system on identifying category-specific ROIs when particular abnormalities are present on chest X-rays, labels were collected from the U.S. board-certified radiologists on a subset of 400 to 800 positive cases per category as determined by the reference standard. The abnormality labels were collected on a random subset of positive cases, therefore, not all abnormalities were represented in the investigation and only those found on over 20 cases were analyzed. The volume of cases with each type of abnormality identified can be found in Supplementary Table 6 along with the AI systemā€™s sensitivity in detecting the corresponding category-specific ROI.

NIH ChestX-ray8 dataset

The AI system performance was evaluated on a subset of cases from the ChestX-ray8 dataset from the National Institutes of Health to confirm the robustness of the AI system. The complete ChestX-ray8 dataset is made up of over 100,000 de-identified chest X-rays obtained from over 30,000 patients59. We randomly sampled 1,000 cases and 78 cases were removed because they did not meet the inclusion criteria for the AI systemā€™s indications for use. Thirteen U.S. board certified radiologists with a median of 14 years post-residency labeled each of the 922 cases for the presence or absence of suspicious ROIs for each of the eight categories. We established the reference standard using majority opinion by randomly selecting labels from three of the U.S. board certified radiologists. See Table 1 for details about the labels, as well as the patient and image characteristics of the NIH dataset. The sensitivity, specificity, and AUC of the ROC curve was calculated overall and per category.

Physician evaluation

To evaluate whether physician accuracy improved when aided compared to unaided by the AI system, we conducted a multi-reader, multi-case (MRMC) study consistent with FDA guidelines79. Twenty-four physicians were enrolled to represent the intended use population. The physician specialties included radiologists (nā€‰=ā€‰6), internal medicine physicians (nā€‰=ā€‰6), family medicine physicians (nā€‰=ā€‰6), and emergency medicine physicians (nā€‰=ā€‰6). Physicians had a mean 14.9 years of experience (range: 1ā€“38 years).

Each physician read 238 X-ray cases (190 cases with at least one abnormality and 48 cases with no abnormalities) unaided and aided by the AI system. Physicians were not given access to additional patient information beyond the chest X-ray. The chest X-ray cases used in the clinical testing dataset were retrospectively randomly sampled from 12 hospitals and healthcare centers between July and December of 2017. Enrichment procedures ensured that there were at least 10 positive cases for each category in the clinical testing dataset. See Table 1 for details about the labels used to create the reference standard, as well as the patient and image characteristics for the clinical testing dataset. The reference standard labels were developed with the same methods outlined in the standalone testing dataset section above. See Fig. 6 for data sampling procedures for the clinical testing dataset. There was no case overlap between the development dataset, the standalone testing dataset, and the clinical testing dataset.

A power analysis was conducted in advance of the study, and it determined that using 24 physicians and 238 cases would provide over 90% power to detect a difference in aided versus unaided AUCs of 0.04. The study consisted of two independent reading sessions separated by a washout period of at least 28 days to reduce memory bias. Physicians read all cases twice. In the first session, half the cases were aided by the AI system and the other half were unaided. In the second session, all cases were read in the opposite condition. Cases were assigned using randomized stratification to reduce case order effects. Each physician was asked to determine the presence or absence of an abnormal region of interest for each of the eight categories for every case and provide a confidence score (0ā€“100) for each of their judgments.

To determine whether there was a statistically significant difference between physician performance when aided versus unaided, we used the Dorfman, Berbaum, and Metz (DBM) model. The DBM model is one of the most common methods for estimating differences in the area under the curve (AUC) of the receiver operating characteristic (ROC) curve and for calculating the corresponding confidence intervals for MRMC studies80,81. We also calculated the sensitivity and specificity for each physician type, as well as 95% bootstrap confidence intervals. In addition, we calculated the relative reduction in miss rate rate by subtracting physiciansā€™ aided miss rate (1 - sensitivity) from their unaided miss rate (1 - sensitivity), and then dividing by their unaided miss rate (1 - sensitivity).

To examine the impact of the AI system on physicians who specialize in medical image interpretation versus those who do not specialize in medical image interpretation, we separated the physicians into two groups, radiologists and non-radiologist physicians (internal medicine, family medicine, and emergency medicine physicians). The ā€œoverallā€ AUCs for radiologists and non-radiologist physicians were found by aggregating the data per physician group from all categories and all cases by micro-averaging78. To evaluate the performance between radiologists and non-radiologist physicians AUCs in the unaided and aided setting, we performed the Delong test82 and evaluated the p-values against a Bonferroni-adjusted alpha level of 0.025 (0.05/2). We again calculated the relative reduction in miss rate for radiologists and non-radiologist physicians, respectively, by subtracting each groupā€™s aided miss rate (1- sensitivity) from their unaided miss rate (1 - sensitivity), and then dividing by their unaided miss rate (1 - sensitivity). To evaluate if the AI system significantly improved case read times, we first took the log transform of the read times to create a normal distribution. Then, we performed a paired t-test to compare log read times in the unaided and aided settings for radiologists and non-radiologist physicians, respectively.

Ethics approval

IRB approval was obtained from the New England IRB (study #1283806).