Deep Convolutional Neural Networks for Accurate Diagnosis of COVID-19 Patients Using Chest X-Ray Image Databases from Italy, Canada, and the USA

Introduction: Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), famously known as COVID-19, has quickly become a global pandemic. Chest X-ray (CXR) imaging has proven reliable, fast, and cost-effective for identifying COVID-19 infections, which presents with atypical unilateral patchy infiltration in the lungs like typical pneumonia. We employed the deep convolutional neural network (DCNN) ResNet-34 to detect and classify CXR images from patients with COVID-19, other viral pneumonias, and normal controls. Methods: We created a single database, containing 781 source CXR images for COVID-19 (n=240), other viral pneumonias (n=274), and normal controls (n=267) from four different international sub-databases: the Società Italiana di Radiologia Medica e Interventistica (SIRM), the GitHub Database, the Radiology Society of North America (RSNA), and the Kaggle Chest X-Ray Database. Images were resized, normalized without any augmentation, and arranged in m batches of 16 images before supervised training, testing, and cross-validation of the DCNN classifier. Results: The ResNet-34 had a diagnostic accuracy as of the receiver operating characteristic (ROC) curves of the truepositive rate versus the false-positive rate with the area under the curve (AUC) of 1.00, 0.99, and 0.99, for COVID-19, other viral pneumonia, and normal control CXR images, respectively. This accuracy implied identical high sensitivity and specificity values of 100%, 99%, and 99% for the three groups, respectively. ResNet-34 achieved identical sensitivity and specificity of 100%, 99.6%, and 98.9% for classifying CXR images of the three groups, with an overall accuracy of 99.5% for the testing subset for diagnosis/prognosis. Conclusion: Based on this high classification precision, we believe that the output activation map of the final layer of the ResNet-34 is a powerful tool for the accurate diagnosis of COVID-19 infection from CXR images.


Introduction
Since December 2019, the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), famously known as COVID-19, has quickly become a global pandemic. In fact, as of October 14, 2021, reports to the World Health Organization (WHO) have amounted to 239,007,759 confirmed cases, including 4,871,841 deaths. [1] The pandemic continues to have disastrous effects on health, industry, and social life worldwide. The disease's long-term symptoms range from flu-like signs to severe respiratory distress syndrome. Patients exhibit evident anomalies (i.e., opacities and lung consolidations) in chest X-ray (CXR) images. [2,3] Rapid and accurate detection of COVID-19 is crucial to control outbreaks in the community and hospitals. [4] Reverse transcriptase quantitative polymerase chain reaction (RT-qPCR) testing, which can detect SARS-CoV-2 RNA in different types of clinical specimens, is the primary technique used for diagnosis. Nevertheless, RT-qPCR testing is time-consuming, complex, and its sensitivity is highly variable with a high probability of false detection. [2,5] Medical imaging techniques such as CXR and computed tomography (CT) have proven reliable, fast, and cost-effective for detecting lung diseases and identifying COVID-19 infections compared to RT-qPCR. [6] Al-  [11] Radiology CT ML Lungs [12] Radiology MRI ML Brain [13] Radiology X-ray ML Chest [14] Hematology Laboratory ML Biological hematology [15] Hematology HSCT ML aGVHD [16] Hematology Flow cytometry AI Leukemia [17] Hematology Flow cytometry AI Leukemia [18] Hematology Cytometry AI AML [19] Hematology Laboratory DM PV Abbreviations: aGVHD, acute graft-versus-host disease; AI, artificial intelligence; AML, acute myeloid leukemia; CT, computed tomography; DM, data mining; HSCT, hematopoietic stem cell transplantation; MRI, magnetic resonance imaging; PV, Polychthemia Vera.
though CXR images do not show any irregularities at the beginning of infection, as COVID-19 proceeds, it slowly presents with atypical unilateral patchy infiltration in the lungs, which exhibit similarities to the characteristics of other pneumonias. [7] Computer vision algorithms have shown high performance in object detection, image classification, and indexing for retrieval and annotation at a large scale. [3] Of these, deep convolutional neural networks (DCNN) with transfer learning have distinguished themselves in many computervision tasks, such as image classification, especially in the field of radiology for computer-aided diagnosis of chronic lung diseases from CT Hounsfield unit values or texture features. [3,8,9] Transfer learning is a supervised machine learning technique, which reuses parts of a previously trained model in a specific field for another task in a new model, with the advantages of using less training data, better generalization, and affordable deep learning. [10][11][12][13][14][15] Examples of the machine learning and artificial intelligence techniques already being tested or applied in various medical fields are summarized in Before applying the database of random inputs, CXR images were all pre-processed-i.e., resized to 224 × 224 to match input requirements and normalized, without any image augmentation, according to standards of the ResNet-34-and were arranged in m batches of 16 images. That is, we randomly divided the 781 CXR images into 49 batches of 16 images each; we carried out training out using 29 image batches (60%), validation using ten batches (20%), and testing for diagnosis using the remaining ten batches (20%). As we passed CXR image batches through the ResNet-34 for training, batch normalization in between skips double-or triple-layers containing nonlinearities for image classification. [20] To localize and highlight a critical region of interest (ROI) for a particular category within an input CXR image, we employed the Gradient-Weighted Class Activation Mapping (Grad-CAM) method to explain the DCNN prediction visually. It uses the gradient information of a target class flowing back into the last convolutional layer to generate visual explanations from the DCNN model. It generates a class activation map from a residual block of the DCNN model ( Figures 1A  and 1B), where it feeds the global average pooled convolutional feature maps into the fully connected final output layer, which is defined as follows: [20] where x and y are the input and output vectors of the building block, F k (x, w c k ) is the residual mapping, F k ∈ R u×v represents the global average pooling that spatially averages the kth feature map of width u and height v produced from the last convolution layer of the DCNN, and w c k is the weight connecting the kth feature map to an output node corresponding to class c. For the residual block considered in Figure 1B, F is given by: where ReLU(.) is the rectified linear unit and is given by: The prediction score (i.e., the input to the activation function Softmax) at the output layer, S c , can be expressed as a weighted sum of the global average pooling concerning the kth feature map given by: where f k (x, y) denotes the spatial element (x, y) activation in the kth feature map. The class activation map of class c, M c ∈ R u×v , is defined as a weighted sum of activation at spatial element (x,y) from all feature maps.
The definition of the Grad-CAM for a class c is the weighted sum of all feature maps resulting from the last convolution layer in a DCNN. We applied the ReLU function (Eq. 3) to remove the potential influence of negative weights on the class of interest, considering that the spatial elements in the feature maps associated with the negative weights were likely to belong to other categories in the image.
where α c k is the weight obtained by computing the gradient of a prediction score and is given by: Figure 1C shows the DCNN model (ResNet-34) block diagram for diagnosis/prognosis, which consists of a three-step pipeline for 1) detecting random CXR images, where the first layers detect edges and shapes while the last layers work on the details; 2) identifying all suspicious patchy infiltrates in the CXR image of the lung; and 3) classifying (i.e., diagnosing) of COVID-19, other viral pneumonias, and normal controls.
The performance of the ResNet-34 was monitored graphically, including the training loss, validation loss, and error rate along epochs. We assessed the accuracy of classification (i.e., diagnosis) by the receiver operating characteristic (ROC) curve, the area under the curve (AUC), sensitivity, and specificity analysis,    as well as a chi square for actual versus predicted values of COVID-19 versus normal control CXR images, and another for other viral pneumonia, versus normal control CXR images. We showed samples of grayscale X-ray input images and output colored activation map grids for COVID-19, other viral pneumonias, and normal controls. Finally, we carried out k-fold cross-validation tests to classify all 781 source CXR images from COVID-19 (n=240), other viral pneumonias (n=274), and normal controls (n=267), divided equally randomly among each one-to five-fold, respectively.

Results
The performance of the ResNet-34 model, where the training and validation loss of random subsets of CXR images and the error rate with the number of epochs converge at the end of the training phase. (Figures 2A-C) An epoch is a measure of the cycles through the full training process using image batches once to update the ResNet-34 weights. We found that ten epochs were sufficient to avoid both under-and over-fitting, reflecting the high precision of testing in the subset of random CXR images used for the diagnosis/prognosis.

Discussion
X-ray imaging for computer-aided medical detection, classification, and diagnosis of many diseases is an active field of research. It is accessible and can be performed at a low cost using portable units in clinics or homes. [27,28] Digital image processing effectively visualizes and extracts essential details, with many available techniques for enhancing the quality of a standard X-ray image before the diagnosis. Elsharkawy et al.
have recently proposed a pre-processing technique for correcting CXR images for non-uniformity of illumination and delimiting the boundaries of lungs using an automated segmentation routine or manual delineation by radiologists. [28] The technique uses estimates of the Markov-Gibbs random field energy at several radii to compute the cumulative distribution function describing the lung nodules and determine the severity of infection using a neural-network fusion and diagnostic system. Vieira et al. have also proposed image resizing using the maximum window function, which preserves anatomical structures of the chest, to increase the classification rate regardless of the deep learning architectures used. [29] Radiologists rely on the appearance of CXR images to identify any atypical unilateral patchy infiltration in the lungs of COVID-19 patients, although there no irregularities in the first grades of infection, and, if present, they may be confused with pneumonia. [7] Thus, to train a high-performance DCNN for diagnosis/prognosis of COVID-19 infection, using large datasets for training is essential. However, large repositories containing enough CXR images are not always available. Thus, transfer learning is the base for initializing a DCNN model, fine-tuned using the available limited medical datasets with results that outperform fully trained models under particular conditions. [20] The ResNet-34 model showed a very high performance, as of the convergence of the training loss, validation loss, and the error rate along with the training phase with the number of epochs to zero (Figures 2A-C). The distinctive nature of ResNet-34, which builds on constructs from pyramidal cells in the cerebral cortex using skip connections or shortcuts to jump over some layers, permits the advantageous reduction of over-fitting and leads to faster optimization of and better overall performance of the model. That is, only ten epochs were required to attain a high precision of testing and cross-validation of the subset of random CXR images without under-or over-fitting ( Table 2).
AUC analysis of ROC curves (Figure 2D-F) of truepositive rate (i.e., sensitivity) versus false-positive rate (i.e., 1-specificity) shows that points in the upper left corner signify sensitivity and specificity as high as 100, 99, and 99% for COVID-19, other viral pneumonias, and normal controls, respectively, by three-fold cross-validation. Generally, ROC curves lie between the lines connecting (0,0) to (0,1) and (0,1) to (1,1) extremes, where the area under the ROC curve is a summary measure that essentially averages the diagnostic accuracy of model predictions. The RSNA and other COVID-19 open access CXR image repositories have formed the basis for many studies for detecting pneumonia and COVID-19. [7,24] One of these is the recent study by Zhang et al. [30] They have successfully employed a pre-trained 18-layer ResNet using ImageNet weights for screening CXR images to detect, identify, and classify 106 COVID-19 patients and 107 normal controls without any fine-tuning. They showed that the ResNet achieved an AUC of 83.61% and sensitivity of 71.70%, which they maintain is comparable to the radiologists' performance reported in the literature.
The ResNet-34 had identical sensitivity and specificity of 100, 99.6, and 98.9% in classifying all source CXR images, without any image augmentation, of COVID-19, other viral pneumonias, and normal controls, respectively (Figure 3). The overall accuracy of the ResNet-34 for testing the subset of random CXR images used for diagnosis/prognosis was as high as 99.5%. The error rate of 0.5% in the testing phase is much lower than the 3.6% showed for the benchmarked ResNet-34. [21] Our group has lately shown similar high-performance results for estimating bone mineral density from DXA-DICOM images with an accuracy of 100% for the training, cross-validation, and testing phases of an ANN, yet with much-elaborated image processing for feature extraction using both histogram and binary algorithms.
[27] The highest accuracy yielded by the automated framework recently proposed by Akram    Distinguishing morphological anomalies in CXR images of COVID-19 patients from typical pneumonia patients is not easy. We employed the Grad-CAM method to localize and visualize an ROI within CXR images, which consistently highlights a specific area common in shape, pattern, or location among CXR images within the same class but distinct or different from images in other modality classes. [20] The activation map (Figures 4D-F) is thus a proper visual diagnostic representation, highlighting the morphological anomalies of a CXR image that contribute the most to the ResNet-34 classification process. Li et al. have recently addressed this problem of automatically distinguishing between COVID-19 and typical pneumonia using a U-Net pre-processor network for lung field segmentation, followed by a 3D ResNet-50 using transferred ImageNet weights. [34] Their model attained an accuracy of 87% using a dataset collected from 6 hospitals. Furthermore, the DenseNet-169 CNN model, tested to identify COVID-19 from CT slices without and with segmentation, showed 79.5% and 83.3% accuracy, respectively. [35] A model based on a combination of U-Net and 3D CNN has identified the presence of COVID-19 with a 90% accuracy using CT images.
[36] An accuracy of 94% has also been achieved with a ResNet-50-based CNN using transfer learning weighted from the ImageNet. [37] In conclusion, distinguishing morphological anomalies in CXR images (i.e., opacities and lung consolidations displayed as atypical unilateral patchy infiltration in the lungs) of COVID-19 patients from those of other pneumonia patients is cumbersome. It usually requires a highly expert radiologist and typically ends with ordering other expensive CT scans, which burdens the national health systems of all countries and delays the diagnosis of COVID-19 patients. In the present study, we employed the ResNet-34 for the detection and classification (diagnosis/prognosis) of CXR images of COVID-19, other viral pneumonias, and normal controls from four international sub-databases: the Società Italiana di Radiologia Medica e Interventistica (SIRM), the GitHub database, the Radiology Society of North America (RSNA), and the Kaggle chest X-ray database. Images were resized, normalized without any image augmentation, and arranged in m batches of 16 images before supervised training and validation of the DCNN classifier. The ResNet-34 model achieved identical sensitivity and specificity of 100%, 99.6%, and 98.9% in classifying CXR images of COVID-19, other viral pneumonias, and normal controls, respectively, with an overall accuracy of 99.5% for the testing subset for diagnosis/prognosis. Based on this high classification precision, we believe that the output activation map of the final layer of the ResNet-34 is a powerful tool for the accurate diagnosis of COVID-19 infection from CXR images.