The local phase-based image enhancement method extracts the phase signatures from the image and are intensity invariant (Alessandrini et al. (2012); Li et al. (2018)). Therefore, the enhancement results and subsequent image analysis methods are not affected by the intensity variations due to patient characteristics or machine acquisition settings. The local phase image features which provide more structural information about the local anatomy. Especially the consolidations are more dominantly represented and highly localized in the enhanced images.

In the context of enhancing the appearance of CXR data, Qi et al. (2020) developed a local phase-based image enhancement method by designing bandpass quadrature filters. During this work, we deploy this image enhancement method for improving the representation of the CXR lung data (Qi et al. (2020)). The enhanced CXR image, termed multi-feature image $MF(x,y)$, is obtained by combining three different local phase image features: 1-Local weighted mean phase angle ($LwPA(x,y)$), 2- Weighted local phase energy ($LPE(x,y)$), and 3-Enhanced local energy attenuation image ($ELEA(x,y)$). $LPE(x,y)$ and $LwPA(x,y)$ image features are extracted by filtering the CXR image in frequency domain using monogenic filter and $\alpha$-scale space derivative (ASSD) bandpass quadrature filters (Qi et al. (2020)). $ELEA(x,y)$ image is extracted from the $LPE(x,y)$ image by modeling the scattering and attenuation effects of lung tissue inside a local region using L1 norm-based contextual regularization method (Qi et al. (2020)). We have used the filter parameters reported in Qi et al. (2020) for processing all the CXR images used in this work. Investigating Figure 1, we can see that structural features inside the lung tissue are more dominant for COVID-19 CXR images compared to healthy lung tissue (last column Fig.1). The next section explains how the $MF(x,y)$ images are used to improve the diagnostic accuracy of our proposed deep learning model.

Figure 2 (b) illustrates parallel-attention (PA) block for the original CXR images ($CXR(x,y)$), and the same procedure is performed for the enhanced CXR images ($MF(x,y)$) by simply swapping the index cxr and enh. Two embedded feature maps $\textbf{F}_{cxr}$, $\textbf{F}_{enh}$ $\in\mathbb{R}^{H\times W\times C}$ from $CXR(x,y)$ and $MF(x,y)$ are fed to a PA block. In a PA block, $\textbf{F}_{cxr}$ is fed to a 1$\times$1 convolution layer for feature adaption to produce a query feature map Q $\in\mathbb{R}^{H\times W\times C}$, and meanwhile $\textbf{F}_{enh}$ is fed to another 1 $\times$ 1 convolutional layer for feature adaption to produce a key feature map K $\in\mathbb{R}^{H\times W\times C}$, as shown in Eq.1. Q and K are then reshaped to $\mathbb{R}^{HW\times C}$. Then, matrix multiplication is performed between Q and K followed by a softmax layer to produce a attention matrix $\textbf{M}_{enh{\scriptscriptstyle\rightarrow}cxr}$ $\in\mathbb{R}^{HW\times HW}$, as shown in Eq.2. Next, $\textbf{F}_{enh}$ is reshaped to $\textbf{F}^{{}^{\prime}}_{enh}$ $\in\mathbb{R}^{HW\times C}$ and multiples $\textbf{M}_{enh{\scriptscriptstyle\rightarrow}cxr}$ to produce output feature map $\textbf{O}_{cxr}$ $\in\mathbb{R}^{H\times W\times C}$, as shown in Eq.3. Through matrix multiplication between Q and K, our PA can effectively encode feature correlation between $\textbf{F}_{cxr}$ and $\textbf{F}_{enh}$ into attention map M, and thus two attention maps ( $\textbf{M}_{enh{\scriptscriptstyle\rightarrow}cxr}$ and $\textbf{M}_{cxr{\scriptscriptstyle\rightarrow}enh}$) are generated by our PA block. Correspondence can then be captured by multiplying its corresponding F as shown in Eq.3.

The network architecture of our propose multi-scale parallel-attention (MS-PA) fusion network is shown as Figure 2 (a). A $CXR(x,y)$ image and its corresponding $MF(x,y)$ image as inputs are fed to two same CNN architectures, then several PA blocks are used for the fusion of intermediate feature maps from convolutional feature extractors between $CXR(x,y)$ and $MF(x,y)$ images, and in the end a ViT with cross-attention is used for making the final decision. The convolutional feature extractors for the $CXR(x,y)$ and $MF(x,y)$ inputs encode different aspects of the image at each scale. Therefore, we fuse these features at multiple scales throughout the convolutional encoder.

As shown in Figure 2 (a), the intermediate feature maps $\textbf{F}_{cxr},\textbf{F}_{enh}$ $\in\mathbb{R}^{H\times W\times C}$ are fed to a PA block, which produces two outputs $\textbf{O}_{cxr}$, $\textbf{O}_{enh}$ $\in\mathbb{R}^{H\times W\times C}$. The each output is then fed back into each of the individual branch using an element-wise summation with the existing feature maps. The mechanism described above constitutes feature fusion at a single scale. This fusion is applied multiple times throughout the convolutional feature extractors of the $CXR(x,y)$ and $MF(x,y)$ branches at different resolutions. However, processing feature maps at high spatial resolutions is computationally expensive, specially when your inputs have a high resolution. Therefore, we downsample higher resolution feature maps from the early convolutional feature extractors using average pooling (H$=$W$=$7) to the resolution of feature map from the last convolutional feature extractor before passing them as inputs to a PA block and upsample the output to the original resolution using bilinear interpolation before element-wise summation with the existing feature maps.

After PA fusion at multiple scales, feature maps $\textbf{F}_{cxr},\textbf{F}_{enh}$ $\in\mathbb{R}^{H^{{}^{\prime}}\times W^{{}^{\prime}}\times C^{{}^{\prime}}}$ are obtained at the last convolutional feature extractors from the $CXR(x,y)$ and $MF(x,y)$ branches. Then, we projected encoded feature maps $\textbf{F}_{cxr}$ and $\textbf{F}_{enh}$ of dimension $C^{{}^{\prime}}$ into $\textbf{F}_{cxr}^{p}$ and $\textbf{F}_{enh}^{p}$ with dimension of 256 using 1 $\times$ 1 convolution kernel. Then $\textbf{F}_{cxr}^{p}$ and $\textbf{F}_{enh}^{p}$ are reshaped to $\mathbb{R}^{H^{{}^{\prime}}W^{{}^{\prime}}\times 256}$. We prepended additional CLS tokens and added positional embedding for incorporating global context to $\textbf{F}_{cxr}^{p}$ and $\textbf{F}_{enh}^{p}$. Two cross-attention blocks are applied to further exchange information between $CXR(x,y)$ and $MF(x,y)$ (Qi et al. (2022)). As $\textbf{F}_{cxr}$ and $\textbf{F}_{enh}$ already encodes the representations for important findings from $CXR(x,y)$ and $MF(x,y)$, we adopted relatively simple architecture for Transformer encoder with two encoder layers with four attention heads. In the end, CLS tokens from two branches are projected into two 3-dimensional feature vectors and summed together.

In this study, all images were collected from six public data repositories, which are BIMCV (de la Iglesia Vayá et al. (2020)), COVIDx (Wang et al. (2020)), COVID-19-AR (Desai et al. (2020)), MIDRC-RICORD-1c (Tsai et al. (2021)), COVID-19-IR (Winther et al. (2020)) and COVID-19-NY-SBU (Saltz et al. (2021)). Total dataset includes three classes: normal, pneumonia, and COVID-19. The numbers of images for each class are summarized in Table 1. COVID-19 CXR images from both (BIMCVde la Iglesia Vayá et al. (2020)) and COVIDx (Wang et al. (2020)) datasets plus 2567 randomly selected CXR images from normal and pneumonia classes composed the ’Evaluation Dataset’. To evaluate the generalization ability of models in various institutional settings, we used the rest normal and pneumonia CXR images from Wang et al. (2020) and four COVID-19 datasets collected from different institutions with different devices as ’Test Dataset-2’.

We choose ResNet-50 pretrained on ImageNet that has the best performance reported by Qi et al. (2020) and DenseNet-121 with PCAM pooling pretrained on CheXpert (Irvin et al. (2019)) that achieved the first place in CheXpert challenge in 2019 (Ye et al. (2020)) as the backbone network architectures. CheXpert (Irvin et al. (2019)) dataset has 223,415 CXR images containing 13 labeled observations including no finding, enlarged cardiomediastinum, cardiomegaly, lung opacity, lung lesion, edema, consolidation, pneumonia, atelectasis, pneumothorax, pleural effusion, pleural other, and support devices. Binary cross-entropy (BCE) losses were used as the loss function. For training of the proposed model for COVID-19 diagnosis, we used SGD optimizer (momentum = 0.9) with learning rate of 0.001. The model was trained for 35 epochs with cosine warm-up scheduler (warm-up epochs = 4), with batch size set to 32. Five-fold cross-validation was performed on ’Evaluation Dataset’ for training and testing the proposed methods. Our augmentation method includes random translation, rotation, changing lighting condition, normalization, horizontal flip and resizing. We adopted mean overall accuracy (Acc) as our primary evaluation metric, but also calculated sensitivity, precision, F-1 score and area under the ROC curve (AUC). All experiments were performed with Python version 3.6 and PyTorch 1.10 on two Nvidia 1080Ti.

Res50-PA-mid, shown as Figure 5, has the same feature extractor as the Res50-PA-ViT, but it uses middle fusion with Convolutional operation reported in Qi et al. (2020) instead.

Res50-PA-late, shown as Figure 6, also has the same feature extractor as the Res50-PA-ViT. After feature extraction, the features are feed into image classifier, which has the same structure of ResNet50. The final decision is made based on the joint correspondence using sum operation reported in Qi et al. (2020).

To reinforce the performance of proposed methods in the detection of COVID-19, we also report the results of AUC tested on Evaluation Dataset and Test Dataset-2 in comparison with baseline and SOTA models shown as Table 5. Dense121-PA-ViT achieves the highest average AUCs compared to other methods testing on two datasets. Res50-PA-mid and Res50-PA-late show a better performance compared to Res50-mid and Res50-late, two multi-feature fusion models without PA blocks reported in Qi et al. (2020). It demonstrates using PA blocks at multi-scales for feature fusion can further improve the model performance.

An publicly available US dataset (Born et al. (2021)) was choose as an additional dataset. The dataset has 2825 US images, including 852 COVID-19, 707 bacterial pneumonia, and 1266 healthy images, which are extracted from 157 convex ultrasound videos at a frame rate of 3Hz with maximal 30 frames per video and includes 56 convex ultrasound static images. All these images are split into five folds on a video level since the consecutive US frames are highly correlated. The reported local phase-based image enhancement method was applied to the US images to obtain the multi-feature $MF(x,y)$ images. Figure 7 shows the original US images and the corresponding enhanced US images for each class.

Table 6 shows the diagnostic performance of our proposed method tested on the US dataset. Dense121-PA-ViT achieves the average AUC of 0.944, average precision 0.880, average sensitivity 0.877, average F-1 score 0.877, and average accuracy of 89.52%. Dense121-PA-ViT has the highest precision, sensitivity, F-1 score, AUC, and accuracy for all classes compared to Res50-PA-ViT.

Table 7 presents the diagnostic performances of our proposed models compared to the baseline and SOTA models in average accuracy and AUC. As shown in Table 7, Dense121-PA-ViT obtains the highest overall accuracy and class accuracy for all classes. And the proposed Dense121-PA-ViT achieves statistically significant improvement compared with the best baseline and SOTA methods with paired t-test ($p<0.05$) in overall accuracy and class accuracy. Investigating Table 7, we can also see the Dense121-PA-ViT obtains the highest overall AUC and class AUC except normal class. The Dense121-ViT-Enh achieves the highest AUC in identification of normal class.