In this work, we fixed an MR sequence of interest for which extensive, clinical acquisition of fully sampled data is infeasible and conducted an extensive, realistic validation of state of the art self-supervised reconstruction methods through two novel, overarching experiments.

1. 1.

   In contrast to the literature, we acquired phantom data with both full sampling and prospective acceleration. This allowed us to quantitatively and qualitatively evaluate both prospective and retrospective reconstructions using both pixel-wise and perceptual metrics for fidelity to ground truth, allowing us to study them individually as well as to see any relevant differences.

2. 2.

   In contrast to the literature, we tested the generalizability of the methods using an extensive, prospectively accelerated dataset with changes in contrast, hardware, field strength, and anatomy. Furthermore, we evaluated the results both quantitatively, using no-reference image quality metrics, and qualitatively, using rating by MR scientists and a radiologist.

The first self-supervised method we examine is called DeepDecoder. DeepDecoder is based on a seminal work in the machine learning literature called Deep Image Prior (DIP) (Ulyanov et al., 2018) which showed that untrained CNNs could be used to effectively solve inverse problems without ground truth. Concretely, let $f_{\theta}$ denote a randomly initialized CNN with parameters $\theta$. Let $\mathbf{z}$ be a sample of a random, Gaussian vector. Then DIP solves Equation 3 by

This formulation is equivalent to setting $R(\mathbf{x})$ to the indicator function with support over the range of the neural network; this assumes that the convolutional network $f_{\theta}$ itself provides a strong prior on the space of image solutions, such that only the data consistency term needs to be minimized. However, since only the noisy signal $\mathbf{y}$ is used during training, minimization can overfit the noise in the signal, depending on the inverse problem being solved (e.g. denoising, super-resolution), thus requiring early stopping (Ulyanov et al., 2018). DeepDecoder (Heckel and Hand, 2019) is a CNN with a simplified architecture (only upsampling units, pixel-wise linear combination of channels, ReLU activation, and channel-wise normalization) which is amenable to theoretical analysis and was shown to be competitive with other architectures for solving inverse problems in a DIP framework.

In (Heckel and Soltanolkotabi, 2020), the authors theoretically showed that for the case of image recovery from compressed sensing measurements, CNNs (in particular, CNNs with the structure of DeepDecoder) are self-regularizing with respect to noise and can simply be trained to convergence with gradient descent without early stopping or additional regularization, provided that the true, underlying image has sufficient smoothness/structure. In a knee MR example, they showed that early stopping would have only provided a marginally better solution than running to convergence. Hence, from a theoretical and practical standpoint, DeepDecoder is attractive for self-supervised reconstruction from undersampled measurements. We emphasize that DeepDecoder entails training a separate network for each separate acquisition/slice, rather than training a single network over a dataset of undersampled acquisitions.

The second self-supervised method we examine is called Self-supervised learning via data under-sampling (SSDU). SSDU uses an unrolled, iterative architecture, with alternating neural network and data consistency modules, to reconstruct MR images using only undersampled measurements, with the adjoint image corresponding to the input k-space measurements as an initial guess. It solves Eqn 3 using an iterative, variable splitting approach where the $k$th iteration consists of

where the superscript denotes the iteration, CNN denotes a generic CNN, and $\mathbf{\hat{x}}^{k}$ denotes an auxiliary variable. The regularization parameter $\lambda$ is learned during training. Let $f_{SSDU}$ denote the function defined by the unrolled network. In each training step of SSDU, the k-space of the data is split into two, random disjoint sets, denoted by $\mathbf{y}_{\Theta}$ and $\mathbf{y}_{\Lambda}$. $\mathbf{y}_{\Theta}$ is passed to the unrolled network as input. The loss function for SSDU compares the simulated k-space measurements of the corresponding image output $f_{SSDU}(\mathbf{y}_{\Theta})$ to $\mathbf{y}_{\Lambda}$:

where $A_{\Lambda}$ is the measurement operator corresponding to sampling the locations of $\Lambda$, and $L$ is an equally weighted combination of the $L_{1}$ and $L_{2}$ loss. Hence, during each training step, $f_{SSDU}$ only sees information from $\mathbf{y}_{\Theta}$, and the loss is only computed over a disjoint set $\mathbf{y}_{\Lambda}$. We note that at inference time, the entire, acquired k-space measurements are given as input. While the authors of SSDU give an intuitive explanation of this approach as similar to cross validation in order to prevent overfitting to noise or learning the identity, results from the machine learning literature on blind, signal denoising can help give a theoretical explanation.

In the Noise2Self framework (Batson and Royer, 2019), the authors prove that a neural network can be trained to denoise a noisy signal, using solely the noisy signal for training. In the following, we describe a special case of the general theory proven in (Batson and Royer, 2019). Let $\mathbf{y}^{\delta}=\mathbf{y}+\mathbf{n}$ denote a noisy signal, where $\mathbf{y},\mathbf{n}$ are the noise-free signal and Gaussian noise respectively. Partition $\mathbf{y}^{\delta}$ into disjoint sets, $\mathbf{y}^{\delta}_{J}$ and $\mathbf{y}^{\delta}_{J^{C}}$, where the subscript indicates restriction of the corresponding vectors to the disjoint subsets of indices $J,J^{C}$, with other indices being zero-filled. Then the authors showed that that a neural network (denoted as $f$) can be trained to denoise the noisy signal, using solely the noisy signal, by using the following loss function:

We emphasize that the right hand side of Equation 8 is composed of the mean squared error between the signal predicted by the network and the ground-truth signal and a constant independent of the network. Hence the Noise2Self strategy allows to minimize the error between the predicted signal and the ground truth signal with only access to the noisy signal, by iteratively giving a partition of the noisy signal as input to $f$ and computing the MSE over a disjoint partition. Identifying $\Theta,\Lambda$ with $J,J^{c}$, we can see that the training of SSDU conforms to the Noise2Self framework with the k-space measurements acting as the noisy signal, albeit with SSDU using an $L_{1}$ loss in addition to the $L_{2}$ loss. Thus, SSDU takes as input the noisy, acquired k-space measurements, and is optimized to output an image whose simulated k-space measurements are the acquired k-space measurements without noise. In this way, SSDU avoids overfitting to noise. This, combined with the powerful image prior from using a CNN as the neural network as well as the interleaving of the data consistency term, explains SSDU’s demonstrated ability to provide denoised images which retain image sharpness, as compared to traditional methods. We can interpret SSDU as an iterative method which interleaves the application of a denoising network and a data consistency step. We note in contrast to DeepDecoder, that we can train different networks for separate acquisitions or train a single, reusable network on a dataset of undersampled acquisitions. In this paper, we do the latter.

In conclusion, both self-supervised approaches accomplish noise robust MR reconstruction using only noisy, undersampled MR measurements;

To mimic a realistic scenario with a sequence for which fully sampled, ground truth data is difficult/infeasible to acquire, and where the training dataset is limited in size and variability, we acquired for ten healthy subjects a 5x accelerated 3D MPRAGE prototype sequence (Mussard et al., 2020) of the brain at 3T (MAGNETOM Prisma^Fit, Siemens Healthcare, Erlangen, Germany) using a 64ch Rx Head/Neck coil. These incoherently undersampled data were used for training/tuning the hyperparameters of all reconstruction methods. In what follows, all training/inference is done on 2D slices of both phase-encoding directions formed from performing the inverse Fourier transform along the readout direction. We emphasize that in the absence of prior knowledge/heuristics, the hyperparameters of the methods should also be tuned in a self-supervised way, as the common method for hyperparameter tuning, i.e. using a hold-out set of data for which the ground truth is known, is not available in our scenario. We use the Noise2Self framework, which also underlies SSDU, for selecting hyperparameters (regularization parameter of CS-L1Wavelet and the network parameters of DeepDecoder and SSDU), as it optimizes for preventing overfitting to the noise in the measurements. Details on the hyperparameter tuning can be found in the Appendix.

In our first experiment, using the aforementioned 3D MPRAGE prototype sequence used for acquiring the training/tuning data, we acquired both fully sampled and 5x prospectively accelerated scans of the following:

1. 1.

   Siemens multi-purpose phantom E-38-19-195-K2130 filled with $MnCl_{2}\cdot 4H_{2}O$ doped water

2. 2.

   Assortment of fruits/vegetables (Pineapple, tomatoes, onions, brussel sprouts)

This allowed us to reconstruct prospective, retrospective (applying the same mask as in prospective sampling on the fully sampled data), and fully-sampled images.

No in-vivo data was used in this experiment since subject motion could bias the results. Furthermore, we used fruits/vegetables as a second phantom since they have more complex structures than a water filled container.

In our second experiment, we examined the generalizability of the reconstruction methods. To that end, we scanned three, healthy subjects with the following prospectively accelerated sequences(anatomy):

1. 1.

   1.5T MPRAGE (Brain)

2. 2.

   3T MPRAGE (Brain)

3. 3.

   7T MPRAGE (Brain)

4. 4.

   3T MPRAGE with 1Tx/20Rx Coil (Brain)

5. 5.

   3T MPRAGE with Subject Motion (Brain)

6. 6.

   3T MPRAGE with Different Parameters (Brain)

7. 7.

   3T, $T_{1}$ SPACE (Brain)

8. 8.

   3T, $T_{2}$ FLAIR SPACE (Brain)

9. 9.

   3T, PD SPACE (Knee)

10. 10.

    3T, $T_{2}$ SPACE (Knee)

The brain scans at 1.5T, 3T and 7T (MAGNETOM Sola, Vida, and Terra, Siemens Healthcare, Erlangen, Germany) were done using a 1Tx/20Rx, 1Tx/64Rx (unless otherwise stated), and 8pTx/32Rx (Nova Medical, Wilmington, MA, USA) head coil, respectively. The knee scans at 3T were done with a 1Tx/18Rx coil. All detailed sequence parameters can be found in the Appendix in Table 2.

As ground truth data is not available since motion would render quantitative comparison difficult due to blurring from image co-registration, we evaluated the reconstructions from the above data quantitatively through no-reference image quality metrics and qualitatively through rating by four MR scientists and a radiologist. In total, 120 reconstructions (40 per subject) were evaluated.

For all quantitative metrics/ratings, we use the Wilcoxon signed rank test with significance level $\frac{0.05}{6}$ (Bonferroni correction with 6 pair-wise comparisons among the 4 methods) to determine statistical significance.

In Fig. 2 and Fig. 3, we can see spatial distortions of hyper/hypo-intense features in the prospective reconstructions and changes in contrast in comparison to the ground truth reconstruction; this distortion is not present in the retrospective reconstructions; however, they are similar across all reconstruction methods.

Retrospective reconstructions have significantly higher mean scores for all metrics in comparison to the prospective reconstructions in both acquisitions (see Table 1).

Comparing the methods, in the phantom, the prospective/retrospective reconstructions of DeepDecoder have the highest pixel-wise fidelity to the ground truth with a mean PSNR of (18.67/23.44) and SSIM of (0.49/0.52); however, qualitatively, it has more spatially varying oversmoothing than those of CS-L1Wavelet and SSDU. SSDU and CS-L1Wavelet perform similarly, with the highest qualitative similarity to the ground truth, with SSDU having a higher mean PSNR overall (17.79/21.95). In contrast to the PSNR/SSIM results, with the PercDis score, SSDU has the highest fidelity to the ground truth (0.63/0.61).

Qualitatively and quantitatively (with PSNR and SSIM), the differences between the methods are much less in the fruits/vegetables. The main qualitative difference is the greater denoising capabilities of SSDU and CS-L1Wavelet in comparison to CG-SENSE and DeepDecoder. Quantitatively, there are only minor differences between the methods with respect to PSNR and SSIM. In contrast, the PercDis scores clearly indicate that CS-L1Wavelet and SSDU (with similar scores) are perceptually more similar to the ground truth than CG-SENSE and DeepDecoder (with similar scores).

Figures 4, 5 show axial MPRAGE brain slices at the different field strengths and corresponding closeups of the cerebellum and the left frontal lobe. Figure 6 shows a sagittal PD knee slice (3T) with closeups of articular cartilage interfaces in sagittal (femur) and axial (patella) views. These show the generalizability of the methods to different magnetic field strengths as well as changes in anatomy and contrast. Example reconstructions for the other sequences can be found in the Appendix 8, 9.

Concerns about the differences between prospective and retrospective reconstructions were also raised in (Muckley et al., 2021), in the context of end-to-end, supervised methods for parallel MR image reconstruction. In particular, they noted that retrospective undersampling neglects potential differences in signal relaxation across echo trains, and verification should be performed before clinical use. From our results using both fully sampled and prospectively accelerated data, it is clear that for the 3D MPRAGE sequence, prospective vs. retrospective reconstructions can differ meaningfully, with retrospective reconstructions having greater fidelity to the fully sampled reconstruction; prospective reconstructions exhibit spatial distortions and local changes in contrast, with respect to the ground truth. This is despite the methods being tuned/trained on prospectively accelerated data; hence, this can be attributed to the differences in the prospectively vs. retrospectively sampled k-space data, potentially due to the different gradient patterns used in the sequences. This difference is relevant both for self-supervised and supervised machine learning methods; indeed, end-to-end, supervised methods which are trained on retrospective data may yield even greater distortion than self-supervised methods when prospective data is used for inference. However, the performance ranking of the different methods was the same in both prospective and retrospective reconstructions. Therefore, retrospective image quality cannot necessarily be taken as a reliable proxy for prospective image quality; however, they can be used to show differences between methods.

The quantitative results in the phantom show how ranking by PSNR and SSIM can be misleading, as images that are perceptually/qualitatively more similar to the ground truth (SSDU,CS-L1Wavelet) can have significantly worse or almost identical mean PSNR/SSIM scores than images which are less qualitatively similar (CG-SENSE,DeepDecoder). In contrast, ranking with the PercDis score, which measures distances between the feature activations within a pretrained classification network of the images rather than the images themselves, better matches with the perceptual quality of the images, showing that SSDU or SSDU/CS-L1Wavelet are better, by a significant margin (relatively with respect to the same differences in PSNR/SSIM), than the other methods. The PercDis score or perceptual loss (Johnson et al., 2016) was created precisely because they found this metric better suited for measuring perceptual similarity than PSNR/SSIM. This apparent tradeoff between PSNR/SSIM and perceptual similarity is well-known in the computer vision community, where it is called the perception-distortion tradeoff (Blau and Michaeli, 2018). This concept has also recently been explored in MR; in (Adamson et al., 2021), the authors train an in-painting network on the Fastmri dataset, and use the features of intermediate layers for quantitative evaluation, producing a perceptual distance tailored for MR images. In (Wang et al., 2019), the authors propose a new reconstruction method which uses distances in feature space (trained from ground truth MR reconstructions) to better recover textures/perceptual appearance than using just pixel-wise metrics.

We note that as our generalizability study is conducted on prospective reconstructions, which we showed can exhibit distortions relative to fully-sampled reconstructions, it cannot be considered as clinical validation; however, as all methods are affected the same way, this study still can give a good idea of how well each method generalizes. While one might conjecture that generalizability is less of a problem for self-supervised methods, if the parameters/hyperparameters of the methods are tuned for a specific sequence/anatomy as in our case, this could potentially impact the robustness of the methods, as these parameters/hyperparameters are obtained from training/tuning on 3D, brain MPRAGE scans acquired at 3T. This is despite the data consistency inherently embedded in CS-L1Wavelet, DeepDecoder, and SSDU.

Generalizability and robustness of reconstruction methods have been studied in the context of end-to-end, supervised methods for MR reconstruction in (Knoll et al., 2019; Hammernik et al., 2021; Antun et al., 2020). We briefly summarize some relevant conclusions from these articles. (Knoll et al., 2019) found that that different domain shifts reduced performance more than others (e.g. changing SNR vs. image contrast), and that transfer learning is a viable strategy for handling distribution shifts. (Hammernik et al., 2021) found that data consistency is important for robustness, and that at acceleration factor 4, distribution shifts are less of an issue. (Antun et al., 2020) found that supervised methods are vulnerable to adversarial perturbations, i.e. perturbations constructed such that minimal changes in the input data result in significant changes in the output.

In (Darestani et al., 2021), the authors examine the robustness of end-to-end methods, compressed sensing, and variations of Deep Image Prior/DeepDecoder to distribution shifts, adversarial perturbations, and recovery of small features. They found that for both supervised and self-supervised methods, distribution shifts resulted in decreased PSNR/SSIM scores; in addition, the decrease was roughly the same for each method, preserving the ranking of the methods. Finally they found that all methods, including self-supervised methods, were vulnerable to adversarial attacks, including CS-L1Wavelet and DeepDecoder. Furthermore, Zhang et al. showed the vulnerability of SSDU to adversarial attacks, showing that this was primarily due to the data consistency term. Thus, CG-SENSE can also be assumed to be vulnerable. Therefore, all the methods used in this paper have been shown to be vulnerable to adversarial attacks. We note that these works are based on retrospective reconstructions/retrospective sampling from fully-sampled datasets for their validation.

In line with (Knoll et al., 2019), we found that different distribution shifts affected generalization differently; changing anatomy/contrast worsened the overall image quality rating in comparison to changing the field strength for all methods according to the MR scientists; in contrast, the radiologist found that only SSDU/CS-L1Wavelet worsened. However, as the mean scores in the knee scans for CG-SENSE/DeepDecoder were already 4 (the worst score), the decrease may not reflect any substantial difference in quality. As in (Hammernik et al., 2021), data consistency is crucial for the robustness of self-supervised methods as network parameters are trained solely through the modelling/the acquired undersampled data; in particular, we do not see any hallucination that can occur with end-to-end networks without data consistency. Furthermore, we see that as CG-SENSE produces a plausible image with acceleration factor 5, this can explain why distribution shifts were not so troublesome, as the self-supervised methods mainly needed to denoise, rather than recover anatomy/missing high frequency details.

In contrast to (Darestani et al., 2021), our PSNR/SSIM results on the phantoms do not preserve the ranking between methods, although the PercDis results do, approximately. However, the qualitative metrics between distribution shifts over the different brain/knee scans seem to preserve ranking according to the no-reference image metrics/human ratings; this is consistent with PercDis being a better measure for perceptual image quality/similarity than PSNR/SSIM. In addition, the distribution shift in (Darestani et al., 2021) was between two, similar datasets of knee MRI, as compared to our distribution shifts, where we change anatomy, contrast, etc.

For a clinical scenario, it was of interest to see if self-supervised methods could potentially work, without retraining, on other sequences, as retraining after deployment could be impractical. Furthermore, while adversarial perturbations are valuable for studying the input stability of reconstruction methods, they need to be manually constructed for each method and added to the input data. As clinical MR reconstruction is a closed loop, this kind of manual perturbation would require hacking the internal MR computer. Therefore, transfer learning and adversarial perturbations were outside the scope of this work, although from (Hammernik et al., 2021; Knoll et al., 2019; Darestani et al., 2021), we would expect an increase in image quality from transfer learning and vulnerability to adversarial perturbations for the methods considered in this paper. For example, (Darestani and Heckel, 2021) found, in a retrospective study, that DeepDecoder had different optimal (judged by PSNR/SSIM) hyperparameters for brain vs. knee scans. However, SSDU and CS-L1Wavelet, tuned only on 3T MPRAGE brain data, are able to achieve an overall image quality of fair to good on a diverse dataset.

From a perceptual viewpoint (PercDis score, no-reference image metrics, human rating), SSDU and CS-L1Wavelet performed the best, with an edge to SSDU in the PercDis score/no-reference image metrics. From a pixel-wise metric viewpoint (PSNR,SSIM), DeepDecoder was better than or similar to all methods, as was also found in (Darestani et al., 2021). CG-SENSE consistently performed the worst or similarly to all methods over all metrics. With respect to validation, both approaches have their advantages and disadvantage; while pixel-wise metrics are the natural way to compare against a ground-truth, they may not correlate well with the perception of a radiologist. While perceptual metrics may be intuitive, the absence of ground truth can make it less objective. To our knowledge, current state of the art MR image reconstructions are generally not evaluated with perceptual metrics such as PercDis or (Adamson et al., 2021), which require ground truth, or the no-reference image quality metrics. However, given the close correspondence of the image quality metrics/PercDis to the human ratings/perceptual evaluation, as well as other evidence from the literature (Woodard and Carley-Spencer, 2006; Zhang et al., 2018), perceptual metrics could be used as a complement to pixel-wise metrics/human ratings.

We note that while SSDU generally outperformed DeepDecoder, SSDU’s denoising network was trained on a dataset of 3T MPRAGE, thus learning a prior over multiple subjects. In contrast, DeepDecoder only learns/performs inference over a single slice at a time, thus limiting the amount of information in comparison to SSDU. In Korkmaz et al. (2022), the authors show that a Deep Image Prior based reconstruction can be fused explicitly with prior information from a dataset of fully sampled acquistions to increase performance; such fusion of prior information could potentially also benefit DeepDecoder and other self-supervised methods which operate on a per slice basis. In addition, from the qualitative results, CS-L1Wavelet with regularization parameter tuned using the Noise2Self framework is competitive with SSDU. At lower acceleration factors, such as the one used in this paper, it is plausible that this result generalizes, such that compressed sensing reconstructions with optimally tuned regularization parameters can be competitive with state of the art machine learning methods, at least on a qualitative basis. For future validation, we conclude that appropriate regularization parameter tuning strategies should be used when comparing compressed sensing reconstructions to new methods. Finally, we note that as the theory behind the self-supervised methods we used (Deep Image Prior and blind denoising) form the basis for or are conceptually similar to many other self-supervised methods, it is plausible that the impressive robustness showed by these methods to a diverse range of realistic distribution shifts would generalize to future self-supervised methods.

However, whatever metrics or datasets are used for validating methods, the ultimate test for reconstruction methods is the usefulness to radiologists for reliably diagnosing pathology in comparison to currently used methods (Recht et al., 2020; Roux et al., 2019). This can imply many things, including fine grained analysis of small textures/details/pathologies as well as tissue specific analysis, requiring novel datasets with extensive annotations by radiologists. (Zhao et al., 2021; Desai et al., 2021) are two recent works in this direction, providing datasets with bounding box annotations/pathology annotations to further validate reconstructions. To assist validating future methods, the datasets acquired for this paper will be made available online; see https://www.melba-journal.org/papers/2022:022.html for details.

For example, to set the regularization parameter of CS-L1Wavelet, we treat it as a function with a single parameter ($\lambda$). We can then optimize this parameter using the Noise2Self training framework to estimate the $\lambda$ which minimizes the noise-free error between simulated measurements and the acquired measurements. Concretely, we fix 20 logarithmically spaced values from 0.00001 to 0.1. We set each value as $\lambda$ and run 50 image reconstructions corresponding to different, random masks and average the corresponding errors with respect to the complementary mask in order to approximate the true measurement error associated with using each value. We then select the value with the lowest measurement error as the optimal regularization parameter. This is done for each slice in each subject; the final regularization value which is used throughout this paper is the average over all subjects. The hyperparameters of DeepDecoder and SSDU are set similarly with a grid search over the network hyperparameters, albeit over a much smaller set of data due to the high computational demand.