Authors

Jueqi Wang, Jacob Levman, Walter Hugo Lopez Pinaya, Petru-Daniel Tudosiu, M. Jorge Cardoso, Razvan Marinescu

Abstract

High-resolution (HR) MRI scans obtained from research-grade medical centers provide precise information about imaged tissues. However, routine clinical MRI scans are typically in low-resolution (LR) and vary greatly in contrast and spatial resolution due to the adjustments of the scanning parameters to the local needs of the medical center. End-to-end deep learning methods for MRI super-resolution (SR) have been proposed, but they require re-training each time there is a shift in the input distribution. To address this issue, we propose a novel approach that leverages a state-of-the-art 3D brain generative model, the latent diffusion model (LDM) from [21] trained on UK BioBank, to increase the resolution of clinical MRI scans. The LDM acts as a generative prior, which has the ability to capture the prior distribution of 3D T1-weighted brain MRI. Based on the architecture of the brain LDM, we find that different methods are suitable for different settings of MRI SR, and thus propose two novel strategies: 1) for SR with more sparsity, we invert through both the decoder D of the LDM and also through a deterministic Denoising Diffusion Implicit Models (DDIM), an approach we will call InverseSR(LDM); 2) for SR with less sparsity, we invert only through the LDM decoder D, an approach we will call InverseSR(Decoder). These two approaches search different latent spaces in the LDM model to find the optimal latent code to map the given LR MRI into HR. The training process of the generative model is independent of the MRI under-sampling process, ensuring the generalization of our method to many MRI SR problems with different input measurements. We validate our method on over 100 brain T1w MRIs from the IXI dataset. Our method can demonstrate that powerful priors given by LDM can be used for MRI reconstruction. Our source code is available online: https://github.com/BioMedAI-UCSC/InverseSR.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43999-5_42

SharedIt: https://rdcu.be/dnwwV

Link to the code repository

https://github.com/BioMedAI-UCSC/InverseSR

Link to the dataset(s)

https://brain-development.org/ixi-dataset/

Reviews

Review #2

Please describe the contribution of the paper

Authors proposed a novel diffusion model to increase the resolution of clinical MRI scans, with a deterministic DDIM sampling approach, a corruption function, two versions of InverseSR’s.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

1- The limitations are discussed. 2- The trained model is validated in an external dataset, showing the generalization. 3- The performances are quite good. 4- The proposed approaches are technically sound (while the proposed approaches are not very novel to me, the good performances and generalizability are what truly matter).

Based on these reasons, I recommend accepting this paper.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

I did not see obvious weaknesses of this paper.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The answers to “For all code related to this work that you have made available or will release if this work is accepted” are all “yes”.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

The back quotes in this papers are all wrong.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The good performances and the generalization test on external dataset.
Reviewer confidence

Not confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

In this paper, a novel method is proposed for brain MRI super-resolution using latent diffusion models and a DDIM sampler. The performance of this method was assessed on publicly accessible datasets, subject to a known form of corruption, and was found to be effective in generating high-quality, high-resolution brain MRI images.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

-The problem that this method focused on is of clinical importance. -The proposed method is expected to be practically useful as it employs LDM and DDIM to address the computational complexity of conventional DL-based methods, while also reducing the inference time steps of DDPM. -An innovative approach was presented in the work, which used conditional LDM for the iterative optimization of a super-resolution image.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-The proposed InverseSR pipeline has a potential flaw in that it involves iterative optimization of the latent code representation of a high-resolution image using low-resolution input. However, because there is no way to provide guidance to the optimization process using high-resolution images, it may introduce “unexpect” objects in the output image. This flaw is evident in Fig. 2, where both InverseSR(LDM) and InverseSR(Decoder) introduced bright objects near the bottom of the skull, which were not present in the high-resolution ground truth image. Moreover, the anatomies of the generated brains produced by the proposed methods do not match the anatomies of the ground truth image. -The proposed method has another potential issue in that it only works on a known corruption function, which could render it unavailable if knowledge about the corruption is infeasible. Additionally, the method requires per-image optimization and a relatively time-consuming reverse diffusion process, which makes the proposed method slow in practice and less efficient for real-world applications. These limitations significantly reduces the practical usefulness of the proposed method. -There lacks comparison to state-of-the-art brain MR SR methods, e.g., 1,2, and
1. Zhao, Can, et al. “SMORE: a self-supervised anti-aliasing and super-resolution algorithm for MRI using deep learning.” IEEE transactions on medical imaging 40.3 (2020): 805-817.
2. Iglesias, Juan Eugenio, et al. “Joint super-resolution and synthesis of 1 mm isotropic MP-RAGE volumes from clinical MRI exams with scans of different orientation, resolution and contrast.” Neuroimage 237 (2021): 118206.
3. Iglesias, Juan E., et al. “SynthSR: A public AI tool to turn heterogeneous clinical brain scans into high-resolution T1-weighted images for 3D morphometry.” Science advances 9.5 (2023): eadd3607.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The reproducibility of the paper is good.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

Please consider addressing the comments mentioned in the weakness section.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The proposed method may have some inherent factors that could potentially hinder its practical usefulness. Nevertheless, due to its novelty and potential interest to the MICCAI society, I recommend a weak acceptance of this paper.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #1

Please describe the contribution of the paper

This manuscript leverages the recently published “latent diffusion models” (LDM) and presents a method of modeling slice imputation and lesion inpainting in MRI. The authors develop a 3d extension of LDM. The autoencoder part of LDM is trained with scan volumes from UK Biobank dataset. The latent representations built by the autoencode are subjected to DDIM (operating in the latent space) conditioned on four scalar variables, to produce an altered latent code, which is decoded to produce the restored image. Evaluation on 100 images from IXI dataset for imputing 3 consecutive slices (4mm to 1mm) and imputing 7 consecutive slices (8mm to 1mm) is done comparing PSNR and SSIM improvements, against cubic interpolation, UniRes, and two proposed variants (trained decoder without DDIM, and DDIM with pretrained decoder. Improvements shown are significant.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

3d extension of LDM is novel. The trained weights of LDM autoencoder will be useful as a method of model-based representation of brain structure, almost as a new way of atlasing brains. The use case shown here is an initial step, and is also of good value, e.g., to standardize volumes, and can be a potential staple preprocessing step, if shown to not introduce artifacts or hallucinate structures.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

All good
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Largely reproducible, as the base model and inspirations are clearly indicated and cited.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Good use case, apt method and significant results.

Feedback points:
1. More implementation details in place of Algorithm 1, might be helpful, specifically about 3d extension of the LDM.
2. Reporting 17 hours for restoring a single LR image sounds incredulous, is possibly a lax use of an inappropriate computer. Please remove the statement, to prevent this from get cited.
3. Will 2d LDM also solve the same problem? if not, please add a statement to reason out.
4. Comparing against one more generative method (possibly GAN) would add great value to demonstrate the capability of your 3d LDM.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Well written, clearly indicates the background, selected model, application, solution method and experiments are good. Results are significant. Good work overall.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper proposes a 3D extension of the Latent Diffusion Model (LDM) and demonstrates its potential for model-based representation of brain structure, akin to atlasing brains. The trained weights of the LDM autoencoder can be used for standardizing volumes and serve as a preprocessing step. The proposed method shows good performance and generalizability, making it a valuable contribution.

The limitations of the proposed method are discussed, including the potential introduction of artifacts or hallucinated structures. The trained model is validated on an external dataset to demonstrate its generalization capability. The performances of the proposed approaches are considered quite good. While the proposed approaches may not be entirely novel, their effectiveness and generalizability are the main factors of importance.

However, there are concerns raised by reviewers. The proposed InverseSR pipeline involves iterative optimization of a high-resolution image’s latent code representation using low-resolution input, which may introduce unexpected objects and discrepancies with the ground truth anatomy. Additionally, the method relies on knowledge of the corruption function and requires per-image optimization and a time-consuming reverse diffusion process, making it impractical and less efficient for real-world applications.

Another limitation is the lack of comparison to state-of-the-art brain MR super-resolution methods, which should be included for a comprehensive evaluation. The cited papers (1, 2, and 3) provide relevant works for comparison.

In summary, the proposed 3D extension of LDM is seen as a novel contribution with potential clinical importance. The method’s practical usefulness, computational complexity reduction, and inference time reduction are considered valuable. However, concerns are raised about the potential introduction of artifacts, the reliance on a known corruption function, and the efficiency for real-world applications. The paper’s strengths lie in its performance and generalizability. It is recommended to address the limitations and include comparisons with state-of-the-art methods to strengthen the evaluation.

Author Feedback

We appreciate the reviewers for their careful evaluation of our work. We have thoroughly read all the feedback. Below, we will respond to the comments raised by reviewers and describe minor changes to our paper.

Lack of comparison to state-of-the-art brain MR super-resolution methods(R3.6.3, R1.9.4): While we have already compared our method to UniRes and Cubic, we will compare our method to the brain MR SR method SynthSR (Iglesias et al, NeuroImage, 2021) in the final version. We apologize for not adding this comparison during the first submission. However, we were unable to train other generative-prior models such as GAN because we do not have access to the UK Biobank dataset, so the ability to use the pre-trained model from (Pinaya et al, Deep Generative Models, 2022) was key to developing our method. Nevertheless, a comparison against GANs is very good also because sampling from GANs does not involve a long Markov chain of sampling steps like with diffusion models, so it can restore the LR image much faster and require fewer computational resources.

InverseSR may introduce unexpected objects and discrepancies with the ground truth anatomy (R3.6.1): This can indeed happen, making an interesting direction for future study. In general, the unexpected objects and discrepancies with the ground truth anatomy could be solved by acquiring more slices or with a better generative model which could capture more generalizability of the brain MRI.

InverseSR requires a relatively time-consuming reverse diffusion process(R1.9.2, R3.6.2): Due to the lack of a suitable GPU, we had to run the inference of our algorithm on a CPU which was significantly slower. Running the 3D brain LDM on a GPU required cards with 80GB memory, which we did not have at the time we were developing the method. We have since gained access to an 80GB GPU, and will run our method on that GPU and report the time in our final version.

More implementation details in place of Algorithm 1, might be helpful, specifically about the 3d extension of the LDM (R1.9.1): We will add more implementation details of Algorithm 1. The 3D brain LDM we used is from (Pinaya et al, Deep Generative Models, 2022), and interested readers can find more information about the 3D extension of the LDM there.

back to top

InverseSR: 3D Brain MRI Super-Resolution Using a Latent Diffusion Model