Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews Back to top

List of Papers By topics Author List

Paper Info

Reviews

Meta-review

Author Feedback

Post-Rebuttal Meta-reviews

Authors

Julian McGinnis, Suprosanna Shit, Hongwei Bran Li, Vasiliki Sideri-Lampretsa, Robert Graf, Maik Dannecker, Jiazhen Pan, Nil Stolt-Ansó, Mark Mühlau, Jan S. Kirschke, Daniel Rueckert, Benedikt Wiestler

Abstract

Clinical routine and retrospective cohorts commonly include multi-parametric Magnetic Resonance Imaging; however, they are mostly acquired in different anisotropic 2D views due to signal-to-noise-ratio and scan-time constraints. Thus acquired views suffer from poor out-of-plane resolution and affect downstream volumetric image analysis that typically requires isotropic 3D scans. Combining different views of multi-contrast scans into high-resolution isotropic 3D scans is challenging due to the lack of a large training cohort, which calls for a subject-specific framework. This work proposes a novel solution to this problem leveraging Implicit Neural Representations (INR). Our proposed INR jointly learns two different contrasts of complementary views in a continuous spatial function and benefits from exchanging anatomical information between them. Trained within minutes on a single commodity GPU, our model provides realistic super-resolution across different pairs of contrasts in our experiments with three datasets. Using Mutual Information (MI) as a metric, we find that our model converges to an optimum MI amongst sequences, achieving anatomically faithful reconstruction.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43993-3_17

SharedIt: https://rdcu.be/dnwNi

Link to the code repository

https://github.com/jqmcginnis/multi_contrast_inr/

Link to the dataset(s)

https://www.med.upenn.edu/cbica/brats2020/data.html

http://portal.fli-iam.irisa.fr/msseg-challenge/

Reviews

Review #1

Please describe the contribution of the paper

This paper with title: Multi-contrast MRI Super-resolution via Implicit Neural Representations leverages NIF to enable 3D subject specific multi-contrast super-resolution from low-resolution scans.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. This paper propose an interesting approach to perform 3D subject-specifc super resolution from multi-contrast 2D low-resolution scans. Due to the beauty of NIF, this approach won’t the supervision of high-resolution ground-truth and doesn’t require pretraining.
2. The authors use mutual information as a assessment metric, which is a clever idea.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. One of my main concerns of this approach is that since its predicting the contrast from the coordinate, I assumed that the images have to be perfectly alianed. I wonder how could you achieve that and curious how sensitive is the results to the registration quality. forexample, if the constrat images are not perfectly aligned, what will the results be? Would appreciate if you could elaborate on this.
2. In the methods section, f is decomposed to g - for contrast and q for anatonical properties, however, this paper lack the analhsis on how f and q do their jobs. What would be output of q only, would appreciate this ablation and visualization.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Yes, the authors claimed to release the code.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

Would appreciate the visualization of g1/g2 and q learned using this approach.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Very interesting work, and definitely has the potential for clinical applications. I have a few minor concerns, in general, feel positive on the quality of this paper.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

The manuscript proposes a novel MRI super-resolution using the implicit neural representation. The method learns two contrasts of complementary views and exchanges anatomical information between them. The method achieves realistic and faithful super-resolution and can handle missing data and partial views.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The manuscript provides a clear motivation, a thorough literature review, a detailed technical description, and a comprehensive experimental evaluation.
2. This manuscript is novel and interesting in improving MR image resolution with INR. Although not the first, it further presents the feasibility of improving image quality by fusing information from multi-contrast and multi-view MRI with INR.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. My major concern is the inadequate experimental design. 1.1 Baseline methods in single-contrast MRI super-resolution:
  - Authors compared their proposed method with cubic-spline interpolation, which was invented decades ago. Authors are encouraged to compare their proposed method with more recent methods.
  - For example, prior to the deep-learning era, model-based SR methods can up-sample LR MR images well with a defined image degradation model and hand-crafted image prior, without any training data, e.g., the LRTV method (10.1109/TMI.2015.2437894).
  - Further, although it is difficult to train a neural network without real HR images, it is not IMPOSSIBLE. For example, SMORE (10.1109/TMI.2020.3037187) is a self-supervised MRI super-resolution which trained a SR model with LR images only.
  - Also, it will be interesting to compare the proposed INR-based method to a full-supervised deep-learning SR method, to know the performance gap between SOTA SR solution and the proposed subject-specific method.
1.2 Data.
- All experiments were conducted on sagittal/coronal/axial T1w/T2w images synthesized from isotropic/HR acquisition, instead of real clinical multi-view MRI, which are far more complicated than down-sampling HR images.
- For example, authors assume the sagittal/coronal/axial scans to be orthogonal, which is definitely not the case.
- Further, it is not clear how authors down-sample HR images, but obviously the partial volume effect is not taken into consideration. What are the slice thickness and slice spacing after and before down-sampling?
1. Also, other than experimental designs, it is also not clear that what prior knowledge is used in the reconstruction of HR images in single contrast INR? It is well known that SR is an ill-posed problem, thus it is not possible to reconstruction HR image from LR ones without any prior knowledge. Model-base SR methods such as LRTV assumes MR images to be low-rank, and full-supervised deep-learning SR method learns prior knowledge from LR-HR exemplifiers. What about INR?
2. Further, I still have a minor comment in the hyper-parameter tuning. It is not clear how hyper-parameters such as \alfa and \beta in equation (1) determined.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

This manuscript seems to be reproduceable with publicly available dataset and open-source code.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

Refer to “main weaknesses” for more details.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

4
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Although method proposed in the manuscript is very attractive and novel, and SR with INR is promising, the experimental designs in this manuscript are faaaaaaaaaaaaar from satisfactory (i.e. weakness slightly weigh over merits). Personally, I was very excited before reaching the experiment section, but I ended up feeling very disappointed after reading the whole manuscript.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

This work proposes a novel solution to this problem lever- aging Implicit Neural Representations (INR). Our proposed INR jointly learns two different contrasts of complementary views in a continuous spatial function and benefits from exchanging anatomical information between them. The paper is easy to follow, the author also provide some impovements over the baselines.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The author use the Mutual Information (MI) [24] as an evaluation metric and find that our method preserves the MI between high-resolution ground truths in its predictions. Further observation of its convergence to the ground truth value during training motivates us to use MI as an early stopping criterion.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. My most critical concern is regarding the review of the related works. The authors totally lacked extensive multi-contrast MRI works. I suggest that the author should review some related works in the introduction, please see an incomplete list here,
  - Exploring Separable Attention for Multi-Contrast MR Image Super-Resolution[J]. arXiv preprint arXiv:2109.01664, 2021.
  - Feng C M, Fu H, Yuan S, et al. Multi-contrast mri super-resolution via a multi-stage integration network MICCAI 2021
2. My second critical concern is regarding the baselines. I strongly suggest the authors use more relevant state-of-the-art multi-contrast methods (see Q1) to evaluate your method.
3. My third concern is the downsapmling patterns. The author do not describe the details of the LR. In my opinion, the MRI LR image is very different from natural image, please see Multi-contrast mri super-resolution via a multi-stage integration network MICCAI 2021.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors have not provided the parameter settings. Did the authors implement a cross-validation procedure to optimize parameter selection for individual baselines? Omission of this step might introduce biases.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

I strongly suggest the authors use more relevant state-of-the-art multi-contrast methods (see Q1) to evaluate your method.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

4
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

lack of most related works for review and comparison, thereby leveraging the effectiveness of the proposed method to be doubtful.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

Although R1 is very positive about the novelty, reviewers generally have strong concerns about the evaluation and comparison to the state-of-the-art. Please address these concerns in your rebuttal.

Author Feedback

We thank all reviewers for their comments and excitement for INRs in patient-specific multi-contrast (MC) super-resolution (SR). We are encouraged reviewers found our method novel and interesting (R1,R2,R3); doesn’t require high-resolution (HR) ground truth (R1); aptly uses mutual information (R1,R3); is comprehensively evaluated (R2) and has clinical potential (R1). Below we address open questions.

Problem setting & related work (R3): There are misunderstandings about our problem and experiment settings. 1) We deal with single-subject SR, not population-based, i.e. we only use low-resolution (LR) images from one subject for training. 2) We perform 3D anisotropic out-of-plane upsampling, not 2D isotropic in-plane upsampling. 3) We DO NOT have access to HR scans from any contrast. All of these are highly relevant to real-world clinical problems. Since MINet & SANet are 2D, population-based, fully supervised methods and require auxiliary HR contrast in the input, they are NOT APPLICABLE as baselines in our setting. Although MINet was already cited in our submission, we have now added SANet in related work.

Single-contrast (SC) Baselines (R2,MR2): Thank you for the suggestion, we have now added LRTV & SMORE for all datasets in the manuscript. Due to space constraints, we report PSNR, SSIM & LPIPS here.

Method | PSNR C1 | SSIM C1 | LPIPS C1 | PSNR C2 | SSIM C2 | LPIPS C2 BraTS LRTV | 21.32 | 0.91 | 0.05 | 24.20 | 0.91 | 0.05 SMORE | 26.26 | 0.94 | 0.03 | 28.46 | 0.94 | 0.03 MSSEG LRTV | 22.84 | 0.86 | 0.05 | 23.92 | 0.87 | 0.04 SMORE | 25.72 | 0.93 | 0.03 | 27.43 | 0.94 | 0.02 cMS LRTV | 28.72 | 0.90 | 0.03 | 22.76 | 0.83 | 0.05 SMORE | 28.93 | 0.92 | 0.04 | 25.33 | 0.92 | 0.03

SMORE performs on par or slightly worse to SC INR baseline, while LRTV struggles as observed in [1].

Downsampling (R2,R3): To synthetically create 2D LR images, one needs to anisotropically downsample out-of-plane in the image domain [1] while preserving in-plane resolution. To R3: MINet uses isotropic in-plane downsampling and is not applicable in our anisotropic 2D clinical protocol. To R2: To mimic realistic 2D clinical protocol, which often have higher in-plane details than that of 3D scans, we use spline interpolation to model partial volume and downsampling. We also experimented with averaging-based downsampling and did not find changes in the relative performance among methods. In all experiments, slice thickness before and after downsampling is 1 mm and 4 mm respectively with a gapless acquisition.

Datasets (R2): Since public datasets lack 3D HR and corresponding 2D LR, we follow common practice [1,2] and downsample images. We highlight our INR’s versatility on different datasets and contrasts (T1w,T2w,DIR,FLAIR). Upon submission, clinical partners have initiated acquisition of both 2D LR & 3D HR scans, and ongoing experiments on this data have replicated our findings indicating clinical feasibility.

SC INR Prior (R2): Experiments with priors, e.g. Laplacian, led to worse performance, which we attribute to implicit smoothness prior in INRs.

Registration & Orthogonality (R1,R2): This is an important aspect, in particular for clinical use. Even though we used orthogonal views, our method can exploit non-orthogonal scans without any modification since INRs operate on global coordinates, unlike CNNs that require orthogonal slices. We observed that rigid registration has reliably aligned non-orthogonal contrasts into a shared global coordinate system in all our experiments.

q & g decomposition (R1): Since a single MLP implicitly models them jointly, we do not have explicit access to q & g. Investigating such decomposition is relevant to our future work.

Hyperparameters (R2,R3): We tune hyperparameter on hold-out sets (c.f. supplementary). Alpha/beta are ratios of the number of voxels between two contrasts.

[1] Zhao, Can, et al. “SMORE … deep learning.” TMI 2020 [2] Wu, Qing, et al. “IREM … representation.” MICCAI 2021

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The use of INR in reconstruction is a novel application in medical imaging. The authors have addressed the concerns about experiments in the rebuttal. It would be an interesting work to discuss on the MICCAI avenue.

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

After reading reviews from reviewers and the authors’ feedback, this work still needs to be improved in certain aspects. The paper cannot meet the standard of MICCAI in its current form. Therefore, I would recommend the rejection of this work.

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

In the rebuttal, the authors have addressed the comments satisfactorily. They have included more comparison methods. They have also provided valid justifications for why they cannot do some of the modifications required by the reviewers.

back to top

Single-subject Multi-contrast MRI Super-resolution via Implicit Neural Representations