Authors

Wanyu Bian, Qingchao Zhang, Xiaojing Ye, Yunmei Chen

Abstract

Generating multi-contrasts/modal MRI of the same anatomy enriches diagnostic information but is limited in practice due to excessive data acquisition time. In this paper, we propose a novel deep-learning model for joint reconstruction and synthesis of multi-modal MRI using incomplete k-space data of several source modalities as inputs. The output of our model includes reconstructed images of the source modalities and high-quality image synthesized in the target modality. Our proposed model is formulated as a variational problem that leverages several learnable modality-specific feature extractors and a multimodal synthesis module. We propose a learnable optimization algorithm to solve this model, which induces a multi-phase network whose parameters can be trained using multi-modal MRI data. Moreover, a bilevel-optimization framework is employed for robust parameter training. We demonstrate the effectiveness of our approach using extensive numerical experiments.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16446-0_34

SharedIt: https://rdcu.be/cVRTt

Link to the code repository

N/A

Link to the dataset(s)

https://www.med.upenn.edu/sbia/brats2018/data.html

Reviews

Review #2

Please describe the contribution of the paper

In this paper, a learnable variational model is proposed for joint multimodal MRI reconstruction and synthesis. Numerical experiments are performed to validate this method.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

(1) The proposed joint MRI reconstruction and synthesis comes with theoretical analysis. (2) A bilevel optimization is used for parameter updating. (3) Experimental results, especially some visual results zoomed in on tumor regions, shows its potential utility and value in practice.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

(1) The Introduction is not well organized. It seems more like stack the review of compressed sensing MRI and MRI synthesis without presenting their link. What is the value of study MR image synthesis under the setup of partially acquired MRI measurements of other modalities. In my opinion, this is the key issue to be addressed to show the value of this paper. (2) Following the first concern, is this paper the first work to explore the joint MRI reconstruction and synthesis? If so, it should be stressed since it is a major contribution. If not, related works should be discussed. (3) Another major issue is that the rationale or analysis should be provided on why the third term in Equation (1) is a reasonable design? Citing related papers, visualized analysis or explanations should be useful. (4) The technical details should be better organized and concise to make the paper more readable and understandable to MICCAI community.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The method presented in this paper is loaded with technical details. I think it would be challenging to reproduce this work without the help of publicized codes.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

(1) The authors are encouraged to strengthen the introduction section. (2) A detailed analysis and explanation of the proposed model is recommended. (3) Paper should be reorganized to make the presentation of the optimization uncluttered.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The theoretical analysis, the novelty w.r.t. application and the presented experimental results are advantageous for this work. However, the listed weakness of the work prevented a higher rating.
Number of papers in your stack

5
What is the ranking of this paper in your review stack?

2
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

In this paper, the authors present a learnable optimization algorithm to jointly do the image reconstruction and image synthesis, the method takes undersampled radial k-space MRI acquisition and outputs the reconstructed image as well as the synthesized third-modality image.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. This paper delivers an solid work on jointly reconstruct undersampled MRI images and perform the image synthesis, the paper is well written and include a thorough analysis and metric comparison. The method is also well described, In general, a very solid work.
2. In terms of both quantitative metric, as well as the visual image quality, the proposed method provide superior results compared to other methods, also, including the results in the regions of pathologies (Fig 2) delivers more diagnostic value of the paper.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. One weakness is that the paper doesn’t clearly describe how the k-space data was generated, since, as far as I know, the BRATS 2018 dataset is not for raw-kspace, please carefully describe the data processing steps. Also, please check carefully on this paper: Implicit data crimes: Machine learning bias arising from misuse of public data, which shows some potential data crime for MRI reconstruction.
2. The undersampling strategy for this paper is radial sampling, and 40% sampling rate is kind of considered as a “low” acceleration factor. Would be nice to elaborate more on how does the undersampling matters and what happened if the sampling is cartesian.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors didn’t claim that they will release the code.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1. Please add clear description on how you pre-processed the data and generate the k-space inputs, which is important for people to use it and evaluate it.
2. I have a comment on those contrast synthesis works in general: usually, deep learning model can succeed in medical image synthesis because the output information is encoded in the inputs (e.g., High Fidelity Direct-Contrast Synthesis from Magnetic Resonance Fingerprinting in Diagnostic Imaging). But for those T1w –> T2w, I don’t think the contrast information of T2w is encoded in T1w, nor FLAIR, which means that its hard to guarantee that the output contrast is authentic, can you elaborate on this point?
3. In the paper, you showed a few examples on from 2 contrasts to 1 contrast, I wonder do you have any experience on generating 2 contrasts from 1 contrast?
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The authors provide a novel framework for joint reconstruction and image synthesis, which is of interest to radiologist. Though more evidence/analysis on different sampling patterns and acceleration rate is needed, this work is insightful and solid.
Number of papers in your stack

8
What is the ranking of this paper in your review stack?

2
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #4

Please describe the contribution of the paper

Based on the theoretical analysis, this paper proposes a new deep model, which can simultaneously reconstruct the source modal image from partially scanned k-space MR data and synthesize the target modal image without any k-space information. In addition, the network proposed in this paper adopts the double-layer optimization training algorithm to optimize the network parameters and super Cen book, and a large number of experiments on different modes of brain magnetic resonance data verify the effectiveness of the proposed algorithm.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

In this paper, the convergence of multi contrast joint reconstruction and generation learnable algorithm is guaranteed by controlling the super parameters, and the idea of meta learning is used to optimize the super parameters to obtain the balance of multiple regular terms.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

This paper proposes to use the idea of meta learning to optimize the hyperparameter, but it does not show that the parameter is the main limitation affecting the problem, and there are not enough experiments to verify the advantages of the optimized parameters over the general hyperparametric design.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

This paper can be reproduced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1. The convergence of the learnable algorithm is proved in the paper, which can show the convergence curve of the optimization objective, so as to ensure the consistency of the theoretical and experimental results.
2. Add experiments to verify the advantages of double-layer optimization superparametric design over general fixed superparametric design.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This paper proposes a learning algorithm with theoretical analysis, and the generation accuracy of reconstruction accuracy has been greatly improved. However, the main contribution of this paper has been proposed, so the innovation of this paper should be improved.
Number of papers in your stack

6
What is the ranking of this paper in your review stack?

4
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper presented a novel deep learning framework to jointly conduct MRI reconstruction from undersampled k-space MRI acquisition and multi-modal MRI synthesis. All reviewers acknowledged the methodological contributions of the proposed method, the theoretical analysis to guarantee the optimization convergency, and the promising experimental results of potential values in practice. They agreed that this work is solid and insightful, and above the acceptance level, although it could be further strengthened with some detailed suggestions from the reviewers.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

1

Author Feedback

N/A

back to top

A Learnable Variational Model for Joint Multimodal MRI Reconstruction and Synthesis