List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Siyuan Dong, Eric Z. Chen, Lin Zhao, Xiao Chen, Yikang Liu, Terrence Chen, Shanhui Sun
Abstract
High-quality MRI reconstruction plays a critical role in clinical applications. Deep learning-based methods have achieved promising results on MRI reconstruction. However, most state-of-the-art methods were designed to optimize the evaluation metrics commonly used for natural images, such as PSNR and SSIM, whereas the visual quality is not primarily pursued. Compared to the fully-sampled images, the reconstructed images are often blurry, where high-frequency features might not be sharp enough for confident clinical diagnosis. To this end, we propose an invertible sharpening network (InvSharpNet) to improve the visual quality of MRI reconstructions. During training, unlike the traditional methods that learn to map the input data to the ground truth, InvSharpNet adapts a backward training strategy that learns a blurring transform from the ground truth (fully-sampled image) to the input data (blurry reconstruction). During inference, the learned blurring transform can be inverted to a sharpening transform leveraging the network’s invertibility. The experiments on various MRI datasets demonstrate that InvSharpNet can improve reconstruction sharpness with few artifacts. The results were also evaluated by radiologists, indicating better visual quality and diagnostic confidence of our proposed method.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16446-0_55
SharedIt: https://rdcu.be/cVRTY
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposed an invertible sharpening network (InvSharpNet) that adapts a backward training streategy that learns a blurring transform from the fully-sampled MR images to the underssampled images to improve the visual quality of MR image reconstruction.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper newly adopted an invertible sharpening network for reconstructing undersampled MR images which could be different from conventional deep-learning-based MR image reconstruction methods.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) The proposed model lacks novelty. It seems that the proposed model was built by simply using pre-existed invertible networks. To highlight the novelty and demonstrate the effectiveness of the proposed model, more explicit explanations and rigorous experiments would be needed. 2) Although results of radiologists’ evaluation showed better performance than the baseline (Table 1), it is difficult to see the performance increment in the presented figures (Fig. 2, 3). Also, quantitative evaluation metrics showed degraded performance compared to other models.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors provided details about the proposed models, datasets, and evaluation. However, the reference code was not provided and and does not seem to be released after the review process.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1) The specific reasons for the increase in performance fore reconstructing MR images from undersampled data by replacing an existed network with an invertible network is unclear. More explicit explanations and rigorous experiments would be needed. In particular, it seems that the proposed model was built by simply changing the existing CNNs with a pre-existed invertible network. 2) The experiemtns were conducted only with cWGAN and InvSharpNet models and lacked comparison with other deep learning-based MR image reconstruction methods. To show the effectiveness of the invertible networks, more rigorous experiments would be needed by applying them to other deep learning-based MR reconstruction methods and compared with other methods. 3) Although adiologists’ evaluation showed outperformed than the baseline in Table 1, it is hard to see the performance increment in Fig. 2 and 3. Especially, the difference between cWGAN and InvSharpNet seems to be very minor. In addition, the quantitative metrics that did not follow the trend of the radiologists’ evaluation. Though the authors mentioned about this, but still it is not clear. The quantiative metrics are widely used for evaluating the reconstruction performance in various papers. Please provide more supporting results and discussion about this.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Even though the experiments showed interesting aspects with the proposed method, the overall opinion is “weak reject” because of the mentioned weaknesses and execution of the idea.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
(1) This paper proposed an invertible network for MRI reconstruction enhancement to make the image sharper. (2) The experiments demonstrate that the proposed InvSharpNet can improve reconstruction sharpness.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
(1) This proposed invertible network is a one-to-one mapping which relieves the ambiguity of the final reconstructed image. (2) The motivation of this work is clear and the method adopted is intuitive. (3) Experiments verify the effectiveness of the proposed method.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
(1) There is already some work using invertible network for image denoise, for example[1]
[1] Invertible Denoising Network: A Light Solution for Real Noise Removal, CVPR 2021.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The dataset used in this paper is publicly available. This paper has the reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
(1) Please fully investigate the relevant literatures and compare with them in experiments. (2) The generalization validation part is not very clear.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The motivation of this work is clear, the method adopted is intuitive, and the experimental verification is sufficient, so I recommend to accept.
- Number of papers in your stack
6
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
An invertible sharpening network (InvSharpNet) is proposed to improve visual quality of reconstructed MRI images. The authors propose a backward training strategy to train InvSharpNet from the ground truth image to the reconstructed image, and the inference is in the opposite direction.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The idea of applying the invertible neural network (INN) to enhance undersampled MRI reconstruction is novel. The INN improves the visual quality of the reconstructed image, which is not fully discussed in the current related works.
- The reconstructed image using the InvSharpNet has a better qualitative results, as shown in Table 1.
- The experiments are comprehensive - evaluation on 3 datasets, comparison on SOTA algorithm and GAN-based method.
- Generalizability & network size are discussed, the Lipschitz constant experiments are included.
- The paper is easy to follow.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-Quantitative results with the proposed methods is inferior to the comparison method, as shown in Table 2. -Fig.1(a) is not clear, and the font is too small.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The training details are missing. (optimizer, # of epochs etc.) The evaluation is on 2 public datasets and 1 private dataset.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- Overall it is an interesting paper, and discussion on the visual quality is an important topic.
- More insights on the InvSharpNet will strengthen this paper, for example: -More discussion and analysis of how is this different from a loss function is necessary. Given that additional training and parameters are needed, is the proposed methods overweight the loss function? (such as perceptual loss etc.)
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
It is a novel paper with extensive experiments. While the quantitative results is not superior, the presentation, analysis and discussion worth presenting in the conference.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
The rebuttal address most of my concerns, therefore, i keep my accept score. Specifically,
- The claim of converting “one-to-many problems” into “one-to-one problems” is a good intuition/motivation, but i suggest to include more mathematically proof in future work.
- The quantitative score problems are explained by the authors, and this remains an open problem in the community.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The invertible sharpening network proposed for the MRI reconstruction is interesting and shows some merits. However, there are some concerns about its technical novelty and its performance against other competing methods. Therefore the work is invited for rebuttal to address the major concerns that the reviewers have raised, particularly on the methodological novelty and its differences from previous works, its degraded performance against other approaches and insights of the strength of the proposed method.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6
Author Feedback
We thank the constructive feedback from all reviewers and appreciate the very positive comments from two reviewers.
Insights of the proposed method and differences from previous works (R1, R2, R3) One fundamental reason for the blurry image problem is the ambiguity of one-to-many mapping. To approach this problem, we propose a new learning framework by converting the traditional one-to-many problem to a one-to-one problem, differentiating it from previous works that propose new loss functions but still in a conventional learning fashion. The rationale is that given a fixed under-sampling mask, one fully-sampled image corresponds to one under-sampled image. Given a fixed reconstruction (recon) network, one under-sampled image corresponds to one blurry recon. Therefore, InvSharpNet learns a one-to-one mapping from a fully-sampled image to a blurry recon. Since the Lipschitz constraint guarantees that this learned mapping is invertible, the inference can be performed in the opposite direction. The paper “Invertible Denoising Network” (R2) focuses on disentangling clean and noisy signals in the latent space, which is very different from the learning framework we propose.
Novelty (R1) Our work does not “simply use pre-existed INN” (R1) but provides a learning framework with INN to solve the blurry image problem. To the best of our knowledge, this work is the first to learn an inverse mapping from ground truth to input data, which converts a one-to-many problem into a one-to-one problem and overcomes the blurry issue. Besides, we are the first to combine an INN with a DC layer to ensure that the sharpened image is strictly consistent with the measurement, which is crucial for medical images. Our work also provides new insights on how Lipschitz constraints and network size can affect the performance of an INN (Fig. 3(b)).
Degraded quantitative scores (R1, R3) We aim to improve the perceptual sharpness of MRI recon. PSNR/SSIM are often degraded as sharpness improves due to the well-known tradeoff described in the paper “The perception-distortion tradeoff,” making those metrics ineffective in evaluating our improvement. Our results were evaluated from a clinical perspective using radiologists’ ratings, which was considered as a key evaluation criterion in fastMRI competitions [8][13]. The radiologists examined the artifacts and sharpness of the possible pathological areas and rated that our method provides the highest diagnostic confidence (Table 1). Besides, we quantified image sharpness using a “contrast” metric (Table 2 and A1), for which InvSharpNet outperforms the recon models (p-value<0.05). In our final version, we will present an additional metric LPIPS (proposed in “The unreasonable effectiveness of deep features as a perceptual metric”), which measures images’ high-level similarity and correlates well with human perceptual judgment. InvSharpNet achieves better LPIPS on all three datasets compared to the recon models and refinement network (e.g. on fastMRI knee dataset, Recon model 1: 0.078±0.032, InvSharpNet(backward): 0.068±0.025, lower is better).
Comparison with more MRI recon methods (R1) We clarify that our work is not a recon method, i.e. from under-sampled to fully-sampled images. Instead, InvSharpNet is an image enhancement method, which directly sharpens the recon given by previous recon methods, so our method is not directly comparable to other recon methods.
Improvement over cWGAN (R1) cWGAN with γ=0.02 tends to generate artifacts (Fig. A3), which is confirmed by the low artifacts score from radiologists. InvSharpNet achieves high radiologists’ scores for both sharpness and artifacts, which indicates our method can improve the sharpness while maintaining minimal artifacts compared to cWGAN.
Generalizability (R1, R2) We applied InvSharpNet to enhance results from two recon models. Fig. 3(a) and Table A1 show that our method has a similar performance on recon model 2 as on recon model 1.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The work is well motivated and shows promising results. The rebuttal has further clarified the approach and explained about the concerns on its novelty and quantitative performance. Though the technical novelty might not be significant, its application to the problem can still be interesting to the readers. Therefore acceptance is recommended.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This manuscript proposed to use an invertible network to perform image sharpening in the context of knee MRI. The major concerns were the novelty, as invertible networks have been proposed for other tasks, and only incremental improvements in quantiative scores.
The rebuttal addressed the challenges with quantitative scores, especially when trying to assess image quality that is often a combination of image properties making it difficult to assess mathematically with measures such as SSIM and PSNR. Overall, this paper demonstrates the use of an invertible network in an interesting way with sufficient experimental evaluation to demonstrate its utility.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Overall, the authors have carefully addressed the main concerns of the reviewers, especially on clarifying the novelty of the proposed methodology. With the authors’ commitment to incorporate all reviewers’ feedback in their revised manuscript, I recommend accepting this paper without further questions.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3