List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Jiamiao Zhang, Yichen Chi, Jun Lyu, Wenming Yang, Yapeng Tian
Abstract
Limited by the drawbacks of imaging systems, the reconstruction of Magnetic Resonance Imaging (MRI) images from partial measurement is an important problem in medical imaging research. Benefiting from the diverse and complementary information of multi-contrast MR images in different imaging modalities, multi-contrast super-resolution (SR) reconstruction is promising to yield SR images with higher quality. In the medical scenario, to fully visualize the lesion, radiologists are accustomed to zooming the MR images at arbitrary scales rather than using a fixed scale, as used by most MRI SR methods. What’s more, existing multi-contrast MRI SR methods often require a fixed resolution for the reference image, which makes the acquisition of reference images difficult and imposes limitations on arbitrary scale SR tasks. To address these issues, we proposed a dual-arbitrary multi-contrast MRI super-resolution method, called Dual-ArbNet. First, we decouple the resolution of the target and reference images by a feature encoder, enabling the network to input target and reference images at arbitrary scales. Then, an implicit fusion decoder fuses the multi-contrast features and use a Implicit Decoding Function~(IDF) to obtain the final MRI SR results. Furthermore, we applied the curriculum learning to train our Dual-ArbNet in several stages. Extensive experiments in two public MRI datasets demonstrate that our method outperforms state-of-the-art approaches under different scale factors and has great potential to be applied in clinical practice.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43999-5_27
SharedIt: https://rdcu.be/dnwwG
Link to the code repository
https://github.com/jmzhang79/Dual-ArbNet
Link to the dataset(s)
http://brain-development.org/ixi-dataset/
Reviews
Review #1
- Please describe the contribution of the paper
This paper introduces a multi-contrast MRI superresolution method based on implicit neural representations to attain arbitrary scale SR. The technical contributions included an implicit fusion decoder to merge results from separate contrasts, and a curriculum learning strategy for training.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The use of implicit neural representations fused across multiple contrasts is a rather new approach to MRI superresolution. Comprehensive demonstrations are provided against prior art in two public datasets for various SR factors.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Components of the network architecture (RDN, CA, SA, and the sinuosoidal implicit layers) are relatively well known, as well as the curriculum learning strategy.
Arbitrary-scale superresolution has some practical applications in computer vision, but for radiological imaging one cannot go beyond the precribed spatial resolution of the modality, so the motivation is not very clear.
Several important prior art in the domain of multi-contrast MRI reconstruction with the intent to improve resolution (e.g., doi: 10.1109/JSTSP.2020.3001737) is omitted in the paper.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors have provided sample code and a demo that aids reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The demonstrations should be expanded to cover distinct types of MRI contrasts (e.g., T1, T2, FLAIR). Reliability against motion between the separate contrasts of a given subject should be examined/discussed. Literature survey can be expanded. Motivation for arbitrary scale SR in MRI can be clarified.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
While the overall approach makes sense in terms of adopting the recent implicit representation for SR in MRI, and relatively comprehensive demonstrations are included, there are several major issues that should be addressed.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The paper proposes a dual-arbitrary multi-contrast MRI super-resolution method, called Dual-ArbNet, for the task of MRI super resolution. It decouples the resolution of the target and reference images by a feature encoder to enable arbitrary scales. And an implicit fusion decoder fuses the multi-contrast features and uses an Implicit Decoding Function (IDF) to obtain the final MRI SR results. The method achieves SOTA results on two public datasets.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The experimental results are strong, as illustrated by the tables.
- Ablation studies are extensive and convincing. All the modules are verified by the experimental results.
- The idea using implicit neural representations for the task of arbitrary-scale SR is novel.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The difference between the proposed method and existing methods is not sufficiently discusssed. For example, [21]. At the same time, [4] and [16] also employ implicit neural representations for the SR task. What is the major difference?
- The idea using curriculum learning is not well illustrated.
- The presentation is not clear. For example, what does the “dual” refers to in the title?
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Not sure about this aspect.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
1, in the full loss function, as there are only two terms, only one hyper-parameter is needed (the other one can be taken as default 1).
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The idea is interesting and the experimental results well support the method.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
The authors well addressed my concerns. I vote for acceptance.
Review #3
- Please describe the contribution of the paper
This manuscript proposes a framework for multi-contrast MRI SR with the implicit neural representation, called Dual-ArbNet which supports arbitrary scale SR at any reference MRI resolution. Besides that, it introduces a curriculum learning technique named “Cur-Random” to increase the network’s stability, generalization, and multi-contrast fusion performance. On two benchmark datasets, fastMRI, and IXI, Dual-ArbNet got the SOTA performance.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
Dual-ArbNet which supports arbitrary scale SR at any reference MRI resolution. It consists of an encoder and an implicit fusion decoder. The encoder performs feature extraction and alignment of the target LR and the reference image. The implicit fusion decoder predicts the pixel values at any coordinate by fusing the features and decoding through the implicit decoding function, thus achieving reconstruction.
-
Cur-Random” uses common scale(2,3,4) for warmup training and then changes the task difficulty to random scale for the rest training. It has been proven to be effective in their ablation study.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The experiments contain only PSNR values for the evaluation metrics of the experimental results, and the single evaluation metrics may not show the objectivity of this experiment. Please consider to add other metric such as MS-SSIM Besides, in the ablation study, the experimental results of “Random with HR” seems to be comparable to those of “Cur-Random with HR”, which makes the Cur-Random learning strategy may not so that convincing, please give detailed explained or more supportive evidence
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
This paper has very good reproducibility which provides testing code, training code, and data. Besides that, it also writes down the specific environment needs in a text file.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Please address my concerns in weakness parts.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
It is a good manuscript overall, please still enhance it by addressing my concerns in weakness
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
In the paper the authors present a method that targets super-resolution for multi-contrast MRI. Although the reviewers consistently recommend the paper for acceptance (a weak accept and two accepts), they still raised several major concerns that are required to be addressed before acceptance. I looked at the paper as well. I do not think the paper in its current version is that strong to get an early accept at this stage. Therefore, I invite the authors to clear the concerns from all reviewers in the rebuttal phase.
Author Feedback
We thank all reviewers for their insightful comments. We address the raised concerns below. (R1) Components of the network and curriculum learning strategy. To the best of our knowledge, we are the first to apply implicit neural representation in multi-contrast MRI super-resolution. To achieve the decoupling of the reference image scale and the target image scale, we find that using the implicit function is a good tool. In addition, we are the first to introduce curriculum learning into SR due to the complex scale pairs of target and reference images. In the future, we will explore more effective module in MRISR. (R1) Motivation of MRI super-resolution. Great question! Our work aims to super-resolve MR images at arbitrary scales rather than a fixed one. However, it’s important to note that these scales will fall within the modality’s prescribed spatial resolution, as constrained by the training data. (R1) About omitting of prior art. Thanks for the suggestion! We have cited and discussed the suggested work in our revised paper. Since its code is not currently available, we cannot include it in the comparative experiment. (R1) Constructive comments. Thanks for your constructive advice, we will further clarify the motivation for arbitrary scale MRI SR. And we will further expand the application to distinct types of MRI contrasts and discuss the motion between separate contrasts. (R2) Difference between ours and existing methods. [21] is based on simple matrix multiplication upsampling, and the process did not take into account the locality of the image. The major difference between our method and [4]/[16] is that we achieved joint SR for any scale reference image and any scale input image in multi contrast MRI. Additionally, our Dual-ArbNet specially designed fusion branches and dual branch implicit neural networks. (R2) Idea of using curriculum learning. The curriculum learning strategy imitates the human learning process, allowing the network to learn from simple samples and gradually transition to complex samples. In practice, we consider HR reference images as simple samples because they are rich in texture and can provide more useful information to the network. Similarly, we consider low quality LR reference images as complex samples. Specifically, we divide network training into three stages. The ablation study in Table 2 demonstrates the quantitative effect of curriculum learning and we also plotted the PSNR and SSIM results of the validation set during the entire training process at 2×, 3×, 4× SR scales, and we found that the convergence speed of curriculum learning was significantly faster than that of random learning. (R2) The meaning of “Dual” in title. The network achieves multi-contrast MRISR for “arbitrary” reference resolution and “arbitrary” input resolution. Thus, it is called “Dual”-ArbNet. (R2) Loss function. Thanks for your advice, we will simplify the expression of the loss function. (R3) Evaluation metrics. Due to space constraints, we only provide PSNR in Table1, but we also measured MS-SSIM values in our experiments. For SR scales of 1.5/2/3/4/6/8, the MS-SSIM scores are 0.977/ 0.949/ 0.915/ 0.890/ 0.848/ 0.801 for fastMRI dataset, and 0.997/ 0.993/ 0.988/ 0.984/ 0.971/ 0.949 for IXI dataset. The results are significantly higher than other comparison methods. We will add these results. (R3) Evidence of Cur-Random strategy. “Cur-Random with HR” is more efficient and stable. Starting with a high-resolution (HR) reference, not a low-resolution (LR) one, ensures steadier output from the Implicit Decoding Function (IDF). This approach also converges faster than “Random with HR”, as evidenced by our PSNR and SSIM results of the validation set during the entire training.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors provided a nice rebuttal to address the reviewers’ concerns. The questions raised by the reviewers in the initial review phase have been well clarified. I happy to recommend to accept this paper for the publication of MICCAI23.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper presents a novel technique for enhancing MRI superresolution through implicit neural representations. The key highlight lies in the introduction of Dual-ArbNet, an architecture capable of achieving superresolution at any scale, backed by robust experimental outcomes. The paper excels in terms of reproducibility, offering testing code, training code, and data to facilitate replication of the results. Additionally, it provides a detailed text file specifying the necessary environment requirements.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have sufficiently addressed the comments regarding the motivation of their work, the differences over existing methods, and experimental details. They have noted that the reviewers’ comments will be incorporated into the final version of the paper. Hence, I suggest accepting this paper for MICCAI.