List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Beliz Gunel, Arda Sahiner, Arjun D. Desai, Akshay S. Chaudhari, Shreyas Vasanawala, Mert Pilanci, John Pauly
Abstract
Unrolled neural networks have enabled state-of-the-art reconstruction performance and fast inference times for the accelerated magnetic resonance imaging (MRI) reconstruction task. However, these approaches depend on fully-sampled scans as ground truth data which is either costly or not possible to acquire in many clinical medical imaging applications; hence, reducing dependence on data is desirable. In this work, we propose modeling the proximal operators of unrolled neural networks with scale-equivariant convolutional neural networks in order to improve the data-efficiency and robustness to drifts in scale of the images that might stem from the variability of patient anatomies or change in field-of-view across different MRI scanners. Our approach demonstrates strong improvements over the state-of-the-art unrolled neural networks under the same memory constraints both with and without data augmentations on both in-distribution and out-of-distribution scaled images without significantly increasing the train or inference time.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16446-0_70
SharedIt: https://rdcu.be/cVRUe
Link to the code repository
https://github.com/ad12/meddlr
Link to the dataset(s)
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposed scale-equivariant unrolled neural networks by modeling the proximal operators of the networks with scale-equivariant CNNs to improve the data-efficiency and robustness to variants for reconstructing MR images from undersampled k-space data.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper newly combined scale-equivariant CNNs with unrolled nerual networks for reconstructing undersampled MR images which could be different from conventional deep-learning-based MR image reconstruction methods.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) The proposed model lacks novelty. It seems that the proposed model was built by simply changing the existing CNNs with pre-existed scale-equivariant CNNs. To highlight the novelty and demonstrate the effectiveness of the proposed model, more explicit explanations and rigorous experiments would be needed. 2) Although quantitative results showed better performance than the baseline (Table 1, 2), it is difficult to see the performance increment in the presented figures (Fig. 1).
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors provided details about the proposed models, datasets, and evaluation. As the authors stated, the reference code seems be released after the review process.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1) The specific reasons for the increase in performance by combining the scale-equivariant CNNs and an unrolled architecture is unclear. More explicit explanations and rigorous experiments would be needed. In particular, it seems that the proposed model was built by simply changing the existing CNNs with pre-existed scale-equivariant CNNs. 2) The experiemtns were conducted only with vanilla, scale-eq, and rotation-eq models and lacked comparison with other deep learning-based MR image reconstruction methods. To show the effectiveness of the scale-equivariant CNNs, more rigorous experiments would be needed by applying them to other deep learning-based MR reconstruction methods and compared with other methods. 3) Although the quantitative results of the proposed network showed better performance than the baseline in Table 1 and 2, it is difficult to see the performance increment in the presented Fig. 1. Especially, the difference between Vanilla+ and Scale-Equivariant+ seems to be very minor. Please provide figures that can show the effectiveness of the proposed model.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Even though the experiments showed better performance with the proposed method, the overall opinion is “weak reject” because of the mentioned weaknesses and execution of the idea.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
(1) This paper proposed a scale-equivariant unrolled network for MRI reconstruction. (2) The experiments demonstrate that the proposed approach outperforms state-of-the-art unrolled neural networks
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
(1) The paper is very well written and the presentation is clear. (2) As far as my knowledge goes, inserting a scale-equivariant proximal network for MRI reconstruction is a novel and interesting idea. (3) Good evaluation: comparison to state-of-the-art unrolled neural networks, generalization validation.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
(1) Some important references are missing, for example, unrolled neural networks [1][2], equivariant network for inverse problem [3].
[1] Deep ADMM-Net for compressive sensing MRI, 2016 [2] A deep cascade of convolutional neural networks for MR image reconstruction, 2017 [3] Equivariant Imaging: Learning Beyond the Range Space, 2021
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The dataset used in this paper is publicly available. This paper has the reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
(1) Please fully investigate the relevant literatures. (2) Give more qualitative comparison results.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The motivation of this work is clear, the method adopted is intuitive, and the experimental verification is sufficient, so I recommend to accept.
- Number of papers in your stack
6
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
The paper introduces the use of scale-equivariant (scale and translation) networks for learned unrolled reconstruction methods, where the proximal operator is replaced by a neural network.
The geometric constraint on the network offers a better performance without the need for data augmentation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The proposed network architecture provides better reconstruction results even under constraints on the network.
The use of scale equivariance is well motivated for practice.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
It is said that the proposed changes provide data efficient training options. Unfortunately, the only comparison that is done in the study is based on data augmentation vs none. I would have hoped for a stronger support of that claim by reducing the training data size.
Reconstruction results are all in supplementary, some results would have been also good in the main body.
A limitation to only 90 degree rotations does not take slight rotations of patients into account.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Data is publicly available and codes will be provided
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The topic of designing networks that work better for smaller amount of data is important, especially when considering limited possibilities to obtain ground-truth references in medical imaging.
A curious result is that the authors achieve better performance with the geometrically constrained network than the vanilla version. This is curious, usually slightly worse performance is observed under additional constraints of the network (here it is by including a scale invariance) I would have welcomed a short note on this in section 5.
As mentioned above, one of the aims of the authors is to provide data efficient learning. Sadly a study on the influence of data size is missing. The argument is only underlined by eliminating the need for data augmentation, but that is done from the same data set anyways, so it does not reduce the amount of data.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
It is well motivated and results are good. Novelty lies mainly in transferring the use of scale-invariant networks to MRI reconstructions, which is limited in novelty.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The scale-equivariant proximal network for MRI reconstruction is well motivated and is an interesting idea. Results shown are also promising. However, there are some concerns about the novelty of the approach and the influence of data size in the proposed data-efficient setting. The work is invited for rebuttal to address the raised concerns from reviewers, particularly on its novelty and comparison against the provided reference, the data size influence and its qualitative performance.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
8
Author Feedback
We thank the reviewers for their helpful feedback. We are encouraged they find our work novel & interesting (R2), well-motivated for practice (R3); and our evaluation (R2) & results (R2,R3) good.
Regarding the novelty of our work (R1,R3), we note that we are the first to study scale-equivariance in the context of any inverse problem to the best of our knowledge. We believe introducing scale-equivariance to the image reconstruction community and validating its utility over the state-of-the-art accelerated MRI reconstruction networks across a variety of settings (with and without appropriate data augmentations on both in distribution and out-of-distribution scaled images) is both a novel and a significant contribution. We also emphasize the practical relevance of our proposal, given it enables reconstruction networks that are data-efficient (important as suggested by R3) and robust to drifts in scale of the images that could be caused by the variability of patient anatomies or change in field-of-view across different MRI scanners.
R1 suggested that the qualitative performance improvement with scale-equivariant networks as shown in Fig 1 was marginal. We believe the error map visualizations make it clear that the reconstruction of the Vanilla+ network is considerably worse than that of the Scale-Eq+ network. However, for the final version, we will replace this with another example to address R1’s comments, and move the figure to the main paper as suggested by R3.
R3 requested a study on the influence of data size, which we agree is a valuable analysis to show. While we cannot provide additional results as per the rebuttal instructions, intuitively, the positive impact of scale-equivariance would be even stronger at smaller dataset sizes. This is because scale-equivariance enforces additional priors onto the convolutional weights, and these priors would intuitively become less important as more training data is provided. We previously validated this claim, and could include results in the final version, if allowed.
R3 expressed a concern about how constrained weights in the scale-equivariant networks would improve the performance over vanilla networks. We emphasize that this is because it enforces priors in the structures of the network weights that would be difficult to learn otherwise, especially with small datasets. As we discussed in Section 3.3., vanilla CNNs are equivariant to translation, which is understood as the reason they are often more effective for computer vision problems compared to standard fully-connected networks, despite constraining the network weights. In other words, scale-equivariance encodes a lack of prior knowledge of the scale of the structures in the images. More broadly, the improved performance through using scale and rotation equivariant convolutions has been replicated in a variety of works (Sec. 2).
R1 suggested we apply scale-equivariance ideas to other DL-based MR reconstruction methods. We would like to clarify that state-of-the-art for this task is achieved by unrolled neural networks motivated by classical variational regularization approaches, hence Vanilla+ unrolled neural network is a particularly strong baseline here, given it also leverages the state-of-the-art data augmentation method, MRAugment.
R3 raised a concern regarding being limited to only 90 degree rotations. We consider rotation-equivariant networks a baseline here, following Celledoni et al.’s work where they picked 90 degree rotations based on a reconstruction performance ablation with respect to the degree of rotation. To clarify, as stated in our paper, “We do not aim to argue that scale symmetry is more useful to encode than rotation symmetry here, in fact, we demonstrate that encoding scale symmetry is as helpful as encoding rotation symmetry, if not more.”
Lastly, we thank R2 for their comments. As space permits, we will include the suggested literature and more qualitative figures.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The key strength of the work is the proposal of the scale-equivariant unrolled network for the MRI reconstruction, which has not yet been explored in this field and shows promising results. The rebuttal has further well clarified the approach, though the data size influence would be more convincing if results could be provided. In general, the work will be interesting to the community despite minor drawbacks.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This research developed scale-equivariant unrolled neural networks by modelling the networks’ proximal operators with scale-equivariant CNNs in order to increase data-efficiency and resilience to variations while reconstructing MR images from undersampled k-space data. As pre-rebuttal mentioned, the work is interesting and I can see the author’s efforts to clear reviewers’ concerns. However, no reviewer has changed their scores that brought up final scores as 4/7/5 which is not high-ranked in my deck. I am inclined to accept this work accredit to its scientific merits.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors propose a scale-equivariant unrolled network for MRI reconstruction. The authors have addressed the concerns on novelty and influence of data-size in the rebuttal. Therefore, I recommend acceptance of the work.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6