List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Weitong Zhang, Berke Basaran, Qingjie Meng, Matthew Baugh, Jonathan Stelter, Phillip Lung, Uday Patel, Wenjia Bai, Dimitrios Karampinos, Bernhard Kainz
Abstract
Abdominal MRI is critical for diagnosing a wide variety of diseases. However, due to respiratory motion and other organ motions, it is challenging to obtain motion-free and isotropic MRI for clinical diagnosis. Imaging patients with inflammatory bowel disease (IBD) can be especially problematic, owing to involuntary bowel movements and difficulties with long breath-holds during acquisition. Therefore, this paper proposes a deep adversarial super-resolution (SR) reconstruction approach to address the problem of multi-task degradation by utilizing cycle consistency in a staged reconstruction model. We leverage a low-resolution (LR) latent space for motion correction, followed by super-resolution reconstruction, compensating for imaging artefacts caused by respiratory motion and spontaneous bowel movements. This alleviates the need for semantic knowledge about the intestines and paired data. Both are examined through variations of our proposed approach and we compare them to conventional, model-based, and learning-based MC and SR methods. Learned image reconstruction approaches are believed to occasionally hide disease signs. We investigate this hypothesis by evaluating a downstream task, automatically scoring IBD in the area of the terminal ileum on the reconstructed images and show evidence that our method does not suffer a synthetic domain bias.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43999-5_12
SharedIt: https://rdcu.be/dnwwq
Link to the code repository
https://github.com/vito1820/MoCoSR
Link to the dataset(s)
https://portal.gdc.cancer.gov/projects/TCGA-LIHC
Reviews
Review #2
- Please describe the contribution of the paper
This paper proposes a deep learning-based approach to improve the quality of abdominal MRI for patients with inflammatory bowel disease. The proposed approach uses a staged reconstruction model with cycle consistency and a low-resolution latent space for motion correction.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors compare their proposed approach to conventional, model-based, and learning-based MC and SR methods, and investigate the hypothesis that learned image reconstruction approaches occasionally hide signs of disease. I think the evaluate a downstream task, which automatically scoring IBD in the area of the terminal ileum on the reconstructed images, is very important and show evidence that their method does not suffer from synthetic domain bias.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The main weakness I see in this work is the complexity of the system and the lack of comparison with simpler approaches based on traditional super-resolution CNN. Additionally, the ablation study conducted is quite minimal.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors could have provided more information about important hyperparameters such as batch size and learning rate. Unfortunately, no code seems available to replicate or build upon the proposed approach.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Major:
My main concerns are related to the use of high-resolution images in the pipeline. Since you have these high-quality HR volumes Y that are fed into the system, it is not clear to me why you did not simply use a paired SR approach obtained by degrading these HR images and train a super-resolution network that maps LR to HR, instead of proposing this complex adversarial training approach. Additionally, there is no comparison with such paired approaches, for example, Enhanced Deep Super-Resolution Network and Fast Super-Resolution Convolutional Neural Network (FSRCNN) and it is hard to understand the contribution of this work in comparison to this type of approaches.
The improvement over some existing approaches is minimal, despite the large complexity added to the system.
The ablation study presented in the supplementary material is based on the downstream task rather than on quality metrics such as SSIM or PSNR. In my opinion, it would have been important to investigate these metrics as well.
Many details on the pipeline are missing, and this could affect the reproducibility of this work.
Minor: There are some problems with the references (i.e., in Table 2, you are referring to work [25], which is a survey. I believe you are referring to [24] instead).
Regarding the statement about handling multiple tasks at different scales, it would be helpful if you could provide more specific information about what you mean there.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
No comparison is provided with paired approaches, and since the HR images are available and used in this context, such a comparison is essential to understand the contribution of the paper.
The ablation study is also minimal and should have been extended to include all the quality metrics considered in the main work.
More details on the training of the pipeline need to be provided for better reproducibility of this work.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
4
- [Post rebuttal] Please justify your decision
The authors have not adequately address my concerns. Although I acknowledge that paired approaches may have limitations in some contexts in comparison to the un-paired ones, your work has not clearly demonstrated or explained these limitations in your context.
Your responses regarding the motivations provided and in the rebuttal are still unclear to me. For example, I am unable to understand how your un-paired approach can effectively address the issue of insufficient high-quality reference images with respect the paired methods, as this has not been adequately demonstrated. Additionally, the claim that “predetermined degradation kernels on high-resolution data (EDSR, FSRCNN) fail to consider factors such as motion corruption and the anisotropic nature of MRI” requires validation, and these baseline approaches should have been compared in your experiments. Considering the availability of high-resolution images and degraded images within your context and dataset, it would have been important to include a paired approach in your experiments. This would have allowed for a comprehensive comparison of the advantages and disadvantages of the paired approach in relation to your proposed method. This comparison, is not unfair as all the methods are trained on the same dataset. Differently was the case if you couldn’t train the paired approaches at all (i.e. different context).
Moreover, the authors have not discuss the minimal improvement over existing approaches, despite the significant complexity introduced. Also there are numerous crucial details that are still missing from the manuscript.
Minor issue. You have overlooked my point regarding the sensitivity study. While it is good that you considered the downstream task, my question was why the other metrics used in the main experiment were not included in the ablation study presented in table 1 of the supplementary material.
Taking all of these factors into consideration, I would like to maintain my initial score.
Review #3
- Please describe the contribution of the paper
The paper proposes MoCoSR, a deep adversarial super-resolution reconstruction approach for abdominal MRI that addresses respiratory motion and organ movements. Major contributions are 1. accounting for non-isotropic voxel sizes to reconstruct spatial dimension, 2 eliminating the need for semantic knowledge and manual paired-annotation, 3. avoiding the requirement for acquiring multiple image stacks. MoCoSR demonstrates superior performance compared to various image SR reconstruction algorithms while preserving clinically relevant features.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
A thorough evaluation of the proposed method using publicly available datasets with simulated respiratory motion and an internal dataset with inherent motion, demonstrating superior performance compared to state-of-the-art methods.
The demonstrated clinical feasibility of the method through a downstream disease prediction task, which showed no decrease in prediction performance compared to traditional reconstruction methods.
The clear presentation and organization of the paper, making it easy to understand and follow the proposed method and evaluation process.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
While the evaluation process is strong, it would be beneficial to have a more extensive clinical validation of MoCoSR’s performance in diagnosing specific diseases or conditions. The paper does not provide a detailed discussion of potential limitations or challenges associated with implementing MoCoSR in clinical practice.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Code will be available on GitHub.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
It would be helpful to provide more specific details about the simulation of motion using PRMR.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The novelty and significance of the proposed method, MoCoSR, which outperformed existing state-of-the-art algorithms for reconstructing high-quality SR MRI in the presence of motion artifacts.
The thorough evaluation of MoCoSR on two different datasets with varying degrees of motion, as well as the evaluation of clinical relevance using a downstream disease prediction task.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
My concerns have been addressed in rebuttal. I trust the authors would act proactively to add downstream task-related limitations and challenges in the final version.
Review #4
- Please describe the contribution of the paper
This paper proposes an unsupervised deep learning framework MoCoSR for joint motion correction and super-resolution of abdominal MRI. The low-resolution motion-corrupted images are first motion-corrected with a CLR encoder and then upscaled to high-resolution images via an SR decoder. Another pair of encoder of decoder is used in the reverse direction for cycle consistency, which allows learning from unpaired data. The method alleviates the need for manual paired annotation and was evaluated on two datasets as well as a downstream classification task.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The idea of jointly optimizing motion correction and super-resolution to alleviate the need for paired data is novel. The proposed cycle consistency learning and loss functions are convincing and justified with sufficient comparisons and ablation studies.
- The performance was further justified with a downstream classification task, showing that the proposed method can successfully reconstruct the features of Crohn’s disease and produce negligible classification degradation compared to real high-resolution images.
- Very detailed description of datasets and data-acquisition parameters.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Some descriptions of the equations and figures are missing (see below). The writing could be improved.
- More qualitative comparisons between different methods should be provided in Fig.4.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- As stated in the paper, the code will be available on GitHub.
- The work depends on private datasets. It is unsure whether they will make it public.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- The caption of Fig.1 is not enough. What does each colored arrow mean? What does the “first pair” refer to? The definitions of the symbols, such as Y_hat and Y_tilde, are not found in the paper.
- Some typos. For example, the second line above Equation 2, the first Z_{tilde}_Q should be Z_Q.
- The symbols used in Equation 4 are confusing. The discriminators in Fig.1 are applied to ZQ and Y_hat, but here are three different ones.
- Is the architecture design of GLRB original? Perhaps it was based on some previously proposed modules, such as residual learning and multi-scale learning. Please provide some citations.
- Fig.4 definitely needs at least one or two more examples, from different subjects. Have you studied other upscaling factors?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Novel and convincing work which needs improvement in writing.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
The rebuttal clarifies some technical details. I keep my original opinion to accept the paper. Please try to include more examples for Fig.4 in the final version.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper addresses the importnat need of isotropic MRI reconstruction of the abdomen from anisotropic data. Direct isotropic acquisition can be time consuming and not always clinically feasible. Therefore the authors suggest to leverage deep neural netwrok architetcure for generatign high-resolution images (at the through slice plane) from low resolution images. To cope with motion in the abdomen additional motion correction sub-network is proposed. Two revewiers were in favor of the paper while the thrid raised some concerns. Main strengthes is clinically-relevant problem, an innovative approach, and rigorous evalution. Yet, the concerns raised by the thrid reviewer should be addressed by the authors, including clarity of the method, explanation of motivation to use adverserial training rather than fully supervised approach and the comparison with simple baseline method.
Author Feedback
We are grateful that the reviewers and the AC appreciate our novel approach to combining motion correction (MC) with super resolution (SR), and that we have verified our methods clinical viability by evaluating its influence on downstream tasks. We address the remaining reviewers’ questions below.
Advantage over supervised approach [R2] Although fully supervised methods have a simpler pipeline, their reliance on paired data is a limitation. (1) In a clinical setting it is difficult to acquire sufficient high-quality reference images to train with a pairwise correlation, due to both physical limitations (resolution) and the human cost (time). (2) Methods used of predetermined degradation kernels on high-resolution data (EDSR, FSRCNN) fail to consider factors such as motion corruption and the anisotropic nature of MRI. This puts an upper bound on the ability of a model trained under the simple inverse problem formulation, as the forward model is not representative of the real-world degradation process.
As noted by R3 & R4, our methods use of adversarial training to avoid using paired data is therefore a great strength. In contrast to traditional SR CNNs that rely on paired SR and require semantic knowledge about the intestines, our approach, MoCoSR, offers a solution that mitigates these requirements. Although the learnt kernels increase the complexity, they are able to better model the true degradation process, meaning that our learning of the inverse problem is more valid.Sensitivity study [R2] While our framework is one of the first to take clinical needs into account, it also requires a unique experimental setup and evaluation that has not been previously employed across image post-processing domains and automatic Crohn’s disease diagnostic. The evaluation on both datasets and the ablation study in Tab.2 have provided a comprehensive analysis of the SR reconstruction performance with respect to SSIM and PSNR. For our downstream task experiments, they are sensitivity-oriented rather than reconstruction-oriented with respect to clinical evaluation metrics. We believe that the ultimate metric for a successful method lies in improved utility for clinical downstream tasks.
Lack of comparison with simpler SR [R2] Comparison to further SR methods with fixed kernels, beyond those presented in the paper, would be unfair, since these simpler methods are highly sensitive towards the chosen degradation kernel and data fit. They cannot be directly compared due to the requirement to learn from paired data and the inability to perform the MC and SR joint tasks. Instead, a more comprehensive study including conventional interpolation, model-based, and learning-based methods is feasible for joint MC and SR, as demonstrated in Fig.3,4, and Tab.2.
Clarification of methods [R4] The flow of inputs to the corresponding-colored loss function is illustrated by the colored arrows. The “first pair” refers to the encoder E_{CLR} and decoder D_{SR}, with the other pair used only for training. The orange flow, \tilde{Y} is generated from the high-quality Y using the encoder-decoder pair. These will be clarified in the main body of the paper and Fig.1 caption. We’ll include the references for GLRB. We conducted experiments with 2x scaling, however 4x is more challenging, hence we present the results of 4x here. These will be clarified and put the results into Fig.4 and appendix.
More discussion [R3] Our manuscript will also add downstream task-related limitations and challenges into the discussion, as well as diseases-specific validation and performance issues.
Reproducibility [R2, R3] The specific details of simulation will be provided in code. We will clarify the pipeline and statement to offer a comprehensive understanding.
Writing typo [R2, R3, R4] Thank you for your diligent work in spotting typos that we will correct all errors.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Authors addressed in their rebuttal the reviewers concerns. Overall a paper that can be of interest to the MICCAI community.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The main strengths of the paper include the novel formulation of jointly optimizing motion correction and super-resolution without the need for paired data, the thorough evaluation using publicly available datasets and an internal dataset, and the demonstration of clinical feasibility through a downstream disease prediction task. The authors compare their proposed approach to conventional, model-based, and learning-based methods, and provide evidence that their method does not suffer from synthetic domain bias. In general, this paper is at the borderline.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper received diverse ratings (2 accepts and 1 reject). In the rebuttal phase, the authors addressed those issues raised in the initial reviews. Unfortunately, R2’s big concerns still maintained in the post-rebuttal evaluation, including the clarifications on the motivations, limitations, trade-off between performance gain and complexity increase. Therefore, I agree with R2 on the above issues. I do not think the paper is ready to publish in MICCAI until those big concerns are cleared. I have to reject the paper at the current phase.