List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Jiale Wang, Alexander F. Heimann, Moritz Tannast, Guoyan Zheng
Abstract
Deep learning-based algorithms for single MR image (MRI) super-resolution have shown great potential in enhancing the resolution of low-quality images. However, many of these methods rely on supervised training with paired low-resolution (LR) and high-resolution (HR) MR images, which can be difficult to obtain in clinical settings. This is because acquiring HR MR images in clinical settings requires a significant amount of time. In contrast, HR CT images are acquired in clinical routine. In this paper, we propose a CT-guided, unsupervised MRI super-resolution reconstruction method based on joint cross-modality image translation and super-resolution reconstruction, eliminating the requirement of high-resolution MRI for training. The proposed approach is validated on two datasets respectively acquired from two different clinical sites. Well-established metrics including Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Metrics (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) are used to assess the performance of the proposed method. Our method achieved an average PSNR of 32.23, an average SSIM of 0.90 and an average LPIPS of 0.14 when evaluated on data of the first site. An average PSNR of 30.58, an average SSIM of 0.88, and an average LPIPS of 0.10 were achieved by our method when evaluated on data of the second site.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43907-0_48
SharedIt: https://rdcu.be/dnwdu
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes an unsupervised MRI super-resolution reconstruction method based on joint cross-modality image translation and super-resolution reconstruction using HR CT images as guidance. The network design features a SRNet and a CITNet that work jointly to generate high-quality pseudo HR MR images. This approach eliminates the need for paired LR and HR MRI images for training, which can be difficult to obtain in clinical settings. Experimental results show that the proposed approach outperforms state-of-the-art methods on two different clinical datasets.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Significant performance boost.
- The proposed method is validated on two different clinical datasets.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- CT and MRI are different modalities. It would be hard to prove the info learned from CT is not hallucination
- The computational complexity and runtime of the proposed method are not discussed in detail, which could be an important factor for clinical adoption.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Code will be released and the reproduce is possible
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- Would be good to discuss the hallucination problem mentioned above
- Would be good to report the runtime of the model
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Consider the cross-modality hallucination issue, I give the above rating
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This paper proposes a CT-guided, unsupervised MRI super-resolution reconstruction method based on joint cross-modality image translation and super-resolution reconstruction. Even though the idea of cross-modality image translation is old, the idea of merging it with super-resolution reconstruction is novel.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The presented method is interesting and also has some limited novelty. The method has been presented well. Results were compared with statistical tests, and the three stages of the methods were evaluated with the help of an ablation study. Improvements over the baselines are statistically significant, and qualitative comparisons are convincing.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
“The acquired medical images have high in-plane resolution but low inter-plane resolution, i.e., large spacing between slices” - this statement in the introduction isn’t always completely true. Undersampling is usually performed to speed-up MR acquisitions that might decrease the in-plane resolution or even introduce artefacts. So, the statement “The acquired medical images have high in-plane resolution but low inter-plane resolution” might be true, even might not be true. The wording must be changed to accommodate this. “will lead to poor visual experience” - isn’t a scientific way to define the problem. It’s better to rephrase it to address the issues (e.g. misdiagnosis).
Baselines should be updated. Method, such as “PARCEL: Physics-based Unsupervised Contrastive Representation Learning for Multi-coil MR Imaging” by Wang et al. was published in 2022 - would have been an interesting baseline. Moreover, the supervised baseline used in the manuscript is from 2018. For example, “ReconResNet: Regularised residual learning for MR image reconstruction of Undersampled Cartesian and Radial data” by Chatterjee et al., published in 2022, might be a better baseline.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The method has been explained well, should be sufficient to reproduce.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The authors should present a summary of the results (e.g. scores of their method vs baseline) in the abstract. Fig. 1 A.1 SRNet: the input image does not look low resolution but rather distorted. This should be updated with an accurate image. The same goes for B.1 and B.2. If this is because the undersampling was in the slice direction, and then now they are plotted by changing the plane, this should be clarified in the figure caption. Minor grammatical improvements are required (e.g. missing articles).
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper has its merits, improvements over the baselines are statistically significant, and qualitative comparisons are convincing. But needs some minor improvements to make it more acceptable.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper propose a CT-guided, unsupervised MRI super-resolution reconstruction method based on joint cross-modality image translation and super-resolution reconstruction, eliminating the require- ment of high-resolution MRI for training. Experimental results demonstrate the superior performance of the proposed approach over the state-of-the-art methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The proposed CT-guided, unsupervised MRI super-resolution reconstruction method based on joint cross-modality image translation and super-resolution reconstruction, eliminating the requirement of HR MRI for training.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
My most critical concern is regarding the review of the related works. The authors lacked extensive multi-contrast MR imaging works. I suggest that the author should review some related works in the introduction, please see an incomplete list here,
- Feng C M, Yan Y, Liu C, et al. Exploring Separable Attention for Multi-Contrast MR Image Super-Resolution[J]. arXiv preprint arXiv:2109.01664, 2021.
- Feng C M, Fu H, Yuan S, et al. Multi-contrast mri super-resolution via a multi-stage integration network MICCAI 2021
The second concern is the implementation details. The authors have not provided the parameter settings. Did the authors implement a cross-validation procedure to optimize parameter selection for individual baselines? Omission of this step might introduce biases.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper has present details on how to implement the method from scratch.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Were the experiments repeated with different seeds and did the trend in the results remain the same? Were the differences between the baselines and the proposed method statistically significant?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The novelty seems a little limited, as CT-guided reconstruction of MRI is a traditional technique.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper provides an interesting perspective on super-resolving 3D MRI. The method is well presented and reasonably designed. It is also well validated on two datasets with promising results. The authors are invited to respond to the concerns of the reviewers. In particular, 1) discuss the hallucination issue of learning MRI from CT (Reviewer #1), 2) update the selection of the baseline (Reviewer #2), and 3) introduce and compare the proposed method to related works on multi-contrast MR imaging (Reviewer #3). Besides, the expression about in-plane resolution and inter-plane resolution may be revised as suggested by Reviewer #2.
Author Feedback
We thank meta-reviewer (MR) and all reviewers for their comments
MR,R1 Hallucination issue Following [C1], hallucination refers as “artifacts or incorrect features that cannot be produced from the measurement”. We argue that this is not the case for our method from following two aspects. First, we design a three-stage training strategy where the first two stages aim for pre-training SRNet and the last stage aims for joint optimization of CITNet and SRNet which is done via disentangled representation learning (DRL). It has been shown previously in [14] that DRL can be used to achieve unsupervised super-resolution reconstruction when trained with unpaired low resolution (LR) and high resolution (HR) MR images. Second, if info learned from the CT would be hallucination, then it would be reflected in the results and should not be influenced by the size of training data. However, quantitative results presented in Table 1 show that our method achieves the best results that are close to the supervised method, and that our method achieves better results on site1 (35 training data) than on site2 (13 training data), reflecting the influence of the size of training data. This indicates that info learned from the CT is not hallucination.
[C1] Bhadra et al. On hallucinations in tomographic image reconstruction. IEEE Trans Med Imaging. 2021 Nov; 40(11): 3249–3260.
MR,R3 Related works on multi-contrast MR imaging (MR,R3) and limited novelty (R3) We will cite the related works as suggested by R3. Here we would like to point out the differences between these works and ours. Both related works are supervised methods leveraging diverse yet complementary information embedded in different MR imaging. Thus, the LR-HR multi-contrast image pairs need to be co-registered in advance, which is tedious and time-consuming. In contrast, our method is an unsupervised method, eliminating the requirement of HR MR images and the cross-modality LR-HR image registration, which is novel and has a clear advantage.
MR,R2 Supervised baseline update Both PARCEL and ReconResNet are MR reconstruction methods using undersampled k-space data while our method does not need to access k-space data and is based on pure image data. Additionally, ReconResNet is a supervised method while our method is unsupervised cross-modality method. As ReconResNet contains an image super-resolution module, we adapt it as a supervised baseline and evaluate it on our data. Our method achieves results close to ReconResNet on site1 (PSNR: Ours 32.23 vs. theirs 32.93; SSIM: Ours 0.90 vs. theirs 0.88; LPIPS: Ours 0.14 vs. theirs 0.09) but better results than ReconResNet on site2 (PSNR: Ours 30.58 vs. theirs 29.97; SSIM: Ours 0.88 vs. theirs 0.84; LPIPS: Ours 0.10 vs. theirs 0.07).
MR,R2 Concern on expression about in-plane resolution and inter-plane resolution We agree with MR&R2. Thus, we will revise it accordingly.
R1 Computational complexity (CC) and runtime The CC is 153 GFlops. The average runtime is 20ms for super-resolving one single image from site1 dataset with an upscale factor of 4 on a NVIDIA RTX A6000 GPU.
R2 Presenting summary results in Abstract and concerns on Fig.1 We thank R2 for the kind suggestion. We will revise the abstract accordingly. Your observation on Fig. 1 is also correct. Following your kind suggestion, we will clarify it in the figure caption.
R3 Implementation details on parameter settings As presented in Paragraph 1, Page 8, we split the dataset of site 1 into training (35 volumes), validation (5 volumes), and testing sets (10 volumes). We optimize parameter selection on the validation dataset of site1 for individual baselines before we conduct comparison study on testing dataset.
R3 Statistically significant differences? As presented in the caption of Table1, we did paired t-tests on all metrics to compare our method and other unsupervised methods. The obtained p-values are all smaller than 0.0001, indicating statistically significant difference.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have adequately addressed the concerns of the reviewers. Most importantly, the hallucination issue is explained, and the reason why the suggested baseline is out of scope is given.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper proposes a deep learning model for reconstructing high-resolution 3D MR images guided by CT. The method is clearly presented, the validation is reasonable and the results are good. Concerns raised by the reviewers have been addressed in the rebuttal.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
After going through the reviews and the author’s rebuttal, the concerns of the reviewers have been addressed adequately. While one of the reviewer thought the method had limited novelty, they also thought that the paper was well written, the results were compared with statistical tests, and the three stages of the methods were adequately evaluated with an ablation study. The method yielded a significant performance boost. The idea of using a CT-guided, unsupervised MRI super-resolution reconstruction method based on joint cross-modality image translation and super-resolution reconstruction was perceived as a strength. Importantly, the authors showed significant improvements over the baseline methods, and the comparisons were convincing.