List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Jun Lyu, Bin Sui, Chengyan Wang, Yapeng Tian, Qi Dou, Jing Qin
Abstract
Multi-contrast magnetic resonance imaging (MC-MRI) has been widely used for the diagnosis and characterization of tumors and lesions, as multi-contrast MR images are capable of providing complementary information for more comprehensive diagnosis and evaluation. However, it usually suffers from long scanning time to acquire multi-contrast MR images; in addition, long scanning time may lead to motion artifacts, degrading the image quality. Recently, many studies have proposed to employ the fully-sampled image of one contrast with short acquisition time to guide the reconstruction of the other contrast with long acquisition time so as to speed up the scanning. However, these studies still have two shortcomings. First, they simply concatenate the features of the two contrast images together without digging and leveraging the inherent and deep correlation between them. Second, as aliasing artifacts are complicated and non-local, sole image domain reconstruction with local dependencies are far from enough to eliminate these artifacts and achieve faithful reconstruction results. We present a novel Dual-Domain Cross-Attention Fusion (DuDoCAF) scheme with recurrent transformer to comprehensively address these shortcomings. Specifically, the proposed CAF scheme enables deep and effective fusion of features extracted from two modalities. The dual-domain recurrent learning allows our model to restore signals in both k-space and image domains, and hence more comprehensively remove the artifacts. In addition, we tame recurrent transformers to capture long-range dependencies from the fused feature maps to further enhance reconstruction performance. Extensive experiments on public fastMRI and clinical brain datasets demonstrate that the proposed DuDoCAF outperforms the state-of-the-art methods under different under-sampling patterns and acceleration rates.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16446-0_45
SharedIt: https://rdcu.be/cVRTE
Link to the code repository
https://github.com/XAIMI-Lab/DuDoCAF
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
Towards better multi-contrast MRI reconstruction, this manuscript proposes a dual-domain cross-attention fusion mechanism to make full use of a reference image, and a recurrent transformer to remove the non-local aliasing artifacts.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Manuscript is well organized and easy to follow. Motivation is clear and persuasive. Methods are well formulated and presented. Experiments are comprehensive and informative. The proposed method is compared to various baselines and is evaluated on different sampling strategies and accelerating ratios.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Experiments are conducted on real-valued single-coil MRI dataset (i.e. simulated dataset), which it is far from real scenarios where reconstruction algorithms deal with complexed-valued multi-coil MRI. Also, it is not clear how this proposed method can be extended to multi-coil MRI. This should be clarified.
The proposed method contains a so-called Swin Transformer Layers, but from my point of view, the key ideas of swin transformer (patch merging and W-MSA/SW-MSA) are not mentioned in the main text. If I misunderstand, patch merging and W-MSA/SW-MSA
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Although some details are missing, it is still reproduceable, considering that the proposed methods is well presented, and authors promised to release code after acceptance.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
It is not clear to me how to extend the proposed method to complex-valued and multi-coil MRI, which is more practical in real MRI reconstruction scenarios. (c.f. main weaknesses of this paper.)
Also, authors claim the proposed method is evaluated on fastMRI dataset, which is somehow misleading. Usually the fastMRI dataset is reference to the raw complexed-valued multi-coil data, while actually only simulated data are used in this manuscript.
Extending the description of residual swin transformer block (RSTB) would make this method more clear. For example, details like strategy patch merging and W-MSA/SW-MSA in Section 2.3.
Is the TGR sub-module necessary in the proposed CAF module? It seems that the target of multi-contrast MRI reconstruction is to borrow information from reference image to target image, why the proposed method needs a target guided reference (TGR) sub-module?
I wonder why authors still need CNN layers instead of powerful non-local Transformer layers at the end of RRT and CAF modules in Fig. 1? In the context of Transformer, element wise addition and linear layers are preferred for image information fusion and feature extraction.
Minor Comments Inconsistent under-sampling mask ($k_u$, Cartesian mask) and artifacts (Radial mask) in intermediate reconstruction result ($\tilde{x}_{u_1}$) in Fig. 1.
Content and contribution of section 2.3 is too weak for a subsection. This subsection should be further extended or simply merged/scattered into other sections.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The organization, motivation, novelty look good to me. For experiments, while selection of dataset has some defects, other parts, the baselines and under-sampling strategies are still reasonable and comprehensive.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
Thanks for addressing the previously raised concerns.
The response looks good to me except the description on dataset. If authors do NOT use the complex-valued multi-coil MRI cohort, this should be clearly and explicitly emphasized in the manuscript, instead of just saying the “Public fastMRI dataset [9], following…”, which is quite MISLEADING (cf. the second comment in “detailed and constructive comments”).
Also, I would encourage the authors to discuss their future works / limitations on multi-coil MRI in the Conclusion section.
Review #2
- Please describe the contribution of the paper
The authors present a novel dual-domain cross-attention fusion network with recurrent transformer for fast multi-contrast MR imaging. The cross-attention fusion scheme enables deep and effective fusion of features extracted from two modalities. The dual-domain recurrent learning allows the proposed model to restore signals in both k-space and image domains by removing the artifacts effectively. The recurrent transformers can capture long-range dependencies from the fused feature maps for improving reconstruction performance.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1) The cross-attention fusion mechanism fuses the features extracted from two contrast images in a bidirectional way to harness complementary information of these two contrasts. 2) The residual-reconstruction transformer to model the long-range dependencies based on the fused feature maps in both domains to counteract aliasing artifacts and faithfully reconstruct the target images. 3) The recurrent dual-domain learning makes the reconstruction results more interpretative.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1)The authors need to analyze the computational complexity of the proposed method in comparison with baseline methods. 2)The authors need to discuss the impact of the differences between MR images from healthy subjects and those from patients on the performance of the proposed method. 3)The authors need to provide in-depth discussion on the consistence of multi-contrast image synthesis obtained by the proposed method.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The results of this paper may be reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1)The authors should provide the analysis on the computational complexity of the proposed method. 2)The authors need to explain whether the proposed method has consistent performance for different modalities. 3)The authors need to give clear explanations on whether Individual health status affects the performance of the proposed method. 4)Font size of Fig. 1 is too small. 5)There are a few typos and grammar errors, such as “the attention outputs is introduced by using one contrast image ”, “The TGR runs in the same way and thus form a”, “. In i-th image domain restoration block”, “groups in out network” and “which means that learn the interaction between two different contrasts step by step is optimal”.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The authors need to provide enough experimental data for method validation.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
This paper proposes a dual domain deep learning framework for MRI multi-contrast super-resolution. According to the results, the proposed method can generate superior results when compared with some other methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is well organized and has presented enough figures and tables to support authors’ ideas.
It is novel of the proposed dual domain network.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
In k-space CAF, feature maps are cropped into smaller patches and then embedded and input into RGT and TGR. Mathematically, each data point in k-space will contribute to every pixel in the image domain after Fourier transform. Please example the reason for cropping k-space.
It is very interesting to know the improvement of each recurrent block. Please consider replacing Fig, 3 to show results from each recurrent block.
In the ablation study, please provide more details on the network structure design for each experiment show in Table 2 as the statement of w/o CAF, w/o DD in the paper is confusing.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Reproducibility is good.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
This paper proposes a dual domain deep learning framework for MRI multi-contrast super-resolution. According to the results, the proposed method can generate superior results when compared with some other methods. The paper is well organized and has presented enough figures and tables to support authors’ ideas. However, there are some major concerns.
In k-space CAF, feature maps are cropped into smaller patches and then embedded and input into RGT and TGR. Mathematically, each data point in k-space will contribute to every pixel in the image domain after Fourier transform. Please example the reason for cropping k-space.
It is very interesting to know the improvement of each recurrent block. Please consider replacing Fig, 3 to show results from each recurrent block.
In the ablation study, please provide more details on the network structure design for each experiment show in Table 2 as the statement of w/o CAF, w/o DD in the paper is confusing.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Novelty, experimental design, result presentation.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposed a dual-domain cross-attention fusion network with recurrent Transformer for fast multi-contrast MR imaging. All reviewers acknowledged the technical contribution of this work in proposing dual-domain cross-attention fusion mechanism to harness the complementary information of different contrasts and recurrent-transformer to remove the non-local aliasing artifacts. However, there are still mixed comments that need to be clarified during rebuttal. For example, Reviewer #1 requested comments about how to extend the proposed method to complex-valued, multi-coil MRI that is more conformable to real scenarios; Reviewer #2 asked for the computational complexity, the performance consistency for different modalities, and the clarification about whether pathology could affect the model performance; and Reviewer #3 requested the explanation of the reason for cropping k-space.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2
Author Feedback
We thank the AC and reviewers for their time and effort in reviewing our paper. In general, the comments raised by the reviewers, as mentioned by the AC, are very positive and supportive, highlighting that “the motivation is persuasive”, “the proposed network is novel”, “the experiments are comprehensive and informative”, etc. Here, we attempt to address main concerns raised by the reviewers. How to extend the method to multi-coil reconstruction (R1). We think it can be realized in two ways. The most direct way is to compress the multi-coil data into a single complex image by using coil compression (Z. Tao et al., Magn Reson Med, 2012) and then applied DuDoCAF to this image. The other way is to reshape the dimension of multi-coil data from NCHW to (NC)H*W, where N represents the number of coils, and then fed them into DuDoCAF. The output of the network should be reshaped back correspondingly. We thank the reviewer for this valuable suggestion and consider it as one of our future works. On the fastMRI dataset (R1). The reason that we did not use multi-coil fastMRI k-space data is that paired multi-contrast MRIs are not available. Thus, we transformed the multi-contrast DICOM images into pseudo k-space images using FFT (Do et al. Med Phy, 2020; Kim et al. Med Phy, 2018). However, it is not hard to adapt our model to multi-coil k-space data when the data are available. On the TGR sub-module (R1). The aim of the proposed CAF module is to utilize the complementariness of the two contrasts by computing the cross attention between them via bidirectional guidance. In this sense, TGR can further drive the network to exploit multi-contrast correspondences and enhance the feature aggregation. Why choose CNN layers at the end of RRT and CAF (R1). These CNN layers are used to perform dimensional transformations of features. We did not employ transformers because (1) the CNN layers have already achieved satisfactory performance and (2) harnessing transformers here may increase training difficulties. However, we appreciate your valuable suggestions and shall try it in the future.
On the computational complexity (R2). The calculated FLOPs (G) and Parameters (M) of all mentioned models are listed below. Y-net: 104.683/12.545; UF-T2:76.028/1.893; MINet:856.192/6.566; DuDoRNet:42.602/325.314K; Ours:169.571/1.248. We further decreased the parameters of our model to 50.942/324.802K, and named it as ‘DuDoCAF-small’. The PSNR and SSIM values of the DuDoCAF-small are 26.82/0.84 (8X random) on the fastMRI dataset, indicating that it still outperforms other comparison methods. The performance consistency for different modalities (R2). In principle, our network can be used for reconstruction of any modalities as long as the reference modality is available. Here, we choose T1W and PDWI as reference modalities to reconstruct FLAIR and FS-PDWI as, in clinical practice, T1W and PDWI can be acquired with relatively shorter time.
Whether pathology could affect the model performance (R2). No. This is partly because the proposed DuDoCAF contains a data consistency (DC) layer, which is used to constrain the image generated by the network to be consistent with the real acquisition. Why cropping k-space (R3). The k-space feature maps are divided into a set of non-overlapping local windows. Each window, in principle, contains information of different frequencies. Thus, the proposed network can extract features of different frequencies via the self-attention mechanism. Furthermore, the patch unembedding operator will make sure the whole k-space data can be leveraged. On the results from each recurrent block (R3). The PSNR/SSIM value of each recurrent block is 38.70/0.96, 39.26/0.96, 39.41/0.96 under 8X random sampling. We will revise Fig. 3 with these results in the final version. On ablation study (R3). As shown in Table 2, (B) w/o CAF adds dual-domain (DD) learning to the Baseline network. (C) w/o DD adds the CAF block to the Baseline architecture.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Reviewer#2 would like to change the score from “weak reject” to “accept” during the communication with the Meta-reviewer. The authors have successfully cleared the previously raised major concerns. This paper presented a novel work of fast multi-contrast MR imaging via dual-domain fusion using recurrent Transformer. The proposed method shows promising performance and has been carefully validated.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Authors have provided a comprehensive response to address all points raised, including additional quantitative data to back up their claims. Novelty of approach in leveraging Transformers in a new application and reasonable data to back up utility suggests an excellent fit for the MICCAI audience.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Authors have addressed the major concerns.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3