List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Ziqi Huang, Li Lin, Pujin Cheng, Kai Pan, Xiaoying Tang
Abstract
Contrast-enhanced T1 (T1ce) is one of the most essential magnetic resonance imaging (MRI) modalities for diagnosing and analyzing brain tumors, especially gliomas. In clinical practice, common MRI modalities such as T1, T2, and fluid attenuation inversion recovery are relatively easy to access while T1ce is more challenging considering the additional cost and potential risk of allergies to the contrast agent. Therefore, it is of great clinical necessity to develop a method to synthesize T1ce from other common modalities. Current paired image translation methods typically have the issue of requiring a large amount of paired data and do not focus on specific regions of interest, e.g., the tumor region, in the synthesization process. To address these issues, we propose a Difficulty-perceived common-to-T1ce Semi-Supervised multimodal MRI Synthesis network (DS3-Net), involving both paired and unpaired data together with dual-level knowledge distillation. DS3-Net predicts a difficulty map to progressively promote the synthesis task. Specifically, a pixelwise constraint and a patchwise contrastive constraint are guided by the predicted difficulty map. Through extensive experiments on the publiclyavailable BraTS2020 dataset, DS3-Net outperforms its supervised counterpart in each respect. Furthermore, with only 5% paired data, the proposed DS3-Net achieves competitive performance with state-of-theart image translation methods utilizing 100% paired data, delivering an average SSIM of 0.8947 and an average PSNR of 23.60.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16446-0_54
SharedIt: https://rdcu.be/cVRTO
Link to the code repository
https://github.com/Huangziqi777/DS-3_Net
Link to the dataset(s)
https://www.med.upenn.edu/cbica/brats2020/
Reviews
Review #1
- Please describe the contribution of the paper
This paper presents a Difficulty-perceived common-to-T1ce Semi-Supervised multimodal MRI Synthesis network (DS3-Net), which promotes the synthesis task by predicting a difficulty map. The predicted difficulty map can guide a pixelwise constraint and a patchwise contrastive constrain. Experiments on BraTS2020 dataset have been performed to investigate the effectiveness of DS3-Net.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
There are two major strengths in this work:
- Multi-modal MRI synthesis is able to make full use of multimodal information for more accurate MRI synthesis, which therefore improves the performance over the traditional single-modal MRI synthesis.
- The paper is well organized and easy to follow.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The technical contribution of this work is not very clear. Some key parameter settings seem adhoc. The experimental results are not well explained.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
-
The authors should provide more detailed explanations on the differences between their work and the existing semi-supervised learning frameworks, especially those based on the teacher-student strategy. Currently, it is hard to understand the true technical contribution of this work. The proposed framework seems a straightforward application of existing semi-supervised learning work in multi-modal MRI synthesis.
-
Page 5: “Here we empirically set λ id , λ fd , λ pad and λ GAN as 100, 1, 1 and 1.” How to determine these parameters? What is the influence once any of them is changed?
-
Table 1: The results for the case of 100% paired percentage look odd. The second best method, Pix2pix, provides the best image quality with highest PSNR and SSIM, however, the corresponding segmentation results are not as good as DS3-Net. Why are two sets of results inconsistent? What is the underlying reason?
-
Following the last comment, another strange thing is that the case of 50% paired percentage outperforms 100% case in terms of TC Dice, which is inconsistent with the conclusions observed in other results, i.e., larger paired percentage better TC Dice score. Please clarify.
-
It is helpful to provide the model size (e.g., number of parameters) of different networks.
-
Some failure cases should be provided in the manuscript. The authors should discuss the potential reasons for the failure of their model, which is helpful for the further improvement of the proposed model.
-
Finally, the authors can consider to provide some results regarding the computational time of different methods.
-
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Good.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper focuses on the multi-modal MRI synthesis task, which is interesting and has a number of practical applications on both research and clinical sides. The proposed method has some novelty and provides satisfactory performance. However, the experiments should be strengthened and more efforts should be made to clarify the contributions of this work. Considering these factors, I would like to give this paper this credit.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Somewhat Confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
The authors have addressed most of my concerns in the rebuttal. I would like to raise my rating.
Review #3
- Please describe the contribution of the paper
1) To the best of our knowledge, this is the first semi-supervised framework applied to multimodal MRI synthesis for gliomas. 2) In light of the teacher-student network, we make full use of unpaired multimodal MRI data through maintaining consistency in spaces of both high and low dimensions. 3)They innovatively estimate a difficulty-perceived map and adopt it to dynamically weigh both pixelwise and patchwise constraints according to the difficulty of model learning, and thus the model can assign the difficult-to-learn parts (such as the glioma region) more attention. 4) Extensive comparison experiments are conducted, both quantitatively and qualitatively.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
They proposed a novel difficulty-perceived semi-supervised multimodal MRI synthesis pipeline to generate the difficult-to-obtain modality T1ce from three common easy-to-obtain MRI modalities including T1, T2, and FLAIR. Difficulty-perceived maps are adopted to guide the synthesization process of important regions, and dual-level distillation enables the model to train a well-performimg network with limited paired data.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The authors only tested the proposed method on the BraTS2020 dataset. More tests are required. Otherwise, the quantitive evaluations for the proposed method on Table 1 are not over all for other methods. The authors should apply the CUTGAN, pGAN, MedGAN and Pix2pix methods on the different Paired percentages. But, the proposed method are interesting for the clinical applications.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Positive for the reproducibility of the paper. It’s easy to be reproduced, because they have shared the source codes on GitHub. But, the authors should describe more details about the network.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
It’s easy to be reproduced and this is very nice work. But the modality T1ce is not popular in the general diagnose. More details of T1ce imaging should be introduced for authors. Otherwise, the authors only gave some values of lamda parameters in the loss function. More tests of different values of lamda parameters should be discussed for other readers and users.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
They proposed a Difficulty-perceived common-to-T1ce Semi-Supervised multimodal MRI Synthesis network (DS3-Net), involving both paired and unpaired data together with dual-level knowledge distillation to address issues related to acquire T1ce.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #5
- Please describe the contribution of the paper
This paper introduce an multimodal MRI synthesis network to generate T1ce based on given modalities. By introducing an attention map, the proposed method can flexibly focus on the important area in the generation, and through a distillation procedure, the method incorporate more unpaired data to help this synthesis task.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The strengths of the paper are summarized as follows: 1) The authors introduce a semi-supervised framework to make full use of unpaired multimodal MRI data for T1ce generation based on the easy-to-acquire modalities, which demonstrates efficiency to save the cost of the paired data significantly.
2) The authors designs a specific attentional mechanism to relax the generation process so that the model focused on the important area from both pixel-wise and patch-wise.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The main concern of this paper is their ablation studies is not enough, which is summarized as follows.
1) Regarding the attention map defined in Eq. (1), a small constant is set to 0.2 at x_i^T=0 lack of sufficient clear explanations e.g., some empirical validation results.
2) It seems that the authors do not show the performance of the plain DS^3-net by only preserving the GAN loss and the distillation loss in Eq.(4) and Eq.(7), i.e., LAGAN with distillation.
3) The hyperparameter is very intuitive in Eq.(4) and Eq. (7) where the authors do not shows how they motivated to be set.
4) It should be useful to show the gap of the model over previous baselines by also conducting their experiments with only 5% paired data in Table 1.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The training schedule including the learning rate and optimizers etc are not well clarified.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The authors should add more ablation study to well characterize the merits of the proposed method. Especially, the constant in Eq.(2) that is set to 0.2 is not well explained, since it actually affects the weighting in the remaining loss terms. Besides, the hyperparameters for most of loss terms are set by 100, 1, 1, which is also empirically due to the large value range.
Addressing these concerns about DS^3-Net could further improve the submission.
I would like to raise the score if these critical concerns could be well solved.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Overall the performance of DS^3-net with only 5% paired data and 95% unpaired data is appealing compared with the baselines with all paired data. The advices above could make this draft easy-to-follow and easy-to-generalize in other image synthesis tasks.
- Number of papers in your stack
1
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Somewhat Confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
I have read the response of the authors. Most of my concerns have been addressed. I would like to raise the score.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposes semi-supervised multimodal MRI synthesis method to generate the difficult-to-obtain modality T1ce from three easy-to-obtain MRI modalities including T1, T2, and FLAIR. The overall reviewers’ opinion about the proposed method is positive, including satisfactory experimental performance and some novelty of the proposed method. However, the reviewers unanimously suggested that the experiment part of the paper should be strengthened, such as more ablation studies and discussion on the experimental observations. and more efforts should be made to clarify the contributions of this work. Moreover, R1 also suggest to clarify the technical contribution of this work. The authors should carefully address these concerns in their rebuttal.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6
Author Feedback
1.Hyperparameters of the weights of different losses (R1 Q5.2, R3 Q6, R5 Q3.3) The hyperparameters were identified empirically. We test six different combinations of the five weights in Eq. (4) and Eq. (7). None of them are as good as our adopted combination, in terms of both SSIM and PSNR. For example, when changing λ_fd to 10, SSIM decreases from 0.8947 to 0.8842 and PSNR decreases from 23.6 to 22.71; when changing λ_id to 1, SSIM decreases to 0.8732 and PSNR decreases to 21.59. We will provide more details in our final version.
2.Main contributions (R1 Q5.1) As far as we know, this work is the first to synthesize the very important T1ce modality in the brain tumor analysis setting in a semi-supervised way (using both few paired data and more unpaired data during training). Existing works [Dalmaz et al., TMI2022, Huang et al., MIA2022] are fully-supervised; they consider paired and complete modalities during training but missing modalities only at inference. Moreover, we take into account the prior knowledge of challenging tumor regions and innovatively incorporate a difficulty-perceived map to make our model learn at both pixel and patch levels.
3.Confusing results in Table 1 (R1 Q5.3, R1 Q5.4) Firstly, SSIM/PSNR/MSE focus on global similarities between a synthesized image and a reference image, whereas Dice measures the overlapping degree of a local tumor region between the two images. These two types of metrics measure different “similarity” properties, and thus slightly inconsistent results are plausible. To test our conjecture, we conduct an extreme experiment. For all synthesized/reference image pairs, we mask out the tumor region in each synthesized image and add Gaussian noise to the masked region. We then calculate the two types of similarity metrics between each modified synthesized image and the corresponding reference image. Still, we obtain 0.8823 SSIM and 21.94 PSNR while the ET and TC Dice scores are almost 0. Secondly, we observe that semi-supervised learning tends to converge when the labeled data reach a certain proportion. Keeping adding more paired (labeled) data may lead to overfitting, and thus for some evaluation metrics the 50% paired data case may work slightly better than the 100% paired data case. Similar observations have also been reported elsewhere [Zhou et al., MICCAI2020].
4.More experimental results (R3 Q3, R5 Q3.1, R5 Q3.2, R5 Q3.4) We perform a variety of additional experiments: a) We evaluate all compared methods with all different percentages of paired data. Our proposed method still outperforms all those compared methods; compared with the second-best method, we obtain +0.0256 SSIM at 5% paired level, +0.0098 SSIM at 10% level, and +0.0010 SSIM at 50% level. b) We successfully establish the effectiveness of our proposed pipeline on the BraTS18 dataset and a private dataset. Due to the space limit, we will present those results in our journal extension. c) We test two other thresholds (0.1 and 0.3) for the background constant in Eq. (1). When the constant is 0.1, it tends to generate out-of-bounds pixels on the background and decrease the image quality (-0.0122 SSIM and -0.60 PSNR). When it is 0.3, it is likely for the difficulty map to lose the score discrepancy between the foreground and the background and induce decreased performance (-0.0137 SSIM and -0.67 PSNR). Overall, 0.2 works the best. d) We preserve only the GAN loss and the distillation loss, leading to -0.0089 SSIM and -0.19 PSNR, which clearly validates the necessity of L_pad.
- More details (R1 Q5.5, R1 Q5.6, R1 Q5.7, R3 Q6, R5 Q5) We will provide all requested details in our final version, including the parameters (33M), the inference time (0.1s/slice), detailed descriptions on T1ce. And we will supplement related comparisons in Table 1. Due to the time and space limit, we will provide failure cases and discuss the potential reasons in our journal extension.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The reviewers are generally positive about the paper, especially regarding the novelty and significance. With more experimental results provided and the contributions of this work better clarified, the concerns were addressed in the rebuttal to a reasonable level. So I would recommend acceptance of the paper.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Initially a borderline paper with somewhat mixed reviews. Most of the concerns were addressed by authors in the rebuttal. Overall interesting work with relevance to the MICCAI community, though demonstration of clinical relevance would strength the paper further. Based on the majority of the reviews I would consider acceptance.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
15
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The work proposes a semi-supervised approach for multi-modal MRI synthesis, which tackles an interesting problems and shows promising results. Previous concerns were mainly on its lack of ablation studies and discussion on the experimental observations. These have been well addressed in the rebuttal, and two reviewers have increased their scores based on the authors’ rebuttal. Given these have been addressed, the work is recommended for acceptance.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5