List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Ling Huang, Thierry Denoeux, Pierre Vera, Su Ruan
Abstract
As information sources are usually imperfect, it is necessary to take into account their reliability in multi-source information fusion tasks. In this paper, we propose a new deep framework allowing us to merge multi-MR image segmentation results using the formalism of Dempster-Shafer theory while taking into account the reliability of different modalities relative to different classes. The framework is composed of an encoder-decoder feature extraction module, an evidential segmentation module that computes a belief function at each voxel for each modality, and a multi-modality evidence fusion module, which assigns a vector of discount rates to each modality evidence and combines the discounted evidence using Dempster’s rule. The whole framework is trained by minimizing a new loss function based on a discounted Dice index to increase segmentation accuracy and reliability. The method was evaluated on the BraTs 2021 database of 1251 patients with brain tumors. Quantitative and qualitative results show that our method outperforms the state of the art, and implements an effective new idea for merging multi-information within deep neural networks.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16443-9_39
SharedIt: https://rdcu.be/cVRyT
Link to the code repository
https://github.com/iWeisskohl/Evidence-fusion-with-contextual-discounting
Link to the dataset(s)
http://braintumorsegmentation.org/
Reviews
Review #2
- Please describe the contribution of the paper
The paper proposes a method to process multi-MRI image separately and merge the segmentation results using the formalism of Dempster-Shafer theory. The merging part is learnable.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper estimates the importance of each input (different MRI modality), without treating them equal (common practice), which is to help with the final accuracy.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
the method lacks substantial algorithmic novelty, and has weak evaluation
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
the paper is well written, and should be reproducable
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1) conceptually the idea is simple - train 4 independent encode-decoder networks (with slight modifications) , and learn the fusion (merging) of segmentations at the end. The paper frames it from the Dempster-Shafer theory angle, but essentially presents a simple trainable merging approach. 2) Evaluation/comparison is lacking modern sota approaches (including nnUnet). Instead the method is compared to many U-net like methods of 2018 or older, which is not necessary.
3) Since the brats2021 dataset is chosen for evaluation - why not report the results on brats2021 official evaluation hidden test set (their server accepts submissions) - Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
the paper explores the segmentation and fusion of multi model MRI from a Dempster-Shafer theory, but essential trains 4 encoder-decoder networks with a fusion module, which lack a substantial novelty. the evaluation lacks sota method comparisons.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
Authors propose a DST and deep learning-based multi modal evidence fusion framework with contextual discounting. the method was evaluated on the BraTs 2021 dataset. In particular, they claim their method is able to take into account the uncertainty of the different sources of information compared to probabilistic approaches.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The authors proposes a multi-modal deep learning and DST based framework or brain tumor segmentation.
- They investigated the contribution of each module component in their framework.
- Quantitative and qualitative evaluation was provided.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The limitation of the study is not discussed.
- Discussion on qualitative results on the scans with less dice score will be helpful.
- The results shown in Table 1 does not show significant improvement.
- Memory footprint is not discussed, the complexity of the network is increased in this method while not giving significant improvement (around 1%).
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors agree to provide the code, so the results can be reproduced. they demonstrate the performance of their method on widely-used publicly available dataset.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- It will be helpful if the authors can discuss the limitation of the proposed method.
- Some examples where the dice score is less can be visualized with possible explanation.
- In results section, few sentences can be added to explain why the proposed method is useful if it is not showing significant improvement in dice score.
- Few lines on how it can be used together with any state-of-art method? Minor comments: Typo in the first line of Page 5.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The paper has good potential.
- It is well written.
- Thorough experiments and ablation study are done.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #4
- Please describe the contribution of the paper
The authors propose a novel evidence fusion framework with contextual discounting for multi-modality brain tumor medical image segmentation. For the first time, the authors proposed an evidence discounting mechanism. The experiment demonstrates that their method outperforms the best previously published results for this task.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper designs an evidential segmentation module based on Dempster-Shafer theory and an evidence discounting mechanism take into account the ability of each modality.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The descriptions about evidence discounting mechanism is very vague. The writing is confused for readers. A large number of errors exist.
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The description of the proposed method is not clear and the paper may be not reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- The number of citation is disorder, such as number of the first citation is [23].
- English spelling problems :“the to uncertainty”should be replaced by“to the uncertainty”.
- “can be extend”should be replaced by “can be extended”
- The two important compared articles (“Peiris, H., Hayat, M., Chen, Z., Egan, G., Harandi, M.: A volumetric transformer for accurate 3d tumor segmentation. arXiv preprint arXiv:2111.13300 (2021)” and “Zhou, H.Y., Guo, J., Zhang, Y., Yu, L., Wang, L., Yu, Y.: nnformer: Interleaved transformer for volumetric segmentation. arXiv preprint arXiv:2109.03201 (2021)”) are all published in arxiv. More comparison with published papers with peer-review should be given.
- “2.1 Evidential segmentation module” is same with subtitle “Evidential segmentation module”.
- Dempster-Shafer theory be combined into segmentation module. However, the Dempster-Shafer equation is not differentiable for optimization, how to deal with the model optimization?
- equation 9a uses different fonts for Typesetting.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The proposed method is novel, but the description and writing is relatively poor.
- Number of papers in your stack
2
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The authors propose a new deep framework to merge multi-MRI image segmentation results using the formalism of Dempster-Shafer theory while taking into account the reliability of different modalities relative to different classes. While the reviewers agreed that there is merit in the proposed approach, concerns were raised with regards to experimental design as well as poor writing. The major concern is with regard to comparison with state-of-the-art which happens to be nnUnet in this space. Some concerns were also raised regarding algorithmic novelty.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
8
Author Feedback
We thank the AC and the three reviewers for their insightful comments. The two major concerns are about novelty, which was not well understood by some of reviewers, and about the comparison with the SOTA. We first answer the two concerns (A1 and A2) A1: To our knowledge, our work is the first attempt to apply evidence theory with contextual discounting to the fusion of deep neural networks. The idea of taking multimodality images as independent inputs and quantifying their reliability is simple and reasonable. However, modeling the reliability of sources is an important issue. Our model computes mass functions assigning degrees of belief m({ωk}) to each class and an ignorance degree m(Ω) to the set of possibilities. It thus has one more degree of freedom than a probability model, which allows us to model source reliablity. A2: One of the main contribution of our paper is the ability to estimate the reliably of each modality when segmenting different tumors, e.g., the evidence from T1 is more reliable for NRC and ET, which is consistent with domain knowledge. This approach can overcome the “black box” limitation and produce interpretable results. We already got some improvement in accuracy as compared to SOTA, and we will refine our proposal to further improve our results. We also thank the reviewers for pointing out the typos and mistakes. We will carefully revise the paper in the future version. According to the R2’s comment on the comparison with nnUNet”, we tested the performance of nnUNet and MMEF-nnUNet. We obtained a dice score of 89.68 for nnUNet and 90.05 for MMEF-nnUNet. When calculating the Expected Calibration Error (criteria to test the model’s reliability), we obtained 4.46% 4.05%, and 2.04% for nnUNet, MMEF-nnUNet and MMEF-UNet respectively. Our results are, thus, more reliable. Dear R2, the answers to your concern about novelty and SOTA are given in A1 and A2. We will explain to you some other questions. Q1: The idea is simple that trains 4 independent networks and learns the fusion The idea is, indeed simple, but the method to fuse unreliable evidence is new. The main contributions of our paper are the mass functions with uncertainty estimation, the contextual discounting, and the fusion of discounted plausibility. This is the first work that combines these three ideas with DNN. Q2: Didn’t report the results on brats2021 official evaluation test set The organizers closed the online evaluation after the competition and reopened it on 7 April. We could not get the test set and evaluate the validation set when writing the paper. Dear R3, thanks for your positive comments on the paper’s potential and experiments. It encourages our future exploration of DST in the medical domain! We will address your comments “further discussion on advantages and limitations, the memory cost” in the final version. Dear R4, thanks for confirming the novelty and strengths of our paper. We will answer some of your questions. Q1: The description of the evidence discounting mechanism is very vague Explaining the contextual discounting in detail is very challenging in such a short paper because it requires a lot of background knowledge on belief function. This why we refer to the paper by Mercier et al. [13]. The main idea is the definition of a degree of belief β_k which represents the reliability of a source, given that the true class is class k. The Dempster-Shafer calculus allows us to combine this “meta-knowledge” with the mass function provided by the source, to compute an updated, weaker mass function. Q2: The Dempster-Shafer equation is not differentiable and the paper may be not reproducible We are afraid do not get the point of the reviewer here. Dempster’s rule involves only the basic arithmetic operations, so there is no differentiability issue. The gradient calculation is detailed in [5]. The dataset used in the paper is public and we will release our code online, so our results will be reproducible.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have adequately addressed some of the concerns raised by the reviewers regarding novelty.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
1
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors present an interesting idea to develop an evidence based method to fuse modality information. They adequately tackle the comments regarding validation and novelty raised by the reviewers. I believe this paper can be of interest to the community and therefore lean towards acceptance if the authors include as stated some more discussion on the limitations of the method.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3