List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Ke Zou, Xuedong Yuan, Xiaojing Shen, Meng Wang, Huazhu Fu
Abstract
Despite recent improvements in the accuracy of brain tumor segmentation, the results still exhibit low levels of confidence and robustness. Uncertainty estimation is one effective way to change this situation, as it provides a measure of confidence in the segmentation results. In this paper, we propose a trusted brain tumor segmentation network which can generate robust segmentation results and reliable uncertainty estimations without excessive computational burden and modification of the backbone network. In our method, uncertainty is modeled explicitly using subjective logic theory, which treats the predictions of backbone neural network as subjective opinions by parameterizing the class probabilities of the segmentation as a Dirichlet distribution. Meanwhile, the trusted segmentation framework learns the function that gathers reliable evidence from the feature leading to the final segmentation results. Overall, our unified trusted segmentation framework endows the model with reliability and robustness to out-of-distribution samples. To evaluate the effectiveness of our model in robustness and reliability, qualitative and quantitative experiments are conducted on the BraTS 2019 dataset.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16452-1_48
SharedIt: https://rdcu.be/cVVp3
Link to the code repository
https://github.com/Cocofeat/TBraTS
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper brings a new method to be able to provide uncertainty in addition to the segmentation. They use Dirichlet distribution in a very nice way for this aim. Furthermore, the robustness of the network is improved.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
• This paper provides a smart way to be able to predict uncertainty related to the resulting segmentation • comparisons with existing methods are detailed
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
I did not detect any real weakness in this pape except the lack of references in matter of U-Net variants.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
This paper seems to be reproducible on any segmentation architecture.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The only remarks that I have is that some references relative to U-Net variants are missing, and some sentences are not well constructed; please reformulate them.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I really appreciated the way uncertainty is computed, I think this approach promising.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
This paper proposed an end-to-end trusted model for brain tumor segmentation by quantifying the voxel-wise uncertainty and introduced the confidence level for the image segmentation in disease diagnosis. A series experiments were conducted and the results verity the reliability of the model.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The idea of using Dempster-Shafer theory for uncertainty quantification is attracting and the proposed methods show good performance on model’s reliability with Entropy, ECM and UEO.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The experiment section is not well organized and the results are not well explained. For example, Fig2 a) is complex and hard to understand. In paragraph”Differences from similar methods.”, the authors should compare with other similar Dempster-Shafer-based methods. See detailed comments below.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper is reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
-
Besides subjective logic theory, many methods have been proposed to assign belief of mass, for example, Shafer’s model [1], Evidential KNN [2], and Evidential neural classifier [3]. Why did the authors choose subjective logic theory instead of other methods to assign belief of mass? The discussion and comparison between different belief assignment methods should be interesting and meaningful. [1]. Shafer, Glenn. A mathematical theory of evidence. Princeton university press, 1976. [2].Denoeux, Thierry. “A k-nearest neighbor classification rule based on Dempster-Shafer theory.” Classic works of the Dempster-Shafer theory of belief functions. Springer, Berlin, Heidelberg, 2008. 737-760. [3]. Denoeux, Thierry. “A neural network classifier based on Dempster-Shafer theory.” IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans 30.2 (2000): 131-150.
-
why did the authors only choose two modality data as the experiment dataset? And the Gaussian noise was added to which modality? Why only add noise on one modality but not modality together?
-
Fig 2 a) is hard to follow. The performance of Dice is not as good as others. The authors should give some explanation for it.
-
In Fig 2 a), the most stable method is UE. The authors should give some explanation for it.
-
The results of Fig2 are obtained during training. Where are the results of the test set?
-
Fig 4 only visualize the uncertainty comparison with the slice containing enhanced tumor (class 4) results. The brain2019 dataset is a three-class segmentation task. The authors should offer an example slice with three class information included to show the model’s reliability better.
-
There are some methods that use the Dempster-Shafer theory for uncertainty quantification in medical image segmentation tasks. Some discussion beyond those methods is necessary, i.e., [1]Huang, Ling, et al. “Evidential segmentation of 3D PET/CT images.” International Conference on Belief Functions. Springer, Cham, 2021. [2]Huang, Ling, Su Ruan, and Thierry Denoeux. “Belief function-based semi-supervised learning for brain tumor segmentation.” 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE, 2021.
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper proposed a novel direction for uncertainty quantification by assigning the belief of mass with the Dempster-Shafer theory. The proposal was evaluated with four metrics, Dice score, Entropy, ECE and UEO. The authors also compare their proposal with the popular uncertainty quantification method, i.e.,MC dropout, model ensemble.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
This paper proposes a trusted brain tumor segmentation relied on the evidential deep learning method. The proposed method estimates uncertainty without excessive computational burden and modification of the backbone networks. Experiments show improved performance.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
The application of evidential deep learning on medical image segmentation is an interesting use case and can be helpful for the community.
-
The experimental results demonstrate the effectiveness of the proposed method over other methods.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Lack of novelty. The proposed method is a direct application of the existing evidential deep learning method [22] to the brain tumor segmentation task. Although [22] is 2D image classification-oriented, and this paper focuses on 3D medical image segmentation, the overall difference is small. Generalizing techniques from image-level classification to pixel-level classification (segmentation) is straightforward. The authors are suggested to elaborate more on their contributions.
Is there any other paper that also relies on evidential deep learning [22] in medical images? If yes, the authors should discuss them as well.
- The writing should be improved. The authors are suggested to proofread carefully and enhance the writing.
Minor issues: Scale Fig. 2 to a proper range. It is hard for readers who prefer to print the paper and read.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
No special issue here.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Considering elaborating why the proposed/used uncertainty estimation method is more effective, in medical image segmentation (e.g., brain tumor), than the existing ones. In your experience, is there a conscious difference between different estimation methods? Such kinds of discussions can provide insights to the community and increase the contributions and impact of this paper.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
See the weakness section.
- Number of papers in your stack
7
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Not Confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
Based on the authors’ response, I would like to raise my score.
Review #4
- Please describe the contribution of the paper
In this paper, the authors presented an end-to-end trusted segmentation model, TBraTS, for reliably and robustly segmenting brain tumor with uncertainty estimation. The model learns predicted behavior from the perspective of evidence inference, through the connection between uncertainty quantification and belief mass of the subjective logic. The model is evaluated using BraTS 2019 dataset.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1 Uncertainty in segmentation is an important research area 2 The paper is easy to read 3 This study uses public dataset
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1 The methodological novelty of this paper is very limited. It seems to use the method proposed in [6] and change the problem setting from multi-view classification to segmentation. 2 Limited quantitative evaluation on why the proposed method is better than MC dropout or other uncertainty estimation.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Code is not publicly available
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The BraTS dataset provides 4 modalities. Why did the authors only tested two modalities? Why is the proposed method better than MC dropout in terms of estimated uncertainty? Further analysis is encouraged.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
2
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This work is very similar to [6], it just changes the problem setting from multi-view classification to segmentation
- Number of papers in your stack
8
- What is the ranking of this paper in your review stack?
7
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
2
- [Post rebuttal] Please justify your decision
(1) Limited methodological novelty: As the authors acknowledeged, this work is an application of [6] [22]. It’s easy to change the problem setting from [6] [22] to image segmentation. Thus the methodological novelty is very limited.
(2) Low clinical value: This work claims to be able to calculate a different kind of uncertainty (than MC dropout/ensemble). In order to prove its clinical usefulness, the authors should show that the proposed uncertainty is positively correlated to the segmentation error (or at least better than MC dropout/ensemble). Otherwise, clinicians can’t use it to infer the quality of the segmentation. Because the authors didn’t include or discuss this important aspect, I still think the clinical impact of this work is very limited.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposes a trusted brain tumor segmentation using the evidential deep learning method. Even though the reviews are mixed, the idea of using Dempster-Shafer theory is interesting and the proposed method shows good performance. Some concerns about organization of the paper have been raised. Relationship to other works, especially [6] as mentioned by reviewer 4 should be clarified.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5
Author Feedback
We thank you for taking time to review our paper and truly appreciate reviewers for their positive supports to our technical novelty（e.g., “They … in a very nice way…” by R1, “The application of … is an interesting use case and can be helpful for the community.” by R2）and effectiveness of method (e.g.,” the robustness of the network is improved” by R1, “the proposed methods show good performance on model’s reliability…” by R3).
Compared with [6] and [22]: 1) [22] and [6] are proposed to handle the problems of single-/ multi-view classification, while we focus on 3D medical image segmentation and provide uncertainty estimations for voxels. 2) Our outputs are 3D uncertainty quantifications and segmentation results for brain tumors, which is essential for interpretability in disease diagnosis. 3) We develop a general learning framework with a flexible network design for brain tumor segmentation, which has a potential contribution to be beneficial to all researchers in the trustworthy medical domain.
We list additional results of 4 modalities. We choose two competitive baselines and report the Dice and UEO scores, where our method consistently outperforms other baselines under the noised condition even though testing on four modalities. Segmentation (Dice) Methods 0/1.5 PU/83.28/54.34 UDO/84.21/61.09 AU+Our/82.09/ 73.96
Uncertainty (UEO) Methods 0/1.5 PU/0.83/0.58 UDO/0.80/0.61 AU+Our/0.85/0.82
QThe Gaussian noise was added to which modality? Why not add noise to modalities together? (R3) R: We list additional results of 2 modalities, where our method also exhibits robust performance than others. Constrained by space, we only report the results of adding vary noise to T2 modality.
Segmentation (Dice) Methods /0.5/1.0/1.5 PU/64.56/45.17/23.41 UDO/70.19/51.35/44.31 V/ 69.05/52.08/43.46 AU/73.63/57.40/45.83 V+Our/75.95/59.73/ 41.11 AU+Our/74.15/61.50/51.98
Uncertainty (UEO) Methods 0.5/1.0/1.5 PU/0.66/0.61/0.33 UDO/0.73/0.52/0.44 V+Our/0.79/0.66/0.46 AU+Our/0.80/0.72/0.66
2) UE’s performance: From the Fig. 2 (a), the performance of UE under the normal or low-level condition is not good. The test running time of UE for one sample is 3.258 mins, while AU with our method is 0.015 mins. From the Fig. 2 (b)-(d), our method gets better performance than UE on uncertainty estimations.
2) Compared with the evidential segmentation methods listed by R3, we use the SL theory to explicitly model uncertainty with Eq. 2. Dirichlet distribution parametrizes the density of each such probability assignment on a simplex. SL with the Dirichlet distribution explicitly models the second-order probability and uncertainty of the output.
2) Since MC dropout indirectly infers uncertainty of the segmentation during the inference stage, it is difficult to seamlessly train a model with high accuracy, robustness and reasonable uncertainty in a unified framework.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
I think the authors have made a good argument about the concerns of reviewer 4. Other reviewers have recommended acceptance. In my opinion, despite one reviewer showing strong reject, the paper has merit. I think there is a sufficient novelty, relative to other papers I have read and reviewed.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
4
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper proposes using evidential deep learning to provide trusted brain tumor segmentation. As several reviewers point out, the novelty of the paper is limited, and the clinical utility of the method seems limited.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
9
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors introduce a novel model for segmentation of brain tumor and provides a method to estimate reliability of the segmentation method. The approach is new. The authors answered most of the reviewers concerns. I agree with the authors that this is an innovative intriguing approach that fits MICCAI conference.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
9