List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Mohammad Mahdi Kazemi Esfeh, Zahra Gholami, Christina Luong, Teresa Tsang, Purang Abolmaesumi
Abstract
Left Ventricular Ejection Fraction (LVEF) as a critical clinical index is widely used to measure the functionality of the cyclic contraction of the left ventricle of the heart. Limited amount of available specialist-annotated data, low and variable quality of captured ultrasound images, and substantial inter/intra-observer variability in gold-standard measurements impose challenges on the robust data-driven automated estimation of LVEF in echocardiography (echo). Deep learning algorithms have recently shown state-of-the-art performance in cardiovascular image analysis. However, these algorithms are usually over-confident in their outputs even if they provide any measure of their output uncertainty. In addition, most of the uncertainty estimation methods in deep learning literature are either exclusively designed for classification tasks or are too memory/time expensive to be deployed on mobile devices or in clinical workflows that demand real-time memory-efficient estimations. In this work, we propose Delta Ensemble Uncertainty Estimation, a novel sampling-free method for estimating the epistemic uncertainty of deep learning algorithms for regression tasks. Our approach provides high-quality, architecture-agnostic and memory/time-efficient estimation of epistemic uncertainty with a single feed-forward pass through the network. We validate our proposed method on the task of LVEF estimation on EchoNet-Dynamic, a publicly available echo dataset, by performing a thorough comparison with multiple baseline methods.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16452-1_50
SharedIt: https://rdcu.be/cVVp5
Link to the code repository
N/A
Link to the dataset(s)
https://echonet.github.io/dynamic/index.html#dataset
Reviews
Review #1
- Please describe the contribution of the paper
This work makes several interesting contributions in form of an epistemic uncertainty estimation method for deep learning regression tasks applied to cardiac echo, including agnostic to neural network architectures, real-time and deterministic; memory-efficient, scalable and computationally fast for large models, and provides high-quality uncertainty estimation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- well written and motivated
- novel use (as far as I can tell) of Delta Method with model ensembles for uncertainty estimation
- use of public dataset EchoNet-Dynamic
- comparison of ResNet-18 and its use for Ensemble of Methods (EM) and Deep Bayesian Neural Networks, and a single deterministic deep regression model (DDR)
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- no novel DL model per se, but method to be used on top of trained models
- mainly a benchmarking paper where it is not clear what the alternative uncertainty estimation adds and how would be helpful for in diagnostics, ie a further downstream task is missing
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Everything is ticked “yes” but I don’t see own code or results being made available?
Otherwise:
- Detailed model/parameter description
- use of state-of-art methods for comparison
- public dataset
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
This is interesting work which will add to existing uncertainty estimation methods, while providing a fast, scalable, single forward-pass solution, so should be of some interest. The application of regressing ejection fraction for cardiac echo is an important one, and the method is not limited to this - moreover it can run on pretrained models. However the conclusions are nor clear to me as the compared methods all seem to be in the same ballpark (Figure 3 - predicted RMS vs % of eliminated data, all showing similar quality of levels of uncertainty (apart from DDR) and a similar test error and need for calibration, so it is not clear what extra value DEUE is then bringing in in clinical practice.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Interesting method for uncertainty estimation in trained-up models, that could be of interest to wider applications.
- Number of papers in your stack
6
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The authors present a method for estimating the uncertainty in a Deep Learning estimate of a real valued prediction. The method computes the empirical variance of the parameters over 5 trained versions of the same network model. The authors explain how such an estimate can be used as an approximation to the network covariance which would be used as part of a Taylor expansion of the expected squared predictive error. The approach is applied to the estimation of ejection fraction from apical 4-chamber ultrasound views from a publicly available database.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The benefits of this approach over prior work are relative computation tractability especially at inference time since only one pass of inference is required and there is no special architecture required of the Deep Learning network.
The exposition is very clear. The approach is one others could easily adopt and evaluate.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
My main criticism is that it is unrealistic to expect the computation of variance over 5 samples to be a good estimate of the actual variance of the network parameters even when the expected variance of the parameters is low (reference [4] in the paper). Nevertheless, the authors are able to show that the estimates are in line with other approaches to estimating the variance. These other methods also have issues with sampling but it is encouraging that the results across methods seem consistent. Also, the few examples of data producing high and low uncertainty estimates are very reasonable. But I would like to see more trained networks be used for sampling the parameter variance despite the computational challenge.
The authors assume that the network model weights are independent. But I would be curious to see how large some of the non-diagonal terms of the covariance matrix are. Is the independence assumption reasonable for their network? This is a minor issue for the publications of this conference paper but this is something I would expect to see in a journal paper.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The reproducibility of this paper is very high. The data is publicly available and the implementation is straighforward.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The references are good. There is so much work in uncertainty estimation that not all references could possibly be given but the paper has a reasonable selection.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although a great deal of work has appeared in the last couple years on uncertainty, this is a new approach that other researches could easily evaluate on useful problems that require a fast run-time and limited computational resources during inference.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
In this paper, the author mainly proposes an epistemic uncertainty estimation method for Deep regression networks in the context of EF learning. Compared to other methods, this method only requires one forward-pass at the inference time.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The main strength of this paper is that it proposes a memory/time-efficient estimation of epistemic uncertainty with a single feed-forward pass through the network. The method was based on the Taylor’s expansion of the expected squared error. The model parameter and the covariance matrix were determined by multiple feed forward propagation. Then during test time, on one forward computation is required to obtain the uncertainty.
The method can identify samples with poor quality. - Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The definition of epistemic uncertainty in Eq. 2 and 7 needs further clarify. Experiments of this paper is inadequate. See detailed comments below.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
relatively easy to reproduce. Not all details provided.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The definition of epistemic uncertainty in Eq. 2 and 7.
- From the definition in the paper, the epistemic uncertainty in equation 2 is the expected squared error. However, the prediction error and the epistemic uncertainty can not be viewed equivalent to each other. A prediction can be of high error and low uncertainty in the meantime, or low error with high uncertainty. Please clarify.
- Ignorance of the prediction error \epsilon_x in Equation 7. While W* is the optimal model parameter, when the network converges, W will be very close to W, which makes the (W-W) equal to 0. Then U_L(x) will be equal to prediction error \epsilon_x
- Is there any difference between U_L(x) and E(\epsilon_x^2).
- During training stage, the network is initialized from different points, thus leading to multiple local optimal estimation of W*. How to obtain these different initializations? I am wondering the results of Eq. 7 maybe very high if they are extremely far from each other. Besides, how about computing the diagonal entries of the covariance matrix \Sigma directly from these initializations? Will this be similar to or quite different from the diagonal entries of \Sigma_M that computed from the converged Ws?
Experiments of this paper is inadequate.
- From figure 3, the proposed method shows inferior performance compared to its competitors.
- And there are no quantitative results to demonstration the advantage of the proposed method.
- Table 3 is not convincing without any quantitative results of the memory and running time. It will be helpful to test the real memory cost and time cost during inference.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The task is interesting and the method of uncertainty estimation is efficient in time and memory.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper provides a method for estimating epistemic uncertainty for deep learning regression tasks. The reviewers appreciate the paper as well written and covering uncertainty for a new type of application, demonstrated with excellent reproducibility.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
1
Author Feedback
N/A