List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Amoon Jamzad, Fahimeh Fooladgar, Laura Connolly, Dilakshan Srikanthan, Ayesha Syeda, Martin Kaufmann, Kevin Y. M. Ren, Shaila Merchant, Jay Engel, Sonal Varma, Gabor Fichtinger, John F. Rudan, Parvin Mousavi
Abstract
PURPOSE: The use of intra-operative mass spectrometry along with Graph Transformer models showed promising results for margin detection on ex-vivo data. Although highly interpretable, these methods lack the ability to handle the uncertainty associated with intra-operative decision making. In this paper for the first time, we propose Evidential Graph Transformer network, a combination of attention mapping and uncertainty estimation to increase the performance and interpretability of surgical margin assessment. METHODS: The Evidential Graph Transformer was formulated to output the uncertainty estimation along with intermediate attentions. The performance of the model was compared with different baselines in an ex-vivo cross-validation scheme, with extensive ablation study. The association of the model with clinical features were explored. The model was further validated for a prospective ex-vivo data, as well as a breast conserving surgery intra-operative data. RESULTS: The purposed model outperformed all baselines, statistically significantly, with average balanced accuracy of 91.6\%. When applied to intra-operative data, the purposed model improved the false positive rate of the baselines. The estimated attention distribution for status of different hormone receptors agreed with reported metabolic findings in the literature. CONCLUSION: Deployment of ex-vivo models is challenging due to the tissue heterogeneity of intra-operative data. The proposed Evidential Graph Transformer is a powerful tool that while providing the attention distribution of biochemical subbands, improve the surgical deployment power by providing decision confidence.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43990-2_53
SharedIt: https://rdcu.be/dnwL8
Link to the code repository
https://github.com/med-i-lab/evidential_graph_transformers/
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
An interesting design that makes use of graph model of Mass spectrum for cancer detection.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
A novel and interesting design that makes use of graph model of Mass spectrum for cancer detection. The results indicated improved performance. Clinically useful.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The generation and details of the experimental data is not clear. So it is hard to judge the clinical significance.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
It is reproducible. Experimental design and detailed introduction of the datasets for validation is needed.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
A novel and interesting design that makes use of graph model of Mass spectrum for cancer detection. The results indicated improved performance. It is clinically useful. However, experimental design and detailed introduction of the datasets for validation is needed, as illustrated in the Fig. 1 (without details). So it is hard to judge the clinical significance.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Interesting idea, and clinically useful (Potentially). However, the experiments and details of the validation dataset are not clear, it is hard to judge the clinical significance.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The primary contribution of this paper is that it proposes an Evidential Graph Transformer network (EGTNet) with attention and uncertainty estimation to increase the performance and interpretability of surgical margin assessment.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Strength:
- The evidential graph transformer network seems novel. Graph transformer is not new, but evidential graph transformer is a new scheme on graph transformer.
- It uses evidential graph transformer to increase the performance and nterpretability of surgical margin assessment. It is a new method.
- The discussion, statistc analysis and visualization are sufficient.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Weakness:
- Proposed evidential graph transformer model, and the surgical margin assessment method by evidential graph transformer are not sufficiently presented.
- The methods compared with proposed one seem too less, only GT, GCN and a non-graph CNN.
- The introduction to related works are also too less, only 15 references.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The reproducibility of this paper seems moderate.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The primary contribution of this paper is that it proposes an evidential graph Transformer network (EGTNet) with attention and uncertainty estimation to increase the performance and interpretability of surgical margin assessment. The scheme based on evidential graph Transformer seems novel. However, some obvious weaknesses exist in this paper:
- Proposed evidential graph transformer model, and the surgical margin assessment method by evidential graph transformer are not sufficiently presented. Current presentation seems coarse. More details should be given out. Especially, the computation & reasoning mechanisms of evidential graph transformer are two primary light points. However, they are clearly presented.
- The methods compared with proposed one seem too less, only GT, GCN and a non-graph CNN. More state-of-the-art ones should be considered.
- The introduction to related works are also too less, only 15 references. More introduction to related methods should be made.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- Proposed evidential graph transformer model, and the surgical margin assessment method by evidential graph transformer are not sufficiently presented. Current presentation seems a little coarse. More details should be given out. Especially, the computation & reasoning mechanisms of evidential graph transformer are two primary light points. However, they are clearly presented.
- The methods compared with proposed one seem too less, only GT, GCN and a non-graph CNN. More state-of-the-art ones should be considered.
- The introduction to related works are also too less, only 15 references. More introduction to related methods should be made.
- Reviewer confidence
Somewhat confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The contribution of this work is the development and validation of the Evidential Graph Transformer model, which outputs uncertainty estimation and intermediate attentions. The model’s performance was compared with different baselines in an ex-vivo cross-validation scheme, and its association with clinical features was explored. The model was further validated using prospective ex-vivo data and breast conserving surgery intra-operative data. Extensive ablation study was conducted to assess the performance of the model.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors proposed EGT to increase the performance and interpretability of surgical margin assessment. The paper is well written and the study is extensive.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The evaluation matrices are not comprehensive. Lack of baseline comparison.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Code was not provided. Unclear for reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- Please enlarge the font in Figures to improve the readability.
- Please use more comprehensive evaluation matrices.
- Please compare with more baseline models.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The novelty of the paper, extensive study and quality of writing.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper presents a noval contribution through the introduction of an evidential graph Transformer network (EGTNet) that incorporates attention and uncertainty estimation. This approach aims to enhance the performance and interpretability of surgical margin assessment. The utilization of an evidential graph Transformer scheme adds novelty to the proposed method. Nonetheless, it is important to acknowledge the presence of certain apparent weaknesses within this paper. The experiments are however limited.
Author Feedback
We thank the reviewers for their invaluable insights and comments. All reviewers acknowledged the novelty and contributions of our proposed method (MetaR,R1,R2,R3), its clinical utility (R1), state-of-the-art performance (R1,R2), and clarity (R1,R3), as well as the extent of the presented study (R3).
The reviewers also asked for i) more details on the mechanism of the proposed method (R2), ii) additional baseline models (MetaR,R2,R3), iii) further elaboration on dataset (R1) and iv) minor editorial clarifications (R2,R3), all of which we will address within the page limit. The paper summary and our response are below.
Paper summary: We proposed evidential graph transformer network (EGTNet) to improve the performance and interpretability of surgical margin assessment for breast cancer. We presented baseline comparison via cross-validation, prospective data validation, intraoperative data validation, network structure ablation, graph structure ablation, uncertainty filtering ablation, interpretability, and clinical relevance.
i) Mechanism of EGTNet: There are two mechanisms embedded in EGTNet: i) node-level attention calculation - via aggregation of neighboring nodes according to their relevance to the predictions, and ii) graph-level uncertainty estimation - via fitting the Dirichlet distribution to the predictions. In the context of surgical margin assessment, the attentions reveal the relevant metabolic ranges to cancerous tissue, while uncertainty helps identify and filter data with unseen pathology. Specifically, the attentions affect the predictions by selectively emphasizing the contributions of relevant nodes, enabling the model to make more accurate predictions. On the other hand, the spread of the outcome probabilities as modeled by the Dirichlet distribution represents the confidence in the final predictions. Combining the two provides interpretable predictions along with the uncertainty estimation.
The in-depth details on the theoretical framework and computation of these mechanisms can be found in their original papers, properly cited in our study. We will add these high-level clarifications on the mechanism of EGTNet in the final version.ii) More baseline models:
Our paper showcases the exceptional performance of our EGTNet (achieving a remarkable balanced accuracy of 91.6%) in comparison with 3 baselines: non-evidential graph transformer (88.7%), graph convolution (88.1%), and vanilla non-graph convolution (87.6) networks. To address reviewers’ recommendation, we will add 3 additional baselines: MC dropout (86.1%), deep ensembles (88.5%), and masksembles (89.2%). These baselines were already implemented as part of a thesis; we did not have them in the initial submission to maintain anonymity as it would not be possible to cite a thesis anonymously. These extra state-of-the-art baselines serve as ideal comparisons for our EGTNet, as they are also capable of uncertainty estimation with different mechanisms. EGTNet outperforms all six baselines. We will cite and include these in our final manuscript.iii) Detail on dataset: We will ensure that Figure 1 is adequately annotated, providing the necessary details of our data. While we offered a concise description of the dataset, for more in-depth understanding of specimen preparation, spectra collection, data curation and labeling, and graph conversion, we will properly reference our earlier publications in the final submission. Please note that we already referred to them in the Introduction but deliberately refrained from citing them in Data Curation to safeguard the anonymity.
iv) Other comments: As requested by the reviewers, we will add additional references on the related works, increase the font size on the images, and report sensitivity and specificity in the results table.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Based on the review and rebuttal. I recommend rejection.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The evidential graph transformer is an interersting approach and the authors’ rebuttal adds additional explanation that may address R2’s concerns about lack of detail (though I understand space limitations in the manuscript). It is an interesting application though as this is outside my field I’m not entirely clear of the clinical relevance. that being said, the novelty of the approach tips me toward the accept decision.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The rebuttal addressed most critical concerns raised by reviewers, especially the insufficient experimental comparison. Though there still exist the concerns, given the contributions of the paper with interesting topic and novel method, I recommend acceptance.