List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Jiajin Zhang, Hanqing Chao, Giridhar Dasegowda, Ge Wang, Mannudeep K. Kalra, Pingkun Yan
Abstract
Various saliency visualization methods have been proposed to explain artificial intelligence (AI) models towards building the trustworthiness of AI-driven medical image computing applications. However, an important question has yet to be answered - are the saliency maps themselves trustworthy? The trustworthiness of saliency methods has largely been overlooked. This paper first proposes the criteria and methods to evaluate the trustworthiness of saliency maps. Then, a series of systematic studies are performed on a large-scale dataset with a commonly adopted deep neural network. The results show that: (i) Saliency maps may not be relevant to the model outputs; (ii) Saliency maps lack resistance and can be tampered without changing the model output. By demonstrating these risks of the current saliency methods, we suggest the community using saliency maps with caution when explaining AI models.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16437-8_43
SharedIt: https://rdcu.be/cVRut
Link to the code repository
https://github.com/DIAL-RPI/Trustworthiness-of-Medical-XAI
Link to the dataset(s)
https://stanfordmlgroup.github.io/competitions/chexpert/
Reviews
Review #1
- Please describe the contribution of the paper
Paper attempts to quantify the trustworthiness using the resilience and resistance of saliency maps. Inspired by adversarial attack methods, they propose a formulation to study how saliency maps change if the adversarial attack changes the classes (relevance) and if the adversarial attack changes the saliency map but maintains the same class (resistance). A comparison with six different saliency maps empirically highlights the drawbacks of current methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Important class of problems where in saliency maps for trustworthiness is quantified and explored.
- First attempt at quantifying the lack of trustworthiness by saliency maps.
- Good experiments and coverage of the saliency maps techniques used to demonstrate their case.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Some of the details are missing.
- A few questions on the basic premise.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
ok.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- For the relevance experiments, the authors fine-tune the network and enforce the condition that the image looks within an \epsilon of the truth image while also optimizing for the saliency map to look identical. The result ends with an adversarial attack where the class changes, but the saliency maps stay the same.
- For some saliency maps, e.g., Grad-CAM, which back propagates the highest output class probability, is it correct to think the network is somehow learning the saliency map invariance? What would happen if you took the Grad-CAM of the truth class, i.e., atelectasis, in the authors’ example? How would that change the Grad-CAM? Can relevance be established then?
- Further, can the authors comment on the real-life scenario where such an attack can happen? For both methods, there was a need to fine-tune the network for a few epochs to enforce this saliency relationship and create the adversarial image. Without fine-tuning, if you have a white box or a black-box attack, isn’t the expectation that the performance of a regular model drops, and the saliency will also be meaningless? What would be the outcomes if an adversarial image was run through this model without the fine-tuning of the model eq 2-4 and using a standard attack?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I question some scenarios where the paper demonstrated the saliency relevance and resistance. Nevertheless, this is an interesting topic that needs exploration, and I base my evaluation on these points.
- Number of papers in your stack
3
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
- This work focus on evaluating the robustness and trustworthiness of common interpretability saliency maps.
- The authors propose two fundamental properties to evaluate the trustworthiness of interpretability saliency maps: relevance (if a model’s prediction change due to alterations in the input, the saliency map should also change) and resistance (if the model’s prediction does not change with alterations in the input, the saliency map should also remain the same).
- In the experiments, the authors show in a Chest X-ray application, that several common interpretability saliency maps demonstrate both a lack of relevance and a lack of resistance.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This work helps to draw the community’s attention to the (lack of) trustworthiness of most interpretability saliency map methods.
- It presents a novel analysis of the interpretability saliency map methods’ quality by introducing two desired properties: relevance and resistance.
- Conclusions are well-supported by the experiments.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Although there is novelty in the work presented, the authors fail to recognize previous works that analyse the reliability of saliency maps (e.g., Adebayo et al. [https://proceedings.neurips.cc/paper/2018/file/294a8ed24b1ad22ec2e7efea049b8737-Paper.pdf]).
- Even though the authors present seven different saliency map methods, some of them are very similar (e.g., VG and VG * Image) and they do not present the results with some other different and well-know methods (e.g. LRP, DeepTaylor). It would be interesting to check if the lack of relevance and resistance still happens in these “relevance”-based methods.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- Experiments were done with a public dataset (CheXpert).
- Code will be made available.
- Reproducibility is guaranteed.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- There are some minor typos in the paper: page 5, “Each adversarial image xp is is”, “optimizerfor”, “Adversarial images xs for evaluating resistance is”; page 8, “The propose properties”.
- In terms of the existence of transferable saliency map attacks, it would be interesting to see a more theoretical analysis of the methods and results obtained (something in the line of Ancona et al. [https://arxiv.org/pdf/1711.06104.pdf?ref=https://githubhelp.com]). Maybe to consider for a future journal version of the work.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- This work helps to draw the community’s attention to the (lack of) trustworthiness of most interpretability saliency map methods. They propose two novel properties to evaluate the trustworthiness of saliency map methods. Experiments are well-performed and demonstrate the lack of robustness of several widely used saliency maps. This work is worth discussing at MICCAI as it demonstrates clear failures of currently used interpretability methods.
- Number of papers in your stack
3
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #4
- Please describe the contribution of the paper
This work investigates the problem of trustworthiness of the saliency maps in the medical imaging research domain via proposing quantitative criteria (relevance and resistance). Experimental studies demonstrate the effectiveness of the criteria in revealing the problems of the overlooked trustworthiness of saliency maps.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The authors propose the relevance and resistance criteria to evaluate the trustworthiness of saliency maps.
- The authors experimentally demonstrate that some popular saliency map-based methods either lack relevance or resistance qualitatively and quantitatively.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The novelty of this work is limited. The fundamental equations (2) and (4) are similar to the equation (4) in the paper [1]. There are many existing works studied on the trustwoorthy of interpretations, e.g., [2-6] and many others.
- Although this work is solid with experimental evaluations, there is no mathematical analysis for the proposed criteria and no discussion of how to improve the trustworthiness of saliency maps.
[1] Improving Deep Learning Interpretability by Saliency Guided Training. Advances in Neural Information Processing Systems, 34. [2] Ghorbani, Amirata, Abubakar Abid, and James Zou. “Interpretation of neural networks is fragile.” In Proceedings of the AAAI conference on artificial intelligence, vol. 33, no. 01, pp. 3681-3688. 2019;
[3 Dombrowski, Ann-Kathrin, Christopher J. Anders, Klaus-Robert Müller, and Pan Kessel. “Towards robust explanations for deep neural networks.” Pattern Recognition 121 (2022): 108194; [4] “Proper network interpretability helps adversarial robustness in classification.” International Conference on Machine Learning. PMLR, 2020; and many others. [5] Kindermans, Pieter-Jan, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. “The (un) reliability of saliency methods.” In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pp. 267-280. Springer, Cham, 2019. [6] Kindermans, Pieter-Jan, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. “The (un) reliability of saliency methods.” In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pp. 267-280. Springer, Cham, 2019. - Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors provide enough information for reproducing the reported results.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- Please refer to the weaknesses of the paper.
- Although this work has solid with experimental evaluations that is valuable for the MICCAI community, how to apply the concepts and results to improve the trustworthiness of saliency maps in medical image application is unclear.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
NA
- Number of papers in your stack
6
- What is the ranking of this paper in your review stack?
5
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Strengths:
- This paper introduces an interesting problem of the trustworthiness of saliency maps
- Experiments are well-designed to analyze and compare different saliency maps by introducing two desired properties of relevance and resistance
Weaknesses:
- Some details are missing
- The authors are missing the discussion with relevant studies such as sanity checks [Adebayo et al. NeurIPS2018].
[Adebayo et al. NeurIPS2018] Adebayo, Julius, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. “Sanity checks for saliency maps.” Advances in neural information processing systems 31 (2018).
Overall: Reviewers agree that experiments are well-designed and the results show interesting findings where several widely used saliency maps lack the robustness for trustworthiness. However, Reviewer #4 makes comments that this work is highly relevant with previous studies and the novelty is limited. Also, more discussion on theoretical analysis of the methods and how to improve the trustworthiness of saliency maps might be needed to improve the paper.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5
Author Feedback
We are grateful to the reviewers and AC for acknowledging our contributions and considering our work “a novel analysis of the interpretability of saliency maps by introducing two desired properties: relevance and resistance.” The reviewers concerned about “discussion with relevant studies” and “some missing details”, as summarized by the AC. This rebuttal clarifies these key points.
- Discussion with relevant studies(R2&R4) We thank R2&R4 for pointing out the relevant works(R2[1], R4[1-5]). The motivation and technical innovation of our work is remarkably different from the references. We will reflect the discussions in the introduction of the camera-ready version.
*Motivation different from R4[1] Ismail; R4[3] Dombrowski; R4[4] Boopathy Our work proposes two systematic criteria for examining the overlooked trustworthiness of saliency maps. Our motivation is different from R4[1] (to obtain noiseless saliency maps), R4[3] (to stabilize saliency maps by model smoothness) and R4[4] (to improve model adversarial robustness). In addition, we clarify that the loss objective in our work differs from R4[1]. Eq.4 in R4[1] aims to optimize the model. However, our Eqs.2&4 optimize input image to generate adversaries by regulating both model output and saliency maps for a given model. The optimization of Eqs.2&4 results in altered images.
*Technical innovations over R2[1] Adebayo; R4[2] Ghorbani; R4[5] Kindermans Previous works used intuitive approaches, such as randomizing models and labels(R2[1]) or adding constant shift to images(R4[5]), to show the irrelevance of saliency maps. In contrast, our work makes two innovations. 1) We systematically examine the trustworthiness of saliency maps by analyzing the bilateral relationship (relevance & resistance) between model prediction and saliency map. The two proposed criteria are the necessary conditions that a trustworthy saliency method should satisfy. 2) We utilize adversarial images to efficiently decouple saliency maps from model predictions. One step further from R42, our targeted adversarial images show the possibility of tampering saliency regions to arbitrary locations without changing model predictions.
Theoretical analysis and future work(R2&R4) We thank the reviewers for the great suggestions. It is fascinating to further analyze why adversaries exist and how they decouple saliency maps from model predictions. Starting from Eqs.2&4, we can see that the saliency attack back-propagates 2nd order derivatives w.r.t. the image. However, the model prediction attack (such as PDG) utilizes 1st order derivatives. Such a difference makes it possible to manipulate saliency map independent of model prediction. We will perform a more formal theoretical analysis to demonstrate this important finding, which can be a great addition to a future journal version as suggested by R2. Furthermore, the observation also implies a potential solution of using 2nd-order Hessian smoothing with adversarial training to mitigate the revealed issues. We’ll reflect the above content in the discussion section.
Model finetuning & real-life case(R1) We clarify that we do not finetune a given model but only optimizes images with the parameter-fixed models to generate adversarial examples. In addition, we evaluate the relevance and resistance of saliency maps only regarding the true class as in Eq.1. One real-life case may involve the adversarial images is to make AI models look more ‘explainable’ without affecting accuracy. Other scenarios are described as in [5,28] in our paper.
Results on other XAI(R2) We were aware of other XAI methods, such as LRP and DeepTaylor, but did not include them for two reasons. 1) The two methods are not commonly used in medical imaging research. 2) LRP requires crafting for each individual network and its implementation for complex networks, e.g. DenseNet, was not discussed in the original work. We’ll include this discussion in the introduction section.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Reviewers agree that experiments are well-designed and the results show interesting findings where several widely used saliency maps lack the robustness for trustworthiness. Some concerns are pointed out in the review but the authors address them during the rebuttal. I believe this paper has more pros than cons, so I recommend acceptance.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
10
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
- the paper studies the resiliance and resistance of saliency map wrt to purturbation in the image
- There are discussion about the novelty of the paper. I am not sure I am convinced by the discussion that the paper is truely novel. It seems the difference from the previous method is limited. However, given that it is important problem and the reviewers agree that the experiments are well-designed, I think the novelty is sufficient for MICCAI.
- One of the reviewers asked for mathematical analysis. That is difficult in 8-page miccai format. Although the reply by the authors is not particularly convincing, I don’t hold that against them.
- I voted for accept.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
na
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Paper attempts to quantify the trustworthiness using the resilience and resistance of saliency maps via proposing quantitative criteria (relevance and resistance). Experimental studies demonstrate the effectiveness of the criteria in revealing the problems of the overlooked trustworthiness of saliency maps. The major concern is the limited novelty and lack of mathematical analysis. The author response partially address the first concern but not to the second concern. So this paper has some merit, but does not differentiate itself well from existing work.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
12