List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Moinak Bhattacharya, Shubham Jain, Prateek Prasanna
Abstract
We present GazeRadar, a novel radiomics and eye gaze-guided deep learning architecture for disease localization in chest radiographs. GazeRadar combines the representation of radiologists’ visual search patterns with corresponding radiomic signatures into an integrated radiomics-visual attention representation for downstream disease localization and classification tasks. Radiologists generally tend to focus on fine-grained disease features, while the radiomics features provide high-level textural information. Our framework first ‘fuses’ radiomics features with visual features inside a teacher block. The visual features are learned through a teacher-focal block, while the radiomics features are learned through a teacher-global block. Next, a novel Radiomics-Visual Attention Feature loss is proposed to leverage this joint radiomics-visual representation of the teacher network to the student network. We demonstrate the efficacy of GazeRadar on disease localization and classification tasks in 4 large scale chest radiograph datasets comprising multiple diseases.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16437-8_66
SharedIt: https://rdcu.be/cVRuR
Link to the code repository
https://github.com/bmi-imaginelab/gazeradar
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The paper combines basic radiomic features and visual attention from gaze maps to localize disease.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Combining radiomics and gaze information.
- Ablation study of the contribution of each component
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Using only a few radiomic features
- The metrics used for comparison are not typically used for localization/detection tasks.
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Reproducible
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- The methods section can be restructured for clarity. It is currently confusing as to how the GAF is pretrained and what was the input data for pretraining.
- Standard metrics such as IoU, mAP can be used for comparison.
- As a baseline, a simple model that does a basic multiplication of the radiomic image with the gaze map and then thresholded for localization can be used to show the benefit of a deep learning model.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Combining radiomics and attention without the segmentation aspect. Unclear methods description. insufficient metrics for comparison.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
5
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
A novel architecture to fuse radiomics and visual attention is proposed for classification and localization tasks. A novel loss is proposed to calculate the distance between the student block attention distribution, and the joint representation. Experiments demonstrate the effectivenesss of the proposed method.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1, The proposed problem is clinically relevant, as model explanation is an important task. 2, The proposed method is extensively evaluated on several benchmarks, demonstrating its effecitveness. 3, The problem is clearly motivated and formulated, and each component is well explained. 4, The proposed methods properly addressed the gaze difference problems of the ragiologists.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The performance boost seems rather negligible. It would be ideal if the authors can demonstrate that the proposed method performance increase is significant.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors are willing to share the training and testing code. Thus, it’s reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
It would be ideal if the authors can demonstrate that the proposed method performance increase is significant.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- Clinically relevant
- The network is novel
- Properly evaluated on several benchmarks and the results demonstrate the effectivness.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The authors present GazeRadar, a novel global-focal student-teacher architecture for disease localization based on radiomics information and visual attention features. The representation is used to train a student block for downstream classification and localization tasks. The authors develop novel Radiomics Attention Fusion and Gaze Attention Fusion strategies to fuse radiomics features and gaze features.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors came up with a novel loss function to transfer the joint radiomics-visual knowledge from the teacher block to the student block. The authors claim that the proposed work is the first work that incorporates radiomics and radiologists’ search patterns into a decision-making pipeline. The authors provide ablation study for their GAF-RAF strategies.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
There is a need to work on the clarity of the text. Many abbreviations and incomprehensible references within the text make confusion and make it difficult to understand.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper seems to be reproducible since the authors intend to provide the code and open-source datasets were used for evaluation (RSNA Pneumonia Detection Challenge Dataset, SIIM-FISABIO-RSNA COVID-19 Detection Dataset, NIH Chest X-rays Dataset, and VinBigData Chest X-ray Abnormalities Detection Dataset).
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The authors present the global-focal student-teacher architecture for disease localization based on radiomics information and visual attention features. The authors explore an interesting topic related to eye-tracking. The developed model can be of practical importance, as it improves the results of disease localization using easily collected eye-tracking data. The authors provide a solid ablation study but the paper lacks clarity.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Technical novelty, reproducibility, and results achieved.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper combines radiomics features with visual gaze features for learning the characteristics of the anomaly in chest X-ray images. Pre-generated gaze feature datasets are effectively utilized. The reviewers have seen this as an interesting reuse of the eye gaze datasets provided in Physionet and other open sources. However, one consideration to address in their revision, besides taking the reviewers’ comments, is to explain how the inference is done and whether it needs eye gaze data or only radiomics data for prediction. While eye gaze data is being acquired for this data generation purpose, it is not typically used in clinical practice, so this aspect could be important. Can those cropped regions based on the anomalies be made available in open source?
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3
Author Feedback
We thank the reviewers for their comments and suggestions, and note their acknowledgment of the clinical relevance, network novelty, and practical applicability of the proposed work. The point-by-point responses to the reviewers’ critiques and new changes in the updated manuscript are provided below:
1) Does GazeRadar need eye gaze or radiomics data during inference? (Meta-Reviewer): The eye-gaze or radiomics features are not needed during inference. These are only used to pre-train the Teacher network.
2) Using only a few radiomic features (R1): In this proposed work, as stated in Section 2.1 Global-Focal Network, this method can be generalized to k radiomic features. As a proof-of-concept, we used two radiomics features to demonstrate the efficacy of GazeRadar.
3) Restructure for clarity (R1): We apologize for the confusion. As suggested, we will restructure the pre-training of GAF in the camera-ready submission.
4) Standard metrics such as IoU, and mAP can be used for comparison (R1): Thank you for the suggestion. AP and MSE are shown for localization, and Accuracy, AUC, and F1 are shown for classification (in Tables 1, 2 in the main paper and Table 1 of the supplementary section). Furthermore, the authors will include GIoU scores in the supplementary section for the camera-ready version.
5) Will multiplying radiomic image with the gaze map and then thresholding for localization help a deep learning model? (R1):
We would like to clarify that the image obtained post multiplication will not help the deep learning model to focus on relevant regions and, in turn, is irrelevant for downstream localization tasks.6) The performance boost seems rather negligible/insignificant (R2): We respectfully disagree. As shown in Table 1, GazeRadar significantly outperforms SOTA architectures in 3 out of 4 datasets for classification, and in 1 out of 4 datasets for detection. Also, we show (in Figure 2) the qualitative improvement in the detection of the pulmonary pathologies (as bounding boxes).
7) Incomprehensible abbreviations and references (R3): Apologies for the confusion. The authors will ensure that the camera-ready version is more comprehensible.