List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Gaurang Karwande, Amarachi B. Mbakwe, Joy T. Wu, Leo A. Celi, Mehdi Moradi, Ismini Lourentzou
Abstract
Despite the progress in utilizing deep learning to automate chest radiograph interpretation and disease diagnosis tasks, change between sequential Chest X-rays (CXRs) has received limited attention. Monitoring the progression of pathologies that are visualized through chest imaging poses several challenges in anatomical motion estimation and image registration, \ie spatially aligning the two images and modeling temporal dynamics in change detection.
In this work, we propose \modelname, a neural model that can track longitudinal pathology change relations between two CXRs. \modelname incorporates local and global visual features, utilizes inter-image and intra-image anatomical information, and learns dependencies between anatomical region attributes, to accurately predict disease change for a pair of CXRs. Experimental results on the Chest ImaGenome dataset show increased downstream performance compared to baselines. Code is available at https://github.com/PLAN-Lab/ChexRelNet.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16431-6_55
SharedIt: https://rdcu.be/cVD7b
Link to the code repository
https://github.com/PLAN-Lab/ChexRelNet
Link to the dataset(s)
https://physionet.org/content/mimic-cxr-jpg/2.0.0/
https://physionet.org/content/chest-imagenome/1.0.0/
Reviews
Review #1
- Please describe the contribution of the paper
The authors present CheXRelNet, an anatomy-aware model for tracking longitudinal relationships between chest X-rays. The proposed approach allows the authors to perform diagnosis and monitoring of a patient through comparisons of sequential chest X-rays. The method takes 2 sequential chest X-ray images of a patient and evaluate disease change.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The proposed model utilizes both local and global anatomical information to output accurate localized comparisons between two sequential chest X-rays examinations. The authors came up with the graph construction to capture correlations between anatomical regions from a pair of chest X-rays. The proposed model outperforms baselines.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
There is no ablation study for the classification module.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper seems to be reproducible since it is written pretty clearly, the authors intend to release the code and an open-source dataset was used for evaluation.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The authors present CheXRelNet, an anatomy-aware model for tracking longitudinal relationships between chest X-rays. The technical novelty of the paper is at a good level. Evaluation is reasonable but lacks an ablation study for the classification module. The paper is well written and easy to reproduce. The authors investigated an important problem in the field of medical image analysis and achieved state-of-the-art results.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Technical novelty, reproducibility, and results achieved.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
This paper works on monitoring the changes of pathological findings in Chest X-rays (CXRs), i.e., given a pair of two sequential CXRs, to predict whether a finding is improved or worsened. The authors propose to leverage both global information from the whole image and local information from the given anatomical region bounding boxes for the prediction. They use pathology co-occurrence to construct a graph and build CheXRelNet based on the graph neural network to merge information. Experiment results show that the proposed CheXRelNet outperforms the separate global and local baselines on the Chest ImaGenome dataset.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Clinical relevance: The sequential monitoring of pathological findings in CXR can potentially help patient follow-up. This topic has limited research before.
- Global-local relation consideration: It is promising if the prediction from the neural network can leverage more local information since the pathology is usually localized in specific regions. The inter/intra-image relation proposed in the paper can provide complementary information for the global decision.
- The proposed CheXRelNet is based on the graph neural network that can merge information from different regions and from past to current.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Not significant improvement compared to the Global baseline: From Table 2, we can see that the proposed CheXRelNet slightly outperforms the Global baseline (0.68 vs 0.67 for All).
- Lack of statistical analysis: Given such a small gain, the authors should give some statistical analysis, e.g., confidence interval, to verify the effectiveness of the proposed method.
- Limited interpretability: A critical factor in using local information is that it can provide precise localization, which can largely help clinicians understand the outcome. The authors also claim that the CheXRelNet can output accurate localized comparisons (contribution 1). But no such demonstration is provided. The result is still largely performance-driven.
- The way to build the graph: The authors use pathology co-occurrence to build the graph. However, this process is somewhat “hard-coded,” i.e., a manually defined threshold is used to determine the connectivity.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors have shown assumptions/ implementation details, etc. The evaluation is on a public dataset [26]. The authors are encouraged to make code publicly available.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- For the graph construction, the authors should give some ablation study or discussion on how to choose the threshold. I also wonder if the authors can propose some soft version to merge information among regions.
- For the comparison, the proposed global-local network needs more computation and parameters compared to the plain Global model. So it is not clear whether the performance gain is from the global-local consideration or more parameters. I recommend that the authors think of a way to justify the proposed network’s effectiveness further.
- Zero-shot evaluation results are presented for comparison. The authors can show more details for the inference during zero-shot evaluation for completeness.
- The authors use the ResNet101 autoencoder [26] to extract features, and the autoencoder is pre-trained on several imaging datasets. I am not clear if the features are extracted from the encoder.
- For the two off-diagonal k by k blocks, the authors can further clarify the explanation, e.g., with the s, t index range. It is a bit hard to understand at first glance.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The monitoring topic and global-local consideration are interesting for CXR. But, there exists some weakness in the method design and experiment results.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
The authors address some of my major concerns about the significance of the proposed ChexReINet against the global baseline. The method is novel in terms of local region and graph construction. However, there still exists some room for performance improvement in the design of the graph network. For the interpretability, it would be interesting if the authors could provide some visualization techniques such as heat maps to show that the incorporation of local regions can indeed focus on the cause region rather than the analysis on the correct label. Overall, the topic and the method are interesting but need more careful effort.
Review #3
- Please describe the contribution of the paper
The study proposes using a graph attention neural network to learn the correlations between the follow-up CXRs to track the pathological change. The model considers both local and global visual features. The experiment compares the proposed approach (local+global) with the local-only model and global-only model.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The study is nicely explained and formulated.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
I could not see any major weakness in the study.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
the study is developed and test on a publicly available dataset. the implementation details and parameters are provided in the manuscript. The codes did not shared.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
nice study.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
nice study.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Somewhat Confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper received three mixed reviews and is recommended for rebuttal. Please address the following adequately in the rebuttal.
”- Not significant improvement compared to the Global baseline: From Table 2, we can see that the proposed CheXRelNet slightly outperforms the Global baseline (0.68 vs 0.67 for All).
- Lack of statistical analysis: Given such a small gain, the authors should give some statistical analysis, e.g., confidence interval, to verify the effectiveness of the proposed method.
- Limited interpretability: A critical factor in using local information is that it can provide precise localization, which can largely help clinicians understand the outcome. The authors also claim that the CheXRelNet can output accurate localized comparisons (contribution 1). But no such demonstration is provided. The result is still largely performance-driven.
- The way to build the graph: The authors use pathology co-occurrence to build the graph. However, this process is somewhat “hard-coded,” i.e., a manually defined threshold is used to determine the connectivity.”
”- For the graph construction, the authors should give some ablation study or discussion on how to choose the threshold. I also wonder if the authors can propose some soft version to merge information among regions.
- For the comparison, the proposed global-local network needs more computation and parameters compared to the plain Global model. So it is not clear whether the performance gain is from the global-local consideration or more parameters. I recommend that the authors think of a way to justify the proposed network’s effectiveness further.
- Zero-shot evaluation results are presented for comparison. The authors can show more details for the inference during zero-shot evaluation for completeness.
- The authors use the ResNet101 autoencoder [26] to extract features, and the autoencoder is pre-trained on several imaging datasets. I am not clear if the features are extracted from the encoder.
- For the two off-diagonal k by k blocks, the authors can further clarify the explanation, e.g., with the s, t index range. It is a bit hard to understand at first glance.”
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5
Author Feedback
We thank reviewers and ACs for their insightful comments and constructive feedback, and for recognizing the paper’s clarity, organization, novelty, and adequacy of methods for the considered tasks.
R1 Ablation studies were limited mainly due to space constraints, we performed ablation analysis on ChexRelNet, we report accuracy: original ChexRelNet 2 GAT layers (0.686), 1 GAT layer (0.682), 3 GAT layers (0.686), without GAT (0.680). Will add these with more details (#params, etc.) to camera-ready.
R2 Statistical analysis: Accuracy and standard deviation (SD) over 6 trials for ChexRelNet (0.683 +/- 0.0024), Global (0.672 +/- 0.0046) and Local (0.602 +/- 0.0059). Unpaired t-test between ChexRelNet and Global predictions pval=0.049 implies significant difference between predictions of the two models. The one-tailed t-test between ChexRelNet and Global accuracies (over 6 trials) pval=0.018 shows our model’s significant improvement over Global. From the SD and t-test results, the proposed method is effective and outperforms baselines. We will add these evaluations to our paper.
R2 Limited interpretability: Supplementary material presents qualitative results. Multiple groups have published on detection and localization of disease in CXR. While we agree that detecting the precise localization is desirable, this work is focused on the extremely challenging task of detecting change. From a clinical point of view, many diseases in CXRs are diffused in terms of their features and are not always well defined. Many of the lung conditions appear across a large area of the lungs. Diseases can spread outside the locality where they first existed. In other words, the disease might remain unchanged in one part but also appear in another. Given these complications, a localized study of temporal change of disease features needs careful clinical design and a dataset that is tracked for changes at local and neighborhood levels.
R2 Graph threshold: Indeed, as the reviewer suggests, there exist several options for constructing an adjacency matrix and hence merging information among regions, including soft-versions, e.g., weighted, exponentially weighted or dynamic adjacency matrices. There is an inherent trade-off between computational complexity and accuracy. Soft-versions densely connect all nodes. Simpler binary versions are more computationally efficient. How well long-tailed dependencies are captured and how much overfitting occurs depends on the filtering threshold. Similarly, there exist several options for graph construction. We have added some ablation studies regarding the R1 concern. We note that further exploration of the design choices is beyond the scope of this work.
R2 Model capacity: We performed ablation study with deeper global and local baselines (performance in parentheses). The pretrained ResNet backbone is kept the same across all methods and then we stack more layers in the classification module. ChexRelNet/Ours 38.6M params (0.686), Deeper Local 41.3M params (0.639) and Deeper Global 41.3M params (0.673). Results show ChexRelNet performance gains are not simply from more parameters.Will add more to the supplementary.
R2 Zero-shot evaluation: This experiment involves unseen diseases. We train our model (ChexRelNet) on a set of diseases and test it on a different set of diseases. Our classification labels are not changing but the disease labels are.We will expand more on this in the final draft.
R2 ResNet101 pre-trained autoencoder and diagonal blocks:Features were indeed extracted from the encoder part.The off-diagonal k by k blocks (k number of anatomical regions) represent relationships between same anatomical regions for every pair of images.We will further clarify these parts in the camera-ready.
R3 code:Our code is already hosted on a private GitHub, we will share the link publicly.We have added an anom.link placeholder but refrained from sharing during the review process due to double-blind anonymity requirement.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
After rebuttal, all three reviewers accepted this submission. AC also agrees with this decision.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The strength is the proposed global-local relation consideration for the longitudinal matching of lesions. Most concerns are responded in the rebuttal. Acceptance is recommended.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
4
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Major concerns were addressed in the rebuttal.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
NR