List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Favour Nerrise, Qingyu Zhao, Kathleen L. Poston, Kilian M. Pohl, Ehsan Adeli
Abstract
One of the hallmark symptoms of Parkinson’s Disease (PD) is the progressive loss of postural reflexes, which eventually leads to gait difficulties and balance problems. Identifying disruptions in brain function associated with gait impairment could be crucial in better understanding PD motor progression, thus advancing the development of more effective and personalized therapeutics. In this work, we present an explainable, geometric, weighted-graph attention neural network (xGW-GAT) to identify functional networks predictive of the progression of gait difficulties in individuals with PD. xGW-GAT predicts the multi-class gait impairment on the MDS Unified PD Rating Scale (MDS-UPDRS). Our computational- and data-efficient model represents functional connectomes as symmetric positive definite (SPD) matrices on a Riemannian manifold to explicitly encode pairwise interactions of entire connectomes, based on which we learn an attention mask yielding individual- and group-level explainability. Applied to our resting-state functional MRI (rs-fMRI) dataset of individuals with PD, xGW-GAT identifies functional connectivity patterns associated with gait impairment in PD and offers interpretable explanations of functional subnetworks associated with motor impairment. Our model successfully outperforms several existing methods while simultaneously revealing clinically-relevant connectivity patterns. The source code is available at https://github.com/favour-nerrise/xGW-GAT.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43895-0_68
SharedIt: https://rdcu.be/dnwzA
Link to the code repository
https://github.com/favour-nerrise/xGW-GAT
Link to the dataset(s)
N/A
Reviews
Review #2
- Please describe the contribution of the paper
This work deals with data-limited, multi-class scenarios where severely impaired disease states have few data samples and there are often imbalanced classes. Specifically, severity of gait impairment in PD patients was investigated here. Two key innovations are proposed: (1) a learning-based, stratified sample selection strategy to mitigate sparse and imbalanced sampling, (2) an interpretability technique that involves creating a global mask per class, generated from a modified version of GAT that incorporates edge attributes.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Although only demonstrated on PD, the methods proposed should be widely applicable to other diseases, many of which have issues with limited data size and class imbalanced. Thus, the impact of the work wrt learning-based sample selection could be significant.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
This work is interesting but it tries to put 2 new ideas (a sample selection technique and a custom GAT model) into a single paper, resulting in limited space left for clearly explaining experimental procedures (e.g. range of values used for parameter tuning), discussion (individual-level biomarkers were generated but only class-level biomarkers seem to be shown) and thorough evaluation of both ideas (very brief ablation study for the sampling technique). It might be a better idea to present them as separate papers especially in view of the page limit.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
No significant concerns, code not released yet during the reviewing phase.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
-
Why is it so crucial to have edge weights to be incorporated in the case of PD? Using the FC matrix as the graph would result in the issues mentioned in the introduction, since it is fully connected. However, there are ways to overcome it (e.g. keep the FC features in the node vectors and use a different modality like structural/diffusion MRI to build the graph, or even population graphs)
-
Comparisons were made with GCN, BrainNetCNN and BrainGNN but how does the proposed GAT-based architecture compare with other architectures that allow edge information to be incorporated?
-
Evaluation of the learning-based sample selection strategy seems limited to a pair of experiments (done vs not done). A more detailed study would have been desirable (and possible, if the paper were to be split into 2 separate submissions). Choice of distance metric and node features construction seem arbitrary and would have benefitted from a more detailed ablation study.
-
Dataset is understandably small but if only 1 set of results are shown without any way to incorporate multiple runs (presumably due to the small size), it’d be much better to evaluate both proposed ideas on an additional dataset. Otherwise, it’s difficult to be certain that the improved performance did not happen by chance.
Issues with equations
- between Eqn 4 & 5, in the definition of exp…, exp seem to be missed out on the right hand side (or perhaps remove the exp on the left hand side and it will then be correct)
- Eqn 5 has a typo, \sigma in the large bracket should be \alpha
- Above eqn 7, why is there a subscript > ?
- Write the 2 parts in Eqn 8 separately for better clarity.
- Link between g to prediction \hat{y} not formally presented in the text. Although straightforward, it will be good to at least write a line to explain it. If space is an issue, there’s still the appendix that has not been used yet (2 pages).
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I still feel that the paper should be split up into 2 and each part should have been more thoroughly evaluated. However, it might be sufficient to attract some discussion during the conference.
- Reviewer confidence
Somewhat confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
Some of my concerns are addressed (first 3 points in their rebuttal), but I still hold some reservations wrt the point about dataset and validation. What was mentioned in the rebuttal is the minimal effort expected in terms of ensuring robustness of findings on a small dataset. It would have been an even stronger submission if they showed that these findings are replicated in another dataset (or even in a simulated dataset).
Considering that they haven’t used the space in the supplementary materials, it should be ensured that these additional details they have given in the rebuttal should be added to the manuscript (especially since they are only using 1 small dataset) in the event that this paper gets accepted.
Review #3
- Please describe the contribution of the paper
The proposed xGW-GAT has two main contributions. 1. Mitigate the lack of enough clinical data across all different classes of gait impairment and data imbalance; 2. Produces a global, shared explanation mask per gait category and soft assigns nodes to community clusters.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The proposed stratified, learning-based sample selection can help to mitigate the lack of enough clinical data across all different classes.
- The proposed method can handle the multi-class classification task on PD, and provide the explainability.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Because the number of samples is too small and there is only one disease, it is not persuasive enough to show the proposed sample selection method is useful.
- The proposed explanation mask method is not novel, and similar to the previous paper “Interpretable Graph Neural Networks for Connectome-Based Brain Disorder Analysis”, which is published at MICCAI 22.
- The explainability of salient ROIs is not clear. i.e. the paper doesn’t talk about which specific ROIs are salient in each neural systems.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper describes they will release the code upon acceptance.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
see above.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The novelty of the explanation mask method, and the effectiveness of sample selection are not persuasive enough.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
The authors have answered my concern and I decided to increase my rating to weak accept, based on all reviewer’s comments and the responses of the authors.
Review #4
- Please describe the contribution of the paper
The paper proposes a novel explainable geometric weighted-graph attention network (xGW-GAT) that characterizes discriminative attributions of edge encodings to attention-based, transductive classification tasks and predicts the progression of gait difficulties in Parkinson’s Disease patients. The model uses a stratified, learning-based sample selection method to mitigate the lack of enough clinical data and dataset imbalance and produces a global, shared explanation mask per gait category and soft assigns nodes to community clusters for clinical interpretability. The model successfully identifies connectivity patterns associated with gait impairment in Parkinson’s Disease and offers interpretable explanations of functional subnetworks associated with motor impairment. The proposed framework outperforms existing methods while simultaneously revealing clinically-relevant connectivity patterns.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The main strengths of the paper are:
- Novel formulation: The paper proposes a novel explainable geometric weighted-graph attention network (xGW-GAT) that characterizes discriminative attributions of edge encodings to attention-based, transductive classification tasks. This formulation is a unique combination of graph attention networks and geometric deep learning that is tailored to model gait impairment in Parkinson’s Disease patients.
- Clinical feasibility: The formulation is designed to predict the progression of gait difficulties in Parkinson’s Disease patients and successfully identifies connectivity patterns associated with gait impairment. The framework offers interpretable explanations of functional subnetworks associated with motor impairment, providing a clinically relevant interpretation of the results.
- Sample selection method: The xGW-GAT model uses a stratified, learning-based sample selection method to mitigate the lack of enough clinical data and dataset imbalance. This is an important aspect as datasets in clinical research can often be unbalanced and lack enough data, and this method helps overcome these issues.
- Evaluation: The proposed framework outperforms existing methods while simultaneously revealing clinically-relevant connectivity patterns. This strong evaluation demonstrates the effectiveness and usefulness of the xGW-GAT model for predicting gait impairment in Parkinson’s Disease patients. Overall, the paper offers a novel and effective approach to model gait impairment in Parkinson’s Disease patients and has the potential to help clinicians better understand the underlying connectivity patterns associated with motor impairment.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
While the authors provide clinically-relevant interpretations of the results and identify functional subnetworks associated with motor impairment, they do not provide any experimental validation of these findings. Additional studies should be conducted to validate these interpretations and subnetworks in order to confirm the clinical relevance of the xGW-GAT model in predicting gait impairment in Parkinson’s Disease patients.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
They did well
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Overall, this paper presents an interesting new method for predicting gait impairment in Parkinson’s Disease patients using graph neural networks. The authors demonstrate good performance on their dataset and provide clinically-relevant interpretations of the results. However, an area where the paper could be improved. While the clinical interpretation of the results and identification of functional subnetworks associated with motor impairment is promising, experimental validation would be necessary to confirm the clinical relevance of the identified subnetworks. Additional studies could be conducted to validate these interpretations and subnetworks in order to support the clinical utility of the xGW-GAT model. Overall, I believe this paper is an interesting contribution to the field of Parkinson’s Disease research, and with some additional clarifications and experimental validation, it could have a significant impact on the development of methods for diagnosing and predicting gait impairment in these patients.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Overall, this paper presents an interesting new method for predicting gait impairment in Parkinson’s Disease patients using graph neural networks. The authors demonstrate good performance on their dataset and provide clinically-relevant interpretations of the results.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Strengths:
- The proposed stratified, learning-based sample selection can help to mitigate the lack of clinical data and dataset imbalance. The approach could be applied in many other limited data applications, thus could have potentially significant impact.
- The proposed approach incorporates prediction with explainability to give clinically relevant interpretation of results.
- Good performance on tested dataset.
Weaknesses:
- Unclear explanation of experimental procedures and discussion of salient ROIs.
- Limited evaluation of methods, with experiments performed on 1 small dataset and 1 task.
- Concerns regarding novelty of explanation mask method.
In rebuttal, please:
- Clarify reviewer concerns regarding details of experimental settings (set values, explain model design choices e.g. why edge weights incorporated why chosen node features)
- Clarify novel aspects of the proposed method and particularly the explanation mask approach
Author Feedback
We greatly appreciate the AC’s & reviewers’ efforts for their thoughtful feedback & constructive criticisms. We address their comments below:
Edge Weights (R2): Edge weights indicate the connectivity strength between ROIs, & changes in these weights can reflect symptom/disease severity, which is our objective. Assuming that edges with higher weights exert greater functional connectivity (& vice versa), we can encode how ROIs & their neighbors across various individuals possess similar attributes. Other modalities like diffusion MRI offer complementary information about structural connectivity, and do not represent functional neural circuitry. Similarly, while population graphs are a powerful tool for studying disease states, they do not capture the individual variability in PD progression. Retaining FC edge weights as attributes & attention-weighting message passing innovatively captures individual variabilities.
Benchmark Comparisons (R2): Our GAT-based architecture uses attention-based, edge-weight message passing to depict neighborhood influence from local node embeddings. Standard edge-weight message mechanisms, such as in BrainGNN, BrainCNN, & GCN only capture information from the Euclidean neighborhood between ROIs. Our model also uses GATv2 layers, which introduce dynamic attention for capturing more complexity. We outperformed standard GAT (AUC=0.71) by a significant margin of 0.12 in AUC. Will be clarified in the final paper.
Learning-Based Sample Selection (R2): Our study aims to address sparse & imbalanced data challenges in multi-class tasks. We conducted an ablation study on different node features (Tab. 1). Due to page constraints, we did not include degree profile (AUC=0.53), which performed similarly to the degree centrality feature. The LogE distance metric was adopted to perform sample selection, not node feature construction, hence no ablation study. Although this metric results in the best performance, we will include more testing in the appendix.
Dataset & Validation (R2,R4): Our experimental pipeline incorporated 4-fold-cross validation, oversampling of minority classes, across 100 trials, & macro-weighted average metrics (Tab. 1). We also include a detailed ablation study. We stress that we rigorously validated the results ensuring generalizability & robustness.
Limited Samples/Disease (R3): Our study was designed for a data-limited, multi-class task of disease progression. The proposed stratified sample selection aims to mitigate sparse & imbalanced sampling, common in datasets with severe disease states. Despite those challenges, our method demonstrated significant improvements.
Explanation Mask Novelty & Method (R3): The paper mentioned by R3 proposes an interpretable GNN-based framework with an explanation generator that learns a globally shared mask. However, our approach differs in significant ways. Whereas the paper uses a GCN-based layer & TopK pooling loss for detecting ROI communities, we propose high-dimensional modeling of functional connectomes as SPD matrices to explicitly encode pairwise interactions of entire connectomes, attention-based, edge-weighted, GATv2 layers for locally-influenced message passing, & attention-based explanation masks of each gait class for interpreting relative importance of each ROI per class (as opposed to global mask). We also provide a new, stratified sample selection strategy. These components are acknowledged by R2 as novelties. We will further clarify these points.
Salient ROIs (R3,R4): In our Discussion, we identify salient ROIs, such as the Cerebellar Network contributing lower attention in predicting Moderate to Severe gait severity. To do so, we soft threshold the top-10 attention coefficients in edge masks across layers per gait class. We provide clinically relevant interpretations by reviewing literature & linking them to our findings. We concur with R4 that more clinical experiments are required for the clinical evaluation of these ROIs.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Most concerns were successfully addressed in the rebuttal, with some concerns regarding validation/dataset remaining, which reasonable given the small dataset. However, I feel the experiments performed on the available dataset were well done, and the paper topic addressing limited data with learning-based sample selection I think is of high interest. Following rebuttal, all reviewers now tend toward accept, and I follow their recommendations.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper proposed a learning-based sample selection method to help mitigate the lack of clinical data and dataset imbalance. The key strength is incorporating prediction with explainability to give clinically relevant interpretation of results, and achieving satisfying performance on test dataset.
The authors have satisfyingly addressed the reviewers’ concerns in the rebuttal and all reviewers and AC recommend acceptance of this paper.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This interesting work presented an explainable, geometric weighted-graph attention neural network. It was validated in resting-state functional MRI dataset of Parkinson’s disease subjects. The paper writing is clear and easy to follow. After the rebuttal, all reviewers recommended its acceptance. It will become an important work with strong potentials.