List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Shuting Liu, Baochang Zhang, Rong Fang, Daniel Rueckert, Veronika A. Zimmer
Abstract
The number of stroke patients is growing worldwide and half of them will suffer from cognitive impairment. Therefore, the prediction of PSCI becomes more and more important. However, the determinants and mechanisms of PSCI are still insufficiently understood, making this task challenging. In this paper, we propose a multi-modal fusion model to solve this task. First, dynamic graph neural representation is proposed to utilize and integrate multi-modal information, i.e. clinical tabular data and image data, which can divide them into node-level and global-level properties, instead of processing these data uniformly. Second, considering the variability of patient brain anatomy, a subject-specific undirected graph is constructed based on the connections among 131 brain anatomical regions segmented from image data, while first-order statistical features are extracted from each brain region and internal stroke lesions as node features. Meanwhile, a novel missing information compensation module is proposed to reduce the impact of missing elements in tabular data. In dynamic graph neural representation, two kinds of attention mechanisms are embedded, which encourage the model to automatically localize brain anatomical regions that are highly relevant to this task. One is node attention established between global tabular neural representation and nodes, the other is multi-head graph self-attention which changes the static undirected graph to several dynamic directed graphs and optimizes the broadcasting process of the graph. The proposed method achieves the best overall performance with a balanced accuracy score of 79.6%, outperforming the competing models.The code is publicly available at github.com/fightingkitty/MHGSA
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43993-3_33
SharedIt: https://rdcu.be/dnwNy
Link to the code repository
https://github.com/fightingkitty/MHGSA
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The author proposed a multi-model graph fusion model to predict Post-Stroke Cognitive Impairment (PSCI) and the proposed method achieves the best overall performance with a balanced accuracy score of 79.6% on PSCI prediction, outperforming the competing models.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1).It proposes a dynamic graphs neural representation that integrates multi-modal information and considers subject-specific brain anatomy. 2).It includes an effective missing information compensation module to reduce the impact of incomplete clinical data.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1).The authors did not provide scanning parameters for MRI data in the dataset, which is not conducive to the reproduction of the article. 2).The authors do not demonstrate the clinical value of this approach.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
In section 3.1 on Data and Preparation, the author failed to provide a comprehensive description of the index variable division and scoring rules for stroke patients’ clinical data. This inadequacy has made it challenging to replicate the results presented in the article.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The paper proposes a novel multi-modal fusion model to solve the challenging task of predicting PSCI. However,the paper does not discuss or compare the proposed method with widely-known baselines in the field.And the authors did not demonstrate the clinical utility of this approach. In the other hand,if the author can clearly explain the method of data collection, processing, and division, the reproducibility of the article will increase.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The determinants and mechanisms of PSCI are still insufficiently understood, making the prediction of PSCI challenging.The proposed method was able to achieved a balanced accuracy score of 79.6% on PSCI prediction.However, there are still some issues that need to be addressed. For instance, the clinical usefulness of the proposed method has not been sufficiently demonstrated. Additionally, the description of the dataset is somewhat lacking in detail.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
some issues have been replied, but a few issues are still not.
Review #3
- Please describe the contribution of the paper
This work proposes a framework to combine the advantages of imaging data and health reports for the diagnosis of post-stroke cognitive impairment. Specifically, since the clinical data may contain incomplete recordings, the authors design a missing information compensation module to recover the information loss. Further, a self-attention-based module is introduced for fusing the multimodal data. The effectiveness of the proposed method is validated via the ablation study on an in-house dataset with 418 subjects.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
(1) The research topic is meaningful and seldom investigated. (2) The key challenge and basic idea of this study are well-explained in the introduction. (3) The formulation for the algorithm is clear and easy to follow.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
(1) The writing structure of the method part is not reasonable. (2) The motivation for network structure design is unknown. (3) The experiment result is not sufficient enough. Please refer to the comments below for details.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
This work has relatively high reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
(1) This work could have more impact if they consider the situation with only image data or clinical data when they design the missing information compensation module. (2) It would be better to reorganize section 2.3 to introduce the node attention block and multi-head graph self-attention block before the training hyperparameters. (3) Please provide more introduction and motivation on why did the authors design the node attention block and multi-head graph self-attention block as such. (4) Only some simple baselines were included in the experiments. The experiment results would be more convincing if they could employ more baselines, even SVM and random forest. (5) It seems that the improvement of the missing information compensation module is not significant by comparing I-A and I-B in Table 1. (6) It would be interesting if they can provide some analysis of the missing information compensation module. For example, what is the average missing rate for the failure cases? What is the most important item in the tabular data? (7) The image resolution in the manuscript is relatively low.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Insufficient experiments and unreasonable paper structure.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
4
- [Post rebuttal] Please justify your decision
This work provides an interesting study on the incomplete modality for cognitive outcomes prediction. Some information was further provided through the rebuttal to improve the quality of this paper, but it would be better if they could provide the detailed description and evaluation of their network designs, which would help the readers to better understand the novelty of this work.
Looking forward to their final version.
Review #4
- Please describe the contribution of the paper
This work presents innovative methods for integrating MRI data and clinical information to predict PSCI. The approach models MRI features as separate graphs based on the spatial locations of brain regions, while assigning attention weights to these graphs, taking into account their structure and the intricate interplay between clinical and MRI data. The compensation module also effectively handles missing data. The proposed method is original, thought-provoking, comprehensive and clinically useful.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1) This work applies attention graph neural networks to integrate MRI data and clinical data for PSCI prediction, presenting a novel application. 2) Instead of merely concatenating MRI and clinical embeddings, as is common in existing literature, this work characterizes MRI features as individual graphs according to brain region spatial locations. It uses both clinical information to assign attention weights to graph nodes and Multi-Head Graph Self-Attention Block to assign attention weights to subgraphs. Additionally, its compensation module effectively handles missing data. 3) The carefully designed ablation study evaluates the compensation module, graph network structure, and the fusion of attention modules separately. An effective missing information compensation module is proposed to reduce the impact of incomplete clinical data, suggesting the superiority of each step of the proposed approach. The method appears original, thought-provoking, and comprehensive. 4) Visual attention maps allow for the identification of the contributions of different brain structures to this prediction task. The top brain structural regions strongly associated with PSCI, as uncovered by the proposed method, align well with previous literature, suggesting clinical significance. 5) The evaluations are rigorous, employing five measures including Balanced Accuracy (BAcc), Classification Accuracy (Acc), Precision (Pre), Sensitivity (Sen), Specificity (Spe), and the area under the receiver operating characteristic curve (AUC). Standard deviations are provided for all metrics, and a 5-fold CV is used to split the training and validation sets. 6) The paper is generally well-written, with clear descriptions of the workflow, mathematical notations, formulas, and real data analysis.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) In the methods section, could more details be provided on the Multi-Head Graph Self-Attention Block? For instance, how would the number of heads be selected? “The shapes of query qg, key kg, and updated graph feature G′ are reshaped as [h, N, C/h].” Is this reshaping based on splitting the original graph into subgraphs? How are the edges used in this “Multi-Head Graph Self-Attention Block”? Since the “Multi-Head Graph Self-Attention Block” is a key novelty of this paper, a more detailed description is needed. 2) If hyperparameters were tuned based on the validation data, it is unclear how the proposed method would perform on an independent test set.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Code will be provided; clear mathematical modeling, dataset description and split, evaluation metrics, hyperparameter configuration and clinical significance discussed; no computational cost was included.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Add more details to the Multi-Head Graph Self-Attention Block, especially on how you incorporate the graph (edge) structure?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The proposed method is generally original, thought-provoking, comprehensive and clinically useful.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper proposes a novel multi-modal fusion model to predict PSCI with strengths lying in its dynamic graphs neural representation, subject-specific brain anatomy, and effective missing information compensation. In addition, the research topic is meaningful and seldom investigated, the key challenge and basic idea of this study are well-explained in the introduction, the formulation for the algorithm is clear and easy to follow, ablation study is well designed. Weaknesses include inadequate description of dataset and need for improvement in demonstrating clinical value, more details should be provided on the Multi-Head Graph Self-Attention Block, etc. Please consider addressing the following points (not limited to the following points) in the rebuttal: 1) The authors did not provide scanning parameters for MRI data in the dataset; 2) Please provide more introduction and motivation on why did the authors design the node attention block and multi-head graph self-attention block as such; 3) It would be interesting if they can provide some analysis of the missing information compensation module. For example, what is the average missing rate for the failure cases? What is the most important item in the tabular data; 4) Simple baselines were included. Overall, this paper is generally well-written. Modification is expected.
Author Feedback
We appreciate the reviewers’ comments and suggestions. They found our approach novel (R1), original and thought-provoking (R4) with rigorous evaluation and a carefully designed ablation study (R4), addressing a meaningful and seldom investigated research topic (R3). We elaborate on their comments below.
Motivation of network design (R3&MR): Inspired by [14] and in order to optimize the broadcasting of the graph, the Multi-Head Graph Self-Attention Block (MHGSAB) is proposed to learn the edge weights during training. The node attention block fuses information from tabular (global level) and image data (local/node level) by measuring the correlation between tabular and node features. Tab1-III shows that the two attention mechanisms result in a substantial improvement. The attention mechanisms help to explore the contribution of each brain region (Fig 2).
More details in method (R4): In MHGSAB, subgraphs share the same adjacency matrix E with the size of [N, N]. Based on the number of heads h, the computed query, key and updated graph features are reshaped to [h, N, C/h], and then assigned to h subgraphs (i.e., each subgraph has a different sub-query, sub-key and sub-updated graph feature with the same size of [1, N, C/h]). Attention weight A2 is computed for h subgraphs using Eq. (7). We noticed a typo in Eq. (8) (missing multiplication sign between E and A2). The adjacency matrix E will be broadcasted automatically according to the size of A2, multiplied elementwise with A2, and then multiplied with the updated graph G’. Using grid-search on the 1-fold validation set we select h=2. The method is evaluated via 5-fold cross validation. We report the average performance on an independent set.
Analysis of missing data compensation (R3&MR): To evaluate the missing data compensation, we performed an ablation experiment. An improvement in the performance can be seen (Tab 1), which is however relatively small due to the fact that the majority of our tabular data is complete. Following the reviewer’s suggestion, we selected all the complete samples from the test set and randomly removed some elements under different missing rates. We found that if the missing rate is about 30%, we observe a 10% performance drop (i.e., about 0.72 of BAcc). Using single-element-out degradation measurement on these complete samples, we find that the top-3 most important elements according to BAcc are atrial fibrillation, sex, and Montreal Cognitive Assessment (MoCA).
More details in data description (R1&MR): We provide a more detailed description of the dataset. Images were acquired using a 3T MRI Scanner (Siemens Healthineers, Erlangen, Germany) within 5 days of stroke onset. T1w (TR=2500ms, TE=4.33-4.37ms), FLAIR (TR=5000ms, TE=393-398ms) and DW-MRI (TR=12700-13400ms, TE81-84ms) are rigidly aligned and resampled to 1x1x1mm3. The clinical data includes age, sex, education, smoking, drinking, hypertension, diabetes, atrial fibrillation, stroke history, BMI, low-density lipoprotein levels, stroke severity, pre-stroke function, MoCA, stroke lesion volume, lacunes number, Fazekas score of WMH in periventricular and deep regions, cerebral microbleeds number, perivascular spaces level, which are collected from clinical laboratory.
Baseline (R3&MR): We compare our method with SVM and random forest (RF) as suggested by R3. SVM achieves 0.60 BAcc and 0.75 AUC; RF achieves 0.61 BAcc and 0.69 AUC. Both are outperformed by our method.
Discussion on clinical value (R1): There are several areas where the proposed method is highly clinically relevant: (1) Risk prediction to identify (i) low-risk patients, and (ii) high-risk patients (for those early and targeted treatment and rehabilitation is recommended). (2) Detailed phenotyping and characterisation of high-risk patients and follow-up. (3) Identification of novel biomarkers and risk factors.
Thanks again for your insightful comments and constructive suggestions.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper proposes a novel multi-modal fusion model to predict PSCI with strengths lying in its dynamic graphs neural representation, subject-specific brain anatomy, and effective missing information compensation. In addition, the research topic is meaningful and seldom investigated, the key challenge and basic idea of this study are well-explained in the introduction, the formulation for the algorithm is clear and easy to follow, ablation study is well designed. Weaknesses include inadequate description of dataset and need for improvement in demonstrating clinical value, more details should be provided on the Multi-Head Graph Self-Attention Block, etc. Although some weaknesses are occurred, most of them are addressed by the author, it is an interesting paper where merits slightly weigh over weakness.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Most of the major concerns (such as motivation of network design, more details in data description) have been well clarified.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Key strengths:
- Novel approach to integration of MRI and clinical data via GNNs, assigning attention to brain nodes according to clinical data, among other innovations.
- Considers missing information problem for the clinical data
Key weaknesses:
- Missing some key methods details and clinical motivation for prediction problem
- Lacking some comparisons in the experimental validation
A number of clarifications were given in the rebuttal, which are helpful for understanding methods, experiments, and clinical value for specific application. Unfortunately, some of rebuttal was devoted to new experimental results, which we have been instructed to not consider. Even without, I found the validation ok, with some baselines and some important ablations evaluated with multiple metrics. Given the reviewers’ overall assessments and the interesting methodology, I recommend accept.