List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Jiaxing Gao, Lin Zhao, Tianyang Zhong, Changhe Li, Zhibin He, Yaonai Wei, Shu Zhang, Lei Guo, Tianming Liu, Junwei Han, Tuo Zhang
Abstract
Brain functional connectivity under the naturalistic paradigm has been demon-strated to be better at predicting individual behaviors than other brain states, such as rest and task. Nevertheless, the state-of-the-art methods are difficult to achieve desirable results from movie-watching paradigm fMRI(mfMRI) induced brain functional connectivity, especially when the datasets are small, because it is diffi-cult to quantify how much useful dynamic information can be extracted from a single mfMRI modality to describe the state of the brain. Eye tracking, becoming popular due to its portability and less expense, can provide abundant behavioral features related to the output of human’s cognition, and thus might supplement the mfMRI in observing subjects’ subconscious behaviors. However, there are very few works on how to effectively integrate the multimodal information to strengthen the performance by unified framework. To this end, an effective fu-sion approach with mfMRI and eye tracking, based on Convolution with Edge-Node Switching in Graph Neural Networks (CensNet), is proposed in this arti-cle, with subjects taken as nodes, mfMRI derived functional connectivity as node feature, different eye tracking features used to compute similarity between sub-jects to construct heterogeneous graph edges. By taking multiple graphs as differ-ent channels, we introduce squeeze-and-excitation attention module to CensNet (A-CensNet) to integrate graph embeddings from multiple channels into one. The experiments demonstrate the proposed model outperforms the one using single modality, single channel and state-of-the-art methods. The results suggest that brain functional activities and eye behaviors might complement each other in in-terpreting trait-like phenotypes.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43895-0_27
SharedIt: https://rdcu.be/dnwyn
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes A-CensNet, which predicts the cognitive scores of subjects. In this approach, subjects are taken as nodes, mfMRI-derived functional connectivity serves as node feature, and various eye-tracking features are utilized to compute similarities between subjects, thereby constructing heterogeneous graph edges.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
(1) It is interesting and innovative to utilize multiple modes of brain functional connectivity data and eye-tracking data simultaneously. (2) The experiment results are highly significant, as they suggest that brain functional activities and eye behaviors may complement each other in interpreting trait-like phenotypes.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
(1) The paper lacks clarity on how other works that incorporate both fMRI and eye-tracking data fuse these two modes of information. Additionally, the author’s rationale for utilizing the fusion approach in this study could be elaborated further. (2) The method’s effectiveness is evaluated using too few metrics, with only AUC utilized as an evaluation metric. (3) The amount of data used in this study is relatively small.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
According to the reproducibility checklist, the reproducibility of the paper is acceptable.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
(1) To enhance the credibility of the results, it is recommended to increase the amount of data used in the study. (2) To better evaluate the proposed method’s performance, it is recommended to incorporate additional evaluation metrics. Furthermore, it would be beneficial to compare the proposed approach with other methods that utilize eye-tracking data. (3) The paper would benefit from improved legibility.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The research significance of this paper is significant.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The paper proposes a new deep neural network architecture called A-CensNet to predict subjects’ cognitive score with movie-watching fMRI and eye tracking data, which outputforms SOTA in quantitative evaluation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper proposes a novel method to integrate information from eye-tracking data on the foundation of the method in Ref 16. It takes multiple features in eye-tracking data to form multiple graph structures that represent similarity between subjects and uses A-CensNet to integrate mutliple graph structures into one hybrid graph.
- The paper shows thorough experiment results including comparison with multiple SOTAs and ablation studies to validate effectiveness of multiple design choices.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The paper fails to discuss the effectiveness of the method from a biological perspective. In other words, it would be beneficial to show what features the network uses to predict cognitive scores and what biological mechanisms they reveal.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Good reproducibility. Public data were used and codes will be released.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
As mentioned in Q6, it would be beneficial to show what features the network uses to predict cognitive scores and what biological mechanisms they reveal. Some data visualizations can do the job. In addition, experiments that drop out some identified features are also good ways to validate the findings.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
As listed in Q5, the paper proposes a novel architecture to predict cognitive score with movie-watching fMRI and eye tracking data, which outperforms SOTA methods. The proposed architecture is well investigated with ablation studies.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper addresses the problem of detecting individual’s behaviors by using multimodal data. Specifically, authors propose combining two deep learning methods to unify movie-watching paradigm fMRI data with eye tracking trajectory and eye pupil size data of the individuals and use this data in classification of behavior of subjects into 4 classes. Authors demonstrate the utility of their method over a dataset of 81 subjects (from HCP dataset) and compare their results with the state of the art methods to demonstrate its superiority.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Uses multi-modality data and deep lerning to combine the data in prediction of cognitive state
- Experiments are done on a decent size dataset to demonstrate the utlility of the method.
- Proposed method is extensively compared against state of the art methods.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The paper has two major flows, one in its writing and the other in its study design. 1) Writing: While introduction and methods sections are clearly written and the results presented are promising, the style and language of results and discussion sections are more akin to that of a technical report, with several grammar mistakes, which sometimes introduce ambiguities in meaning. 2) Study design: Considering the small size of the dataset, cross validation would have made the analysis more robust. However, it is not clear whether the authors did cross validation or not.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Authors provided details of data processing and how the models were trained. I would assume, one could repeat the experiments by following the steps explained in the paper.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Writing: While introduction and methods sections are clearly written and the results presented are promising, the style and language of results and discussion sections are more akin to that of a technical report, with several grammar mistakes, which sometimes introduce ambiguities in meaning. This gave me the impression that the latter section are written by a different author than the rest of the paper. An example to ambiguous meaning could be how the data is split for training, validation, and testing. But pretty much any sentence in these sections could be an example of a grammar mistake. I would strongly consider revising the writing of these sections. The problem with results section is not only about the language but also about the presentation. Ablation study, for example, presents various version of the proposed model in a way that is hard to follow.
Study design: In terms of study design, considering the small size of the dataset, cross validation would have made the analysis more robust. However, it is not clear whether the authors did cross validation or not. Section 3.1 mentions picking a random set of 10 subjects from each cognitive group, and then picked 20 and 21 of the remaining subjects for training and testing, respectively, but do not mention a cross validation. End of section 3.1 merely mentions “AUC is adopted for each cross-validation test to evaluate the prediction performance”, which is confusing, whether 1) cross validation is used or not, 2) if used, whether mean AUC is reported, etc.
Also, determination of four cognitive groups is vaguely explained as simply PCA is used to determine the first component and then a classification is mentioned. Neither the variation explaoined by the first component, nor what type of a classification used is mentioned.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The idea is interesting, problem being addressed is relevant, first half of the paper is well written, and the experiments are mostly convincing about the merit of the method. However, results part of the paper is poorly written and cross validation of the experiments might or might not be done properly (as it is hard to figure out due to how the paper is written on those parts).
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Strengths:
- Proposed a novel graph neural network approach to integrate eye-tracking and movie-watching fMRI data for cognition prediction.
- Extensive experiments with comparisons to SOTA methods and ablation studies to to evaluate model design.
- Interesting experimental results showing promise of incorporating eye-tracking data in predicting brain phenotypes.
Weaknesses:
- Writing quality needs improvement, particularly in results/discussion sections, as errors lead to ambiguous statements.
- Relatively small dataset size used in experiments, combined with only 1 performance metric (AUC) reported and unclear experimental details regarding other eye-tracking/fMRI fusion methods compared and cross-validation setup.
- Lacks discussion of biological interpretation, e.g., which features are important for predicting cognitive scores.
Please proofread for grammatical errors and revise the writing particularly in results/discussion sections for clarity. Also please clarify experimental methods regarding cross-validation approach and how comparison methods fuse eye-tracking/fMRI data.
Author Feedback
We thank the meta reviewer and reviewers for appreciation of our work. We would like to respond to the major concerns.
Q1(AC&R1&R3)：Writing quality and errors lead to ambiguous statements. A：We make some explanations to two major concerns.
- Fusion method of eye-tracking/fMRI data: We represent the subjects as a graph, in which each subject is the node, associated with an fMRI feature vector. The relation between these subjects (specifically, similarity between subjects) is represented by the edge as well as edge weights of the graph. In this work, we using eye movement behavior to measure the subjects’ similarity. On this graph representation, we adopted CensNet as the primary model that updates node features and edge features in turn. By this work, these two modalities are fused.
- Cross-validation on small dataset. On the small dataset of 81 subjects, We divided them into four cognitive levels, with about 20 subjects in each cognitive group. In the experiment, we randomly selected 10 subjects from each group, a total of 40 subjects (about 50% of the total) to form the training set. Among the remaining 41 people, we randomly selected 20 people to be the verification group and 21 people to be the testing group (each accounting for 25% of the total). Such a random division of training/verification/testing subsets was repeated 100 times, independently. The results (AUCs) were the average of 100 independent replicates.
Q2(AC&R1&R3)：Only 1 performance metric (AUC) reported and why classification scheme is used. A: Regression is the ultimate goal of prediction. It was demonstrated in He et al., 2020 that the behavior score prediction accuracy via regression drops dramatically when subject number is below 100 (no more than r=0.1 via Spearman correlation). Since we only have 81 subjects, a low regression accuracy cannot be a trustworthy to be used to compare with state-of-the-arts. As a compromise solution, we adopted classification scheme to demonstrate the effectiveness of our proposed framework. In fact, we have divided them into more groups(not just four groups) and the classification results are show as follows.(-:placeholder) C1: Numbers of Classes&—–4—–&—–6—–&—–8—–&—–10—–&—–12—– ———AUC——–&54.63±0.65&56.34±0.65&56.95±0.48&51.53±0.60&45.52±0.38 ———ACC——–&27.57±0.80&22.20±0.82&16.89±0.63&10.39±0.38&6.65±0.16 Note that we also added classification accuracy (ACC) as another metric. ACC is relatively low and drops with the increase of class numbers. This is consistent with the conclusion in He et al., 2020 on a small dataset via regression method. In contrast, AUC is relatively stable and starts to drop only when class number is larger than 10.
Q3(AC&R2)：Lacks discussion of biological interpretation. A: In our graph representation, node features relatively play a more important role than edge ones, since the latter is only used to estimate the similarity among nodes. In our previous study (anonymous), we compared the two experiment settings: 1) fMRI as node and eye movement as edge feature; 2) eye movement as node and fMRI as edge. The former performs better than the latter, demonstrating that fMRI feature could be relatively more important for predicting cognitive scores. Such a mapping from MRI-based brain activity to cognitive scores has gained large supports from variety of previous studies (Finn & Bandettini, 2011; He et al., 2020). For eye movement, since we used two types of features: eye movement trajectory and pupil size. Our results in Table 1 showed that pupil size imposes a slightly more important effects on the prediction accuracy than trajectory. Eye movement trajectory might be passively guided by the content of a movie, such that it might be more subject to the compliance to the visual input, while pupil size variation might be more involuntary and has been demonstrated to serve as a marker of cognitive processes (Einhäuser, 2017).