Authors

Chongyue Zhao, Liang Zhan, Paul M. Thompson, Heng Huang

Abstract

Brain large-scale dynamics is constrained by the heterogeneity of intrinsic anatomical substrate. Little is known how the spatio-temporal dynamics adapt for the heterogeneous structural connectivity (SC). Modern neuroimaging modalities make it possible to study the intrinsic brain activity at the scale of seconds to minutes. Diffusion magnetic resonance imaging (dMRI) and functional MRI reveals the large-scale SC across different brain regions. Electrophysiological methods (i.e. MEG/EEG) provide direct measures of neural activity and exhibits complex neurobiological temporal dynamics which could not be solved by fMRI. However, most of existing multimodal analytical methods collapse the brain measurements either in space or time domain and fail to capture the spatio-temporal circuit dynamics. In this paper, we propose a novel spatio-temporal graph Transformer model to integrate the structural and functional connectivity in both spatial and temporal domain. The proposed method learns the heterogeneous node and graph representation via contrastive learning and multi-head attention based graph Transformer using multimodal brain data (i.e. fMRI, MRI, MEG and behavior performance). The proposed contrastive graph Transformer representation model incorporates the heterogeneity map constrained by T1-to-T2-weighted (T1w/T2w) to improve the model fit to structure-function interactions. The experimental results with multimodal resting state brain measurements demonstrate the proposed method could highlight the local properties of large-scale brain spatio-temporal dynamics and capture the dependence strength between functional connectivity and behaviors. In summary, the proposed method enables the complex brain dynamics explanation for different modal variants.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16431-6_33

SharedIt: https://rdcu.be/cVD5f

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #2

Please describe the contribution of the paper

This paper proposed a novel spatio-temporal graph transformer model to explore the dependency of functional connectivity on anatomical structure.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The proposed study incorporates the areal heterogeneity map (T1-to-T2-weighted MRI) to improve the model fit to structure-functional interactions. The authors also introduced a novel graph transformer pooling layer to learn the global representation of the entire graph. Meta-analysis was adopted to evaluate the proposed model to explore the behavioral relevance of different brain regions.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Some critical technical details are missing. For example, how was the T1-to-T2 heterogeneity map incorporated in the model? How do the fMRI and Meg graphs relate to functional correlation matrices in Figure 1? Where does the functional gradients in section 3.2 come from? Considering the space limit, part of 2.2 can be removed to Supplementary materials.
2. What do the results tell us about the dependency of functional connectivity on anatomical structure, or structural connectivity?
3. Model training details are missing, which may degenerate the reproducibility of the study.
4. The authors showed the connectivity patterns for one hemisphere. Why not both hemispheres as there exist both strong structural and functional connections between some symmetric brain regions?
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The reproducibility of the study can be improved by providing more technical details.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

Please see comments on the main weakness of the paper.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The study is interesting. The framework is novel. However, some technical details are missing. The justification about experimental results is relatively weak.
Number of papers in your stack

4
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

The paper proposed a novel spatio-temporal graph Transformer model to integrate the structural and functional connectivity in both spatial and temporal domain, and exemplified its application in HCP data.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. adopt graph transformer with multi-head attention
2. use contrastive learning to integrate multimodal imaging details
3. evaluation with meta-analysis
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Some explanation about the concepts are not detailed provided in the paper, such as heterogeneity, structure-function coupling index.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The reproducibility is acceptable.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1. some details should be added to make the paper easier to understand.
2. the preprocessing of SC is not mentioned.
3. if the findings in the paper could be validated in another dataset, the method may be more reliable.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The research topic is very important to the research field, and the proposed method provides a new avenue to deepen the understanding of brain dynamical organization.
Number of papers in your stack

4
What is the ranking of this paper in your review stack?

2
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #4

Please describe the contribution of the paper

This work proposed a novel spatiotemporal graph Transformer model to learn the heterogeneous node and graph representation via contrastive learning based on multimodal brain data (i.e. fMRI, MRI, MEG and behavior performance). The experimental results reveal the significance of regional heterogeneity in modeling structure-function relationship of brain dynamical organization.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

a) Incorporating the heterogeneity map constrained by T1-to-T2-weighted (T1w/T2w) to improve the model fit to structure-function interactions is novel. b) Contrastive learning is used to associate the different data modalities. c) The experimental results of behavior related global gradient are very interesting.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The major weakness of the paper is the writing quality, which significantly confuses audience. Here are some examples: a) G_i = (V, E_i) is used to represent the heterogeneous graph representation. According to the illustration, i=1,…,N represents the brain ROIs, which means for each ROI, there is a graph G_i. However, I think G_i represents the graph for different modality in the first paragraph of Methods section. b) In the same paragraph, what are the multivariate values X_A and X_B? Are they features of nodes in graph? Similarly, what are the Y_A and Y_B? c) In Section 2.1, N was used to represent the number of neurons. What is the meaning of neurons here? Is it corresponding to the brain ROIs mentioned above? d) In Section 2.1, it is mentioned that fG, L^j_G, L^y_G are graph encoder networks. What is graph encoder network? How is it implemented? Z^+_t is introduced to represent the value after discrete operation. What is the discrete operation? I think Z^t+_t is not used in Eq.2 and afterwards. e) How can we reach Eq.4 from Eq.3? The graph encoder shows up here again. Is it same as the graph encoder network in Section 2.1? f) … Overall, it is hard to figure out the input of each module, data flow, implementation details of the whole framework.
Please rate the clarity and organization of this paper

Poor
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

I think the paper is not reproducible based on the current submission because the method section is vague and too many details are missing.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

a) The paper writting should be greatly improved. b) Ablation studies should be included to evalute each part of the proposed method.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

4
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The method is novel but the writting quality is very poor.
Number of papers in your stack

5
What is the ranking of this paper in your review stack?

3
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposed a spatio-temporal graph Transformer model to integrate structural and functional connectivity in the spatio-temporal domain and to learn the heterogeneous node and graph representation via contrastive learning based on multimodal brain data. They tested the effectiveness of the model in modeling structure-function relationship of brain dynamical organization in HCP data. The key strength of this paper is to adopt contrastive learning to associate different data modalities for graph transformer with multi-head attention, which is novel for brain connectivity analysis as well as its association with behavior. The reviewers have some concerns and confusions: 1. The writing quality should be improved to provide clear details such as the input of each module, data flow, implementation details of the whole framework. 2. Hard to evaluate the performance of this model since there are no ablation studies or independent testing data. 3. The model justification is limited and unconvincing. The authors need to improve the paper in their final version.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

3

Author Feedback

We are grateful for the valuable comments from all reviewers and meta-reviewer. We will refer to the suggestions to revise the manuscript.

back to top

Revealing Continuous Brain Dynamical Organization with Multimodal Graph Transformer