List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Chongyue Zhao, Liang Zhan, Paul M. Thompson, Heng Huang
Abstract
Understanding the intrinsic patterns of human brain is important to make inferences about the mind and brain-behavior association. Electrophysiological methods(i.e. MEG/EEG) provide direct measures of neural activity without the effect of vascular confounds. The blood oxygenated level-dependent (BOLD) signal of functional MRI (fMRI) reveals the spatial and temporal brain activity across different brain regions. However, it is unclear how to associate the high temporal resolution Electrophysiological measures with high spatial resolution fMRI signals. Here, we present a novel interpretable model for coupling the structure and function activity of brain based on heterogeneous contrastive graph representation.The proposed method is able to link manifest variables of the brain (i.e. MEG, MRI, fMRI and behavior performance) and quantify the intrinsic coupling strength of different modal signals. The proposed method learns the heterogeneous node and graph representations by contrasting the structural and temporal views through the mind to multimodal brain data. The first experiment with 1200 subjects from Human connectome Project (HCP) shows that the proposed method outperforms the existing approaches in predicting individual gender and enabling the location of the importance of brain regions with sex difference. The second experiment associates the structure and temporal views between the low-level sensory regions and high-level cognitive ones. The experimental results demonstrate that the dependence of structural and temporal views varied spatially through different modal variants. The proposed method enables the heterogeneous biomarkers explanation for different brain measurements.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16431-6_34
SharedIt: https://rdcu.be/cVD5g
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposed a model to coupling the structure and function activity of brain on graph representation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
A lot of fancy features in the model: dynamic, heterogeneous, explainable, causal model…
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Too much details are missing for the whole model. e.g. rare hyper parameters were provided; how edge of graph was defined? no time-window used for dynamic FC? why distillation is necessary? And for sex classification result, it’s better to compare to other published methods, since a lot methods have been proposed for this problem.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Too much details are missing for the whole model, so it will be hard to reproduce this method only from paper.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- Much more details should be provided. e.g. hyper parameters, how edge of graph was defined? Was time-window used for dynamic FC? why distillation is necessary?
- I don’t think the model can find “cause” instead of correlation. And granger causality is hard to applied in resting fmri with large number of node.
- 22 major ROIs for fmri is not enough.
- sex classification is not that good compared to current FC based result.
- where the “structural connectivity” come from?
- ablation study is necessary for this model.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Too much details are missing for the whole model. e.g. rare hyper parameters were provided; how edge of graph was defined? no time-window used for dynamic FC? why distillation is necessary? And for sex classification result, it’s better to compare to other published methods, since a lot methods have been proposed for this problem with better results.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
This paper used GCN for contrastive learning of structural and functional multimodal data and used the model results to analyze the strength of the structure-function coupling patterns between functional connectivity, structural connectivity and behavioral performance.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
In general, this paper is comprehensive and technical sound that will surely inspire future research along similar lines. The main message is brought across fully supported by the presented results.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
However, I still have a few major concerns before possible publishment. (1) In the introduction, the authors mention several reasons why the previous method is not applicable, and in the article authors should emphasize how the method proposed in this paper solves these problems and why it has advantages over the previous methods. (2) Authors should add the parameters of the methods. It would be better to add some necessary arguments for Equations to make them easier to understand. The overall schematic illustration needs to be clear and easy to understand and highlight the innovative points of the model. (3) The first experiment used the causal explanation model to obtain the regions that play an important role in the classification, and the authors should further show and analyze them. (4) In the second experiment, three different FC data were used to assess whether the analytical approach of the role of SC is the innovative approach of this paper. Please add some details of this experimental approach or relevant literature.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
This paper used GCN for contrastive learning of structural and functional multimodal data and used the model results to analyze the strength of the structure-function coupling patterns between functional connectivity, structural connectivity and behavioral performance.
In general, this paper is comprehensive and technical sound that will surely inspire future research along similar lines. The main message is brought across fully supported by the presented results. However, I still have a few major concerns before possible publishment. (1) In the introduction, the authors mention several reasons why the previous method is not applicable, and in the article authors should emphasize how the method proposed in this paper solves these problems and why it has advantages over the previous methods. (2) Authors should add the parameters of the methods. It would be better to add some necessary arguments for Equations to make them easier to understand. The overall schematic illustration needs to be clear and easy to understand and highlight the innovative points of the model. (3) The first experiment used the causal explanation model to obtain the regions that play an important role in the classification, and the authors should further show and analyze them. (4) In the second experiment, three different FC data were used to assess whether the analytical approach of the role of SC is the innovative approach of this paper. Please add some details of this experimental approach or relevant literature.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Interesting topic and novel idea.
- Number of papers in your stack
3
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
The paper mainly proposed a novel heterogeneous contrast subgraphs representation learning based method to exploit the coupling of structural and functional connectivity from different brain modality.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
the fusion manner of multimodal imaging in the paper is novel, effective and interpretable, which shows great potential and extensibility in many related clinical tasks.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
there are not enough validation tasks for the proposed framework, e.g. to use external dataset or different combinations of imaging modality.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The reproducibility is good for the clear elaboration in method detail.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Several minor limitations are as flollowing:
- the authors should testify the fusion of other modalities in HCP dataset for the sex discrimination to demonstrate its generalization of the proposed framework, such as fmri and dti, or smri and fmri.
- there is no colorbar in Fig 4, and it is not clear how to compare them.
- the used parcellation atlas is not detailed enough for fMRI.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
the fusion manner of multimodal imaging is impressive
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
The idea behind the paper is good and attractive, although some minor details were not mentioned in the paper.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposed a heterogeneous contrast subgraphs representation learning based method based on structural and functional multimodal data in order to explore the coupling of structural and functional connectivity and how it related to gender and behavioral performance. The key strength of this paper is to integrate dynamic, heterogeneous, explainable, causal model etc. which are hot topics for brain network modeling into a whole framework, and has potential and extensibility in related clinical applications. Although it is an interesting paper, the reviewers have some concerns and confusions. The meta-reviewer therefore invites the authors to provide the rebuttal to clarify these major concerns: 1. The whole model details and parameter setting are missing. 2. Hard to evaluate the model performance since there are no comparisons with other methods or ablation studies. 3. Whether the proposed model can represent the “cause” relationship. Please also refer to the detailed comments from each reviewer.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
4
Author Feedback
We are grateful for the valuable comments from all reviewers. We address all reviewers’ major critiques as follows. (1) We give details of the whole framework in Section 2. Firstly, the proposed method uses dynamical neural graph encoder framework to associate the spatial and temporal patterns of multimodal brain measurements. Then, we introduce the contrastive graph learning method to maximize mutual information of different views. Finally, we discuss the interpretable causal explanations for the proposed method on graph. The causal representation is only used in the distillation process to explain the learned graph representation. (2) We test the performance of the proposed method with sex classification task using HCP data. We compare the proposed method with several state-of-the-art methods in Table 1. Comparing with the baseline methods, the proposed method could achieve the highest sex classification performance of 85.2%. (3) We use the causal model to explain the multimodal graph representation approach based on granger causality. The explanation part is divided into two steps, the distillation process and explainer training process. We discuss the details in Section 2.3 and supplementary material. Specifically, we use Equation 4 to get the causal contribution of each edge. After that, we could sort the top-K most relevant edges for model explanation. In Fig. 2, We show the top-K brain regions that contribute to the sex classification. We will refer to the suggestion from the meta-reviewer to ‘revise the manuscript’ by adding more details about the mathematical notations to our final paper.
To R1: Thanks for the constructive comments and advice. We will revise the mathematical notions to make it clear in our final paper. (1) As discussed in Section 2.1, we use dynamical graph model to represent the whole time-series of fMRI and MEG. We do not use time window. We define some parameters such as the edge of graph on page 3, Section 2. We use the definition “structural connectivity” to define the high spatial information from complementary modalities (i.e. fMRI and MRI). (2) We use causal model to explain the learned graph model. As shown in Fig. 1, our whole framework could be divided into two steps: the graph representation learning and the causal explanation learning. The causality is only used in the explanation process to highlight the importance of the brain ROIs and encourage the reasonable node selection process. (3) We use HCP MMP 1.0 for parcellation [7]. We will revise our final paper to give the sex classification results with 180 areas per hemisphere ROIs [7]. (4) For sex classification, we compare the proposed method with SOTA methods in Table 1. We also show the top K important brain regions in Fig. 2. We will give more details of the model interpretation experiment in our final paper. (5) For ablation study, we show the single model and multimodal performance in Table 1. The single model does not use contrastive learning method. We also show the single modal and multimodal results in Fig. 3 and Fig. 4. We will add more details in our final paper.
To R2: Thanks for constructive suggestions and comments. (1) We highlight the model details at the beginning of our rebuttal. We discuss the advantages of the proposed method on Page 2. We give details of the causal explanation model in the supplementary material. We will revise the mathematical part in our final version to help readers understand the paper. (2) We give some details in section 3.2 about how to use NeuroSynth meta-analysis on the same topic in [15] to assess the structural-decoupling index. We will give more details in the final paper.
To R3: Thanks for constructive suggestions and comments. We show sex classification result with single modality (fMRI) or multimodality (fMRI and MEG) in Table 1. We will give more details of model explanation steps and experimental setting in our final paper.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have adequately and reasonably addressed the major concerns of all reviewers, especially Reviewer 1, and is convincing to the meta-reviewer. The key strength of this paper is to integrate dynamic, heterogeneous, explainable, causal model etc. which are hot topics for brain network modeling into a whole framework, and has potential and extensibility in related clinical applications. Therefore, I would suggest acceptance of this paper and revise the final version of this paper to integrate all useful comments.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The idea of using a graph embedding within a neural network is novel for the purpose of combining heterogeneous multiple input modalities from brain imaging for brain activity modeling. The authors mention causal explanation, however, the experimental results do not conclusive prove this phenomenon. Nevertheless, the contribution of this paper is original and is supported by the author’s response after the rebuttal phase.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This is a borderline paper with strong discrapency in the reviews. This discrapncy is well addressed by the authros in their rebuttal. Given the clear explanation and the majority of strong positive rates by the reviewers I would consider accepting this paper.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5