Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews Back to top

List of Papers By topics Author List

Paper Info

Reviews

Meta-review

Author Feedback

Post-Rebuttal Meta-reviews

Authors

Tingting Dan, Hongmin Cai, Zhuobin Huang, Paul Laurienti, Won Hwa Kim, Guorong Wu

Abstract

The functional neural imaging technology sheds new light on characterizing the neural activity at each brain region and the information exchange from region to region. However, it is challenging to reverse-engineer the working mechanism of brain function and behavior through the evolving functional neuroimages. In this work, we conceptualize that the ensemble of evolving neuronal synapses forms a dynamic system of functional connectivity, where the system behavior (aka. brain state) of such a reaction-diffusion process can be formulated by a set of trainable PDEs (partial differential equations). To that end, we first introduce a PDE-based reaction-diffusion model (RDM) to jointly characterize the propagation of brain states throughout the functional brain network as well as the non-linear interac-tion between brain state and neural activity manifested in the BOLD (blood-oxygen-level-dependent) signals. Next, we translate the diffusion and reaction processes formulated in the PDE into a graph neural network, where the driving force is to establish the mapping from the evolution of brain states to the known cognitive tasks. By doing so, the layer-by-layer learning scenario allows us to not only fine-tune the RDM model for predicting cognitive tasks but also explain how brain functions support cognitive status with a great neuroscience insight. We have evaluated our proof-of-concept approach on both simulated data and functional neuroimages from HCP (Human Connectome Project) database. Since our neuro-RDM combines the power of deep learning and insight of dynamic systems, our method has significantly improved recognition performance in terms of accuracy and robustness to the non-neuronal noise, compared with the conventional deep learning models.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16452-1_35

SharedIt: https://rdcu.be/cVVpQ

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper introduces a new method for extracting brain states from fMRI via a dynamical systems model. The authors blend model-based and data-driven strategies by using neural networks to implicitly learn the parameters of a reaction-diffusion equation. The “reaction” component is implemented using an ANN, and the “diffusion” component is mapped onto a graph convolutional network for joint training. The authors evaluate their model on simulated data and task-based fMRI from the Human Connectome Project. The results demonstrate improved recognition accuracy compared to recurrent architectures.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Interesting premise of inferring functional brain networks based on steady-state solutions of a reaction-diffusion equation.
- The authors map each component of the reaction-diffusion model onto a neural network that can be optimized using standard gradient-based techniques.
- The approach blends structured model-based assumptions with the representational power of deep learning. The authors demonstrate that this structure provides robustness over RNNs.
- The model achieves higher state-recognition accuracy than two RNN methods and a PDE solver.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- While the authors weave a nice story to motivate their work, I find the claims about neurobiology to be overstated. For example, the statement “the ensemble of evolving neuronal synapses forms a dynamic system of functional connectivity (abstract)” is a simplification of the complex neural and hemodynamic attributes leading to the BOLD response. Likewise, the statement “uncovering hidden brain states…becomes the gateway to understanding flexible and adaptive human cognition (page 2)” is an exaggeration at best. Also, the statement “we develop the machine intelligence of system-level explainability into a deep learning model…to yield new underpinnings of biological processes (page 2)” is unsupported by the experimental results, which consist of a few predictive analyses.
- The is little acknowledgement of existing methods to analyze dynamic functional interactions or extract brain states. There is also no acknowledgement of existing work on interpretable AI or methods that blend model-based and data-driven techniques. This is a large oversight in an already popular field.
- Several details of the proposed method are unclear. First, how is the graph w_ij generated? Is it subject-specific or fixed across the cohort? Second, the diffusion process does not require an attention model, so why is it implemented? Third, how are the “ground truth” cognitive states defined for training and evaluation? Fourth, why is the BOLD time series truncated? I would expect a dynamical model to accommodate different acquisition lengths. Fifth, what does the statement “the training data is mixed with the test/retest fMRI data” in the evaluation section mean? If the data is truly mixed, then the results are optimistic due to data leakage.
- There is no ablation study to quantify the impact of the different modeling components (e.g., choice of W, attention vs. no attention, neural network sizes, etc.).
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Reproducibility statement is acceptable.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The “weaknesses” section details my major concerns and constructive suggestions.

Here are some additional minor comments:
- The use of “functional neural imaging technology” in the abstract is awkward. I would suggest rephrasing.
- The characterization of fMRI as allowing us to characterize coupling mechanisms of brain functions on to of the structural connectomes is also overstated (page 1). FMRI quantifies hemodynamic changes, which are not necessarily linked to the structural connectome.
- The motivational comparison on pages 2-3 seems unfair. The one-particle trajectory is generated from a PDE, so naturally embedding a priori knowledge of this process into the model will perform better. The key question is whether such a model would perform well if the data were NOT generated from a PDE.
- It is unclear to me why an ANN is an appropriate model for the “Reaction Process”. I would think neuronal firing patterns are more complex than a linear model with sigmoid activation.
- The similarity between test and retest in Fig. 5 may indicate that the model is learning a “mean” representation for the attention weights, rather than capitalize on subject-level differences. I would suggest the authors explore this phenomenon.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Overall, I appreciate the methodological novelty in this work. I particularly like how the authors map elements of the reaction diffusion equation onto different neural network architectures. However, my enthusiasm is dampened by the overstated claims, lack of an ablation study, and several unclear details.
Number of papers in your stack

4
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

5
[Post rebuttal] Please justify your decision

While I appreciate the author’s response, I do not believe it alleviates my original concerns.

Review #3

Please describe the contribution of the paper

The article rewrites the reaction-diffusion model originally expressed in partial differential equations in terms of a deep neural network. The new network is used to study changes in brain states from BOLD fMRIs.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The idea of re-expressing differential equation models in a neural network via component decomposition or equivalent counterparts
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- It is not clear what is gained with the new network compared to the original model of differential equations
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Very high. As aforementioned, with a little more description in the generation of the model forward, I think one should not have problems to replicate the experiments and reproduce the results
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- If one already has the model of differential equations, what is the benefit of re-expressing it in a neural network (other than for the beauty of doing it)?
- How much error is there between the trained NN and the original PDE model?
- although from figure two I think I can intuit how the diffusion-reaction model of equation 1 has been translated into the neural network, but perhaps a little more step-by-step explanation explaining the methodology to take the first model to the second would be convenient. to be able to generalize the idea.
- I suppose that the generic idea of decomposing differential equation models into fundamental elements with counterparts in neural networks is in principle generalizable to other models. Although I understand that this goes beyond the purpose of the article that is presented, may I ask the authors to what extent they believe that this idea can be developed? Is it possible to affirm that any model of partial differential equations can have an equivalent counterpart in neural networks?
- Add the PDE model to figure 1
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The flow of the article is excellent, the mathematical formalization is correct and elegant, the rationale described in the first paragraph of section 2 is outstanding, the images are very well taken care of and clearly contribute to the message of the article, the two experiments presented are well executed in my opinion and perhaps with a little more description in the generation of the model forward, I think one should not have problems to replicate the experiments and reproduce the results. In general, a very good article, from which I have learned a lot, and my congratulations to the authors.
Number of papers in your stack

3
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

Not Answered
[Post rebuttal] Please justify your decision

Not Answered

Review #4

Please describe the contribution of the paper

This work examines the problem of modeling regional fMRI BOLD signals as a dynamical system governed by reaction-diffusion process which is formulated as a set of trainable PDEs. They introduce a reaction-diffusion model (RDM) which characterizes evolution of brain states across the connectome and captures the interplay between brain states and neural activity. This in turn is formulated as a graph neural network optimization which is guided by the cognitive task being performed.

They evaluate their framework on simulated and real HCP data against baselines on the basis of the ability to identify the latent generating states and identify the cognitive task being performed respectively
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Novelty in the formulation is the salient point of this work. The utilization of GNNs as Neural ODE solvers for modeling the dynamics of functional connectivity is an interesting perspective in the field and a departure from more popular neural architectures such as RNNs, LSTMs or transformers. In my opinion, this line of analysis may provide interesting new insights in the field.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Architectural Description: There is very little information provided in terms of implementation details, eg. number of GNN layers, width of the embeddings, type of graph convolutions used, non-linearities etc. They also do not clearly mention the paradigm for training the models (epochs, learning rate, optimizers etc). This would make it very difficult to adopt their framework for any future applications.
2. Evaluation is performed on a single dataset split into train/test/validation as opposed to a cross validated, which may not provide a reasonable estimate of the robustness of the improvements. Additionally, since a single dataset split is used, it is unclear how the distribution of the accuracies in Fig. 3 and 4 are generated .
3. Attentional selection: a) A contribution of the work is in introducing graph attentions into the Neuro-RDM framework for discovery of new links. However, there is no experimental comparison that establishes whether having this additional component (increased parameterization) is necessary, for example via an ablation study b) The authors average patterns learned by the attentional framework as a proxy for studying replicability of the patterns. Since evaluation was performed only on a single data split, this does not provide a good indication for whether these patterns would be consistently replicated for a different subset of the population
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Upon reading the paper and the checklist, I found several inconsistencies.

For example, I found no description of the parameter settings, architecture, range of parameter values, sensitivity to parameters, memory footprint or compute software in the paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1. I would encourage the authors to improve the clarity of Fig.2 to clearly indicate the input BOLD signals and outputs, to aid readers in parsing the methodological description
2. Several error bars in Fig 3 appear to be cut off. Perhaps the range on the y axis can be changed to prevent this
3. Clarifications for the following would be helpful:
a) Are the regional connectivities thresholded to generate W? If so, what is the threshold? In practice, how would this threshold affect the optimization of the Neuro-RDM and generalization.

b) What statistical test is used to compare the performance of the methods pairwise in Fig. 4?

c) “we truncate the long time course of BOLD signals (usually includes more than eight functional tasks) into a set of segments where each segment primarily covers one functional task”

From my understanding, this implies that the time series was broken up into segments of a fixed length. How was this chosen, and how would the choice of this length impact performance?

c) Definition of a latent brain state: For the purposes of this paper, it seems like the intrinsic brain states and cognitive tasks are used synonymously. Therefore it is unclear whether this method could be applied for a study of dynamic functional connectivity in a more broad context, for example for resting state-fMRI data. In this space, several models such as the sliding window or dynamic conditional correlation aim to address the problem of identifying the “latent brain states” from the regional time series. It would be great if the authors could discuss if and how their method could be applicable beyond the specific task evoked fMRI paradigm examined here.

d) Scalability: From my understanding of the experiments, the Neuro-RDM was tested on Functional Connectomes generated by grouping regions into 8 subsystems such as the DMN, CEN etc. A natural question to ask here is whether the method would scale to using finer parcellation schemes such as the Yale functional atlas, which is also the scale at which several GNNs developed for functional connectivity work with
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

3
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

While the methodology is interesting and novel, the major weaknesses of the paper are in the evaluation of their contributions and the design of experiments. The complexity of their model is not sufficiently justified. Additionally, several key details for implementation are missing.
Number of papers in your stack

7
What is the ranking of this paper in your review stack?

6
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

Not Answered
[Post rebuttal] Please justify your decision

Not Answered

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposed a computational framework for modeling brain functional states by combining the reaction-diffusion model and graph convolutional networks. They tested the effectiveness of the proposed model based on both simulated data and task-based fMRI from HCP, and demonstrated superior recognition accuracy compared to existing methods. The key strength of this paper is to integrate both model-based and data-driven methods for learning the parameters of a reaction-diffusion equation via neural networks, which is novel. Although it is an interesting paper with potential significance, the meta-reviewer as well as the reviewers have some concerns and confusions, and invite the authors to provide the rebuttal to clarify these major concerns: 1. Several claims about neurobiology, e.g., “the ensemble of evolving neuronal synapses forms a dynamic system of functional connectivity (abstract)”, “uncovering hidden brain states…becomes the gateway to understanding flexible and adaptive human cognition (page 2)”, etc. (See reviewer 1) seem to be overstated. 2. No acknowledgement of existing methods for analyzing dynamic functional interactions or extract brain states, and interpretable AI or methods which integrate model-based and data-driven techniques. 3. Detailed descriptions of the method and implementation is not provided. 4. No ablation study. 5. What is gained with the new network compared to the original model of differential equations. 6. Evaluation based on only one dataset might be unconvincing. Please also refer to the detailed comments from each reviewer.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

6

Author Feedback

We thank for the constructive comments. We will incorporate all feedback in the final version and provide the code to the public to aid reproducibility.

Clarification for meta-reviews:

We will rephrase the neurobiology statements.

Our work is more concerned with integrating model-based and data-driven techniques in both ways. We will include recent(one-way) works in the final version.

Regarding the method details, we briefly answer each question from R1&R4 as follows. (1) w_ij is the pre-calculated connectivity degree which is subject-specific. (2) We add an attention component on graph nodes(not on links). So it does not affect graph diffusion. (3) The “ground truth” is the pre-defined task schedule in the neuroscience experiment. (4) Since we aim to recognize functional tasks, we truncate the whole fMRI scan into several task-specific sequences. Another reason is to reduce the learning complexity by keeping the trajectory length on a manageable scale. (5) Since the task schedules of test/retest data are different, using mixed test/retest data in the training stage is to push the model to learn task-specific features rather than scheduling variance. (6) We deploy a one-layer GNN with four PDE solvers. The width of the embeddings is the same as the input. We use the sigmoid function and Adam optimizer, with a learning rate of 0.01 and epoch 200 (on Intel(R) Core(TM) i7-9700k CPU).

Regarding the ablation study, the recognition accuracies of w/ vs. w/o attention are 0.691 and 0.676. The recognition accuracies of network layer of 1, 2, 4, 8, 16 are 0.691, 0.707, 0.716, 0.728, 0.723, respectively. We will add it in the final version.

Regarding the gain over original PDE, we would argue that the observed improvement is due to the way we turn empirical PDEs into trainable PDEs, which can be learned from the experimental biological data. Furthermore, we link PDE to GNN in an integrated math framework, thus gaining the model explainability.

Regarding testing on more datasets, we are extending our model to the resting state and aiming for disease diagnosis. Due to the page limit, we only can make comments in the final version.

R1 • We appreciate the comments on the toy example in Fig. 1. Considering the human brain is a complex system with a lot of unknowns, we don’t have the intention to model the brain function using PDEs nor lay such a strong assumption on the method development. Instead, we are interested in using the PDE-like behaviors to formulate the machine learning into a well-posed problem. The reaction-diffusion model is widely used in neuroscience field at the neuron level. In this work, we land the physiological insight at the meso-scale to the macro-scale brain network science. • In this pilot work, we use ANN to simplify the complex reaction process. But we completely agree with R1 more sophisticated deep model could further improve our method.

R3 • The predicted mean error of the original PDE model is 0.36(our model is 0.097). We will add it in Fig. 1.

R4 • The major contribution of our work is PDE-based GNN backbone. We demonstrate the performance in a brain state recognition scenario. We are working on extending our model to then resting stage in classifying normal and disease connectome which is the driving force of supervised learning. • We used 5-fold cross-validation to evaluate our model, the accuracies of each fold(testing set) are 0.691, 0.690, 0.691, 0.691, 0.691, respectively, where the attention maps of each fold is replicable. We will correct it and refine Fig. 3, 4, 5 in the final version. • We specifically evaluate the contribution of w/ and w/o the attention component, 0.691 vs. 0.676. • We are actually using the whole-brain fine parcellation of Yale atlas. We will refine Fig. 2 and 3 in the final version. • We have not applied any threshold on the connectivity value (but we discard the negative values as most of fMRI studies). We use paired t-test. We will make it clear.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors have adequately and reasonably addressed the major concerns of all reviewers, and is relatively convincing to the meta-reviewer. The key strength of this paper is to integrate both model-based and data-driven methods for learning the parameters of a reaction-diffusion equation via neural networks, which is novel. Although there are still some unclearness of details, I think it is novel enough for acceptance of this paper. I would strongly suggest the authors revise the final version of this paper to integrate all useful comments.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

4

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The reaction diffusion model has been extensively used for modeling dynamical systems in other fields as well as brain imaging (see Hurdal et al. Lefevre et al.). However the approach of using these models for understanding of cognitive tasks from imaging (fMRI) is novel. The experimental results have also shown a test-retest validation. All reviewers commented on the novelty of the contribution. I recommend the paper be accepted.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

10

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper presents a computational framework to extract brain functional states by integrating PDE-based reaction-diffusion model and a graph neural network and blending the model-based and data-driven approaches. This paper receives split reviews and major concerns focus on the lack of recognition of existing work, and lack of model implementation details and blation study. The authors were able to address most of the questions raised by reviewers and meta-reviewer in the feedback. I would recommend acceptance of this paper, though I would strongly suggest the authors carefully revise their paper to fully address all review comments in their final version.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

4

back to top

Neuro-RDM: An Explainable Neural Network Landscape of Reaction-Diffusion Model for Cognitive Task Recognition