Authors

Chongyue Zhao, Liang Zhan, Paul M. Thompson, Heng Huang

Abstract

The transformation and transmission of brain stimuli reflect the dynamical brain activity in space and time. Compared with functional magnetic resonance imaging (fMRI), magneto- or electroencephalography (M/EEG) fast couples to the neural activity through generated magnetic fields. However, the MEG signal is inhomogeneous throughout the whole brain, which is affected by the signal-to-noise ratio, the sensors’ location and distance. Current non-invasive neuroimaging modalities such as fMRI and M/EEG excel high resolution in space or time but not in both. To solve the main limitations of current technique for brain activity recording, we propose a novel recurrent memory optimization approach to predict the internal behavioral states in space and time. The proposed method uses Optimal Polynomial Projections to capture the long temporal history with robust online compression. The training process takes the pairs of fMRI and MEG data as inputs and predicts the recurrent brain states through the Siamese network. In the testing process, the framework only uses fMRI data to generate the corresponding neural response in space and time. The experimental results with Human connectome project (HCP) show that the predicted signal could reflect the neural activity with high spatial resolution as fMRI and high temporal resolution as MEG signal. The experimental results demonstrate for the first time that the proposed method is able to predict the brain response in both milliseconds and millimeters using only fMRI signal.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16431-6_32

SharedIt: https://rdcu.be/cVD5e

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper proposed a method to predict high resolution brain response in space and time using fMRI data.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The paper proposed a novel framework based on Siamese network to predict high temporal and spatial resolution brain activities using fMRI data. This framework is interesting since current brain observation techniques have limitations in either spatial or temporal resolution. The current study adopts Siamese network to establish the relationship between cross-modal network prediction and original brain activity time series, and uses Polynomial projection operators to overcome the problem of gradient vanishing when long time series are taken into consideration.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Some technical details are missing. It is not clear how the fMRI graph and MEG graph in Figure 1 were generated; It is also not clear what is the role of the connectivity matrix in model training. As consequences, the theoretical soundness and the reproducibility of the study could be contaminated, and it is a bit confusing when matching Figure 1 with its descriptions in the main text.
2. It seems the model was trained at ROI-level as the time series within a brain atlas ROI were averaged before input to the model. Considering the functional heterogeneity of the cortex, this average operation could largely degenerate the specificity of brain activities. I have a sense that this average operation is related to high dimensionality of the fMRI and MEG data, but how would this operation affect the prediction performance of the proposed model? The authors are encouraged to justify this issue explicitly if my understanding is correct.
3. The authors provided qualitative and quantitative evaluations of how well high-resolution fMRI signals can be predicted. Besides Table 1, the authors are encouraged to provided more details about the performance difference among different brain ROIs. If possible, they are also encouraged to provide fMRI time series prediction for a single voxel rather than averaging over ROI.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The reproducibility of the paper can be improved after the author completing the missing technical details.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

Please see the comments on the main weakness of the paper.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The study is interesting. The framework is novel. However, some technical details are missing. The experimental results are relatively not sufficient.
Number of papers in your stack

4
What is the ranking of this paper in your review stack?

2
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper
1. In this paper, a recurrent memory optimization method for predicting the behavioral state of spatio-temporal brain activity is proposed. The proposed method uses Optimal Polynomial Projections to capture the long temporal history with robust online compression, and predicted the recurrent brain states through a Siamese network based on fMRI data and MEG data in the training phase.
2. During the testing phase, using only fMRI data to predict the spatiotemporal corresponding neural response of each voxel within the brain, millisecond and millimeter-scale brain responses were predicted.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. This paper proposes a general framework for discretizing time points and projecting them onto a polynomial basis by a novel recurrent memory optimization method, which connects the mapping between brain regions and time points.
2. This method enables high-resolution temporal signal prediction for each voxel of the brain using only fMRI data.
3. The method can simultaneously consider the spatial and temporal aspects of neural activity.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

This paper compares the proposed method with only two baselines, LSTM and GRU-D, with few comparisons.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Reasonable
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

This paper can further improve the comparative experiment.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This paper has a rich theoretical derivation process, and has a more detailed explanation for the model proposed in this paper, which has strong persuasive force.
Number of papers in your stack

4
What is the ranking of this paper in your review stack?

2
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #4

Please describe the contribution of the paper

This work proposed a novel framework to predict the brain response with high spatial and temporal resolution only based on fMRI data. In the training stage, the proposed method uses Optimal Polynomial Projections with robust online compression to take the fMRI and MEG data as inputs and uses a Siamese network to predict the brain state. In the testing stage, only the fMRI data is used.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

a) Modeling the brain dynamics is an interesting and important problem. b) The proposed method can predict the brain state with both high spatial/temporal resolution. c) A feasible approach for dealing with gradient vanishing in modeling MEG data with RNN.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

a) It’s better to include more experiments or discussions about how behavioral representations evolve with the proposed model. I think this is the major point of the paper, however, the experiments only compared the spatial and temporal patterns from fMRI and MEG, respectively. b) Only 3 brain networks are shown ion Section 3.1. It’s better to show more networks in Fig.2. Meanwhile, the quantitative measurements should be included for a comprehensive evaluation. c) In Section 3.2, MSE may not truly reflect the similarity between the prediction and ground truth. It’s better to include more measurements such as PCC. In addition, it is not clear the similarity is averaged over all subjects? Or the time series is firstly averaged and then similarity is computed.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

I think this work is reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

a) The details of how LSTM and GRU-D are implemented for comparison are missing. At least, it should be included in the supplemental material. b) It’s better to include more visualizations in Fig. 2 and Fig. 3. For example, the time series in Fig. 3 is from one of visual networks. What about the other ROIs? c) I think the prediction performance varies from region to region. In which ROI, the proposed model performs worse? It’s interesting to exploring the performance degeneration. d) Does the number of parcels have an effect on the model’s performance? e) What is the limitations of the proposed model? f) Correct the grammar errors.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This work proposed a novel approach in modeling the brain dynamics. The results seems promising, and it can be potentially applied for super resolution of fMRI data. The experiments and discussions may not be suffient.
Number of papers in your stack

1
What is the ranking of this paper in your review stack?

7
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposed a novel framework to predict brain response with high spatial and temporal resolution based on fMRI and MEG, which is recognized by all three reviewers. The key strength is to adopt a Siamese network to establish the relationship between cross-modal network prediction and original brain activity time series. Although there are several weaknesses such as unclearness of some technical details, potential ROI signal heterogeneity, lack of more comparisons, etc, the meta-reviewer agrees with all three reviewers that this paper has enough technical novelty as well as promising applications. The authors should consider integrating all reviewer comments into the final paper.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

1

Author Feedback

We thank you for all the reviews and meta-reviews. The professional comments help us to improve the research. We will refer to the suggestions from the reviewers and meta-reviewer to revise the manuscript.

back to top

Predicting Spatio-Temporal Human Brain Response Using fMRI