Authors

Shijie Zhao, Long Fang, Lin Wu, Yang Yang, Junwei Han

Abstract

Decoding brain states under different task conditions from functional mag-netic resonance imaging (tfMRI) data has attracted more and more atten-tions in neuroimaging studies. Although various methods have been devel-oped, existing methods do not fully consider the temporal dependencies be-tween adjacent fMRI data points which limits the model performance. In this paper, we propose a novel group deep bidirectional recurrent neural net-work (Group-DBRNN) model for decoding task sub-type states from individ-ual fMRI volume data points. Specifically, we employed the bidirectional re-current neural network layer to characterize the temporal dependency feature from both directions effectively. We further developed a multi-task interac-tion layer (MTIL) to effectively capture the latent temporal dependencies of brain sub-type states under different tasks. Besides, we modified the training strategy to train the classification model in group data fashion for the indi-vidual task. The basic idea is that relational tfMRI data may provide external information for brain decoding. The proposed Group-DBRNN model has been tested on the task fMRI datasets of HCP 900 subject’s release, and the average classification accuracy of 24 sub-type brain states is as high as 91.34%. The average seven-task classification accuracy is 95.55% which is significantly higher than other state-of-the-art methods. Extensive experi-mental results demonstrated the superiority of the proposed Group-DBRNN model in automatically learning the discriminative representation features and effectively distinguishing brain sub-type states across different task fMRI datasets.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16431-6_23

SharedIt: https://rdcu.be/cVD45

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper uses a single classification model to simultaneously complete multiple brain function network decoding tasks under different external stimuli. The authors proposed a multi-scale random segment preprocessing and multi-task interaction layer to realize a single model’s perception of varying brain functional states.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The authors propose Group Deep Bidirectional Recurrent Neural Network that decoding the functional brain for group-wise tasks is novel.
2. The proposed Multiple-scale Random Fragment Strategy and the multi-task interaction layer for decoding functional brain states are innovative.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The description in the methods section lacks some necessary explanations, e.g. The motivation for data preprocessing is unclear.
2. There is a lack of discussion on the importance of context information from both directions to decode brain network states.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The method is clear and reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1. The authors should add some explanations that describe the intrinsic relevance between the different external stimulus tasks and the properties of brain functional networks.
2. Why did the authors use bidirectional GRUs instead of unidirectional ones? It seems to be to learn temporal context information from both directions.
3. In the data preprocessing, the authors divided the fMRI into multiple time slices. The motivation for doing so is unclear, and the authors are advised to add corresponding explanations.
4. The multi-task interaction layer proposed in this manuscript is not introduced in detail. I suggest the authors to add corresponding diagrams to explain the mechanism of the multi-task interaction layer in detail.
5. The descriptions for data propressing are not clear. In the proposed data pre-pressing method, as shown in Fig2, the input T7-T1 refers to 7 different tasks? For the training phase, does the method need to choose 7 sub-tasks from different tasks? Or arbitrarily choose 7 subtasks?
6. In addition, the article lacks a description of the testing process. Does the proposed method need to enter 7 different tasks at one time during the testing phase?
7. The article also includes some errors such as “Fig. 1. Confusion matrix for classification accuracy of 24 events” should be fig 3. and “Fig. 2. Classification comparison chart of 24 events” should be Fig.4. It is recommended that the author make careful revisions.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This manuscript proposed a novel method to identify brain state. It shows great improvement over current SOTA methods. However, it still need to be carefully revised to make the method to be more clear. Therefore, I think this manuscript should be weakly accepted.
Number of papers in your stack

7
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

1.The authors proposed a group-wise brain function state classification network by capturing the temporal-spatial-wise context information. 2.The proposed multi-task interaction layer can help the classification network to learn the temporal-task-wise context information between multiple brain activation states. 3.The proposed method improves the performance of classifying the brain activation states of multiple subtasks.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

1.The proposed bidirectional GRU-based group-wise brain functional state classification method is innovative. 2.The proposed multi-task interaction strategy for mining temporal-task-wise context information is novel. 3.The experimental results show that the proposed method has good classification performance.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

1.The structure of the manuscript needs improvement. The methods section does not correspond closely to each module in Figure 2, e.g. lack of detailed description about MITL. 2.The article lacks a description of the motivation for the proposed method. The motivation of proposing random combinations of multi-sub-type tasking MRI sequences is unclear.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The method proposed in this paper has been validated on the HCP dataset and is highly reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

From the manuscript, the advantage of the group-wise functional brain states decoding method compared with the traditional method seems to be the improvement of the classification accuracy. I think the group-wise analysis method seems able to discover the internal relations between the sub-type functional brain states caused by different stimulus tasks. It is recommended that the authors add relevant experiments to analyze the ability of a group-wise manner to discover the intrinsic associations of subtype brain states.

The experiment results reported in table2 that some combinations of tasks can improve the accuracy. However, the possible reasons are unclear, and the author is advised to increase the discussion about the possible reasons for the improved classification performance caused by the group-wise manner.

What is the method of DSRNN in Table3? There is no citation in the manuscript. I suggest the authors add the introduction for the relevant methods and mark the cited references.

Also, the comparison methods in Experiment 3.3, such as SVM, MVPA, and SoftMAX, also lack reference citations.

The resolution of the pictures in the article is very low, such as Fig2. The color labels of different methods in the figure are not clear. the authors should improve the quality of the figures in the manuscript.

The experimental results in Table 3 require further analysis. The classification performance of the proposed method for class Emt and Lng is reduced. I suggest that the authors discuss the reasons for the decline in classification performance.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This paper proposes a novel group-wise brain function state classification network. Experiments demonstrate the excellent performance over current methods. It is important to this field.
Number of papers in your stack

4
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper
1. This manuscript proposed a two layers-bidirectional GRU network to extract time-series features and the inter-class association features.
2. The proposed method improves the classification performance of deep neural networks for functional brain activation states decoding.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. In the manuscript, the authors proposed a novel group-wise Bidirectional Recurrent Neural Network for analysis of the brain function sub-type states.
2. It is innovative that the authors implement the classification task of different subtypes of brain networks using a single model.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The introduction of this manuscript is very difficult to follow and the motivation of the study was not well articulated.
2. The article has some grammatical errors, such as: “Group-wise brain sub-type states convenient our method…”
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

It is easy to follow
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1. Introduction needs to be carefully re-written; it is very difficult to follow and the motivation of the study is not well articulated.
2. What are the intrinsic correlations between brain activity states under different external stimuli? Why is the group-wise brain network state decoding strategy helpful to explore this interconnectedness?
3. Which steps in data pre-processing are referred to as “Random combination” and “Signal extraction” in Fig.2? Authors are advised to describe in detail in the manuscript.
4. Meanwhile, the multi-task interaction layer mentioned in Figure 2 is not introduced in detail in the article. For the proposed method, The multi-task interaction is more important than GRU, and it is recommended that the author explains it in detail.
5. The authors should check and revise the entire manuscript for grammatical errors and typos.
6. How is the multiscale manifested in the Multi-scale Random Fragment Strategy (MRFS)? The work seems to just randomly splice different fMRI sequences and does not use multi-scale MRI sequences.
7. In the experiment, the authors only reported the classification accuracy of brain network decoding, which was improved by the group-wise strategy. Whether the authors add some visual classification results in the revised manuscript? So that the readers can intuitively discover the interpretability of group-wise brain network decoding.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

In the manuscript, the authors proposed a novel group-wise Bidirectional Recurrent Neural Network for analysis of the brain function sub-type states. It is important to this field.
Number of papers in your stack

1
What is the ranking of this paper in your review stack?

4
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

o In this paper, a single classification model is utilized for multiple brain function network decoding tasks via different external stimuli. A multi-scale random segment preprocessing and multi-task interaction layer are expolored for the perception of varying brain functional states. The proposed method using a novel group-wise Bidirectional Recurrent Neural Network to analyze the brain function sub-type states is novel. The implementation of the classification task of different subtypes of brain networks using a single model is novel too. However, this paper still has the following issues: 1) the unclear description of methodology; 2) failure to discuss the importance of context information from both directions to decode brain network states. 3) Unorganized structure of the manuscript for further improvement; 4) lacks of descrbing the motivation for the proposed method.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

1

Author Feedback

We thank area chair and all reviewers’ constructive comments and appreciation of our work’s potential to effectively decode Task Sub-type States with Group Deep Bidirectional Recurrent Neural Network. All the constructive suggestions will be adopted and the writing will be checked carefully in the final version.

back to top

Decoding Task Sub-type States with Group Deep Bidirectional Recurrent Neural Network