Authors

Chuhang Zheng, Wei Shao, Daoqiang Zhang, Qi Zhu

Abstract

Emotions are closely related to many mental and cognitive diseases, such as depression, mania, Parkinson’s Disease, etc, and the recognition of emotion plays an important role in diagnosis of these diseases, which is mostly limited to the patient’s self-description. Because emotion is always unstable, the objective quantitative methods are urgently needed for more accurate recognition of emotion, which can help improve the diagnosis performance for emotion related brain disease. Existing studies have shown that EEG and facial expressions are highly correlated, and combining EEG with facial expressions can better depict emotion-related information. However, most of the existing multi-modal emotion recognition studies cannot combine multiple modalities properly, and ignore the temporal variability of channel connectivity in EEG. In this paper, we propose a spatial-temporal feature extraction framework for multi-modal emotion recognition by constructing prior-driven Dynamic Functional Connectivity Networks (DFCNs). First, we consider each electrode as a node to construct the original dynamic brain networks. Second, we calculate the correlation between EEG and facial expression through cross attention, as a prior knowledge of dynamic brain networks, and embedded to obtain the final DFCNs representation with prior knowledge. Then, we design a spatial-temporal feature extraction network by stacking multiple residual blocks based on 3D convolutions, and non-local attention is introduced to capture the global information at the temporal level. Finally, we adopt the features from fully connected layer for classification. Experimental results on the DEAP dataset demonstrate the effectiveness of the proposed method.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43993-3_38

SharedIt: https://rdcu.be/dnwND

Link to the code repository

N/A

Link to the dataset(s)

http://www.eecs.qmul.ac.uk/mmv/datasets/deap/

Reviews

Review #3

Please describe the contribution of the paper

This paper proposed a novel framework for multi-modal emotion recognition based on prior-driven Dynamic Functional Connectivity Networks (DFCNs) and the extraction of emotion-related spatial-temporal information. The experimental results on the DEAP dataset demonstrated the effectiveness of the proposed method. Overall, this paper is easy to follow and well-written.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. Construction of dynamic functional connectivity networks.
2. Introducing prior knowledge into DFCNs.
3. Exploration of global information.
4. Extensive experiments.
5. This paper is well-written.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Evaluated only on one dataset with 32 subjects.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

I think it can be easily reproduced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
1. To model the long-distance dependencies, Transformer might be a better choice. Can authors provide more justification for using non-local blocks with 3D-CNN?
2. The dataset used for validation only has 32 subjects. However, I am unsure how many parameters the proposed framework has. Is there overfitting problem?
3. Can the proposed framework generalize to other tasks?
4. The discussion of the limitation is missing.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

A very good paper with high-quality.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

In this paper, authors proposed a spatial-temporal feature extraction framework for multi-modal emotion recognition by constructing priordriven Dynamic Functional Connectivity Networks (DFCNs). This is very important for emotion recognition.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

1 The method is interesting for emotion recognition field. 2 The netword structure is fit to the problem.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The dataset is a bit small.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

there need a bit more detailed information for reproducibility of the paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

In this paper, authors proposed a novel framework for emotion recognition. The experiments demonstrated the proposed methods. However, I only have a few suggestions. 1) more subjects. If the result could be validated on a larger dataset, it would be better. 2) more details about the method.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The methods and topic is qutie interesting.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #1

Please describe the contribution of the paper

The authors proposed a spatial-temporal feature extraction framework, which used EEG and facial expression to achieve emotion recognition. The main contributions are the prior-driven dynamic functional connectivity networks and a STFENet for spatial-temporal feature extraction.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The main contributions are the prior-driven dynamic functional connectivity networks and a STFENet for spatial-temporal feature extraction.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. In abstract and Introduction, the authors emphasized their contributions of DFCN, but the DFCN method using sliding windows is not novel. They should pay more attention on their contribution on how to generate a prior-driven DFCN by using EEG and facial expression.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
1. In Fig. 1, what is the ‘spatial and temporal’ features extracted by STFENet? It is unclear to readers.
2. In the 2nd paragraph of Results, the authors said ‘there comes a significant performance degradation’. This statement without a statistical test is not rigorous. The authors should apply statistics on the performance metrics in order to demonstrate statistically the superiority of the proposed framework. Table 1 and 2.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
1. Abbreviations need to be given when they first appear, for example, STEENet, LSTM and DFCN.
2. The comparing methods shown in Table 1 are not from recently published studies.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

4
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. It is suggested to explain what does the bold text in Table 1 and 2 represent?
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposed a novel framework to extract emotion-related spatial-temporal information and for multi-modal emotion recognition based on prior-driven Dynamic Functional Connectivity Networks (DFCNs). The key strength includes that it is well-organized and clear written, with sufficient experiments and interpretation, as recognized by a majority of the reviewers. Although there are several weaknesses such as limited sample of subjects, unclearness of some technical details or novelties, etc, the meta-reviewer agrees with a majority of the reviewers that this paper has considerable contribution to the EEG and emotion recognition field. The authors should consider integrating all reviewer comments into the final paper.

Author Feedback

N/A

back to top

Prior-driven Dynamic Brain Networks for Multi-Modal Emotion Recognition