List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Zhongyu Huang, Changde Du, Yingheng Wang, Huiguang He
Abstract
Brain signal-based affective computing has recently drawn considerable attention due to its potential widespread applications. Most existing efforts exploit emotion similarities or brain region similarities to learn emotion representations. However, the relationships between emotions and brain regions are not explicitly incorporated into the representation learning process. Consequently, the learned representations may not be informative enough to benefit downstream tasks, e.g., emotion decoding. In this work, we propose a novel neural decoding framework, Graph Emotion Decoding (GED), which integrates the relationships between emotions and brain regions via a bipartite graph structure into the neural decoding process. Further analysis shows that exploiting such relationships helps learn better representations, verifying the rationality and effectiveness of GED. Comprehensive experiments on visually evoked emotion datasets demonstrate the superiority of our model.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16452-1_38
SharedIt: https://rdcu.be/cVVpT
Link to the code repository
https://github.com/zhongyu1998/GED
Link to the dataset(s)
https://doi.org/10.6084/m9.figshare.11988351
Reviews
Review #1
- Please describe the contribution of the paper
This paper introduces a novel method, Graph Emotion Decoding (GED), to decode emotions by integrating emotion scores of the video stimulus and brain responses from brain regions. By stacking layers of the GED, the model uses the relationships of the brain regions and neighboring emotions to decode. The model showed reasonable performance by comparing with other models and gradually improved by staking the layer of the model.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This study showed a novel method integrating emotional responses and brain responses of fMRI. Even using the GED, the model can decode using emotion and using the connections of the neighboring emotions. Additionally, this paper showed reasonable performance by interpreting and comparing with the other state-of-the-art model. GED uses more integrated information, making the performance better and even gradually improving performance by stacking layers.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Although this paper showed interesting results and a novel method, the number of fMRI subjects is small (n=5), which may cause reproducibility issues when applied to other datasets. Also, since this paper didn’t use the emotion scores from the individuals that performed fMRI sessions, which means that the emotion decoding they did was the emotions that people mostly feel from videos, not the emotions of the individual itself This paper doesn’t have a conclusion section. It would be worth showing the conclusions by summarizing the results and the interpretation to emphasize the main point of this study. Lastly, it would be better for this paper to show the relevance of this method by interpreting more of the embeddings from emotions and the brain regions, such as interpreting which of the ROIs showed meaningful results to interpret the corresponding emotions.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
As mentioned in the weakness section, this paper only used five subject data from fMRI, it would be better to use more data to show the reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
It would be better if they could provide the conclusions by summarizing their works and their interpretation. Also, to convince the model interpretation of which emotion scores and the regions in the brain are important for decoding emotions would be needed. Additionally, since the number of the subject is small it would be better to perform an analysis on the other dataset to show the reproducibility.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The data they used for the fMRI was obtained from a small number of participants to represent the objective emotions. The results may be affected by the subjective emotions of the participants. The reviewer recommends performing an analysis using more fMRI data to show the novelty and its reproducibility. Additionally, as this method uses embedding of the emotions and brain regions, it would be better to interpret the results from the brain regions. Such as explaining the meaningful regions for each corresponding emotion.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
The authors proposed a neural network to predict scores of individual emotions for presented videos from fMRI recordings. The network architecture is simple but the results are better than some of the state-of-the-arts methods. Overall this paper is interesting and clearly written.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors proposed a relatively simple network architecture to achieve a relatively difficult problem, and proved the effectiveness of their method.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The experimental design is not clearly stated in article. Also authors should compare their results to some of the newest models.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
acceptable.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Overall this paper is interesting and the authors sort to increase the accuracy of a difficult prediction problem using a simple way. The proposed emotion-brain region architecture is novel, and very straightforward to understand, i.e. the biological interpretation of the model is very clear and understandable. However I do have some questions and concerns about the experimental design and the results. 1, it is not clear what value were authors trying to predict, emotional score for audience like me is quite vague, and subjective, authors should clearly indicate how the emotional values were measured. 2, how the input vector was generated was not clear, emotional stimuli (from vedios) and fMRI signals are both time series, did authors segment the time series into pieces? How? and what is the length for each time window? 3, is the output (the emotional score) a vector or a scaler? If it was a vector, then how the accuray was measured? If it was a scaler, what does it represent in a time series window? 4, the prediction is based on a fMRI voxel signals but not a functional brain network or graph. However two methods authors compared (GCN and BrainNetCNN) are both graph based, how authors ran the two models using the same input is unknown. More information is needed. 5, I don’t think the models authors compared are most state-of-the-arts methods, authors should compare their methods with more recent methods.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The prediction results are better than some state-of-the-arts methods, but the network architecture are much simpler than the methods they are comparing.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #4
- Please describe the contribution of the paper
The Neural Decoding framework is proposed to find the association between Emotion and brain regions via a bipartite graph structure.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- It seems an interesting idea to perform embedding propagation.
- t-SNE Visualization generates interesting visualization, similar emotions tend to be closer.
- It shows significant improvement above the state-of-the-art brain-network-based models.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
I don’t get a clear understanding of how the evaluation is done. If there are five subjects, are we doing leave one subject out validation? What are 10 folds here? How this 10 fold is formed? Or one question did BrainGNN apply a similar evaluation setup? or it’s run by the authors with this idea.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
It seems reproducible. Data is available.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Please refer weakness
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The idea seems interesting to exploit relationships between emotions and brain regions.
- The embedding propagation approach is interesting.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Somewhat Confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
4
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposed a Graph Emotion Decoding (GED) model to decode emotions by adopting both emotion scores of video stimulus and brain responses as well as their relationships via a bipartite graph structure. They tested the superiority of the model compared to other methods and by staking the model layers. The key strength of this paper is to integrate both emotional responses and brain responses of fMRI for decoding, which is novel to some extent (although it is not the first paper to use fMRI for emotion decoding). Although reviewers acknowledged the novelty and impressive performance of this paper, the meta-reviewer has some significant concerns and confusions, which are also raised by the reviewers. Therefore, the meta-reviewer would like to invite the authors to provide the rebuttal to clarify these major concerns: 1. Limited sample size. There are only 5 subjects which is quite small and may cause reproducibility issues when applied to other datasets. 2. The emotion scores used in this paper are from a range of raters using self-report scales, not the emotion scores from the 5 subjects. 3. The paper format is incomplete. There is no discussion/conclusion section, which makes the readers cannot well digest the major findings of this paper. 4. More model and result interpretations are needed to help better understanding. 5. More advanced and latest models are needed in the comparison study. 6. The evaluation step is unclear. How to perform 10-fold cross-validation based on five subjects? Please also refer to the detailed comments from each reviewer.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
7
Author Feedback
A1. Reviewer 4 and AC are confused about the evaluation protocol. We need to clarify that we perform the cross-validation (cv) on each subject’s data (a within-subject design), but do not perform it between subjects. Specifically, we perform the 10-fold cv on 2181 pairs of {fMRI recording, emotion score} for each subject. In each trial of 10-fold cv, we use 1963 random pairs (9 folds) as the training set to train a model and then use the remaining 218 pairs (1 fold) as the testing set to evaluate the model. Such a cross-validation manner is used by the original dataset [8]. We follow this evaluation step and apply it to BrainGNN and other baselines.
A2. Reviewer 1 and AC question the small number of subjects. Since the fMRI data acquisition is demanding and expensive, the number of subjects is usually small when each subject is required to perform a large number of tasks. For example, the large-scale fMRI dataset BOLD5000 (Chang et al., 2019) has only 4 subjects, which demonstrates a small number of subjects is common in fMRI experiments. For the dataset used in this work, there are extensive tasks (watching 2181 unique videos) and each corresponds to 34 rich emotion categories, leading to a relatively small number of subjects. In addition, as explained in A1, this work does not use a between-subjects design but the within-subject design, which is independent of the number of subjects, thus it can be easily reproduced on new datasets.
A3. Reviewers 1, 2, and AC raise concerns about emotion scores. The measurement of emotion scores is described in Sec. 3.1, “Human raters used binarized scores to report whether these emotions exist in each video, and the final emotion category scores were averaged among these raters”. In general, if the number of emotion categories is large, then emotions are labeled by multiple raters. Since there are large numbers of tasks and emotion categories in this work, suppose the subjects are required to report 34 rich emotion categories after collecting each fMRI response, it will significantly increase their burden and affect the subsequent fMRI acquisition. Thus, [8] uses a wide range of raters to report emotion scores, which would also be more objective and less susceptible to the subjective influence of a single subject.
A4. For Reviewer 2, the whole of each short video stimulus corresponds to an emotion score vector, whose each element (a scalar) represents an emotion category rather than time. We finally output a predicted emotion score vector, whose each element is a scalar calculated by Eq. 2, and the performance is measured by Mean Absolute Error (Table 1) and Pearson Correlation (Fig. 6 in Appendix). As for input graphs, we construct an emotion-brain bipartite graph (Sec. 2.1) as the input graph of our model. In contrast, the brain-network-based baselines use fMRI to calculate the functional connectivity between ROIs and construct functional brain networks as their input graphs, consistent with [1, 14].
A5. Reviewer 2 and AC suggest comparing the latest model. In fact, there are few works related to fMRI-based emotion recognition, and we have compared the most recent baseline BrainGNN [14] proposed in 2021. To the best of our knowledge, there are not any other open-source state-of-the-art methods to be compared.
A6. Reviewer 1 and AC suggest providing more model interpretations. In fact, the constructed emotion-brain bipartite graph reflects which regions are important for decoding emotions. From the construction results, fear and joy are connected with the amygdala and precuneus in the graph, consistent with the conclusions in the literature [12, 17]. In addition, we also provide t-SNE visualization and interpretation for emotion embeddings in Sec. 3.5. We will add more interpretations in the revision according to your valuable suggestion.
A7. For Reviewer 1 and AC, considering that an extra page will be allowed in the final version, we will add the conclusion section in the revision.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have addressed the major concerns of meta-reviewer and reviewers. The key strength of this paper is to integrate both emotional responses and brain responses of fMRI for decoding, which is novel to some extent (although it is not the first paper to use fMRI for emotion decoding). The major weakness is the very small number of subjects and lack of comprehensive comparisons with other methods. I would suggest the authors significantly improve the new version of this paper by integrating all useful comments.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
9
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
I can not be convinced that only 5 subjects (even each subject has intensive data) can achieve reliable emotion decoding result. The CV is conducted within each subject instead of between subjects. I believe the result is due to overfitting.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
17
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The major weakness is the limited sample size. The rebuttal explained that each sample has tons of fmri tasks. I would suggest to create another dataset that picking several major tasks and largely increase the sample size, as an extended experiment. Considering the strength in the proposed methodology, I would suggest acceptance.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5