List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Lin Zhao, Zihao Wu, Haixing Dai, Zhengliang Liu, Tuo Zhang, Dajiang Zhu, Tianming Liu
Abstract
BOLD fMRI has been an established tool for studying the human brain’s functional organization. Considering the high dimensionality of fMRI data, various computational techniques have been developed to perform the dimension reduction such as independent component analysis (ICA) or sparse dictionary learning (SDL). These methods decompose the fMRI as compact functional brain networks, and then build the correspondence of those brain networks across individuals by viewing the brain networks as one-hot vectors and performing their matching. However, these one-hot vectors do not encode the regularity and variability of different brains, and thus cannot effectively represent the functional brain activities in different brains and at different time points. To bridge the gaps, in this paper, we propose a novel unsupervised embedding framework based on Transformer to encode the brain function in a compact, stereotyped and comparable latent space where the brain activities are represented as dense embedding vectors. The framework is evaluated on the publicly available Human Connectome Project (HCP) task based fMRI dataset. The experiment on brain state prediction downstream task indicates the effectiveness and generalizability of the learned embeddings. We also explore the interpretability of the embedding vectors and achieve promising result. In general, our approach provides novel insights on representing regularity and variability of human brain function in a general, comparable, and stereotyped latent space.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16431-6_35
SharedIt: https://rdcu.be/cVD6R
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #4
- Please describe the contribution of the paper
They adoptted a transformer-based framework that encoded the human brain function measured by fMRI data into vectors of latent layer in transformer.Then they evaluated the proposed framework in brain state prediction downstream task, and found that the embedding vectors are relevant to the response of task stimulus.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The work provides an approach on representing regularity and variability of human brain function in a latent space.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The authors stated that the embeding vectors represent regularity and variability of human brain function, maybe they should show the variability part for different brains and time points.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The reproducibility of the paper is good.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
If the embedding size was 64, all fMRI signals of whole brain were represented the 64 vectors, or the fMRI signal of one voxel were represented the 64 vectors? or others? When predicting ADHD, how to divide the training, validation and test data, I mean that if author use some brains as training data, and the others as test data, or any other manner?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The work provides an approach on representing regularity and variability of human brain function in a latent space of Transformer, and uses the embeding vectors to identify the brain state, experimental results demonstrate it effective.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
7
- [Post rebuttal] Please justify your decision
The author answered my questions very well and addressed my doubts.
Review #5
- Please describe the contribution of the paper
In this submission, the authors propose a transformer-based learning model for analyzing human brain functions, specifically, they focus on learning a canonical embedding for better predicting human brain states through fMRI inputs. They address the issue of regularity and variability of different instances, and choose transformer as the backbone to solve this issue.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
A good implementation of introducing cutting-edge computer vision and natural language processing technology to medical image analysis. This submission provides an option of analyzing medical time series data via transformer.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
One major drawback of the submission is the experimental design. If I remember correctly, HCP contains 7 tasks, but only the results of two are given in the paper. Regarding to measure the prediction performance, more statistical metrics are expected to use, but I only see accuracy and pcc. Besides VAE-based model, I see no other comparison methods. Such an experimental design is not convincing to me. From the perspective of methodology, the proposed method did not well address the issue they mentioned in Introduction. How does the proposed method encode the regularity and variability of different brains is not clearly answered.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
No code is provided
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
(1) Transformer shows a more flexible and expressive ability compared with CNN, so I’d like to see its application in medical image analysis. This work utilizes transformer in a basic way: switch CNN modules with transformer, which is not impressive. But applying transformer to fMRI is a good topic. LSTM-based or RNN-based methods show a good prediction results on HCP tasks, the authors could explore them and add some of them as comparison methods. (2) Test on all tasks, evaluate with not only accuracy but also precision, recall, f1 score. Statistical results could be tricky, so always provide as much information as possible. (3) Fig2 is redundant, provide some results from comparison methods.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I give this score mainly based on the soundness of the methodology and the experimental design.
- Number of papers in your stack
6
- What is the ranking of this paper in your review stack?
5
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
3
- [Post rebuttal] Please justify your decision
Thanks for the rebuttal. The total number of experiments is not my main concern. But the number of involved tasks in HCP is somehow a cared point for me. As the authors can only select tasks that their method has good performance on, it might be biased for performance evaluation. It’s always a good move to include all results and highlight the good ones. Also, the statistical evaluation is not sufficient and trustful enough for me. I didnt find explanations On general, I dont think the current version of submission and the rebuttal can convince me to give a higher score, so I remain my previous score of this submission.
Review #6
- Please describe the contribution of the paper
DL method is proposed based on transformers for building a compact representation of human brain functions from fMRI. 3D volume of fMRI data can be embedded as a dense vector which profiles the functional brain activities at the corresponding time point. Regularity and variability of brain functions at different time points and across individual brains can be measured by the distance in the embedding space. The method is evaluated on the Human Connectome Project task fMRI dataset for brain state prediction. A comparison with various baselines is reported, demonstrating the increase in performance of the proposed method, with a neglectable computational cost increase as compared to a standard auto-encoder.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is well written, easy to follow and the method and task are well motivated. The method is simple and sound. The results demonstrate the benefit of the proposed approach over a simple auto-encoder, for a limited computational cost increase.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The evaluation is limited to a single dataset/task, which may however be sufficient for a conference paper.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Method evaluated on a publicly available dataset. The code is not shared and minor implementation details are missing for reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Some more comments are provided in the following.
-
Have the authors tried to train simultaneously the embedding and the specific task (despite the stated motivation of not being task-specific) ?
-
In 3.2. Since the methods is in two stages (train the embedding, then train the brain state prediction), it is not clear what some parameters refer to in the implementation details. Details should be given for both trainings. Also, is there no early-stopping on the validation set?
-
Figure 1, keep t either columns or rows in both the 2D signal matrixx and the learned embedding.
-
In 3.3. “But the performance gain is significant”. It reads as if a statistical test was performed but I think it is not.
-
In 3.4. Interesting analysis of the correlation of individual digits with individual stimuli. It would also be good to report the specificity of these digits, as some may just respond to any stimulus, while others may be more specific? I would mention the limitations of the “interpretation”, it does not interpret the internal behavior of the transformer, it is just a correlation analysis between extracted features and stimuli.
-
“general, comparable, and stereotyped space” is repeated in introduction. In 2.3. “We fix the weights of pre-trained model” -> the pre-trained model In 3.2. “the size of two FC” -> of the two
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I think this simple application of transformer to fMRI is valuable to the MICCAI community. It is well written, an interesting approach with good results.
- Number of papers in your stack
7
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
The authors answered my comments. It does not modify my evaluation, with minor improvements in the revision.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The idea of using transformers for embedding fMRI signals spatiotemporally is interesting. A concern brought up by the reviewers is that the experimental results are limited. For e.g. the authors show results on the emotion and language tasks from the HCP dataset instead of the 7 tasks that are present. Additionally, the table 1 only lists accuracy measures and the figure 2 shows correlation measures without detailed metrics. Thus it is harder to gauge the performance of the method. The authors make a statement “It is observed that all baselines have an accuracy over 0.6, suggesting the effectiveness of deep learning for high-dimensional data embedding.”. This is hardly showing the effectiveness of deep learning methods. Such claims should be toned down.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
9
Author Feedback
We appreciate the reviewers’ acknowledgement of our contribution and their guidance and constructive feedback. The point-to-point responses are as follows: #R4: We thank the reviewer for many positive comments. [Variability] In Fig. 2, the variability has been demonstrated. We will show more visualizations in the supplemental materials. [Embedding Size] All voxels in a fMRI volume of a time point are represented as a vector with 64 digits. [Dataset Division] We divided different subjects into different datasets. We hope our responses resolve the reviewer’s concerns.
#R5: We would like to clarify a few key points that might confused the reviewer. [HCP Tasks] Including two tasks in fMRI studies is typically sufficient for a MICCAI conference paper, due to space limit. For example, Zhang et al., 2019, Li et al., 2020 included only one task; Zhao et al., 2021 included Emotion and Motor tasks. [Comparison Methods] Our task is to represent the fMRI data in an embedding space, which have not been done in any previous studies. Predicting brain state is the downstream tasks of learned embedding for evaluation, not the major contribution and task. We have included several methods that can be extended for our task. DSRAE model is based on RNN/LSTM; DRVAE is based on VAE and LSTM model. Our experiments have already demonstrated the superiority over these LSTM-based methods. [Regularity and Variability] The fMRI data from brains of different subjects are embedded into the same latent space, and the vector of this space encodes the regularity and variability. [Evaluation Metrics] We will include the mentioned metrics for the camera-ready if the paper is accepted. The current improvement of accuracy was evaluated by two-sample one-tailed un-pair-wise t-test (p<0.025, corrected) and was significant. We will clearly mention it. [Redundant Fig.2] It is for the interpretation of learned embedding, which is an important part of our framework. In addition, demonstrating the temporal pattern is a common practice for fMRI studies. [Reproducibility] We will release the code. Please check the Reproducibility Response in our submission. We hope that our response resolved the reviewer’s concerns.
Zhang et al. “Identify hierarchical structures from task-based fMRI data via hybrid spatiotemporal neural architecture search net”. MICCAI 2019. Li et al. “Neural Architecture Search for Optimization of Spatial-Temporal Brain Network Decomposition”. MICCAI 2020. Zhao et al. “Exploring the Functional Difference of Gyri/Sulci via Hierarchical Interpretable Autoencoder”. MICCAI 2021.
#R6: We thank the reviewer for many positive comments. [Evaluation Dataset/Task] We will include more dataset and tasks in the future journal version. But two tasks are typically sufficient for MICCAI submission, due to space limit. [Simultaneous Train] We have not tried that because we are not working on a specific task. But we agree it deserves a try for the follow-up works. [Training Details] We will add these details and release our code. Our implementation provides the option for the early stopping on validation dataset. In the experiments of current submission, the testing was performed on model with the lowest loss on validation dataset. [Statistical Test] We performed two-sample one-tailed un-pair-wise t-test (p<0.025, corrected) on the performance gain and will clearly mention it. [Interpretation] We will add the specificity of digits in the revision. Interpretation here refers to the dimension of embedding space, not the internal of transformer. We will extend this work to interpret the internal behavior of the transformer in future journal version. [Other Minor Issues] We will revise them. Thank you for these constructive comments to help us improve the paper.
Meta-Reviewer: We thank the reviewer for constructive comments. Please refer our response to #R5 for experimental results. We will tone down these claims in the camera ready if it is accepted.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors addressed all the concerns in the rebuttal.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
8
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The rebuttal addressed all concern very convincingly, and I would recommend it for acceptance.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
7
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
I agree with the reviewers who think that this work on a transformer based latent space embedding of brain function will be of interest to the MICCAI community. From my perspective, the most important remaining argument against this paper would be R5’s suspicion that results might have been cherry-picked. I’m recommending acceptance under the assumption that the limitation to two fMRI tasks was simply due to limited time and space, and that these are the only tasks on which the authors tried their method before submission. I believe this point merits explicit (and of course honest) clarification in the final version.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
4