List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Yoni Choukroun, Lior Golgher, Pablo Blinder, Lior Wolf
Abstract
The relationship between blood flow and neuronal activity is widely recognized, with blood flow frequently serving as a surrogate for neuronal activity in fMRI studies. At the microscopic level, neuronal activity has been shown to influence blood flow in nearby blood vessels. This study introduces the first predictive model that addresses this issue directly at the explicit neuronal population level. Using in vivo recordings in awake mice, we employ a novel spatiotemporal bimodal transformer architecture to infer current blood flow based on both historical blood flow and ongoing spontaneous neuronal activity. Our findings indicate that incorporating neuronal activity significantly enhances the model’s ability to predict blood flow values. Through analysis of the model’s behavior, we propose hypotheses regarding the largely unexplored nature of the hemodynamic response to neuronal activity.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43895-0_35
SharedIt: https://rdcu.be/dnwyO
Link to the code repository
Model code in Suplementary Material
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The paper uses a transformer-based model to predict blood flow in a cube of brain tissue, given current neural activity and previous blood flow. The idea is to leverage the close relationship between neural activity and blood flow, which is the basis, in a coarse way, for fMRI studies).
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is strongly grounded in the domain, the domain descriptions are thorough and clear, and the program leverages various specific properties of the system.
The paper appears well-grounded in the literature, and draws on multiple threads to develop a model (I say “appears” because I am not versed in this literature).
The whole conception and architecture are interesting, creative, and tailored to the task.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Just what the wider applicability of the method is unclear, because the method depends on neural readings from electrodes.
The highly interesting question of how to better characterize the relationship of neural activity to blood flow at a unit level, to better inform fMRI interpretation, is left to a single final paragraph.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
good
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The comments below are not intended to warrant point-by-point “correction”. I do believe that addressing them might increase the paper’s impact.
Misc comment: The Allen Institute hosts a variety of events at which this content would fit well.
Introduction: thank you for citing reviews rather than isolated papers for large topics.
Page 3 bottom: The flows in segments of blood vessel are presumbably highly correlated if they are from the same vessel. Are these segments connected, or is the structure such that the cube of tissue contains several unrelated vessels so that “same vessel” correlation can be ignored? Or could it be encoded?
Page 4: (top) inter-elements (typo) subsection Transformers: I totally did not follow this subsection Equn 2: what is the role of sqrt(d)? is q a row of Q? What is D (at end of subsection)?
Page 5: Subsection “vascular decoding”: Should this title start the next paragraph? Or should it say “vascular encoding and decoding”? The paragraph talks about encoding. Line 2: …by the neuronal encoder… (if that’s the intended meaning) I did not follow paargraphs 2 and 3 of this subsection. subsection Architecture and Training: why is d = 64? …encoder and decoder are each defined as … (if that’s the intended meaning) Fig 1: Aspects are unclear: what is Nx? is the left hand box the same module as the encoder part of the vascular track? If yes, could you show this module on the left as “detail of A”, and insert a smaller unit “A” as a part of the overall architecture schematic? For vascular, which part is the encoder and which is the decoder? Or maybe the schematic is exact - I have trouble matching the pieces up to the description on page 5.
Table 1: Are you able to use k-fold data splits to generate std devs for MSE and NRMSE? this gives valuable insight into result variability, and whether the differences are meaningful. Removing a decimal place in MSE would clarify the table. Again, are the decimal places justified by the variability envelopes?
Fig 2: I did not understand the import of (b). what is the take-away? Could the data be reordered to have clearer meaning? If you zoomed in on (a), would relevant fine structure show up, or is it a straight forward exponential? Does plotting log(phi) vs d_y show anything useful?
Page 8: My main regret in this paper is that the implications for standard fMRI conversion functions is restricted to just this one paragraph.
Conclusion: (ablation) -> (shown by ablation studies) this is a limitation of our imaging resources and not of the ML method. - Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Especially commendable is the careful embedding of the methods in the domain context and in the literature.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This paper proposes a novel bimodal transformer model for reconstructing the hemodynamic response function (HRF), which has not been extensively explored but may have a significant impact on neuronal activity. The model takes as inputs both the hemodynamic response and neuronal spikes, and incorporates spatial positions into the transformer architecture. To enhance the attention mechanism, the model introduces both neural encoding and vascular encoding into the self-attention and cross-attention layers. Experimental results demonstrate that the proposed method outperforms baseline models.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper puts forth hypotheses regarding the largely unexplored nature of the hemodynamic response to neural activity, which may be affected by ongoing spontaneous neuronal activity. The authors address this issue by integrating both blood flow and neuronal activity information and feeding them into a specialized transformer that can effectively model the relationship between the two. The proposed transformer takes into account both types of information, resulting in a more comprehensive model.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
The performance gain of the proposed method appears to be relatively small. Specifically, when compared to the linear model, the improvement in terms of the mean squared error (MSE) metric and normalized root-mean-square error (NRMSE) at 30.03 Hz is only 0.2 and 0.001, respectively.
-
The paper lacks an ablation study of the designed modules, such as Neuronal encoding and Vascular decoding. It is difficult to determine whether the improved performance is solely due to these modules or simply the result of an increase in the number of model parameters (i.e., model capacity).
-
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Although the author did not provide open-source code, the paper is well-written and contains clear descriptions of the methodology used. Therefore, I believe that this study is reproducible with the information provided in the article.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
-
Although this paper achieves state-of-the-art performance in accurately modeling blood flow, the improvement compared to previous methods is not significant. To address this issue, the authors may consider collecting more data to verify the bottleneck of the current performance.
-
The paper lacks an ablation study of the proposed neuronal encoding and vascular decoding modules. Specifically, it is not clear from eq.3 and eq.4 whether the transformer will perform worse or not without these modules’ modulation.
-
The experimental section could benefit from reorganization to improve the readability and clarity of the paper.
-
It would be helpful for readers to have a few sample visualizations of the dataset to facilitate a quick understanding of the paper.
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
-
The performance improvement of the proposed method in this article is not significant, particularly given the weak baseline used in the experiments.
-
Although the paper includes ablation experiments, the results do not fully demonstrate the source of the model’s performance gains.
-
The paper’s novelty lies in the introduction of neural activity into the hemodynamic response.
Based on the above critiques, I recommend a weak reject decision for this article.
-
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
A novel transformer is proposed for reconstructing the Hemodynamic Response Function. Experiments show that the proposed transformer works better than existing methods like linear or RNN method.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
A novel transformer architecture is constructed for reconstructing the Hemodynamic Response Function.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The motivations for the new transformer architecture are not discussed in detail. In the experiments, the comparisons might not be fair.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
No code is available during review.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The motivations for the new transformer architecture are not discussed in detail. For example, how are the functions $\psi()$ in (3) and (4) modeled? Why prefer such modeling to insert spatial encoding?
In the experiments, do the HRFT models have much more parameters than the baseline methods? If so, are the comparisons fair?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
It’s not clear that why the proposed transformer is better than existing ones. The baseline methods in the experiments are kind of weak. More specifically, they might have much fewer trainable parameters the proposed one.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The authors introduce a transformer based architecture to predict HRF from neural activity. The architecture is seemingly novel, and it does better than baselines for predicting HRF. However, some reviewers have noted that the improvement in prediction over a linear baseline seem small. Also, the authors do not provide intuition as to which modules of the architecture lead to the greatest performance gain (i.e. an ablation study). If these are explained better, the paper might have higher impact.
Author Feedback
We thank the AC and the reviewers for the helpful feedback. As discussed below, the concerns are to a large extent addressed in the manuscript. We respectfully disagree with the claim that the improvement in prediction over the linear baseline is not large. The reviewer cites the 30.03Hz results, where, as noted in the text (page 8), the prediction is very short-term, and therefore it is hard to show the full potential of any method. On the more interesting ranges, we have a 5% improvement for the 1st test set and a 18.5% improvement for the test set from the separate session. Following the review’s focus on the significance of the results, we have also performed a long-range (50 frames ~1500ms) analysis at 30 Hz. Performance. is better than for 10 frames at 30Hz in the paper, meaning that higher resolution helps. Very reassuringly, the gap in performance between the methods (in percent) is very similar to the 10 frames experiment. The significance of our contribution also follows from showing that a computational model that is tuned to predict results in a temporal filter is remarkably in agreement with the current knowledge on the timing of neuronal influences. This is a level of validation that is seldom presented for algorithmic work. Intuition and motivation: Our spatiotemporal modeling provides a) space-time-invariance (3D vs traditional 2D), b) cross-modality analysis that is different from the literature, c) natural integration of the 3D geometry via attention, and d) efficient geometric cross-analysis. No other transformer provides these properties. See the penultimate paragraph of the intro for a comparison to other multi-modal transformers. These contributions also allow better analysis and explainability as shown in Fig.2. Ablation: we ablate the neuro-vascular contribution, which is the main contribution of our work (HRFT-S). Also, reg. Eq. 3&4 that R2 asks about, we visualize the elements of the learned HRFT and demonstrate in Fig. 2(a,b) that the model learns distance filters that look reasonable, and in Fig. 2(c) that it fits what is known from the literature surprisingly well. Addressing additional concerns: R1: k-fold was not used since it would give overly optimistic results, due to the inclusion of nearby samples in both train and test. Instead, we provide 2test sets: (1) the last 7.5% of the 1st session, and (2) a separate session after a 30min break, which was held out during the development of the method. This is a more challenging and accurate depiction of the generalization capabilities. R1: Vessels structure: the data includes segments that are scattered across the tissue volume. Based on the vessel distribution it is unlikely to observe repeated or sibling vessels. R1: Sqrt(d) is for unit variance, q is a row of Q, D_# are the pairwise distance matrices (cf. Sec.3-3.1). Nx=N\times the module, d = 64 is a model hyperparameter (cf.Sec.3.4). R1: Fig. 5(b) shows the neuro-vascular impact on the prediction. Fine-grained structures are indeed present in Fig. 5(a). We will add in the Appendix. R2: Improvement over linear. Please see response to AC. R2: Ablation and Eq.3&4. Please see response to AC. R2: Manuscript. More visualizations of the dataset, attention maps, and learned filters will be provided in the Appendix. Code will be released upon acceptance. R2&R3: Model capacity: Our transformer is very shallow (d=64, N=3), with a single modulated initial embedding enabling both low capacity and generalization to arbitrary time and element ranges. RNN has a similar capacity and performs poorly. FC NN has 653K parameters, while ours is only 542K. FC lacks time invariance and interpretability and clearly overfits. R2&R3: Motivation. Please see response to AC. R3: \psi(). From the arch. Sec. “[it] is a FC NN with 250D hidden layers and GELU non-linearities, expanded to all the heads of the self-att. module”. We prefer to learn it with a generic network since the literature does not provide a known prior.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Post-rebuttal, the authors have adequately adressed concerns about ablation studies and visualizations of data. Therefore, the reviewers have agreed to improve their scores, raising this paper’s overall evaluation to an accept.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
strengths: the use of transformer to reconstruct the Hemodynamic Response Function; weaknesses: presentation; they authors may consider to pro ide a clear explanation of why the proposed transformer idea is superior, and to perhaps improve the baseline methods. lack of ablation study; lack of clear & detailed analysis of the empirical performance The rebuttal provides some info, but does not influence my decision here in a major way. Perhaps this is due to the rather rigid length limitation.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The rebuttal addressed the reviewers concerns, especially about the ablation study and visualization of the dataset. They also addressed the meta-reviewers questions satisfactorily. The paper makes a good contribution of using a transformer based architecture to predict hemodynamic response functions from neural activity. While the improvement in prediction over linear baselines was small, the architecture was deemed novel.