List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Seong-A Park, Hyung-Chul Lee, Chul-Woo Jung, Hyun-Lim Yang
Abstract
Attention mechanisms are widely used to dramatically improve deep learning model performance in various fields. However, their general ability to improve the performance of physiological signal deep learning model is immature. In this study, we experimentally analyze four attention mechanisms (e.g., squeeze-and-excitation, non-local, convolutional block attention module, and multi-head self-attention) and three convolutional neural network (CNN) architectures (e.g., VGG, ResNet, and Inception) for two representative physiological signal prediction tasks: the classification for predicting hypotension and the regression for predicting cardiac output (CO). We evaluated multiple combinations for performance and convergence of physiological signal deep learning model. Accordingly, the CNN models with the spatial attention mechanism showed the best performance in the classification problem, whereas the channel attention mechanism achieved the lowest error in the regression problem. Moreover, the performance and convergence of the CNN models with attention mechanisms were better than stand-alone self-attention models in both problems. Hence, we verified that convolutional operation and attention mechanisms are complementary and provide faster convergence time, despite the stand-alone self-attention models requiring fewer parameters.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16431-6_58
SharedIt: https://rdcu.be/cVD7e
Link to the code repository
N/A
Link to the dataset(s)
Reviews
Review #1
- Please describe the contribution of the paper
The paper investigates several self attention mechanisms in a range of CNN and multi-head self attention based architectures in application to ECG signals combined with other physiological data. Experiments are performed on a classification test for predicting hypotension and a regression task to predict intraoperative cardiac output. The authors identify best performing self attention mechanisms in each task and provide hypotheses as to why these models work the best.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper seeks to compare the performance of several self-attention mechanisms in CNN architectures in 1D physiological (mostly ECG) signals. While extensive experimentation has been done in the computer vision literature regarding such architectures, the benefits and drawbacks are less well established for 1D signals. As a researcher in EEG and ECG I have personally searched the literature for papers in the physiological signal domain addressing similar architecture design related questions and found a dearth of resources. I believe that work presented here addresses questions that may be common in the 1D physiological signal domain that there are currently few resources for.
The applications in clinical/medical practice are generally well motivated. 1D signal monitoring is commonplace in many clinical settings and the work presented here could be of relevance to researchers working in a variety of application areas.
While the paper does not propose new methods or architectures, the way it evaluates and compares existing methods is intuitive and well thought out. The experimental procedure is well described, results are presented in an interpretable way and discussed understandably in ways that might be informative for other researchers in the field.
The paper is clear and free of misspellings and other errors.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The paper is not novel. No methods are presented that have not been presented before by the literature. However, it is my opinion that experiments presented are valuable and informative.
I’m not sure the inclusion of the multi-head self attention model makes sense in the experiment. Transformer architectures and multi-head self attention has recently become widespread in the literature. Their applications are widely varied. They have been used as sequential models, models that pool information spatially, enhanced with convolutional blocks, and applied in a wide variety of novel architectures. As such it is hard to draw conclusions about the overall efficacy of transformer models from the single architecture explored in the paper.
Similarly, each of the CNN architectures employs some method of aggregating over the final output of the CNN encoders (flattening or GAP). Architectures that replace this aggregation and fully connected classification layers with transformers have already appeared in the literature. Therefore it is possible to envision combining the CNNs presented in the paper with transformers and multi-head self attention. Given these two points it may be better to limit the scope of the experiments to just CNN based models.
It would be good to perform the experiment on a wider number of tasks. The authors choose two tasks with different that require information from separate phenomena in the ECG for the network to identify. Different self attention mechanisms are more effective in each task according to the phenomena of interest. The inclusion of more experiments in different ECG applications would help verify that the results presented in the classification and regression tasks hold in a variety of situations.
The paper relies on citations for explaining the self-attention mechanisms investigated. The paper does a decent job motivating these mechanisms in writing but as these mechanisms are central to the understanding of the paper, further detail should be presented. This detail could come in the form of equations or schematic diagrams.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper seems to be pretty reproducible. I would be concerned that if there are any necessary adjustments to apply the self attention methods in the 1D setting as opposed to the 2D setting where they were proposed, these details have been omitted. Otherwise, network diagrams are given for each of the CNNs and the dataset used seems to be open source.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The self-attention mechanisms should be described in mathematical detail or schematics.
More citations for the conditions of interest should be provided. The authors cite a few deep learning papers that are applied to similar problems but it would be good for the authors to include some more clinical citations and background.
Some more description of the data would be good. It is not clear if the ECG is 12 lead. Similarly it’s not clear how the PPG was collected. It would be good to include these details. The dataset is not cited.
In figures 2 and 3, the fact that higher is better and lower is better, respectively, lead me to a bit of confusion. Explicitly stating this, as the figures look very similar, would be good.
Similarly, the discussion mentions that different self-attention mechanisms work better in each task because of the phenomena that must be identified for the tasks to be solved. Clinical citations for these phenomena would be good to verify these assertions.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
While the paper does not propose new methods, the authors identify an area of the literature that has been scarcely investigated and perform an intuitive and interpretable experiment. The experiment is likely to be of use to other researchers who seek to apply CV methods to CNN architectures for 1D physiological signals. Because of the clarity of the experiments and the potential value to other researchers working on similar problems, I would recommend this paper for acceptance.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
Four attention mechanisms were compared.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The manuscript was written clearly and organised well.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
There are no deep insights.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
It is reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The manuscript presents the comparison results of four attention mechanisms: squeeze and excitation, non-local, convolutional block attention module, and multi-head self-attention.
Although the samples were randomly split into the training set and testing set, it still has bias. Cross-validation is a more fair way to evaluate the models.
The demographic data were used as features for the classification. Statistical analysis should be performed to check whether or not the differences between patients and healthy people in these demographic data were significant.
The results shown in Fig. 2 demonstrate that the attention mechanism did not significantly improve the performance. The authors should provide insights. Does it imply that the attention mechanism is not effective in this case?
It is better to acronymise area under the receiver operating characteristics curve as AU-ROC.
I strongly suggest that the authors identify typos and correct them. For example, “mean percentage absolute error (MAPE)”
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although the manuscript’s contribution is not significant, it still provides help in the selection of the attention mechanisms.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #4
- Please describe the contribution of the paper
This paper comprehensively investigates four attention mechanisms fused with three convolutional neural network (CNN) architectures for two processing physiological signal prediction tasks. This paper aims to provide a good guide for researchers who use attention-based convolutional networks for physiological signal prediction tasks.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The problem studied in this paper is important and needs to be solved in ECG
- Extensive case studies
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Need more justifications about the novelty claims
- Need to include more related work that are highly important
- Need to check for grammatical errors and typos.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The dataset and code are not disclosed in this paper, so the reproducibility of this paper needs to be further improved.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Comments:
- Need to include more related work that are highly important
[1] Chen W, McDuff D. Deepphys: Video-based physiological measurement using convolutional attention networks[C]//Proceedings of the European Conference on Computer Vision (ECCV). 2018: 349-365.
[2] Zhu X, Cheng D, Zhang Z, et al. An empirical study of spatial attention mechanisms in deep networks[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 6688-6697.
[3] Wang S, Li B Z, Khabsa M, et al. Linformer: Self-attention with linear complexity[J]. arXiv preprint arXiv:2006.04768, 2020.
- The authors need to introduce some related work on the self-attention mechanisms in the introduction.
- Need more justifications about the novelty claims
- This paper comprehensively investigates four attention mechanisms fused with three convolutional neural network (CNN) architectures for two processing physiological signal prediction tasks. Although this paper can provide a good guide for researchers, the novelty still needs to be further justified and improved.
- It would be better if the authors could devise some novel ways of combining attention mechanisms with convolutional networks efficiently.
- The authors should explore the combination of efficient self-attention mechanisms with convolutional networks.
- Need to check for grammatical errors and typos
- The editorial quality of this paper is largely unsatisfactory. It contains quite a lot of inconsistent/non-precise description, as also reflected in the above comments.
- Need to include more related work that are highly important
[1] Chen W, McDuff D. Deepphys: Video-based physiological measurement using convolutional attention networks[C]//Proceedings of the European Conference on Computer Vision (ECCV). 2018: 349-365.
[2] Zhu X, Cheng D, Zhang Z, et al. An empirical study of spatial attention mechanisms in deep networks[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 6688-6697.
[3] Wang S, Li B Z, Khabsa M, et al. Linformer: Self-attention with linear complexity[J]. arXiv preprint arXiv:2006.04768, 2020.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although this paper provides a high-quality experimental guide on combining attention mechanisms with CNNs, novel designs and ideas are still lacking.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
The reviewer revised the decision to weak accept. The authors address the reviewers’ concerns well. Thanks to the authors for their detailed responses.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Summary & Contribution: This work proposes the use of an attention mechanism for processing two physiological signal prediction tasks: classification task for hypotension and regression for cardiac output. The main motivation is that attention mechanisms are widely used to improve performance in other fields but this has not been evaluated with physiological signals yet. In the study, the authors consider 4 different attention mechanisms (squeeze-and-excitation, non-local, convolutional block attention module and multi-head self-attention) and 3 CNNs (VGG16, ResNet18, InceptionV1), and evaluate multiple combinations for performance and convergence of physiological signals on a public dataset with more than 6,000 patients. Results show that the performance of the CNN when using attention mechanisms is improved.
The main contribution of this paper is an evaluation of attention mechanisms for physiological signal analysis with deep learning.
Key strengths:
- Strong comparison and evaluation analysis
- Clear clinical application and well-motivated
- Clinically important problem
Key weaknesses:
- Limited technical novelty
- Evaluation on two tasks; reviewers believe that more tasks should have been included to demonstrate the increased performance when using an attention mechanism
Evaluation & Justification: Reviewers agree that this work is well motivated and presents a clear clinical application on a clinically relevant problem. However, the authors have detected some issues with this work. Firstly some relevant works seem to have been missed in the literature review, and the self-attention mechanisms should be described in the paper.
If a rebuttal is submitted, please consider all authors comments. In particular, please clarify the potential bias between the samples in training, validation and test datasets and provide an explanation on why cross-validation was not considered. Please clarify results shown in Fig 2, did the attention mechanism help in this case?
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
4
Author Feedback
We would like to thank the AC and the reviewers for their constructive comments. Below, we address some major concerns raised by reviewers.
Potential bias on datasets and reason why not considered cross-validation (AC&R2): We initially considered the cross-validation, however, we found that it might be a hindrance to comparing model performances without any other external factors. Specifically, as the data we used were from real-world perioperative records, physiological status was significantly different among operation types of patients. The patients who underwent biliary or pancreas-related surgeries were more likely to have hypotension (p<0.001), and patient who had liver or kidney-relate surgeries tended to have high CO (p<0.001). If a fold for testing had biased toward severe surgeries, it might contaminate the general tendency of our results. Therefore, we manually selected a random seed and conducted random selection of testing set which was not biased toward specific surgeries.
More explanations about result shown in Fig. 2 (AC&R2): In Fig. 2, we demonstrated AU-ROC and convergence time (sec.) in hypotension prediction. The attention mechanisms did not significantly improve the performance (1.46%) compared to increased time overhead even in the best case (ResNet+NL). However, the maximum performance gain was 4.02% (Inception+NL) which implies potential benefit of attention model. In our experiments, we inherited most hyperparameters from canonical CNNs except depth, thus, there are still have room to improve performance through hyperparameter tuning. Lastly, we note that NL module (spatial attention) was beneficial for all CNNs, which means spatial information is crucial for hypotension prediction.
Require details about self-attention mechanisms (AC&R1): We cannot agree more that we should have included detailed mathematical descriptions and illustrations. We omitted it and relied on citations due to space limit, however, we would like to add rigorous explanations in camera-ready paper if allowed.
More citations for conditions of interest and technical components (AC&R1&R4): There were many other major clinical tasks adopting deep learning. In our manuscript, we tried to only focus on the most representative clinical problem (hypotension and CO) for two tasks (classification and regression) due to space limit. We specially thank the R4 for informative advice with examples of articles. However, a few articles were not directly related to our aim and scope. The article #2 analyzed attention mechanisms itself and #3 proposed a novel attention, while we tried to make suggestions of attention when we make a new application. Yet, our manuscript still has considerable scope for expansion as like the rigorous description in abovementioned articles. We will be pursuing more investigation of literatures and add mathematical details or schematics in our manuscript if we allowed.
About our scope and future work (include response to R1&R4): Our intrinsic and empirical research question was what the key disagreement between CNN for computer vision (CV) task and for physiological signal analysis task is. In computer vision area, several self-attentions (NL (2017), SE (2017), CBAM (2018)) were introduced first. ViT (2020) proposed that MSA-only model can achieve comparable performance to CNNs, and then, other papers (like CoAtNet (2021), CMT (2021)) comes up with combination of CNN and self-attentions to improve model performance. We tried to follow the track of development history in CV area, and this manuscript corresponding to the first milestone which find best pair of CNN and self-attentions or verify whether MSA-only model was better. For our next step, we plan to investigate the combination of convolution and self-attention. Furthermore, we are considering adding experiments with more clinical tasks.
We believe that every criticism pointed out by respected reviewers will be of great help to future research. Thank you.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The main contribution of this paper is an evaluation of attention mechanisms for physiological signal analysis with deep learning for two prediction tasks: classification of hypotension and regression for cardiac output.
Key strengths:
- Strong comparison and evaluation analysis
- Clear clinical application and well-motivated
- Clinically important problem
Key weaknesses:
- Limited technical novelty, but this is a CAI paper with a strong comparison analysis.
Review comments & Scores: After rebuttal, the score of the paper increased. R4 believes that the authors addressed all concerns during rebuttal
Rebuttal: Authors have provided a clear explanation on the bias between samples (e.g., different surgeries), the choice of validation and a clear explanation on the results shown in Figure 2.
Evaluation & Justification: The rebuttal has addressed all my concerns regarding data, and I believe this is a work worth publishing at MICCAI.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper seems weak in comparison to the other papers on my stack. I am wondering if a short conference paper is the right format for something like this? The evaluation here would need to be much more thorough to justify the lack of technical novelty and the paper might become more akin to a survey paper. I think a more extensive journal submission, e.g. to IEEE TMI might be more appropriate here.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
16
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper addresses an important problem and, although the methodology is not new, is a valuable guide for researchers who use attention-based CNNs. The justification of not using cross-validation is not totally satisfactory, since cross-validation can be stratified. But overall, the rebuttal addresses well the comments raised by the reviewers.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
NR