List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Md Sirajus Salekin, Ghada Zamzmi, Dmitry Goldgof, Peter R. Mouton, Kanwaljeet J. S. Anand, Terri Ashmeade, Stephanie Prescott, Yangxin Huang, Yu Sun
Abstract
Artificial Intelligence (AI)-based methods allow for automatic assessment of pain intensity based on continuous monitoring and processing of subtle changes in sensory signals, including facial expression, body movements, and crying frequency. Currently, there is a large and growing need for expanding current AI-based approaches to the assessment of postoperative pain in the neonatal intensive care unit (NICU). In contrast to acute procedural pain in the clinic, the NICU has neonates emerging from postoperative sedation, usually intubated, and with variable energy reserves for manifesting forceful pain responses. Here, we present a novel multi-modal approach designed, developed, and validated for assessment of neonatal postoperative pain in the challenging NICU setting. Our approach includes a robust network capable of efficient reconstruction of missing modalities (e.g., obscured facial expression due to intubation) using an unsupervised spatio-temporal feature learning with a generative model for learning the joint features. Our approach generates the final pain score along with the intensity using an attentional cross-modal feature fusion. Using experimental dataset from postoperative neonates in the NICU, our pain assessment approach achieves superior performance (AUC 0.906, accuracy 0.820) as compared to the state-of-the-art approaches.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16437-8_72
SharedIt: https://rdcu.be/cVRuX
Link to the code repository
N/A
Link to the dataset(s)
https://doi.org/10.1016/j.dib.2021.106796
Reviews
Review #1
- Please describe the contribution of the paper
The paper proposes an automated approach to asses pain in the neonatal intensive care unit. It is a multi-modal approach designed, developed, and validated for assessment of posoperative pain. The multiple modalities include visual and auditory inputs.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper is well written, the target application is excellent and it combines multiple modalities in the analysis. The architecture design seems appropriate for the task of combining multiple modalities and fusing them in a lower-dimensional latent space.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The approach is novel and derives from similar work using CNN-LSTMs. The innovation is a small step as it also uses a combination of similar techniques.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The data used is public and the methods are clearly explain. With these 2 elements the results should be reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
I enjoyed this paper and it was hard to find issues in the experimental design as well as any criticisim in the techniques used to accomplish the given task of pain estimation. The ablation experiment is interesting, although I would expect that the system should require all inputs as it achieves the higuest classification accuracy.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper is clearly written, the problem is challenging and uses multiple modalities which are combine using autoencoders and attention layers. The architecture is interesting and the proposed approach reached the higuest reported accuracy in the literature.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Somewhat Confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
Authors propose a novel approach for neonatal postoperative pain assessment. The proposed model is consist of three stages: spatio-temporal feature learning (Stage 1), joint feature distribution learning (Stage 2), and attentional feature fusion (Stage 3). The main technical contributions are as follows:
- develop a deep feature extractor followed by an RNN Auto-Encoder network
- design a novel generative model that combines all the modalities
- use a transformer-based attentional model Compared to other state-of-the-art, the proposed method outperforms in the classification task. Authors also show the results of the pain intensity estimation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Thorough evaluations: Authors evaluate the proposed method by using many types of scores and conducting ablation experiments. These results show the efficacy of the proposed approaches in the paper.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Lack of detailed information about the experiments: For example, authors do not describe how many data is used for the evaluation, how many pain/no pain cases there are. These information would help to understand the results exactly.
- The results of the pain intensity estimation are seemed to be not good enough.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- Authors use many public dataset and pre-trained models such as USF-MNPAD-I.
- The implementation details are well described.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- There are many types of scores in the results. However, which is the most important score from clinical point of view? What cases can the model classify correctly (but other state-of-the-art fail to classify)? These discussions might help to understand and improve the model.
- The pain intensity needs to be estimated so accurately? If not, is it usuful to estimate the range of the pain intensity? For example, range1(0-1), range2(2-4), and range3(5-7), it become a bit easier, and the model might achieve the performance good enough.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- Lack of detailed information about the experiments: For example, authors do not describe how many data is used for the evaluation, how many pain/no pain cases there are. These information would help to understand the results exactly.
- The results of the pain intensity estimation are seemed to be not good enough.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
Authors answered all the issues by all reviewers and are planning to revise it. My opinion is on the assumption that the paper is revised satisfactorily.
Review #2
- Please describe the contribution of the paper
The paper proposes a novel multi-modal approach for the assessment of postoperative pain of neonates. A deep feature extractor followed by an RNN Auto-Encoder network is used to extract spatio-temporal features from both visual and auditory modalities. A novel generative model is designed to combine all the modalities while learning to reconstruct any missing modalities. Instead of using early or late fusion techniques, a transformer-based attentional model is adopted to learn cross-modal features and generates final results.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1) This paper handles the problems of missing modalities by reconstructing them using a generative model. 2) It is novel to use a transformer-based attentional model to generate final pain label and its intensity.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) The performance of the proposed approach when dropping each modality was discussed in the paper, however, if some samples of one modality and other samples of another modality were missing at the same time, can the model reconstruct features? 2) Did the authors consider different weights of three modalities during the training process? As shown in Table 3, when the sound modality was dropped, it seems that the performance decreased more.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Implementation details have been described.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1) Page 2, Paragraph 3, “i.e., it makes final assessment of pain” should be “i.e., it makes the final assessment of pain” 2) Page 4, Paragraph 2, “Finally, feature sequences of each modality are used to train the LSTM-based AE …” should be “Finally, feature sequences of each modality were used to train the LSTM-based AE …”. 3) Page 6, Paragraph1, “All the models are developed based on PyTorch using GPU” should be “All the models were developed based on PyTorch using GPU”. The past tense is suggested to be used, please modify the tense of the whole paper.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper proposes a novel multi-modal approach for the assessment of postoperative pain of neonates. Instead of using early or late fusion techniques, a transformer-based attentional model is adopted to learn cross-modal features and generates final results.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #4
- Please describe the contribution of the paper
The paper proposes a method for neonatal postoperative pain assessment. In doing so, the authors focus on constructing missing signals, which is a common situation in NICU settings.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- evaluation on a public/real world dataset
- comparison to other methods
- ablation study
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- no future work
- no hardware, computation, running times
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper can be reproduced.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
There could be a few more sentences about future work and the hardware, computation and running times.
Minor comment: Page 8: Intbaution -> intubation
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper is well written, presents an algorithmic novelty and a good evaluation with an ablation study on a public dataset. That makes it also comparable to the state of the art.
- Number of papers in your stack
7
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Not Confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The authors propose an approach to asses pain in the neonatal intensive care unit from multi-modal data. It ddresses the problems of missing modalities by reconstructing them using a generative model. The method is novel and the application is very interesting. Please, address the points that reviewers comment on the evaluation and the results. Please remind that the purpose of the rebuttal is to provide clarification or to point out misunderstandings, it is not to promise additional experiments.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
NR
Author Feedback
We are thankful for all reviewers and their assessment of several strengths including important contributions, sound/comprehensive evaluations, and excellent writing/organization. We have corrected all the minor typos and seek to address the critiques here.
Two missing modalities (R2): Yes, the proposed method has the capability of reconstructing any number of missing modalities (i.e., one or more) by adjusting ELBO loss in Eq. 5. We provided results of only 1 missing modality due to the page limit.
Use of different weights for modalities (R2): We emphasize that we experimented with both equal weights and different weights, but the results were almost similar in both cases.
The past tense is suggested to be used (R2): We modified the tense of the whole paper as suggested.
More experimental details (R3): We presented the same number of data-points as the previous SOTA [14]. Our experiments included total 218 videos (50% pain) and we also followed the same leave-one-subject-out protocol for evaluation as SOTA [14]. These points are clarified in the revised manuscript.
Many metrics to report the results (R3): It has been reported [C1] that a single metric/score (no matter what is the metric) could give a false impression of the model’s actual performance, and in turn, yield unexpected results when deployed to a clinical setting. Therefore, we followed the recommendation in a recent Nature paper [C1] and used a combination of several metrics to interpret the performance holistically. [C1] Hicks SA, Strümke I, Thambawita V, Hammou M, Riegler MA, Halvorsen P, Parasa S. On evaluation metrics for medical applications of artificial intelligence. Scientific Reports. 2022 Apr 8;12(1):1-9.
Result of pain estimation (R3): The only available dataset does not have enough distribution of pain intensity (0-7), therefore, developing any regression model on this dataset is very challenging. We, however, followed an approach similar to what R3 has suggested and grouped several intensities and map the distribution over 0-1 and 0-4 ranges. We included these results already in the appendix (Section 3, Table 2) due to the space limit. The performance was better with lower range distributions. We moved some results from the appendix to the revised manuscript.
Cases our model can classify but SOTA fails (R3): Our model can classify any case with missing modalities as it can reconstruct these modalities and integrate them into the assessment. Previous SOTA cannot reconstruct missing modalities, and hence, it performed the assessment without them. We included this discussion in the revision.
Future work, hardware computation, running time (R4): Our future work will focus on further evaluation using a large-scale multi-site neonatal multimodal postoperative pain dataset collection. We included this future direction in the revised manuscript. During the experiments, we used intel core i7-7700K @4.20GHz, 32GB RAM, and NVIDIA GV100 TITAN V 12GB GPU machine. During the inference time, our approach (including 3 steps) takes approximately 1.623 ms to make a decision.
Address the points that reviewers’ comments on the evaluation and the results (MR1): We tried our best to address each point on the evaluation and results. We also added these points to the revised manuscript.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The rebuttal has satisfactorily addressed the reviewer’s concern and now they all agree to accept the paper
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
NR
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Apparently rebuttals do work. Apart from that I also find the paper of high enough quality for MICCAI, and a fresh and important topic.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
1
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The rebuttal has cleared the major concerns.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2