List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Mark Endo, Kathleen L. Poston, Edith V. Sullivan, Li Fei-Fei, Kilian M. Pohl, Ehsan Adeli
Abstract
Parkinson’s disease (PD) is a neurological disorder that has a variety of observable motor-related symptoms such as slow movement, tremor, muscular rigidity, and impaired posture. PD is typically diagnosed by evaluating the severity of motor impairments according to scoring systems such as the Movement Disorder Society Unified Parkinson’s Disease Rating Scale (MDS-UPDRS). Automated severity prediction using video recordings of individuals provides a promising route for non-intrusive monitoring of motor impairments. However, the limited size of PD gait data hinders model ability and clinical potential. Because of this clinical data scarcity and inspired by the recent advances in self-supervised large-scale language models like GPT-3, we use human motion forecasting as an effective self-supervised pre-training task for the estimation of motor impairment severity. We introduce GaitForeMer, Gait Forecasting and impairment estimation transforMer, which is first pre-trained on public datasets to forecast gait movements and then applied to clinical data to predict MDS-UPDRS gait impairment severity. Our method outperforms previous approaches that rely solely on clinical data by a large margin, achieving an F1 score of 0.76, precision of 0.79, and recall of 0.75. Using GaitForeMer, we show how public human movement data repositories can assist clinical use cases through learning universal motion representations. The code is available at https://github.com/markendo/GaitForeMer.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16452-1_13
SharedIt: https://rdcu.be/cVRYQ
Link to the code repository
https://github.com/markendo/GaitForeMer
Link to the dataset(s)
https://rose1.ntu.edu.sg/dataset/actionRecognition/
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposed a novel method GaitForeMer that forecasts motion and gait (pretext task) while estimating impairment severity (downstream task). By pre-training on NTU dataset, it can improve performance of early diagnosis of Parkinson’s disease (PD) on a small dataset.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper proposed a novel method. It is the first one to used motion data from normal people to pretrain the model and then fineturn on the skeleton-based motor impairment estimation task, which can well solve the problem of small sample size. With a strong and reasonable motivation, they designed a reasonable and feasible transformer-based model and achieved remarkable results. I think it deserves to be promoted in this field.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Some important details are missing. For example, in Sec 2, despite citing previous work, it is not clear enough for the model architect, and in Fig. 1, some details are not shown, such as positional embeddings, self- and encoder-decoder attention in decoder, add operation in residual connection and so on. And the classification result after linear should be “Activity when pretrain and MDS-UPDRS Score when fineturn”. All these confuse me when I read the paper.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
I think this paper has a strong reproducibility with a complete description of the training process, as well as its reasonable model design.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
It is a good job that shows us the potential of pre-training models in this field. It could be better if the paper can describe the proposed model in more detail. Besides, Fig. 1 is somewhat simple and is very similar to the picture in the referenced previous work POTR. Redesign the picture in a different way could be better. And I suggest that “z0” should not be put in the same box as “z1,z2,…,zT” for it will not serve as input to the decoder.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
A novel method that first use pre-training models to deal with small sample size problem in this field.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This paper presents a method to predict Movement Disorder Society Unified Parkinson’s Disease Rating Scale (MDS-UPDRS) scores. To this end, the authors propose Gait Forecasting and impairment estimation transformer (GaitForeMer), a transformer model using motion forecasting as a self-supervised pre-training task. The proposed system achieves an F-1 of 0.76, which is 0.18 higher than OF-DDNet [1].
[1] Lu,M.,Poston,K.,Pfefferbaum,A.,Sullivan,E.V.,Fei-Fei,L.,Pohl,K.M.,Niebles, J.C., Adeli, E.: Vision-based estimation of mds-updrs gait scores for assessing parkinson’s disease motor severity. In: Medical Image Computing and Computer Assisted Intervention (2020)
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1) The author presents an original idea of self-supervised learning of a given model by predicting the latter half of the video. 2) Detailed experimental results are presented (e.g., comparing various existing methods, showing results while changing the fine-tuning strategy, and showing performance changes as the amount of training data changes).
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) The reproducibility of this paper is low. 2) The dataset used is not disclosed, but an accurate explanation is still lacking. 3) Since only simple explanations are listed for the experimental results, it is difficult to grasp the advantages and disadvantages of the proposed method based on the results.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
A lot of parameters and design details are missing. It seems that it cannot be reproduced only by reading the paper.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
In contrast to good ideas, the explanation of the proposed design is not thorough. Compared to the cited paper [1], the novelty of this paper is incremental.
[1] Lu,M.,Poston,K.,Pfefferbaum,A.,Sullivan,E.V.,Fei-Fei,L.,Pohl,K.M.,Niebles, J.C., Adeli, E.: Vision-based estimation of mds-updrs gait scores for assessing parkinson’s disease motor severity. In: Medical Image Computing and Computer Assisted Intervention (2020)
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The manuscript is clear and easy to follow, and the results are plausible. But the novelty is incremental.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
It is an interesting paper. The paper develops models to predict MDS-UPDRS gait impairment severity and these models are first pre-trained on public datasets to forecast gait movements.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
the use of pretraining has enhanced the performance
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Discussion on possible misclassification missing. Also novel contributions in the work are unclear. The use of pretraining using public datasets is fine but this is general idea that pretraining improves performance which is already established.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
reproducible
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
It is an interesting paper. The paper develops models to predict MDS-UPDRS gait impairment severity and these models are first pre-trained on public datasets to forecast gait movements.
Comments are below: Illustrate and discuss the misclassifications from the proposed method? And also suggest possible ways to make it better? Any discussion to make the model more interpretable would add value.
Report areas under roc as well and areas under the pr curve for all methods. What is the threshold used for all classification methods? How was it decided? Is that threshold optimal?
Provide more description on the subjects in the study such as their age, gender and severity of Gait Impairment etc.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
further evaluation needed in terms of performance metrices such as area under roc and area under precision recall curve
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
All three reviews tend toward weak accept.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6
Author Feedback
We thank the reviewers and the AC for their encouraging and constructive comments. In regards to adding more clarity to our proposed design and model architecture, we will make the following changes to Section 2 of the final paper. We will change x_T to x_t and y_1 … y_M to x_t+1 … x_T to emphasize the fact that these are forecasted movements from the same subject. This allows us to represent the MDS-UPDRS score labels as y which can hopefully make the score/activity classification and motion prediction branches more distinct. We will also add the modifications suggested by the reviewers. Furthermore, we will release the source code of our method, which will aid reproducibility and clarification of the technical details.
With respect to the novelty of our approach, it is important to note that to the best of our knowledge, we are the first to use motion prediction from public human activity data as an effective self-supervised pre-training task for the downstream task of motor impairment severity estimation. We demonstrated how our model can obtain state-of-the-art results even in few-shot learning settings. Please note that this is not any pretraining on a clinical dataset or in a supervised manner as regularly done by other works. We will add these clarifications in the final paper.
Regarding the dataset details, for the blind submission, we were not able to disclose certain information about the dataset. That said, we will include more information in the final version.