List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Yaojia Zheng, Zhouwu Liu, Rong Mo, Ziyi Chen, Wei-shi Zheng, Ruixuan Wang
Abstract
Accurate automated analysis of electroencephalography(EEG) would largely help clinicians effectively monitor and diagnose patients with various brain diseases. Compared to supervised learning with la- belled disease EEG data which can train a model to analyze specific diseases but would fail to monitor previously unseen statuses, anomaly detection based on only normal EEGs can detect any potential anomaly in new EEGs. Different from existing anomaly detection strategies which do not consider any property of unavailable abnormal data during model development, a task-oriented self-supervised learning approach is proposed here which makes use of available normal EEGs and expert knowledge about abnormal EEGs to train a more effective feature extractor for the subsequent development of anomaly detector. In addition, a specific two-branch convolutional neural network with larger kernels is designed as the feature extractor such that it can more easily extract both larger-scale and small-scale features which often appear in unavailable abnormal EEGs. The effectively designed and trained feature extractor has shown to be able to extract better feature representations from EEGs for development of anomaly detector based on normal data and future anomaly detection for new EEGs, as demonstrated on three EEG datasets. The code is available at https://github.com/ironing/EEG-AD.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16452-1_19
SharedIt: https://rdcu.be/cVRY1
Link to the code repository
https://github.com/ironing/EEG-AD
Link to the dataset(s)
https://physionet.org/content/chbmit/1.0.0/
https://www.kaggle.com/c/seizure-detection/data
Reviews
Review #1
- Please describe the contribution of the paper
his paper proposed a task-oriented self-supervised learning approach to train a feature extractor based on normal EEG data and key properties of the abnormal EEGs. Two-branch of CNN with larger kernels is designed for effective extraction of both small-scale and large-scale features. The method is tested on two public and one internal EEG datasets with reasonable good performance.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Self-learning for anomaly detection of EEG based on CNN may have some novelties The way of generating abnormal EEG signals is interesting even if there exist deficiencies.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The simulated abnormal EEGs are generally different from real abnormal EEGs, e.g., more irregular and include combinations of various anomalies. It is expected that more realistic amplitude-abnormal and frequency-abnormal EEG data can be generated for anomaly detection. Hence, it is in doubt whether or not the proposed method is capable of detecting more complex noisy abnormalities in EEG signals.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Codes not open, should have some issues in reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1) What’s the criterion in defining the range of amplitude scaling factor, e.g., \alpha_l and \alpha_h in the second paragraph of page 3, “Generation of self-labeled abnormal EEG data”? Also applies to the scaling factors for frequency scalar factor. Is this range different for different types of anomalies? 2) Please compare with method [25], and indicate the major advancement of the present method in comparison to [25]. Noticed that there are some results presented in Fig. 3. 3) In general, what’s the dimension for the shortcut branch from the output of the 1st convolutional layers to the penultimate layer? Does this design need to change for the detection of different anomalies of EEG disease? 4) Regarding the methodology discussed in page 5, what’s the performance if you retain the classifier head instead of using Mahalanobis distance on the extracted features?? And what’s the criteria of determining the threshold for anomaly detection? 5) As for the experiments, please indicate the disease present in these datasets.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The moderate novelties of self-supervised learning for anomaly detection of EEG signals and the way of generating abnormal EEG signals. However, the abnormalities generated may not be practical abnormal EEG signals, hence limit its applicability of application.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
The rebuttal response from the authors have clarified most of my doubts even if I am still not totally convinced with the explanation of the use of more complex combinations of anomalies of EEG. The inclusion of two new baselines in the comparison and the clarification of the two recently proposed baselines help appreciating the contribution of the paper.
Review #3
- Please describe the contribution of the paper
In this paper, the authors demonstrate the use of converting domain knowledge into SSL transformation rules to augment the data, so that the anomaly detection result is improved. The authors demonstrate the effectiveness through two simple transformations.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The idea is simple and natrual. In other words, domain experts may add as many SSL transformation rules based on their understanding of the problem.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- I worry that how deep the knowledge human experts need to have to create valuable but not detrimental SSL rules. For instance, the proposed method works in these three datasets due to the simulated anomalies make some sense under the context. However, do this approach generalize when we inject incompatible simulation rules?
- The baseline are not SOTA. For instance, OCSVM, KDE, and AE are very basic methods. There are a large number of SOTA sequence-based anomaly detection methods in https://github.com/datamllab/tods [1].
[1] Lai, K.H., Zha, D., Wang, G., Xu, J., Zhao, Y., Kumar, D., Chen, Y., Zumkhawaka, P., Wan, M., Martinez, D. and Hu, X., 2021, May. TODS: An Automated Time Series Outlier Detection System. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 18, pp. 16060-16062).
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
GIven the submission, it is possible to reproduce the results even without the code.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- I think the authors need to justify how to and when should one to use the task oriented SSL rules? When do they apply?
- I would highly recommend improve the baseline. The current unsupervised anomaly detection baselines are too old to be effective.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I am mainly concerned due to is applicability and experiment setting.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #4
- Please describe the contribution of the paper
A simulated based anomaly detection method for EEG Data.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
The experimental results are quite good ! It is great to see the simulated based anomaly detection is such effectiveness in EEG modality.
-
The ablation study is well designed.
-
Using Mahalanobis distance as anomaly score is reasonable.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
” Impirical evaluations show that the above-mentioned simple transformations are sufficient to help train an anomaly detector for EEGs. “
The comparsion with fully-supervised method is required to support this claim. This claim may mislead the readers ! Intuitively, more complex anomaly data can improve the performance.
Also, why not choose more complex anomaly similation combining frequency and amplitude abnormal ?
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Good.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- Choose and discuss more complex anomaly similation combining frequency and amplitude abnormal.
- It is suggested to provide 2D t-SNE to visualize the features of normal, simulated abnormal and abnormal features.
- Comparision to supervised ( with fully and partial training label ) methods.
- As a classifier is trained, it is interesting to see the performance of directly using the classifier to detect abnormaly EEG data.
- Another simulated based method in medical imagning is highly related to this paper but has not been cited: (NormNet:) Label-free segmentation of COVID-19 lesions in lung CT, TMI, 2021.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although the idea is similar to CutPaste and NormNet, it is valueable to see such a simulated-based anomaly detection method is effectiveness in EEG modality. The experiments are well designed, especially the ablation study. Also, there is a major concern : lack of comparision with supervised methods. So, I recommend weak accpet.
- Number of papers in your stack
2
- What is the ranking of this paper in your review stack?
6
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
Rebuttal: “model training would cause overfitting to limited simulated complex anomalies” can not convince me.
Despite the weakness, I believe the method has novelty and interest to MICCAI community on EEG data.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper presents a task-oriented self-supervised learning approach to train a feature extractor to detect abnormal EEG signals.
Overall, reviewers found the approach intuitive and adaptable to many different scenarios, by modifying the SSL transformation rules, and overall the method improve over the comparison methods.
However, as R2 and R1 pointed out the comparisons were with quite basic methods and comparisons against the state-of-the-art were missing. The other major critique is the applied SSL rules were quite basic, R4 point out that more complex combinations of anomalies could improve the performance and the algorithm, and R1 was concerned that with the current approach more complex EEG anomalies may be missed as the current simulations do not accurately reflect abnormalities seen in real world data. Relately, more details on the type of diseases in the EEG used to evaluation the approach could help to clarify this point. Finally R2 also mentioned the main weakness of this approach seems to be one could add incorrect task oriented SSL rules and these would determintaly impact performance, this would be an interesting point for the authors to comment on the robustness of this method to different SSL rules.
Overall I recommend this paper for rebuttal, the authors need to justify their selection of the comparison methods and choice of SSL rules.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
8
Author Feedback
Q1 (R#3): comparisons were with quite basic methods and comparisons against SOTA methods were missing A1: Our method focuses on training a feature extractor (stage 1) which can well extract discriminative features for the development of anomaly detector (stage 2), and the proposed SSL idea (stage 1) can be combined with any anomaly detection technique (stage 2). Therefore, we only chose several representative baselines to show effectiveness of the SSL idea. Note baselines also include two recently proposed methods (ScaleNet in BIBM’20, CutPaste in CVPR’21). As suggested by R#3, more SOTA methods were compared on dataset CHB-MIT, e.g., resulting in AUC 0.715 for MSCRED & 0.827 for USAD, all outperformed by ours (AUC=0.924), supporting effectiveness of our method.
Q2 (R#1,#4): the applied SSL rules were quite basic and simulated anomalies do not reflect real anomalies, and more complex combinations of anomalies could improve performance A2: We kindly remind the SSL rules (stage 1) are used to train a feature extractor (Fig.2) for stage 2. A better feature extractor means it can effectively extract discriminative features to help discriminate normal from abnormal EEGs. While real abnormal EEGs are often complex and include combinations of abnormal frequency and amplitude signals, empirical evaluations show the proposed simple SSL rules (based on simulated anomalies in frequency and amplitude individually) performed better than the more complex SSL rules (based on combinations of simulated abnormal frequency & amplitude), with AUC 0.954 vs. 0.901 on dataset Internal, and 0.924 vs. 0.792 on CHB-MIT. One possible reason is, although simulated combinations of anomalies are more realistic, they may not cover all possible anomalies in real abnormal EEGs and so model training would cause overfitting to limited simulated complex anomalies. In contrast, with simple SSL rules, the model (feature extractor, Fig.2) is trained to extract features which are discriminative enough between normal EEGs & simulated abnormal EEGs based on only abnormal frequency or amplitude features, i.e., the feature extractor trained with basic SLL rules is more powerful/effective in extracting discriminative features (because it can extract discriminative features based on either aspect of anomalies rather than combinations of two aspects).
Q3 (R#3): applicability & robustness: How deep knowledge experts need to create valuable but not detrimental SSL rules? Generalizable when injecting incompatible rules? How&when to apply? A3: The merit of our SSL rules is it doesn’t require simulate realistic complex anomalies but is only based on individual aspects of anomalies in abnormal EEGs (also refer to A2 above). Experts know that anomalies in EEGs appear in aspects of frequency and amplitude, and therefore simulated anomalies in each aspect can be easily created and such creation is less likely detrimental. In extreme case, even when injecting incompatible rules, our SSL strategy should still work stably. E.g., during training feature extractor (Fig.2), when a proportion (5%, 10%, 15%, 20%) of simulated abnormal EEGs are replaced by fake abnormal EEGs (each fake EEG randomly from real normal EEGs but used as abnormal), the final performance (AUC) is respectively 0.938, 0.903, 0.899, and 0.891 on Internal, lower than reported AUC (0.954) but still kept at high level.
Q4 (R#1,#4): What if directly using the classifier (of 1st stage) to detect abnormal EEGs? A4: It is worse than the proposed method.
Q5 (R#1): details on type of diseases in EEGs A5: Will include & show performance on each disease.
Q6 (R#1): reproducibility A6: Will add link to full source code in paper.
Q7 (R#4): lack of comparison with supervised methods A7: We focus on anomaly detection where only normal EEGs are available during training, so it is unfair to compare with supervised methods which use both normal & abnormal EEGs during training.
Other comments are also adopted to refine paper.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This manuscript presents a technique for self-supervised learning to detect anomalies in EEG via creating “abnormal” signals using a set of simple, task and domain specific rules. I think in general reviewers had some concerns on why only use simple rules, and not combinations or more complex rules, and on the appropriateness of the comparisons. The rebuttal addressed most of these concerns in a reasonable manner, and most reviewers after the rebuttal felt this manuscript was worth of acceptance. However, some lingering concerns remain especially related to why the network would learn to overfit complex anomaly rules but not simple rules, and perhaps this is a topic the authors should consider more carefully in the final manuscript.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
moderate technical contributions but with good empirical results
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
9
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have done very well in addressing the reviewers’ concerns
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2