List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Sana Ayromlou, Purang Abolmaesumi, Teresa Tsang, Xiaoxiao Li
Abstract
Standard deep learning-based classification approaches require collecting all samples from all classes in advance and are trained offline. This paradigm may not be practical in real-world clinical applications, where new classes are incrementally introduced through the addition of new data. Class incremental learning is a strategy allowing learning from such data. However, a major challenge is catastrophic forgetting, i.e., performance degradation on previous classes when adapting a trained model to new data. Prior methodologies to alleviate this challenge save a portion of training data require perpetual storage of such data that may introduce privacy issues. Here, we propose a novel data-free class incremental learning framework that first synthesizes data from the model trained on previous classes to generate a class impression. Subsequently, it updates the model by combining the synthesized data with new class data. To do so, we incorporate a cosine normalized Cross-entropy loss to mitigate the adverse effects of the imbalance, a margin loss to increase separation among previous classes and new ones, and an intra-domain contrastive loss to generalize the model trained on the synthesized data to real data. We compare our proposed framework with state-of-the-art methods in class incremental learning, where we demonstrate improvement in accuracy for the classification of 11,062 echocardiography cine series of patients. Code is available at https://github.com/sanaAyrml/Class-Impresion-for-Data-free-Incremental-Learning
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16440-8_31
SharedIt: https://rdcu.be/cVRvW
Link to the code repository
https://github.com/sanaAyrml/Class-Impresion-for-Data-free-Incremental-Learning.git
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The author presents a data synthesis strategy using learned neural network parameters for data-free incremental learning. Multiple loss functions are introduced to mitigate the catastrophic forgetting problem. The proposed scheme is applied in echocardiogram view classification and demonstrates its efficacy.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
A. The authors suggest to initialize the neural network with the average batch value of the input. The proposed scheme is an interesting attempt that is distinct from the existing deep inversion method. Through a comparative study, the paper demonstrates that the initialization contributes to the classification accuracy B. The performance of a neural network greatly depends on the scaling of multiple loss functions. The presented automatic loss weighting method mitigates the imbalance problem of multiple losses. C. The authors conduct extensive comparison experiments with diverse incremental learning methods. The proposed class impression (CL) method outperforms existing methods including LUCIR, ABM, and CLBM. The ablation study is presented and demonstrates the efficacy of using multiple loss functions.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
A. The incremental-learning method is assessed with tasks of classifying five views of echocardiography. Such an echocardiogram view classification task has a limited number of classification classes (<=5). The efficacy of the class impression to complicated tasks is concerned. B. The clinical effectiveness of the echocardiogram view classification is limited. C. The exact value of quantitative assessment is not provided.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Author agrees to disclose code and thus the result will be reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
A. The efficacy of the paper would be emphasized with a broader range of applications in medical incremental learning. For example, breast lesion classification, or chest x-ray classification could be a good application to show clinical effectiveness. B. Additional ablation study would help demonstrate the effect of the mean initialization method, and data synthesizing scheme.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The proposed class impression is an interesting approach that mitigates catastrophic forgetting problems in incremental learning. The experiment demonstrates that the class impression outperforms existing schemes. However, the clinical effectiveness of the experiment is not sufficiently demonstrated
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
The authors propose a data-free class Incremental learning method, and they show that the proposed framework which combines the pseudo images generation and three novel losses can provide the accuracy improvement for sequences of medical images classification tasks.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
The authors apply data-free rehearsal-based class incremental learning to the medical image analysis task. The implementation seems interesting.
-
This paper leverages three additional loss terms, which are specifically designed for medical image analysis, to address the catastrophic forgetting problem. One of them, named margin loss, is the first time to be applied to data-free class incremental learning.
-
Large improvements have been made on a datasets, which demonstrates the effectiveness on this task.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
There are parts where it is not clear if the authors are using something “off the shelf,” or if it is part of their contribution. For instance, what is the novelty of Class Impression compared to DeepInversion? It seems they have exactly the same optimization objective. Is the Class Impression specifically designed for medical images? What new challenges does the medical image introduce compared to the popular datasets widely used in computer vision, such as CIFAR, ImageNet, and so on? And how does the Class Impression address them?
-
The organization of the paper is incomplete. Since the paper contains various existing methods, the authors should provide more details in the literature review and background. Besides, the experiments section lacks the necessary discussion to analyze the results.
-
The evaluations are not solid. The four-task setup containing only five classes may be too simple to cause the catastrophic forgetting problem. The longer sequence of learning tasks with more classes should be used to evaluate the effectiveness of the method. From the current experiments, I still doubt if using pseudo images in task T, which are generated from the model in task T-1, for replaying can avoid catastrophic forgetting.
-
They also should double-check the bold statements and references. For instance, the authors claim that Smith et al. train a generative model to synthesize images without considering preserving class-specific information. As far as I am concerned, they not only did not train a generative model but also improved the DeepInversion which is used in this manuscript. Why are the images generated by Smith et al.’s method look so bad as shown in Fig. 1. (d)?
Ref: Smith, J., Hsu, Y.C., Balloch, J., Shen, Y., Jin, H., Kira, Z.: Always be dreaming: A new approach for data-free class-incremental learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9374–9384 (2021)
-
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Good. All methods used for the proposed framework are depicted clearly and noted with appropriate references.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Please double-check the correctness of reference [22] in the manuscript. The experimental evaluation could be expanded, especially for the ablation studies.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The decision are mainly made according to the novelty of the proposed framework and the organization of this paper.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
The authors propose Class Impression, a novel data-free class incremental learning framework. In Class Impression, instead of saving data from classes in the earlier tasks that are not available for training in the new task, they synthesize class-specific images from the frozen model trained in the last task.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors propose novel 1) cosine normalized cross-entropy loss for imbalance issue, 2) margin loss to encourage robust decision boundary, and 3) intra- domain contrastive loss to alleviate domain shift, which empowers Class Impression in addressing the catastrophic forgetting problem. – They conduct extensive comparison experiments and ablation analysis on the echocardiogram view classification task to demonstrate the efficacy.
Class Impression out-performs the SOTA methods in data-free class incremental learning with an improbable gap of 31.34% accuracy in the final task and get comparable results with the SOTA data-saving rehearsal-based methods.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
This is a strong paper. There are no main weaknesses detected.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Highly reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
This is a very interesting approach with real potential in medical imaging applications due to privacy regulations of data storage.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
It is challenging to deploy class-incremental learning for medical image analysis. The proposed approach is novel. To the best of my knowledge there are no existing data- free rehearsal-based class incremental learning work on deep neural networks specifically designed for medical image analysis.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper has split reviews. Reviewers think that the paper has substaintial merits related the novel approach for synthesizing images from trained model. The performance improvement seems convincing. However, the first and second reviewers also pointed out significant weaklesses related the difference between the proposed method and DeepInversion, and insufficient demonstration of clinical applicability, too simple evaluation procedures. After considering all reviewers’ comments, the area chair would like to invite authors to submit a rebuttal addressing the reviewers’ concerns.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
14
Author Feedback
We sincerely thank the reviewers for their valuable comments. We are grateful that R1 & R3 cited the novelty of our approach, and all the reviews agreed on the significant improvement over prior art. We address the concerns of the reviewers below.
1- Novelty over ABD [1] and DeepInversion [2] - R2 Our key innovations over [1,2] are multifold, lying in utilizing the unique class-specific information in medical images: 1) we initialize batches for the image synthesis step with the mean of each class; 2) we mitigate the domain shift between class-wise synthesized images and original images by defining the Intra-domain Contrastive Loss. These innovations were motivated by the observation that medical imaging samples within a categorical classification task normally share similar anatomical landmarks. In contrast, [1,2] do not leverage this class-specific prior, as such observation does not necessarily hold in natural images. We further utilize the class-specific prior to design Intra-domain Contrastive Loss, which aims to bring the mean of latent representation for synthesized and original images for each class close to each other while pushing the mean across classes far apart. The effectiveness of our innovations is demonstrated in Fig. 2, where we substantially improve accuracy via a head-to-head comparison against [1,2]. Note that [1] optimizes ConvNets to generate and sample synthetic images (described in Sec 4, [1]) in a form of ‘generative model’, instead of directly optimizing image batches like ours and [1].
We emphasize that our implementation for [1] uses the publicly available code from the authors, with hyperparameter tuning on our data.
Evaluation - R1&2 We followed the first setting of [1] for our experimental setup, where the performance was compared over 5 tasks. We ran additional experiments with our dataset with up to 10 tasks and observed that while our performance gain over prior arts (shown in Fig. 2.a) is maintained, the drop in overall classification accuracy is such that neither of the methods is likely clinically usable (dropping from 73% accuracy for 5 classes to 39% for 10 classes for our method; in contrast, [1] drops from 33% for 5 classes to 12% for 10 classes). Our decision to not include classification results beyond 5 tasks is also aligned with the comment raised by R1 in terms of clinical effectiveness (see the next comment). We emphasize that in addition to the task-based comparison with 5 prior arts, we also present a detailed ablation study with 4 different settings to justify the necessity of our design.
Justification for the Choice of Clinical Application - R1 We chose the echo view classification task as a test-bed to evaluate our innovations. In contrast to many other medical imaging classification problems, the labels associated with standard echo views are less ambiguous, less noisy, and anatomically interpretable, making the analysis of classification results easier and its failure modes tractable. We believe that our approach is not specific to echo view classification; rather, the innovations are very generalizable to other categorical medical image classification tasks. It is common in clinical deployment that a client inherits a trained model without having access to its training data. Our design enables the client to refine the model with new tasks.
Following the comment from R2 in regards to the paper organization, we will refine the paper structure by separating the discussion of results from the description of the baseline methods. We will also further clarify the literature review and background to distinguish our approach. We will indicate the exact accuracy numbers on Fig.2 and appropriately refer to them in our discussion.
- Smith, J., et al.: Always be dreaming: A new approach for data-free class-incremental learning. CVPR2021
- Yin, H., et al.: Dreaming to distill: Data-free knowledge transfer via deepinversion. CVPR 2020
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper has split reviews. Reviewers think that the paper has substaintial merits related the novel approach for synthesizing images from trained model. However, the first and second reviewers also pointed out significant weaklesses related the difference between the proposed method and DeepInversion, and insufficient demonstration of clinical applicability, too simple evaluation procedures. After the rebuttal, the majority of reviewers favor acceptance. The rebuttal seems to adequately address main concerns. The AC recommends to accept t his paper.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
NR
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have defended well their novelty statement in the rebuttal, as well as their choice for clinical application. I think this should remove the reviewers’ concerns. Overall a solid paper.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
4
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The author presents a data synthesis strategy using learned neural network parameters for data-free incremental learning. Following my reading of the paper, reviews, and rebuttal, it seems the authors have addressed most of the concerns, especially the novelty over ABD and DeepInversion. Recommend to accept and the authors should integrate the clarifications given in the rebuttal in the paper if finally accepted.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3