List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Wei Feng, Lie Ju, Lin Wang, Kaimin Song, Zongyuan Ge
Abstract
Existing deep learning models have achieved promising performance in recognizing skin diseases from dermoscopic images. However, these models can only recognize samples from predefined categories, when they are deployed in the clinic, data from new unknown categories are constantly emerging. Therefore, it is crucial to automatically discover and identify new semantic categories from new data. In this paper, we propose a new novel class discovery framework for automatically discovering new semantic classes from dermoscopy image datasets based on the knowledge of known classes. Specifically, we first use contrastive learning to learn a robust and unbiased feature representation based on all data from known and unknown categories. We then propose an uncertainty-aware multi-view cross pseudo-supervision strategy, which is trained jointly on all categories of data using pseudo labels generated by a self-labeling strategy. Finally, we further refine the pseudo label by aggregating neighborhood information through local sample similarity to improve the clustering performance of the model for unknown categories. We conducted extensive experiments on the dermatology dataset ISIC 2019, and the experimental results show that our approach can effectively leverage knowledge from known categories to discover new semantic categories. We also further validated the effectiveness of the different modules through extensive ablation experiments. Our code will be released soon.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43987-2_3
SharedIt: https://rdcu.be/dnwJl
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The authors propose to tackle the online new class classification tasks in skin lesions clustering.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The proposed method includes dual contrastive learning pretraining, pseudo labeling and refinement, and local neighbor information aggregation. Experimental results show the effectiveness of the proposed method.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The overall idea of this paper comes from the robust pattern recognition field, yet the authors applied the current research algorithms to a new field, discovering new semantic classes from dermoscopy image datasets. Most of these techniques are borrowed from the current incremental pattern recognition tasks.
The presented task is good, but the authors take these blurred concepts, classification, and clustering, from the title to the main content. Generally, the main task is to use the labeled and unlabelled data for precisely predicting the new categories. However, it is not clearly introduced what are the relations of classification and clustering in this algorithm.
Generally, the authors utilized too many existing algorithms that make the proposed method too complex, including contrastive learning networks, pseudo-labeling, label denoising with the mixup, and momentum-based optimization.
The compared algorithms come from the 2021, yet there are many incremental learning algorithms that have been published in 2022 and 2023. This is not the main reason, but the technical improvements should be carefully taken into account.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Too many parameters should be tuned, so not easy to reproduce.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
See weakness.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The authors combined many reserach techniques from the current researches in incremental pattern recognition, transfer learning, contrastive learning and label denosing, into one learning scheme, yet not novel enough.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
All the comments from different reviewers have been checked and the rebuttal almost tackled my concerns.
Review #2
- Please describe the contribution of the paper
The authors provide a framework for skin lesion classification with novel class discovery by combining contrastive learning for feature representation, uncertainty-aware multi-view cross-pseudo-supervision, and similarity-based local information aggregation. Their method consistently outperforms other methods for this problem.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper proposes a novel framework for skin lesion classification with a focus on novel class discovery, which is a challenging problem because novel classes lack expert-annotated labeled datasets. This paper compares its results to other approaches and consistently outperforms the other methods.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The method proposed in this paper is somewhat complex and deserves a better overview figure to help explain it. Figure 1 appears busy and difficult to read with small text, many arrows crossing, and multiple merge points throughout the process.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors provide implementation details in their Experiments section which make it relatively straightforward to reproduce, and they mention that they intend to publish their code soon which will further assist in reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
This paper is well-written overall and addresses an important problem in problem spaces with the potential for novel classes. The approach is interesting and integrates multiple methods to improve upon existing models. The experiments and ablation studies are sufficient to support the claims made by the authors and they report multiple metrics following examples from the field. One improvement that could be made is to make Figure 1 easier to read. The text and arrow heads are small, and it is difficult to get a sense of the flow through the process. Also, the “push” and “pull” arrows in the red area of Figure 1 are not necessarily well explained. Another way the authors could add some strength to their paper, since they are already averaging results across multiple runs, is to perform significance testing to see if their model performance is a significant improvement over the other results.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The major factors that led to me choosing “Accept” were: the novelty and complexity of the formulation, the experimental design and results, the clarity of the implementation details, and the quality and clarity of the writing.
- Reviewer confidence
Somewhat confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper proposed a deep-learning driven clustering method that can identify unseen classes before. To do this, this paper first use contrastive learning to capture discriminative features, and then use the the uncertainty-aware multi-view cross-pseudo-supervision strategy to jointly train data from both the known and unknown categories, followed by the “Local information aggregation to further enhance the alignment of local samples. The effectiveness of the proposed method is numerically validated.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- the research problem is interesting and appear to valuable.
- the proposed method can be easily extended to other disease.
- there is substantial improvement under comparison.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- it is better to provides some real-cases of diagnosis results
- How does it update the memory bank, whether is it using new data or retraining the model?
- in fig.1 what does the share mean? Does it first train the unlabeled data and then share the models parameters to the augmented unlabelled data. Or, two models are trained synchronously?
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
good.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
I may think the methology could be introcued more detailledly.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
There is a substantial improvement in perforamnce compared with SOTA baseline methods.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper received mixed ratings with both strengths and weaknesses. The authors are expected to respond the review comments for the novelty of the proposed method as pointed out by R#1, better explain the key concepts as pointed out by R#1, and provide better description on the overview of the proposed method.
Author Feedback
We thank AC and all reviewers for their constructive comments. This paper receives 3 reviews with scores of 4, 6 and 5 from R1, R2 and R3, respectively. We are encouraged by the positive comments: 1) motivation and ideas are novel and clear (R2) and (R3); 2) methods are helpful and well-linked to the insights (R2,R3); 3) experiments are adequate and rigorous (R1,R2,R3). We also thank R1 for pointing out many potential improvements for our work. Q1:relations of classification and clustering (@R1) A: We use the Sinkhorn-Knopp algorithm to generate pseudo-labels for unlabeled data that can be treated homogeneously with ground truth labels. This allows us to convert the clustering problem into a classification problem where we use the cross-entropy loss to operate on both labeled and unlabeled sets [1]. [1] Fini, Enrico, et al. “A unified objective for novel class discovery.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021. Q2: novelty concern (@R1) A: To the best of our knowledge, we are the first to explore the novel class discovery task in skin lesion classification. The methods in incremental learning cannot be directly applied to the NCD task. Because incremental learning assumes that the data of new categories are labeled and aims to prevent catastrophic forgetting, it is not possible to discover new categories from unlabeled data as NCD does. Although our method has many components, each component is necessary. We first learn general feature representations based on contrastive learning over all data. We then propose Uncertainty-aware multi-view cross-pseudo-supervision to identify both known and unknown classes in a unified framework. We use the Sinkhorn-Knopp algorithm to generate pseudo-labels for unlabeled data. The SOTA approach [2] does not consider the noise in the pseudo label, which leads to noisy pseudo label affecting the model training. So we also propose to use prediction uncertainty to mitigate the noise in pseudo labels. In addition, we propose to use information from local samples to further refine the pseudo label. We will also release the code for reproducibility. Q3: Comparison with the state-of-the-art algorithm (@R1) A: We implemented a state-of-the-art comparison algorithm, IIC [2] which is published in 2023. Its performance is: Task 0: ACC:0.6216 NMI:0.3149 ARI:0.2853; Task 1: ACC:0.5084 NMI:0.1721 ARI:0.1834. our performance: Task 0: ACC:0.6654 NMI:0.3372 ARI:0.3018; Task 1: ACC. 0.5271 NMI:0.1826 ARI:0.2033. In contrast, IIC does not consider the noise in the pseudo-label and ignores the local information, so our method achieves better performance. [2] Li, Wenbin, et al. “Modeling Inter-Class and Intra-Class Constraints in Novel Class Discovery.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. Q4: Clarification of the training procedure. (@R3) A: The memory bank is a first-in-first-out queue which stores the most recent samples and their corresponding pseudo-labels, updated based on each mini-batch data and length is fixed. Q5: Explanation of Figure 1 (@R2 @R3) Q5-1. Arrows confusion @R2 A: We apologize for the confusion. The “push” and “pull” arrows in the red area of Figure 1 refer to the strategy in unsupervised contrastive learning where we pull samples closer to their augmented samples and push samples away from other samples. In supervised contrastive learning, we pull samples with the same label closer and push samples with different labels away. We will polish the figure in the final version. Q5-2. “share” confusion @R3 A: “share” means that we use the same feature extractor to extract features from unlabeled data and augmented unlabeled data. The parameters of the feature extractor are first pre-trained using contrastive learning and then fixed and we update the parameters of the category classification head. We will polish the figure in the final version.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The issues/concerns raised by the reviewers have been addressed by the rebuttal. As all the reviewers consistently rate this paper positively, accept!
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Reviewers agree on the decision of acceptance.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Overall, the authors have addressed the concerns from reviewers.