List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Qian Zhou, Hua Zou, Zhongyuan Wang
Abstract
More and more people are suffering from ocular diseases, which may cause blindness if not treated promptly. However, it is not easy to diagnose these diseases for the barely visible clinical symptoms. Even though some computer-aided approaches have been developed to help ophthalmologists make an accurate diagnosis, there still exist some challenges to be solved. For example, one patient may suffer from more than one retinal disease and these diseases often exhibit a long-tailed distribution, making it difficult to be automatically classified. In this work, we propose a novel framework that utilizes the correlations among these diseases in a knowledge distillation manner. Specifically, we apply the correlations from three main aspects (i.e., multi-task relation, feature relation, and pathological region relation) to recognize diseases more exactly. Firstly, we take diabetic retinopathy (DR) lesion segmentation and severity grading as the downstream tasks to train the network backbone for the findings that segmentation may improve the classification. Secondly, the long-tailed dataset is divided into several subsets to train multiple teacher networks according to semantic feature relation, which can help reduce the label co-occurrence and class imbalance. Thirdly, an improved attention mechanism is adopted to explore relations among pathological regions. Finally, a class-balanced distillation loss is introduced to distill the multiple teacher models into a student model. Extensive experiments are conducted to validate the superiority of our proposed method. The results have demonstrated that we achieve state-of-the-art performance on the publicly available datasets. ode will be available at: https://github.com/liyiersan/RLKD.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16434-7_68
SharedIt: https://rdcu.be/cVRsA
Link to the code repository
https://github.com/liyiersan/RLKD
Link to the dataset(s)
https://csyizhou.github.io/FGADR/
https://odir2019.grand-challenge.org/
Reviews
Review #1
- Please describe the contribution of the paper
This paper focuses on mitigating the long-tailed effect in the multi-label classification of eye images. The proposed method first divides multiple diseases into several groups. A teacher model is trained for each group. Then a student learns from all the teacher models through knowledge distillation.
The basic idea of this paper is similar to [16], with several refinements, e.g., mutli-task backbone pretraining, region-based attention, automatical relational subsets generation, and class-balanced distillation loss.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This work is dealing with an important problem, i.e., the long-tailed effect in multi-label classification.
- The newly added components are reasonable for this specific task.
- Comparably thorough related works.
- Well written and easy to follow.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Given the main idea is similar to [16], the methodology contribution is limited.
- The performance improvement of the proposed method is marginal.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors provide sufficient implementation details, but no absolute guarantee for reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
If possible, provide the ground-truth locations of diseases in Fig.3 in comparison to the CAMs.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Please see the strengths and weaknesses.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
The authors proposed a framework that contains multi-task relation, feature relation, and region relation in a knowledge distillation manner to recognize ocular diseases. The framework uses lesion segmentation and grading information from subjects with diabetic retinopathy to perform an initial classification task. The authors tested the proposed method with one dataset and reported performance metrics and class activation maps.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The ablation study and the qualitative and quantitative analysis performed for the authors.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The class activation maps are not useful to highlight the obtained results. For example, In Figure 3 (a), (c) and (d) the highlighted areas are not specific to the diseases. The results are good but some issues are missing in the paper.
- Why CCT-Net method was not explored using ResNet-50 backbone?
- Which results are obtained using healthy images as input?
- What about the CAM in healthy images?
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Although the dataset are free public available, the authors should consider to release the code and some models in order to reproduce the results!
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- The class activation maps are not useful to highlight the obtained results. For example, In Figure 3 (a), (c) and (d) the highlighted areas are not specific to the diseases.
- Why CCT-Net method was not explored using ResNet-50 backbone?
- Which results are obtained using healthy images as input?
- What about the CAM in healthy images?
- The code and some model weights are missing, so the reproducibility of the paper would be impossible!
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The framework uses lesion segmentation and grading information from subjects with diabetic retinopathy to perform an initial classification task. Then, the dataset is split into several subsets to train different teachers and to apply attention mechanisms, then multiple teacher-student is distilled. The authors tested the proposed method with two datasets with the performance metrics and class activation maps.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
In this paper, the authors propose a novel knowledge distillation framework for long-tailed multi-label retinal disease recognition. First, a network is pretrained with classification and segmentation tasks on a well-labeled public dataset. Second, the long-tailed dataset is automatically divided into three subsets to train multiple teacher networks, which can help reduce the label co-occurrence and class imbalance. Besides, an spatial attention mechanism and a class-balanced distillation loss is introduced. The superiority of the proposed method is evaluated on a public dateset.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The motivation, problem description, and the proposed solution are very clear
- The paper is well-written and pleasant to read
- The method is evaluated on a public datasets and outperforms other relevant approaches.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The framework in this paper is very similar with existing methods, such as the “Relational subsets knowledge distillation for long-tailed retinal diseases recognition (MICCAI 2021)”.
- In section 2.4, the authors extracted image features with a pretrained ResNet50 and employed k-means algorithm to cluster the images into subsets. I wonder if the samples are directly clustered into three subsets (i.e., {D, AMD, H, M}, {G, C}, {N, O}) or first clustered into 8 classes (i.e., D, AMD, H, M, G, C, N, O) then divided into three subsets based on the pre-defined rule? Besides, according to the section 3.1, the images are labeled as the 8 classes. Why not use the class labels to divide all samples into three subsets?
- In section 2.3, the authors preposed a novel region-based attention, which slightly differs from CBAM by combining a trainable convolutional layer. However, the authors did not provide the experimental comparison between the proposed attention and CBAM. I think this is a crucial comparison and suggest the authors to provide the comparison results.
- In section 2.6, the training and testing samples are randomly splitted. The randomness of data partition might affect the results. I would suggest 5-fold cross-validation to produce more solid results.
- In table 1, the performance of the proposed method is slightly worse than the existing method, i.e., CCT-Net. The performance is not satisfactory since a novel method is supposed to outperform existing methods.
- In fig.3, only original images and heatmaps are provided. I suggest the authors to present the pixel-level ground-truth labels to indicate the produced heatmaps are able to locate the lesion regions. Without pixel-level ground-truth labels, the readers do not know the location of lesion regions.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
-The authors declares they will share all the codes for the experiement when accepted.
- The authors also presented most of the relevant setting parameters for the expriments.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Please refer to the “main weaknesses” part.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper is well-written and easy to read. The motivaion and method are clearly demonstrated. However, there lacks of some crucial comparisons, such as comparison between the proposed attention and existing CBAM. Besides, the performance is not superior to existing CCT-Net.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
The authors’ responses have eliminated my concerns about the novelty and performance gains. I think this paper could be accepted if the authors could provide 5-fold cross validation results in the final version.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposes to utilize the relations among diseases for long-tailed multi-label retinal diseases classification. The main weakness of the paper is the technical novelty, especially compared with previous method in [16].
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6
Author Feedback
We sincerely thank all reviewers for their valuable and constructive feedback. A detailed response to each reviewer (R1, R2, R3) is provided as follows and we hope reviewers can change their minds and consider accepting our paper. Q1: Code release. (R1, R2) A1: We will share the codes and pretrained weights when our paper is accepted. Q2: The main idea is similar to [16] with limited technical novelty. (R1, R3) A2: Our work is motivated by [16], but there are still some differences and improvements in the methodology. Firstly, [16] proposed to generate relational subsets by clinical experts. This preprocess is experience-required and thus limiting its application. In contrast, our method uses a simple but effective clustering algorithm to obtain relational subsets. This can be easily expanded into other fields without strong prior knowledge. Secondly, we use transfer learning (i.e., multi-task pretraining) to transfer the common information from the head class (DR) to other rare classes. Thirdly, novel region-based attention is utilized to highlight the important regions related to diseases. Finally, [16] is more concerned about the long-tailed classification, while the multi-label cases are not well considered. In comparison, we proposed a new instance-label-wise weighted distillation loss to help reduce the multi-label effect when training the student model. Q3: The performance improvement is marginal compared to CCT-Net. (R1, R3) A3: We further extend our experiments with the focal loss to address the class imbalance, and the results of our method with focal loss are Kappa:0.765, F1:0.963, AUC:0.972, much better than CCT-Net (Kappa: 0.751, F1:0.951, AUC: 0.958). Q4: No pixel-level ground-truth labels are provided in Figure 3. (R1, R3) A4: The pixel-level ground truth will be added in the final version to help show the superiority of our method. Q5: The CAMs in Figure 3 are not useful to highlight the obtained results. (R2) A5: We will provide the pixel-level ground truth in Figure 3 to indicate the produced heatmaps can locate the lesion regions. Q6: CCT-Net is not explored using the ResNet-50 backbone. (R2) A6: The codes of CCT-Net are not publicly available and the architecture of CCT-Net is a little complex with three main submodules, each submodule contains dense blocks. We are not quite sure to reimplement it with the ResNet-50 backbone. Therefore, we just compare our model with the DenseNet-121 backbone to CCT-Net. Q7: No CAMs in healthy images are included in Figure 3. (R2) A7: Due to the limitation of space, the CAM in healthy images is not included in Figure 3. In the final version, we will add more visualization results including healthy images. Q8: Unclear expression about the clustering to generate subsets. (R3) A8: As described in Sec 2.4, we only conduct the clustering on six diseases (D, G, C, AMD, H, M). Firstly, we select samples whose labels are contained in these six diseases. Then we extract features and use K-Means to cluster them. In each cluster, we count the number of six diseases and use the maximum as the clustering results. For example, for D, 1541 samples in cluster1 and 260 samples in cluster2, therefore D is grouped to cluster1. As for the number of cluster centers, we tried several settings and find 2 is the best option. Q9: Comparison between proposed region-based attention and CBAM. (R3) A9: The results of CBAM are Kappa: 0.694, F1: 0.927, AUC: 0.930. From the results, we can see that our proposed region-based attention (Kappa: 0.710, F1: 0.933, AUC: 0.941) outperforms CBAM. Q10: 5-fold cross-validation to produce more solid results. (R3) A10: We will expand our experiments with 5-fold cross-validation in the final version.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The rebuttal has addressed the major concerns of the reviewers.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have addressed the reviewers’ concerns.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The major concern in the last round is the novelty of this paper. The author addressed this concerns well and the reviewer also raised the grade. Therefore, three reviewers has made a consus decision and I would suggest to accept this paper.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5