List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Minmin Liu, Xuechen Li, Xiangbo Gao, Junliang Chen, Linlin Shen, Huisi Wu
Abstract
Due to the difficulty of cancer samples collection and annotation, cervical cancer datasets usually exhibit a long-tailed data distribution. When training a detector to detect the cancer cells in a WSI (Whole Slice Image) image captured from the TCT (Thinprep Cytology Test) specimen, head categories (e.g. normal cells and inflammatory cells) typically have a much larger number of samples than tail categories (e.g. cancer cells). Most existing state-of-the-art long-tailed learning methods in object detection focus on category distribution statistics to solve the problem in the long-tailed scenario, without considering the ``hardness’’ of each sample. To address this problem, in this work we propose a Grad-Libra Loss that leverages the gradients to dynamically calibrate the degree of hardness of each sample for different categories, and re-balance the gradients of positive and negative samples. Our loss can thus help the detector to put more emphasis on those hard samples in both head and tail categories. Extensive experiments on a long-tailed TCT WSI image dataset show that the mainstream detectors, e.g. RepPoints, FCOS, ATSS, YOLOF, etc. trained using our proposed Gradient-Libra Loss, achieved much higher (7.8%) mAP than that trained using cross-entropy classification loss.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16434-7_11
SharedIt: https://rdcu.be/cVRq9
Link to the code repository
https://github.com/M-LLiu/Grad-Libra
Link to the dataset(s)
https://github.com/M-LLiu/Grad-Libra
Reviews
Review #1
- Please describe the contribution of the paper
- To deal with existing problems in datasets with a long-tailed distribution, a novel concept of sample hardness is introduced in order to help improve the performance of gradient loss. The hardness is calibrated at sample level with the gradient.
- Set in a cancer cells detection task, a novel gradient loss is presented where samples are re-weighted according to their hardness, achieving better gradient balance and much higher mAP than cross-entropy loss, surpassing other SOTA methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- An interesting method to dealing with long-tailed datasets. A complete derivation for the loss function is given and the basic idea behind it is well explained.
- A proper dataset is created to imitate the real-life situations, which is large-scaled and has a strong data imbalance.
- The impact of the hyper-parameters in the loss function is separately studied and a proper analysis explained their significant contribution.
- The Grad-Libra loss is applied to other detectors, showing its generalization ability.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The rule of choosing hyper-parameters of the loss is vague.
- Lack of axis labels in Fig.4.
- The scale of the dataset is missing, hence it’s doubtful that the quantity of rare samples is large enough to make the experiment results convincing.
- More detail should be provided about how the variable g is gained in Eq.3. Where do the output logits come from?
- Lack of insightful analysis. The idea is somewhat similar to Focal Loss, how’s the nature different from that of Focal Loss?
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The dataset didn’t go public, but the implementation details of the main detector are described, so it would be reproduced should the dataset be accessible. The details of other detectors (FCOS, ATSS, YOLOF) are not provided.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
See 5.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The method is novel and the experiment results prove good. But the rationality of gradients being related to sample hardness needs insightful explaining. And the scale of the dataset needs to be clarified. I‘d consider raising my score on this paper should proper explanation be provided.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
The response answers in part my concerns, e.g., the number of rare samples, the relationship with Focal loss. I would like to raise my score although I still have concerns about the rationality of gradients related to sample hardness.
Review #2
- Please describe the contribution of the paper
In this work, the authors propose a method to take the “hardness” of each sample into account and rebalance the gradients of positive and negative samples. Specifically, Grad-Libra Loss is proposed to help put more emphasis on hard samples in both head and tail categories. Experiments show that 7.8% mAP improvement is achieved on TCT WSI image dataset using the proposed loss function.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
(1) The proposed Grad-Libra loss is relatively simple and intuitive. (2) The paper is well written and easy to follow with. (3) Extensive experiments are performed to demonstrate the effectiveness of the proposed loss function on different detectors, both head and tail classes etc.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
(1) There is a lack of systematic methods to determine the parameters \alpha+ and \alpha-. As discussed by the author, \alpha+ and \alpha- are important parameters to tune in order to achieve satisfactory effect, it would be useful to see how these parameters can be determined in a systematic way. (2) Minor: isn’t equation (3) a monotonically increasing function depending on g?
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The experimental dataset is not public available neither with the code. It would be preferable if the author can publish their dataset and the code upon the acceptance of the paper.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
(1) It would be great if the author can provide systematic way to determine the hyper-parameters used in their method. (2) It would be preferable if the author can publish their dataset and the code upon the acceptance of the paper, so the work can be fully reproducible.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
There is few work focusing on the hardness of the samples in the classification task. This work provides a unified framework to take the hardness into account for both head and tail classes. The extensive experiments show the effectiveness of their methods especially for the rare classes.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
Thanks to the authors for providing the detailed response. It is great to see the authors promise to release the dataset so that their work can be fully reproducible. It is understandable that it’s not an easy job to provide a systematic method to determine the hyper-parameter, though I still feel it will be beneficial for the method to be widely applied. Nevertheless I will keep my score unchanged based on the performance improvement by the proposed method.
Review #3
- Please describe the contribution of the paper
1.This paper provides a new loss re-weighting method for tackle long-tailed problem by utilizing gradient information of each sample. 2.This paper proposes to use gradient information to measure the hardness of each sample in long-tailed learning problem. 3.The method of this paper achieves better results than previous methods for tackling long-tailed problems.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The experiments results are comprehensive and the qualitive analysis precisely illustrate the improvement. 2.Author uses gradient information of each sample to re-weight the cross-entropy of each class so that the method can solve long-tailed at sample level. This is an innovative loss re-weighting method. 3.This paper firstly summarizes the existing scheme for solving long-tailed problem and then this paper clearly claims what sets this work apart from other works. 4.Illustrated explanation about how the method works.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1.The performance with different value of modulating factor alpha has obvious fluctuations. The value selection of modulating factor alpha requires more detailed discussions and verification; 2.The method seems almost irrelevant to medicine. This method doesn’t utilize any medical prior or clinical knowledge; 3.Although this method provides an amazing performance boost and shows comprehensive benchmark results, this method is only benchmark on private dataset. The experiment results may be a little bit incredible;
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
1.The codebase of this paper is based on mmdetection. So, the code is well-structured and the results can be easily reproduced; 2.The collection of long-tailed dataset used by this paper is time-consuming. As far as I know, there is not a public dataset about long-tailed cell detection. However, the experiment results are totally dependent on the private dataset. If this dataset will not be open-sourced, the reproducibility of this paper will be greatly reduced.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1.Fig2 provide a visualization example about how the grad-libra loss works. But the weights after F are illustrated unclear by colorful circle. Expressing the weights after F as concrete math number may be better. 2.It is better to provide visualization results of cell detection which can make your paper more impressive.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1.Clear description about how the method works; 2.Clear states about how it differs from existing methods; 3.Amazing performance improvements;
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper presents a cervical cell detection method focusing on long-tailed distribution. A gradient libra loss is proposed, which is applied on MMDetection and other detectors, and compared against other loss functions. A private dataset containing many annotated cells of a long-tailed distribution is used for testing and improved performance is obtained. Reviewers are generally positive about the paper but there are many detailed questions regarding the method design and result evaluation. Also, the dataset description is very confusing. It seems only Table 1 has listed the various cell categories but not anywhere else. What’s the data distribution among these categories? How many whole slide images are acquired exactly? How many patches are cropped for experiments? Are the data distributions following a similar long-tailed trend in all train/val/test subsets? What’s the clinical significance of detecting all these different types of cells? Are some of the categories more important in terms of recall or precision? And how were the cells annotated, using bounding boxes, centroids, etc.?
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
4
Author Feedback
We sincerely thank all of the reviewers for their comments and their acknowledgment of the novelty, methodology, and reproducibility of our work. Our work provides a unified [R2] and innovative [R3] framework to take the hardness into account for both head and tail classes. Our method is simple, intuitive [R2], and fully experimental [R1-R3], showing its generalization ability R1 and amazing performance improvement [R3]. We address the key concerns below and will further improve our paper.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper presents a cervical cell detection method focusing on long-tailed distribution. There were some confusion in method and dataset description and the rebuttal has addressed most of the comments. Moreover, in the rebuttal, the authors have promised to release the dataset, which is in fact one of the main points the reviewers have highlighted. The final version should include the changes made based on reviewers’ comments.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper targets the long-tailed datasets, and proposes a novel loss function by re-weighting the samples according to their hardness. The gradients are then balanced and the performance is improved. The addressed problem is practical in real world, and the proposed method that tackles it from hardness samples is promising. The rebuttal has addressed most of the reviewers’ concerns. It is good that the authors will release the dataset to ensure reproducibility and benefit the community.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper introduces a concept of sample hardness and reweights its contribution accordingly using a novel gradient loss. It uses the method on a cervical cell detection problem with long-tailed distribution and improves the detection results. The rebuttal has addressed most of the reviewers’ concerns in terms of paper presentation and technical details. It is also nice that the author will release the code, as I believe the problem is rather general and it has potential to apply to other problems as well.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6