Authors

Haifan Gong, Hui Cheng, Yifan Xie, Shuangyi Tan, Guanqi Chen, Fei Chen, Guanbin Li

Abstract

Thyroid nodule classification aims at determining whether the nodule is benign or malignant based on a given ultrasound image. However, the label obtained by the cytological biopsy which is the golden standard in clinical medicine is not always consistent with the ultrasound imaging TI-RADS criteria. The information difference between the two causes the existing deep learning-based classification methods to be indecisive. To solve the Inconsistent Label problem, we propose an Adaptive Curriculum Learning (ACL) framework, which adaptively discovers and discards the samples with inconsistent labels. Specifically, ACL takes both hard sample and model certainty into account, and could accurately determine the threshold to distinguish the samples with Inconsistent Label. Moreover, we contribute TNCD: a Thyroid Nodule Classification Dataset to facilitate future related research on thyroid nodules. Extensive experimental results on TNCD based on three different backbone networks not only demonstrate the superiority of our method but also prove that the less-is-more principle which strategically discards the samples with Inconsistent Label could yield performance gains. Source code and data are available at https://github.com/chenghui-666/ACL/.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16440-8_24

SharedIt: https://rdcu.be/cVRvP

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper aims at improving the accuracy of the thyroid nodule classification. The existed methods can’t fit well with the inconsistent information between the label obtained by the cytological biopsy and the ultrasound imaging TI-RADS criteria, so the author propose an adaptive curriculum learning framework, which adaptively discovers and discards the samples with inconsistent labels. Moreover, the authors contribute a new thyroid nodule classification dataset to facilitate future related research on the thyroid nodule.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The motivation of this paper is innovative and attractive. There are few papers about solving the inconsistent label issue between FNA and TI-RADS.
2. The key innovations of this work include designing a function to discriminate the hardness of a sample, and designing a general learning strategy. There are few works about applying curriculum learning on thyroid nodule classification.
3. The authors contribute a new thyroid nodule classification dataset.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The author should supplement more information about the contributed dataset, such as the source of data, device(s) used, image acquisition parameters, instructions to annotators, and methods for quality control.
2. The paper writing can be further improved.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The author should supplement more information about the contributed dataset, such as the source of data, device(s) used, image acquisition parameters, instructions to annotators, and methods for quality control.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

The paper writing can be further improved. (1) In the last paragraph of Introduction, the “;” before “(2)” should be changed to ”.” so as to unifying the formatting. (2) There should be punctuation after each formula. (3) In the first paragraph of 3.3, the left colon for the phrase ”less is more” should be corrected. (4) The backbone names in Table.2 are better changed to uppercases, such as ResNet, VGG, and DenseNet.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The motivation of this paper is innovative and attractive.
Number of papers in your stack

5
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

This paper presents a model for adaptive curriculum learning that can adaptively eliminate the samples with label uncertainty from the prediction function. The paper contributes a dataset to evaluate the model on.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Novel architecture
- Can be used with different backbone networks
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Some unclarities in the methodology
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Dataset will be available Code will be available
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

1- “Let µ and σ be the average value and standard deviation of elements in Qh” You may clarify the intended value to avoid mis-understanding the confidence value vs the number of elements in the queue Qh 2- I didn’t get how can Ci (confidence) will be limited from 0.5 to 1, since as per Algorithm 1, the confidence uses the probability after SoftMax which can go outside the 0.5-1 range? 3- Could you please elaborate more how can the probability produced from the DNN represents the confidence of the samples 4- I mean, that T=the two queues Qc and Qh use the same shared prediction probability, however they are by definition representing two different concepts. 5- What is the network that generate yi and yi’ in Equation 8? How this is related to P of Algorithm 1 and Equation 1 and 2. I understand that there is a network that should estimate the Ci independently from the backbone network, but the architecture of this network is not clear
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

For the conceptual innovation, novel architecture and dataset
Number of papers in your stack

5
What is the ranking of this paper in your review stack?

2
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

This paper proposed an adaptive curriculum learning for thyroid nodule classification based on ultrasound images. The proposed confidence queue and certainty queue can help the model pick out hard samples. Moerover, the authors provide a new dataset for thyroid nodule classification. The experimental results show the superiority of the proposed method against SOTA methods.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The proposed adaptive threshold function T_{ada} can effectively reflect the predicting state of hard samples during training, and this has been illustrated in the experiment. The quantitative results show the superiority of the proposed method.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The motivation of the paper is to solve the inconsistency between labels of dataset and the ground truth of biopsy. However, I cannot see any biopsy informations during description of method, e.g., in Eq.(2), there are predictions and the label information, respectively. Please clarify this confusion.
2. What is the setting of Baseline method in experiments?
3. How the model perform on the training setting without discarding hard samples? This will help exhibit the superiority of the proposed method.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

I think this paper can be reproduced according to the proposed training algorithm. It will be better to provide the new dataset used in this paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1. Proofreading is needed, e.g., in Page 5, “we propose to embeded” –> “….to embed”
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. The new dataset which contains a large amount of samples and the corresponding biopsy labels.
2. The training algorithm is clear and this guarantees the reproducibility.
3. The better performances obtained by this paper.
Number of papers in your stack

5
What is the ranking of this paper in your review stack?

3
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

There are three high quality reviews pointing to strength of this paper and recommending Accept. I also agree that the focus on Curriculum Learning enabling the use of domain and knowledge, the generalization with flexible backbone, and the well crafted manuscript make this a strong submission. There is also a new dataset.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

1

Author Feedback

N/A

back to top

Less is More: Adaptive Curriculum Learning for Thyroid Nodule Diagnosis