Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews Back to top
List of Papers By topics Author List
Paper Info Reviews Meta-review Author Feedback Post-Rebuttal Meta-reviews

Authors

Chaoyi Li, Meng Li, Can Peng, Brian C. Lovell

Abstract

This paper presents an innovative approach to curriculum learning, which is a technique used to train learning models. Curriculum learning is inspired by the way humans learn, starting with simple examples and gradually progressing to more challenging ones. There are currently two main types of curriculum learning: fixed curriculum generated by transfer learning, and self-paced learning based on loss functions. However, these methods have limitations that can hinder their effectiveness. To overcome these limitations, this article proposes a new approach called Dynamic Curriculum Learning via in-domain uncertainty (DCLU), which is derived from uncertainty estimation. The proposed approach utilizes a Dirichlet distribution classifier to obtain prediction and uncertainty estimates from the network, which can be used as a metric to quantify the difficulty level of the data. An uncertainty-aware sampling pacing function is also introduced to adapt the curriculum according to the difficulty metric. This new approach has been evaluated on two medical image datasets, and the results show that it outperforms other curriculum learning methods. The source code for this approach will be released at https://github.com/Joey2117/DCLU.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43904-9_72

SharedIt: https://rdcu.be/dnwIk

Link to the code repository

https://github.com/Joey2117/DCLU

Link to the dataset(s)

N/A

Reviews

Review #3

  • Please describe the contribution of the paper

    The authors have proposed the use of evidence-based uncertainty as a measure of data difficulty in a curriculum learning setting to rearrange the data from easy to hard in each epoch and let the network learn from the most certain (easier) samples first. They have evaluated their method on two datasets and compared with a number of curriculum learning methods outperforming those approaches.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • Novel approach for curriculum learning: introducing uncertainty as a difficulty measure
    • The approach is generalizable and can be used with any kind of medical data.
    • Well-written and easy to follow text: The language is simple (although needs to be polished for some minor errors) and there are multiple subsections that make the paper more understandable.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    • The method is only compared to other curriculum learning methods and this does not show the need for these complicated approaches.
  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
    • Authors have mentioned that they will make their code public. Also they have shared a pseudo-code in the Appendix and there are enough details about the implementation in the text.
  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
    • Authors could compare the performance of their model to simpler baselines to justify going through more complicated methods like curriculum learning approaches.
    • Figures should be self-contained. Captions should describe everything that is seen in the figure.
    • Text should be edited for language and punctuation.
  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    6

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The authors have proposed a general and novel approach.

  • Reviewer confidence

    Somewhat confident

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #1

  • Please describe the contribution of the paper

    The proposed method utilizes in-domain uncertainty to measure the difficulty of data and employs an uncertainty-aware sampling pacing function to present training data during the learning process. The dynamic curriculum learning approach improves the training process by presenting data in a more effective manner, allowing the model to learn more efficiently and effectively.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    This paper proposed the novel approach for curriculum learning in medical image classification, the use of in-domain uncertainty to measure the difficulty of data, and the uncertainty-aware sampling pacing function to present training data effectively. The experimental results show that the proposed approach outperforms existing curriculum learning methods and effectively reduces uncertainty, which is crucial for medical applications where accurate diagnosis is essential. This paper provides a valuable tool for medical professionals in their work, and the proposed approach has the potential for further development and optimization. The proposed method is evaluated on two medical image datasets, and the results demonstrate that it outperforms existing curriculum learning methods. Additionally, the method effectively reduces uncertainty, which is crucial for medical applications where accurate diagnosis is essential. The paper includes sufficient experiments and ablation study to analyze the impact of key components on the performance of the model.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    Although the paper proposes using in-domain uncertainty to measure data difficulty, it does not thoroughly define or explain how this uncertainty is measured. For instance, does it include abnormalities in medical imaging or out-of-distribution data, such as rare cases?

    The way to choose the dataset is tricky. How to define the training, validation and testing, how much uncertainty should be included, and how to prevent the information leaking to training and validation?

    The pacing function is tied to the adjustment of the difficulty measurer. Specifically, how should we define the first and second phases in the training process? The network should prioritize learning hard examples to improve training. How can this process be optimized to leverage these two components?

    Minor: The font in Table 2 is too small.

  • Please rate the clarity and organization of this paper

    Excellent

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    The author claimed to release the code and data necessary to reproduce the work. The paper’s description is also sufficient to do so.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

    Please provide more details for the questions mentioned in limitations (section 6).

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    5

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper uses in-domain uncertainty, uncertainty-aware sampling pacing function, and rigorous evaluation make it a contribution to the field of medical image classification and curriculum learning. The method is practical and robust to apply to different user cases.

  • Reviewer confidence

    Somewhat confident

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #2

  • Please describe the contribution of the paper

    This paper proposed a new approach called Dynamic Curriculum Learning via in-domain uncertainty(DCLU) to achieve curriculum learning. This approach Utilizes a Dirichlet distribution classifier to obtain prediction and uncertainty estimates from the network as a metric to quantify the difficulty level of the data and an uncertainty-aware sampling pacing function to adapt the curriculum according to the difficulty metric. The proposed pipeline achieved promising results in two medical image datasets.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. The DCLU provides a dynamic measure during the training process. The author uses a Dirichlet distribution classifier to provide uncertainty estimates and predictions simultaneously, eliminating the need for additional networks for evaluation.
    2. The experiment was reasonable and sufficient.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    1. The authors mention that difficulty measuring machines based on loss functions may be difficult to quantify at an early stage due to inadequate training. Then whether the dynamic difficulty measurer(DDM) has the same problem.
    2. The author proposed an Uncertainty-aware sampling pacing function(UAS) to specify simple samples for training. But in the experimental part, it seems to prove that better results can be obtained by using the full dataset.
  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    The paper is reproducible as the authors will provide the code and the experiment involved two publicly available datasets.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

    In response to the shortcomings presented in 6, I have the following suggestions:

    1. Discussion of DDM from the theoretical or early training performance.
    2. Explain UAS more clearly to avoid misunderstandings.
  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    5

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper is generally understandable and the research topic is valuable, and the experimental results are convincing. But its technicalities still need to be debated.

  • Reviewer confidence

    Confident but not absolutely certain

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Primary Meta-Review

  • Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    Strengths: good technical novelty for the use of in-domain uncertainty to measure the difficulty of data in curriculum learning; superior performance seen in sufficient experiments with ablations studies; the proposed method can be extended for other medical applications;

    Weaknesses: Lack some of important details on how to define phases in the training processes, how to define training/validation/testing data, and how to measure uncertainty in relation to rare diseases and abnormalities; lack discussion of the method with regards to its early training performance; lack results of simple baseline methods to justify needs of curriculum learning.



Author Feedback

We sincerely thank all reviewers and Area Chair for your time and comments. In the final version, we will (1) describe the details on how to define training/testing data, define or explain how in-domain uncertainty is measured, and better clarify details on how to define phases of UAS in the training processes (for R1); (2) provide results of experiments of our method in the early training process to indicate our method is more stable than other curriculum learning methods based on a loss function at an early training stage (for R2); (3) clarify that we already have compared with a simple baseline method that is vanilla in our results of experiments, improve figure and caption (for R3).



back to top