List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Vishwesh Nath, Dong Yang, Holger R. Roth, Daguang Xu
Abstract
Which volume to annotate next is a challenging problem in building medical imaging datasets for deep learning. One of the promising methods to approach this question is active learning (AL). However, AL has been a hard nut to crack in terms of which AL algorithm and acquisition functions are most useful for which datasets. Also, the problem is exacerbated with which volumes to label first when there is zero labeled data to start with. This is known as the cold start problem in AL. We propose two novel strategies for AL specifically for 3D image segmentation. First, we tackle the cold start problem by proposing a proxy task and then utilizing uncertainty generated from the proxy task to rank the unlabeled data to be annotated. Second, we craft a two-stage learning framework for each active iteration where the unlabeled data is also used in the second stage as a semi-supervised fine-tuning strategy. We show the promise of our approach on two well-known large public datasets from medical segmentation decathlon. The results indicate that the initial selection of data and semi-supervised framework both showed significant improvement for several AL strategies.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16452-1_29
SharedIt: https://rdcu.be/cVRZb
Link to the code repository
N/A
Link to the dataset(s)
Reviews
Review #1
- Please describe the contribution of the paper
The manuscript deals with the problem of a cold start in active learning. The problem addressed here is which sample we shall start labeling given a set of unlabeled data.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The manuscript deals with an interesting research topic, i.e., how to select samples for being labeled by the user. The manuscript is well written (primarily). The main idea is to use a self-supervised approach to pseudo-label the data. Then, a fully supervised training procedure takes place for further fine-tuning under a semi-supervised learning framework. That final model is used to select data for annotation. Pseudo labels are created by thresholding and other standard image operators. The proposed approach is good as well. Several and different concepts are used to compose the final methodology.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The manuscript suffers from several weaknesses. First, the figures are small and have a poor resolution. Secondly, the experimental section does not compare the proposed approach against others in the literature. The authors could consider similar approaches that have been proposed for other applications and try them for 3D medical data.
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
It seems reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
I recommend the authors improve the paper presentation. Although they provided an algorithm to describe the proposed approach better, which is nice, they failed in providing a more robust experimental section.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Proper paper presentation and a more robust experimental section are lacking.
- Number of papers in your stack
3
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
4
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
This work applied self-supervised training for cold-start and semi-supervised training for active learning. The pseudo-labels in self-supervised task are collected by thresholding and the task is to perform binary segmentation. The pseudo-labels in the semi-supervised task are collected from the trained network. During the process, images need to be ranked based on network’s uncertainty using MC. The proposed method outperformed the baselines on two datasets.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The proposed method is simple and straightforward. The results from the figures are strong, meaning at least one setting is outperforming the baselines.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
It is not clear what are the different settings compared in the experiments. Especially, how the variance and entropy methods are used together with semi-supervised method. In algorithm 1, line 6, 10, 12, there are three places where data needs to be selected and the comparison/ablation study setting is not clear.
“proxy ranking” has also be introduced but the exact definition is not clear.
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors state that the code will be shared. The data is also public.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Explaining the different setting better and uniform the setting name between figures and tables would largely increase the paper’s clarity. In sec 2, it may be better to provide some arguments or related work for the claim that mean teacher and shape constraints are not ideal for AL. The figures 2 & 3 are not easy to read, may be better to increase font size of the legend.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The method is good. The results are good. Explaining the experiment setting should be minor changes.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
The paper presents a method to label 3D medical image volumes without any labelled data to begin with. A proxy segmentation task on pseudo-labelled data provide uncertainty measure to rank unlabelled data for initial annotation. Following this, a two stage combination of supervised segmentation model and a semi-supervised fine-tuning model take over selection of unlabelled data to be annotated next.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Use of uncertainty from some proxy task to tackle cold-start problem is an interesting take of the problem.
- The build up of method from initial to no labels to using unlabelled data to select for annotation is applicable to many medical image data in clinical setting.
- Extendability to other tasks - Changing the pseudo label process and proxy task could help implement the same method to other medical image modalities.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
It seems that from figure 2, the improvement / performance gain of semisupervised method with proxy is tied to choice of data and acquisition function. For example, the ProxyRank-Semi performed well above supervised method and other semi-supervised method (e.g. entropy) for Liver & Tumor but for Hepatic Vessel data, the ProxyRank-Semi came close to ProxyEnt and at the end of fourth round was well below it as well as ProxyRank-EntSemi
-
Limited discussions
- From figure 3 (eg. 3C and 3F), the combination of gain of semi-supervised method with proxy ranking was seen more on Hepatic dataset than liver and tumor which could have been explored and explained better.
- Semi-supervised added with proxy better only for early rounds (fig 3C)?
-
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors have satisfied the reproducibility checklist.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- The method is three-staged at this point i.e. proxy ranking, supervised, unsupervised. One future direction could be making the method end-to-end
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The presented method proposed a methodology that tackled the problem at hand i.e. cold-start annotation / active learning and exploitation of unlabelled data for better annotation candidates
- The paper is well written but some parts of the results could have had a better exploration and discussion
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This work addresses the cold start problem of active learning techniques. The framework uses self-supervised learning to generate pseudo-labels of a proxy task, allowing to rank the unlabeled data for annotation. Following this, a two stage framework consisting of a supervised segmentation step and a semi-supervised fine-tuning step that allows selection of the unlabeled data to be annotated with in an active iteration. The reported results are good.
The reviewers agree on the relevance of the problem tackled and the good obtained results. However, they also agree that the paper lacks clarity in several sections, in particular, the experiments section, that require improvement or further discussion. I consider these points could be addressed in the rebuttal. Please revise carefully the points raised as weaknesses by the different reviewers.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
7
Author Feedback
Meta-Reviewer 2 ( MR2)
Thank you for providing constructive feedback to our work. We appreciate the reviewers & the meta-reviewer for acknowledging the relevance of the problem that we are addressing. As per MR2 : “The reported results are good. The reviewers agree on the relevance of the problem tackled and obtained good results.”
We address the 5 major concerns raised by the reviewers below:
Concern 1: Paper Presentation & Alg Clarity (R1, R2)
We will enlarge the figures,improve their resolution, and also increase the size of the text in the plot legends. Regarding the naming of the approaches we will ensure that the names from captions of Fig. 2 and the naming in Table 1 are consistent. Additionally, we will add the same naming in the text.
Proxy ranking means that “uncertainty was generated for all unlabeled data by the proxy model M_p, thereafter the data was sorted for selection based on uncertainty generated by M_p per data point.” This will be added to the paper.
For Alg. 1, lines 6, 10 and 12, we will improve the clarity by introducing dedicated symbols for the “data selection part” & also introduce a table to ensure the clarity of the ablation study.
Concern 2: Lacking comparisons against baselines (R1)
The beginning of the sub-section “SSL AL with Proxy Ranking” contains all the information regarding baselines: “To gauge the benefit of the combined strategies of semi-supervised learning (SSL) with proxy ranking, we compare them with random based acquisition and also with two well-known active learning (AL) acquisition functions of entropy and variance [6,5,16, 30].” The entropy and variance are the most commonly used baselines in the AL literature & they have been used for both 2D [30] & 3D [16] medical imaging tasks. The citations will be added to captions of Fig. 3 & 2 to clarify that we already compare to these common baselines. The main objective of our work was to show that proxy ranking and semi-supervised learning are beneficial to basic components of AL. It would be out of the scope to compare with other SSL methods.
Concern 3: Improve the experiment design (R1, R2)
We will improve this section by providing a summary table that shows the settings for each experiment in our ablation study as stated in concern 1; further illustrating how the baselines of entropy and variance were also used by us as mentioned in concern 2 as well. We will also simplify the experiment section by breaking it down into further sub-sections for ease of understanding. To improve upon the interpretation of the results, we’ll add to figure captions that “solid lines are the baselines in our experiments, the dashed lines are our proposed methods” for Fig 2 & 3.
Concern 4: More detail on results (R3)
Regarding results of Fig. 2, the proxy ranking primarily provides the benefit for the initial AL rounds in both datasets. Our goal was to show that the maximum advantage is achieved by combining proxy ranking with SSL into AL. For every dataset, one can tune the type of acquisition function and choose from one of the proposed ones as AL is often sensitive to the specific task.
Results of Fig. 3 (C & F), yes the benefits are more for hepatic vessels dataset, as that is a relatively more challenging task. This shows that AL and SSL provide more benefit towards challenging tasks. We also explored more active iterations for hepatic vessels as shown in the supplementary material, where the results portray the superiority of our proposed method.
Concern 5: Discussion & Related work improvement (R2, R3)
We mentioned shape constraints as being “not ideal” because they are unlikely to work well for objects with irregular boundaries, e.g., tumors. It is difficult to model hepatic vessels with shape models as the topology of the vessel varies from case to case.
Student-teacher as a SSL technique for AL, brings a burden of tuning more hyper-parameters as it require 2 models.
We will integrate all points raised above to Discussion.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have addressed the remarks raised by the first round of reviews with clarity. The paper presents interesting ideas that could be worth of discussion, so I recommend acceptance.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Paper strengths: 1) Interesting research topic 2) Paper is well written 3) Strong results
Paper weaknesses: 1) Unclear settings in the experiments (variance and entropy methods) 2) Undefined proxy ranking 3) The improvement / performance gain of semisupervised method with proxy is tied to choice of data and acquisition function 4) Limited discussions
The rebuttal addresses most of the issues above, so I’m recommending the paper to be accepted.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
8
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper presents self-supervised training for cold-start and semi-supervised training for active learning, which are straightforward and effective. In the rebuttal, the authors included intensive clarification to well address the initial concerns raised by the reviewers. I recommend accept.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
1