List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Longxiang Tang, Kai Li, Chunming He, Yulun Zhang, Xiu Li
Abstract
This paper studies source-free domain adaptive fundus image segmentation which aims to adapt a pretrained fundus segmentation model to a target domain using unlabeled images. This is a challenging task because it is highly risky to adapt a model only using unlabeled data. Most existing methods tackle this task mainly by designing techniques to carefully generate pseudo labels from the model’s predictions and use the pseudo labels to train the model. While often obtaining positive adaption effects, these methods suffer from two major issues. First, they tend to be fairly unstable - incorrect pseudo labels abruptly emerged may cause a catastrophic impact on the model. Second, they fail to consider the severe class imbalance of fundus images where the foreground (e.g., cup) region is usually very small. This paper aims to address these two issues by proposing the Class-Balanced Mean Teacher (CBMT) model. CBMT addresses the unstable issue by proposing a weak-strong augmented mean teacher learning scheme where only the teacher model generates pseudo labels from weakly augmented images to train a student model that takes strongly augmented images as input. The teacher is updated as the moving average of the instantly trained student, which could be noisy. This prevents the teacher model from being abruptly impacted by incorrect pseudo-labels. For the class imbalance issue, CBMT proposes a novel loss calibration approach to highlight foreground classes according to global statistics. Experiments show that CBMT well addresses these two issues and outperforms existing methods on multiple benchmarks.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43907-0_65
SharedIt: https://rdcu.be/dnwdM
Link to the code repository
https://github.com/lloongx/SFDA-CBMT
Link to the dataset(s)
https://refuge.grand-challenge.org/
http://medimrg.webs.ull.es/research/retinal-imaging/rim-one/
https://cvit.iiit.ac.in/projects/mip/drishti-gs/mip-dataset2/Home.php
Reviews
Review #5
- Please describe the contribution of the paper
This paper1 studies source-free domain adaptive fundus image segmentation which aims to adapt a pretrained fundus segmentation model to a target domain using unlabeled images. The paper proposes the Class-Balanced Mean Teacher (CBMT) model which addresses the unstable issue by proposing a weak-strong augmented mean teacher learning scheme where only the teacher model generates pseudo labels from weakly augmented images to train a student model that takes strongly augmented images as input. The teacher is updated as the moving average of the instantly trained student, which could be noisy. This prevents the teacher model from being abruptly impacted by incorrect pseudo-labels. For the class imbalance issue, CBMT proposes a novel loss calibration approach to highlight foreground classes according to global statistics.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- the article is well-written and easy to follow
- the teacher-student learning method is interesting for discussing its application in optic disc and cup segmentation
- the results are impressive
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
the datasets and benchmarks used in the paper are too easy. The training set and test set of REFUGE are collected from the same domain. please use REFUGE-2[1], which the test test come from another domain instead. Also the paper has not compared to recent SOTA OD/OC segmentation methods[2][3][6]. It is not saying a paper is unacceptable if it is not SOTA, different methods are suitable in different scenarios. But the authors should give them a comparison or discussion of the differences.
-
The pixel-level accuracy of optic disc and cup segmentation is objective, what’s the performance of its results used for the glaucoma classification? As REFUGE2 provide both glaucoma classification label and OD/OC segmentation labels, authors could easily verify the result.
-
Authors have not discussed other related works. There aren’t many teacher-student based methods for OD/OC segmentation[4][5], but the authors discussed none of them. Plus, authors used a calibrated loss tech but not discuss the differences between the current calibration technologies[6][7][8] in this field.
[1]”Refuge2 challenge: Treasure for multi-domain learning in glaucoma assessment.” arXiv preprint arXiv:2202.08994 (2022). [2] “MedSegDiff: Medical Image Segmentation with Diffusion Probabilistic Model.” MIDL 2023 [3] “Opinions Vary? Diagnosis First!.” MICCAI 2022 [4] “CADA: Multi-scale Collaborative Adversarial Domain Adaptation for unsupervised optic disc and cup segmentation.” Neurocomputing 469 (2022): 209-220. [5] “Leveraging undiagnosed data for glaucoma classification with teacher-student learning.” MICCAI 2020 [6] “Learning calibrated medical image segmentation via multi-rater agreement modeling.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. [7] “Calibrate the inter-observer segmentation uncertainty via diagnosis-first principle.” arXiv preprint arXiv:2208.03016 (2022).] [8] “Learning self-calibrated optic disc and cup segmentation from multi-rater annotations.” MICCAI 2022
-
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
yes
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
address the concerns in point 6
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
interesting idea with well-written article, but need to supply more verifications and discussions, see my point 6.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #4
- Please describe the contribution of the paper
Authors propose a fundus image segmentation method applicable to different domains. The proposal consists of two segmentation models where one (the teacher) generates weak segmentation and help the other (the student) to obtain stronger segmentations. Also, the student provides a bit of feedback to the teacher model. Compared to other segmentation models, this work has the ability of be trained on unlabeled data preventing the teacher model from being abruptly impacted by incorrect pseudo labels. Authors validate the method comparing different datasets and state-of-art methods, including an ablation study.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Simplicity: the main strength is that the algorithm is quite simple. The method consist on two identical models trained together where the back-propagation is passed through weighted losses according to global segmentation knowledge.
- Extendibility: The method can be applied to any deep architecture and to any other type of dataset.
- Validation: the proposed model has been validated with several fundus datasets, 7 different segmentation models, and also an ablation study is done to check the best performance of the model parameters.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Discussion and conclusions: the discussion of the results and the conclusions are very short. It would be better if reduces a bit the method section removing the comments about how people address the problem and include a better discussion of the results.
- Justification: the model was applied for fundus images, but there is no reason to be used with any other type of datasets. Authors should justify better the applicability of the method on medical images compared to others.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- Models and algorithms: the models are mathematically correct, and descriptions of the model parameters used are provided, which helps for the reproducibility.
- Datasets: They used cited public datasets.
- Code: no code was attached.
- Reported experimental results: tables are very descriptive and include the essential metrics for evaluation and comparison.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- The proposal is very good detailed mathematically and validated with many experiments. I would improve the discussion explaining better the comparisons with the other methods.
- Include future works in the conclusion section.
- There are some weird sentences, please revise them:
- “We update the student model by back-propagation the loss defined in…”
- “Since most predictions are usually highly confident (very close to 0 or 1), and thus less informative. We need to only include pixels with relatively…”
- The acronym “EMA” is not defined.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The contribution of the paper is good, and the experiments validate the findings. In addition, the methodology is correct and novel. Thinking just on the medical part, more justification is needed to say that the proposal is only focused on medical images.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #1
- Please describe the contribution of the paper
to propose the Class-Balanced Mean Teacher (CBMT) model to address the unstable issue by proposing a weak-strong augmented mean teacher learning scheme where only the teacher model generates pseudo labels from weakly augmented images to train a student model that takes strongly augmented images as input. To proposes a novel loss calibration approach to highlight foreground classes according to global statistics
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
We propose the weak strong augmented mean teacher learning scheme to address the stable issue of existing methods. We propose the novel global knowledge-guided loss calibration technique to address the foreground and background imbalance problem. Our proposed CBMT reaches state-of-the-art performance on two popular benchmarks for adaptive fundus image segmentation
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
not so much
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
not so much
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
not so much
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
good study.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
- The paper introduces the Class-Balanced Mean Teacher (CBMT) model for source-free domain adaptive fundus image segmentation, where a pretrained model must adapt to a target domain with only unlabeled target images.
- CBMT addresses the challenges of noisy pseudo labels and class imbalance through a weak-strong augmented mean teacher scheme and loss calibration approach.
- Experimental results demonstrate that CBMT outperforms existing methods on multiple benchmarks.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper has a clear motivation to address the challenges of dealing with noisy pseudo labels and class imbalance in source-free domain adaptive fundus image segmentation.
- The paper effectively demonstrates how the weak-strong augmented mean teacher can be used to reduce the impact of noisy pseudo labels on the model, through consistency regularization.
- The paper provides fair comparisons between the proposed method and existing approaches, along with a detailed analysis of various components and hyperparameters, enhancing the reliability of the results.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The use of strong augmentation such as random eraser in mean teacher can result in pixel-level information loss that may impact the segmentation.
- The proposed loss calibration approach involves storing pixel predictions from all images, which could be sensitive to different datasets. And the proposed method does not consistently outperform other methods on different datasets, particularly in optic cup segmentation, as shown in Table 1.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
I believe that the obtained results can, in principle, be reproduced.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- Add more details of augmentations used in the mean teacher approach.
- Investigate the potential impact of using the prediction bank approach on model efficiency and provide further analysis.
- Remove some of the repeated claims about the benefits of the proposed method to make the paper more concise and focused.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The methodology could benefit from more clarity, especially regarding the augmentations used. Additionally, the proposed method’s effectiveness in addressing class imbalance may vary on different datasets, lacking generalizability.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The proposed Class-Balanced Mean Teacher (CBMT) model is generally considered an interesting and novel solution for adaptive fundus image segmentation. The model addresses the challenge of dealing with noisy pseudo labels and class imbalance by using weak-strong augmented mean teacher learning and a novel loss calibration approach. The paper achieved state-of-the-art performance on two popular benchmarks and was well-organized, clear, and reproducible. However, there were some weaknesses, such as potential pixel-level information loss due to strong augmentation, sensitivity of the proposed loss calibration approach to different datasets, and lack of consistency in outperforming other methods on different datasets. Reviewer 1 and Reviewer 3 recommend acceptance, with Reviewer 1 suggesting the paper is suitable for an oral presentation, while Reviewer 2 suggests improvements to make the paper more concise and focused and does not consider it suitable for an oral presentation. Reviewer 4 suggests addressing additional concerns, including using a more challenging dataset and discussing the applicability of the proposed method for glaucoma classification and other related works in the field. Overall, this paper is recommended for provisional acceptance.
Author Feedback
We thank all the reviewers for their constructive comments. We appreciate that the reviewers recognize the importance of our studied problem and the effectiveness of our proposed method. Here we address the main concerns: Q1: Potential pixel-level information loss due to strong augmentations.
- Since fundus image segmentation is a binary segmentation task, and the optic cup and disc are two continuous areas, a small random eraser is considered not to lose much information. Rather, it can boost the model’s ability to extract contextual information. Q2: Proposed method does not consistently outperform other methods in Table 1.
- Our proposed CBMT achieves the best performance among all SFDA methods. The BEAL reaches better results for cup segmentation on the Drishti-GS dataset, but it is under UDA setting, which means it can access labelled source data and is unfair to compare with our CBMT model. Q3: Results on more datasets and tasks, such as other type medical data, REFUGE-2, and so on.
- Thanks for constructive suggestions. We will further evaluate our proposed CBMT model on these datasets in our future work. Q4: Comparation to recent SOTA OD/OC segmentation methods.
- Our contribution mainly focuses on how to adapt a pretrained fundus segmentation model to a target domain without using source domain data, which is different from the common supervised segmentation task. The underlying segmentation model used here can be replaced with any SOTA OD/OC segmentation models. We adopt Deeplabv3+ for the sake of fair comparisons. Q5: Contents arrangement, method details, related works, writing suggestions and typos.
- Thanks for all the suggestions for improving our paper; we will try to revise them in our final version.