Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews Back to top
List of Papers By topics Author List
Paper Info Reviews Meta-review Author Feedback Post-Rebuttal Meta-reviews

Authors

Yicheng Wu, Zhonghua Wu, Qianyi Wu, Zongyuan Ge, Jianfei Cai

Abstract

Semi-supervised segmentation remains challenging in medical imaging since the amount of annotated medical data is often scarce and there are many blurred pixels near the adhesive edges or in the low-contrast regions. To address the issues, we advocate to firstly constrain the consistency of pixels with and without strong perturbations to apply a sufficient smoothness constraint and further encourage the class-level separation to exploit the low-entropy regularization for the model training. Particularly, in this paper, we propose the SS-Net for semi-supervised medical image segmentation tasks, via exploring the pixel-level smoothness and inter-class separation at the same time. The pixel-level smoothness forces the model to generate invariant results under adversarial perturbations. Meanwhile, the inter-class separation encourages individual class features should approach their corresponding high-quality prototypes, in order to make each class distribution compact and separate different classes. We evaluated our SS-Net against five recent methods on the public LA and ACDC datasets. Extensive experimental results under two semi-supervised settings demonstrate the superiority of our proposed SS-Net model, achieving new state-of-the-art (SOTA) performance on both datasets. The code is available at https://github.com/ycwu1997/SS-Net.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16443-9_4

SharedIt: https://rdcu.be/cVRya

Link to the code repository

https://github.com/ycwu1997/SS-Net

Link to the dataset(s)

http://atriaseg2018.cardiacatlas.org

https://www.creatis.insa-lyon.fr/Challenge/acdc/databases.html

Reviews

Review #2

  • Please describe the contribution of the paper

    This paper proposes a modified semi-supervised method for image segmentation. They design two losses to separately promote model regularization and inter-class separation. They also conduct comprehensive experiments to show the effectiveness of the proposed method.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. The paper is very well organized.
    2. The motivation and ideas of the two losses are clear and straightforward.
    3. The proposed methods are helpful as verified in the experiments.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    1. The main concern is that the proposed two methods are both existing semi-supervised solutions. The authors adopt adversarial noises from VAT [13] with a small modification of discrepancy measurement and the inter-class separation strategy is the same as [1]. Most of the equations are very similar to the original papers. The authors should clearly explain the novelty of their method compared to [13] and [1].
    2. The performance gains are limited.
  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    As the authors claim that they will release their codes, the reviewer has no other concerns.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

    Overall, I think this paper has merit on the proposed method and adequate experiments. However, the paper can be improved in a few ways.

    1. Some details about Inter-class Separation are missing in Sec 2.2.
      • How to select the subset of F_l according to the predictions’ confidence? Is there any threshold?
      • What is the detailed structure of the attention modules?
    2. What is the starting value of lambda and the specific warming-up function?
    3. Backbone selection. Why do the authors use different backbones for LA and ACDC? In addition, the authors should demonstrate whether all the comparisons were conducted with the same backbone.
    4. It seems from both Table 1 and Table 2, the performance increase is very limited under 8 labeled settings. It’s difficult to assess whether the model is significantly better than the baselines. It would be helpful to provide the variance or testing models under more labeled/unlabeled settings.

    A few other minor comments:

    1. What do the red arrow and orange block represent in Fig.2?
    2. Some of the notations in Fig.2 do not correspond to notations in the paper, e.g. X_{L} and X_{l}, Y and y. Some descriptions are missing, e.g. P_L, P_X.
    3. What do ‘y’ in Eq(2) and ‘y_c’ in Eq(3) represent? The reviewer thinks they might be typos. Please double-check all the notations and equations.
  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    5

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper applies previously proposed methods to a new scenario, medical imaging. From a theoretical perspective, the novelty is very limited. However, considering the motivation is clear and inspiring, I recommend a weak accept based on the understanding that the authors will release their codes.

  • Number of papers in your stack

    5

  • What is the ranking of this paper in your review stack?

    1

  • Reviewer confidence

    Very confident

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    5

  • [Post rebuttal] Please justify your decision

    It is helpful to have provided more results under different settings. Although I still believe it shows very limited novelty, I understand it is non-trivial to combine them. Based on the clarification that they will release their codes, I will keep the original weak acceptance.



Review #3

  • Please describe the contribution of the paper

    This paper proposes the SS-Net for semi-supervised medical image segmentation via exploring the pixel-level Smoothness and inter-class Separation at the same time. The pixel-level smoothness forces the model to generate invariant results under adversarial perturbations. Meanwhile, the inter-class separation constrains individual class features should approach their corresponding high-quality prototypes, in order to make each class distribution compact and separate different classes. The methods are evaluated on two public benchmarks.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. I found this paper well-written and easy to follow. Especially, it is good to use a toy exemplar data to help readers understand their two key insights and motivations.
    2. The combination of VAT-based strong perturbation with class separation is well linked to their two insights.
    3. The experiments are well designed with rigorous ablation study.
    4. The method will not introduce obvious computational complexity.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    Since the paper is highly related to two directions, i.e., strong perturbation and class prototype learning, related work should be more comprehensive, with more relevant methods regarding to their insights. For strong perturbations, authors can discuss more relevant papers about it besides the VAT they borrowed. For example, French, Geoff, et al. “Semi-supervised semantic segmentation needs strong, varied perturbations.” BMVC (2020).

    Also, for the class prototype-based methods to push feature separation and compactness, authors can also discuss relevant papers, including but not limited to: Xu, Z., et al, All-Around Real Label Supervision: Cyclic Prototype Consistency Learning for Semi-supervised Medical Image Segmentation. IEEE Journal of Biomedical and Health Informatics (2022).

    The method introduces some other parameters compared with previous methods, like in their adversarial perturbation and the tradeoff. It will be good to see some sensitivity analysis to help readers better understand the method and tune for their applications.

    The supervised results are very low with your lowest portion of labeled data. It is sometimes due to the batchnorm module will have negative effect trained with very limited data. Like, if you train MT model but turning off the L(unlabeled), it can also serve as the supervised method, and the results will be normal. I suggest using InstanceNorm here to avoid this problem.

  • Please rate the clarity and organization of this paper

    Excellent

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    The authors promise to release their codes.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

    More comprehensive and relevant literature should be discussed, i.e., other strong perturbation SSL and prototype-based SSL. Sensitivity analysis is needed since there are many hyper-parameters.

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    6

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    I appreciate the insights and good use of figures in this paper to support their claims. Therefore, I lean to acceptance.

  • Number of papers in your stack

    5

  • What is the ranking of this paper in your review stack?

    1

  • Reviewer confidence

    Confident but not absolutely certain

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    7

  • [Post rebuttal] Please justify your decision

    The authors have well addressed major concerns. I raise my score to 7.



Review #4

  • Please describe the contribution of the paper

    The authors proposed a deep learning framework for semi-supervised medical image segmentation. Pixel-level smoothness and inter-class separation are explored in the proposed framework. In the experiment, two public datasets are used for evaluation.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. Two different datasets are used for evaluation.
    2. The proposed method has been compared to five other semi-supervised segmentation methods.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    1. The contributions are a little bit over-claimed.
    2. The technical novelty of this work is limited. Detailed comments please see below.
  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    The supplementary material provides detailed algorithms. No code is provided. Not sure about reproducibility.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

    1. Regarding to the first contribution, “fewer labels” and “blurred targets” are two common challenges in medical image segmentation task and existing semi-supervised methods always aim to address these challenges. In addition, the idea of using “constrain the pixel-level smoothness and inter-class separation” to address these challenges is not new, as the authors mentioned in the Introduction section (paragraph 2) that existing methods did the same thing.

    2. For the second contribution, using adversarial perturbations for medical image segmentation has appeared in previous works (e.g., [A], [B]), which is not the first in this paper as claimed. [A] Adversarial Perturbation on MRI Modalities in Brain Tumor Segmentation, IEEE Access, 2020 [B] Towards Robust General Medical Image Segmentation, MICCAI, 2021

    3. The proposed framework is more like a combination of two existing works and has limited technical novelty. For the two parts of the proposed method, the “Pixel-level Smoothness” part is basically the work from [13] and the “Inter-class Separation” part is very similar to [1].

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    4

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The contribution is a little bit over-claimed and this work has limited technical novelty.

  • Number of papers in your stack

    5

  • What is the ranking of this paper in your review stack?

    4

  • Reviewer confidence

    Confident but not absolutely certain

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    4

  • [Post rebuttal] Please justify your decision

    Thank the authors for their reply. After reading the rebuttal, I still concern about the methodological novelty. The authors agree that their method is a combination of existing works [1] and [13] and the novelty is applying them to medical tasks. Previous works [1] and [13] addressed the same challenges (few labeled data and close classes /blur boundaries) which also appear in medical tasks. Therefore, the proposed method, combining existing algorithms to solve same challenges for just a different application, has quite incremental methodological novelty.



Primary Meta-Review

  • Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    There are non-converging review recommendations. The authors are encouraged to address esp. the issues raised by the reviewers including the novelties & technical contributions (e.g. the proposed are existing solutions), related works (as pointed by the reviewers), empirical evaluations (e.g. limited performance improvement), among others.

  • What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    7



Author Feedback

We thank AC and all reviewers for their constructive comments. This paper receives 3 reviews with scores of 5, 6 and 4 from R2 (very confident), R3 and R4 (not absolutely certain), respectively. We are encouraged by the positive comments: 1. motivation and ideas are clear (R2) and appreciated (R3); 2. methods are helpful (R2) and well-linked to the insights (R3); 3. experiments are adequate (R2) and rigorous (R3). R4 mainly concerns our technical novelty. Below we first address the major issues.

Q1: Is our model a simple combination of [1] and [13]? (AC, R2&R4) Although the two main components are adapted from prior arts, applying them to medical tasks is new. More importantly, it is non-trivial to come out with the idea of combining them, which is based on our careful observations of the challenges in medical imaging: fewer labels and blurred targets, see Fig. 1. Our key insight is that it is crucial to enforce the strong perturbations to encourage the pixel-level smoothness while at the same time shrinking each class distribution to separate different classes, which motivates us to combine [1] and [13]. However, directly employing [1] or [13] won’t give us desired performance, while our carefully customized model can. For example, Table 3 shows that using the original [1] with 5% training labels can only obtain an 82.27% dice on LA, while our model obtains an 86.33% dice.

Q2: Motivation (R4) Although existing works like [18,19] use both consistency and low-entropy constraints for SSL, they do not explicitly analyze the effects caused by the two above challenges for medical tasks. For example, when the data is blurred and with scarce labels (e.g., 5%), the feature manifolds of different classes may be inter-connected, see Fig. 1. In this case, the performance of [18] drops significantly, which is mainly caused by insufficient perturbations and incorrect co-training. In contrast, we proposed an effective SSNet in the extremely-scarce-labels regime by applying strong perturbations and prototype learning at the same time to achieve superior performance. Both R2 and R3 appreciate our motivation and insights.

Q3: Limited performance gains (AC, R2) We focus on the scarce label scenario. Compared to existing methods, our SSNet improves the performance significantly (e.g., 2.74% and 2.98% dice gains over the previous SOTA model [18] with 5% training labels on LA and ACDC). Meanwhile, as R2 suggested, we further show the results on LA, obtained by SSNet and [18] under more settings as:

Model-Dice(%) (4 labels) [18]–83.59 Ours–86.33 (+2.74)

(6 labels) [18]-85.43 Ours–87.34 (+1.91)

(8 labels) [18]-87.62 Ours–88.55 (+0.93)

(10 labels) [18]-87.24 Ours–88.42 (+1.18)

We can see that our model always surpasses [18] on LA.

Q4: Related work (R3&R4) To our knowledge, existing adversarial models like [A] & [B] are not applied in semi-supervised medical image segmentation and our modified VAT model is the first one to do so. Meanwhile, we will discuss the references suggested by R3&R4 and other relevant works.

Q5: Parameter sensitivity (R3) For simplicity, all experiments used the same hyper-parameters. As R3 suggested, we conducted the discussion about N (# of prototypes) as an example:

(10% labelled data on ACDC) N-Dice(%) 32-86.78 64-86.58 128-86.27 256-86.81

It shows that the dice values of different N vary slightly, indicating our model is robust with different N on ACDC. Future work will discuss the effects of more hyper-parameters.

Q6: Different backbones (R2) Following the public methods [9,10], we selected VNet and UNet as the backbones on LA and ACDC, respectively and all compared methods on each dataset shared the same backbone for fair comparisons.

Q7: Other normalization (R3) It is interesting to try different normalized techniques, for which we will leave for our future work.

Q8: Model details and typos (R2) Thanks for these suggestions and we will address them.

Q9: Reproducibility (R4) We will release our codes.



Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    This paper deals with semi-supervised medical image segmentation using deep learning methods, with good empirical results. Admittedly building upon existing methods, this work has perhaps limited technical contributions. The novel use in medical imaging setting with good empirical results still provide reasonable justification. Meanwhile, the reviewers have raised a number of concerns including providing more results under different settings, making code publicly available, discussing mentioned related efforts, sensitivity analysis, tuning down the claimed contributions, among others. The authors need to seriously go through the issues raised by the reviewers and address them properly.

  • After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

    Accept

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    8



Meta-review #2

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    Although there is weakness in methodological novelty by combining the components from existing studies, the motivation is driven the observation from medical tasks and the experimental results over two tasks demonstrated better performance. Thus I am inclined to acceptance.

  • After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

    Accept

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    2



Meta-review #3

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    This paper studies semi-supervised image segmentation problem by two losses to separately promote model regularization and inter-class separation. While the paper is generally well-written and easy to follow, the following concerns prevent me from recommending this paper.

    1. By checking ref [1] and [13], I agree with Reviewer #3 that this paper combines existing algorithms to solve same challenges with a different application, thus it has quite incremental methodological novelty.
    2. Experimental results to compare ref [1] and [13] should be included.
    3. Both image segmentation and semi-supervised image segmentation have been studied for decades. There existing many effective methods for these two segmentation problems. Yet the comparative studies only reported the neural network based segmentation methods. Comparing some benchmark methods not using neural works can be included for better convincing the readers.
  • After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

    Reject

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    5



back to top