List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Jinhua Liu, Christian Desrosiers, Yuanfeng Zhou
Abstract
In semi-supervised medical image segmentation, the limited amount of labeled data available for training is often insufficient to learn the variability and complexity of target regions. To overcome these challenges, we propose a novel framework based on cross-model pseudo-supervision that generates anatomically plausible predictions using shape awareness and local context constraints. Our framework consists of two parallel networks, a shape-aware network and a shape-agnostic network, which provide pseudo-labels to each other for using unlabeled data effectively. The shape-aware network implicitly captures information on the shape of target regions by adding the prediction of the other network as input. On the other hand, the shape-agnostic network leverages Monte-Carlo dropout uncertainty estimation to generate reliable pseudo-labels to the other network. The proposed framework also comprises a new loss function that enables the network to learn the local context of the segmentation, thus improving the overall segmentation accuracy. Experiments on two publicly-available datasets show that our method outperforms state-of-the-art approaches for semi-supervised segmentation and better preserves anatomical morphology compared to these approaches.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16452-1_14
SharedIt: https://rdcu.be/cVRYW
Link to the code repository
https://github.com/igip-liu/SLC-Net
Link to the dataset(s)
https://www.creatis.insa-lyon.fr/Challenge/acdc/databases.html
http://promise12.grand-challenge.org/
Reviews
Review #1
- Please describe the contribution of the paper
This study proposed a semi-supervised segmentation framework consisting of two Unet networks that generate pseudo-labels for each other. The study also proposed a loss function, local context loss, as an extension of the dice loss. The framework was evaluated using two public datasets and showed superior results compared to other approaches with semi-supervision (n=7,14) while underperformed the baseline method with full supervision.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The proposed local context loss is new and showed improved segmentation performance.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The novelty of the paper is limited. Although with a handful of adaptions, the overall architecture of using two Unet networks and feeding the prediction to the other is not new.
- The gains of using the proposed architecture are marginal. Overall, the proposed method underperforms the baseline method with full supervision.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors will provide the code after acceptance. Public dataset is used.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- More theoretical explanations for the framework architecture or how this design sufficiently addresses the lacking of dataset issue, i.e. overcoming variability and complexity issue of medical images, is needed.
- For the comparison experiments, the choice for the number of training samples, 7 or 14, needs to be justified. Would this be clinically relevant, as a larger number of training sets could be reasonably obtained for the segmentation tasks targeted? Overall, the proposed experiments showed inferior results compared with the baseline ones using full supervision.
- When the local context loss is used, the DSC increases substantially for RV, while only slightly for Myo ad LV. An explanation for this difference is helpful.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Please refer to the strength(4), weaknesses (5), comments (8) for justification.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
We thank the authors for thoughtfully addressing and clarifyting the concerns. It would still be helpful to see how the results distribute for training sets with different patients as a small number of labeled cases were used.
Review #2
- Please describe the contribution of the paper
This paper introduces a semi-supervised segmentation approach. It extends a prior art of cross-modal supervision by incorporating two things. First is the share awareness by co-teaching a shape agnostic network and a shape aware network. The second is to adopt local context constraints using patch level dice loss. The approach is applied on two public datasets in a semi-supervised setting, and demonstrates superiority over other methods
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The design of the framework is clear and reasonable
- The results look positive compared to other methods
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The author does not seem to be clear on what exactly are novel on top of their citations [4] and [9]
- The ablation studies can be improved.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The method is clearly described, so the paper should be reproducible in terms of implementation.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The paper is well written, the idea is simple and clear, the key is how much the author’s contribution helps on performance gain. It will be nice if the authors can elaborate a bit what exactly is new compared to citations [4] and [9]. Also elaborate a bit on the ablation studies, specifically in Table 1, what exactly is the “Baseline”? what is “UE” (the authors mentions only U1 and U2 for the unsupervised loss), what is the impact if no threshold is applied to the entropy?
There is also a conflict when the author describes the implementation and experiments. The author claims that they use only 7 labeled samples to generate the results in Table 2. But in section 3.2, the author states that during training a batch size of 24 is used, 12 labeled, 12 unlabeled.12 is more than 7. This is confusing
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper presents a simple but effective semi-supervised approach. It is not exceptional in terms of novelty, but it is solid given the benchmark results.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
The authors answered some of my questions in the rebuttal. The novelty is still limited to me, but the overall quality of the paper is good enough for acceptance.
Review #3
- Please describe the contribution of the paper
This manuscript is about incorporating shape priors in the context of segmentation with neural networks. For this purpose two networks are used. One network is shape agnostic. The second network is shape aware and receives as input the original image as well as the output of the first agnostic network. The second network also encodes local information since it considers different parts of a shape. The method is validated with two datasets.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
It addresses a valid problem, namely incorporating shape information in neural networks segmentation. It is motivated by the publications referenced in [4][9], that are interesting.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
It uses two different networks rather than one. Why not one?
In the unsupervised loss training function (page 5) for unlabeled data the authors claim that the labels provided by the shape network are more accurate because of the shape information. This is not explained and substantiated.
In general the cost terms and their terminology are explained after the whole cost is introduced. They should state any pre-processing necessary for the shapes whether the prior shapes must be aligned and the robustness of the method after rotations.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Precise reproducibility is not a particular strength of neural networks. The networks are described in the implementation section and can be reimplemented.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The unsupervised loss function should be better justified.
Could state the robustness of the method with respect to rotations or whether they have to be aligned and in the same pose. The preprocessing necessary for shapes.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
A good method to incorporate prior shape to segmentation with neural networks that is currently not sufficiently addressed. Could be more forthcoming for pre-processing conditions.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
Concerning whether the prior shape has to be of a specific pose and the sensitivity to similarity transformations, the authors response seems to be that this is irrelevant to their method. I would say, if the pose and shape instance has to be precisely specified, then there might be simpler segmentation algorithms that might do even better, without involving the complications of neural networks. In any case, neural networks seem to have a potential for development, so for some time we can ignore some issues. In the long term it is better not to be fixated to a particular technique.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper introduces a semi-supervised method for image segmentation that applies two networks such that they produce pseudo-labels for each other. Overall, the reviewers gave positive feedback, however key concerns on novelty were raised as well. In the rebuttal, please address the novelty concerns, especially as compared with [4] and [9], the ablation questions and the clarity questions raised by the reviewers.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
8
Author Feedback
We thank the Reviewers and AC for their comments and positive feedback.
- Novelty (R1 & R2) As underlined by R2 and R3, we propose a concise and robust SSL method achieving outstanding results on two datasets. Instead of designing a complex architecture, we use Unet as the base network to build a novel paradigm. Compared to [4], or method adds two innovative strategies to make the segmentation pipeline more robust and boost performance: a shape awareness for enhanced guidance and a local context loss for enforcing accuracy in local regions. While the instance segmentation method of [9] also exploits shape priors, we do not use such priors in an iterative process but instead through the cross-teaching of shape-agnostic and shape-aware networks.
- Comparison with others (R1) We argue that our gains are not marginal. The full supervision (FS) baseline requires the entire training dataset to be labeled, which is extremely laborious and time-consuming. In contrast, our SSL method requires only a few labeled data for training. Hence, it cannot be directly compared to this “upper bound”. However, as shown in Tables 1 and 2, our method outperforms all other SSL approaches by a large margin for all segmentation datasets/classes and metrics (significant with p < 0.05).
- Ablation questions (R1 & R2) Compared to Myo and LV, which are mostly circular, the RV has a more complex and irregular shape (see Fig. 2). Our local context loss is thus more useful to guide the segmentation in this harder to segment region. In Table 1, “Baseline” is the cross-modal supervision without shape awareness, context loss, and uncertainty estimation. The 1 vs. 3, 3 vs. 5, and 2 vs. 3 (rows) of the ablations indicate that each of these techniques lead to large gains. Moreover, if no threshold is applied to the entropy in our uncertainty estimation (“UE”), unreliable regions are not filtered out. The negative impact of this can be seen in row 2 vs. 3 of ablations.
- Theoretical explanation (R1) A detailed theoretical analysis is beyond the scope of this paper, however we can explain our method’s performance as follows. First, our cross-model pseudo-supervision approach acts as a regularization prior on unlabeled data, similar to other SSL methods like co-training and Mean Teacher. Unlike these methods, our shape-aware approach also feeds the prediction of a model as input to the second model, providing additional guidance. Second, our local context loss, which imposes a high accuracy in every local region, alleviates the learning bias toward some regions due to the labeled data scarcity in SSL.
- Experiments (R1 & R2) The 7 and 14 labeled samples correspond to 10% and 20% of the training set, which follows similar works on SSL. This setting is well suited for clinical scenarios where obtaining labeled data is laborious but unlabeled data is often available. Note that, although there are few labeled 3D volumes, as described in Section 3.1, we train the networks with 2D slices of these volumes (test images are from volumes not seen in training). Thus, the number of actual training examples is greater.
- Why not one network? (R3) Having two networks cross-teaching each other enables to have a consistency regularization prior (our unsupervised loss) exploiting unlabeled examples. It is also necessary because the two models are asymmetric (one only has the image as input while the other also receives a segmentation prediction).
- Unsupervised loss (R3) We validate that shape-aware network is more accurate by reporting the mean (stdev) DSC on ACDC: Shape-agnostic: RV 72.61(1.76) | Myo 79.51(0.80) | LV 85.20(0.29) Shape-aware: RV 74.21(1.11) | Myo 80.28(0.40) | LV 85.68(0.12)
- Shape alignment (R3) Our method does not require shape alignment and is robust to rotation. Both networks receive input from the same pose and the shape-aware network receives the output of the shape-agnostic network. So, the shapes are inherently aligned and not affected by rotation.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The rebuttal addressed the main concerns raised by the reviewers and convinced the reviewers to reach consensus on accept. Please try to take the final comments when preparing the final version.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
8
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The reviews have highlighted the quality of the paper in terms of clarity, reasonably interesting approach and promising results.
However, the potential lack of novelty and the proximity to previous papers on the subject raised questions.
I found the authors’ rebuttal well explaining how different their approach was, and thus quite well responding to the concerns raised by the reviewers. This is is also agreed upon by the 3 reviewers, who all gave a « weak accept » ultimately. Hence I recommend acceptance for this paper.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors propose an SSL approach here that is relatively straightforward and produces a good margin of improvement over alternatives. Although differences compared to [4] and [9] could be better elucidated, I think there enough novelty here to be of interest to the MICCAI community. I also agree with the authors’ rebuttal that their inferior performance compared to a fully-supervised upper bound is not a negative, as that it what we would expect.
Thus, while the paper certainly has some shortcomings, I think it passes the MICCAI bar and I would recommend acceptance.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6