List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Junchao Zhu, Yiqing Shen, Haolin Zhang, Jing Ke
Abstract
The Bethesda System for Reporting Thyroid Cytopathology (TBSRTC) has been widely accepted as a reliable criterion for thyroid cytology diagnosis, where extensive diagnostic information can be deduced from the allocation and boundary of cell nuclei. However, two major challenges hinder accurate nuclei segmentation from thyroid cytology. Firstly, unbalanced distribution of nuclei morphology across different TBSRTC categories can lead to a biased model. Secondly, the insufficiency of densely annotated images results in a less generalized model. In contrast, image-wise TBSRTC labels, while containing lightweight information, can be deeply explored for segmentation guidance. To this end, we propose a TBSRTC-category aware nuclei segmentation framework (TCSegNet). To top up the small amount of pixel-wise annotations and eliminate the category preference, a larger amount of image-wise labels are taken in as the complementary supervision signal in TCSegNet. This integration of data can effectively guide the pixel-wise nuclei segmentation task with a latent global context. We also propose a semi-supervised extension of TCSegNet that leverages images with only TBSRTC-category labels. To evaluate the proposed framework and also for further cytology cell studies, we curated and elaborately annotated a multi-label thyroid cytology benchmark, collected clinically from 2019 to 2022, which will be made public upon acceptance. Our TCSegNet outperforms state-of-the-art segmentation approaches with an improvement of 2.0% Dice and 2.7% IoU; besides, the semi-supervised extension can further boost this margin. In conclusion, our study explores the weak annotations by constructing an image-wise-label-guided nuclei segmentation framework, which has the potential medical importance to assist thyroid abnormality examination. Code is available at https://github.com/Junchao-Zhu/TCSegNet.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43987-2_56
SharedIt: https://rdcu.be/dnwKb
Link to the code repository
https://github.com/Junchao-Zhu/TCSegNet
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This work proposes a new cytopathology nuclei segmentation approach that can address the bias in training on unbalanced datasets.
An extension of the presented method can employ image-wise (instead of pixel-wise) labels to train the model with a reduced reliance on training
This work also provides the first publicly available thyroid cytopathology dataset with both image- and pixel-level labels
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The proposed approach utilizes image-level labels to enhance single cell segmentation. This is made possible by using a dual path network with a CNN and a transformer. Using labels at different scales helps in identifying cell boundaries more accurately.
The extension of the proposed model in the semi-supervised manner, can be used when only image-level labels are available. This integration widens the applicability of the method and makes it usable with more real-life scenarios.
The mentioned dataset of annotated cytology images can play a key role in training new models for (thyroid) nucleus segmentation and helps developments in the field
The paper reads well and the methodology is explained up to a sufficient level of details.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The proposed methods have been compared to the baseline. However, given the small margins of improvement, using cross validation to report the results can help in reporting more reliable metrics. The authors mention two main challenges with thyroid nuclei segmentation at the beginning. The experiments and results do not address those challenges sufficiently. The reduction of model bias is not well discussed in the text. Further, the generalization of the model is not evaluated in the experiments.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The code is provided and the model is explained well, helping with the reproducibility of the method.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Minor grammatical errors are found in the text. These can be corrected by a proofreading tool. It would have been interesting to see how extending other supervised methods by taking the same approach in utilizing image-level labels would stand compared to Semi-TCSegNet on the same dataset.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The authors address valid issues in thyroid cytology image analysis. The proposes method is novel and they have shown the improved performance of it. The experiments remain limited in justifying the claims to address the current challenges mentioned at the beginning of the paper. The authors are expected to rectify this to make recommendations based on more solid results.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper proposes a novel method for segmenting nuclei in cytology imaging using a technique that involves minimizing the bias by deploying category label guidance block based on the transformer branch, and also involves using a semi-supervised technique where both the student and teacher share the same network and the weights of the teacher are updated with the exponential moving average of the weights of student.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Innovative methodology.
- Well designed semi-supervised technique.
- CNN branch + Transformer branch network to combine local and image level learning.
- Comprehensive benchmarking with multiple models.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- While the paper uses large number of image patches for training, the authors didn’t provide much details whether these patches are extracted from the same section / sample, and the conducted techniques implemented to avoid data leakage between correlated patches (if any)
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Code is available. Data will be available upon acceptance
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The paper is well written with comprehensive analysis and experiments. Just few comments
- Please consider adding qualitative illustration to show the independence between training and testing patches.
- In Fig3, it’s not clear to what is the red bounding box referring to in the images.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
8
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Innovative, cell segmentation is extremely important task, well designed and presented methodology and experiments.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #4
- Please describe the contribution of the paper
The paper proposes an ensemble of models based on CNN and Transformer architecture with a semi-supervised learning strategy to improve the accuracy of nuclei segmentation in thyroid cytology, which is limited by unbalanced distribution of nuclei morphology and the insufficiency of densely annotated images.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
The proposed segmentation method outperforms the current state-of-the-art method.
-
The paper provides clear method illustrations and results.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
The contribution of the CNN-Transformer network architecture is not well explained. It is suggested to clarify the innovation and the motivation of TCSegNet compared to the original Conformer.
-
The paper lacks references to support the theories of the two functional blocks. Without those two blocks, it seems that the performance didn’t drop much, and the unbalanced issue was solved by the capability of the Conformer network architecture itself. It is suggested to provide performance comparison between Conformer and TCSegNet on six diagnostic categories and clarify the necessity of the two blocks.
-
The semi-supervised learning strategies should be compared fairly with the same backbone. It is suggested to perform experiments with various sizes of image-wise label datasets to demonstrate the functionality of semi-supervised learning.
-
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The illustration of the method is clear to reproduce the method. And the dataset will be released on GitHub after the review process.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- The colors of the lines and legends in Fig 4 are not consistent (ClusterSeg, etc.).
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
-
The motivation of the method that whether it addresses a significant problem rather than simply increasing the performance.
-
The contribution and innovation of the proposed design.
-
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
The author addressed my concerns in the rebuttal, leading to a change in my final decision to a weak accept. Please make sure to update the clarification, references, discussion, and latest ablation results in the final version of the manuscript.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
To enhance this study, we offer the following suggestions regarding its strengths and weaknesses:
Summary of Key Strengths:
- The proposed approach effectively uses image-level labels to improve single cell segmentation by employing a dual path network consisting of a CNN and a transformer. Leveraging labels at different scales enhances the accuracy of cell boundary identification.
- The extension of the proposed model in a semi-supervised manner enables its use when only image-level labels are available, expanding its applicability to real-life scenarios.
- The annotated cytology image dataset mentioned in the study plays a crucial role in training new models for nucleus segmentation, particularly in the context of (thyroid) pathology.
- The paper is well-written, and the innovative methodology is adequately explained.
- The semi-supervised technique is well-designed.
- The combination of a CNN branch and a Transformer branch in the network allows for the integration of local and image-level learning.
- The study includes comprehensive benchmarking with multiple models.
- The proposed segmentation method outperforms the current state-of-the-art approach.
- The paper provides clear illustrations of the method and its results.
Main Weaknesses:
- Reporting results using cross-validation can improve the reliability of metrics.
- The challenges mentioned in thyroid nuclei segmentation are not adequately addressed in the experiments and results.
- The discussion about reducing model bias is insufficient in the text.
- The generalisation of the model is not thoroughly evaluated in the experiments.
- More details are needed regarding the extraction of image patches, whether they are from the same section/sample, and measures taken to avoid data leakage between correlated patches (if applicable).
- The contribution and innovation of the CNN-Transformer network architecture in TCSegNet need clearer explanation compared to the original Conformer model.
To address these concerns and improve the study, we suggest the following recommendations:
- Address minor grammatical errors found in the text.
- It would be interesting to compare the extension of other supervised methods using the same approach of using image-level labels with Semi-TCSegNet on the same dataset.
- Consider adding qualitative illustrations to demonstrate the independence between training and testing patches.
- Clarify the reference of the red bounding box in the images shown in Figure 3.
- Ensure consistency in the colours of lines and legends in Figure 4 (e.g., ClusterSeg, etc.).
Key points to focus on in rebuttals:
- Provide a detailed explanation of the motivation behind the method and its significance in addressing a significant problem rather than solely improving performance.
- Specify the specific contribution and innovation of the proposed design.
- Include references to support the theories of the two functional blocks and provide performance comparisons between Conformer and TCSegNet on six diagnostic categories to demonstrate the necessity of the two blocks.
- Conduct fair comparisons of semi-supervised learning strategies with the same backbone network and experiment with various sizes of image-wise labeled datasets to showcase the functionality of semi-supervised learning.
- Acknowledge the suggestion of using cross-validation to report more reliable metrics.
- Address the lack of discussion about model bias reduction and evaluate the generalisation of the model in the experiments.
- Provide more details about the extraction of image patches, including whether they are from the same section/sample.
- Specify the techniques implemented to prevent data leakage between correlated patches (if applicable).
- Provide further explanation of the contribution of the CNN-Transformer network architecture.
- Clarify the innovation and motivation of TCSegNet compared to the original Conformer model.
Author Feedback
1.Significance/motivation: Nuclei identification is crucial for thyroid cancer analysis. However, clinically unbalanced nuclei distribution across samples of different TBSRTC categories causes model bias. Our multi-task formulated approach can learn from lightweight image-wise labels to alleviate the demand for pixel-wise labels.
2.Contribution: TCSegNet utilizes lightweight labels to enhance learning from unbalanced sets. A novel semi-supervised model leverages exclusive image-wise labels to facilitate segmentation, where cytology-specific noise incorporates prior knowledge of stain distribution for practical color perturbation.
3.References for functional blocks: Pseudoseg(PS)[24] and mean teacher(MT)[18] support our Label Guidance Block and HSV-Intensity Noise respectively. We advance the PS with direct supervision of image-wise labels rather than re-processing info. The proposed noise improves MT to boost the model’s generalization ability to the cytology dataset.
4.TCSegNet’s innovation compared with Conformer: While Conformer is confined to classification, our TCSegNet stands out as a multitask dual-path U-shaped network for both classification and segmentation to address the learning from unbalanced data.
5.Contribution of dual-path architecture: 1)It benefits segmentation by the integration of complementary information from both branches in both the encoder and decoder. 2)It correlates global information in image-wise labels and local information in pixel-wise labels, maximizing the utilization of the multi-labels in our dataset. 3)It enables the design of a novel consistency constraint between CNN and transformer branch for further performance improvement.
6.Comparisons for semi-supervised: Comprehensive comparisons of our strategy against SOTAs with different backbones are summarized in Tab.1. However, TCSegNet is not applicable to all the compared methods as the backbone since they require specific network architecture modifications e.g., PS-ClusterSeg needs clustered contour predictions. Different data sizes were reported: +1k: 0.879; +2k: 0.882; and +3.5k: 0.889 (dice). Performance improvement is accumulated with increasing images.
7.Cross-validation: We conducted random runs where the STD of test Dice <= 0.35% over 5 runs, yielding significant improvements over SOTAs, along with an accuracy improvement exceeding 2.0%. Importantly, we did perform cross-validation, observing that the results were similar to random runs.
8.Discussion about bias & generalization: Unbalanced distribution refers to samples from different catalogs varying significantly in quantity (Fig.2), which is inevitable in clinical practice. However, all current methods failed to address this critical issue, hence unsatisfied biased models are resulted. One important cause is that cell morphological characteristics are exclusively extracted and memorized. Innovatively, image-wise labels allow a more generalization ability in our TCSegNet, and its straightforward output is the accuracy in nuclei segmentation of various sizes and shapes across all diagnostic grades (Fig.3). The reduction of model bias is further demonstrated in Fig.4 with an average higher performance while the less difference in metrics.
9.Patch extraction: Low-grade patches, suspicious patches, and malignant patches followed different criteria in benchmark construction. I/II were randomly selected from their WSIs using Monte Carlo Sampling and cellularity threshold. In the curation of III/VI, we sampled by cellularity first and then filtered out I/II, as diagnostic clusters are sparse in any cytology image. We achieved a benchmark without a considerable gap between the number of patches in TBSRTC catalogs. Meanwhile, the modest difference in quantity allowed an unbalanced set to prove our advantage.
10.Data leakage: WSI/patient-level images were partitioned first for training and test images, and patch-level curation was performed. It effectively avoids data leakage.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The responses of the authors during the rebuttal process were convincing to accept the paper. We hope that the constructive remarks will help you to improve the work pls. consider them carefully. Please also make sure to update the clarification, references, discussion, and latest ablation results in the final version of the manuscript.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper introduces a novel method for segmenting nuclei in cytology imaging. The proposed technique involves minimizing bias by deploying a category label guidance block based on the transformer branch. Additionally, a semi-supervised technique is employed where both the student and teacher networks share the same architecture, and the teacher’s weights are updated using the exponential moving average of the student’s weights.
During the first round of review, the reviewers and AC appreciate the method’s effectiveness, high quality of writing, and its potential clinical applications. However, they also raise questions regarding clarity, rigor of the evaluation process, and the need for further discussion. The author provides a comprehensive rebuttal, summarizing and addressing these concerns. As a result, the paper receives three positive reviews.
I believe this paper tackles an important clinical scenario where only a small-scale annotation is available. While the idea of semi-supervised learning is not new, the design choice of incorporating both a CNN branch and a Transformer branch holds significance. The author’s rebuttal successfully addresses the concerns raised during the review process.
Based on these reasons, I recommend accepting this paper.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors propose a method to improve Thyroid nuclei segmentation. The experimental results show that the proposed method outperforms other methods and semi-supervised learning aids in improving the performance. However, I do believe that the authors present their work in an inappropriate manner. The authors emphasize two issues: 1) unbalanced nuclei distribution and 2) lack of sufficient annotations. Semi-supervised learning shows the improvement in the performance, and thus lack of sufficient annotations can be resolved to some extent. However, whether the proposed model can address the unbalanced nuclei distribution problem was not shown and discussed in the paper. As per the rebuttal, it is a major motivation for their work, but there is no attempt to show or prove that the proposed method is able to deal with this. Even, how the imbalance affects the model performance was not clearly shown in the paper. Also, DICE and IoU are not the optimal metric for evaluating nuclei segmentation. These should be properly addressed.