Authors

Pingyi Chen, Chenglu Zhu, Zhongyi Shui, Jiatong Cai, Sunyi Zheng, Shichuan Zhang, Lin Yang

Abstract

The success of supervised deep learning models on cell recognition tasks relies on detailed annotations. Many previous works have managed to reduce the dependency on labels. However, considering the large number of cells contained in a patch, costly and inefficient labeling is still inevitable. To this end, we explored label-free methods for cell recognition. Prior self-activation maps (PSM) are proposed to generate pseudo masks as training targets. To be specific, an activation network is trained with self-supervised learning. The gradient information in the shallow layers of the network is aggregated to generate prior self-activation maps. Afterward, a semantic clustering module is then introduced as a pipeline to transform PSMs to pixel-level semantic pseudo masks for downstream tasks. We evaluated our method on two histological datasets: MoNuSeg (cell segmentation) and BCData (multi-class cell detection). Compared with other fully-supervised and weakly-supervised methods, our method can achieve competitive performance without any manual annotations. Our simple but effective framework can also achieve multi-class cell detection which can not be done by existing unsupervised methods. The results show the potential of PSMs that might inspire other research to deal with the hunger for labels in medical area.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43993-3_54

SharedIt: https://rdcu.be/dnwNZ

Link to the code repository

https://github.com/cpystan/PSM

Link to the dataset(s)

N/A

Reviews

Review #2

Please describe the contribution of the paper

The author proposed novel label-free cell segmentation methods. They demonstrated the effectiveness of the proposed method on two datasets and achieved competitive performance with supervised or weakly-supervised comparisons.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Unlike previous method which use color of image, the method use extracted feature of self-supervised model in addition to color of image. It seems more robust than previous work.

The proposed framework achieved comparable performance on cell detection and segmentation with supervised methods.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The detail of pseud label for multi-class detection is unclear. As shown in Fig.2, there is no difference between green and red cell in the attention map. The definition of the pseudo mask I_sg is ambiguous. How to generate I_sg? How to use the semantic clustering result S? Please clarify the relationship between clustering results and labels.

There is no discussion with conventional clustering such as color clustering which is widely used for weakly-supervised nuclei segmentation [1]. It is difficult to understand the effectiveness of proposed clustering.

[1] Qu et al., Weakly supervised deep nuclei segmentation using points annotation in histopathology images.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The author will upload code if this work is accepted. Therefore, the reproducibility is ok.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- Please clarify the relationship between I_sg and S.
- The reviewer recommend to add discussion about pseudo label quality between color clustering and the proposed clustering. The discussion would clarify the contribution of the proposed method.
[1] Qu et al., Weakly supervised deep nuclei segmentation using points annotation in histopathology images.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Although there are some unclear explanations, the proposed method is simple and effective, and experiments demonstrate the effectiveness of the proposed method.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

The author describe a unsupervised segmentation and detection pipeline with self-activation map. The prior self-activation maps act as the segmentation mask. Furthermore, the author validate the proposed method on two different tasks.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The annotation (segmentation) for cell is time-consuming by manual efforts. So this paper is a nice contribution to this fields. The author proposes to use prior self-activation maps as the target for segmentation model training. The paper is in good quality. The method is easy to follow.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The evaluation of the mothods are not discussed enough. Since the annotation for model training is not provided by manual efforts, if the backbone used for vision task will affect the model performance. (notice the transformer-based model perform worse than CNN-based model).
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The method is easy to follow. The configuration for model training is provided in manuscipt.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

The evaluation part in manuscipt can be further discussed. For example, the difference between methods and the performance gaps can be discussed.

Compared with the baseline experiments, the effect of backbone could be discussed. For example, the CellProfile has relative low performance compared with other methods. The reason of proposed methods outperform other baseline model can be discussed.

The motivation to abstract the self-activation map is very important.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper is in good quality. The manuscript is well-written and figure is easy to follow. So if the result evaluation can provide better discussion, it would be a good paper for MICCAI.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #1

Please describe the contribution of the paper

The paper proposes an unsupervised learning approach to nuclei segmentation in histopathology images. The proposed methods takes advantage of feature maps in the shallow layers to generate pseudo segmentation labels. They evaluate their method on 2 public datasets: MoNuSeg and BCData. On MoNuSeg, the authors show competitive performance to several supervised, weakly supervised and improvement over unsupervised methods. On BCData, they compare to another set of weakly supervised methods and show competitive performance on the cell detection task and multi-class counting task.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper address an important problem in the medical domain, that is the unsupervised nuclei segmentation in histopathology data.
- The method proposed is interesting and novel. They use activation maps from the shallow layers to generate pseudo labels. They do that by aggregation, re-weighting, and clustering of the activation maps.
- The proposed method shows competitive performance to supervised and weakly supervised methods on 2 public datasets.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The evaluation in BCData for multi-class cell detection only uses the counts. There is only per-class evaluation on the counting task and not the individual cell detection.
- In the BCData, the staining for different cell types is clearly different. How well does the proposed approach work when classifying cells with the same staining, for example different cell types in H&E images (such as in BRCA-M2C or CoNSeP datasets)
- The unsupervised methods compared against are relatively old and not the SoTA. For example the following are more recent methods: Self-Supervised Nuclei Segmentation in Histopathological Images Using Attention Unsupervised Nuclei Segmentation Using Spatial Organization Priors
- There are a few design choices that are not intuitive and not justified, such as:
  - Why similarity is give better results than contrastiveness?
  - Why not use the contrastive loss formulation in the contrastiveness task?
  - Would the use of the contrastive loss formulation improve contrastiveness performance?
- There are several details that are not mentioned or not clear from the text:
  - What is I_raw, is it the RGB image?
  - When compute the fused semantic map I_f (eq. 5), the PSM I_am does it have the same size as I_raw or does it need scaling?
  - In the K-means clustering, what is the value of number of clusters K and how is it selected? is it number of cell types + background?
  - How are the clusters mapped to foreground and background? Is it manual?
  - In the cell detection, when getting the peaks from local regions (eq. 7), what is the size of the local region D_m,n
  - The pseudo mask I_sg used in cell detection and cell segmentation tasks, how is it obtained? It was never mentioned before the cell detection paragraph on page 5.
  - In the ablation studies, what is the experimental setting? which dataset?
- The qualitative results do not show the ground truth or compare to other methods.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Difficult given that there are several details that are not clear enough (please refer to Q6). However, the authors promise to provide the code.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

Please refer to weaknesses in Q6.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The paper proposes a novel approach to unsupervised cell segmentation and shows promising results.
- Comparison to SoTA self-supervised nuclei segmentation methods is missing.
- Intuition and justification for some of the design choices is missing.
- Several essential methodology details are not clear.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This submission received generally positive comments, especially on the usage of activation maps from low layers of a neural network for pseudo cell/nuclei segmentation or detection label generation. The neural network is based on self-supervised learning, without any manual annotation. Another strength is that the proposed method can produce cell/nuclei segmentation or detection performance that is competitive with some other supervised and weakly supervised approaches.

Despite the strengths above, the reviewers raised several concerns as follows: the motivation of algorithm design is not clear or not well justified such as why similarity gives better results than contrastiveness (Reviewer #1), some key technical details are missing (Reviewers #1 and #2), lack of sufficient discussion about experimental results (Reviewer #3), and so on. In addition, although the authors do not need to provide more comparison experiments with recent relevant state-of-the-art approaches (e.g., those listed by Reviewer #1), it would be helpful if the paper can provide a discussion about the relationship or difference between the proposed method and those relevant approaches. Please improve the manuscript by considering these comments when preparing the final version.

Author Feedback

We sincerely appreciate the constructive suggestions from reviewers. The reason why similarity wins contrastiveness (Review 1), why we outperform other baselines (Review 3) are all related to the characteristic of cell-level pathological images (dense distribution and semi-regular shape). We will add detailed discussion in the final version. In addition, some missing technical details (Review 1 and 2) will also be included.

back to top

Exploring Unsupervised Cell Recognition with Prior Self-activation Maps