List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Loïc Le Bescond, Marvin Lerousseau, Ingrid Garberis, Fabrice André, Stergios Christodoulidis, Maria Vakalopoulou, Hugues Talbot
Abstract
In digital pathology, various biomarkers (e.g., KI67, HER2, CD3/CD8) are routinely analyzed by pathologists through immuno-histo-chemistry-stained slides. Identifying these biomarkers on patient biopsies allows for a more informed design of their treatment regimen. The diversity and specificity of these types of images make the availability of annotated databases sparse. Consequently, robust and efficient learning-based diagnostic systems are difficult to develop and apply in a clinical setting. Our study builds on the observation that the overall organization and structure of the observed tissues is similar across different staining protocols. In this paper, we propose to leverage both the wide availability of hematoxylin-eosin stained databases and the invariance of tissue organization and structure in order to perform unsupervised nuclei segmentation on immunohistochemistry images. We implement and evaluate a generative adversarial method that relies on high-level nuclei distribution priors through comparison with largely available hematoxylin-eosin stained cell nuclei masks. Our approach shows promising results compared to classic unsupervised and supervised methods, as we quantitatively demonstrate on two publicly available datasets. Our code is publicly available to encourage further contributions.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16434-7_32
SharedIt: https://rdcu.be/cVRrP
Link to the code repository
https://github.com/loic-lb/Unsupervised-Nuclei-Segmentation-using-Spatial-Organization-Priors
Link to the dataset(s)
https://zenodo.org/record/4751737#.Yr2mtN869hG
https://warwick.ac.uk/fac/cross_fac/tia/data/her2contest
https://sites.google.com/view/bcdataset
https://warwick.ac.uk/fac/cross_fac/tia/data/pannuke
Reviews
Review #3
- Please describe the contribution of the paper
The authors propose a novel unsupervised nuclei segmentation method for IHC images. It utilizes the consistency of nuclei distribution between HE and IHC images and imposes the spatial organization prior via generative adversarial learning. Experiments on three datasets demonstrate its effectiveness
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
The way to leverage spatial organization prior between HE and IHC images is interesting and shows good performance.
-
The experimental details are clearly described.
-
Ablation studies show the effectiveness of different components.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
Missing the description of the HE database.
-
Lack of comparison with other adversarial learning methods for nuclei segmentation[1][2].
[1] Liu, Dongnan, et al. “Unsupervised instance segmentation in microscopy images via panoptic domain adaptation and task re-weighting.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020. [2] Mahmood, Faisal, et al. “Deep adversarial training for multi-organ nuclei segmentation in histopathology images.” IEEE transactions on medical imaging 39.11 (2019): 3257-3267.
-
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The discription of training details is clear. The reproducibility will be good if the author add the details for HE database construction.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
In general, this paper is well written and easy to follow. It utilizes the consistency of nuclei distribution between HE and IHC images and imposes the spatial organization prior via generative adversarial learning. The motivation is clear and experiments on 3 datasets show its effectiveness.
Other questions:
-
How to build the HE mask database? Cells in different organs may have large appearance variance. Will it affect the performance a lot?
-
The main purpose of ICH images is to estimate the immunofluorescence correspondences. How can the method be generalized to multi-class nuclei segmentation?
-
The consistency loss only considers the colour augmentation. Have you tried other types of perturbations such as the VAT?
-
Will the processed Warwick HER2 dataset be published available?
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The motivation of the method is clear. Results on 3 datasets demonstrates the effectiveness.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #4
- Please describe the contribution of the paper
This paper propose an unsupervised segmentation method for nuclei segmentation. The proposed approach train a Unet based segmentor in an adversarial training setting. The segmentor is trained to generate nuclei segmentation masks. The predicted masks and the real masks from some other nuclei dataset are used to train the discriminator. To avoid false negative segmentation, a reconstruction network is trained to reconstruct the input image based on the predicted segmentation mask. The proposed method is verified on three public datasets. The results seems promising. This paper conveys novel contribution and is well written. The reviewer has a little concern that how practical such system can be. Segmentation networks are usually used to extract the shape information of the nuclei. This requires the segmentation network to be accurate enough to capture the variance in the nuclei shapes. It is unclear if this segmentation method can achieve that level of accuracy.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Unsupervised segmentation is a challenging problem. This paper present a working system for this task.
- The experiment is extensive and sufficient.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The method proposed in this paper may be better termed as unsupervised nuclei detection. It may be hard to make a reliable segmentation method. Segmentation is usually used to extract the shape information of the cells or nuclei. If the segmentation is not accurate enough, it could extract misleading information for diagnosis.
- What are the training settings for the supervised and unsupervised in Table 1? Actually, the proposed method also makes use of the labeled data. What data and how many samples are used to train the supervised methods? What and how many labeled and unlabeled samples are used to train the proposed method? Is the proposed method using less labeled data than the Unet and Nuclick in Table 1?
- Precision and Recall may not be good metrics separately. Why not computing F1 score based on the precision and recall?
- How is the accuracy balanced?
- How is the object level segmentation achieved using Unet?
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Some details of how the metrics are computed are not present. For example:
- How is the accuracy balanced? 2.How is the object level segmentation achieved using Unet?
- The proposed method uses some labeled masks to train the discriminator. How many such masks are used?
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- The method proposed in this paper may be better termed as unsupervised nuclei detection. It may be hard to make a reliable segmentation method. Segmentation is usually used to extract the shape information of the cells or nuclei. If the segmentation is not accurate enough, it could extract misleading information for diagnosis.
- What are the training settings for the supervised and unsupervised in Table 1? Actually, the proposed method also makes use of the labeled data. What data and how many samples are used to train the supervised methods? What and how many labeled and unlabeled samples are used to train the proposed method? Is the proposed method using less labeled data than the Unet and Nuclick in Table 1?
- Precision and Recall may not be good metrics separately. Why not computing F1 score based on the precision and recall?
- How is the accuracy balanced?
- How is the object level segmentation achieved using Unet?
- It can be helpful to show some failure cases for the method to help understanding the limit of the proposed method.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper bears some novelty. The experimental results has some gaps compared to supervised and semi-supervised method, but looks promising for unsupervised methods.
- Number of papers in your stack
7
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #5
- Please describe the contribution of the paper
This paper presents a novel unsupervised method for nuclei segmentation incorporating priors from public H&E datasets for use in IHC-stained images. The method exploits available information at the segmentation level by encoding and identifying the histological tissue characteristics that are independent of the staining.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- novel unsupervised method
- stain independence
- good results on 3 datasets in comparison with SOTA methods
- good results with nuclei and membrane staining images
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- impact of postprocessing included in proposed method is not discussed
- lack of evaluation in relation to tissue with different compactness
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Loss functions and hyperparameters are present in the paper so the idea could be roughly reconstructed. But taking in consideration the amount of parameters and size of the model without actual code it is impossible to fully reproduce the results. There is no information about public release of the code so the reproducibility of the paper is limited.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The overall idea of the novel unsupervised method based on GANs is quite well described. I have but a few comments:
- the figure reference on p.3 should bo to fig. 1 instead of fig.2
- I suggest a redesign of figure 1 to be more consistent with the in-text description (generator-segmentator) and to possibly include the other discriminator
- It would be also interesting to evaluate the influence of the included post-processing in the proposed metod. How much the evaluation metrics change without erosion and watershed. Maybe Unet+watershed would be even better but since only proposed method has postprocessing it is unknown. Please provide further explanation/results in either main paper or supplementary material.
- Since this method is based on spatial organisation priors, it would be interesting to evaluate the consistency of the results in relation to tissue architecture compactness. (this is just a suggestion for future investigations)
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The idea is novel and well presented. The description is clear. The results are significantly better than reference SOTA methods, and based on 3 very different datasets. Overall, the presented novel unsupervised method for nuclei segmentation that can learn from H&E and then can be used for IHC-stained tissue images is very promising for the computational pathology.
- Number of papers in your stack
7
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
All the three reviewers appreciated the idea of using spatial distribution priors for unsupervised nuclei segmentation and the good results on different image datasets. While the reviewers gave high scores (one weak accept and two accepts), they raised some questions or suggestions to improve the manuscript: describe the H&E data construction, explain how to handle multi-class nuclei segmentation, clarify whether the method segments nuclei with accurate nuclei shape preservation, add more details about experimental setup for Table 1 and the metric computation, and discuss the effects of the post-processing used in the proposed method. Please consider these suggestions to improve the paper when preparing the final version.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
1
Author Feedback
Firstly, we would like to thank all the reviewers and the area chair for their careful assessment of our paper and their comments. While all the reviewers appreciated the novelty and the promising results of our approach, there were also a number of questions raised during the review process that we would like to address. Minor concerns such as typos will be addressed in the camera-ready version. Moreover, we would like to highlight that the code for the method will be published online, along with the code used to create the HER2 dataset and ground truth masks constructed for the test set.
R3 pointed out the lack of comparison with two specific adversarial methods. With respect to these, we argue that a comparison is not so straightforward. Specifically, the work of Mahmood et al. uses some ground truth masks to train the segmentation network, which is not the same as the unsupervised setting to which our method belongs. On the other hand, Liu et al.’s approach relies on a domain adaptation idea through a stain transfer scheme. Even though such an approach could be used in this study, it would require the model to transfer the stain from the H&E domain to IHC to train an additional model using the artificially IHC-generated tiles. This task is quite challenging for the IHC case due to the inherent differences between the different types of possible IHC staining protocols (e.g., membrane - HER2, nuclear - Ki67). Additionally, such a multi-step approach can be prone to accumulative errors related to the different models. We believe that a fair comparison with the proposed method would not be feasible for the aforementioned reasons.
R4 and R5 underlined the need to provide further details regarding the post-processing used for Unet. Both the thresholding and Unet results were post-processed using the same pipeline as the one used in the proposed method, including morphological operations and the watershed algorithm, to perform instance segmentation. That way, we ensure a fair comparison between the tested pipelines.
R3 and R4 highlighted the need for a more precise description of training settings and H&E database composition. For table 1, we trained Unet using 709 512x512 labeled samples and 303 samples for hyperparameter tuning. For Nuclick, we used the publicly available pre-trained network and manually annotated each nuclei centroid on the test set. The proposed method leverages spatial organization information from around 7700 256x256 nuclei segmentation masks of H&E images collected across 19 different tissue types (Pannuke database). The datasets used are publicly available, and one can refer to [4] and [5] for an extensive presentation.
R4 raises the point that our approach is more a detection method than a segmentation method because our architecture does not enforce a precise output shape that corresponds to the input. This is not entirely true since the CycleGAN model we use ensures appearance similarity, thus encouraging shape similarities between input and output.