Authors

Yi Lin, Zeyu Wang, Kwang-Ting Cheng, Hao Chen

Abstract

Nuclei Segmentation within histology images is a fundamental prerequisite in digital pathology workflow. However, deep-learning-based nuclei segmentation methods often suffer from limited annotations. This paper aims at a realistic data augmentation for nuclei segmentation, named InsMix, that follows a Copy-Smooth-Paste principle and performs morphology-constrained generative instance augmentation. Specifically, we propose morphology constraints that enable the augmented images to acquire luxuriant information about nuclei while maintaining their morphology characteristics (e.g., geometry, location). To fully exploit the pixel redundancy of the background, we further propose a background perturbation method, which randomly shuffles the background patches without disordering the original nuclei distribution. To achieve contextual consistency between original and template instances, a smooth-GAN is designed with a foreground similarity encoder (FSE) and a triplet loss. We validated the proposed method on two datasets, i.e., Kumar and CPS dataset. Comprehensive experimental results demonstrate the effectiveness of each component and the superior performance achieved by our method to the state-of-the-art methods. Codes will be made publicly available upon acceptance.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16434-7_14

SharedIt: https://rdcu.be/cVRre

Link to the code repository

https://github.com/hust-linyi/insmix

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

Submission 807 proposes a new method for generating data augmentations to aid the task of instance segmentation of nuclei in histology images. The method contains 3 major components, each of which contributes into the final improvement. Extensive evaluation is made available, including a comparison to state-of-the-art. The proposed method clearly aids the segmentation network in all datasets shown.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Strengths:
- a difficult and important problem is addressed
- the method is well motivated, the strategy of the authors is clear
- several novel ideas are composed into a single pipeline
- the evaluation includes quantitative and qualitative results on multiple datasets
- the method clearly outperforms other state-of-the-art augmentation strategies
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- not critical: the discussion is missing an outlook to other applications of this strategy. Can it be used for other histology segmentation tasks?
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

the code will be made available, the data is public
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Suggestions:
- I did not fully understand how the background perturbation works, i.e. what do the authors mean by shuffling with a ration of alpha? Could you elaborate or provide a formula?
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The method is clearly superior to prior art, interesting new ideas are proposed
Number of papers in your stack

5
What is the ranking of this paper in your review stack?

2
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

The paper describes a novel image augmentation method, that is similar to Mix-based methods.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- image augmentation specifically for nuclei instance segmentation
- great idea with applied morphology constraints (SSD)
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- some details needs further explanation
- very little improvement in quantitative evaluation
- background perturbation in not proven enough to have positive impact on the results
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

the idea seems reproducible - the authors claim to make the code publicly available
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Comments:
- (p.1) “some studies [10,12,21] proposed to regress the distance map […]” there are more interesting and novel approaches than referenced [21] I would suggest substituting it with https://doi.org/10.1186/s13640-020-00514-6 and https://doi.org/10.1109/ICCVW54120.2021.00081
- the text about the steps of method on p.3 is redundant
- (p.4) “The parameters […] are determined by cross-validation” - please provide further explanation
- par2.2 (p.4) the background perturbation needs further explanation and/or reference to be properly proven (especially the last sentence in that paragraph)
- The results in table 2&3 would be more reliable if they were cross-validated
- it is unclear if the results presented in Table 2 were obtained with TAFE or Hovernet, please provide this detail
- The results in Table4: the claimed increase in AJI value should be confirmed with proper statistical analysis, for now the increase is questionable
- background perturbation in not proven enough to have positive impact on the results
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Overall, well structured and well presented idea, providing novel insight to image augmentation.
Number of papers in your stack

5
What is the ranking of this paper in your review stack?

3
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

This paper proposed InsMix which is a realistic data augmentation approach follows a Copy-Smooth-Paste principle. Compared with previous Copy-Paste methods, the InsMix mainly has three different components: 1. Morphology constraints (scale, shape, distance) to maintain nuclei’s morphology characteristics. 2. Background perturbation is used for exploit effective use of the background information. 3. Smooth-GAN with triplet loss is designed for generate realistic augmented nuclei images. The experiments were done in two public datasets. Ablation studies on each component were also provided.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The InsMix is based on Copy-Paste augmentation method, with some unique improvements which were especially designed for nuclei segmentation problem, such as SSD (scale, shape, distance) constraints. All the three components presented a complete pipeline and show performance increasements. The overall novelty of this paper is good.
2. The experiments were done in two widely used public datasets, also with the detailed ablation studies. The evaluation is strong.
3. The paper is well organized and easy to follow.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The actual values of some hyper-parameters such as ϵ, ρ, δ and γ which occurred in SSD constraints are not mentioned in the experiments section. The text said “determined by cross-validation.”, maybe it’s better to give the actual values of these hyper-parameters in experiments.
2. In experiments, why all the model only trained for 300 epochs? Does all the model convergent?
3. In Table 4, the performance of Cowout drops evidently, what’s the possible causes? Although this method is not proposed by the paper, I encourage the authors to analysis and discuss it.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Good. Datasets are public and code will be public.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

Please see the weakness section.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

An augmentation method based on Copy-Paste with some unique improvements which were especially designed for nuclei segmentation problem.
Number of papers in your stack

4
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper describes a novel way of augmenting data for nuclei segmentation that incorporates domain specific constraints. Experiments on two publicly available datasets demonstrate the utility of this apptroach. The background augmentation step needs to be more clearly explained. Statistical tests to determine whether the differences in results are significant would strengthen the claim that this method improves performance.
The hyperparameters used should be given – if space is too restrictive then this could be in a table in supplementary information.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

5

Author Feedback

We appreciate all the reviewers for their valuable feedback. We have carefully addressed the issues and suggestions point by point, which will be elaborated on in the camera-ready version. Below we discuss some of the questions the reviewer raised:

“The background augmentation step needs to be more clearly explained.” The idea of background perturbation is similar to the Jigsaw puzzles problem in self-supervised learning. In [1-3], the authors proposed a pretext task that recognizes the shuffled sequence of patches from the same image. It would benefit the network to learn the global information of the image. The critical difference between the proposed background perturbation to prior work is that the perturbated background area would enable the network to neglect the artifacts caused by the patching process during the training and inference. Meanwhile, it will produce more samples for the network to discriminate the contrast between the background and the foreground. The background perturbation is generated by randomly shuffling the small patches in the background area with an alpha ratio. The shuffling is done in the following way: 1. Randomly select a portion (20% in the experiment) of small patches in the background area; 2. Randomly shuffling the location of the selected patches.

[1] Self-Loop Uncertainty: A Novel Pseudo-Label for Semi-supervised Medical Image Segmentation. MICCAI, 2020. [2] 3D self-supervised methods for medical imaging. NeurIPS, 2020. [3] Self-supervised feature learning for 3D medical images by playing a rubik’s cube. MICCAI, 2019.

“Statistical tests to determine whether the differences in results are significant would strengthen the claim that this method improves performance.” For the Kumar dataset, the providers have made clear data splitting criteria. For a fair comparison, we will use the same criteria. As for the CPS dataset, we conducted five-fold cross-validation as prior works provide no clear data splitting criteria. The details of the experiment results are presented in the supplementary material. In addition, we will provide the statistical tests in the modified manuscript/supplementary material.

“The hyperparameters used should be given.” Sorry for the confusion. The hyperparameters will be provided in the modified manuscript. Specifically, we will clarify the following hyperparameters. In Eq. (1), the scale constraint \epsilon is set to 3, and the shape constraint \rho is set to 0.8. The distance constraints \delta and \gamma are set to 40 pixels and 80 pixels, respectively.

back to top

InsMix: Towards Realistic Generative Data Augmentation for Nuclei Instance Segmentation