List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Zhenrong Shen, Maosong Cao, Sheng Wang, Lichi Zhang, Qian Wang
Abstract
Automatic examination of thin-prep cytologic test (TCT) slides can assist pathologists in finding cervical abnormality for accurate and efficient cancer screening. Current solutions mostly need to localize suspicious cells and classify abnormality based on local patches, concerning the fact that whole slide images of TCT are extremely large. It thus requires many annotations of normal and abnormal cervical cells, to supervise the training of the patch-level classifier for promising performance. In this paper, we propose CellGAN to synthesize cytopathological images of various cervical cell types for augmenting patch-level cell classification. Built upon a lightweight backbone, CellGAN is equipped with a non-linear class mapping network to effectively incorporate cell type information into image generation. We also propose the Skip-layer Global Context module to model the complex spatial relationship of the cells, and attain high fidelity of the synthesized images through adversarial learning. Our experiments demonstrate that CellGAN can produce visually plausible TCT cytopathological images for different cell types. We also validate the effectiveness of using CellGAN to greatly augment patch-level cell classification performance. Our code and model checkpoint are available at https://github.com/ZhenrongShen/CellGAN.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43987-2_47
SharedIt: https://rdcu.be/dnwJ2
Link to the code repository
https://github.com/ZhenrongShen/CellGAN
Link to the dataset(s)
N/A
Reviews
Review #2
- Please describe the contribution of the paper
The authors propose a conditional GAN-based image generation method for generating liquid-based cytology images of different types of cervical cancer.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors introduce a conditional GAN based on FastGAN to generate synthesize cytopathological images of cervical cells. They utilize a non-linear mapping network, which embeds the class labels to perform layer-wise feature modulation in the generator, to inject cell type for fine-grained conditioning. Additionally, they introduce the Skip-layer Global Context module to capture the long-range dependency of cells for precisely modeling their spatial relationship. In the experiment, they also verify the potential of the proposed method in data augmentation by comparing the cell classification task.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
While the proposed method has no major weaknesses, there are limitations that have been mentioned in the article. For example, the method cannot directly control the generation of detailed image attributes such as nucleus size and nucleus-cytoplasm ratio. Additionally, the synthesized image size is currently limited to 256 x 256 pixels, and there has been no attempt to experiment on larger-sized images.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The proposed work is described clearly and logically and has a certain reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
It is best for the author to consider attempting to validate the effectiveness of the proposed method in a publicly available dataset to further demonstrate its generalizability and potential for wider use.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The authors’ description of the proposed method is clear, logical, and reproducible. In the experiment, the authors not only use traditional evaluation indicators and visual display to illustrate the advantages of the proposed method, but also further verify the potential of the proposed method in data augmentation by comparing the cell classification task. However, due to the privacy of the data, the validity of the method cannot be further verified.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #1
- Please describe the contribution of the paper
The manuscript leverages a GAN model (FastGAN) with class-conditioned generation to synthesize image tiles of thin-prep cytologic test slides. The realism of the generated tiles is quantitatively compared against another gan model (BigGAN) and a recent diffusion model (LDM). The authors show a novel “skip-layer global context” module in the generator, and propose the injection of class embeddings in the generator feature maps at multiple scales. A dataset has been collected and classified, then used to train the proposed model. Comparative visual results and FID score show significant improvement over BigGAN and LDM.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- A well structured method of generating realistic medical (cytology) images that are conditioned by desired class indicator
- The method is shown to aid baseline image classifiers
- Ablation study to indicate the impact of the three proposed modifications to the base model
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Although the results are good, the application of simulating cytology image tiles, when a whole slide image is available (random tiles can be collected from a wsi) seems to be going in a tangential direction.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Mostly reproducible. Sufficient references and details are provided in the paper for the proposed method, but not the implementation details of the compared methods.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Generally well written manuscript, the introduction should include prior work, cutting down a bit of the long winded background. The connection to cervical cancer is motivational, and seems to not influence the methods proposed, perhaps the title is written in a limiting manner. The references to literature and methods which inspire the proposed method are clearly indicated. Comparison experiments and ablation are good and results are significant.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The structure of the paper is very good, and matches the typical MICCAI paper. The wordings are well measured to provide enough information. The methods in the paper are not new, but the authors have put together a seemingly well-performing and useful model, gathered a dataset and realized the application.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
Generating synthetic cellular images are usually challenging, as it requires the model to learn the local cellular morphology features, also need to understand the global correlations of the cells. The proposed CellGAN included an innovative skip-layer global context module trying to learn the spatial correlations of cell-cell interactions, which is critical in improving the synthetic images quality.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors provide a solid comparison quantitative metrics, including frechet inception distance, and other general metrics, which could convince the audience with the conclusions.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The information of the dataset does are relatively insufficient, such as how many institutions/hospitals involved in contributing the dataset, what are the scanners used, etc. I also doubt the input with a random vectors instead of real images in training the GAN, does this could cause the mode collapse? If so, how to address it?
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors did include the sufficient details of the computational infrastructure, however, the hyper-parameters provided might not be enough to reproduce the model training. The authors did not provide sufficient details regarding data preprocessing.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Generating synthetic cellular images are usually challenging, as it requires the model to learn the local cellular morphology features, also need to understand the global correlations of the cells. The proposed CellGAN included an innovative skip-layer global context module trying to learn the spatial correlations of cell-cell interactions, which is critical in improving the synthetic images quality.The information of the dataset does are relatively insufficient, such as how many institutions/hospitals involved in contributing the dataset, what are the scanners used, etc. The authors did include the sufficient details of the computational infrastructure, however, the hyper-parameters provided might not be enough to reproduce the model training. The authors did not provide sufficient details regarding data preprocessing.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Detailed illustration of the architecture, and nicely created pipeline diagram.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper proposed a conditional GAN-based image generation method for generating liquid-based cytology images of different types of cervical cancer. The authors show a novel skip-layer global context module in the generator, and propose the injection of class embeddings in the generator feature maps at multiple scales. The high quality of experiments and methodologies are appreciated by all reviewers. Moreover, the paper is well written and easy to follow. However, we still encourage the author to address some detailed concerns from reviewers, such as the clearness of the methodology and experimental design and missing details.
Author Feedback
Thanks for the valuable comments. We appreciate the recognition of our work and have addressed each of the reviewers’ concerns as follows.
Regarding the application limitation when WSIs are available (R1): Thanks for the feedback. Though WSIs can provide a large number of random tiles, the abnormal cells within them are limited in number. Our method, in contrast, can synthesize arbitrary number of abnormal cells, which increases the overall quantity and diversity of available data and thus improves the ability of cytological classifiers to identify abnormal cells in real-world scenarios.
Regarding the implementation details of the compared methods (R1): We apologize for not including the implementation details due to the page limitation. Herein we provide a detailed description: (1) BigGAN. We selected the 256×256 architecture, and set the same hyper-parameters as our proposed method except for the batch size (set to 32) due to the GPU memory. (2) LDM. We selected KL-reg autoencoder with a downsampling factor of 8, and used the learning rate of 1.5e−6, batch size of 32, and Adam optimizer for training 100k iterations. For U-Net, we used the learning rate of 2.0e−5, batch size of 128, Adam optimizer, and DDPM objective for training 100k iterations. And we took 100 DDIM steps for sampling images.
Regarding the effectiveness in a public dataset (R2): Thanks for the suggestion. We agree that demonstrating the effectiveness and generalizability of our method on a public dataset would be valuable for wider use and community validation. However, we were unable to conduct such experiments for this paper due to time and page limitations. Nevertheless, we will continue to explore the potential benefits of our method on public datasets in the future work.
Regarding the information of the in-house dataset (R3): Thanks for the comment. Our private dataset was collected from three collaborating clinical institutes, and we will make it clear in the revised manuscript. We cannot disclose the exact digital scanner model, since it is an internal experimental equipment at this moment. However, we strictly followed the standard TCT scanning protocol, and our data was clinically validated.
Regarding the concern about the possibility of mode collapse (R3): Thanks for the insightful question. While we did not encounter this issue in our experiments, we acknowledge that it is a potential concern. There are a few reasons why we believe our approach did not suffer from mode collapse: (1) We built CellGAN upon a stable GAN architecture (FastGAN) that is known to be less prone to mode collapse. (2) We implement spectral normalizations across CellGAN, which improves the stability of GAN training process and mitigates potential mode collapse. (3) We applied differentiable augmentation during training that can also be helpful because it allows GAN to learn from a wider range of data variation, thus increasing the diversity of the generated samples and reducing the risk of mode collapse. Of course, we recognize that mode collapse is still a valid concern and could potentially occur even with these measures. In future work, we will continue to explore ways to address this issue and make our method more robust and reliable.
Regarding the details of data preprocessing (R3): Thanks for the comment. The collected WSIs were divided into multiple 1024×1024 image tiles at first. Two experienced pathologists provided box annotations of abnormal cells for each image tile by manual inspection. We then cropped 256×256 patches at the center of each bounding box and obtained the final dataset.
Regarding the reproducibility of the model training (R3): We apologize for not including sufficient hyper-parameter details due to the page limitation. We understand the importance of hyper-parameters in reproducing the model training process, and we will make our code and model checkpoint publicly available once our paper is finally accepted.