Authors

Amandeep Kumar, Ankan Kumar Bhunia, Sanath Narayan, Hisham Cholakkal, Rao Muhammad Anwer, Jorma Laaksonen, Fahad Shahbaz Khan

Abstract

In this work, we propose a few-shot colorectal tissue image generation method for addressing the scarcity of histopathological training data for rare cancer tissues. Our few-shot generation method, named XM-GAN, takes one base and a pair of reference tissue images as input and generates high-quality yet diverse images. Within our XM-GAN, a novel controllable fusion block densely aggregates local regions of reference images based on their similarity to those in the base image, resulting in locally consistent features. To the best of our knowledge, we are the first to investigate few-shot generation in colorectal tissue images. We evaluate our few-shot colorectral tissue image generation by performing extensive qualitative, quantitative and subject specialist (pathologist) based evaluations. Specifically, in specialist-based evaluation, pathologists could differentiate between our XM-GAN generated tissue images and real images only 55% time. Moreover, we utilize these generated images as data augmentation to address the few-shot tissue image classification task, achieving a gain of 4.4% in terms of mean accuracy over the vanilla few-shot classifier.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43898-1_13

SharedIt: https://rdcu.be/dnwAK

Link to the code repository

https://github.com/VIROBO-15/XM-GAN

Link to the dataset(s)

https://zenodo.org/record/53169

Reviews

Review #3

Please describe the contribution of the paper

This paper proposes a few-shot image generation method for histopathology. The authors introduce a new fusion block that improves the local feature consistency. The paper is evaluated on colorectral tissue image generation by performing qualitative, quantitative and specialist evaluations. The method outperforms the related work.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The paper is well-written and easy to follow.
2. The proposed method seems to be novel to the best of my knowledge and few-shot image generation for histopathology images is not explored as far as I know.
3. The proposed method outperforms the related work.
4. The work is well-motivated.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The method is only compared against one other work (LoFGAN). I suggest the authors to compare the method against more related few-shot image generation methods (e.g. FIGR).
2. The authors claim that the proposed fusion block improves the local feature consistency. Is there any experiment to verify this?
3. The authors claim that the proposed method generated images with higher diversity. This is only shown qualitatively. I suggest the authors to also add metrics that measure the diversity (such as precision and recall).
4. The method is evaluated using FID and LPIPS. Adding more metrics (IS, KID, Precission and Recall) can improve the paper.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The architecture may be a bit complex to implement from scratch, but the authors mention they will release the code publicly.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

See weaknesses.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper is interesting and has merits. There are some weaknesses that can be improved.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #1

Please describe the contribution of the paper

Authors propose a few-shot image generation method for histopathological image. The proposed method can simultaneously focuse on generating high-quality and diverse images.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Authors construct a cross-attention based feature aggregation framework and cross- modulated layer normalization for few-shot histopathological image generation. It is a novel formulation.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

There is no explanation why cross-attention and cLN can process few-shot problem? More comparison experiments are needed.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Code and data do not seem to be available. Sounds good.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

The few-shot image generation is a novel research interesting, especially for histopathological image. The Method is described with implementation details. However, the expriments is limited. Explaining more what was done, than highlighting the relevance of the study.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Authors construct a cross-attention based feature aggregation framework and cross- modulated layer normalization for few-shot histopathological image generation. It is a novel formulation.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

The paper present a Cross-modulated Few-shot image generation module, XM-GAN, for colorectal cancer classification. The model use a base image and a number of several images to generate diverse tissue images, which are then used to augment the training dataset of unseen categories, to improve the tissue classification performance.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The network strucutre seems to be clear and the quality of generated images seem to be better than available LoFGAN, the classifiation performance using the augmented data also seems to be improved.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

1) What about the discriminator of the proposed XM-GAN, there is no information provided. 2) It’s good to show the FID and LPIPS of the generated tissue images, but how are they calculated, which network do you use to calculate the distance? How many generated images are used to calculate the FID and LPIPS? No details are given. 3) There are lots of style based image-to-image translation GAN models like FUNIT, the authors shall include more related GAN models for comparison. 4) The tissue dataset used for comparison is quite small, with 8 categories (40 images/category) available, the number of unseen category is small as well.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Seems to be reproducible, not 100% sure
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

As listed in the weakness section, there are a numer of important details missing. The results are not so convincing, due to the small scale of the tissue classification dataset.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The whole framework seesm to be well motivated, the results suggest that the proposed GAN model generated higher quality images, and can help improve the classification performance.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper presents a few-shot image generation method for histopathology. The proposed method is interesting and novel. The reviewers however also noted the extent of evaluation is limited. Comparison with more state-of-the-art methods and more datasets would be good.

Author Feedback

Our Response

We thank the reviewers and meta reviewer for their time, valuable feedback, and insightful comments on our submission. Code: We will make our well-documented code and pre-trained model publicly available.

Reviewer #1

Cross-attention and cLN used for few-shot problems : We introduce cross-attention within our controllable fusion block (CFB) to densely aggregate local region feature of the reference tissue images based on their congruence with the base image tissues. Our cLN learns sample dependent modulation weights and strive to perform feature normalization for generating images that are similar to the few-shot samples.

More comparison experiments are needed : Thank you for your suggestion. Considering the unavailability of existing approaches for our novel problem setting, we leverage an existing few-shot image generation technique (LofGAN) for cancer tissue image generation. Our experiments and human study with subject experts (pathologists) demonstrate the benefits of the proposed method. In the present limited page conference version we could not include many of these comparisons and studies since it is non-trivial to adapt many generation methods targeting natural images since the problem of few shot colorectal tissue generation require capturing fine-grained tissue details. We are planning to include more detailed studies in our future work.

More experiments to highlight the relevance of the study : Table 3 presents an ablation study, which outlines the inclusion and purpose of each module in our architecture. This study aims to elucidate the rationale behind incorporating these modules to generate images of superior quality and diversity. We are planning to include more ablation studies and comparisons in our future work.

Reviewer #2

Discriminator D of the proposed XM-GAN : we use the same Discriminator as Liu. et. al. [Few-Shot Unsupervised Image-to-Image translation, ICCV 2019].

Network for calculating the FID Score : Similar to LoFGAN, we extract features from real and fake images using a pre-trained Inception network. The mean and covariance of these feature representations are computed and the Wasserstein-2 distance between the two distributions are used to obtain FID scores. For more details, please refer to LoFGAN paper.

Number of Images to Calculate the FID and LPIPS : Following LoFGAN, we randomly generate 128 images for each class and combine the images from unseen classes to calculate the FID and LPIPS scores.

Comparison with Few-Shot Image Generation : Thanks for the suggestion. In the present limited page conference version we could not include many of these comparisons and studies since it is non-trivial to adapt many generation methods targeting natural images since the problem of few shot colorectal tissue generation require capturing fine-grained tissue details. We plan to carefully adapt more GAN models for colorectal tissue generation in our future work.

The tissue dataset used is small : We agree with the reviewer. The datasets for colorectal tissue classification are generally small due to the scarcity of labeled positive cancer images. This clearly shows the necessity of few-shot image generation for this important problem setting. We are also planning to release a synthetic image dataset in our future work.

Reviewer #3

Adding metrics that measure the diversity : Please note that the LPIPS metric we reported indicates the diversity of generated images (the relation between LPIPS and diversity is also mentioned in the LoFGAN paper). Moreover, our experiments with subject experts (pathologists) demonstrate the merits of images generated by our methods.

Adding more metrics : Thanks for the suggestion. More evaluation metrics will be included in our future work.

back to top

Cross-modulated Few-shot Image Generation for Colorectal Tissue Classification