Authors

Fuyong Xing, Toby C. Cornish

Abstract

Due to domain shifts, deep cell/nucleus detection models trained on one microscopy image dataset might not be applicable to other datasets acquired with different imaging modalities. Unsupervised domain adaptation (UDA) based on generative adversarial networks (GANs) has recently been exploited to close domain gaps and has achieved excellent nucleus detection performance. However, current GAN-based UDA model training often requires a large amount of unannotated target data, which may be prohibitively expensive to obtain in real practice. Additionally, these methods have significant performance degradation when using limited target training data. In this paper, we study a more realistic yet challenging UDA scenario, where (unannotated) target training data is very scarce, a low-resource case rarely explored for nucleus detection in previous work. Specifically, we augment a dual GAN network by leveraging a task-specific model to supplement the target-domain discriminator and facilitate generator learning with limited data. The task model is constrained by cross-domain prediction consistency to encourage semantic content preservation for image-to-image translation. Next, we incorporate a stochastic, differentiable data augmentation module into the task-augmented GAN network to further improve model training by alleviating discriminator overfitting. This data augmentation module is a plug-and-play component, requiring no modification of network architectures or loss functions. We evaluate the proposed low-resource UDA method for nucleus detection on multiple public cross-modality microscopy image datasets. With a single training image in the target domain, our method significantly outperforms recent state-of-the-art UDA approaches and delivers very competitive or superior performance over fully supervised models trained with real labeled target data.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16449-1_61

SharedIt: https://rdcu.be/cVRXz

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper presents a generative adverserial network (GAN) based unsupervised domain adaptation strategy for limited target training data case. The main contribution is achieved by a stochastic data augmentation module integrated to the GAN network. Unsupervised domain adaptation performance is increased.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The problem is defined well. The proposed stochastic data augmentation is formulated well. Experiments are quite comprehensive; they are conducted on 5 different datasets (4 public, 1 in-house) and consists of comparison with 4 other methods and ablation studies.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The proposed method has limited novelty. It adds a stochastic data augmentation module to a GAN network. In numerous papers, data augmentation techniques have already been shown to be an effective solution for training deep neural networks on small datasets. Also, the paper studies a particular case (limited target training data case) of unsupervised domain adaptation. If one has enough target training data, the proposed data augmentation module will not be necessary. Thus, the proposed method does not increase the general capabilities of GANs for unsupervised domain adaptation. Hence, this paper only repeats known contributions of data augmentation by proposing a new data augmentation module and integrating it to a GAN network. Compared to existing works, this paper proposes a slightly modified GAN architecture and a slightly improved data augmentation module.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The method is defined clearly and all implementation details are given.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

The paper is well-organized and well-written. Also, the experiments are carried out extensively. It is a pleasure to read such a paper.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This is a well-conducted research work with good results. It proposes an improvement for unsupervised domain adaptation with GANs when the target training data is limited. However, its novelty is rather limited; a new data augmentation module is integrated to a GAN network. Moreover, it only addresses the low-resource case of the unsupervised domain adaptation (UDA). It is very likely that the proposed method won’t be needed in the case of sufficiently big target training data. Thus, the proposed method does not offer a general improvement to GANs for UDA, although it presents an effective solution to GANs for UDA with limited target training data. It should also be noted that data augmentation techniques have been numerously proved to be an efficient way to efficiently train deep neural networks on small datasets. This paper mainly repeats known contributions of data augmentation and then proposes a new data augmentation module, which is a case-specific solution.
Number of papers in your stack

4
What is the ranking of this paper in your review stack?

2
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper
- The authors propose a domain adaptation workflow to enable nuclei detection across different microscopy image modalities. Although the approach is not new, the authors propose a workflow that can provide accurate results using small annotated datasets.
- They contribute a differentiable data module that could be connected to different approaches as it is agnostic to the architecture and the loss function.
- The accuracy of the proposed approach is compared with other existing approaches showing that it can outperform them in the situation of having a few target training images.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The authors present a creative approach of transforming a source image, force this transformation to have a coherent structural meaning and back (i.e., get a target image, transform it into the source’s domain and compare the structural meaning again). In this case, the structural information is forced by training a nuclei detector. This way, they guide the network on the creation of results that structurally reassemble more to each other, despite changing the domain.

The authors have worked on the definition of a fully differentiable data augmentation that works on the
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

While the authors presented an original work, I found it difficult to understand what’s the real motivation or the gaining that drive this wok. First, from the title it seems that the work is specially designed to allow nuclei detection when there’s lack of annotated to train a specific model for it. However, the issue about lacking data is to train the domain adaptation (DA). DA is an unsupervised task for which you would only need the input images. For example, the following sentence “In this paper, we propose a novel GAN-based UDA method (see Fig. 1) for cross-modality cell/nucleus detection in a low resource setting, where unlabeled target training data is scarce, a more realistic case but rarely explored.” is confusing because the low resource setting to be explored here is about the unsupervised method (non-labelled data). Also, the content of this sentence may relative: scarce labelled data in bioimage analysis is the common situation and there are plenty of works targeting it, so the authors may want to be more specific in this statement. Same for the conclusions “With limited unlabeled target training data (e.g., 1 image),…”, you do not need to have the labeled data (i.e., nuclei detection) for the target to train the method, but only to evaluate it’s performance. Second, the authors motivated this by the fact that acquiring data is expensive, but could it be possible to use publicly available datasets of the same modality? (even if they are acquired with different devices). Finally, they are not considering any other biological meaning of the generated image despite the nuclei detection, so it should be possible using an heterogeneous dataset of a single modality and biological organism.

Related to the previous comment, along the text, the authors say that the domain adaptation is supported by the nuclei detection task. However this work was proposed to get nuclei detection supported by the fact that we may need domain adaptation.

It is important to specify in the text that this approach cannot be applied to all type bioimages as it potentially requires, by definition, all nuceli in the image to be visible.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

As far as I saw (please, correct me if I’m worng), the authors do not provide any link to the dataset or the code, which makes it incredibly difficult to reproduce due to the complexity of the architecture.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

When defining y^S_i in page 3, shouldn’t it take into account the pixel size in images?

The authors define a new approach for the differentiable data augmentation but I strongly recommend them to include a graphical illustration of this process (Section 2.2) to really understand what they are doing. In the limit, it could be in the supplementary materials.

When explaining the data augmentation process, the authors sometimes speak about differentiable data augmentation and others, about invertible. Although these two properties are important to integrate the process in the training, I would specify in the text why you need both features and when they are used.

Please, correct me if I am wrong, is the augmentation applied to the source image the same one as for the target one? The question raised from this sentence “To address this issue, we apply data augmentation to both real and translated images before feeding them to the discriminators and conduct the augmentation when training both generators and discriminators”

Please correct the following sentence: “… data distribution ONLY if the augmentation contains invertible transformations…” Please correct: “training data can posE serious challenges”

Please, elaborate more on this sentence as it is not clear to what you are referring “This target task-based augmentation can also reduce the effects of data for which the discriminator has no access to the labels.”

What is E in Eq 1,2,3? Are you referring to the estimated mean value? please, define it.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I like the idea presented in the paper and the approach followed by the authors. I think they need to work more on graphically show their contributions and technical approaches, as well as better introduce the motivation and what is the specific problem for which they propose this solution.
Number of papers in your stack

5
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

This manuscripts presents a method for domain adaptation for the task of nucleus detection from microscopy data, for the case of small amount of labelled target data. The authors present several improvements for state-of-the-art, including usage of task-specific model to support target-domain discriminator, stochastic data augmentation, and targeted consistency constraint. This results in superior performance with respect state-of-the-art domain adaptation methods.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

• Strong and complete submission: clear and advanced methodology, convincing validation on versatile data. • Results superior to the current state-of-the-art and on pair with direct training in the target domain.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

• Visualization of the qualitative results needs to be improved.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

All the methods developed in this paper are clear and valid. Also the implementation details are properly described and values of all the parameters are reported.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1. Presentation of the qualitative results needs to be improved, as the images are too small and hardly readable on print.
2. It might be useful to add the size of each of the four validation data sets (in terms of number of images) to the header of Table 1.
3. It would also be interesting to see comparison between the proposed method and the reference ones for the less extreme case, when more training images are available.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This is a very strong submission, with advanced methodology, clear presentation and convincing validation. In particular, the authors have introduced several methodological improvements including task-specific model for supporting the source-target discriminator, target consistency constraint and stochastic data augmentation scheme (in both source- and target domains). This allows efficient training under the scenario of low availability of (unlabeled) data in the target domain. The authors have demonstrated the power of their approach by using a single image from the target domain. The results of this experiment were much better compared to the state-of-the-art methods (trained on the same data) and comparable to training directly in the target domain. In addition, the authors perform an ablation study demonstrating, in particular, importance of the introduced stochastic data augmentation step.
Number of papers in your stack

4
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #4

Please describe the contribution of the paper

The paper tackles domain adaptation (staining of microscopy slides) given very few annotated target samples. It combines a GANS with (i) a data augmentation module between the translated image generator and the loss function; (ii) co-training of a nucleus detector; and (iii) a loss to reward consistency of nucleus labels cross-domain. They test against several current methods, reporting mean +/- std dev of various metrics. Their F1 scores are superior and/or competitive in this regime.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The use-case considered is very important: domain adaptation to a target domain with few labeled samples. The paper appears to have a strong foundation in the literature, drawing on both the strengths and weaknesses of various prior methods. The incorporation of a nucleus detector into the optimization program is clever - it leverages information which is otherwise ignored (it of course assumes that the prediction task is known). The construction of the data augmentation is also clever, as a means to gain the benefit of augmentation without derailing the GANS to learn the augmentations.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The lack of a publicly-available codebase is a regret, given how touchy GANs can be. Any weaknesses require more expertise than mine to see :)
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper makes no mention of a public codebase, which is a substantial regret.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

“Low resource setting” (LRS) also refers to situations in poorer countries, rural areas, lack of electricity or internet, etc. If LRS is a standard term for the sparse target domain data situation addressed in the paper, great. If not, perhaps use a different term. “Sparse target data regime”?

The layout of mean and std devs in Table 1 is a bit cluttered. Is there a way to format them in one line as $\mu \pm \sigma$?
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Please see the “Strengths” section. On a personal level, this paper addresses a particular need in my professional work.
Number of papers in your stack

3
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper proposes a method for nucleus detection across modalities with a domain adaptation strategy. The paper seems well written and all 3 reviewers agree that this has a significant contribution that can be of interest to the MICCAI audience.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

1

Author Feedback

We thank the area chair and the reviewers for their positive and constructive comments. We will consider the comments/suggestions and revise our manuscript accordingly for the final version.

back to top

Low-Resource Adversarial Domain Adaptation for Cross-Modality Nucleus Detection