Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews Back to top
List of Papers By topics Author List
Paper Info Reviews Meta-review Author Feedback Post-Rebuttal Meta-reviews

Authors

Yash Sharma, Sana Syed, Donald E. Brown

Abstract

In this work, we propose a mutual information (MI) based unsupervised domain adaptation (UDA) method for the cross-domain nuclei segmentation. Nuclei vary substantially in structure and appearances across different cancer types, leading to a drop in performance of deep learning models when trained on one cancer type and tested on another. This domain shift becomes even more critical as accurate segmentation and quantification of nuclei is an essential histopathology task for the diagnosis/ prognosis of patients and annotating nuclei at the pixel level for new cancer types demands extensive effort by medical experts. To address this problem, we maximize the MI between labeled source cancer type data and unlabeled target cancer type data for transferring nuclei segmentation knowledge across domains. We use the Jensen-Shanon divergence bound, requiring only one negative pair per positive pair for MI maximization. We evaluate our set-up for multiple modeling frameworks and on different datasets comprising of over 20 cancer-type domain shifts and demonstrate competitive performance. All the recently proposed approaches consist of multiple components for improving the domain adaptation, whereas our proposed module is light and can be easily incorporated into other methods.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16434-7_34

SharedIt: https://rdcu.be/cVRrR

Link to the code repository

https://github.com/YashSharma/MaNi

Link to the dataset(s)

PanNuke - https://jgamper.github.io/PanNukeDataset/

CoNSep - https://warwick.ac.uk/fac/cross_fac/tia/data/hovernet/

TNBC - https://zenodo.org/record/1175282#.YrvcJ-xBy3J

KIRC - https://becklab.hms.harvard.edu/software/psb-crowdsourced-nuclei-annotation-data-1

TCIA - https://wiki.cancerimagingarchive.net/display/NBIA/Downloading+TCIA+Images



Reviews

Review #1

  • Please describe the contribution of the paper

    This paper addresses an interesting topic of unsupervised domain adaptation applied in Nuclei segmentation. The core idea of this paper is to use a lower bound based on Jensen-Shannon Divergence (JSD) to optimize Mutual information between different classes of source and target images, using the ground truth segmentation masks and pseudo-labels. Two series of experiments are made to show the effectiveness of the proposed method.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    • While employing mutual information for domain adaptation is not a new idea (i.e. 1, 2,3), the proposed approach, using masked average pooling, contrasting the averaged representation, and leveraging JSD as a surrogate loss for MI, can be considered a novel contribution.

    • Results show clear improvements over recent approaches for UDA in segmentation.

    • The paper is well written and easy follow. The motivation and contributions of the work are clearly described in the introduction, and the method is presented with sufficient details.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    • The paper motivates the choice of using JSD as a lower bound of MI by discussing its advantages over InfoNCE and by saying that JSD maintains good performance without using a large number of negative samples. However, this evidence is not supported experimentally. If one carefully checks their cited papers, 4 said that InfoNCE provided better results compared with JSD in classification tasks, while 5 indicated in its supplementary material that reducing the batch size (thus the negative examples) degraded the performance. Hence, it is not clear to me whether this statement holds.

    • In addition, I am not entirely convinced about their provided experimental results. The performance values in the comparison methods were taken from reference papers, but the authors should at least report results using their own runs. Even while using the same codebase, many parameters such as the library version, random seeds, etc. can impact the results.

    • Authors could better motivate the applicability of the JSD lower bound on MI to UDA. Specifically, I wonder how to model and sample the joint distribution in Eq (1) since z_s and z_t come from independent images. Isn’t the joint same as product of marginals in this case?

  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
    • I am a little concerned about the reproducibility of this paper. I would suggest the authors cite the values from previous papers and report their own performances using the same code based on their own runs. Furthermore, I would be happy to see that the author can make their code and dataset open-source.
  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

    • The second paragraph of the introduction mentioned multiple methods, but did not properly cite them. The authors are invited to carefully cite these papers.

    • Regarding the related works, the paper only presented methods using adversarial training and pseudo-label. However, more methods should be mentioned, such as those using mutual information (2,3), using metric learning (6,7,8), and using co-training based frameworks (9,10)

    • As mentioned previously, the authors used JSD instead of InfoNCE or MINE [11] to provide a stable estimation of MI, however, the authors do not verify this claim experimentally in the ablation study. Moreover, only one negative pair is used per positive pair, and the paper claimed this would not decrease the performance. However, no evidence has been shown to support this hypothesis. I would like to see experimental elements for both claims.

    • The projection head is 1 x 1 convolutional layer. Parallel works usually used nonlinear heads. The authors may want to explain the motivation for this choice.

    • The Word “Actual” on the top line of Fig. 1 should be “ground truth”?

    • The discriminator’s structure should be explained clearly.

    • There is no “source only baseline” for Table 1.

    • As mentioned previously, the values reported in Tables 1 and 2 should include the runs in the authors’ own codebase/environment. I encourage the authors to provide further results on these settings.

    • For the presented tables, the authors can bold the best performing values in order to provide a better visual understanding.

    • For the ablation study, the authors only tested MI loss weight and the different sampling/pooling methods. For table 3, I would like to see the values with a weight higher than 1, to see if it helps the performance.

    • The authors should provide the mean and std for each experiment to show the variance of each method.

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    5

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The proposed method is interesting and comprises novel elements. Results are promising. However, some issues should be addressed (see comments).

  • Number of papers in your stack

    5

  • What is the ranking of this paper in your review stack?

    2

  • Reviewer confidence

    Confident but not absolutely certain

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    5

  • [Post rebuttal] Please justify your decision

    The authors have addressed several concerns, however they did not clearly motivate the applicability of the JSD lower bound on MI to UDA, as requested. Nevertheless, I still believe the paper has more merits than flaws.



Review #2

  • Please describe the contribution of the paper

    In this work, the authors proposed a domain adaptive nuclei instance segmentation method based on a two-stage training strategy. The proposed method facilitates the target feature generation by mutual information maximization mechanism. Extensive experiments on UDA nuclei semantic and instance segmentation benchmarks have indicated the effectiveness of the proposed method by achieving state-of-the-art performance.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1)The proposed method has achieved state-of-the-art performance under various UDA nuclei segmentation benchmarks.

    2)This work has studied an important research topic, i.e., UDA nuclei semantic and instance segmentation tasks, which can enhance the models’ generalization ability.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    1) The ablation study on removing the MI loss is missing. Since utilizing the MI maximization learning to facilitate the domain adaptation is a major novelty in this paper, it is important to report the model’s performance without this loss function, to further indicate its effectiveness.

    2) The motivation for enlarging the mutual information for the cross-domain features to facilitate the domain adaptation has been proposed in [16] and [a].

    [a] MI2GAN: Generative Adversarial Network for Medical Image Domain Adaptation using Mutual Information Constrain, in MICCAI 2020

    Although the MI maximization loss function in the proposed MaNi has a different implementation, detailed discussions and comparisons with [16] and [a] are missing.

    3) Visualization results are missing. In addition to the quantitative comparisons, qualitative analysis of the instance segmentation prediction is also an important metric to evaluate the model’s cross-domain segmentation performance.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    All datasets used in the experiments are public. Source code is not available during review.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

    1) Please include a computational complexity analysis of the proposed method.

    2) For the experimental results in Table 2, [27] also reported the performance under the nuclei classification and detection. Please also report the results under these tasks.

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    4

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The overall paper lacks novelty. Important ablation experiments are missing.

  • Number of papers in your stack

    8

  • What is the ranking of this paper in your review stack?

    6

  • Reviewer confidence

    Very confident

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    5

  • [Post rebuttal] Please justify your decision

    Based on the rebuttal, the authors claim that novelty of the proposed method is that it can be adapted into various architectures. I think it indicates the contribution is on the implementation. It can somehow emphasise the overall contribution. However, it is suggested to include more evidence to indicate the proposed method is better than the existing UDA methods based on MI maximisation.

    In addition, for the consep and pannuke datasets, they have category labels for each nucleus. Please include the details on whether the original nuclei categories labels are used for the experiments.

    Overall, I think the authors’ feedback has addressed some of my concerns, and improve the overall rating after the rebuttal. However, some details and changes should be included in the updated manuscript.



Review #3

  • Please describe the contribution of the paper
    • The paper proposes to use a contrastive learning based mutual information maximization approach to maximize the mutual information between source and target domains
    • The proposed approach is empirically validated on using different architectures on various datasets: TNBC -> KIRC/TCIA, TCIA -> KIRC/TNBC, CoNSep -> PanNuke.
  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. The paper is well-written and well-motivated.
    2. The use of Jensen-Shannon divergence (JSD bound) based mutual information estimator in the context of nuclei segmentation is novel. JSD has been used in feature alignment for text and image data. To the best of my knowledge, applying JSD for domain adaptation for nuclei segmentation is novel.
    3. The paper has strong empirical results on different datasets. The paper evaluates on Nuclei Semantic Segmentation (TNBC-> KIRC/ TCIA and TCIA -> KIRC/ TNBC shift) and Nuclei Instance Segmentation (CoNSep -> PanNuke) and demonstrates competitive results over the baseline approaches.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    1. It would be interesting to compare the empirical results of JS with other types of MI estimators, e.g., infoNCE, Donsker-Varadhan representation 1.
    2. The selection of negative pairs in the MI estimator lacks motivation. What are the different ways to select negative pairs and why prefer this particular approach?

    1 Belghazi, Mohamed Ishmael, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R. Devon Hjelm. “Mine: mutual information neural estimation.” arXiv preprint arXiv:1801.04062 (2018).

  • Please rate the clarity and organization of this paper

    Excellent

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    Not sure how easy/difficult it is to reproduce the paper.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

    Please refer to main weaknesses.

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    7

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    I think the novelty in proposing the JSD based adaptation approach and the empirical results outweight the weakness of not evaluating against other types of MI estimators.

  • Number of papers in your stack

    5

  • What is the ranking of this paper in your review stack?

    1

  • Reviewer confidence

    Confident but not absolutely certain

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    5

  • [Post rebuttal] Please justify your decision

    After reading other reviews, I update my rating to “weak accept” due to the lack of comparison with other mutual information based estimators.



Primary Meta-Review

  • Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    This manuscript introduces a mutual information (MI) estimation method for cross-domain nuclei segmentation. The reviewers appreciated the idea of using a Jensen-Shannon divergence (JSD)-based lower bound for MI estimation in domain adaption and the good experimental results on different image datasets. However, the reviewers also raised some questions or concerns. For instance, Reviewer #1 pointed out that the evidence to select JSD as a lower bound for MI estimation is insufficient or unclear, the experimental results are not convincing, some technical details are missing, and many relevant methods are not discussed. Reviewer #2 mentioned that using MI estimation for domain adaptation has been reported in some other literature and the difference between the proposed method and others should be clearly explained, the major contribution of using MI maximization is not well justified by the experiments, and an analysis of computational complexity should be provided in the paper. Reviewer #3 commented that the motivation of selecting negative pairs for MI estimation is missing. Please consider addressing the reviewers’ concerns/comments in the rebuttal.

  • What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    4



Author Feedback

We would like to thank reviewers for their constructive comments. R#1 and R#3 have raised concerns about the motivation for using JSD and its comparison. We refer to Fig. 9 of the Deep InfoMax paper 10 comparing the number of negative samples vs. accuracy and highlight that JSD is insensitive to the number of negative samples. In contrast, other bounds become unstable and decline in performance when the number of negative samples is reduced. In segmentation model, the computational burden of contrasting each data point with high numbers of negatives is extremely high, motivating us to use JSD bound and constraining us from testing other bounds. Further, in the supplementary table of CLIP-lite paper [22], the authors demonstrate that JSD converges even with small batch sizes. However, please note that this does not indicate a reduction in the number of negatives which is still one for each positive in the batch. Also, we agree with R#3 that multiple strategies can be used for determining negative pairs. We used the standard approach of random sampling negative pairs to demonstrate our methodology and plan to study other techniques in future. R#1 and R#2 have raised concerns about a lack of literature comparison to prior approaches. In our work, we specifically proposed a method for UDA nuclei segmentation via MI maximization, and MICCAI has a 2-page reference limit; hence for related works, we focus on recently proposed UDA segmentation methods and relevant MI literature. Further, we agree with R#2 that MI loss for cross-domain feature alignment has already been proposed in multiple works such as PDAM and MI2GAN. However, these methods are architecture and use-specific with a limited scope of adoption. PDAM maximizes MI between the matching object instances by fusing the box and mask branch features, and MI2GAN maximizes MI between the encoded image content features. In contrast, we proposed a generic formulation that can be easily incorporated into any segmentation architecture. We demonstrated its performance for multiple nuclei datasets with different architectures. As suggested by R#2, we will add this comparison in our paper. R#2 has further pointed out the missing nuclei classification task reported in the paper [27]. In our work, we trained models specifically for segmentation tasks (highlighted in Section 4.2 ). We didn’t include the instance classification task as it requires further extensive experimentation for determining the appropriate MI set-up, which was beyond the scope of this work. Also, this highlights that for the nuclei instance segmentation task, our model trained with only segmentation masks is able to perform competitively with prior approaches trained with both segmentation masks and classification labels. The reviewers have raised concerns about the reproducibility of the work. We strongly believe in reproducible research and will open-source the code with our paper. Our PyTorch implementations adopt open-sourced standard HoverNet, Unet, and DeepInfoMax implementation. Also, as suggested by R#1, initially, we were not able to rerun the comparison approaches in our codebase as none of them had their code open-sourced. However, we agree that the model environment can substantially impact the performance. So for TNBC, TCIA, and KIRC experiments, we wrote our best implementation of other approaches and have updated the scores. For reference, dice scores in TNBC to TCIA experiment - source-only: 0.681, DA_ADV: 0.734, CellSegUDA: 0.765, SelfEnsemblingUDA: 0.75, MaNi: 0.776. We have added the “source only” baseline in Table 1 for comparison. Further, as suggested by R#2, we have added computational complexity in Table 1 to highlight the number of parameters and FLOPs in each model. Further, we will incorporate all the edits suggested by R#1, such as highlighting the non-linearity in the discriminator. Also, as suggested by R#2, we will include a qualitative comparison in the supplementary material.



Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    This paper introduces an interesting unsupervised domain adaptation method for nuclei segmentation, which is achieved via mutual information maximization with a Jensen-Shannon divergence (JSD)-based lower bound. The benefit of using JSD is that it is insensitive to the number of negative samples. Although the paper does not provide a comparison with other mutual information based estimators in the experiments, the rebuttal has addressed most of the reviewers’ concerns, and all reviewers recommend (weak) acceptance.

  • After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

    Accept

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    1



Meta-review #2

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    The authors mostly addressed reviewers concerns. However, some remaining concerns such as the motivation for the applicability of the Jensen Shannon Divergence lower bound on the mutual information and its relation to UDA could be better discussed, why the proposed method is better than existing UDA methods using MI maximization should be addressed in the discussion. Otherwise, this is an excellent manuscript with a novel methodology that is potentially applicable beyond the nuclei image segmentation used in this work.

  • After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

    Accept

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    4



Meta-review #3

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    While the authors have failed to address all the comments raised during the review process, the reviewers consider that the strengths of the proposed work outweigh its weaknesses. I side with the reviewers, and recommend acceptance of this work despite several points that could have been improved. For example, the motivation of using the JSD is not very strong and not supported by the experiments. Note that existing literature points to InfoNCE outperforming JSD, so authors should have done a better job motivating the choice of the JSD, as well as validating this choice empirically.

  • After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

    Accept

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    4



back to top