Authors

Chen Yang, Yifan Liu, Yixuan Yuan

Abstract

Unsupervised domain adaptation has drawn sustained attentions in medical image segmentation by transferring knowledge from labeled source data to unlabeled target domain. However, most existing approaches assume the source data are collected from a single client, which cannot be successfully applied to explore complementary transferable knowledge from multiple source domains with large distribution discrepancy. Moreover, they require access to source data during training, which is inefficient and unpractical due to privacy preservation and memory storage. To address these challenges, we study a novel and practical problem, named multi-source model adaptation (MSMA), which aims to transfer multiple source models to the unlabeled target domain without any source data. Since no target label and source data is provided to evaluate the transferability of each source model or domain gap between the source and the target domain, we may encounter negative transfer by those less related source domains, thus hurting target performance. To solve this problem, we propose a transferability-guided model adaptation (TGMA) framework to eliminate negative transfer. Specifically, 1) A label-free transferability metric (LFTM) is designed to evaluate transferability of source models without target annotations for the first time. 2) Based on the designed metric, we compute instance-level transferability matrix (ITM) for target pseudo label correction and domain-level transferability matrix (DTM) to achieve model selection for better target model initialization. Extensive experiments on multi-site prostate segmentation dataset demonstrate the superiority of our framework.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43895-0_66

SharedIt: https://rdcu.be/dnwzy

Link to the code repository

https://github.com/CityU-AIM-Group/TGMA

Link to the dataset(s)

N/A

Reviews

Review #2

Please describe the contribution of the paper

The paper proposes a novel framework for medical image segmentation that can transfer knowledge from multiple pre-trained source models to an unlabeled target domain without accessing the source data. The paper introduces a label-free transferability metric to evaluate the relevance of each source model to the target domain based on attentive masking consistency. The paper also uses this metric to guide the pseudo-label correction and model selection for better adaptation. The paper claims that the proposed framework can eliminate negative transfer from less related source domains and achieve superior performance on a multi-site prostate segmentation dataset.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. This paper addresses a challenging and practical problem of domain adaptation for medical image segmentation without accessing the source data, which can benefit many real-world applications where data privacy and storage are important issues.
2. This paper proposes a novel and effective framework that can leverage multiple pre-trained source models and adapt them to an unlabeled target domain using a label-free transferability metric based on attentive masking consistency, which measures the relevance of each source model to the target domain in a self-supervised manner.
3. This paper demonstrates the superiority of the proposed framework over existing methods on a multi-site prostate segmentation dataset, and provides extensive ablation studies and analysis to validate the effectiveness of each component of the framework. This paper also shows that the proposed framework can eliminate negative transfer from less related source domains and improve the robustness and generalization of the segmentation model.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Some weaknesses of this paper are:
1. The paper uses a limited number of datasets for the experiments, which may not be sufficient to demonstrate the generality of the algorithm for medical segmentation problems.
2. Although the paper provides a detailed description of the algorithm design, it lacks a clear explanation of the motivation and insight behind the design choices.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper’s algorithm implementation is clear and well-documented, making it relatively easy for others to reproduce the results.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
1. What’s the difference between the source free domain adaptation with multi-sources and the multi-source model adaptation in this paper? It seems the same.
2. The experimental setting is similar with the leave-one-domain-out setting in domain generalization. Similar descriptions can be used to make the table information easier to read
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. Novelty
2. Well-structured and well-written
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #1

Please describe the contribution of the paper

This paper proposes a multi-source model adaptation model to solve problems of medical image segmentation. The research problem studied is how to complementarily explore transferable knowledge from multiple clients to the unlabeled target domain. Because of storage and privacy concerns, the proposed method uses only pre-trained source models without accessing source data. In order to eliminate negative transfer, the authors design metrics to evaluate the importance of different source domains.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. ITM and DTM are proposed to solve the multi-source model adaptation problem.
2. The experimental results seem good on medical segmentation problems.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The performance highly depends on the pre-trained source model.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The reproducibility seems good according to the reproducibility response.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
1. The results may be influenced by the quality of pre-trained source models, which are used to generate pseudo labels without fine-tuning. It would be better to explore the effects of the pre-trained model on the source-free domain adaptation problem.
2. The design of transferability metrics seems good based on the assumptions in Section 2.1.
3. DTM is the average of ITM, which ignores the different contributions or effects of the instances. It could be a better choice to adopt a weighting approach.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

A reasonable method is proposed to address the problem of multi-source domain adaptation without accessing source data.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

The authors propose a new problem, multi-source model adaptation (MSMA), which aims to transfer multiple source models to an unlabeled target domain without using any source data. They introduce a transferability-guided model adaptation (TGMA) framework to eliminate negative transfer and improve target performance. They design a label-free transferability metric (LFTM) to evaluate the transferability of source models and compute instance-level transferability matrix (ITM) and domain-level transferability matrix (DTM) for better target model initialization. Experiments on a multi-site prostate segmentation dataset show the effectiveness of the proposed framework.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The experimental results are promising, as the proposed method outperforms existing methods by 3.5% on the dice metric.
2. This paper introduces the consideration of the relevance between source and target domains, and uses it to improve performance, which is an interesting and reasonable approach.
3. The proposed schemes align well with the paper’s motivation.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Although this paper is well-performed and self-contained, there are still some minor errors that need to be addressed. Firstly, in the Transferability-guided Model Selection, the initialization weights are missing, which strongly influences the initial performance of the target model. Additionally, it seems unnecessary to distinguish between the main network and auxiliary network since no larger weight is assigned to the main network. Secondly, there are some expression errors in the paper, such as using ‘where’ after Eq. 1 and presenting a strange expression for the segmentation map.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

I am convinced of the reproducibility of the paper. However, due to the complexity of the framework, I find it difficult to understand some details. Therefore, I suggest that the authors publish their code to improve reproducibility and facilitate understanding of the methodology.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

As shown in weakness
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The motivation, methods, and experimental results are reasonable and interesting, but exist minor error
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposes a multi-source model adaptation model to solve problems of medical image segmentation, which is a challenging and practical problem of domain adaptation. In addition, the storage and privacy concerns are alleviated by using only pre-trained source models without accessing source data. To eliminate negative transfer, the authors design metrics to evaluate the importance of different source domains.

The paper is well orgnised and easy to follow. The novelty and performance are well recognised by the reviewers. There are several single source-free/black-box UDA for medical image segmentation in recent MICCAI/ISBI/SPIE MI etc., the authors are encouraged to discuss clarify the difference in this paper.

Author Feedback

We sincerely thank all reviewers and meta-reviewer for their valuable comments and positive feedback regarding the idea. Here are responses to their invaluable suggestions and remaining concerns. Review #1: Q1: Influence of the quality of pre-trained source models. A: We will try different backbones of pre-trained source model to explore the influence in the futher work. Review #2: Q1: Limited datasets. A: We will try more datasets in the futher work. Q2: Lack of a clear explanation of the motivation. A: In this paper, we aims to solve the negative transfer problem in multi-source model adaptation. Since no target label and source data is provided in this setting, we design a label-free transferability metric (LFTM) to evaluate transferability of source models. This metric is designed based on two assumptions: 1) Sample relevance: similar samples should hold identical predictions; 2) Model stability: if a source model makes accurate decision on this sample, little permutation on irrelevant regions will not influence the prediction.Therefore, we construct augmented sample by attentive masking, and compute the consistency as the transferability. Q3: Difference between the source free domain adaptation with multi-sources and the multi-source model adaptation in this paper? A: They represent the same setting; we abbreviate source free domain adaptation with multi-sources as multi-source model adaptation for convenience. Review #3: Q1: Initialization weights for model selection. A: Weights are initialized as the top two values of Tdomain. Q2: Expression errors. A: We will revise these errors in the final version. Q3: Reproducibility. A: We will release the code soon.

back to top

Transferability-Guided Multi-Source Model Adaptation for Medical Image Segmentation