List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Jiajin Zhang, Hanqing Chao, Amit Dhurandhar, Pin-Yu Chen, Ali Tajer, Yangyang Xu, Pingkun Yan
Abstract
Domain shift is a common problem in clinical applications, where the training images (source domain) and the test images (target domain) are under different distributions. Unsupervised Domain Adaptation (UDA) techniques have been proposed to adapt models trained in the source domain to the target domain. However, those methods require a large number of images from the target domain for model training. In this paper, we propose a novel method for Few-Shot Unsupervised Domain Adaptation (FSUDA), where only a limited number of unlabeled target domain samples are available for training. To accomplish this challenging task, first, a spectral sensitivity map is introduced to characterize the generalization weaknesses of models in the frequency domain. We then developed a Sensitivity-guided Spectral Adversarial MixUp (SAMix) method to generate target-style images to effectively suppresses the model sensitivity, which leads to improved model generalizability in the target domain. We demonstrated the proposed method and rigorously evaluated its performance on multiple tasks using several public datasets.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43907-0_69
SharedIt: https://rdcu.be/dnwdQ
Link to the code repository
https://github.com/RPIDIAL/SAMix
Link to the dataset(s)
https://wilds.stanford.edu/datasets/#camelyon17
https://refuge.grand-challenge.org/Home2020/
Reviews
Review #2
- Please describe the contribution of the paper
The paper proposes a plug-in module to enhance the performance of existing unsupervised domain adaptation methods in the few-shot unsupervised domain adaptation scenario. The proposed method relies on a spectral sensitivity map and an adversarial mixing scheme to guide the generation of hard-to-learn target-style images from each source image. The proposed approach is evaluated on two publicly available datasets and outperforms previous methods on both classification and segmentation tasks.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is well organised and easy-to-follow and the proposed method is well motivated.
-
The results are very promising - The proposed method has been evaluated on two publicly available datasets, consistently enhancing the performance of existing unsupervised domain adaptation in the few-shot unsupervised domain adaptation scenario and outperforming previous few-shot unsupervised domain adaptation methods.
-
The paper includes detailed ablation studies that analyse the impact of various values of critical parameters on the performance.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The paper does not provide details about the experimental settings used to train existing methods included in the comparison, which raises concerns about the fairness of the comparisons.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The method is clearly described, and the authors intend to make their code publicly available.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
See weaknesses.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper proposes a very interesting approach for few-shot unsupervised domain adaptation, provides sufficient experiments to demonstrate the effectiveness of the proposed method on both segmentation and classification, and outperforms existing studies by a large margin.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #1
- Please describe the contribution of the paper
The paper proposes a few-shot unsupervised domain adaptation framework. The framework utilizes Fourier Transform, MixUp, and adversarial training to augment the data. The proposed method can be used a plug-in module to improve the performance of existing methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
The technical writing is good.
-
The experimental results are promising.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
The technical contributions are incremental as all components are based on the existing studies. However, this doesn’t mean this is not a good work. Efficiently integrating the existing studies into a successful work is also a valuable achievement.
-
Missing related works. For example, [a] utilizes human structure similarity in adversarial UDA and works for low-data regime; [b] is a UDA work on fundus datasets used in this work.
[a] Unsupervised domain adaptation for automatic estimation of cardiothoracic ratio, Dong et al., MICCAI 2018. [b] WGAN domain adaptation for the joint optic disc-and-cup segmentation in fundus images, Wang et al., International Journal of Computer Assisted Radiology and Surgery, 2020.
-
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
I think this paper can be reproduced with enough efforts.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
I have no other comments on this work.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Technical novelty
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The authors propose a few-shot unsupervised domain adaptation (UDA) method that extends spectral sensitivity-based single domain generalization to source-to-target domain adaptation in the few-shot scenario i.e., using limited target samples. To mitigate domain shift and avoid overfitting on limited samples, a domain specific distance map (module) is first estimated from source-target data in the frequency space, and later used for adversarial spectral mix-up (module) to augment target data with hard target-style samples. These former modules enable improved generalization to target data and improve sensitivity of existing UDA methods. Extensive evaluation on benchmark datasets in the 1-10 shot scenario reveals impressive results across all settings.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
The paper is very clear, easy to follow and addresses a relevant problem in medical image analysis i.e., few-shot learning with limited target domain samples.
- While spectral sensitivity and adversarial attacks in natural images via Fourier domain analyses is fairly well studied, its applicability for this task is both novel and interesting.
- Data augmentation is a common strategy to improve model robustness, though not uniform across corruption types – the proposed spectral mix-up is a clever solution to ensure target-style consistency.
- The augmented images qualitatively appear consistent with the target domain.
- Extensive experiments on benchmark datasets with ablations sufficiently that support the proposed ideas. Also, this work has consistent and impressive results in each of the evaluated few-shot scenarios.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
Unclear how sensitive the design is to the selected transformations in domain distance measurement. The authors mention employing only geometric transformations for this step (DD measurement) as this does not change intensity for retinal fundus images. This is reasonable; however, it is unclear how severe intensity-based transforms would affect performance. Did the authors evaluate the scenario??
-
More on transforms – For Camelyon dataset, most tissue patches from different sites will exhibit different staining due to the acquisition process; thus, for this task I assume a combination of geometric and intensity-based transformation would work better. Clarifying this important, as the reviewer assumes transforms may be dataset specific- suggesting the requirement of domain knowledge.
-
Unclear how many augmentations are employed in SAMix step. Is the fixed, and how sensitive is the model to the number of augmentations?
-
The domain shift in retinal images does not appear to be very challenging. For instance, SM-PPM reports fairly consistent scores in the varying-shot scenario (Fig.3 (a)), this begs the question of whether the model is truly learning generalizable representations. To assess the impact of this approach, evaluation of more datasets with severe domain shifts would be helpful.
-
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors included all relevant details regarding reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Minor suggestions:
Abstract : “suppresses –> suppress”
(Sec. 3.2, Fundus) “(adverage DSC) –> (average DSC)”
As I cannot list all the minor grammatical errors, I recommend the authors to carefully check the manuscript for spelling errors.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I found this work easy to read, with clear motivations for a challenging task in medical image analysis. Overall, this is a well-structured paper with sufficient empirical evidence that supports the proposed ideas well. I appreciate the extensive ablations and insights. Aside from the minor weaknesses, and additional clarifications (see. weaknesses above), I feel this work will appeal to a larger part of the MICCAI audience interested in few-shot learning and UDA.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper investigates few-shot unsupervised domain adaptation. The framework integrates Fourier Transform, MixUp, and adversarial training to augment the data. The method could be used as a plug-in module to improve the performance of existing methods.
All reviewers found the paper easy to read, with clear motivations and sufficient empirical evidence (as well as promising performances and intensive ablations). I concur with this and agree that this work could appeal to a large MICCAI audience interested in few-shot learning and unsupervised domain adaptation.
Author Feedback
We are grateful to the reviewers and AC for acknowledging our contributions and considering our work as “easy to read, with clear motivations and sufficient empirical evidence as well as promising performances and intensive ablations.” The reviewers’ questions mainly regard related works, detailed experimental settings and clarification of some technical details. This response tries to address these excellent questions.
R1: Missing related works.
We would like to thank R1 for pointing out the references. Both papers are on UDA-based segmentation methods. We will include them together with a brief discussion in our camera-ready version.
R2: Details about the experimental settings.
In our experiments, we followed the original settings (model architecture, learning rate and learning rate schedule) of the existing methods. Due to the page limit, we didn’t provide all these details in the paper. We will clarify this point in the section of Implementation Details for interested readers to refer to the details. We will also share the source code of our work for reproducibility. The link will be included in the camera-ready version.
R3:
- Q1&Q2: Impact of intensity-based transformation on performance.
Thanks for the questions! Intensity-based transformations are excluded when measuring the DoDiSS map for the following reason. The proposed domain difference measurements aim to quantify the image style difference across different domains to guide the following SAMix algorithm for improved domain adaptation performance. As R3 mentioned, intensity-based transformations may change the image color and brightness, which are the factors closely related to the image style. Thus, applying intensity-based transformations when measuring the DoDiSS map will lead to the distorted measurement on the real domain difference, which needs to be avoided.
- Q3: Unclear how many augmentations are employed in SAMix step.
The four geometric augmentations are only used when computing DoDiSS, but not the following SAMix step. In our experiments, SAMix only uses the proposed spectral augmentation, i.e., mixing the source and target domain image amplitude spectra for intensity transformation.
- Q4: The domain shift in retinal images does not appear to be very challenging.
In our experiments, we employed two datasets to demonstrate the reliable performance of our proposed method on different tasks and medical imaging modalities. More specifically, we chose the Fundus dataset to evaluate the method classification performance on the optical data, and used Camelyon dataset to quantify the method segmentation performance on the microscopic data. The Fundus dataset is from the MICCAI2020 REFUGE2 challenge, which is a widely used segmentation task in medical cross-site domain adaptation problems. The Camelyon dataset is from the WILDS benchmark dataset, which is also commonly used for evaluating domain adaptation and domain generalization methods for medical image classification tasks. In our future work, we will evaluate the broaden impact of our method by extending to more challenging medical image datasets.