List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Yuncheng Yang, Meng Wei, Junjun He, Jie Yang, Jin Ye, Yun Gu
Abstract
Transfer learning is a critical technique in training deep neural networks for the challenging medical image segmentation task that requires enormous resources. With the abundance of medical image data, many research institutions release models trained on various datasets that can form a huge pool of candidate source models to choose from. Hence, it’s vital to estimate the source models’ transferability (i.e., the ability to generalize across different downstream tasks) for proper and efficient model reuse. To make up for its deficiency when applying transfer learning to medical image segmentation, in this paper, we therefore propose a new \textbf{Transferability Estimation} (TE) method. We first analyze the drawbacks of using the existing TE algorithms for medical image segmentation and then design a source-free TE framework that considers both class consistency and feature variety for better estimation. Extensive experiments show that our method surpasses all current algorithms for transferability estimation in medical image segmentation. Code is available at https://github.com/EndoluminalSurgicalVision-IMR/CCFV.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43907-0_64
SharedIt: https://rdcu.be/dnwdL
Link to the code repository
https://github.com/EndoluminalSurgicalVision-IMR/CCFV
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The contribution of the paper is to propose a transferability estimation (TE) measure that can be used to compare pre-trained models.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- proposing a new transferability estimation measure.
- comparing with existing measures.
- visualizations are provided.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The descriptions in captions and labels of figures and tables are missing some details. Reader has to go through the paper and the supplementary file to understand these details.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
All details are provided.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
-
Please consider adding all details to the captions and labels of figures and tables. For example, task 03, etc. in Table 1 and Table 2. You can add a footnote.
-
For Fig. 2, define the title (which task), define the legend, etc. You may summarize the four plots into one plot given that the vertical axis does not change for all of them.
-
define es in EQ (3)
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The paper is well written and the results are comprehensive.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The paper novelly proposes a transferrability estimation (TE) method for medical image segmentation from two aspects: class consistency and feature variety. Experiments on a Medical Segmentation Decathon dataset validates the effectiveness of the proposed TE method.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1) Research on TE is valuable and necessary. The TE methods for medical image segmentation is rarely studied. 2) Experiments results on the Medical Segmentation Decathon dataset is effective and convincing.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) After reading Section 2 and ablation studies, I am still not sure which components of the proposed TE contribute most, especially for medical image segmentation tasks. 2) Running time performance of the proposed method is not shown in the experiments.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper propovides sufficient implementation details.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
1) Refind contributions of the proposed TE method for medicial image segmentation. 2) Show run time performance of the proposed method.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Research on TE methods for medical image segmentation is rarely studied, and the experiments results on the Medical Segmentation Decathon dataset is effective and convincing.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper proposes a new Transferability Estimation (TE) method to improve the efficiency and effectiveness of transfer learning in medical image segmentation. The authors analyze the limitations of existing TE algorithms and design a source-free framework that considers class consistency and feature variety for better estimation of transferability. The proposed method is evaluated through extensive experiments and is shown to outperform all current algorithms for transferability estimation in medical image segmentation. The contribution of this paper lies in its development of a new method to estimate the transferability of models for medical image segmentation, which can lead to more efficient and effective use of existing models and resources.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The proposed method introduces a novel approach to estimate the transferability score of pre-trained models, utilizing class consistency and feature variety constraints to rank pre-trained models based on their performance on the target dataset.
- The proposed method is evaluated on clinically relevant 3D medical imaging datasets for segmentation and outperforms in terms of transferability score estimation.
- The authors conduct extensive experiments on the widely used Medical Segmentation Decathlon dataset, demonstrating the applicability of the proposed method to multiple modalities of medical imaging datasets. They provide thorough evaluation and quantitative analysis to demonstrate the clinical feasibility of the proposed method.
- Multiple evaluation metrics, including the dice score coefficient, Kendall’s tau, and Pearson correlation coefficients, are utilized to thoroughly evaluate the performance of the proposed method. The authors also conduct an ablation study to analyze the impact of different parts of the proposed method and validate the effectiveness of the multi-scale strategy.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The lack of a comprehensive comparison with the state-of-the-art transfer learning methods for medical image segmentation is a significant limitation that hinders a complete understanding of the proposed method’s effectiveness.
- The absence of a comparison with state-of-the-art methods specifically designed for class imbalance problems, which may not accurately reflect the current state-of-the-art in addressing class imbalance problems in medical image segmentation.
- Information on the computational requirements of the proposed method, such as training time, memory usage, and inference time, is not provided, limiting the practical guidance for researchers and practitioners interested in applying the method.
- The limited evaluation of the proposed method on only 5 of the 10 datasets in the Medical Segmentation Decathlon dataset may limit the generalizability of the results to other datasets or imaging modalities.
- A qualitative analysis of the segmentation results is not included, which could provide additional insights into the clinical usefulness of the proposed method.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper has provided sufficient information and resources to allow for reproducibility of the proposed method.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- A more detailed comparison with the latest and most advanced methods in the field would strengthen the paper.
- Including additional datasets would help evaluate the generalizability of the proposed method.
- It would be beneficial to compare the proposed method with the state-of-the-art methods addressing class imbalance problems.
- Adding a discussion of the clinical implications of the results would improve the paper.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The authors propose a novel approach for estimating transferability scores of pre-trained models in medical image segmentation by introducing class consistency and feature variety constraints. The proposed method is evaluated thoroughly and outperforms existing state-of-the-art methods on clinically relevant 3D medical imaging datasets. However, some limitations need to be addressed, including the lack of a comprehensive comparison with state-of-the-art methods for medical image segmentation and class imbalance problems, limited evaluation on only a subset of the Medical Segmentation Decathlon datasets, and the absence of qualitative analysis of segmentation results. Overall, this paper constitutes a significant contribution to the field and could be accepted with minor revisions. The authors are advised to address the aforementioned limitations and improve the readability of some sections of the paper.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #4
- Please describe the contribution of the paper
This paper introduced a method for transferability estimation problem. In order to evaluate the ability of transfer learning of a pre-trained model without fine-tuning.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper proposed an interesting problem named transferability estimation. It is a realistic problem while processing the target data. To do this, several metrics are proposed for this problem.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
It should be a problem, but in fact, it is a huge task. Firstly, does such a task a necessary problem for medical image? Transfer learning is a method for deep network training, to evaluate the performance of transfer learning, plenty of methods are available. Why not directly fine-tuning on the target dataset? Why not directly select the models trained on similar source datasets? Is it true that if the TE score is high, the harm data samples are less? Secondly, the evaluation is not believable. The selected segmentation datasets are small, but the problem is huge, especially on medical datasets. Thirdly, the visualization is not evident but for easy understanding. Inverse, the proposed metrics is easy to understand but may not be the key to this problem.
Overall, this paper is interesting, but it would be far from this conference.
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The proposed metrics is simple and can be reproduced.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The proposed metric is good and shows improvement over existing methods. But currently the problem is not to overfit the dataset. As a theory paper, the analysis is poor. As an engineering work, the experiment is poor.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The writing is hard to understand, the result is not related.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Overall, the paper proposes a novel transferability estimation (TE) method for medical image segmentation that is well-organized and thoroughly evaluated using various metrics. The strengths of the paper include comparing the proposed method with existing measures and providing visualizations. However, weaknesses include missing details in the descriptions of figures and tables, and limited evaluation on only a subset of the Medical Segmentation Decathlon datasets. Reviewers 1 and 3 recommend acceptance with minor revisions, while Reviewer 2 suggests refining the contributions and showing the run time performance of the proposed method. Reviewer 4 suggests rejecting the paper due to major weaknesses, such as the clarity and organization of the paper and believability of the evaluation and visualization.
Author Feedback
N/A