List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Ning Dai, Lai Jiang, Yibing Fu, Sai Pan, Mai Xu, Xin Deng, Pu Chen, Xiangmei Chen
Abstract
The joint use of multiple imaging modalities for medical image has been widely studied in recent years. The fusion of information from different modalities has demonstrated the performance improvement for some medical tasks. For nephropathy diagnosis, immunofluorescence (IF) is one of the most widely-used medical image due to its ease of acquisition with low cost, which is also an advanced multi-modality technique. However, the existing methods mainly integrate information from diverse sources by averaging or combining them, failing to exploit multi-modality knowledge in details. In this paper, we observe that the 7 modalities of IF images have different impact on different nephropathy categories. Accordingly, we propose a knowledge distillation framework to transfer knowledge from the trained single-modality teacher networks to a multi-modality student network. On top of this, given a input IF sequence, a recruitment module is developed to dynamically assign weights to teacher models and optimize the performance of student model. By applying on several different architectures, the extensive experimental results verify the effectiveness of our method for nephropathy diagnosis.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43904-9_51
SharedIt: https://rdcu.be/dnwHw
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This work presents a novel framework that integrates multi-modal IF images for nephropathy diagnosis. Based on the observation that the single-modality IF image achieves better diagnosis than multi-modality IF images, a customized knowledge distillation framework is proposed to transfer the knowledge from individual modalities to the multi-modal network. Experimental results on the internal and external datasets indicate that knowledge distillation greatly improves the performance of the multi-modal network.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This work is well-motivated by the realistic challenges of the multi-modal nephropathy diagnosis.
- The authors have collected a dataset with multi-modal IF images and promised to release the source data after the reviewing process. The released data may promote the development of the medical AI community.
- Except for the randomly split training and testing sets, the authors collected an external test set, which is very crucial to validate the effectiveness and generalizability of the proposed method.
- It is rational to combine the medical prior weights and learnable weights to customize the contribution of different single-modality teachers.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Fig. 3 contains too many details and is not clear enough. I would suggest the authors omit some details in this figure and add more descriptions in the caption.
- The notation of the modality importance weight is not consistent in this paper. In Eq. (6), it is denoted as w_m. However, in the text above Eq. (6), the weight is denoted as w_t. The authors should check through this paper and unify the notations.
- This paper lacks necessary experiments. Table 2 compares the proposed method with baseline models (e.g., ResNet, DenseNet) and two nephropathy diagnosis methods. I think the authors should compare with more advanced knowledge distillation methods to validate the effectiveness of the proposed customized distillation method.
- The main contribution of the distillation method in this paper is the customized weights of different single-modality teachers. To validate the effectiveness of this weighting strategy, the authors should compare it with equal distillation from all teachers. What’s more, the modality weights are decided by the combination of the medical prior weights and learnable weights. An ablation study should be provided to validate the contribution of these two parts.
- There are some grammatical mistakes in this paper. The authors are recommended to check the grammar thoroughly.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors have promised to release the source codes and data after the reviewing process, so I think this work should be reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Please refer to the major weakness.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This work is well-motivated and has some novelty in the methodology part. However, it lacks some necessary experiments and ablation studies to validate the effectiveness of the proposed method.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This paper proposes a customized knowledge distillation framework for nephropathy diagnosis, which transfers knowledge from the trained single-modality teacher networks to a multi-modality student network. In the proposed framework, a recruitment module is developed to dynamically assign weights to teacher models by taking the different impacts of 7 modalities of immunofluorescence (IF) images on different nephropathy categories into consideration. Extensive experimental results on two datasets which were established by the authors verify the effectiveness of the proposed framework for nephropathy diagnosis.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The authors demonstrate interesting findings about the relationship between different IF modalities and nephropathy.
- The authors established a large-scale PLAG dataset with 1,582 IF sequences and 6,381 images. Besides, they further proposed an external CQDP dataset collected from another hospital, including 109 IF sequences with 348 images.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
The technical novelty of the proposed framework seems a little limited. To select the “best” teacher networks during the knowledge distillation process, the authors simply assign different weights to different teacher networks according to prior knowledge and learnable weights. To achieve the same goal, an existing method[1] utilized reinforcement learning to design more complex weight allocation strategies. Compared with existing methods, the selection strategy of the proposed framework seems not novel. [1] Yuan F, Shou L, Pei J, et al. Reinforced multi-teacher selection for knowledge distillation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 35(16): 14284-14291.
-
The evaluation of experimental results is not convincing enough. In addition to several DNN backbones, the authors only compared the proposed framework with two nephropathy diagnosis methods proposed in 2020 and 2022, respectively. More SOTA methods on nephropathy diagnosis should be compared to demonstrate the superiority of the proposed framework.
-
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors provided a clear description of the proposed method and used a publicly available dataset. The authors will release data after the reviewing process.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
-
In the proposed framework, the teacher networks take single-modality images as inputs and the student network takes multi-modality images as inputs. However, the authors mentioned that for certain combinations, the single-modality IF image achieves better diagnosis accuracy than multi-modality IF sequences over DNN models. Thus, why not involve a multi-modality teacher network and a single-modality student network for nephropathy diagnosis? The motivation for designing a student network with multi-modalities to diagnose nephropathy should be further clarified in the introduction section.
-
What is the input of the student network? Since the collected modalities of an IF sequence are usually incomplete, and the missing modalities are different for different subjects, how did the authors deal with the dynamic input sequences?
-
Comparison with more SOTA methods that focus on nephropathy diagnosis is strongly suggested.
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The limited technical novelty and unconvincing evaluation are my major concerns.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The authors collected and constructed two immunofluorescence datasets from two hospitals and proposed a teacher-student network for the classification of chronic kidney disease. The structure of the article is complete.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Authors construct brand new in-house dataset to explore AI-based immunofluorescence for kidney disease diagnosis For the first time, the author explores the intelligent diagnosis of kidney disease by using the teacher-student network combined with various immunofluorescence data
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
the authors’ method only applied the teacher-student network to their own dataset, and the innovation is somewhat lacking. No comparison with other multimodal fusion methods
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors describe their experimental setup, and even though the code is not disclosed, I think their lab is reproducible
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- What multimodal method was used in 2.2? This was not elaborated in the paper.
- Please provide the accuracy of each class using the proposed method.
- In the framework proposed by the authors, the teacher network needs to be trained on a single modality. However, the dataset provided by the authors has extremely unbalanced data, and some classes have no data. How did the authors address this issue
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The article is logically written and well illustrated. Creatively use the teacher-student network to learn the characteristics of multimodal immunofluorescence to diagnose kidney disease, but the performance of the model has not been compared with other multimodal methods, and the effectiveness cannot be verified. At the same time, the network structure is proposed by previous work
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #4
- Please describe the contribution of the paper
The paper proposes a customized knowledge distillation method for IF-based nephropathy diagnosis. The method transfers knowledge from single-modality teacher networks to a multi-modality student network, utilizing a recruitment module to dynamically select relevant teacher models. Experimental results show the effectiveness of the proposed approach compared to state-of-the-art methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The main strengths of the paper are as follows:
-
Novel Customized Knowledge Distillation: The paper introduces a customized knowledge distillation framework specifically tailored for IF-based nephropathy diagnosis. This approach transfers knowledge from single-modality teacher networks to a multi-modality student network, leveraging the distinct impacts of different IF modalities on nephropathy categories. This novel formulation allows for more detailed exploitation of multi-modality knowledge, improving the diagnosis performance.
-
Adaptive Teacher Selection: The inclusion of a recruitment module to dynamically assign weights to the teacher models based on the medical priors is a notable strength. This adaptive selection process ensures that the “best” teacher networks are chosen for knowledge transfer, enhancing the student network’s performance. This original way of utilizing medical priors adds value to the overall methodology.
-
Extensive Experimental Evaluation: The paper provides a comprehensive experimental evaluation of the proposed method on a large-scale IF dataset. Multiple backbone models are utilized, and the results demonstrate the superior performance of the proposed approach compared to several state-of-the-art methods. The evaluation metrics used (accuracy, kappa, F1-score) effectively assess the diagnosis performance and highlight the method’s effectiveness.
-
Clinical Feasibility: The paper addresses the important clinical application of nephropathy diagnosis, a progressive and incurable disease. By leveraging IF images, which are widely used in the diagnostic process, the proposed method has direct relevance and potential clinical applicability. The evaluation on external datasets further strengthens the clinical feasibility of the approach.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
Limited Comparison with Existing Knowledge Distillation Methods: While the proposed customized knowledge distillation method is novel, the paper does not extensively compare it with other knowledge distillation techniques applied to similar medical image analysis problems. A comparative analysis would strengthen the discussion of the method’s advantages and showcase its superiority.
-
Lack of Discussion on Computational Complexity: The paper does not provide a detailed analysis of the computational complexity or efficiency of the proposed method. Considering the resource limitations in clinical settings, it is important to address the computational requirements and potential scalability issues for real-world deployment.
-
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Good
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
-
Comparison with Existing Knowledge Distillation Methods: While the proposed customized knowledge distillation method is innovative, it would greatly benefit from a more comprehensive comparison with existing knowledge distillation techniques applied to similar medical image analysis problems. Discuss the advantages and limitations of the proposed method in comparison to other relevant approaches, highlighting its superiority and addressing potential drawbacks.
-
Computational Complexity and Efficiency: Provide a detailed analysis of the computational complexity and efficiency of the proposed method. Discuss any potential scalability issues and propose strategies for mitigating them, considering the resource limitations typically encountered in clinical settings. This information is crucial for assessing the method’s feasibility and practicality for real-world deployment.
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
-
Novelty: The paper introduces a novel customized knowledge distillation method for IF-based nephropathy diagnosis. The approach of transferring knowledge from single-modality teacher networks to a multi-modality student network is innovative and addresses the limitations of existing methods that average or combine information from different modalities.
-
Experimental Evaluation: The paper provides extensive experimental results, demonstrating the effectiveness of the proposed method. The comparisons with state-of-the-art models and the improvements achieved in accuracy, kappa, and F1-score metrics validate the superiority of the proposed approach.
-
Potential Impact: The proposed method has the potential to have a positive impact on IF-based nephropathy diagnosis. By improving the accuracy and reliability of diagnosis, the method can aid in early detection and treatment, ultimately benefiting patients and healthcare providers.
-
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper introduces a customized knowledge distillation framework for nephropathy diagnosis. The reviewers acknowledged the well-defined problem setting, thoughtful model design, and incorporation of prior clinical knowledge. However, they raised concerns regarding the level of innovation, the rigor of evaluation and baseline method selection, as well as the simplicity of the weights-based design. For detailed feedback, please refer to the reviewer comments. It is crucial for the author to address these major concerns in order to further enhance the paper’s contribution and address the reviewers’ feedback satisfactorily.
Author Feedback
We sincerely thank the reviewers and AC for the valuable comments.
R1 Notations in Fig.3./Eq.6 and other typos. In revision, we omit unnecessary notations in Fig.3, and fix the notation in Eq.6 as w_t. Besides, we carefully proofread the paper and correct the other typos. Comparison with knowledge distillation (KD) methods. As suggested, we further implement 2 advanced KD methods, MCL [Yang2022AAAI] and ITRD [Miles2022BMVC] in our nephropathy diagnosis framework. Compared with our method, MCL/ITRD degrades 0.05/0.08, 0.06/0.09, and 0.10/0.13, in terms of accuracy, kappa and F1-score, over PLAG test set. We add the above experiments in revision. Ablation study. Actually, the mentioned ablation experiments are reported in Fig.2 of the supplementary. Specifically, the accuracy degrades 0.03, 0.02, and 0.03, when ablating medical prior, learnable weights, and recruitment module (equal distillation). Other ablations, such as the number of teacher networks and distillation loss, are also analyzed in supplementary. In revision, we move the key ablation experiments to main text.
R2 Novelty. Considering the small data scale and clinical feasibility, this paper proposes a simple but efficient KD pipeline for nephropathy diagnosis, in which we set up the first attempt to utilize the single modality to boost the diagnosis on IF images with incomplete modalities. Besides, we further compare with 2 complex KD methods, MCL and ITRD in our framework, with accuracy degradation of 0.05 and 0.08. However, it is an interesting future work to develop more complex KD methods like [1]. Comparison with SOTA. As suggested, we further compare with 2 SOTA nephropathy diagnosis methods, [Hao2023LIFE] and [Wang2023EAAI]. Compared with our method, these 2 methods degarde 0.05/0.08, 0.06/0.09, and 0.10/0.13, in terms of accuracy, kappa and F1-score. The experiments are added in revision. Network input. As in Finding 2, single-modality performs better only under certain combinations of modality and nephropathy. Since GT nephropathy is inacessiable in inference stage, the student network cannot select the best modality for diagnosing if implemented in single-modality manner. Actually, the averaged accuracy over 7 single-modalities is lower than that of multi-modality. Taking MN in Finding 2 as an example, the accuracy of multi-modality and averaged single-modality are 0.81 and 0.53. Thus, we alternatively utilize the medical priors of single-modality during training. Besides, we follow traditional works to compensate the missing input modalities of the student network by zero-valued maps. In revision, we clarify the network input in introduction.
R3 Multimodal method. For fair comparison, we implement multimodal method as the same structure of single-modality but with different inputs. We elaborate it in revision. Single class accuracy. In revision, we further report the accuracy of each single class for all methods. For instance, the MN, IgAN, LN, DN, ANCA, MPGN and AL accuracy of our method are 0.86, 0.96, 0.88, 0.97, 0.60, 0.45 and 0.58. Unbalanced data. For pre-training teacher networks, we conduct data argumentation for the unbalanced classes, and manully set the output logit as 0 for the class with no data.
R4 Comparison with KD methods. Compared with exisiting KD methods, our method is particularly developed to deal with medical images with incompelte modalities, by first adopting single-modality to boost multi-modality. A potential drawback is our method may not be suitable for large-scale datasets, due to the light structure. Computational cost and potential issues. Since only student network is used in practice, our method is very light with 26.87M flops, and can process a IF sequence within 50ms. For meidical application of DNN, an important issue is lack of explainability. In addition to applying visualization method, our method explicitly learns importance weight for each modality, which can explain the diagnosis priority of DNN.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
his paper presents a customized knowledge distillation framework specifically designed for nephropathy diagnosis. The reviewers appreciate the well-defined problem setting, thoughtful model design, and incorporation of prior clinical knowledge. However, they express concerns about the level of innovation, the rigor of the evaluation process and the choice of baseline methods, as well as the simplicity of the weights-based design.
I believe this paper represents a promising step towards integrating clinical knowledge with image analysis. Despite the aforementioned limitations, the author’s rebuttal effectively addresses many of the concerns raised. I am eagerly anticipating the discussions surrounding this paper at the MICCAI conference. This work is particularly noteworthy as it appears to be the first of its kind in the field of renal pathology.
Based on these reasons, I recommend accepting this paper.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
In rebuttal, the authors have provided new comparison with KD methods and state the novelty of the proposed model. Overall, the raised issues can be addressed.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors focused most of their rebuttal on the comparison with STOA and did not actually provide clarification of the novelty. Also, the critical comment by R1 regarding Yuan et al. method is not addressed. The authors promised adding lots of results to the camera reedy also, which I believe should have partially done in the initial submission. Again, supp. materials should contain extra info and all necessary info should be added to the main body of the manuscript