Authors

Ke Zou, Tian Lin, Xuedong Yuan, Haoyu Chen, Xiaojing Shen, Meng Wang, Huazhu Fu

Abstract

Multimodality eye disease screening is crucial in ophthalmology as it integrates information from diverse sources to complement their respective performances. However, the existing methods are weak in assessing the reliability of each unimodality, and directly fusing an unreliable modality may cause screening errors. To address this issue, we introduce a novel multimodality evidential fusion pipeline for eye disease screening, EyeMoSt, which provides a measure of confidence for unimodality and elegantly integrates the multimodality information from a multi-distribution fusion perspective. Specifically, our model estimates both local uncertainty for unimodality and global uncertainty for the fusion modality to produce reliable classification results. More importantly, the proposed mixture of Student’s t distributions adaptively integrates different modalities to endow the model with heavy-tailed properties, increasing robustness and reliability. Our experimental findings on both public and in-house datasets show that our model is more reliable than current methods. Additionally, EyeMost has the potential ability to serve as a data quality discriminator, enabling reliable decision-making for multimodality eye disease screening.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43990-2_56

SharedIt: https://rdcu.be/dnwMb

Link to the code repository

https://github.com/Cocofeat/EyeMoSt

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The authors introduced a multimodality fusion pipeline for eye disease screening based on the mixture of t-distributions that improves the classification performance with the aid of unimodal and global uncertainties.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The main strength of the paper is the novel evidential fusion model that integrates both aleatoric and epistemic uncertainties in unimodality for fusion of multimodal data for classification.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The main weakness is lack of clarification on how unimodal uncertainties and global uncertainty drive the learning process, and lack of ablation study.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The reproducibility of the paper is high because the authors will release the code after acceptance and one dataset is public.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
1. A list of contributions/novelties should be summarized at the end of the introduction.
2. Careful proofreading is required. For example, “we derive the NIG distributions to Student’s t (St) distributions”. What is being derived from what?
3. Why a t-distribution with heavier tailed is preferred? More intuition should be provided for readers not being able to follow the full derivation.
4. Is the fused distribution simply the selection of an unimodality distribution with heavier tails (for M=2), instead of some fashion of combination from the multiple modalities? This is my impression from eqn 6 and figure 3(b).
5. In equation 12, do all modalities contribute to learning, including the one with less reliability? How are uncertainties incorporated in the learning process? I do not see how unimodal uncertainties and global uncertainty can drive the learning process.
6. In Table 1, why kappa is studied for the GAMMA dataset, but ECE for the in-house dataset? TMC for in-house should be colored in red as well.
7. Additional ablation study should be done on eqn 12 to see the effect of using only the unimodal uncertainties, only fusion model uncertainty, and no uncertainty.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The recommendation is based on the much improved results for the GAMMA dataset compared to other uncertainty-based fusion models.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

The paper proposes a novel multimodality evidential fusion pipeline for eye disease screening, EyeMoSt, which provides a measure of confidence for unimodality and elegantly integrates the multimodality information from a multi-distribution fusion perspective. The paper also introduces the Mixture of Student’s t distributions to adaptively integrate different modalities to endow the model with heavy-tailed properties, increasing robustness and reliability.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. This paper describes a new method for reliable and robust screening of eye diseases called EyeMoSt, which uses multimodality fusion of Fundus and OCT data. EyeMoSt places Normal-inverse Gamma prior distributions over pre-trained neural networks to learn both aleatoric and epistemic uncertainty. Additionally, EyeMoSt introduces Mixture of Student’s t distributions to provide robust classification results with global uncertainty and heavy-tailed property awareness.
2. The extensive experiments conducted on GAMMA and in-house datasets demonstrate the robustness and reliability of the proposed method in classification and uncertainty estimation. The competitive performance of EyeMoSt with previous methods is also a testament to its effectiveness.
3. The paper provides a clear and detailed description of EyeMoSt’s framework and its theoretical foundations.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The paper lacks ablation experiments and the validity of the proposed module needs to be verified.
2. The paper does not provide enough background information on the existing methods for multimodality eye disease screening and their limitations. The paper only briefly mentions some methods for different fusion stages or uncertainty estimation without explaining how they work or why they are inadequate for the problem.
3. The paper does not explain how EyeMoSt handles missing or unpaired data from different modalities.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper provides some details on the implementation and training of EyeMoSt but does not share the code or data publicly. The paper should make the code and data available online to facilitate reproducibility and verification of the results. The paper should also provide more details on the hyperparameters used for training EyeMoSt such as learning rate, batch size, and regularization.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
1. The paper should clearly state its research question or hypothesis at the beginning of the introduction section and explain how it relates to existing literature and methods.
2. The paper should perform some ablation studies or analysis to show the contribution of each component of the proposed method and compare it with other alternatives. The paper should also discuss how different choices of parameters or hyperparameters affect the performance of EyeMoSt.
3. The paper should discuss some limitations or drawbacks of the proposed method and suggest possible directions for future work.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper addresses an important and challenging problem of multimodality eye disease screening, which has potential applications in ophthalmology. The paper only briefly mentions some methods for different fusion stages or uncertainty estimation without explaining how they work or why they are inadequate for the problem. The paper should provide a more comprehensive literature review and compare EyeMoSt with more relevant baselines.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #4

Please describe the contribution of the paper

The main contribution of this work is the introduction of a novel multimodality evidential fusion pipeline for eye disease screening, EyeMoSt, which provides a measure of confidence for unimodality and elegantly integrates multimodality information from a multi-distribution fusion perspective. The experimental findings on both public and in-house datasets show that EyeMoSt is more reliable than existing methods and has the potential to serve as a data quality discriminator for multimodality eye disease screening.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The paper is well written with good clarity and formulations.
2. The proposed EyeMoSt is interesting and captures the uncertainty of different modalities when processing the feature fusion, which seems convincing to me.
3. The methodology section is well-explained and provides a clear overview of the proposed approach.
4. The experimental findings section is well-presented, and the use of both public and in-house datasets adds to the credibility of the results. The authors provide a detailed analysis of the experimental results and discuss the implications of the findings, e.g., OOD samples.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The novelty of the proposed methods is limited. For the uncertainty estimation, this work modified the deep evidential regression into a classification form. The same applies to the other components. Although technical novelty is not my first consideration in evaluating this paper, more deep thinkings to the related medical problem, i.e., ophthalmology, while designing your methods.
2. In a multimodal task, I expected to see that how to investigate the relationships between the uncertainties of different modalities to build more justified and valid uncertainty estimates. To this point, this paper looks more like a direct use of uncertainty information in a multimodal task to obtain the gain of either modality in information learning, rather than using uncertainty information to help understand multimodality.
3. The comparison study is more like an ablation study, as it evaluates different counterparts (feature fusion strategies, uncertainty estimation). Similarly, there is no other ablation analysis to investigate how each component makes your methods work. Since the glaucoma datasets are collected from a public challenge, have you compared your methods with the GAMMA challenge benchmarks?
4. Some training details are missing. Are the same data augmentation techniques applied into both the fundus and OCT images? For example, random color jitter.
5. Some typos need to be refined, e.g., where is Table 3? Bold is overused, usually you can bold a few words instead of several sentences.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Although the authors did not provide sufficient implementation details of this work, they promised to release the codes. I therefore consider this work to be reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

Please refer to the comments. Overall, I think this is a good technical paper but some details should be further improved.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The technical novelty, writing and experimental results.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposes a multimodality evidential fusion pipeline for eye disease screening via mixture of students t distributions. Given three consistent positive reviews, I recommend accepting this submission. However, there are some concerns arised by the reviewers, such missed technique details, ablation study and comprison experiments, typos. The authors should address those concerns in the published version.

Author Feedback

We sincerely thank you for provisionally accepting our paper. We express our gratitude to the reviewers for their high-quality reviews and constructive feedback on our manuscript. We have provided point-to-point responses to the comments, which will be integrated into the final version of the paper.

back to top

Reliable Multimodality Eye Disease Screening via Mixture of Student's t Distributions