List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Marla Narazani, Ignacio Sarasua, Sebastian Pölsterl, Aldana Lizarraga, Igor Yakushev, Christian Wachinger
Abstract
Alzheimer’s Disease (AD) is the most common form of dementia and often difficult to diagnose due to the multifactorial etiology of dementia. Recent works on neuroimaging-based computer-aided diagnosis with deep neural networks (DNNs) showed that fusing structural magnetic resonance images (sMRI) and fluorodeoxyglucose positron emission tomography (FDG-PET) leads to improved accuracy in a study population of healthy controls and subjects with AD. However, this result conflicts with the established clinical knowledge that FDG-PET better captures AD-specific pathologies than sMRI. Therefore, we propose a framework for the systematic evaluation of multi-modal DNNs and critically re-evaluate single- and multi-modal DNNs based on FDG-PET and sMRI for binary healthy vs. AD, and three-way healthy/mild cognitive impaired/AD classification. Our experiments demonstrate that a single-modality network using FDG-PET performs better than MRI (accuracy 0.91 vs 0.87) and does not show improvement when combined. This conforms with the established clinical knowledge on AD biomarkers, but raises questions about the true benefit of multi-modal DNNs. We argue that future work on multi-modal fusion should systematically assess the contribution of individual modalities following our proposed evaluation framework. Finally, we encourage the community to go beyond healthy vs.AD classification and focus on differential diagnosis of dementia, where fusing multi-modal image information conforms with a clinical need.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16431-6_7
SharedIt: https://rdcu.be/cVD4O
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #3
- Please describe the contribution of the paper
This paper proposes a framework for the systematic evaluation of multi-modal DNNs and critically re-evaluate single- and multi-modal DNNs based on FDG-PET and sMRI for binary healthy vs. AD, and three-way healthy/mild cognitive impairment/AD classification.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper demonstrate that a single-modality network using FDG-PET performs better than MRI and does not show improvement when combined. This paper gives an evaluation framework for multi-modal fusion to systematically assess the contribution of individual modalities.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The theoretical novelty is very limited and the experiment is not innovative enough.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The reproducibility of the paper is not very well.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1) MCI vs. NC is more challenging than AD vs. NC, why not conduct experiments on MCI vs.NC? And the paper lacks the description of MCI. 2) The experimental results from Table 2 (CN vs. MCI vs. AD) cannot completely conclude that a single-modality network using FDG-PET performs best. Is the opinion too arbitrary? 3) In terms of quantity, the data with three labels is not balanced, how do you solve this problems?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The theoretical novelty is very limited and the experiment is not innovative enough. The experimental results from Table 2 (CN vs. MCI vs. AD) cannot completely conclude that a single-modality network using FDG-PET performs best.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #4
- Please describe the contribution of the paper
The author proposed to investigate the utility of multi-modal MRI + FDG PET for Alzheimer’s disease classification using deep-learning. They conducted a robust comparison of this two modality and different fusion schemes.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Paper is well-written and hypothesis are clearly stated.
- Evaluation framework is adapted and results are convincing.
- Experiments present interesting framework that can be used by the community for a better use of multimodale stratregy.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- No novelty, beside the fact that the validation strategy is novel and validate previous findings.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- The description of method and validation framework provides moderate reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- An argument against the presented findings is that a ResNet is not the correct model to learn structural pattern associated with AD. Indeed, current classification framework obtained better classification performance compared to results for AD classification using MRI readout only presented in this paper.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper proposed an interesting framework and highlight the need for futur paper to justify the fusion of sMRI and PET data. Paper is well-written and hypothesis are clearly stated, tested, and discussed.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #5
- Please describe the contribution of the paper
This paper designs a framework for the systematic evaluation of multi-modal DNNs and critically re-evaluate single- and multi-modal DNNs based on FDG-PET and sMRI for binary healthy vs. AD, and three-way healthy/mild cognitive impairment/AD classification. This study conforms with the established clinical knowledge on AD biomarkers, but raises questions about the true benefit of multi-modal DNNs.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
(1) This paper re-evaluates single- and multi-modal DL models based on FDG-PET and structural MRI for classifying healthy vs. AD subjects. (2) The experimental results show that FDG PET alone is sufficient for AD diagnosis, which conforms with established clinical knowledge about biomarkers in AD.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
(1) The authors design three fusion strategies to show the comparison results, and the results show that FDG PET alone is sufficient for AD diagnosis. However, whether do the results rely on the current classification models? Therefore, more SOTA classification models should be investigated with only using FDG PET. (2) The current comparison experiment is comducted on one dataset. To effectively the effectiveness of FDG PET, it is recommended to emply on more datasets.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors mention in reproducibility statement that they will release code and trained models after acceptance
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
(1) The authors design three fusion strategies to show the comparison results, and the results show that FDG PET alone is sufficient for AD diagnosis. However, whether do the results rely on the current classification models? Therefore, more SOTA classification models should be investigated with only using FDG PET. (2) The current comparison experiment is comducted on one dataset. To effectively the effectiveness of FDG PET, it is recommended to emply on more datasets. (3) When using PET and random MRI, the model still obtains promising performance. However, whether does the random MRI introduce incorrect information? More discussions should be included.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The findings are interesting, but more validations should be conducted on more datasets.
- Number of papers in your stack
6
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Strengths: Robust validation framework, novel validation approach, convincing results, very well written, framework is useful for community, interesting findings
Weaknesses: Limited theoretical novelty, single dataset
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
4
Author Feedback
N/A