List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Yanjie Zhou, Youhao Li, Feng Zhou, Yong Liu, Liyun Tu
Abstract
Construct a generalizable model for the diagnosis of Alzheimer’s disease (AD) is an important task in medical imaging. While deep neural networks have recently advanced classification performance for various diseases using structural magnetic resonance imaging (sMRI), existing methods often provide suboptimal and untrustworthy results because they do not incorporate domain-knowledge and global context information. Additionally, most state-of-the-art deep learning methods rely on multi-stage preprocessing pipelines, which are inefficient and prone to errors. In this paper, we propose a novel domain-knowledge-constrained neural network for automatic diagnosis of AD using multi-center sMRI. Specifically, we incorporate domain-knowledge into a ResNet-like architecture. We explicitly enforce the network to learn domain invariant and domain specific features by jointly training multiple weighted classifiers, so that pixel-wise predictive performance generalizes to unseen images. In addition, the network directly takes segmentation-free and patch-free images in original resolution as input, which offers accurate inference with global context information and accurate individualized abnormalities to further refines reproducible predictions. The framework was evaluated on a set of sMRI collected from 7 independent centers. The proposed approach identifies important discriminative brain abnormalities associated with AD. Experimental results demonstrate superior performance of our method compared to state-of-the-art methods.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43904-9_44
SharedIt: https://rdcu.be/dnwHp
Link to the code repository
https://github.com/Yanjie-Z/DomainKnowledge4AD
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This article presents a novel domain-knowledge-constrained neural network for the diagnosis of Alzheimer’s disease (AD) using multi-center structural magnetic resonance imaging (sMRI). The proposed method incorporates domain knowledge into a ResNet-like architecture and directly takes segmentation-free, patch-free images in original resolution as input. The framework was evaluated on sMRI data from 7 independent centers, showing good performance
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The method integrates domain knowledge into the neural network architecture, which can improve the model’s understanding of the problem and lead to better performance.
- The network can directly process images in their original resolution without the need for segmentation or patch extraction, simplifying the preprocessing pipeline.
- The method captures global context information, which can help identify accurate individualized abnormalities and improve the overall predictive performance.
- The framework was tested on sMRI data collected from 7 independent centers, demonstrating its generalizability and robustness across different datasets.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- authors cropped the input MRI to only the central part of the brain. It is demostrated, though, that also other information present in the remaining part of the brain could be useful (for instance the cortical thickness)
- The incorporation of domain knowledge into the neural network architecture may increase the complexity of the model, potentially making it harder to train and optimize, or could lead to a wrong initial assumption
- The method’s focus on individualized abnormalities could lead to overfitting, especially if the training data is not diverse enough or does not adequately represent the target population. Please add a paragraph about diversity
- the used base network architecuture is very simple and somewhat outdatad: please justify
- no public dataset is used. This hinder the reproducibility and transparency of the reuslts
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
It is very difficult to reproduce this work given that both the code and the data are not available
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Please careful answer the weaknesses highlighted above
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The idea is intersting but there are a few critical weakness.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
The authors respond clearly to my main concerns, thus I increase my previous score by one (though this has still some weakness)
Review #2
- Please describe the contribution of the paper
This paper introduced a domain-knowledge-constrained neural network for automatically diagnosing Alzheimer’s disease (AD) using multi-center structural magnetic resonance imaging (sMRI).
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
In this paper, the authors incorporate domain knowledge into a ResNet-like architecture. They explicitly enforced the network to learn domain invariant and domain-specific features by jointly training multiple weighted classifiers. Besides, the network directly toke segmentation-free and patch-free images in original resolution as input, offering accurate inference with global context information and individualized abnormalities to further refine reproducible predictions.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
More details are required about the used datasets. In addition, the paper needs some modifications in its strcture.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Yes. It can be reprodcused if the datasets are available. The authors mentioned the details of their proposed model.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The authors should consider the following issues in their paper:
- The abstract section is inconsistent and hard to follow. The authors should rewrite the abstract section to mention the paper’s main purpose, primary contributions, experimental results, and global implications.
- At the end of the introduction section, the authors should summarize the paper’s main contributions into concise points.
- All values in Table 1 should be represented as percentages, not mixed.
- The paper needs intensive proofreading as it contains many long, inconsistent sentences. Besides, the manuscript contains many grammar errors and typos.
- The authors should discuss their future research directions at the end of the conclusion section.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The methodology is reprsented in a good way. In addition, the idea of having datasets from different sources is good for generalization.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
In this paper, authors propose ResNet-base model to classify Alzheimer disease or not. The proposed model is evaluated by 7-fold cross-validation and compared to some baseline models such as ResNet.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Good implementation details: The architecture of the proposed model and training condition including hardware information are written properly.
- Saliency map: Authors also show the attention map like Fig. 2 and 3, and try to explain the behavior of proposed model.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Limited technical contribution: The proposed model is basically combination of 3D ResNet and MLP.
- The effects of some parameters are unclear: For example, if cropping size and the number of classifier are changed, how does the performance of the model change?
- Limited evaluation metrics: Authors mainly evaluate models by using only accuracy AUC. However, it is better to add more evaluation metrics such as sensitivity and specificity.
- Limited comparison: Authors do NOT compare the proposed model to other SOTA models.
For instance, the models as below might be candidates for comparing:
- Qiao, H et al (2021) “Early alzheimer’s disease diagnosis with the contrastive loss using paired structural mris.”
- Liu, M et al (2020) “A multi-model deep convolutional neural network for automatic hippocampus segmentation and classification in alzheimer’s disease.”
- Liu, M et al (2018) “Landmark-based deep multi-instance learning for brain disease diagnosis.”
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Authors use their own dataset and train the proposed model by using it.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
For future work, I would recommend:
- Evaluation by using public dataset: ADNI dataset is very popular and useful dataset for Alzheimer disease (https://adni.loni.usc.edu/). It is better to evaluate by using not only authors’ own data but also public dataset.
- More evaluation metrics: In clinical situation, sensitivity and specificity are very useful and important. It is better to compare models by using it.
- The effect of Domain Knowledge Encoding: It is one of authors’ key idea to improve model. However, its effect is not shown in this paper. Thus, it is better to add the effect of Domain Knowledge Encoding to ablation study.
- Comparison with SOTA models, for example models as below might be candidates:
- Qiao, H et al (2021) “Early alzheimer’s disease diagnosis with the contrastive loss using paired structural mris.”
- Liu, M et al (2020) “A multi-model deep convolutional neural network for automatic hippocampus segmentation and classification in alzheimer’s disease.”
- Liu, M et al (2018) “Landmark-based deep multi-instance learning for brain disease diagnosis.”
Miner revision(typo): In Table. 1, areaunder the curve -> area under the curve In Methods section, The model operates in 3 major steps: 1) … b) … c) -> authors should use 1), 2), 3) or a), b), c).
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- Limited technical contribution: The proposed model is basically combination of 3D ResNet and MLP.
- The effects of some parameters are unclear: For example, if cropping size and the number of classifier are changed, how does the performance of the model change?
- Limited evaluation metrics: Authors mainly evaluate models by using only accuracy AUC. However, it is better to add more evaluation metrics such as sensitivity and specificity.
- Limited comparison: Authors do NOT compare the proposed model to other SOTA models.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
4
- [Post rebuttal] Please justify your decision
Some concerns would be removed if the paper are revised satisfactorily. However, some drawbacks are remaining, for example, the proposed model is outdated and is basically combination of 3D ResNet and MLP.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This article presents a novel domain-knowledge-constrained neural network for the diagnosis of Alzheimer’s disease (AD) using multi-center structural magnetic resonance imaging (sMRI). Domain knowledge is incorporated. In addition, the network directly takes segmentation-free and patch-free images in original resolution as input. Experiments have been conducted on a large number of sites (7 centers) to verity the effectiveness.
However, several drawbacks in terms of dataset, methodology and experiment are raised by the reviewers.
- No public dataset is used. This may hinder the reproducibility and transparency of the results. Also, more details are required about the used datasets.
- The proposed model is outdated and is basically combination of 3D ResNet and MLP.
- Lack of evaluation metrics (e.g., sensitivity and specificity) and comparison with other competing methods. The authors are suggested to refer to the review comments carefully to improve the quality of this work. This submission needs to be re-considered after rebuttal.
Author Feedback
We thank all reviewers for their valuable comments that improved the clarity of our manuscript. Below, we address their main concerns in the updated version.
Contributions: We present a novel method for predicting Alzheimer’s Disease (AD) that is both generalizable and interpretable. We thank Reviewers #1 and #2 for their agreements on the strengths of our main contributions: 1) integrating domain knowledge to address data distribution gaps for reliable predictions; 2) direct processing of images in their original resolution, eliminating the need for segmentation or patch extraction and providing a simplified end-to-end trainable model; 3) accurate inference with global context information and individualized abnormalities to further refine reproducible predictions; 4) evaluation on sMRI images from 7 independent sites, demonstrating the method’s generalizability and robustness across different datasets.
Data and Reproducibility: Regarding the dataset concerns raised by all Reviewers, we intentionally omitted specific citations to preserve anonymity. However, the data is available from the corresponding author of [anonymized] published in a reputable journal (IF>17), upon request. Code will be publicly accessible at GitHub.
Model: We thank Reviewers #1 and #3 for the feedback on model complexity. In Section 2.1 of our submitted manuscript, the cropping size is a fixed tuple determined by the maximum bounding box containing informative anatomical objects associated with AD. The complexity arising from the increasing number of classifiers may pose a limitation, this can be easily addressed by clustering similar domains to share a classifier when more data becomes available. We acknowledge the existence of modern and powerful models beyond ResNet and MLP, however, our innovation and formulation are independent of the specific network architecture employed. Notably, even with such limitations, our experiments achieve state-of-the-art overall predictive performance with an average AUC of 0.92 in AD identification and robust generalization across 7 domains.
Evaluation: We thank Reviewer #3’s comments on the evaluation metrics. Typically, accuracy and AUC serve as adequate metrics in evaluating the overall discriminative power of ensemble decision-making for domain generalization [Li et al., AISTATS, 2019; Hoffman et al., ICML, 2018; Balaji et al., NeurIPS, 2018], as the one we presented. In the updated manuscript, we use and cite in Section 3.1 the approach presented in [anonymized], with a reported average sensitivity and specificity of 0.78 and 0.92 respectively on ADNI dataset. As depicted in Fig. 2 of our submitted manuscript, the relationship between predicted probabilities and the clinical metric MMSE highlights true positive and false negative samples (yellow dots). Furthermore, we have enhanced the manuscript to provide clearer clarification that the removal of the domain knowledge encoding module led to a downgrade from the “Proposed” method to the “Baseline” method, resulting in average accuracies of 89.25% and 85.95% respectively.
Comparison with SOTA methods: Reviewer #3 suggested to compare our work with [Liu et al., NeuroImage, 2020; Liu et al., Med Image Anal., 2018; Qiao et al., Comput Methods Programs Biomed, 2021]. These models assume identical and independent data distributions, leading to degraded performance with shifted test domain distributions [Balaji et al., NeurIPS, 2018]. Furthermore, they rely on multi-stage pipelines for landmark or hippocampus detection and prioritize classification over generalization, which is in contrast to our model. Consequently, direct comparisons are not applicable. Additionally, they were developed using single-site datasets, making their effectiveness uncertain in our multi-site scenario. Nonetheless, our submitted manuscript includes comparisons with ResNet and 2 state-of-the-art works [Selvaraju et al., ICCV, 2017; Wang et al., Arxiv, 2022].
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Most of the major concerns (such as model complexity, evaluation metrics) have been well clarified.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper received mainly positive reviews, although there remain some drawbacks like of evaluation on public datasets, I vote for accepting the paper in accordance with the majority of the reviewers.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
It is an interesting work which addresses a novel approach to harmonize data from different data sources for accurate prediction. The authors have been responsive in the rebuttal. Two reviewers have raised their scores. The work may inspire novel ways to integrate multi-center structural MRI data. Acceptance is recommended.