List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Kyriaki-Margarita Bintsi, Vasileios Baltatzis, Rolandos Alexandros Potamias, Alexander Hammers, Daniel Rueckert
Abstract
Brain age estimation is clinically important as it can provide valuable information in the context of neurodegenerative diseases such as Alzheimer’s. Population graphs, which include multimodal imaging information of the subjects, along with the relationships among the population, have been used in literature along with Graph Convolutional Networks (GCNs) and have proved beneficial for a variety of medical imaging tasks. A population graph is usually static and constructed manually using non-imaging information. However, graph construction is not a trivial task and might significantly affect the performance of the GCN, which is inherently very sensitive to the graph structure. In this work, we propose a framework that learns a population graph structure optimized for the downstream task. An attention mechanism assigns weights to a set of imaging and non-imaging features (phenotypes), which are then used for edge extraction. The resulting graph is used to train the GCN. The entire pipeline can be trained end-to-end. Additionally, by visualizing the attention weights that were the most important for the graph construction, we increase the interpretability of the graph. We use the UK Biobank, which provides a big variety of neuroimaging and non-imaging phenotypes, to evaluate our method on brain age regression and classification. The proposed method outperforms competing static graph approaches and other state-of-the-art adaptive methods. We further show that the assigned attention scores indicate that there are both imaging and non-imaging phenotypes that are informative for brain age estimation and are in agreement with the relevant literature.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43993-3_19
SharedIt: https://rdcu.be/dnwNk
Link to the code repository
https://github.com/bintsi/adaptive-graph-learning
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This article proposes a framework for brain age estimation that combines imaging and non-imaging features in an attention-based dynamic graph construction to achieve improved performance and interpretability.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This paper is well organized and the language is clear.
- The task of brain age regression is challenging and meaningful.
- Experiments are comprehensive, including ablation analysis and interpretability.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1.The procedure and loss function about graph construction are mainly based on DGM(22’TPAMI), where I doubt that the novelty is enough.
- As for the experiments, the settings about the baselines are not illustrated clearly, I think they should be compared with best settings and performance.
- And I also wonder why the second baseline choose cosine similarity rather than Euclidean distance. According to the ablation study, Euclidean distance performs better.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The hyperparameters of proposed model are clearly claimed and most of settings about baselines are claimed. And code will be publicly available.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Though experiments are comprehensive, I think it would be more complete if the problems mentioned in weakness are fixed and more latest baselines are considered.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- Experiments are comprehensive and detailed.
- The novelty is limited.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The authors are proposing an end-to-end modeling scheme for brain age estimation, where multi-modality imaging and non-imaging features are extracted from a large public dataset (UK Biobank) and the non-imaging features are utilized to build a better graph representation of the multi-modality imaging features by incorporating similarity with respect to the non-imaging features. Then the resulting graph convolutional network is training using a novel graph loss function
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-The method aims to overcome the limitations of the previous GCN based multimodal-image fusion models for brain age estimation by improving the graph structure upon incorporating non-imaging features -The resulting GCN is trained using a novel graph loss for prediction of brain age that is an important indicator for neurological diseases -Via the attention mechanism, imaging and non-imaging features could be ranked as a means to provide relevance to brain age thus improving interpretability -The paper utilizes a large public dataset
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Based on the reported performance results, it appears that the problem itself still remains difficult.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper utilizes the UK Biobank dataset that is a large dataset and available upon applying as a researcher. The authors state that their code will be available. Adequate details on the architecture as well as experimental procedures are presented in the paper
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
I appreciated the review of the previous graph based method and identifying its short-comings and proposing a logical way to overcoming it. I also appreciated that the resulting population graph looked better in terms of having age groups clustered more coherently. However, the problem itself still looks challenging, maybe not something that can be solved simply by innovating on the algorithmic side on this public dataset.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I enjoyed the review of shortcomings of the previous technique and compelling evidence that their proposed method is indeed an improvement. However, maybe this problem just needs more innovations on the data acquisition, sensing technologies rather than solely on the algorithm side to be able to accurately predict brain age
- Reviewer confidence
Somewhat confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
1) This work combines imaging and non-imaging information in an attention-based framework to learn adaptively an optimized graph structure for brain age estimation. 2) This work proposes a novel graph loss that enables end-to-end training for the task of brain age regression. 3) The attention mechanism ranks all imaging and non-imaging phenotypes according to their significance for the task, thus allowing for good interpretibility. 4) The proposed method was applied to the UK Biobank (UKBB) and outperformed many competitive methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1) This work applied the adaptive graph learning to the medical domain by integrating imaging and non-imaging traits for brain age prediction. This application is novel. 2) From methodology, this work developed a novel graph loss by integrating the reinforcement learning reward/penalty function and the graph convolutional neural network. The method also looks novel. 3) This work used the UKBiobank data to evaluate the proposed methods, whose sample size is sufficient. 4) In this work, training, validation and test set were split and evaluations and comparisons were performed on the independent test set, which is relatively rigid. 5) The methods can give the importance scores of all imaging and non-imaging features, which are clinically useful for interpretations.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) The major weakness lies in whether the proposed method is really state-of-art in the brain age prediction. It can be seen that the authors only compared the proposed method with limited alternative methods including linear regression, the DGM, and the static population-graph models instead of the adaptively modified graph structure. Elastic Net, XGBoost, PC regression, fully connected or fully convolutional neural networks often outperforms a lot over the simple linear model, as when the feature dimension is large, the simple linear model may overfit the data. Linear model may not be suitable as a baseline model to compare. Besides, in the area of brain age prediction, the authors may compare with more existing machine learning methods (such as [1]-[3]) 2) In addition, we are not clear whether the F1-score in the 4 class age classification is high enough, as it is only 0.56. Is this F1-score the Macro-F1 score? how is it defined as a combined score for the 4 classes? 3) Although the sample size seems large enough, there is no standard deviations, confidence intervals or p-values reported when evaluation metrics between different methods were compared. We cannot conclude that the proposed method is significantly better than other existing methods.
[1] Estimation of brain age delta from brain imaging [2] Predicting brain-age from multimodal imaging data captures cognitive impairment [3] Accurate brain age prediction with lightweight deep neural networks
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
1) The paper provides a detailed description of the proposed method, including the architecture of the graph construction and the graph convolutional network (GCN). 2) The dataset used for evaluation, the UK Biobank (UKBB), is a well-known and widely used dataset in the medical imaging domain. 3) The paper presents clear evaluation metrics (MAE, r score, accuracy, AUC, F1-score) and compares the proposed method to relevant baselines. 4) The authors describe the implementation details, including the GCN architecture, optimizer, learning rate, and the number of epochs. 5) The ablation studies give additional insight into the method’s performance under different conditions, enhancing the understanding of the method’s behavior. 6) The exact dataset split (i.e., subject IDs) used for training, validation, and test sets is provided. 7) The code for the implementation will be made publicly available.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
To further improve the paper, the author may make a more comprehensive comparison of the proposed methods with existing state-of-art brain age prediction method, and compare their performance. For each evaluation metric, the author should better provide standard deviations, confidence intervals or p-values when making comparisons. The author should better justify if the 0.58 accuracy or 0.56 F1-score is high enough either by citing literature or a more comprehensive comparison.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The two primary factors I consider when scoring this paper are its novelty in both methodology and application. Despite some limitations concerning the results of this work, which may not surpass certain existing state-of-the-art age prediction models, I believe the paper has enough merit to warrant a tentative acceptance.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposed a new framework for brain age estimation. All reviewers show some enthusiasm on the paper, therefore, I would like to recommend accept but do suggest the authors to address those minor issues raised by reviewers on the final camera-ready version.
Author Feedback
We would like to thank all the reviewers and the AC for the constructive feedback. We plan to improve the clarity of the paper by incorporating the suggestions of the reviewers in the camera-ready version of the paper. More specifically:
Regarding the use of cosine similarity instead of Euclidean distance [R1], it should indeed be clarified and will be added in the paper that both performed similarly for the static graphs (baseline), with the graph created using cosine similarity slightly outperforming the one created using Euclidean distance.
The F1-score estimated for the 4-class age classification task was Macro-F1 score [R2], which will be added to the camera-ready version.
The concern that the linear model is quite simple and may result in overfitting, as well as that other more complex machine learning models would perform better for the task of brain age regression is valid and we agree to some extent with the reviewer [R2]. In our case, linear regression worked well and did not overfit. However, in the literature it is noticed that GNNs can usually perform worse than traditional machine learning models and that building a population graph that is meaningful enough to improve the performance of the GNN to state-of-the-art levels is difficult. The added benefit of constructing an adaptive graph using our method is the interpretability aspect. We will extend the discussion section of the paper accordingly. We look forward to discussing more about that in-person in Vancouver!
Finally, we strongly agree with the reviewer [R3] that brain age regression itself is a very challenging problem that might need more innovations on the side of data acquisition apart from the algorithmic advances.
We appreciate the thoughtful comments that help improve the quality of the paper.