List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Nemo Fournier, Stanley Durrleman
Abstract
We introduce a disease progression model suited for neurodegenerative pathologies that allows to model associations between covariates and dynamic features of the disease course. We establish a statistical framework and implement an algorithm for its estimation. We show that the model is reliable and can provide uncertainty estimates of the discovered associations thanks to its Bayesian formulation. The model’s interest is showcased by shining a new light on genetic associations.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43904-9_58
SharedIt: https://rdcu.be/dnwH4
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The paper adapts the Disease Course Mapping algorithm to include dependencies of trajectories on fixed risk factors, such as genetic profiles. The authors demonstrate the extended algorithm in a simple simulation and on data from the ADNI data set using 69 SNPs as candidate influential factors.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Timely work on a popular and evolving topic.
- A simple but important advance on the existing state of the art.
- The presentation is clear and thorough.
- The experiments are sensible and conclusions reasonable.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Lack of discussion of the wider literature on disease progression modelling beyond the precise model the authors choose to build on.
- Simulation experiment is simplistic.
- Lack of insight into the findings from the ADNI experiment as to whether fitting the model identifies anything realistic/interesting.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Fine
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
This is a nicely written paper introducing a simple enhancement of the disease course mapping model of Schiratti et al JMLR 2017 to capture dependence of biomarker trajectories on risk factors such as genetic markers. The model implements a simple linear adjustment of the trajectory model for each risk factor considered.
The context / literature review is rather narrow. There is a huge body of work on disease progression modelling, but the authors really only talk about the particular model they build on here. It should be expanded to put the work properly in context. In particular, other works also consider the influence of various risk factors on disease trajectory, for example Young et al Nat Comms 2018. I think most previous works analyse risk-factor effects post-hoc after estimating disease trajectories, rather than simultaneously with trajectory estimation, as this paper proposes. However, that discussion is important but missing in the literature review of this paper. Similarly, closer to the content of this paper, follow on work from Disease Course Mapping by Koval et al on the Digital Brain model also appears to show dependence on risk factors and some discussion of how the proposed approach differs to that would seem important in the literature review here.
Experiments demonstrate the method first in a simple simulation with three dynamic biomarkers and three fixed risk factors. The simulation appears to demonstrate that the method recovers adjustments in trajectory associated with the risk factors, although I found figure 2 a little difficult to interpret (unsure what the red curve in panel c is). Second they use the ADNI data set and demonstrate that, as expected, APOE mutation status affects progression according to the model. They then repeat the experiment with 69 SNPs selected as potential risk factors. These appear to show a variety of influences, which are not really interpreted so hard to know if the model recovers anything useful/credible.
Overall the results are a little underwhelming, but they do demonstrate the potential of the model to discover links between risk factors and disease trajectory, which certainly has potential utility.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
It is an simple incremental advance, but potentially important. The experiments and results are a bit weak.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper presents a method for adding covariate adjustment to the manifold-based multimodal disease biomarker progression model of reference 13. That model poses that as a vector of biomarker values evolves over time it traces out a trajectory on a high-dimensional manifold; the population level mean trajectory is cast as a fixed effect, and random perturbations away from this mean trajectory are used to model the trajectories as they occur in individuals (random effects). This paper allows the mean trajectories to be transformed by supplied covariate values, with parameterized functions mapping those covariate values to deformations of the mean trajectory. An MCMC approach is used to estimate all the parameters of the original model of [13], together with the new parameters of these covariate-based transformation functions. Simulated and real data experiments evaluate the ability of the method to back out ground-truth or expected covariate-biomarker relationships.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Improving multimodal biomarker progression modeling is a worthy enterprise. This kind of modeling remains essential to improving our understanding of how brain diseases like Alzheimer’s works in real people.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- It is difficult to escape the impression that this is only a minor modification of [13]. Yes, covariates are now allowed to modify model trajectories, and that seems useful. But it doesn’t seem like this adds very much on top of what was already there, or that the approach to covariate adjustment is non-obvious. The cost of adding these covariates appears to be large, as a weighty MCMC optimization must now be undertaken for parameter estimation.
- The simulated data is difficult to understand. Usually the term “covariate” is used to refer to a variable that is measured, such as age or blood pressure. Here, the covariates are motor-, memory-, and language-related “risks.” Those three seem to be fractions that add up to 1. It’s not clear what those correspond to in real life— are they polygenic risk scores? It also is unclear why these covariates should increase biomarker abnormality in a sigmoid fashion with respect to age.
- In Fig 3, the fact that all AD biomarkers except cognition are at least somewhat abnormal before age 50— regardless of APOE genotype— should suggest that something is wrong with this model. Those biomarkers are widely believed to remain normal in most people until after that age.
- The fact that APOE has a null effect on tau and amyloid, and a borderline effect on hippocampus and ventricle volume, does not inspire confidence in the model either.
- The SNP data is really difficult to understand.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The simulated and real data are from well-described sources, which should help with reproducibility. Several aspects of the proposed algorithm are described vaguely or not at all, especially in the optimization part. So I believe true reproduction would be difficult.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
I believe that really drilling down into why, exactly, covariate adjustment of [13] is a major endeavor, or a non-trivial enterprise, would help a lot. I think a more intuitive formulation of the simulated data would also be a big help, as well as clearer explanations of what lessions we are supposed to draw from the real data.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
At this point it is difficult for me to tell whether this is really a substantial contribution or not, and I am having difficulty figuring out what exactly the simulated and real data results are proving about this particular method.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
3
- [Post rebuttal] Please justify your decision
I remain fairly convinced that the simulated data is totally confusing, the method is only making a small incremental contribution over prior work, and the ADNI data is confusing and/or doesn’t conform to current theories about AD. The “motor risk” of the simulated data seems to be akin to a polygenic risk for motor dysfunction that doesn’t just increase rate / onset of motor decline, but simultaneously decrease rate / onset of memory and language dysfunction. As such, it is unclear what real, actual polygenic risk score this might be considered akin to. The explanation of the ADNI null APOE / non-null APOE effects remains confusing. And the whole thing is just a minor change from what was there in the literature previously.
Review #5
- Please describe the contribution of the paper
This study proposes a framework from a statistical standpoint to model the association between covariates and dynamic features of the disease course. The authors propose both a theoretical framework of how this association could be modeled and further develop this framework. The authors first demonstrate the working of their model on a simulated dataset and further test their model on data from ADNI. Nonlinear data driven modeling of association between covariates and dynamic features in neurodegenerative disorders is a desired feature.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
a) Development of a statistical framework that can model association between covariates and disease progression b) Seemingly robust simulated dataset and testing on a well-characterized clinical cohort.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The lack of interaction between the covariates and their influence on disease progression dampens the enthusiasm.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Not applicable
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
a) The covariates affecting the disease progression are not independent, as shown by the authors, but the authors show no inter-covariate influence and their effects pertaining to how this interaction might affect the disease progression. b) While modelling the effect of any covariate on disease progression is an interesting concept, how does this modelling inform about a participant level monitoring of disease progression, which I thought was the authors original motivation? c) Several predictive algorithms exist that have attempted to predict conversion of MCI participants to AD using multitude of the fixed and random effects with about 90% accuracy (https://www.nature.com/articles/s41598-019-38793-3) . What are the authors opinion on how their model could be used in such predictive algorithms? d) Can the authors should use a different clinical example where the presence or absence of any genetic mutation might alter the progression of disease course such as ALS? This might make their paper stronger.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Clarity, organization, and presentation
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
The response though not satisfactory deems discussion with a broader community.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper’s strengths lie in its clear presentation, robust statistical framework, and the novel disease progression model it introduces. The model’s utility is demonstrated with both simulated and real-world data. However, the paper falls short in providing a comprehensive discussion of the wider literature on disease progression modeling, and there are concerns about the originality of the work. The interpretation of the results from the ADNI experiment is also seen as inadequate, raising questions about the model’s practical implications. The authors are invited to submit a rebuttal addressing the following key points raised by the reviewers:
Interpretation and Presentation of Results: A consistent concern across all reviewers is the interpretation and presentation of the results, especially those from the ADNI experiment. The authors should provide a thorough interpretation of these results and a clearer discussion on the model’s real-world implications and its utility. In particular, the authors should clarify the implications of the findings from the ADNI experiment and how the modelling informs participant-level monitoring of disease progression.
Context and Literature Review: The authors should expand the context and literature review to properly situate their work within the broader field of disease progression modeling. The discussion should include a comparative analysis with other relevant works, particularly those by Young et al. (Nat Comms 2018) and Koval et al.’s work on the Digital Brain model. The authors should highlight how their work differs and contributes to this field in a novel way. (Reviewer 1)
Significance of Covariate Adjustment: The authors should provide a robust justification explaining why the covariate adjustment of the model presented in [13] is a significant contribution to the field. The authors should clarify why this modification isn’t an obvious extension of the original model, and highlight its utility in disease progression modelling. (Reviewer 2)
Author Feedback
We thank the reviewers whose insightful comments will improve this work. They pointed out that the paper is overall well written and its contribution —devising covariate-adjustment in a Disease Progression Model (DPM)— is a worth pursuing effort.
As R1,2,3 pointed out, our experiment settings and results would benefit from clearer discussion. In simulation, the objects’ naming is indeed confusing, as highlighted by R2. We simulate 3 biomarkers (memory, motor & language abnormality) with sigmoid-shaped progressions —reasonable assumption in neurodegenerative DPM— and 2 covariates akin to polygenic risks (memory- and motor-risk). These continuously modulate the biomarkers’ expected trajectory for each patient (eg. the motor-risk balances between the two first profiles of fig 2a, leading to different paces of motor abnormality). We will improve our wording and fig 2’s clarity, thus also addressing R1’s remark.
On ADNI, our main statement is that adjusting DPM to covariates can bring light as to how covariates associated with diagnosis actually modulate the disease progression (be it the pace or level of particular biomarkers, disease onset). Moreover, to address R3’s point on patient-level monitoring: adjusting [13]’s Disease Course Mapping (DCM) with covariates should improve its forecasting abilities: [Maheux et al, 2023] show how an unadjusted DCM allows clinical trial enrichment via progression forecasting from a baseline visit. Adjusting the priors of a DCM with —previously unused— covariates should further increase its predictive performances, to be demonstrated in future work on AD and other ND (ALS, FTD, …) On R2’s comment on borderline/null effect of APOE: we stress that the effects shown in fig 3 relate to v0 (disease pace). Links between APOE and p0/t0 (baseline levels/onset time) were also estimated and significant. One can get convinced from the included material by seeing the trajectories in fig 3 of e.g. amyloid, which are disjoint between APOE status (their difference including slope, baseline level, etc). Relevant effects bar for p0 will be added in fig 3a.
As R1 noted, the literature review would benefit from a broader coverage of SOTA (Young et al, Koval et al), eg. instead of the mixed-effect paradigm exposition. We will complete the discussion by stressing how covariates were mostly considered in a-posteriori association studies while our method enables a-priori covariate-adjustment of the DPM.
R2 expresses concern about our work being a trivial extension of [13. Given the associations of covariates and random-effects exhibited by Koval et al, a logical step is to model a multimodal distribution of random-effects and estimate its modes conditioned by the covariates. Yet, this falls short as when the number of covariates increases, the conditionings have to be estimated in ever smaller subgroups. Non-supervised estimation of DPM mixtures have also been devised (eg. [Poulet et al., 2021] expanding on [13]’s model, or Young et al. on [Fonteijn et al, 2012]’s EBM) but only enable post-hoc associations with covariates. Our approach —built on the hypothesis that the considered covariates have slight and continuous additive effects around the disease average dynamics— leverages these covariates a-priori by continuously adapting the priors and fixed-effects of the DCM. While it retains the geometric roots and interpretable parametrization that made DCM successful, our framework is more general. As illustration: [13] cannot model patients whose progressions are not parallel to the reference geodesic (v0 is common to all patients), while ours can (see fig 2b&c). Our work can be seen as learning both a continuum of DCM models and the link between covariates and trajectories. Moreover, adjusting the progression via a generic link function will allow direct extension to non-linear links between covariates and disease progression (in the same Bayesian setting) and eg. model interactions, thus addressing R3’s comment.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Although improving multimodal biomarker progression modeling is an important and interesting topic, the reviewers agree that the proposed method s an simple incremental advance and there remains some confusion about the synthetic data.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper presents a method for adding covariate adjustment to the manifold-based multimodal disease biomarker progression model.
Key strengths:
- multimodal biomarker progression modeling is a valuable topic. propose a workable solution.
- clear method presentation.
Key weaknesses:
- the novelty seems to be incremental.
- lack in-depth discussion and analysis
The rebuttal somewhat addresses the novelty issues. The simulation experiment is still a little confusing.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper is mostly extension of [13] by adding covariates and studying the genetic effect on progression. The simulation scnerio seems to be realistic and a quite a bit of literature are missing. I think it is a useful contribution to the community but it is disappointing to see large number of ommited work.