List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Benoît Sauty, Stanley Durrleman
Abstract
Disease progression models are crucial to understanding degenerative diseases. Mixed-effects models have been consistently used to model clinical assessments or biomarkers extracted from medical images, allowing missing data imputation and prediction at any timepoint. However, such progression models have seldom been used for entire medical images. In this work, a Variational Auto Encoder is coupled with a temporal linear mixed-effect model to learn a latent representation of the data such that individual trajectories follow straight lines over time and are characterised by a few interpretable parameters. A Monte Carlo estimator is devised to iteratively optimize the networks and the statistical model. We apply this method on a synthetic data set to illustrate the disentanglement between time dependant changes and inter-subjects variability, as well as the predictive capabilities of the method. We then apply it to 3D MRI and FDG-PET data from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) to recover well documented patterns of structural and metabolic alterations of the brain.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16431-6_1
SharedIt: https://rdcu.be/cVD4I
Link to the code repository
https://github.com/bsauty/longitudinal-VAEs
Link to the dataset(s)
Reviews
Review #1
- Please describe the contribution of the paper
The authors proposed to endow the latent space of a VAE with a linear mixed-effect longitudinal model to generate MRI or PET images of elderlies that progress with time. The latent representation, including patient onset time, acceleration factor, and individual space shifting, can be applied to characterize Alzheimer’s disease progression.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- provided a progression model that disentangles temporal changes from changes due to inter-patients variability, and allows sampling patients’ trajectories at any time point, to infer missing data or predict future progression;
- proceeded to dimension reduction using a convolutional VAE with the added constraint that latent representations must comply with the structure of a generative statistical model of the trajectories;
- demonstrated this method on a synthetic data set and on both MRI and PET scans from the Alzheimer’s Disease Neuroimaging Initiative (ADNI), recovering known patterns in normal or pathological brain aging.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The authors proposed a novel way to synthesize images of AD patients, and provided numerical estimations of disease progression for each individual. I like how it combines linear mixed effect model and a generative model, but additional analysis of the performance of the model can be provided to support the model. For example, the authors can generate different average images for different disease stages (NC, MCI, and AD), and try to find the difference between modalities. They can also analysis different onset time and acceleration factors of disease stages, and apply statistical tests to see whether the difference is significant. Sometimes it’s hard to evaluate the results with generated images for a random subject, so quantitative evaluation may be helpful in this scenario.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors used public datasets in the experiment, and they claimed that the code will be made publicly available upon acceptance of the paper.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- What disease stage groups are included in the training and test group, respectively, and why? How does input of different disease stage groups influence the result?
- I like the style and the amount of information delivered through Figure 1, but I think more contents can be added to it to make the model more clear. For example, the authors could add symbols besides the lines and arrows in the middle figure to make it clear. During training, what parameters have actual meaning and what parameters are from a Gaussian sampling?
- In algorithm 1, it would be good to elaborate the simulation and approximation part in the main text, or refer to some citations so that readers would know how this process is simulated.
- In the result section 3.2, the authors claimed that the minimum dimension to capture the dynamics of structural MRI is 16. Could you provide citation where this number comes from?
- In Figure 3, only the synthesized images are shown, and it doesn’t seems to be clear enough for a T1 MRI image of the whole brain. Also the resolution (809680) does not seem high enough. Could the authors provide a sample original image in Figure 3 for comparison?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
It is a good method to generate images along time and predict meaningful parameters that can represent disease progression at the same time.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
This paper proposes a method for the generation of images following a disease progression model. The method combines a variational autoencoder with a temporal linear mixed-effect model that allows learning from the data a latent representation that is able to disentangle age from disease effects in image generation. The ability of the method to correctly generate the images has been demonstrated in a simulated experiment. In addition, the authors show how the method is able to generate images with well known patterns in Alzheimer’s Disease progression from MRI and PET data.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper successfully addresses the difficult problem of generating disease progression models from images.
The solution is original and very innovative. It successfully combines the finding of a latent space representation with the effects of disease evolution in a very original way.
The simulated experiments are smartly selected and they show a proof of concept of the ability of the method to generate appropriate diseased image models. Indeed, the experiments in MRI and PET data show that the method is able to provide patterns typically seen in the real images.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
In my opinion, the method has not apparent weaknesses. To say something, I would like to read something about the use of interpretability with the parameters estimated by the method in a clinical application.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The datasets used for evaluation are publicly available, although the exact images used for training and testing are not provided.
The authors mentioned that the codes will be available upon acceptance. Otherwise, I would not feel able to reproduce the paper from scratch with the information provided in the manuscript.
The parameter values were provided.
For the evaluation, the authors provided a clear description of metrics.
Full memory footprint was not provided, although the size of the 3D images (80x96x80) makes me guess that the method is expensive.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
I don’t have much to say. The paper is very well written, the problem challenging and interesting, and the solution smart and appealing. I would like to read something about the use of interpretability with the parameters estimated by the method in a clinical application, maybe within the last line of the conclusion. I believe this is a great paper for Miccai.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
My recommendation is based on the difficult of the problem and the smart solution proposed by the authors. I believe this is a strong paper suitable for publication in Miccai. I expect that the authors follow on with this interesting research and move into the use of the latent space parameterization for prognosis and interpretability.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
After carefully reading the other reviewer’s opinions and the author’s rebuttal I see the point of Reviewer 3 and I do agree with him/her to some extent. However, the idea of combining mixed-effect models in the generative model is very interesting and it seems to provide coherent results. I would like to see this work published in Miccai so that the authors can extend their work with the reviewer’s suggestions, provide a fair theoretical and practical comparison with the state of the art, and build on the idea of interpretability. I am going to downgrade my assessment to accept since I think it is a more fair grade for this paper.
Review #3
- Please describe the contribution of the paper
The submission works on longitudinal image regression by integrating the variational autoencoder with mixed-effects model in the latent space. The method is evaluated on a synthetic dataset and the ADNI dataset.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Although the idea of mixing VAE with Euclidean regression models is not new, the proposed method is a good try to futher move this direction forward. The results on the synthetic data domenstrates the effectiveness of the proposed method to some extent.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The main weakness of this paper is its negligence of existing tranditional methods on image regression that works on the same problem and misses the comparsoin to traditional methods and the existing deep learning based methods.
For example, this sentence in the abract is misleading, “few progression models for entire medical images have been proposed that allow missing data imputation and prediction at any timepoint.” In the past decade, a bunch of researchers work on image regression for longitudinal/cross-sectional medical images and many papers (just google regression on image time series or image regression) were published and achieved good performance in interpreting and prediting the entire images, images with even much higher resoultions compared to the ones used in this manuscript.
The downsampled images used in the submission show missing details in many brain structures, which also indicates the high computational cost of the proposed method. This questions the motivation of this work. Since both the regression quality and computation cost are not improved by comparing to the traditional methods, what is the motivation for using this method? Just because it is a deep learning based method?
Also, the mixed-effects model in the latent space is linear, how about nonlinear changes? Should it be limited to an age range that the evolution is almost linear? Otherwise, how to handle the nonlinear case, which is very often in brain degeneration or disease evaluation?
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The reader probably has difficulty to reproduce this work, releasing the source code could be a big help on the reproducibility of this work.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Please check the weakness section. A thorough suvery of related work and the comparison to existing methods, both traditional and deep learning based methods are necessary for demonstrating the effectiveness of the proposed method in the work.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
A work has its merits but desiring more work to domenstrate or justify them.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper received a mixed review of positives and negatives. All reviewers agreed that (i) the problem of appropriately modeling disease progression is challenging, and (ii) the proposed idea of developing variational autoencoder with a linear mixed effect longitudinal model seems to be a right direction moving towards the solution. While the presented methodology is technically solid, it lacks a thorough review of literature and a comparison to other existing progression models (as suggested by R3). It would also be good for the author to clarify the motivation behind this work. The authors are encouraged to respond to all reviewers’ questions and concerns, particularly those from R3.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
4
Author Feedback
We thank the reviewers for their insightful comments. They pointed out that the manuscript is overall well written and stressed the main contribution which is to blend a mixed-effect framework with deep learning tools for dimensionality reduction, in order to provide interpretable disease progression models for imaging data.
There are questions, notably from R3, about the difference between our approach and existing literature on progression models for medical image time series. One important distinction is that regression tools for time-series usually model repeated measurements over time for one individual, while mixed-effects models in longitudinal studies analyse a set of patients with repeated measurements. Such longitudinal modeling allows one to estimate trajectories at the individual, that are then averaged at the population level. The average trajectory is thus informed by portions of individual trajectories at various disease stages. Our approach uniquely addresses this problem of estimating a distribution of trajectories combining sound statistical tools and the unsupervised feature extraction techniques of deep learning.
The first advantage is that the forecast of the progression of an unseen individual is informed by the other individuals in the training set. In [Bone et al, Prediction of the Progression of Subcortical Brain Structures in Alzheimer’s Disease from Baseline, MICCAI 2017, GRAIL workshop], authors report that time-series methods such as geodesic regression can suffer from limited prediction capabilities compared to a mixed-effect model.
The second advantage is the disentanglement of time-related changes from inter-patient variability. Our approach provides a normative scenario of temporal alterations, as well as a set of spatial generative factors, independant from time, that represent anatomical differences between individuals. This is demonstrated on the synthetic dataset, although admittedly harder to validate on medical data. The estimated individual parameters such as the onset age and pace of decline are of clinical importance as they position the individual’s trajectory of changes relatively to the training population. We will emphasise this interpretability asset in conclusion as suggested by R1.
We agree with R3 that the sentence he mentioned from the abstract is misleading. We will change it to better stress the difference with time-series modeling.
We would also like to point out that the sampled images in Fig. 3 and 4 come from average population trajectories. As is common in any atlasing method, one could not expect these images to be as sharp and detailed as true images, as they average anatomical details from different subjects. Reconstructed images are not shown but also have a higher degree of smoothing than the observations. We will make this point clearer in the captions of the figures and in the text.
The question on the limitations of linear modeling is legitimate. As mentioned in section 2.3, X can be seen as a Riemannian manifold, the metric of which is given by the pushforward of the Euclidean metric of Z, through the decoder. The generated images follow geodesics for that metric and exhibit dynamics that are not at all linear. The assumption here is that the decoder learns the metric (and hence the non-linearity) observed in the image space.
Regarding implementation, the MCMC estimation procedure is detailed in references [1,20]. We agree with the reviewers that the method might be hard to re-implement and will provide the commented source code upon acceptance of the paper. Computational burden is lower than for most morphometric methods and will be detailed.
To conclude, we will pay attention to better position our work regarding the state of the art by referencing image time-series regression methods, and to emphasize the interest of mixed-effects modeling. We will clarify the points about image smoothness, non-linearity and implementation details.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Overall, the authors have carefully addressed the main concerns of the reviewers, especially on clarifying the difference between the proposed approach and existing algorithms. With the authors’ commitment to incorporate all reviewers’ feedback in their revised manuscript, I recommend accepting this paper without further questions.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Reviewers are positive about the paper and mention generally positive and constructive points. Rebuttal is strong and convincing.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper makes a good contribution to a challenging problem of modeling longitudinal data. Although the results are somewhat lacking and the method is also shown to work on synthetic data, the methodological contribution of the paper is novel and is thus a good fit for MICCAI.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5