List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Haoran Dou, Nishant Ravikumar, Alejandro F. Frangi
Abstract
Generating virtual populations (VPs) of anatomy is essential for conducting in-silico trials of medical devices. Typically, the generated VP should capture sufficient variability while remaining plausible, and should reflect specific characteristics and patient demographics observed in real populations. It is desirable in several applications to synthesize VPs in a \textit{controlled} manner, where relevant covariates are used to conditionally synthesise virtual populations that fit specific target patient populations/characteristics. We propose to equip a conditional variational autoencoder (cVAE) with normalizing flows to boost the flexibility and complexity of the approximate posterior learned, leading to enhanced flexibility for controllable synthesis of VPs of anatomical structures. We demonstrate the performance of our conditional-flow VAE using a dataset of cardiac left ventricles acquired from 2360 patients, with associated demographic information and clinical measurements (used as covariates/conditioning information). The obtained results indicate the superiority of the proposed method for conditional synthesis of virtual populations of cardiac left ventricles relative to a cVAE. Conditional synthesis performance was assessed in terms of generalisation and specificity errors, and in terms of the ability to preserve clinical relevant biomarkers in the synthesised VPs, I.e. left ventricular blood pool and myocardial volume, relative to the observed real population.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43990-2_14
SharedIt: https://rdcu.be/dnwLo
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The authors used a conditional variational auto-encoder for controlled reconstruction of the left ventricle shape. Normalizing flow is used to transform the unimodal Gaussian prior to multimodal, increasing the variability of the reconstructions.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Reconstructing virtual anatomy is an interesting application and is relatively less investigated compared to other medical image computing tasks.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The structuring of the paper could be improved. An in-depth discussion of the methods and results is missing. The method (cVAE+ NF) is not sufficiently novel. The evaluation is also a bit flawed. See point 9 for detailed comments.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Information about model architecture and training procedures is provided.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
A discussion section (about the method and results) is missing. The authors could better utilize the space by removing/shortening some of the repetitive (e.g., ‘controllable synthesis of…’ is mentioned more times than necessary) or redundant paragraphs (e.g., introduction about the basics of VAEA, cVAE, NF in section 2. ). It is also helpful to provide a detailed derivation (or source) of the modified ELBO shown in equation 7 in the main manuscript or in an appendix. Furthermore, besides the intended use, could the method also be useful in data augmentation (e.g., generating synthetic training data)?
How is the method compared with voxel-based reconstruction (i.e., the ventricles are represented as 3D binary voxel grids instead of meshes)? A comparison with other approaches discussed in the Introduction section is also lacking.
Many of the important results shown in the supplementary video could have been incorporated and discussed in the main manuscript. Overall, the paper could improve a lot by carefully restructuring the content.
Some minor issues to correct: (1) Does the ‘vanilla cVAE’ mean a cVAE without normalizing flow? Please specify. (2) How many steps are used for normalizing flow? (3) Formatting issues: inconsistent use of indents in section ‘3 Experimental setup and Results’ (4) typos: clnical measurements – clinical measurements
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The application itself is interesting and important. However, the method used is not very novel and the evaluation procedure is a bit flawed (e.g., no comparison with state-of-the-art).
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The authors propose a conditional flow VAE for controlled synthesis of virtual population of cardiac anatomy. It converts the unimodal latent representation to multimodal distribution with added flexibility and control.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper is well written starting from the problem with current methods to proposed method
- The methodology section is well presented with appropriate block diagrams and derivations
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- How different are training of cVAE and cVAE with NF? Since cVAE introduces control over generation with some control signal, how different are the impact produced by cVAE with NF on actual IST?
- How beneficial would cVAE with NF be in comparison to cVAE assuming just a small boost in reconstruction error?
- What are the computational resource required for each methods?
- What would the representation look like if we choose other than normal distribution?
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The reproducibility criteria is well satisfied
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
See limitations
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Paper writing, experiments available and methodology
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This work focuses on controllable synthesis of virtual populations of anatomy. Firstly, the authors propose the use of normalizing flows to transform the latent variables from a simple unimodal distribution to a multi-modal one to capture a wider range of anatomical variability observed in the real population and generate more diverse virtual patients. They also introduced a conditioning mechanism to the flow-based VAE model that incorporates patient demographic data and clinical measurements. This approach encourages the synthesis of virtual patients that reflect the observed correlations between non-imaging patient information and anatomical characteristics in the real population.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Novelty of the method for the task of controllable synthesis of virtual populations of anatomy
- Comparison with other methods using meaningful metrics (sensitivity, variability)
- The paper is well written and easy to read
- Public and large dataset (UK Biobank)
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Lack of quantification of the difference between the distributions (Wasserstein distance ?) in Fig. 3
- To quantify the variability, why only use the LV volume and not other measure ? Myo Volume, Wall thickness or other measurement available in the UK biobank showcase.
- Fig. 4 not corresponding always to the described expected evolution such as “increasing the individual’s age results in a smaller BPVol, but increased MyoVol” or in the figure MyoVol (Age -10y) > MyoVol (GT) for male and female.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Clear description of the data processing and training strategy. Overall good reproducibility
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The supplementary is not mentioned in the paper or commented in the paper despite representing a good contribution. Also, an analysis on the right ventricle is made despite no mention or visualization in the paper.
In future work, a comparison of the experiment in the supplementary materials with the cVAE method will be valuable.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The novelty of the proposed method for controllable synthesis of virtual population is an interesting contribution. The paper present moderate weaknesses on the analysis of the results.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper propose a method for controlled synthesis of cardiac anatomy meshes (e.g., LV) conditioned on non-imaging covariates (e.g., age, weight, blood cholesterol). Technically, it combines a cVAE with a normalizing flow backbone for more flexible posterior construction. The paper received mixed reviews initially.
The major strengths are the application setup itself (generation of cardiac meshes conditioned on non-imaging covariates) that has not received much attention, the novelty of the method in this specific context (however, concerns about its general novelty exist; see below), and clarity of presentation. A major weakness as identified by R1, and which I also see, is the lack of general methodological novelty as combinations of NFs+VAEs have been proposed before for medical data. See, for example work on causal modeling by Pawlowski et al A and related papers such as B (both uncited). Additional important concerns raised are (1) the lack of comparison to other baselines than a PCA and vanilla (c)VAEs, (2) a lack of justification for the ELBO term used for optimization, (3) the fact that the presented method only marginally outperforms a cVAE while being less specific, (4) and some lack of clarity about the training process/architecture (e.g., number of flow blocks used - see R1; computational ressources needed - see R2) and the presentation of the results (e.g., Fig. 4 not showing what is expected - see R3).
For the rebuttal, the authors should specifically focus on the novelty aspect and place their work in the context of existing work on combining NFs and VAEs for medical data, clarify the ELBO term, and provide additional justification for the baselines chosen and why they are sufficient to showcase the superiority of the method. More clarity regarding the results reported would also strengthen the work (see detailed comments above).
Author Feedback
We thank the reviewers for their valuable feedback. Our responses are included below: Q1: Novelty (R1, Meta-R) A1: The primary contribution of our paper is to propose the first conditional flow VAE approach for controllable synthesis of the virtual populations of anatomy. Compared with the existing studies in this domain, the new equipped normalizing flow (NF) can boost the flexibility of the generative model, thus enabling diverse and plausible generations. In a more general perspective of this approach to medical data, the existing method, i.e., the causal shape model, only leveraged the conditional NF to embed the covariates (e.g., brain volume) for stabilizing training. Additionally, the causal relationships can be complex, unstable, and hard to identify when more covariates are involved. Conversely, our method focuses on improving the flexibility and capacity of the latent space to represent the shape geometry via NF, thus better improving the diversity of the generative shapes. Q2: Derivation of the modified ELBO (R1, Meta-R): A2: We are sorry for the insufficient details due to the limited space. The main difference between the modified ELBO and the original is the second item in equation 7, which computes the determinant of the Jacobian of the NF. Please refer to [3] for details and we will revise it in the future version.
[3] Rezende et al., Variational inference with normalizing flows, ICML 2015. Q3: Baseline chosen and their sufficiency (R1, R2, Meta-R): A3: The existing approaches discussed in Introduction are built on the cVAE with different covariates (ECG) and basic units in the network. Hence, comparison with the cVAE can also validate the effectiveness of our model to the existing approaches. The core challenge of virtual population modeling is how to generate diverse shapes while remaining plausible. Compared to the reconstructed error (quality), the diversity and plausibility of the virtual population significantly contribute to the accuracy of the follow-up virtual simulation in in-silico trials. Following Table 1 in the paper and A4 in the rebuttal, the comparison between cVAE-NF and cVAE shows essential improvements in the diversity and plausibility (29.91 > 28.39 in volume variability, and 11.06 < 19.04, 2.81 < 11.04 in the distances of the distributions). The results illustrate the NF can boost the flexibility of the cVAE to capture enough variability of the real population while remaining plausible. Q4: Clarity of results (R3, Meta-R) A4: We first calculate the difference between the distributions in Fig.3. The Wasserstein distances between cVAE and UKBB are 19.14 in males and 12.52 in females, while those for cVAE-NF to UKBB are 11.06 and 2.81, correspondingly. Additionally, we compute the variability of cVAE and cVAE-NF based on myocardium volume, which is 23.85 and 24.21. This shows the superiority of NF in improving the model’s generative variability and plausibility. Finally, we apologize for the misleading in Fig. 4. Here, we show the model could achieve controllable synthesis and subject-specific prognoses by decoding specific latent variables and manipulated covariates (e.g., age + 10yrs). Hence, the expectation can be seen by comparing manipulated and reconstructed shapes instead of ground truth. We acknowledge the limitation of the inconsistency between the reconstruction and ground truth regarding clinical measurements, which will be revised in the future version. Q5: Clarity about training process/architecture (R1, R2 Meta-R) A5: The step for the normalizing flows was set as 4. We evaluate the computational resource of the investigated methods (VAE, cVAE, VAE-NF, cVAE-NF) in terms of the number of parameters (426.4K, 612.8K, 426.6K, and 612.9K). Q6: More in-depth discussions (R1): A6: In the revised version, we will add more in-depth discussions based on the results illustrated in the supplemental materials and rebuttal.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors prepared a good rebuttal and discuss all the major points raised in my meta-review. However, after the rebuttal I still only see very, very limited novelty here. The novelty of the approach is limited to this highly specific application scenario (cardiac mesh generation) and the authors’ response to my questions regarding the differences to already available causal models involving combinations of VAEs and normalizing flows contains a lot of inaccuracies (e.g., “only leveraged the conditional NF to embed the covariates (e.g., brain volume) for stabilizing training”). The causal models cited in my meta-review can do exactly what the proposed approach is capable of doing while (in addition) being able to respect causal relationships between the variables and the normalizing flow is not only utilized to stabilize training but it is an essential part of the model to handle causal dependencies. The authors are right that it is hard to identify those causal dependencies, but the proposed model completely ignores the causal aspect and fully relies on correlations.
In summary, this is a borderline case for me with a rebuttal that fails to really address my concerns regarding the technical novelty of the paper. However, the paper does not contain any major flaws and all other major points were sufficiently addressed in the rebuttal. I am, therefore, leaning towards acceptance.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The original paper was received positively with a number of concerns on novelty, comparison with other SOTA methods and the ELBO term. The authors have satisfactorily addressed the issues as raised by the reviewers and I am glad to accept the paper for presentation at MICCAI.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Overall, the authors have carefully addressed the major questions and concerns raised by all reviewers, especially regarding the novelty of their work, the comparison of experimental results with existing approaches, and clarification of the described methodology. With the authors’ commitment to implementing all the revisions as discussed, the paper is recommended to be accepted at MICCAI.