List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Nathan Lampen, Daeseung Kim, Xuanang Xu, Xi Fang, Jungwook Lee, Tianshu Kuang, Hannah H. Deng, Michael A. K. Liebschner, James J. Xia, Jaime Gateno, Pingkun Yan
Abstract
Accurate surgical planning for orthognathic surgical procedures requires biomechanical simulation of facial soft tissue changes. Simulations must be performed quickly and accurately to be useful in a clinical pipeline, and surgeons may try several iterations before arriving at an optimal surgical plan. The finite element method (FEM) is commonly used to perform biomechanical simulations. Previous studies divided FEM simulations into incremental steps to improve convergence and model accuracy. While incremental simulations are more realistic, they greatly increase FEM simulation time, preventing integration into clinical use. In an attempt to make simulations faster, deep learning (DL) models have been developed to replace FEM for biomechanical simulations. However, previous DL models are not designed to utilize temporal information in incremental simulations. In this study, we propose Spatiotemporal Incremental Mechanics Modeling (SIMM), a deep learning method that performs spatiotemporally-aware incremental simulations for mechanical modeling of soft tissues. Our method uses both spatial and temporal information by combining a spatial feature extractor with a temporal aggregation mechanism. We trained our network using incremental FEM simulations of 18 subjects from our repository. We compared SIMM to spatial-only incremental and single-step simulation approaches. Our results suggest that adding spatiotemporal information may improve the accuracy of incremental simulations compared to methods that use only spatial information.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43996-4_54
SharedIt: https://rdcu.be/dnwPY
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #5
- Please describe the contribution of the paper
• Authors propose a novel method called spatiotemporal incremental mechanics modeling (SIMM) for simulating facial deformation. • The authors attempt to obtain results of FEM simulation in real time using SIMM. In contrast to other ML (PINN, PointNet++, PhysGNN) approaches, SIMM method is based on the idea that facial deformation is not just a spatial process, but also a temporal one. • The SIMM method takes this into account by using a neural network to capture relationship between spatial and temporal information. This allows the SIMM method to simulate more realistic facial deformation than methods that only use spatial information (single step or incremental methods).
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
• The authors propose a new technique to perform physics-based simulations, finite element method, in real-time. • The introduction and discussion section are well explained and describe the contributions of the proposed within the context of state-of-art techniques. • The methods section along with the supplemental data helps the interested readers understand the ML approach.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
• It is unclear whether having greater number of sub-simulations for the SIMM method resulted in superior results for the work. It would be helpful to get clarification on this aspect using data titration experiments. • The MSE results of 0.42 vs 0.44 vs 0.47 (all in mm) of SIMM, single step and incremental, respectively, seemingly appear marginally different. That being said, the authors use Wilcoxon signed rank test to establish statistical significance of their results. Yet, the absolute difference of 0.44 vs 0.47 mm is greater than 0.42 vs 0.44 mm for single step vs incremental and SIMM vs single step, respectively, which results in lack of robust statistical test. Is it possible the non-parametric test such as Wilcoxon may not be ideal for this analysis? Was data normalization tested by the authors and were any other statistical tests conducted by the authors? • The practical use case of SIMM for surgical planning relative to single step method is also not as convincing given the ME difference is not large compared to ground truth FEM data.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Yes reproducibility report follows the text corresponding to the key points acknowledged by the authors.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The authors are suggested to provide justification for the following points: • It is unclear whether having greater number of sub-simulations for the SIMM method resulted in superior results for the work. It would be helpful to get clarification on this aspect using data titration experiments. • The MSE results of 0.42 vs 0.44 vs 0.47 (all in mm) of SIMM, single step and incremental, respectively, seemingly appear marginally different. That being said, the authors use Wilcoxon signed rank test to establish statistical significance of their results. Yet, the absolute difference of 0.44 vs 0.47 mm is greater than 0.42 vs 0.44 mm for single step vs incremental and SIMM vs single step, respectively, which results in lack of robust statistical test. Is it possible the non-parametric test such as Wilcoxon may not be ideal for this analysis? Was data normalization tested by the authors and were any other statistical tests conducted by the authors? • The practical use case of SIMM for surgical planning relative to single step method is also not as convincing given the ME difference is not large compared to ground truth FEM data.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The rigorous introduction, method and experiments of the study outweighs the weaknesses, which could be addressed with a paper revision.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #1
- Please describe the contribution of the paper
The authors train a GNN which uses temporal information to simulate facial deformation in incremental steps. A comparison is made to a method without temporal information and a method without incremental simulation, both of which are outperformed by the proposed method. An ablation study and cross-validation are performed on 18 simulated facial deformations.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Resulting method is a fast network that drastically reduces simulation time while having a high accuracy
- Very clear motivation, description and evluation of the method. Overall, the paper is very well written in my opinion
- Most figures are very clear and convey a lot of information without being too convoluted.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Although statistically significant, the method does not improve all that much over other DL-based methods, while taking slightly longer to calculate.
The paper suggests that the improvement in accuracy comes from the temporal component in the network, however the temporal network also has the most network weights and is thus the most powerful. Although it’s indeed likely, I believe that there is no proof that the additional accuracy really comes from the temporal aspect, it may just be a more powerful network that would also work on
Evaluation was only performed on the same kind of synthetic data set as was used for training, i.e. generalization to real data was not shown. However, given the nature of the problem, I believe this is acceptable.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Most of the important parameters are given and presented in a clear, straight-forward manner and the authors state that code will be released (in the reproducibility checklist). Data is not available, but the method can be reproduced.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Can you put the final errors of the method into perspective? How large is the target (ground truth) displacement, i.e. how much does the face deform overall?
The paper mentions the method being “explainable” several times, but doesn’t explain what this means exactly. How is the method “explainable”? I assume the authors mean that one can see/output the intermediate steps of the simulation, but it’s not quite clear from the paper. Maybe a sentence could be added which explicitly states how the method is “explainable”?
“For example, starting from timepoint 0, the first incremental timepoint with a maximum deformation exceeding the threshold would be included in the sub-simulation (Fig. S1).” Does this mean that the time that passes between two time steps can vary? If so, does the method improve if equidistant time steps are used instead? I believe for the Single-Step and Incremental method this should not make much of a difference, but with the temporal component of SIMM I’m not so sure… I find Fig. S1. not all that clear - the abscissa seems to be time and deformation, however I don’t think they are always exactly correlated?
“We found a d max of 0.5 achieved the best performance in the incremental method, although the best d max may change for different cases (Tab. S2).” Does this mean the test was performed only for the “Incremental” method? Why not the full SIMM method? Related to my previous point, I think this may matter most for SIMM. (Which model was used could also be added to the caption of Tab. S2).
“…faster than FEM, which can take several minutes to perform an incremental simulation (Tab. 1).” This is not shown in Tab. 1?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The method could be very interesting to the biomechanics community and it is presented in a very clear way that makes it possible for others to build upon.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #4
- Please describe the contribution of the paper
The paper presents a Spatiotemporal Incremental Mechanics Modeling (SIMM) approach to address the passive deformation of soft tissue in the context of orthognathic surgery planning. The proposed method combines a spatial feature extractor with a temporal aggregation mechanism to predict the outcome of the surgery in times inferior to alternative FEM SOTA methods, and with accuracy rates slightly superior to incremental methods that use only incremental spatial information or a single step simulation method.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The addressed problem is clinically relevant and the combined learning approach is well-suited to address this class of problems.
- Well-established evaluation approach with attention to parameter sensitivity.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The novelty of the method is modest, and the obtained accuracy is slightly superior to the baseline methods.
- The Euclidean distance quantitative metric alone is not sufficient to reflect a significant improvement in the prediction accuracy.
- The clinical transfer requires performing FEM-based simulations with all their accumulated numerical approximations and sensitivity to the volume elements and the 3D meshing technique.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Partially reproducible
- The 3D models dataset will not be available.
- It is mentioned in the submission summary that the code of training and validation will be available, adding this information in the paper would be appreciated.
- The networks architectures and hyper parameters are well described to allow the reimplementation.
- The reproducibility of the biomechanical simulations is based on the details from [6].
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The paper presents a Spatiotemporal Incremental Mechanics Modeling (SIMM) approach to address the passive deformation of soft tissue in the context of orthognathic surgery planning. The proposed method combines a spatial feature extractor with a temporal aggregation mechanism to predict the outcome of the surgery in times inferior to alternative FEM SOTA methods, and with accuracy rates slightly superior to incremental methods that use only incremental spatial information or a single step simulation method.
The paper is well-structured and written in a consistent manner, the supplementary materials, the FEM-based simulations which serve as a ground truth (Ref [6]) and Figure 2 allow a clear understanding of the learning approach. Whereas the proposed approach is a valid choice to address the problem, a better positioning wrt. related literature would be appreciated. For instance, “but these methods are limited to grid-like structures and cannot represent irregular meshes used inFEM [11, 13]” That is not accurate for the [11] where the method is able to capture deformations for hyperelastic materials with multi-resolution meshes. Further, please have a look at a related work using spatiotemporal CNNs in prediction of hyperplastic soft tissue deformation [Ref.1].
-
Page 2, “the main limitation of all the aforementioned DL methods is that they perform “single-step” simulations (Fig. 1), which are not ideal for modeling non-linear materials such as facial soft tissue, especially when large deformation is involved [6].” How much deformation magnitude is considered large within this study? It has been mentioned that “maximum deformation threshold d_max” which corresponds to the lowest error equals to 0.5 mm. What is the signification of this deformation? Which constituitve law (behaviour law) has been assigned to the facial soft tissue?
-
The deformation of the soft tissue is driven by the bones. Isn’t it necessary to handle the topological changes which might be caused during the surgery?
-
Figure 3, Could the authors clarify why the number of nodes in the same model is different across the three methods?
-
The Euclidean distance metric between the source and target 3D meshes is an meaningful metric. Nevertheless, additional metrics could be also interesting such as a semantic descriptor (like curvatures), or a metric that assigns more weight to specific regions (for example a weighted Euclidean distance with weights proportional to the magnitude of the nodes’ displacement fields). Last but not least, qualitative metrics set by the surgeons can be also used.
Minor:
- Please insert the references in an ascending reference id order within the text.
[Ref.1] Mohammad Karami, Hervé Lombaert, David Rivest-Hénault, Real-time simulation of viscoelastic tissue behavior with physics-guided deep learning, Computerized Medical Imaging and Graphics,104, 2023, 102165, ISSN 0895-6111, https://doi.org/10.1016/
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The novelty of the method is modest, and the obtained accuracy is very close to the baseline methods.
- The Euclidean distance quantitative metric alone is not sufficient to reflect a significant improvement in the prediction accuracy.
- The clinical transfer requires performing FEM-based simulations with all their accumulated numerical approximations and sensitivity to volume elements meshing.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Overall all the reviewers recommend acceptance. With R4 and R5 making many detailed suggestions for how to revise the manuscript to improve clarity.
Some suggestions for additional experiments to understand how sub-simulations and some questions about the validity of the statistical testing that should be address before publication.
Author Feedback
We are grateful to the reviewers and AC for the thorough and favorable evaluation of our work. We will address the specific suggestions in our camera-ready version. Brief responses are included here for clarification. The reviewers expressed concern regarding the significance of the improvement of SIMM over the alternative methods. In addition, although SIMM significantly improved the mean Euclidean distance error compared to the alternative methods, the reviewers suggested using only this quantitative metric may not highlight the improvement of SIMM. After further consideration, we agree with the reviewers that additional quantitative evaluations would be beneficial, such as accuracy within clinically defined regions of the face, error weighted by the magnitude of displacement, and accumulated error across incremental simulations. The reviewers also asked for clarification on the statistical tests performed. Statistical tests were performed on the mean Euclidean distance errors for all simulations between the tested methods. The distribution was first tested for normality using the Kolmogorov–Smirnov test. After confirming that the distributions were not normal, the Wilcoxon signed rank test was chosen to test for significance between the tested methods with p < 0.05. Reviewers 1 and 4 asked for clarification regarding the maximum deformation for a given simulation and what is regarded as “large” deformation. To address this, we will include details of the maximum deformation for each subject. For our purposes, we define “large” deformation as greater than 1 mm, which is what we consider the threshold for clinically acceptable error. Reviewer 1 asked to clarify what we consider an “explainable” model. For the purposes of this study, we define an explainable spatiotemporal model as one that learns temporal trends from already extracted spatial features. This eliminates the need for creating a separate temporal feature extractor, which may be difficult to elucidate how the network learns such features. We will add further explanation of what we mean by an “explainable” model in the camera-ready version. Reviewer 4 asked for additional details regarding the position of this work with respect to related literature, specifically previous works which incorporated irregular meshes and spatiotemporal learning. We will include additional explanation of the limitations of previous works and how our proposed work fits into this field, including discussion of the reference provided by the reviewer. Reviewer 4 asked about the constitutive law applied to the facial tissue. The constitutive law originates from the finite element simulations used in training, which is fully described in [6]. Put briefly, the finite element simulations assume neo-Hookean material properties with a Young’s modulus of 3,000 Pa and a Poisson’s ratio of 0.47. Reviewer 5 expressed concern regarding the number of sub-simulations used between methods. Both the SIMM and spatial-only incremental methods used the same number of sub-simulations and incremental steps for training. While the single-step method omitted the intermediate steps in the sub-simulations, we believe this is a tradeoff that is inherent to the single-step training scheme. Overall, we would like to thank the reviewers for their valuable insight and suggestions.