List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Connor Elkhill, Scott LeBeau, Brooke French, Antonio R. Porras
Abstract
Quantitative evaluation of pediatric craniofacial anomalies relies on the accurate identification of anatomical landmarks and structures. While segmentation and landmark detection methods in standard clinical images are available in the literature, image-based methods are not directly applicable to 3D photogrammetry because of its unstructured nature consisting in variable numbers of vertices and polygons. In this work, we propose a graph-based convolutional neural network based on Chebyshev polynomials that exploits vertex coordinates, polygonal connectivity, and surface normal vectors to extract multi-resolution spatial features from the 3D photographs. We then aggregate them using a novel weighting scheme that accounts for local spatial resolution variability in the data. We also propose a new trainable regression scheme based on the probabilistic distances between each original vertex and the anatomical landmarks to calculate coordinates from the aggregated spatial features. This approach allows calculating accurate landmark coordinates without assuming correspondences with specific vertices in the original mesh. Our method achieved state-of-the-art landmark detection errors.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16437-8_55
SharedIt: https://rdcu.be/cVRuG
Link to the code repository
https://github.com/cuMIP/3dphoto
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
A graph NN based approach to locating landmarks on photogrammetric images. Takes account of specific nature of data and includes a regression method to deal with landmarks which don’t coincide with a mesh node.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Elegant approach giving SOTA performance (compared to various well known methods).
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
What is the inter- and intra-rater repeatability for the manual annotation? How does your method compare to that?
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Algorithm described in enough detail to reproduce it with some thought (and checking the references). No mention of a public version of the code/data in main paper.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Overall a sensible approach to an interesting problem, with evidence that it gives good performance. What is the required accuracy of the placement for clinical application, and does the system achieve that? Minor points: Section 2.2 Unnecessary precision. Sufficient to say the average number of nodes was 8000+/-4100 “unitary surface normal” -> “unit surface normal”
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Proposes a sensible solution to an interesting problem, and demonstrates that it works.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
The authors describe a novel method for automated landmark placement on 3D point clouds of faces that leverages a graph convolutional neural network architecture. The clinical motivation for automated landmark placement is for enabling automatic and reproducible analysis for 3D photogrammetric data. In and of itself, the spectral model approach used here is an adaptation of the well known Chebnet (Defferrard, M., Bresson, X., & Vandergheynst, P. (2016)). The authors demonstrate the landmark placement accuracy of their framework through an analysis of a large dataset (982) of 3D paediatric faces and they compare their approach, plus variants thereof, to Pointnet++. The results quantitatively indicate the improvement gains in landmark placement accuracy of their approach.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
In and of itself, the spectral model approach used here is an adaptation of the well known Chebnet (Defferrard, M., Bresson, X., & Vandergheynst, P. (2016)). The extension and therefore technical contributions, include a multi-resolution feature quantification approach leveraging different orders of Chebyshev polynomials; a mechanism for weighted spatial feature aggregation where the weighting is cognisant of the local density of vertex data; and a probabilistic regression framework for regressing landmark coordinates as a combination of graph node coordinates.
The method caters for both connected and unconnected nodes which makes their approach more applicable generally for automating point-based data analysis. Their probabilistic framework to regress landmark coordinates as a combination of graph node coordinates provides a general improvement in landmark placement accuracy that improves not only their approach but also Pointnet++.
The use of multi-resolution spatial feature calculation and aggregation schemes makes the proposed approach somewhat robust to local data density issues. Although this reviewer feels that perhaps this issue could be managed more cheaply with some mesh preprocessing (see below in weaknesses)
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
A weakness to the study, in this reviewer’s opinion, is the use of a single expert’s landmark placement as ground truth. Typically in morphometry studies, more than one expert’s landmarks are used to remove subjective bias and this maybe more important when the landmarks are to be used for training an algorithm.
It seems to this reviewer that isotropic vertex placement of the raw image data would reduce the need for the proposed multi resolution approach? Isotropic vertex placement regularises the distance between neighbouring nodes regardless of spatial location thus possibly enabling single order polynomial use. Given that the authors are already doing some preprocessing of the raw image data anyway (section 2.2), could this note gave been done within that step? Please note this is not a major limitation, rather an honest question.
As indicated above, technically, the proposed framework seems to be an extension of Chebnet, albeit with some important contributions to handle the peculiarities of automated 3D landmark location.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
There is a full description of the methods and data used indicating good attention to reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
There is no justification for the orders of polynomials used (k=3 and 7). Where these decided on empirically?
The authors may want to comment on the large standard deviations in their results on average landmark detection error.
Section 5 conclusion. There is a typo in the first sentence. “We presented a novel graph convolutional neural network to for the automatic identification..,”
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper indicated novel extensions of GCN to make it applicable for landmark detection in 3D facial photogrammetry. The paper is very well written, scores highly on clinical significance, has important contributions to the state of the art, and presents a balanced reporting of results.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
The rebuttal provided addressed two concerns from this reviewer which were single-expert “ground truth” annotations, and the possibility of employing mesh preprocessing to eliminate the need for the proposed multi-resolution approach.
For the former they pointed out that it was infeasible to have ground truth annotations from several observers which is understandable. The authors do include a statement addressing this limitation to their study.
For the latter concern the authors explanation that “variable point density between points in the 3D photogrammetry data allows for small geometric craniofacial features (e.g., canthi, tragi) to be properly represented in the surface meshes” is a good explanation justifying why they don’t do uniform mesh resampling.
Review #3
- Please describe the contribution of the paper
This paper introduces a graph-based convolutional neural network to pediatric craniofacial landmarks from 3D photographs (i.e., surface meshes). Three strategies are adopted in the proposed method: 1) Multi-resolution spatial features at every vertex are extracted with Chebyshev polynomials; 2) A novel weighting scheme dependent on the local data density at every surface location to aggregate the spatial features; 3) A new probabilistic regression framework that uses the aggregated spatial features to calculate landmark locations. The authors evaluated the proposed method by detecting 13 landmarks from a set of patients’ 3D craniofacial surfaces. Generally, the three strategies are actually not new, similar ideas are commonly applied by the related methods in the field of geometrical/point-cloud deep learning. In addition, the landmark detection accuracy of the proposed method is too low to meet clinical requirements.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The geometric deep learning was applied for craniofacial landmarks detection from 3D surfaces.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) The literature review is limited. There are a number of point-cloud deep learning methods [1], which are originally designed for the object detection from 3D point cloud, should can be directly applied for the task of this paper. For instance, VoxelNet [2] and PointRCNN [3] should be mentioned in the Introduction and compared in the Experiments. 2) The detection accuracy of this method is relatively low. The landmark localization error can be more than 10mm as reported in the Experiments section, such large errors should be not clinically satisfied. [1] Guo Y, et al. Deep learning for 3D point clouds: A survey. IEEE TPAMI. 2020, 43(12):4338-64. [2] Zhou Y, et al. VoxelNet: End-to-end learning for point cloud based 3D object detection. CVPR, 2018, pp. 4490-4499. [3] Shi S, et al. PointRCNN: 3D object proposal generation and detection from point cloud. CVPR, 2019, pp. 770-779.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The reproducibility of this paper is unclear without the released data and code.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1) More related works should be reviewed and compared. 2) The detection results should be assessed clinical experts to confirm the clinical feasibility of the proposed method. 3) Quantitative evaluation results should be provided.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1) The literature review is limited. There are a number of point-cloud deep learning methods [1], which are originally designed for the object detection from 3D point cloud, should can be directly applied for the task of this paper. For instance, VoxelNet [2] and PointRCNN [3] should be mentioned in the Introduction and compared in the Experiments. 2) The detection accuracy of this method is relatively low. The landmark localization error can be more than 10mm as reported in the Experiments section, such large errors should be not clinically satisfied. [1] Guo Y, et al. Deep learning for 3D point clouds: A survey. IEEE TPAMI. 2020, 43(12):4338-64. [2] Zhou Y, et al. VoxelNet: End-to-end learning for point cloud based 3D object detection. CVPR, 2018, pp. 4490-4499. [3] Shi S, et al. PointRCNN: 3D object proposal generation and detection from point cloud. CVPR, 2019, pp. 770-779.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
5
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This work applies a graph convolutional network to the problem of craniofacial landmark detection based on 3D photogrammetric scans of the face and head. Overall, reviewers are positive about the contribution, however, there are a few points that need to be addressed in a rebuttal:
-
It seems important in this application that several annotators perform landmark annotation to generate ground truth, due to some landmarks being hard to specify. This would also allow comparison of the localization error with inter- (and even intra- if possible) rater errors. This important aspect has to be discussed or at least stated as a limitation
-
It is unclear if the developed methodology using Chebyshev polynomial derived features overcomes mainly the fact that landmark vertices were placed naively, instead of e.g. in an isotropic manner.
-
the literature review seems to have limitations especially regarding aspects of recent point cloud processing methods
-
importantly, it is unclear if the large localization errors are actually clinically acceptable, and where these large errors are coming from
-
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
4
Author Feedback
We would like to thank the reviewers for their valuable feedback and the positive reception of our manuscript. Please, find our answers to the main comments received below.
Reviewers noted that method evaluation would be more robust using ground truth annotations from several observers. Although we agree that quantifying intra- and inter- annotator reliability would benefit our work, it is not feasible for us to have more experts annotating our large dataset of 3D photograms. As suggested, we will incorporate the following sentence as a limitation in section 4 of the final version of the manuscript “A limitation of this work is the use of a single expert to define the ground truth landmark locations”.
Reviewers suggested that uniform mesh resampling could eliminate the need for the proposed multi-resolution feature extraction approach. A variable point density between points in the 3D photogrammetry data allows for small geometric craniofacial features (e.g., canthi, tragi) to be properly represented in the surface meshes. If we resampled our meshes to be spatially uniform, we would need to substantially increase the number of nodes and edges to capture high resolution spatial features, which would affect the computational feasibility of our network. As noted in section 2.2 (page 3), our resampling approach was based on quadric error estimation to preserve the overall mesh appearance. This variable point density motivated our use of multi-resolution feature extraction with two different empirically determined K-order Chebyshev polynomials.
Reviewers suggested some limitations of the literature review, specifically on the aspects of recent advancements in point cloud processing methods. PointNet is the main reference in this domain and the foundation of many point cloud-based networks such as PointRCNN. Additionally, many of the available point cloud-based learning methods such as VoxelNet or PointRCNN are not directly applicable to our problem as they produce bounding boxes for object detection instead of sets of point coordinates. We chose PointNet++ for comparative reference because it is the extension of PointNet that has shown state-of-the-art performance in point cloud segmentation and is applicable to 3D photogrammetry data. In section 1 of the final version of the manuscript, we will clarify this and include a reference to the survey by Guo et al [1] in addition to the current references on graph-based learning, so readers can easily access a thorough review of point cloud-based learning methods.
Reviewers raised concerns about the variable landmark localization error. We reported our results in categories following the Bookstein typing system, which groups landmarks according to their anatomical definition. While our network performs best on Bookstein type I and II landmarks, whose anatomical definition is clear, we believe that it was meaningful to report the performance of our model on all landmark types. As discussed in section 4, Bookstein type III landmarks cannot be reliably placed in 3D photogrammetry because of the lack of clear anatomical definitions, and the high variability in the manual placement of these landmarks substantially affects the quantified error of our model. In Bookstein landmarks type I and II, our network provides an accuracy of 3.39 mm as reported in section 3.2. As we will indicate in section 4 of our final manuscript, this error is within the suggested clinically acceptable accuracy (< 4 mm) in facial images [2]. [1] Y. Guo, H. Wang, Q. Hu, H. Liu, L. Liu, and M. Bennamoun, “Deep Learning for 3D Point Clouds: A Survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 12, pp. 4338–4364, 2021, doi: 10.1109/TPAMI.2020.3005434. [2] Weining Yue, D. Yin, Chengjun Li, Guoping Wang, and Tianmin Xu, “Automated 2-D Cephalometric Analysis on X-ray Images by a Model-Based Approach,” IEEE Trans. Biomed. Eng., vol. 53, no. 8, pp. 1615–1623, Aug. 2006, doi: 10.1109/TBME.2006.876638.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The key contribution of this work is the graph convolutional network for craniofacial landmark detection. Reviewers agreed that this method has its merits and is of interest to the community.
Main concerns from the reviews were addressed in the author rebuttal. The lack of different annotators will be stated as a limitation in the final manuscript. Limitations regarding the literature review will be addressed. The discussion of the results will include mentioning of the clinical utility of the landmark localization method regarding its achieved error.
Overall, given such a minor revision, this meta reviewer thinks that the benefits of this work outweigh the weaknesses.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
8
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The technical contribution is a graph NN based approach for locating landmarks on photogrammetric images. The rebuttal addressed the concerns on accuracy for clinical usage and experiments/validation. I would recommend acceptance.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper presents an interesting methodological development with the use of graph-based network for landmark detection. The authors adequately adressed the comments in their rebuttal and it seems entirely feasible to integrate these points in the camera ready version which warrants a recommendation for acceptance.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
4