List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Hong Xu, Shireen Y. Elhabian
Abstract
Statistical shape modeling (SSM) is an essential tool for analyzing variations in anatomical morphology. In a typical SSM pipeline, 3D anatomical images, gone through segmentation and rigid registration, are represented using lower-dimensional shape features, on which statistical analysis can be performed. Various methods for constructing compact shape representations have been proposed, but they involve laborious and costly steps. We propose Image2SSM, a novel deep-learning-based approach for SSM that leverages image-segmentation pairs to learn a radial-basis-function (RBF)-based representation of shapes directly from images. This RBF-based shape representation offers a rich self-supervised signal for the network to estimate a continuous, yet compact representation of the underlying surface that can adapt to complex geometries in a data-driven manner. Image2SSM can characterize populations of biological structures of interest by constructing statistical landmark-based shape models of ensembles of anatomical shapes while requiring minimal parameter tuning and no user assistance. Once trained, Image2SSM can be used to infer low-dimensional shape representations from new unsegmented images, paving the way toward scalable approaches for SSM, especially when dealing with large cohorts. Experiments on synthetic and
real datasets show the efficacy of the proposed method compared to the
state-of-art correspondence-based method for SSM.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43907-0_49
SharedIt: https://rdcu.be/dnwdv
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #4
- Please describe the contribution of the paper
The paper introduces Image2SSM, a deep-learning-based approach for statistical shape modeling (SSM), which learns radial-basis-function (RBF) representations directly from images. This method enables characterizing biological structures with minimal parameter tuning and no user assistance, outperforming state-of-the-art correspondence-based SSM methods. Image2SSM paves the way for scalable SSM approaches, especially for large cohorts.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1- The paper shows an innovative loss function that ensures the consistency of many components therefore improving the overall results.
2- Each loss function is clearly detailed and justified for the use in this paper which makes it clear to understand.
3- Figure 1 standout and it is easy to understand and follow. - Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1- Comparison figures could have made their way into the paper instead of just being in the supplementary material, these results are very important to be in the main paper. 2- Figure1 seems a bit blurry from esthetic perspective.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper doesn’t seems to be easily reproducible at this stage, details of the model are missing.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The authors could have made several ablatio studies to showcase the importance of each term of the loss function. The authors could have swapped figures from the supp material to the main paper for clearer results. The authors could have shown other shapes for more diverse showcase of the results, some readers might be thinking that showing only this shape might be suspicious, where authors show cherry-picked results.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The loss function is wwhat makes the paper stand out, it covers many issues within the problem to be solved, the paper is well organized and discussed well.
- Reviewer confidence
Not confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper presents the Image2SSM approach that automatically extracts the surface of a target structure in a 3D image. The surface is expressed using a Radial Basis Function (RBF) representation, parameterized with control points and normals.
To compute the RBF parameters, a deep learning framework is trained with a series of 3D images and segmentations of the target structure. More specifically, the 3D image (=a training sample) is input to a series of convolutional blocks that output a flattened vector of the RBF parameters, and whose block parameters are optimized with the minimization of a loss function. The loss function measures the closeness of the RBF-based surface to the segmentation boundary, the similarity of its normals, the point sampling and finally the correspondence of the RBF control points across the training samples.
This last property allows after training to build a statistical shape model (SSM) from control points in correspondence across training shapes.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The CNN-like architecture proposed by the authors coupled with the RBF representation has the merit of being automatic and has a dual utility: 1) to support the construction of SSM at its training stage and 2) to perform segmentation of structures at is inference stage.
The automatic construction of SSM tackling the challenging issue of point correspondence is always a useful contribution.
Another advantage of creating shapes with known point correspondence at the inference stage is that any training shape can be decorated in advance with anatomical knowledge and this prior information will be naturally transferred to the segmented shape thanks to the correspondence property.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The paper suffers from clarity such as various typo errors or undefined parameters in various equations hindering the paper readability (see specific comments later on).
But the major weakness is in my opinion related to the evaluation of the proposed approach.
When comparing methods producing SSM, a common approach is to rely on metrics of generalization ability, specificity and compactness [1]. These metrics involve calculations similar to the experiments performed by the authors, so it should not be difficult to implement and could strengthen the evaluation in future work. Some authors claimed that SSMs should not be compared using Davies’ metrics but they should be instead evaluated in specific applications such as image segmentation [2], as comparing modes of variation brings little information out of a practical (clinical) context of shape analysis.
The comparison of image2SSM vs. DeepSSM in the segmentation context is hence a good thing but I found that the experiments in the paper were not sufficiently informative (see detailed comments).
[1] Davies, R.H., 2002. Learning shape: optimal models for analysing shape variability. Ph.D. Thesis, University of Manchester. [2] T. Heimann and H.-P. Meinzer, “Statistical shape models for 3D medical image Segmentation: A review,” Med Image Anal, vol. 13, no. 4, pp. 543–563, 2009.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Various entries in the reproducibility form are not really reflected in the paper, I wonder if this is related to the fact that authors claimed that code and data will be publicly shared (which is always very appreciated) and as a result little information has been given in the paper. For instance:
A clear declaration of what software framework and version you used. [Yes] Whether ethics approval was necessary for the data. [Yes]
The exact number of training and evaluation runs. [Yes]this information is nowhere to be found.
A lot of key information is not present in the paper and I think that a minimum of it could have been added, regardless of the anonymity status or the fact that authors plan to share data and code (probably on some website).
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Introduction:
Authors wrote: “This method uses accelerated computational resources to perform training and outperforms existing deep learning based methods that constructs PDMs from unsegmented images.”. The experiments however do not support this claim as no figures are provided (e.g. computation time, speedup factor, etc.).
Methodology:
Equations should be written with more attention, especially for new readers less familiar with the topic:
- In Eq (1), while I can guess why the coeff. 3 is present (1 control point and 2 dipoles), in the summation index j cannot reach 3J as there are J points.
- In Eq (4), if I am not mistaken variable M has never been defined, from my understanding M is actually K. Here once again the coeff 3 is not justified except if mu is a mean vector that includes also the dipoles in which case it should P^tilde instead of P.
The fact that a coarse initialization is necessary between all training samples (last sentence of manuscript) should also be brought up in the methodology as this introduces the non-trivial yet well studied issue of co-registration.
Experiments:
Please provide minimum implementation information, even if it can be found in the supplementary material like “The hyperparameters we use for Image2SSM are α= 1e2, β=1e2, γ=1e4, and ζ=1e3 for femurs”. Provide the framework used (pytorch, tflow, etc.) and some optimization information (optimizer, learning rate, number of epochs, etc.). I was also wondering how complex computations were made differentiable before noticing in Fig. 2 that some blocks had a “detached gradients” mention implying that they were not updated in the backprop phase, this should be clarified in the text.
Most importantly, provide basic information on the dataset used for testing such as how many samples were tested. Please also provide values as mean+/-std in addition to graphs (in text or figure captions) as it is a little difficult to estimate them from the figures and this information could ease the comparison between methods.
From a general viewpoint, you should comment the results, to convince the reader that your approach is superior to referenced works such as DeepSSM. I would remove one picture and replace it with text/table/comments, etc.
Specific comments:
- 3.1 Datasets: clarify that you are dealing with proximal femurs instead of femurs.
- Fig. 2 remove in Fig. the arrow from P^tilde_i and the normal loss, I think the normal loss just requires the left and top input.
- Fig. 4c. To what the two femurs on the right part of the figure correspond? (not the zoomed versions).
- conclusion: first sentence “….and predicts SSM from unseen images”. SSM is incorrect here, I guess you wanted to say something like PDM or shape, etc.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Despite the mentioned weaknesses of the paper, I found the Image2SSM framework interesting and with conceptual merit.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This work proposes a method to create sets of corresponding 3D points and surface normals from roughly aligned segmentation masks and images for statistical shape modelling. During inference, only the input image is needed. The method establishes correspondence over different shapes represented by Radial Basis Function control points and using a neural network. Evaluation has been performed using 50 femur shapes and comparing against a state of the art correspondence establishment method.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- the methodology is sound and results indicate decent performance on par with (11)
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- the method has been evaluated on 50 femur datasets only (the supplementary material contains results of 1018 left atrium MRI, though). The shape variability of the femur and left atrium are not very large compared to other anatomical structures and it is unclear how the method would generalise.
- the evaluation is missing standard statistical shape modelling metrics such as model generalisability, compactness, specificity
- the approach has only been compared to one other correspondence establishment method and the results are comparable, but not showing major improvements
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Details should be sufficient for reproduction.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- Balancing 4 loss terms is usually quite challenging. It would be great to know more about how the weighting parameters influence the results and how big of an impact each parameter has
- the evaluation would greatly benefit from a more detailed analysis with more test data and using standard statistical shape modelling metrics like the ones mentioned above. That would help to better compare the results to other state of the art methods. The proposed method is also only compared to one other method with results being on par so it is difficult to judge whether there are substantial benefits from the proposed approach
- a discussion about the computational complexity of the method and particularly the computation of the losses would be helpful
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Optimising radial basis function control points in a NN for correspondence establishment is an interesting concept and leads to decent results in the presented evaluation. However, the evaluation does not present a strong case of highlighting the advantages over existing methods like (5/11) and is lacking of further comparisons to state of the art.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Reviewers were enthusiastic about the methodology and its novelty. The loss function was seen as novel. The idea of predicting the radial-basis functional representations for point distribution models is innovative. However, there were several comments about the technical writing of the paper. Equations had undefined parameters, concepts were not properly explained, and there were typos. The authors are asked to particularly look at detailed comments from the reviewers that bring up all these points.
The results were convincing, but lacked comparisons to existing methods. While the first part of the method (Image to RBF prediction) is quite novel, and no other suitable methods may exist for comparison to my knowledge, the outputs of the network could indeed be compared to classical segmentation-based and shape extraction and representation (PDM) approaches. The authors do compare their results with DeepSSM, however that is another neural network-based approach. Since the method is quite novel, it is easier for the readers to understand classical-based comparisons, especially as detailed parametric representations are involved. An important ingredient for the final downstream SSM construction is the issue of correspondence. This is cleverly taken care of by the correspondence term (Equation 4).
The terminology of SSM is used rather haphazardly. SSM usually referes to the statistical representations on point distributions, thus PDMs can be thought of as SSMs. I presume, the authors understand this distinction, however, that is sometimes not reflected in the paper. One reviewer also brings this point up, for e.g. the line “…predicts SSM from unseen images”. It should be point distributions or shapes. In that regard, the SSM is simply a few extra computed steps on the points. Thus should the paper be re-titled to say something simpler (point cloud models) instead of SSMs, as this may me misleading to the reader?
The authors should carefully go through all the reviewer’s comments and concerns, especially about results, and providing quantitative information in addition to graphs/charts so the performance can be accurately evaluated. However, overall, the paper presents interesting ideas.
Author Feedback
We thank the reviewers for their generous and informative comments on the manuscript. In light of these, we present a detailed plan to address the issues raised to further enhance the quality and clarity of our work.
The most important aspect that was touched upon is the lack of generalizability, specificity and compactness metrics in the results [Davies]. We observed these values in our experiments and they were comparable though different from the approaches we compared against. Along with space limitations, given that some authors claim that SSMs should be evaluated using image segmentation metrics [Heimann] instead of Davies’ metrics, these were initially omitted. Instead, we favored the two-way surface-to-surface distance between the reconstructed and manually segmented shapes, which is a metric of segmentation quality. For completeness, Davies’ metrics will be added to the main manuscript by compacting some graphs into tables following the given suggestions.
One misunderstanding was that the approach was only compared to another deep learning tool in DeepSSM [5] and was not compared with state-of-the-art particle-based shape modeling tools. However, both the “Statistical Shape Model” section and figure 3 contain comparisons to ShapeWorks [11], the premier particle-based shape modeling tool at the time of writing. In fact, Image2SSM is an adequate tool to compute both classical segmentation-based shape extraction PDM as well as image to PDM. If space limitations permit, we will reformat the comparison to the classical ShapeWorks approach to be more visible in the camera-ready.
The computation time of Image2SSM is slightly higher for fewer shapes, but scales very favorably, achieving close to 2X speed up for >1000 shapes. The specific times were excluded in the manuscript because experiments ran in different machines, but will be included in the camera-ready.
One criticism was made on the fact that roughly registered images are necessary to run Image2SSM. This shortcoming is shared with DeepSSM and can be resolved using automatic 3D image registration approaches (e.g., using rigit registation tools in the ANTs toolkit, https://github.com/ANTsX/ANTs ).
All other minor comments and typos will be addressed to improve the readability of the paper.
Although it was pointed out that our approach performs on par with the state-of-the-art without having significant improvements, there are implicit advantages to our approach in relative terms. In particular, our approach performs similarly with the advantages of (a) circumventing iterative optimization and data preprocessing required by conventional PDM approaches, (b) mitigating the need to establish SSM (with its associated challenges) for training DeepSSM, (c) having an implicit notion of the surface that can be used in statistical analysis, (d) representing shapes well with fewer descriptors, and (e) having only 4 robust hyperparameters as opposed to 10+.
All in all, we view this work as the first iteration in a new family of PDM tools given that it performs on par with current tools despite its simple implementation while providing new features and extensions. We believe the comments and suggestions provided will help enhance the evaluation of this and future iterations of Image2SSM, paving the way for higher quality, more precise and more user friendly PDM tools in the future.
[Davies] Davies, R.H., 2002. Learning shape: optimal models for analysing shape variability. Ph.D. Thesis, University of Manchester.
[Heimann] T. Heimann and H.-P. Meinzer, “Statistical shape models for 3D medical image Segmentation: A review,” Med Image Anal, vol. 13, no. 4, pp. 543–563, 2009.