List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Lin Tian, Yueh Z. Lee, Raúl San José Estépar, Marc Niethammer
Abstract
In this work we propose LiftReg, a 2D/3D deformable reg-
istration approach. LiftReg is a deep registration framework which is
trained using sets of digitally reconstructed radiographs (DRR) and com-
puted tomography (CT) image pairs. By using simulated training data,
LiftReg can use high-quality CT-CT image similarity measure, which
helps the network to learn a high-quality deformation space. To further
improve registration quality and to address the inherent depth ambi-
guities of very limited angle acquisitions, we propose to use the lifted
features extracted from the backprojected volume and a statistical de-
formation model. We test our approach on the DirLab lung registration
dataset and show that it outperforms the existing learning-based pair-
wise registration approach.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16446-0_20
SharedIt: https://rdcu.be/cVRS0
Link to the code repository
https://github.com/uncbiag/LiftReg
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes a new network for performing 2D/3D registration from limited view x-ray. Feasibility is demonstrated using CT + DRRs
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This is a very well written paper with some interesting ideas as well as good experimental methods. It does a good job explaining the overall reasoning and details of the methods and goes on to do a good amount of experiments, especially comparing to multiple prior works.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The main weakness of the paper is that the number of and variation of x-ray imaging angles is only briefly described in the introduction and figure 1, but not enough detail is given.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Code will be released upon acceptance, and DIRLAB is public data. In order to reproduce this network, the most difficult step will likely be the spatial warping transformer, so hopefully this will be included in the code release.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
I think this is a very interesting paper with not only novel methods but good experimental results. In the future, it would be good to have more of a description of real applications of this work.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Overall this paper has everything I look for in a good MICCAI paper: Important application, technical novelty, and most importantly good experimental design and results showing improvement over prior published methods.
- Number of papers in your stack
3
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
This paper provides a deformable 2D-3D registration algorithm based on an artificial intelligence algorithm capable of regressing deformation field vector between the source and the target images. A unique aspect of this algorithm is the back projection of 2D projections into a 3D space and regressing the transformation between the back-projected 3D space and the source 3D space. The performance of the developed method has been evaluated on two a publicly available dataset and according to target registration error and dice coefficient metrics. As a core contribution, the training loss is calculated in a 3D space allowing for better representation of the deformation parameters in all directions (specifically along the projection direction).
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The proposed network architecture operates based on first, “lifting” the 2D projections into a 3D space. This 3D space can be noted as a pseudo CT representation, whereby a deformation field is regressed between the lifted 3D space and the source 3D space. This problem formulation is unique and may help in alleviating the spatial ambiguity of traditional 2D-3D registration methods. The training loss is also an interesting point, where it combined a regularization term dedicated to the deformation field’s basis coefficients and the similarity between the warped and the original 3D volume. The provided evaluation study is feasible and compares the performance of this algorithm to the state-of-the-art techniques such as regnet. The paper is clear and follows a proper structure.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Although this work is presented as a 2D-3D registration paradigm, to the best of my comprehension, no evaluation is performed based on real 2D data. The evaluation dataset was in fact created following the same DRR generation framework used during the training process. This point undermines the clinical applicability of the developed algorithm. Furthermore, there appears to be a disconnect between the mathematical expressions and the illustration (e.g., Fig 1). Based on the provided evaluation study, it appears that the improvement made in the Dice coefficient metric compared to the existing networks (e.g., regnet) is very minimal and the authors’ justification for this is not sound. As shown in the ablation study, the back-projection (aka., lifting) process has minimal impact on the accuracy, this is not well justified in the manuscript.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The training datasets and the associated codes are disclosed in this work. The implemented code specific to this method however is not disclosed. The amount of implementation details provided in the main text and the supplementary materials is sufficient.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
An evaluation on real 2D X-ray images should be included to better motivate the clinical usability of this work. In the subsequent publications on this work, a comprehensive evaluation based on real clinical 2D data must be included. Another assumption made in this paper is the presence of X-ray calibration parameters for DRR generation purposes. This may not be a valid assumption in a clinical setting. The authors should amend the conclusion section to include aspects regarding the calibration parameters as well. In general, the illustrations are of poor quality and are not well connected to the mathematical expressions within the text.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This work provides a new perspective on deformable 2D-3D registration using artificial intelligence. The developed network architecture and the back-projection process is unique and of value for the community. No evaluation of real clinical data (specifically 2D X-rays) is provided.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
7
- [Post rebuttal] Please justify your decision
The authors’ response to the reviewers’ questions and concerns are appropriate. The argument about the use of synthetic data is not completely justifiable, but given the fact that the authors may have had limited time and resources for this conference publication, I can support the publication of this paper.
Review #3
- Please describe the contribution of the paper
The manuscript describes a method to predict deformation vector fields between a 3D source image and 2D limited-sampled projections.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The use of deep learning to solve the iterative 2D-3D registration problem.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The use of a population-based PCA model lacks support and validation. Different patients may have very different anatomy distributions and mechanics, which result in very different deformation patterns.
- The Lift3D module mainly serves to back-project. Why not use a standard back-projection layer that is differentiable? The stacking of 3D volumes from all angles lacks justification.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
NA
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The paper needs to justify the use of the PCA to serve a population-based deformation model. Also the use of Lift3D module needs to be justified and compared with standard 3D reconstruction layers. The paper should also make clear how the training/validation/testing division was done and whether they used different patients.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The overall study design lacks support, especially on using the PCA-based DVF model across the full population. The lift3D module also lacks justification.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
The authors explained the rationale of using a population-based PCA subspace, which however needs further studies on its validity and robustness.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
While two of the reviewers are very positive and highlight several commendable aspects of the presented work and I agree that the description is clear and sound, there are certainly shortcomings as highlighted by reviewer #3. I think the achieved accuracy of over 12mm TRE is not sufficient for practical use and the improvements over related work (and the ablation without Lift3d) is minimal. I understand that much information is lost during the projection, but even an affine registration on DirLAB-COPD achieves lower TREs. In addition reviewer #2 remarks on the potential problems when considering real instead of simulated projection X-rays. I would like the authors to give a better reasoning what additional information is gained by simply re-projecting the 2D image into 3D space, since this step does not seem to incorporate any additional anatomical knowledge from (annotate) training data.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
4
Author Feedback
We appreciate the reviewers’ comments and address the main concerns below.
Experimental Setup (R1/3): We use the sDCT geometry and generate 4 projections over a 30-degree angle. The dataset is split on the patient level. The training, validation, and test sets do not contain overlapping patients.
Validation of PCA model (R3,AC): Admittedly, PCA is a simple dimension reduction model and more flexible models (e.g., auto-encoders) may result in an improved subspace. We view our use of PCA only as a starting point and study whether one can learn a subspace representing intra-patient deformation which will be beneficial for 3D/2D registration. Sec. 4.3 shows that the PCA subspace can express most of the desired transformations. When applying PCA to unseen deformations (e.g.,COPDGene-val and DirLab), the landmarks error drops from 3.24mm to 5.34mm. This is significantly better than the landmark error achieved by 2D3D-regnet, the SOTA 2D/3D registration method (5.34mm vs 16.94mm). Hence, while the PCA approximation leads to a slight deterioration in performance it is expressive enough for our 3D/2D registration experiments.
Motivation behind Lift3D (R3,AC): It might be possible to use a 3D reconstruction instead of Lift3D. Note however that due to our limited angle (30 degrees) setup such a reconstruction will be blurry and will lack detail. Instead, Lift3D retains all the information in the projected images and allows the deep network to extract appropriate features. Though we currently only use the original 2D images as the 2D feature maps, other 2D features could easily be integrated into Lift3D, but not into a 3D reconstruction.
Registration Accuracy / SOTA (AC): While the concern of clinical applicability of our method at the current accuracy is valid we note that our method moves the accuracy of 2D/3D registration approaches in the right direction: our approach outperforms the current SOTA by ~25% which is substantial. We now also trained networks for 3D/3D Affine and 3D/2D Affine registration with landmark errors of 13.76mm (3D/3D) and 15.52mm (3D/2D) which is inferior to our LiftReg results (12.74mm): an 18% improvement wrt (3D/2D Affine) and still a 7% improvement compared to the significantly less challenging 3D/3D Affine registration, indicating that we are able to capture deformations beyond affine transformations.
Ablation Study (R2,AC): Tab. 2 shows that using Lift3D results in a 1mm(~7%) reduction in landmark errors and a 2%(~2.3%) improvement in DICE score over the baseline of not using Lift3D. Thus, compared with directly applying convolutional layers to the 2D features (our baseline) Lift3D indeed improves registration results.
Validation on synthetic data (R1/2,AC): Validating on real data is valuable and important. However, to evaluate on real data, one would need a way to obtain accurate 3D annotations (e.g, lung segmentations and landmarks) given only 2D projections of the lung, which is highly non trivial as the reconstructed volumes lack good depth resolution. Directly evaluating on real data would require having perfectly aligned CT and sDCT imaging systems and images (which is not feasible). Thus, we currently evaluate in an idealized scenario without scattering effects, beam hardening, and veiling glare and with precomputed calibration parameters (as pointed out by R2).
Summary: We address limited angle 3D/2D registration in two ways: (1) by introducing prior information via a subspace learned from 3D/3D deformations and (2) by lifting the 2D features to 3D space wrt the acquisition geometry. Our method achieves a ~25% improvement for landmark errors over the SOTA. We hope our proposed method can be a starting point for future work to further improve 3D/2D registration. Our software will be open source.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
I appreciate the responses during rebuttal, but remain slightly cautious about the usefulness and technical maturity of the proposed method. The resulting alignment is just minimally (~7%) better than comparison experiments and further research is definitely necessary to fully employ the projection information for 2D/3D registration. Nevertheless I agree that the paper nicely written and interesting for an in-depth discussion at the conference.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
7
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
LiftReg: Limited Angle 2D/3D Deformable Registration
This submission tackles the 2D/3D registration problem by uplifting 2D content into a 3D space. The originality resides in proposing a Lift3D network module, which allows for better image feature to be uplifted than plain backprojection.
The work has scientific merit in addressing the backprojection from 2D to 3D and rely on a reduced space of deformations fopr regularization. The main concern was that the proposed contributions (2D/3D) wasn’t directly evaluated in real dataset. Instead, overall 3D/3D Dice metrics were performed with 25% improvements from previous methods and an ablation study on simulated 2D data. It is indeed unfortunate that evaluation wasn’t performed on real 2D/3D experiments. The justification that no 2D/3D experiment could be feasible, is indeed deceiving and questionable. The method and results show, however, that this approach may go towards the right direction. The use of PCA may further help in regularizing the plausible deformations. The rebuttal may therefore be considered satisfactory.
For all these reasons, recommendation is towards Acceptance.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
10
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Challenging application and interesting contribution with demonstrated efficacy.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
NR