List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Hastings Greer, Lin Tian, Francois-Xavier Vialard, Roland Kwitt, Sylvain Bouix, Raul San Jose Estepar, Richard Rushmore, Marc Niethammer
Abstract
Inverse consistency is a desirable property for image registration. We propose a simple technique to make a neural registration network inverse consistent by construction, as a consequence of its structure, as long as it parameterizes its output transform by a Lie group. We extend this technique to multi-step neural registration by composing many such networks in a way that preserves inverse consistency. This multi-step approach also allows for inverse-consistent coarse to fine registration. We evaluate our technique on synthetic 2-D data and four 3-D
medical image registration tasks and obtain excellent registration accuracy while assuring inverse consistency.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43999-5_65
SharedIt: https://rdcu.be/dnwxi
Link to the code repository
https://github.com/uncbiag/ByConstructionICON
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
the paper presents a novel Lie-group based operator for multistep registration to create inverse consistent image registration. the operator generalises to deformable (non-rigid) image registration.
the paper presents the results on synthetic data set (MNIST) and on several publicly available medical imaging data sets (brain, lungs, knee) and compares against the most relevant methods for preserving the inverse consistency in the image registration. The presented method achieves satisfactory results
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The main contribution of this paper seems to be a novel operator to generate inverse consistent transformations for multistep deformable image registration. This is an elegant extension of direct regression methods that can estimate inverse consistent transformation in single step using e.g. stationary velocity field representation.
the method is well validated and show advantage when compared to the other relevant methods. The key measures for assessment of thee registration quality has been used and they prove the advantages of the new method.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Many image registration algorithms are estimating (or predicting) the transformation between moving and fixed images in single step (as direct regression), and so if the transformation is using e.g. stationary velocity field for fast inversion, the final estimated transformation is inverse consistent by the construction. Therefore, to some extend it is hard to find a medical application when the multistep inverse consistent image registration could be essential and have advantage over simple use of stationary velocity field as for e.g. proposed in Mok, T.C., Chung, A.C.: Fast symmetric diffeomorphic image registration with convolutional neural networks. In: Proceedings of the IEEE/CV
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
the data sets used for validation are publicly available
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Clarify whether COPDgene data set that has been used is 3D? (please add the size of the images to make description consistent with the other data sets)
fibial cartilage -> tibial cartilage (please correct)
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
An interesting paper because proposes a novel operator to create multistep inverse consistent image registration, and the main claim of the paper are proven. The potential limitation is clinical relevance - not well motivated in the paper
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
In this paper, the authors propose a novel strategy to constraint the inverse consistency of DNN-based registration algorithms. This technique is also extended to multi-step DNN-based registration, where different displacement fields are composed, as for instance in coarse-to-fin schemes. Compared with the nearest approach [10], the authors extend their strategy to simultaneously optimized coarse-to-fine approaches.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is relatively well written and the novelty of the contribution is clear.
The method seems efficient.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Some important parts of the methodology are not clear, in my opinion.
I have some doubts about the fact that the experimental protocol was properly addressed when testing the methods compared to the proposed one.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The tests appear to me as reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
At the beginning of Section 4, the claim « Unfortunately, TwoStep [7,23] is not always inverse consistent even with inverse consistent arguments. » is particularly strong but not supported by any discussion. The results of Table 1 also indicate that the inverse consistency is not purely respected using [23], but is far from being dramatic (up to half-voxel error). In this context, I would say that this claim is exaggerated.
Is Eq. (11) indirectly used as a regularization loss ? If yes, this should be made explicit.
In the subsection « Affine registration convergence », I would recommend the authors to give some intuitions of why they believe that their strategy speeds-up the convergence.
In Table 1, I’m surprised to see the very bad inverse consistency results obtained using ANTs-SyN on DirLab. This method is diffeomorphic, so computing its inverse should be accurate. Is it due to a poor interpolation strategy when composing different deformations at different resolutions?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper could be interesting but lacks maturity. I believe that the authors should provide more discussions and more convincing results in their paper. They could gain a significant amount of space, and probably more impact, by only focusing on non-rigid and diffeomorphic deformation models, since they are also motivated by invertibility concerns.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
4
- [Post rebuttal] Please justify your decision
I thank the authors for their answers and changed my recommendation for this paper. If the paper is accepted, I recommend the authors to temper or justify some of their claims and to give more insights about their method and results.
Review #5
- Please describe the contribution of the paper
Offers a constructive way to provide inverse consistency in a neural registration network. Offers multi-step composition that can combine modular and still maintain overall inverse consistency.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This is overall a well organized paper and presents clear challenge in preserving inverse consistency and the constructive rationale. The extension to multi-step via telescoping has very nice practical implications and has been illustrated in the numerical example.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
It is mostly presentation-wise. Some operators such as tilde was not defined before usage. It would also improve illustrative power Fig2 shows not just the confirmation that the proposed approach yields inverse consistency but also how the naive version fails (in that the concatenation of individually inverse consistent ones may turn out to be not) to numerically echo the statement after eqn. 5.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Good.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
It would also improve illustrative power Fig2 shows not just the confirmation that the proposed approach yields inverse consistency but also how the naive version fails (in that the concatenation of individually inverse consistent ones may turn out to be not) to numerically echo the statement after eqn. 5.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This is actually a relatively simple development and can be reasonably implemented. The construction is quite cleaver and addresses an important issue in ensuring inverse consistency when concatenation of transformation is to be used, which is quite common in Med Im registration.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper received a mixed review of both positive and negative feedback. While all reviewers acknowledged the novelty of the proposed idea, there were several major concerns raised by R1 & R3. During the rebuttal phase, the authors are strongly encouraged to address all questions and concerns raised by all reviewers, with particular emphasis on the following points: (i) Clarifying why the baseline algorithm, such as ANTs-SyN, achieves unexpectedly poor inverse consistency results, as highlighted by R3. (ii) Providing insights into why the proposed method gains benefits such as faster convergence, as pointed out by R3. (iii) Elaborating on the advantages of the proposed model (in any potential real-world medical applications) in comparison to existing methods, as mentioned by R1.
Author Feedback
R3 and R5 were concerned that we had not sufficiently argued that, for inverse consistent Φ and Ψ, TwoStep{Φ, Ψ} is not necessarily inverse consistent. We expand the proof of this result from the second paragraph of section 4. TwoStepΦ, Ψ ◦ TwoStepΦ, Ψ = Φ[IA, IB ] ◦ Ψ [IA ◦ Φ[IA, IB ], IB ] ◦ Φ[IB, IA ] ◦ Ψ [IB ◦ Φ[IB, IA ], IA ] = Φ[IA, IB ] ◦ (Ψ [IA ◦ Φ[IA, IB ], IB ]) ◦ Φ[IA, IB ]^-1 ◦ Ψ [IA, IB ◦ Φ[IB, IA ] ]^-1 And without further assumptions, this cannot be reduced to the identity map. This is in contrast to Eq. (10), which can be analytically reduced to the identity map (Eq. (11)).
In addition, we showed two pieces of numerical evidence that TwoStep does not preserve inverse consistency. First, this failure to preserve is demonstrated in Fig. 3, middle, dashed green line, which is far less inverse consistent than the other networks with inverse consistent components (green, dotted/solid). We cover this in our discussion (end of section 5).
Second, TwoStep’s failure to preserve inverse consistency is demonstrated by the ANTs SyN results. The SymmetricNormalization algorithm from Avants et al. [2008] is shipped in the ANTs library as the SyNOnly preset, which is not the default preset. The default, the SyN preset, which is also most commonly benchmarked, is actually TwoStep{Affine,SyNOnly}. As this two step procedure does not preserve inverse consistency, SyN is not inverse consistent. This is seen in Table 1, section HCP, where ANTs SyN has 50x larger inverse consistency error than ANTs SyNOnly.
R3 was concerned about ANTS SyN’s large inverse consistency error on COPDGene and OAI, but as explained above this is a correct measurement, and is explained by the library’s handling of multistep registration.
R1, R3, and R5 raised concerns about the clinical relevance of inverse consistency. Precise inverse consistency is needed for unbiased time series analysis, (Reuter 2010). Half a pixel of inverse consistency error is not insignificant in lung registration- the SOTA landmark mTRE (0.9 mm) is half a pixel at our resolution. Finally, inverse consistency eliminates the arbitrary choice of specifying a fixed and a moving image.
R3 asks what advantage we provide over SymNet. SymNet performs well on brain registration, but as presented is a fundamentally one step approach. Although we did not train SymNet on OAI or COPDGene, other approaches such as Voxelmorph or GradICON cannot register these datasets in one step, but succeed when made multistep (see Tables 1 and 2 in Tian 2022, where e.g. Voxelmorph goes from DICE of 46 to 66 on OAI when combined with affine preregistration via TwoStep)
Our TwoStepConsistent approach offers the potential to bring inverse consistency to powerful heterogeneous, multi-step registration pipelines such as the ANTs SyN preset. This justifies our choice to cover several transform representations (R3)
We answer R3 that Eq. (11) is not used as a loss - it is a property of our architecture and holds as long as the component networks are inverse consistent. In our work, the component networks are inverse consistent up to numerical error, so a loss is not needed.
We are also interested in the convergence question of R3. This is an open question and likely related to subtle differences in the optimization dynamics when training with SGD- we will investigate this effect in detail in future work.
R3 asks whether ANTS, which is diffeomorphic and thus invertible, is necessarily inverse consistent. A well designed diffeomorphic approach will be invertible, ie (Φ[IA, IB ])^-1 exists, but will not necessarily be inverse consistent such that ((Φ[IA, IB ])^-1 = Φ[IB, IA ]
We thank R1 for pointing out the fibial-tibial mistake, and we agree that the resolution and spacing of the processed 3-D COPDGene data (175 x 175 x 175, 2mm) should be added. We will correct these errors.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have addressed the majority of the major questions and concerns raised by the reviewers in their rebuttal. Nonetheless, this meta reviewer concurs with R3’s comments (after reviewing the rebuttal) about the necessity of providing additional insights into why the proposed TwoStepConsistent method yields improved transformation solutions. It is worth noting that this adjustment can be easily incorporated into the revised version and does not undermine the overall strength of the paper. As such, it is recommended that the paper be accepted to publish at MICCAI.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This is a very solid paper that received very good review scores and solves a relevant problem in (learning-based) medical image registration (inverse consistency). The scope of the experiments (four quite different but all challenging datasets) and the substantial improvements over simpler models (VoxelMorph) in particular for COPD-registration make this work stand out positively amongst other submissions. I fully recommend acceptance.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have tried to address the concerns raised by reviewers about comparison or analysis of existing approaches as well as significance of more accurate inverse consistent algorithm. In my opinion these are major concerns and authors might be correct in their answers but this needs re-review and further more detailed explanation in the paper.