List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Zi Li, Lin Tian, Tony C. W. Mok, Xiaoyu Bai, Puyang Wang, Jia Ge, Jingren Zhou, Le Lu, Xianghua Ye, Ke Yan, Dakai Jin
Abstract
Estimating displacement vector field via a cost volume computed in the feature space has shown great success in image registration, but it suffers excessive computation burdens. Moreover, existing feature descriptors only extract local features incapable of representing the global semantic information, which is especially important for solving large transformations.
To address the discussed issues, we propose SAMConvex, a fast coarse-to-fine discrete optimization method for CT registration that includes a decoupled convex optimization procedure to obtain deformation fields based on a self-supervised anatomical embedding (SAM) feature extractor that captures both local and global information.
To be specific, SAMConvex extracts per-voxel features and builds 6D correlation volumes based on SAM features, and iteratively updates a flow field by performing lookups on the correlation volumes with a coarse-to-fine scheme.
SAMConvex outperforms the state-of-the-art learning-based methods and optimization-based methods over two inter-patient registration datasets (Abdomen CT and HeadNeck CT) and one intra-patient registration dataset (Lung CT). Moreover, as an optimization-based method, SAMConvex only takes $\sim2$s ($\sim5s$ with instance optimization) for one paired images.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43999-5_53
SharedIt: https://rdcu.be/dnww6
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper combines the SAM (self-supervised anatomical embedding) enhanced registration, variable splitting, and a coarse-to-fine optimization strategy to achieve mono-modal image registration.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The evaluation of the work is complete, which include three datasets and five competing methods.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The novelty of the method is limited. The method is based primarily on SAM [23]. The contributions claimed by the authors were already proposed by SAME [13].
What does SAM-based instance optimization mean? There is no description on this in the paper.
Why are there training data for Abdomen and HeadNeck datasets but not for Lung? What training process are these training data involved? The proposed method seems like an iterative approach without training.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The reproducibility of the paper is satisfactory. The paper includes complete description on datasets and implementation details.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Missing definition: what does the notation d in Table 3 mean?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The evaluation of the paper is strong. However, the proposed method lacks novelty, and some technical details are missing in the paper.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
4
- [Post rebuttal] Please justify your decision
Though the authors have addressed my concerns, I still think the lack of novelty of this methodological paper is a major weakness.
Review #2
- Please describe the contribution of the paper
In this paper, the authors utilize a pretrained SAM model to build 3-level 6D correlation volumes, and optimize the deformation field from coarse to fine in feature space for high-performance optimization-based registration. Results on 3 public datasets demonstrate the effectiveness of the proposed method.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This paper is well-written, and the motivation is clear.
- The results look good and have clear advantages over learning-based methods
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The novelty of this paper is a little weak. The Self-supervised Anatomical eMbedding(SAM) [1] and coarse-to-fine[2,3,4] optimization are both existing technologies, the author just optimize the deformation field from coarse to fine in the SAM’s feature space.
- What datasets are the SAM model pre-trained on? Are all 3 CT datasets in this paper used to pre-train the SAM model?
- Is instance optimization based on the highest resolution feature? Please detail this part.
- The hyperparameter (e.g., the searching radius N) should be detailed in the section of Implementation Detail to ensure reproducibility.
- The statistical analysis is necessary to demonstrate the effectiveness of the method.
[1]Yan K, Cai J, Jin D, et al. SAM: Self-supervised learning of pixel-wise anatomical embeddings in radiological images[J]. IEEE Transactions on Medical Imaging, 2022, 41(10): 2658-2669. [2]Mok T C W, Chung A C S. Large deformation diffeomorphic image registration with Laplacian pyramid networks[C]//Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part III 23. Springer International Publishing, 2020: 211-221. [3]Lv J, Wang Z, Shi H, et al. Joint progressive and coarse-to-fine registration of brain MRI via deformation field integration and non-rigid feature fusion[J]. IEEE Transactions on Medical Imaging, 2022, 41(10): 2788-2802. [4]Sun D, Yang X, Liu M Y, et al. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 8934-8943.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Lacking sufficient implementation/code details. The authors could make their source code publicly available if this paper is accepted.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Please address the issues mentioned in “Weaknesses of the paper”.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This is a well-written paper with a clear motivation. The idea seems interesting and the results is promising. However, there are some major issues for the authors to address during the rebuttal phase.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
My concerns have been addressed, and I will keep my original score.
Review #3
- Please describe the contribution of the paper
This paper proposed a deformable registration algorithm that is based on the optimization of feature vector matching result. The method SAM for feature extraction, and divided the optimization into a multi-resolution pyramid process. This registration method is innately explainable and stable.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
The proposed method is a good combination of past milestones in the field of deformable registration. SAM is a strong feature encoder, the pyramid (multi-resolution) process has been proven effective, and the convex decoupled global optimization for deformable registration is also renowned in the field.
-
Image registration is fundamentally the matching of anatomical structures. The proposed method perfectly fits this paradigm and is therefore very explanable and trustworthy.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
The novelty of this paper isn’t very strong. As mentioned in the main strengths, most of the proposed framework comes from previous works. The assembly of these methods is the main novelty.
-
The figures in this paper aren’t very clear. The Fig. 1 that illustrates the overall pipeline does not clearly show in the input and output of the left half,and how it’s connected to the right half (which is relatively clearer). Fig 3 is too subtle to read, and thereby fails to provide an easy way for readers to compare the result with ground truth.
-
The difference between this method and ConvexAdam is not clearly stated. Since the same optimization method is used, how is ConvexAdam different from the proposed method except for feature extraction?
-
In the original convex optimization paper [1], the authors argued that the coarse-to-fine registration workflow is suboptimal, and their convex optimization was partially designed to overcome that limitation. In this work, the authors are using that same convex optimization for for coarse-to-fine registration. There appears to be a logical gap/flaw in this connection.
[1] Steinbrücker, Frank, Thomas Pock, and Daniel Cremers. “Large displacement optical flow computation withoutwarping.” 2009 IEEE 12th International Conference on Computer Vision. IEEE, 2009.
-
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors agreed to share their code.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
As mentioned above in the main weaknesses:
- redesign the figures for better clarity
- thoroughly explain the difference between this work and some benchmarking methods, especially ConvexAdam
- SAM is not the only image feature encoding method. Try experimenting with others and benchmark.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
All the flaws I recognized and listed does not affect the impact of this algorithm to the field. This is still an effective registration method that deserves publicization.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
The author addressed my doubts well, especially the explanation on the coarse-to-fine strategy and distinguishing this work from SAME.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper received mixed comments. Reviewers acknowledge the methodological design of the method and the evaluation setting to prove its soundness. However, all reviewers raised concerns about limited novelty [R1, R2, R3], missing details [R1, R2, R3] and, missing statistical analysis [R2]. Thus, the authors are invited for a rebuttal to address the reviewer’s concerns.
Author Feedback
We thank all reviewers for their thoughtful comments and categorize the main comments (C) followed by responses (R) as follows:
Reviewer 1, 2 and 3. C: Novelty and contribution. R: 1) To the best of our knowledge, this is the first registration work exploring convex optimization with self-supervised learning-based features. Although SAME enhances learning-based registration with SAM (Reviewer 1), our method circumvents the tedious training process and achieves a superior registration performance over SAME on 3 diverse tasks. 2) We incorporate a coarse-to-fine strategy to convex optimization to alleviate the intensive computation burdens of cost volume in existing works. 3) We carry out comprehensive experiments to verify the robustness of the SAM feature against large geometrical transformations and demonstrate our method achieves state-of-the-art accuracy and good generalizability in diverse registration tasks. We believe these contributions are of great interest to the MICCAI community.
Reviewer 1, 2. C #1: SAM-based instance optimization. R: It solves an instance-specific optimization problem (Balakrishnan, et al., TMI 2019) in SAM feature space. The optimization objective consists of similarity (dot product between SAM feature vectors on the highest resolution) and diffusion regularization terms.
C #2: The definition of d and the detail of searching radius N. R: As mentioned in the last paragraph of Sec 2.3., d refers to the voxel displacement within the neighborhood. d=2N+1 and we will make this clear in the final version. We have shown the ablation study on d, namely N, in Table 3. N is set to [2,3,3] in three levels, respectively.
Reviewer 1. C: Training data for Abdomen and HeadNeck but not for Lung. R: Data split for Abdomen and HeadNeck is to compare with learning-based methods. Lung dataset contains 35 CT pairs, which is not sufficient for developing learning-based methods. Hence, it is only used as a testing set for optimization-based methods.
Reviewer 2. C #1: SAM pre-trained datasets. R: SAM is pre-trained on NIH Lymph Nodes dataset. All 3 datasets in this paper are not used for pre-training
C #2: Statistical analysis. R: We achieve statistically significant improvements in DSC than all comparing methods (p<0.001 or p<0.05) on Abdomen and HeadNeck tasks. We will add it to the final version.
Reviewer 3. C#1: Logical gap of convex optimization and coarse-to-fine scheme in [1]. R: As discussed in [1], when handling large motion, inaccurate small-scale low-contrast structure issues may occur in coarse-to-fine settings, which can be alleviated by convex optimization with complete search. However, complete search with a large search window suffers from heavy computational burden. To this end, we build a connection between the two. With the coarse-to-fine strategy, one can sparsely search on a coarser level with a smaller search radius in each iteration, reaching the same search range as the complete search with less computational complexity.
On a fine level, the small-scale structures can still be preserved within the search range. Similarity measure that relies on SAM rather than intensity/contrast also helps to align small-scale structures. Ablation study on pyramid designs (Tab.3) and leading accuracy on large deformation registration tasks (Tab.1) further support the claims.C #2: Differences with ConvexAdam. R: Apart from the SAM extraction module, first, we introduce a cost volume pyramid to the convex optimization framework to reduce the intensive computation burdens. Please refer to the response in C #1 for more details. Second, we explicitly validate the robustness of SAM feature against geometrical transformations and integrate the SAM feature into an instance-specific optimization pipeline. Our method is less sensitive to local minimal, achieving superior performance with comparable running time.
C #3: Figures are not very clear. R: We will improve the figures according to the comments.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper presents a fast coarse to fine discrete optimization method for CT registration that obtain deformation fields based on a self-supervised anatomical embedding feature extractor that captures both local and global information. All reviewers appreciated the proposed method, however they raised concerns about the novelty and missing details. The authors submitted a rebuttal to address these points and all reviewers appreciated their justification. The metareviewer agrees and even if he/ she thinks that the novelty is a bit limited, he/ she agrees that this method can be an interesting contribution for MICCAI.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
No major weakness, authors seem to have addressed the concerns raised.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper proposes a registration method using multi-resolution discrete optimization with SAM feature comparison as a similarity measure. The paper is well written and well motivated. On a first glance, the experiments seem to be extensive with a comparison to often used state-of-the-art methods and ablation study. However, I agree with the reviewers that the novelty is limited. The authors combined well established methods and strategies for image registration into a new method. This includes discrete/convex optimization, multi-resolution strategies (which is a common approach in classical image registration and used for over 20 years), SAM features, etc. The combination into a new method might be still a valid contribution, but then the experiments (and the rebuttal) are not convincing. The authors do not specify which strategies are used for the baselines. Are these multi-resolution approaches, which similarity measures are used, etc? To convince the reader that the proposed registration method is outperforming existing methods, they should show the performance of (i) SAM features as similarity measure in different registration frameworks, (ii) their optimization strategy with other similarity measures, e.g., MIND, MSE. Otherwise it seems that they are comparing apples to pears.