List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Zhehua Mao, Liang Zhao, Shoudong Huang, Yiting Fan, Alex P.W. Lee
Abstract
This paper presents a novel algorithm named Direct Simultaneous Registration (DSR) that registers a collection of 3D images in a simultaneous fashion without specifying any reference image, feature extraction and matching, or information loss or reuse. The algorithm optimizes the global poses of local image frames by maximizing the similarity between a predefined panoramic image and local images. Although we formulate the problem as a Direct Bundle Adjustment (DBA) that jointly optimizes the poses of local frames and the intensities of the panoramic image, by investigating the independence of pose estimation from the panoramic image in the solving process, DSR is proposed to solve the poses only and proved to be able to obtain the same optimal poses as DBA. The proposed method is particularly suitable for the scenarios where distinct features are not available, such as Transesophageal Echocardiography (TEE) images. DSR is evaluated by comparing it with four widely used methods via simulated and in-vivo 3D TEE images. It is shown that the proposed method outperforms these four methods in terms of accuracy and requires much fewer computational resources than the state-of-the-art accumulated pairwise estimates (APE). Codes of DSR are available at https://github.com/ZH-Mao/DSR.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16446-0_10
SharedIt: https://rdcu.be/cVRSQ
Link to the code repository
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
A method for simultaneous mono-modal rigid registration of multiple volumes is proposed, based on a bundle adjustment mathematical formulation. The properties of the presented algorithm are demonstrated on simulated and real TEE ultrasound data from six patients, and compared against a number of other related algorithms.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The simultaneous registration approach is expressed as a bundle-adjustment problem that allows for a Gauss-Newton solver for least-squares problems to be utilized. It’s mathematical properties are thoroughly explained. This is an innovative idea that could be of interest to other researchers.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
There seems to be very little practical use for this method.
For one, the improvement over other methods is marginal. Then, only the most basic portion of a product-grade registration algorithm is addressed, namely a straight-forward SSD cost function, without any overlap normalization, artifact considerations (portions of ultrasound volume will be occluded that are visible in others, which violates the SSD assumption of the intensity relationship); and most important some real-time motion handling, which provides for some of the most interesting research problems. Hence one could argue that the authors are solving a non-problem, despite the elegant mathematical formulation.
No implementation details, runtime, and other computational resource information whatsoever is provided. So I will have to suspect that in light of the highly parallelized GPU implementations that are state of the art today, this method will fare unfavorably, by require dealing with huge sparse matrices and throwing a BLAS library at it (which might be hard to parallelize).
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
There is an algorithm diagram in the supplementary material which helps (it should be moved to the main manuscript), but most of the other implementation details such as numerical libraries, programming language etc are missing.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The mathematical proof in 2.2. looks elegant; however the outcome could be put in context better. To me it sounds intuitive that there is no dependency on the panorama intensities, since they are also reconstructed with a least-squares assumption, hence they will amount to the average value of all overlapping input volumes. Having a gradient iteration that is using the grid of that panorama image but not the intensities is nice - and the key idea of this paper it seems.
Please provide implementation details and run-times such that readers know whether this is only a mathematical feat or something useful for their medical image analysis problem.
The provided presentation video is very nice!
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Nice mathematical idea, but (in light of missing computation times, and similar performance than existing methods) limit practical use.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
The paper proposes a novel simultaneous registration of 3D image data without requiring a reference image or features, specifically for Transesophageal Echocardiography. A predefined panoramic image is used to optimise the global poses of local image frames. The method shows promising results on both simulated and in-vivo datasets.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper is easy to follow
- Clinically very valuable
- Methodology has all essential mathematical details
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Paper needs additional clinical motivation w.r.t 3D Transesophageal echocardiography registration
- Few simulated cases and in-vivo cases
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors should include more details on dataset. It would be great if they can release the dataset together with the code for reproducibility of the work conducted.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- How are these panoramic image predefined even though the solution does not depend on this?
- Other 3D CT or MRI cases would be very relevant examples to see if this can be applied to other imaging
- Are the chosen transformations of +/-15 pixels and +/- 12 degrees realistic range. How did the authors come to these values?
- Could authors include some samples of panoramic images (predefined) that they have taken and how these look after registration?
- What is the estimated time for the proposed SE(3) pose optimisation?
- Multimodality version or changing pixel intensities during simulation can be interesting to include especially to validate that the method is not dependant on intensity variation as claimed.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper hold huge clinical benefit, however, the authors need to brush up their clinical motivation of applying this to 3D TEE image. The results looks promising for both simulated and in-vivo studies.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
DSR (Direct Simultaneous Registration) approach is able to match 3D images without using the definition of keypoints. It is adapted to object (here organs) that have no easy keypoints to detect. The framework of the direct bundle adjustment is used. The first contribution DBA (Direct Bundle Adjustment) consists in redefining BA that jointly optimizes the poses of local frames and the intensities of the panoramic image (instead of 3D point positions in BA) and the second contribution is the proof that the optimization of poses is independent of the image intensities.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Well Written Well Structured mathematically well-founded Rigorous evaluation Code available if the paper is accepted
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Lack of illustrations for the method
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The code will be available after acceptation.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The state of the art is presented simply and briefly: from direct pairwise registration (based on features or intensities) to multiple 3D volume registration in order to emphasize the main contribution in the field of bundle adjustment in registering an image collection. The contribution is conducted in two steps as explained in section 3: DBA based on panoramic images, which consists of an original way to solve the BA, and then the proof that this DBA, thanks to Gauss-Newton optimization, is independent of the image intensities. The contribution is straightforward but clearly presented and the demonstration of the independence to intensity is convincing enough. The experiments were realized on synthetic data (5 simulated sequences of 3D grey heart images) and real data (46 TEE, Transesophageal Echocardiography images). The data amount tested is quite limited, however, regarding the real data, this is probably due to the difficulty of collecting this kind of data. The proposed method is compared to four methods: the pairwise method, the Lie normalization method, the sequential method and the APE method. This comparison seems relatively fair.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The work is interesting for a reduced community.
- Number of papers in your stack
3
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper is about finding the relative pose between 3D images while making a complete image (the panorama). For that, it minimises the LS discrepancy between the generated images from the panorama and the original images. Because the problem involves many parameters, the authors used the classical BA approach. The reviewers are positive or lean to the positive side about the paper. The AC is less positive. In terms of methodology, section 2 is just about standard material from BA, which is available in many textbooks or articles, such as Triggs et al, some of them cited in the paper. In addition, the paper uses a poor/naive image intensity model, which is simply generated by some interpolation of the panorama. Advanced intensity models exist for images, based on a proper point spread function (PSF), especially in superresolution (see LC Pickup and A Zisserman et al’s papers) and varying resolution (see eg Simon Baker’s numerous papers on the subject). The mechanism of sparse solution / LS /GN is widely in computer vision, in many contexts. The AC thus recommends the paper enters rebuttal. The authors should address the following questions: 1) what methodological novelty does the paper bring wrt classical BA solution of sparse structured systems? 2) same question regarding the intensity model and why not using existing PSF / superresolution models ; in addition, what interpolation scheme is being actually used? 3) why not comparing other costs that L2 in the used framework? 4) why providing little experimental results and not testing the framework on other 3D modalities, using some of the numerous available public datasets? The AC thanks the authors for their responses.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
NR
Author Feedback
We thank Reviewer #1 (R1), R2, R3, and the Area Chair (AC) for providing the reviews. They appreciate that our work is “an innovative idea” (R1), “clinically very valuable” (R2), and “mathematically well-founded” (R3).
We now address the questions from AC, and some major comments by R1-R2.
AC asks the methodological novelty of our paper compared to classical BA. (1) Our DBA redefines BA by jointly optimizing the poses of local frames and the intensities of the panorama (instead of 3D point positions in classical BA). (2) The inputs of our DBA are the raw 3D images (DBA uses intensity information directly) instead of the extracted and matched feature points of the local 2D images in classical BA (such as Triggs et al). So our method can deal with images lacking distinct features such as 3D TEE. (3) Importantly, we prove in DBA, the pose estimation is independent of intensities of panorama. Classical BA does not have such property. (4) Based on (3), we derive DSR that ONLY solves the poses WITHOUT solving the intensities of panorama (obtaining the same poses as DBA). Classical BAs need to jointly solve all the parameters (poses and feature positions) together. R3 confirmed our novelties in “contribution”.
AC asks why not using PSF/super-resolution (SR) models and what interpolation scheme is used. In our work, the focus is on registration of 3D images to enlarge FoV, instead of obtaining a high-resolution image for the overlapping area. The resolution of panorama is the same as that of local 3D images. We do not perform interpolation on panorama, instead, we project voxels from panorama to local images and use trilinear interpolation on local images for computing more accurate Jacobians. Using this DBA formulation, we prove that the intensities of panorama do not influence the registration and derive DSR.
When SR is used, one way is to separate registration and high-resolution image reconstruction into two steps. Another way is simultaneous registration and SR (such as LC Pickup and A Zisserman et al), but mainly for 2D optical images. Obtaining a good prior of the blur model is critical for SR, which is more difficult for 3D medical images such as TEE, making simultaneous registration and SR ill-posed. Our immediate future work could use our DSR as the first step and then attempt SR. Thanks AC for reminding us about PSF/SR.
AC asks why not compare other costs. The L2 norm (i.e. SSD similarity) used in the paper has been extensively validated [1,2,6,11] to be effective for monomodal medical image registration.
AC and R2 suggest testing on other 3D modalities and providing more experimental results. We use 3D TEE but not other 3D modalities because (1) the registration is especially valuable for overcoming one major drawback of 3D TEE images, i.e., the limited FoV. (2) registration of ultrasound images is usually more challenging than other modalities like CT and MRI due to the relatively low signal-to-noise ratio. For the number of experiments, in the paper, we used 55 simulated 3D images, and 46 in-vivo 3D TEE images from 6 patients that we can access.
R1 asks the practical use of our work. (1) Achieving a higher accuracy is always valuable for medical image registration. Fig. 1 shows that we can improve more than 50% in accuracy compared to [2,9,10,11] for most cases. (2) Our method can be very useful in practice when real-time performance is not required. The enlarged FoV of 3D TEE has already been used for surgical planning of LAA occlusion. (3) For 3D TEE, main artifacts can be avoided in the imaging stage and real-time motion can also be avoided since clinical experts usually fix probe when capturing 3D TEE images. Our in-vivo datasets are all real clinical images without pre-processing, and our method works well.
R2 asks how panorama is predefined. It is predefined by only creating a 3D grid that covers the increased FoV without intensity values (only voxel positions are used in our DSR).
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors provided a rebuttal to address the questions from the AC. After carefully reading and considering it, the AC believes they have understood the paper well and will stick to their original recommendation. The main recommendation for the authors in the AC’s opinion will be to drop from the paper the technical material on solving a sparse BA like system, which is highly standard and well known for many decades and consider instead more experiments and types of images with the free space, as well as the other points from the initial review.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
NR
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Reviewers were all relatively positive about the work, but the other meta-reviewers were less positive. I go along with them because they seem to have seen similar approaches to this in a textbook.
My own principal objection to the approach would be that it does not properly consider a forward model of the acquired data (I), but instead uses an objective function (equation 1) that interpolates each of the images (I) to match the reconstructed panorama (M), rather than consider how M would be spatially transformed to match each I. This subtly changes the objective function (and its gradients) and would make it less straightforward to incorporate enhancements, such as super-resolution etc, in future (although I realise the work is not about super-resolution).
I have no objections to the use of Gauss-Newton, and see this a useful component for tacking many optimisation problems, although I am unsure if this particular formulation has been published previously (as the other meta-reviewers suggest) . I also see the general purpose utility in the approach, as there are many situations where clinicians may wish to average several scans of the same subject and modality in order to achieve better SNR.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
14
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
DSR: Direct Simultaneous Registration for Multiple 3D Images
This submission is about registering 3D TEE ultrasound in order to acquire large field of view images. The reviews were positive with concerns partially addressed in the rebuttal. One was situating novelty with bundle adjustement use in computer vision. The rebuttal clarified the unique scenario posed by 3D TEE, namely, the lack of obvious keypoints. One was on the clinical benefit of the method. The review has clarified that enlarging a field of view from the registration of multiple images has clinical value, which I also agree, for instance, in transeasophagal imaging in left atrial fibrilations. This is a difficult problem due to poor imaging conditions, different independent motions arising from cardiac and breathing cycles.
The rebuttal has partially addressed this concern as the key clinical value in such case is having a fast real time approach, which the submission does not seem to address. R1 is in that sense very right and spot-on in my opinion. The rebuttal evades this concern and does not provide any computation time - which may raise further doubts. This is the main weakness of the submission, despite the evaluation on six patients. R1 indicates that the work may possibly solve a non-problem despite the elegant mathematical formulation. R3 indicates good paper with reduced community. AC indicates the outcome is less positive.
For all these reasons, and situating this work with the other submissions, recommendation is towards Rejection.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
11
Meta-review #4
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
There was a divergence of recommendations between the reviewers and ACs. While the AC recommendations were unanimous in rejection, the reviewers expressed consensus in supporting acceptance including one reviewer raised score to support acceptance after rebuttal. The PCs thus assessed the paper reviews, meta-reviews, the rebuttal, and the submission. It is noted that the reviewers highly appreciated the novelty, the mathematical elegance, the potential clinical value, and the rigorous evaluation presented in the paper. The ACs raised additional questions such as the differentiating points between the presented work and standard BA methods and the choice of intensity models, to which the PCs considered that the authors have provided a reasonable response. There was a remaining concern on the practical value of the presented method including its unclear computation time – while this was raised by the reviewer before rebuttal, it was not summarized in the metareview as the key points to address during rebuttal, which could be the reason why it was not addressed in the rebuttal. Overall, the PCs felt that, while there are areas to be improved, the weaknesses are outweighed by the enthusiasm the reviewers expressed on the strength of the paper. The final decision of the paper is thus accept, and the PCs strongly encourage the authors to address the practicality and computation time of their work in the final manuscript.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
NR