List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Qingjie Meng, Wenjia Bai, Tianrui Liu, Declan P O’Regan, Daniel Rueckert
Abstract
3D motion estimation from cine cardiac magnetic resonance (CMR) images is important for the assessment of cardiac function and diagnosis of cardiovascular diseases. Most of the previous methods focus on estimating pixel-/voxel-wise motion fields in the full image space, which ignore the fact that motion estimation is mainly relevant and useful within the object of interest, e.g., the heart. In this work, we model the heart as a 3D geometric mesh and propose a novel deep learning-based method that can estimate 3D motion of the heart mesh from 2D short- and long-axis CMR images. By developing a differentiable mesh-to-image rasterizer, the method is able to leverage the anatomical shape information from 2D multi-view CMR images for 3D motion estimation. The differentiability of the rasterizer enables us to train the method end-to-end. One advantage of the proposed method is that by tracking the motion of each vertex, it is able to keep the vertex correspondence of 3D meshes between time frames, which is important for quantitative assessment of the cardiac function on the mesh. We evaluate the proposed method on CMR images acquired from the UK Biobank study. Experimental results show that the proposed method quantitatively and qualitatively outperforms both conventional and learning-based cardiac motion tracking methods.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16446-0_24
SharedIt: https://rdcu.be/cVRS5
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes an image-based mesh motion estimation network for 3D myocardial motion tracking, which estimates 3D mesh displacements from the intensity information of 2D CMR images in an end-to-end manner.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The proposed method models the heart as a 3D geometric mesh and propose a network to estimate 3D motion of the heart mesh from 2D short- and long-axis CMR images. The paper is well-written and easy to follow. The authors provided sufficient details about the proposed method.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
(1) This paper utilizes no regularization loss for the registration step. Though the proposed method imposes Laplacian smoothing loss on predicted mesh, it would be better to provide discussions or analyses of the reason for the absence of the registration regularization loss. (2) Self-supervised differentiable segmentation losses have been widely used in related motion tracking works [1][2]. It will be better for authors to discuss these works in this paper.
[1] Self-supervised Learning of Motion Capture. [2] Self-Supervised Learning for Cardiac MR Image Segmentation by Anatomical Position Prediction.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Good reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Minor point: (1) One more period in the title of Section 2.3. (2) In Section 4, “…propose a image-base…’’ needs to be improved.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper is innovative and the experimental evaluations are convincing.
- Number of papers in your stack
2
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The authors propose a deep learning method for 3D cardiac motion tracking from 2D MRI. Rather than computing a dense displacement field, the proposed method learns to predict the vertex displacements of a 3D mesh from the 2D input images. This requires a novel mesh-to-image rasterizer to translate the motion estimates to 2D. This allows cardiac motion estimation and mesh-based segmentation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
• Experiments are performed on a large cohort of 530 subjects from the UK Biobank (testing set of 80 subjects). • Ablation studies clearly demonstrate the design decisions for both the input image configuration (using 3 views) (Table2) and the loss functions (Table 3). • A (brief) discussion regarding the choice of algorithm hyperparameters is presented (Sec. 3, Discussion). • The proposed method demonstrates improvements in registration accuracy compared to three other cardiac registration methods. • The clinical problem and background literature are covered well.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
• Unclear how well the method performs quantitatively across the full cardiac cycle (results appear to show motion correction between end diastole and end systole). • (minor) Comparison registration methods may also benefit from multi-view inputs, but these are never tested.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The dataset and imaging information used in this study are well described and the model training parameters are detailed in sufficient detail.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Sec. 3 (Tables 1, 2, 3): From the text in Sec. 3 it appears that quantitative motion correction evaluation is calculated at only two time points (end diastole and end systole). It would be interesting to report the values at different times in between as well to verify quantitatively that the method works well across the full cardiac cycle.
Sec. 3: Based on the results of the ablation study to include different views of the heart (Table 2), it appears that a major source of performance improvement comes from having two or more views (the addition of the 2CH and 4CH views). While comparison to the three other registration methods using SAX only images (Table 1) is very strong, and the results in Table 2 demonstrate that SAX only imaging for the proposed method shows improvements over the comparison methods, these results raise the question about the other methods also being improved using multiple views. This may present a technical challenge for the other methods, but it might be worth discussing.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This is a well written paper with good experimental setup and validation of a novel cardiac motion correction method based on deep learning. The authors provide a rigorous set of validation experiments to compare their proposed method to other registration approached and include a strong set of ablation studies to demonstrate their design decisions.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The authors claimed 3 innovations: 1) Achieved cardiac motion tracking on a geometric mesh using a deep network. 2) Used a mesh-to-image rasterizer to summarize related 2D information into 3D motion estimation. 3) Can achieve both motion estimation and image segmentation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The inclusion of the mesh-to-image rasterizer is a bright idea. Usually combining 2D image information into any 3D tracking result needs an underlying interpolation step which takes time and extra computation resource (also introduces error). The rasterizer used in this work extracts 2D shape information in multiple directions and summarizes its together for 3D, which is a computationally more efficient process.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) Cardiac motion tracking on a geometric mesh model is not uncommon. 2) Simultaneous motion tracking and segmentation is not a completely novel achievement. 3) Comparison with only a few previous methods before claiming “the proposed method quantitatively and qualitatively outperforms both conventional and learning-based cardiac motion tracking methods” is not convincing.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Reproducible. Data available. Method detailes described well.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
This is a very interesting work with some good innovation (especially the mesh-to-image rasterizer part mentioned above). The paper is also well written. It is definitely worth being published in some form. Though considering the novelty required in MICCAI, I am not totally convinced to recommend a full acceptance. Cardiac motion tracking is a very wide field with numerous previous methods. The authors tried to separate this work from those “voxel-wise deformation estimation” methods by working in the mesh tracking area. This is a good strategy and vertex tracking is definitely beneficial in many ways. But you cannot help compare it to those voxel-wise tracking methods because they have been used for a long time. The hardest job for the authors is to show the value of this new method and justify its value. But it feels too weak for me.
For example, in the initial introduction the authors briefly mentioned those voxel-wise methods [17,18,2,27], which are pretty recent but are also very limited. They are two MICCAI papers, a CVPR paper, and one journal, which are way less than enough to represent those previous works. Later in related works, the authors introduced more, but they were mostly registration-based “conventional” methods, which, again, are just too limited to cover the 3D cardiac motion tracking field. This is not to mention the overwhelming amount of deep learning-based tracking methods nowadays. Also, in the experiments, the authors compared the method to FFD, dDemons (representing the conventional methods), and U-Net (representing deep learning methods). And the conclusion was “Experimental results show that the proposed method quantitatively and qualitatively outperforms both conventional and learning-based cardiac motion tracking methods.” This makes it a rash conclusion. As said above, conventional methods come in way more categories and FFD and dDemons barely suffice to represent the deformation kind. The same goes for learning-based methods. So the justification of proposing this method is not strong enough for me.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1) This is an overall interesting work. Very good innovation. Manuscript well written. Has scientific value. The novelty is good enough for a weak accept but not outstandingly impressive. 2) The part of comparison to previous works part is still weak. Not too convincing for me to recommend a strong acceptance.
- Number of papers in your stack
3
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposes an image-based mesh motion estimation network for 3D myocardial motion tracking. As a result, 3D mesh displacements can be estimated from the intensity information of 2D CMR images in an end-to-end manner. The consensus is that the work has made interesting contributions and the paper is well written. Please complete the literature and add discussions on the performance over the full cardiac cycle.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
4
Author Feedback
We thank AC and all reviewers for acknowledging the contributions of our work. There are two comments summarized by the AC:
- Complete the literature.
In the revision, we will add additional literature on cardiac motion tracking (see [A,B,C,D]) and further clarify that both conventional registration-based methods and deep learning-based methods are included in the related work. In addition, we will add and discuss another literature on self-supervised learning [E], suggested by R1. The other work [F] (suggested by R1) is not included as it focuses on segmentation instead of motion tracking.
[A] Using synthetic data generation to train a cardiac motion tag tracking neural network, Medical Image Analysis, 2021. [B] DeepTag: An Unsupervised Deep Learning Method for Motion Tracking on Cardiac Tagging Magnetic Resonance Images, CVPR, 2021. [C] Temporal diffeomorphic free-form deformation: Application to motion and strain estimation from 3D echocardiography, Medical Image Analysis, 2012. [D] An Incompressible Log-Domain Demons Algorithm for Tracking Heart Tissue, MICCAI STACOM, 2011. [E] Self-supervised Learning of Motion Capture, NeurIPS, 2017. [F] Self-Supervised Learning for Cardiac MR Image Segmentation by Anatomical Position Prediction, MICCAI, 2019.
- Discuss the quantitative performance over the full cardiac cycle.
We evaluated quantitative performance on the ES frame. But for the full cardiac cycle, we were only able to demonstrate qualitative results in the paper. This is due to the limit of the dataset we used that ground truth meshes were only available at the ED and ES frames. We will add this discussion in the revision.
In addition, we will also discuss the choice of the regularization loss and correct for the typos pointed out by the reviewers.
Thank you for all comments that help improve the quality of the paper.