List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Ali K. Z. Tehrani, Hassan Rivaz
Abstract
Displacement estimation is a critical step of virtually all Ultrasound Elastography (USE) techniques. Two main features make this task unique compared to the general optical flow problem: the high-frequency nature of ultrasound radio-frequency (RF) data and the governing laws of physics on the displacement field. Recently, the architecture of the optical flow networks has been modified to be able to use RF data. Also, semi-supervised and unsupervised techniques have been employed for USE by considering prior knowledge of displacement continuity in the form of the first- and second-derivative regularizers. Despite these attempts, no work has considered the tissue compression pattern, and displacements in axial and lateral directions have been assumed to be independent. However, tissue motion pattern is governed by laws of physics in USE, rendering the axial and the lateral displacements highly correlated. In this paper, we propose Physically Inspired ConsTraint for Unsupervised Regularized Elastography (PICTURE), where we impose constraints on the Poisson’s ratio to improve lateral displacement estimates. Experiments on phantom and \textit{in vivo} data show that PICTURE substantially improves the quality of the lateral displacement estimation.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16440-8_21
SharedIt: https://rdcu.be/cVRvM
Link to the code repository
Link to the dataset(s)
https://users.encs.concordia.ca/~impact/ultrasound-elastography-dataset-for-unsupervised-training/
Reviews
Review #1
- Please describe the contribution of the paper
This paper tackles the problem of displacement estimation from beamformed RF data in ultrasound elastography. The contribution is a physics-based regularization – in particular, using the fact that human tissue has a relatively small range of Poisson’s ratio close to 0.5. This allows the lateral and axial displacements to be correlated.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The problem of displacement estimation in ultrasound elastography (and similar applications) is still an ongoing unsolved problem and this paper’s approach of using biomechanics-motivated constraints is a solid approach.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Biomechanics constraints in displacement tracking is not completely novel. Also, limited testing is performed and only one previous method is used for comparison. The work does advance the field of ultrasound displacement tracking but is not revolutionary in its approach and still needs to be combined with other displacement tracking innovations for reliable accurate tracking.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Reproducibility is adequate from the clear mathematical derivation.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The proposed method is called PICTURE and is compared to unsupervised learning and OVERWIND from reference [2]. The methods are compared on both phantom and in vivo liver scanning data using CNR and SR (strain ratio), two common metrics of comparison. No simulations are performed where the ground truth of displacement is known.
The main contribution of this paper is the development of a learning-plus-physics based approach to displacement measurements that does appear to perform well in these limited test cases with some exceptions where it performs worst. As negatives, the amount of testing is rather limited, there is no sensitivity analysis to the parameters, and there were no simulations performed with known ground truth. This is therefore a good but early contribution that needs more analysis to prove its benefits with confidence
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although the methods are sound, the paper makes only a small contribution to the very large body of literature on ultrasound displacement tracking and is of interest to only the subset of ultrasound researchers in elastography who don’t already have their own solutions.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #4
- Please describe the contribution of the paper
The paper proposes a new loss function to regularise the training of Ultrasound Elastography. It is based on physical properties that need to be present in a pysically correct estimate of Ultrasound Elastography. In terms of network architecture MPWC-Net++ is employed to conduct the experiments. Experimental results demonstrate significant improvements over the state of the art.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper proposes a novel Loss function that enforces physical correctness during unsupervised training of the network. The experimental results demonstrate strong effectiveness.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The regularisation during the training is of course only present at train time. As such here is no guarantee that the predictions will always be physically plausible. Therefore, I recommend the authors to investigate whether known operators [1] could be applies in this scenario. This might lead to another paper on MICCAI 2023…
[1] Maier, Andreas K., et al. “Learning with known operators reduces maximum error bounds.” Nature machine intelligence 1.8 (2019): 373-380.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
OK
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Paper is very good and in good shape. Other than that I recommend to look into hard-coding the physical properties into the network (see weaknesses)
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
See section “strengths”.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #5
- Please describe the contribution of the paper
- The authors propose a physically inspired constraint (PICTURE) to improve lateral displacement estimation on ultrasound elastography.
- The experiment results on phantom and in vivo data show that PICTURE substantially improves the quality of the lateral displacement estimation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The motivation and innovation of this work are good.
- A new constraint based on the Poisson’s ratio for lateral displacement estimation.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The amount of in vivo data is unknown.
- The PICTURE appears to be limited to uniform axial strain (see detailed and constructive comments).
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The code will be publicly available after the paper acceptance. Hence, the reproducibility seems fine.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
-
In Eq. 11, how do you warp I2? How does data loss constrain the displacement W?
-
In Eq. 12, the axial strain is constrained to be near the mean, while the strain in other directions is constrained to be zero, and all first-order derivatives are constrained to be zero. As a result, the strain is the same for all frames, which is not the case in reality.
-
In Eq. 14, how do you select the window, especially the background window?
-
The authors should describe detailed information about the dataset, such as the number of sweeps for experimental phantom and the number of patients for in vivo data.
-
The unsupervised results in Fig. 1 are a complete failure, which questions the validity of data loss and smoothness loss. The authors should analyze the reasons for the failure and conduct further other ablation experiments.
-
The authors should show more cases in Fig. 2.
-
Why does the EPR histogram of PICTURE (Fig. 1 in the supplementary material) range beyond v_emin and v_emax, with two peaks?
-
Can PICTURE be applied to out-of-plane displacements?
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The authors design a new physically inspired constraint to improve lateral displacement estimation, but it is limited to uniform axial strain.
- The experimental results show that the proposed framework is superior to the current methods but lacks methodological innovation.
- Number of papers in your stack
8
- What is the ranking of this paper in your review stack?
5
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The authors propose a novel method from improving motion tracking in ultrasound elastography, combining prior biomechanical knowledge and a learned model. All reviewers agree to the value of the proposal; although the novelty ranges from limited to good, there is consensus that the flaws in the paper are relatively small and that most can be addressed . I suggest authors have a close look at the reviewers remarks to improve their manuscript, mainly on:
- clarify what component of strain is estimated
- physical plausibility at test time
- dependence on other tracking methods.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2
Author Feedback
R1. On limited testing We agree that more testing always strengthens a paper. We tested the method on real (i.e. not simulated) images of tissue mimicking phantoms and in vivo data and provided additional results in the supplementary material. Testing on more data will be included in future work.
R4: On regularisation during the training is of course only present at train time. As such here is no guarantee that the predictions will always be physically plausible. Excellent point. The PICTURE regularization, similar to any other forms of regularization, is only applied during training. Although PICTURE limits the range of Poisson’s ratio, there is no guarantee it is exactly in the considered range during the test time. We added this point in the revised paper and added a citation to the Maier et al. 2019 paper as follows: It should be mentioned that the PICTURE regularization, similar to other forms of regularization, is only applied during the training phase, and methods such as known operators [14] can be used to ensure it is also applied at test time. R5: The amount of in vivo data is unknown. Due to lack of space, the method is evaluated on one patient, which is now clarified on page 7. R5: 3- Method limited to uniform axial strain. We respectfully disagree. As all of the results demonstrate, all axial strains are non-uniform. R5: About Eq. 11. The moved image (I2) is warped by the displacement (W) using bi-linear warping operation. The data loss constrains the displacement W by trying to minimize the loss function. It is now clarified in the revised paper (page 4) R5: About Eq. 12. This is a popular smoothness regularization used in both elastography and optical flow. In fact, it acts as a regularization to penalize high strain. The presence of the data loss and low weight of the smoothness loss avoids the strain to be zero and instead causes a smooth strain. R5: About Eq. 14. The windows are selected in uniform regions of the target (inclusion) and the background regions. To avoid sensitivity to the location of these windows, they are moved around and hundreds of CNR values are calculated based on the location of these windows. The resulting mean and standard values are reported in Table I. R5, dataset information. The dataset is the same as the training dataset in [5] and is available online at data.sonographi.ai R5. On failure of unsupervised results in Fig. 1. We respectfully disagree. The unsupervised method does not fail and it produces high-quality axial strain which was not shown in the main paper due to space limitation, and was instead shown in the supplementary materials of the original submission. The unsupervised method fails in estimating the lateral strain where the motion and sampling frequency are substantially lower than the axial one. This is a common problem for traditional elastography methods too. We showed that the new proposed regularization changes the unsupervised method from a failure for lateral strain into an acceptable method. R5. On the range of EPR. Because the constraint is applied at training time only and is not enforced at test time. We added the following sentence to the revised paper to clarify this point: It should be mentioned that the PICTURE regularization, similar to other forms of regularization, is only applied during the training phase, and methods such as known operators [14] can be used to ensure it is also applied at test time. R1. On limited testing We agree that more testing always strengthens a paper. We tested the method on real (i.e. not simulated) images of tissue-mimicking phantoms and in vivo data and provided additional results in the supplementary material. Testing on more data will be included in future work.
R4: On regularisation during the training is of course only present at train time. As such here is no guarantee that the predictions will always be physically plausible. Excellent point. The PICTURE regularization, similar to any other forms of regularization, is only applied during training. Although PICTURE limits the range of Poisson’s ratio, there is no guarantee it is exactly in the considered range during the test time. We added this point in the revised paper and added a citation to the Maier et al. 2019 paper as follows: It should be mentioned that the PICTURE regularization, similar to other forms of regularization, is only applied during the training phase, and methods such as known operators [14] can be used to ensure it is also applied at test time. R5: The amount of in vivo data is unknown. Due to lack of space, the method is evaluated on one patient, which is now clarified in page 7. R5: 3- Method limited to uniform axial strain. We respectfully disagree. As all of the results demonstrate, all axial strains are non-uniform. R5: About Eq. 11. The moved image (I2) is warped by the displacement (W) using bi-linear warping operation. The data loss constrains the displacement W by trying to minimize the loss function. It is now clarified in the revised paper (page 4) R5: About Eq. 12. This is a popular smoothness regularization used in both elastography and optical flow. In fact, it acts as a regularization to penalize high strain. The presence of the data loss and low weight of the smoothness loss avoids the strain to be zero and instead causes a smooth strain. R5: About Eq. 14. The windows are selected in uniform regions of the target (inclusion) and the background regions. To avoid sensitivity to the location of these windows, they are moved around and hundreds of CNR values are calculated based on the location of these windows. The resulting mean and standard values are reported in Table I. R5, dataset information. The dataset is the same as the training dataset in [5] and is available online at data.sonographi.ai R5. On failure of unsupervised results in Fig. 1. We respectfully disagree. The unsupervised method does not fail and it produces high-quality axial strain which were not shown in the main paper due to space limitation, and were instead shown in the supplementary materials of the original submission. The unsupervised method fails in estimating the lateral strain where the motion and sampling frequency are substantially lower than the axial one. This is a common problem for traditional elastography methods too. We showed that the new proposed regularization changes the unsupervised method from a failure for lateral strain into an acceptable method. R5. On the range of EPR. Because the constraint is applied at training time only and is not enforced at test time. We added the following sentence to the revised paper to clarify this point: It should be mentioned that the PICTURE regularization, similar to other forms of regularization, is only applied during the training phase, and methods such as known operators [14] can be used to ensure it is also applied at test time.