List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Ali K. Z. Tehrani, Hassan Rivaz
Abstract
The displacement estimation step of Ultrasound Elastography (USE) can be done by optical flow Convolutional Neural Networks (CNN). Even though displacement estimation in USE and computer vision share some challenges, USE displacement estimation has two distinct characteristics that set it apart from the computer vision counterpart: high-frequency nature of RF data, and the physical rules that govern the motion pattern. The high-frequency nature of RF data has been well addressed in recent works by modifying the architecture of the available optical flow CNNs. However, insufficient attention has been placed on the integration of physical laws of deformation into the displacement estimation. In USE, lateral displacement estimation, which is highly required for elasticity and Poisson’s ratio imaging, is a more challenging task compared to the axial one since the motion in the lateral direction is limited, and the sampling frequency is much lower than the axial one. Recently, Physically Inspired ConstrainT for Unsupervised Regularized Elastography (PICTURE) has been introduced which incorporates the physical laws of deformation by introducing a regularized loss function. PICTURE tries to limit the range of the lateral displacement by the feasible range of Poisson’s ratio and the estimated high-quality axial displacement. Despite the improvement, the regularization was only applied during the training phase. Furthermore, only a feasible range for Poisson’s ratio was enforced. We exploit the concept of known operators to incorporate iterative refinement optimization methods into the network architecture so that the network is forced to remain within the physically plausible displacement manifold. The refinement optimization methods are embedded into the different pyramid levels of the network architecture to improve the estimate. Our results on experimental phantom and in vivo data show that the proposed method substantially improves the estimated displacements.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43907-0_45
SharedIt: https://rdcu.be/dnwdr
Link to the code repository
Link to the dataset(s)
Reviews
Review #1
- Please describe the contribution of the paper
This work focuses on improving the accuracy of ultrasound elastography (USE) displacement estimation by incorporating physical laws of deformation into the process. The authors use Physically Inspired ConstrainT for Unsupervised Regularized Elastography (PICTURE) as a starting point but introduce iterative refinement optimization methods into the network architecture to ensure the network stays within the physically plausible displacement manifold. The proposed method improves the estimated displacements, as demonstrated in experiments on phantom and in vivo data against the selected baseline methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The incorporation of known operators into network architecture represents a novel approach that not only improves model accuracy but also enhances interpretability by ensuring adherence to underlying physical laws. The well-written presentation of this approach enables a clear understanding of its significance in bridging the gap between data-driven modeling and domain-specific knowledge. Moreover, the thorough evaluation against selected baselines demonstrates the effectiveness of this method, validating its potential to make a substantial impact in the field and offering a potentially reliable solution.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The presented approach, although promising, could benefit from a comparison with the recent work sPICTURE [1], which utilizes the same dataset. The rationale for not comparing with this self-supervised method is not entirely convincing, as it may provide valuable insights into the relative strengths and weaknesses of the two approaches. Additionally, the current study offers limited contributions over the previously published PICTURE method, making it essential to further distinguish and justify the proposed technique’s novelty.
[1] Tehrani, A. K., Ashikuzzaman, M., & Rivaz, H. (2022). Lateral Strain Imaging using Self-supervised and Physically Inspired Constraints in Unsupervised Regularized Elastography. IEEE Transactions on Medical Imaging.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors will provide the code and utilize publicly available datasets, promoting transparency and reproducibility in research. Moreover, the required Institutional Review Board (IRB) approval has been obtained for data collection, ensuring adherence to ethical standards.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
A thorough comparison with sPICTURE is needed to evaluate the proposed approach’s merits and shortcomings, and there is potential to expand the assessment of kPICTURE beyond previous experiments. Examining the frequency and location of threshold values reached could provide valuable insights into the samples outside the feasibility range.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The similarity to prior submissions was a major factor in my overall evaluation of this work.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The paper exploits the concept of known operators to incorporate iterative refinement optimization methods into the network architecture to ensure the network remain within the physically plausible displacement manifold. And this method acquires the best experimental results compared with SOTA.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper presents to incorporate two known operators inside a network for displacement estimation. The proposed method outperforms previous displacement estimation method.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The main weaknesses of the paper is that the author combines the existing technologies in papers [2][5] for displacement estimation.Nevertheless, I concede the authors the credit for adapting this technology to their task at hand.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The reproducibility of the paper is satisfactory .
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
(1) In Fig.1, the author gives the whole framework of MPWC-Net++. However, about the optical flow decoder, please give some information. (2) The author combines the existing technologies in papers [2][5] for displacement estimation. Please further highlight the novelty of the proposed method. (3) In the paper, the alogrithm2, “Guo et al. refinement [2] employed as known operator”. In my opinion, this algorithm seems have been proposed in reference[2]. It is better to remove this algorithm. (4) The experimental results are a little confusing, Whether kPICTURE means the proposed method? (5) How to understand the experimental results in Fig.3(a)?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The topic of this paper is intersting.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
The comparison with the recent work is insufficient.
Review #3
- Please describe the contribution of the paper
In this paper, the authors aim to embed two lateral displacement refinement algorithms in a CNN to improve the lateral strains estimations used for static ultrasound elastography.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
From the results, the proposed approach seems quite relevant and greatly improves the quality of the obtained images. However, given my expertise, I do not think I am fully able to properly evaluate the strengths and assets of this paper.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The Poisson’s ratio is estimated by considering in 2D its definition as the ratio of axial and transverse deformation. However, this parameter is eminently tridimensional, characterizing the conservation of volume of a material under stress. Indeed, the slightest anisotropy or heterogeneity should greatly affect its evaluation (not compensable by a simple assumption of epsilon2 = epsilon3), which is essential here. What exactly is the case?
We have to take into account “the minimum and maximum accepted EPR values, which are assumed to be 0.1 and 0.6, respectively” but biological soft tissue are mainly componed of water, so they are largely incompressible, aren’t they? So, the Poisson coefficient should be at least 0.49, right?
The development of the dataset requires a large number of data (here 600 RF if I am not mistaken), but all with the a priori knowledge of the presence of heterogeneity (dimensions and fixed positions). I currently find it difficult to understand how the CNN can be applied to a “random” case where both the mechanical properties, the position, the number and the dimension of the heterogeneities remain unknown (as is the case most of the time in clinical elastography for detection and characterization of tumors or others).
I am also not sure how boundary conditions (tissue interfaces, intermediate layers, other organs in the vicinity,…) can be taken into account and not compromise the quality of the results.
In view of the results, many points seem to me to be unclear, despite the very clear quality of the results obtained. However, considering my expertise, I do not think I am fully able to properly evaluate the limits and weaknesses of this paper.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Items appear to have been provided fully by the authors to ensure reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
My main comments concern the transition from 2D to 3D, as described above. The order of magnitude of the parameters also raises questions for me.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
My comments are already listing here above.
- Reviewer confidence
Not confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposes a method for measuring tissue displacements in Ultrasound Elastography, and its main focus is on explicitly enforcing constraints based on the physical properties of tissue motion. The starting point is an existing work PICTURE that enforces such physical constraints via regularisation during training. This submission introduces an inference-time operator that guarantees network outputs to adherence to physical models.
Strengths:
- Reviewers acknowledge that the incorporation of physical models in this problem is an interesting and relevant direction
- Reviewers also acknowledge the improvements obtained in the experimental results
Weaknesses:
- One reviewer questions whether all the relevant baselines have been tested (sPICTURE)
- One reviewer questions whether the employed 2D models really generalise to real scenarios where physical displacements happen in 3D.
I recommend this paper for rebuttal, so authors can better justify their choice of baselines and physical model assumptions.
Author Feedback
We thank the Reviewers and the AC for the constructive comments. We will only reply to major comments. Minor comments will be applied in the final submission. Meta Reviewer, R1, R2 & R3: (A) On comparison with sPICTURE (ref [13]): sPICTURE is an extension to PICTURE by adding self-supervision (in the form of cropping) to PICTURE to make the network more robust to cropping, which is out of the scope of this manuscript. Furthermore, sPICTURE was accepted on December 16, 2022 and the code was made available online later in February 2023. At the time of developing the proposed method, the sPICTURE paper and the code were not available; therefore, we built known operators on PICTURE instead of sPICTURE. The proposed known operators can also be applied to the network trained with sPICTURE method. To further address this comment, we will make the proposed network weights available online, trained using both PICTURE and sPICTURE methods. (B) On generalization of 2D models to 3D: The proposed method generalizes well to 3D deformation for isotropic materials, as supported by our phantom experiments and in vivo data (in all our results, the deformations are in 3D). We concur with both the meta reviewer and the reviewer that the performance of the method may be compromised when dealing with anisotropic materials, such as muscles, which exhibit varying properties along the direction of the fibers compared to their properties perpendicular to the fibers. To address this comment, the following text will be added to the discussion section (we had already mentioned that we assume isotropic material, a common assumption when lacking 3D imaging): “The applied known operators and PICTURE assume that the material is isotropic. Their performance on anisotropic materials can be investigated by experiments on anisotropic tissues such as muscles. Furthermore, 3D imaging data can be collected from 2D arrays to have information in out-of-plane direction to be able to formulate known operators and PICTURE loss for anisotropic tissues.” Additional comments: R2: On the contribution: Iterative optimization methods are infused to optical flow CNNs to enforce the physical constraints during the inference for the first time (in both medical imaging and computer vision). The first known operator (Algorithm 1) was proposed in this manuscript, the second one was originally proposed in [2] and the modified variant (adding smoothing in each step and updating the displacement using a hyperparameter weight for stability) was employed as the known operator in each pyramid levels of optical flow CNN. On Algorithm 2: This algorithm is slightly modified in this paper (see contribution part). We respectively believe it makes this work easier to understand and more reproducible. Interpreting Fig. 3(a): This figure shows the histogram of EPR values of experimental sample (1). The histogram indicates the range of EPR values for different methods. The proposed method is contained in the physically possible range of 0.1to 0.6. We also respectively refer to the last paragraph of page 7.
R3: On the range of EPR: Even if the Poisson’s ratio is 0.49 throughout the tissue, the EPR will have a range. To clarify this, we edited the first paragraph of page 3 as follows: EPR is spatially variant, and it is not equal to Poisson’s ratio, particularly in the vicinity of inclusion boundaries or within inhomogeneous tissue. Its value tends to converge towards the Poisson’s ratio in homogeneous regions [9].
On testing on random case: The presented in vivo data is a demonstration of applying the method on a random case in which there was no supervision on the boundary conditions, positions and mechanical properties. On the boundary conditions: It is assumed that the tissue can move freely in the lateral direction. This sentence will be added in the revised manuscript as: Assuming linear elastic, isotropic, and homogeneous material that can move freely in the lateral direction …
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors provided reasonable responses to the reviewers, carefully explaining why comparing against sPICTURE would not be necessary in terms of validating the specific contributions in this paper. They also provide additional comments on the issue of model generalisability to 3D displacements.
The reviewers have not provided any post-rebuttal feedback regarding these two concerns, so I am weighting on my judgement that the rebuttal responses are satisfactory.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The original reviews of the paper were generally positive for the paper, there were only a few questions regarding the choice of baselines and physical model assumptions, which has been addressed by the authors in their rebuttal. I recommend acceptance of the paper.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The proposed work is a novel attempt of fusing physic constraints into the model design. While the impact from tissue anisotropy or heterogeneity would certainly affect the model performance, this paper is suitable to be accepted as a preliminary proof-of-concept work.