List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Pak-Hei Yeung, Moska Aliasi, Monique Haak, the INTERGROWTH-21st Consortium, Weidi Xie, Ana I. L. Namburete
Abstract
Two-dimensional (2D) freehand ultrasound is the mainstay in prenatal care and fetal growth monitoring. The task of matching corresponding cross-sectional planes in the 3D anatomy for a given 2D ultrasound brain scan is essential in freehand scanning, but challenging. We propose AdLocUI, a framework that Adaptively Localizes 2D Ultrasound Images in the 3D anatomical atlas without using any external tracking sensor.. We first train a convolutional neural network with 2D slices sampled from co-aligned 3D ultrasound volumes to predict their locations in the 3D anatomical atlas. Next, we fine-tune it with 2D freehand ultrasound images using a novel unsupervised cycle consistency, which utilizes the fact that the overall displacement of a sequence of images in the 3D anatomical atlas is equal to the displacement from the first image to the last in that sequence. We demonstrate that AdLocUI can adapt to three different ultrasound datasets, acquired with different machines and protocols, and achieves significantly better localization accuracy than the baselines. AdLocUI can be used for sensorless 2D freehand ultrasound guidance by the bedside. he source code is available at https://github.com/pakheiyeung/AdLocUI.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16440-8_20
SharedIt: https://rdcu.be/cVRvL
Link to the code repository
https://github.com/pakheiyeung/AdLocUI
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The authors developed a method to localize a 2D ultrasound image in the head of a fetus during an ultrasound examination.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The algorithm is able to localize the plane in the head of a fetus The analysis is sound
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The clinical motivation is not clear The statistical analysis requires some clarification.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
OK
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
It is not entirely clear why localizing the 2D image in the application you described is needed. Is it to find the optimal plane to perform measurements? Although the 2D ultrasound approach can be variable, does it take much time and is the variability significant? Although plane detection can be useful in many clinical applications, I think the motivation for this work needs some more justification.
What is used to normalize NSTD (normalized standard deviation)? Is it the mean value?
Table 1. It would be good to understand ED in units of mm to give the reader a better understanding of the error.
Table 1. Since it appears you performed statistical tests, it is not clear whether you corrected for multiple t-test to avoid a type 1 error, i.e., Bonferroni correction.
I am not sure how to interpret a NSTD of 0.553. It is lower than what was generated in Yeung at al, but I am not sure if it sufficiently low to be used clinically. It would help to know what NSTD was normalized with.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
My main concern is the clinical motivation. The reason for developing the method is not convincing.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Not Confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
My main concern was the clinical utility of the method. The rebattal addresses this, but I am still a little skeptical.
Review #3
- Please describe the contribution of the paper
This work contributes with a method to adapt a previously trained CNN for the localization a 2D ultrasound plane inside a 3D volume. The contribution allows for the application of the trained CNN to different sets of ultrasound volumes and video sequences.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The method for adaptation of previuosly trained CNN is a novel contribution. Testing was performed in an adequate set of 3D volumes (17) not used for training, as well as testing in 7 video sequences of free hand 2d ultrasound with more than 1000 image frames (planes) in total
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
In my opinion the main weakness is the testing performed on free hand 2D ultrasound images, since the location of the image planes inside the volume is unknown. The authors propose a “rate of change” index to estimate accurate localization of planes, it seems to me that this index can result in optimum values for an smooth sequence of contiguos images, which are wrongly located in the US volume (e.g. an smooth sequence of planes parallel to the trancerebellar plane may have optimum values of “rate of change” for the bottom half of the cerebellum as well as for the top half of the cerebellum)
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The work seems reproducible
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
A possible evaluation of free hand 2D US image location inside a 3D volume, could be performed if the set of 2D images is acquired with a tracking device attached to the US probe. Then the set can be reconstructed accurately and all the plane positions calculated could be evaluated against this reference volume.
What is the voxel size in table 1?
I suggest to include table 3 in the main text
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The original contribution is significant and has a wide scope for application and further improvements, maybe trackless 3D reconstruction if adequately validated.
- Number of papers in your stack
3
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
Authors designed a framework that adaptively localizes 2D ultrasound images in the 3D anatomical atlas. They also fine-tuned their method after getting instances which provides double accuracy in quantitative results.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The method requires limited number of frames to be manually annotated which is good. Unsupervised fashion of the method alow the technique to adapt to 3 different datasets from different US machines.
- Authors claimed that the overall displacement of a video in the 3D anatomical atlas is equal to the displacement from the first image to the last in that video.
- Authors claimed that compare to the baseline methods they fine tuned their method to produce more accurate localization accuracy.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
This might be a little personal idea but it is a fact. In medical imaging, when we are reconstructing 3D image from voxels from 2D freescan Ultrasound frames, we need to be around 100% accurate in reconstruction. The reason is that we are looking at patient organ or lision. We cannot use estimation like the author’s idea or train a model to generalize one image to other image. maybe in test data, we have a new problem in patient which was not in training dataset at all. Therefore, still using sensors and 3D ultrasound probes is the best way to solve this problem, and community should work on decreasing the cost of those techniques and increasing the performance and accuracy. It is not acceptable that every problem can be solve by training a deep learning model. For this reason, the proposed idea is not suitable for medical imaging. Estimating and predicting location of a 2D frame can be used for animation design or other engineering applications. Now this generalization issue can be see more clear when we are generalizing from Atlas to real human organs. This become even more worse when we are using unsupervised techniques for localization of frames in 3D slace.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
I couldn’t fully understand Regression ConvNet because authors didn’t provide any details. Supplementary data was helpful for better understanding the work.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The manuscript was written smoothly, and I like reading that. It is better to focus more on qualitative results than providing many formula for training objectives. Literature has two great papers in this field that can be cite in introduction:
Mohamed, Farhan, and C. Vei Siang. “A survey on 3D ultrasound reconstruction techniques.” Artificial Intelligence—Applications in Medicine and Biology (2019).
Mozaffari, Mohammad Hamed, and Won-Sook Lee. “Freehand 3-D ultrasound imaging: a systematic review.” Ultrasound in medicine & biology 43.10 (2017): 2099-2124.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
In my opinion, working at datasets from different ultrasound machines is valuable. Especially when the goal is training novice sonographers. Although novelty of the work is not completely enough for MICCAI, but publication of the work will attract many attention in ultrasound community readers.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #4
- Please describe the contribution of the paper
-The authors propose a localization framework (AdLocUI) that localizes 2D ultrasound images within a predefined 3D fetal brain atlas. -The authors design a novel cycle consistency to adapt to the target domain of 2D ultrasound images.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The motivation and innovation of this work are good.
- A novel cycle consistency for 2D ultrasound image localization.
- The AdLocUI is trained with minimal manual annotation.
- The experiment results are shown to be competitive compared to SOTA on three datasets.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The framework is not innovative enough.
- The native 2D freehand datasets contain only four/three sequences from two different types of the ultrasound machine.
- Some details about the method and experiments are confusing (see detailed and constructive comments).
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
average level.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
-
During the fine-tuning phase, data and labels from the training set are actually used, which is not called unsupervised training.
-
As the authors state, the displacement Dik can be parameterized as Li-Lk. Then it seems that the displacement prediction structure in the AdLocUI is unnecessary. The AdLocUI can still compute and constrain D from L. The experiment only proves the necessity of multi-task learning, not the structure of displacement prediction.
-
Do the scaling and in-plane translation in the enhancement modify Li simultaneously? If Li is not modified, the AdLocUI will predict the same position for images of different sizes. And how does the AdLocUI handle the various image sizes?
-
In the fine-tuning phase, how is the associated training sequence S selected for each native 2D freehand sequence I? Is the fine-tuning performance affected by the non-smoothness (e.g., Di1 and D3k in Fig. 2) of the connection between S and I?
-
Since Dik=Li-Lk, the symbols in Eq. 5 seem to represent summation.
-
Why are the results of the Yeung et al. method without fine-tuning in Table 1 different under the two settings? Is this due to the testing datasets in the two settings being different? If so, it is pointless to compare the degree of impact on the methods under the two settings.
-
Metric performance analysis is not feasible since the native freehand ultrasound sequences are not labeled. It’s worth experimenting with a compromise validation scheme, i.e., performance validation on unaligned 3D volumes.
-
The authors should give more details of the NSTD results, such as the NSTD standard deviation for different methods. Are the results truly valid when just three or four sequences are tested?
-
The authors believe that AdLocUI can be extended to other anatomical structures and developed as a general sensorless freehand ultrasound guidance tool. However, there is no theoretical or experimental support for this paper. The authors should change the statement.
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The authors design a novel cycle consistency of domain adaptation for 2D ultrasound image localization, but too few native 2D freehand sequences to be statistically significant.
- The experimental results show that the proposed framework is superior to the SOTA methods but lacks methodological innovation.
- Number of papers in your stack
8
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The authors propose a method to localize 2D standard planes in freehand ultrasound imaging. This is relevant and clinically useful, although perhaps authors could insist on why it is clinically useful. The main concern from the reviews is in the validation, using untracked freehand without ground truth. I would suggest to consider the reviews and try to reassure reviewers on this topic specifically, and in othercomments more generally.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6
Author Feedback
We thank all the reviewers for their constructive comments and suggestions. We have grouped our responses by topic.
External tracking sensor (R2, R3) Freehand ultrasound (US) scanning of the fetus is challenged by the fact that the fetus is not stationary, particularly before the third trimester. A tracking sensor, therefore, can only record the probe position but not the plane position due to the relative motion between the fetus and the probe. This limits the tracking sensors’ practical application and use for evaluation.
Clinical motivation (R1, R2, MR) (1) Locating standard planes with freehand US is highly operator-dependent, requiring extensive expertise and anatomical understanding. Our motivation is to facilitate the scanning and minimize inter-operator variability. “Training novice sonographers (R2)” is another motivation, which is particularly important in resource-limited settings. (2) We agree that 3D probe research is valuable. However, given the already-established protocols and practical constraints (e.g. cost, ease of use), it is foreseeable that 2D US will remain the mainstay clinically. Therefore, we believe improving its utilization experience, i.e. our motivation, is “clinically useful (MR)” and “has a wide scope for application (R3)”. (3) The 2 suggested papers will be included in the introduction.
Validation (R3, R4, MR) (1) We agree that the rate of change index (NSTD) alone is not sufficient but it is only 1 of the 3 evaluations presented. For freehand images, we also present qualitative results (Fig. 1 and GIF files in Supp. Materials), which directly demonstrate and compare AdLocUI’s performance to other methods. Although only 7 freehand sequences were tested, they contain over 1000 images, which is agreed by R3 as a merit, together with our detailed quantitative analysis on 2D slices sampled from “an adequate set of 3D volumes … (R3)”. (2) As discussed above, using a tracking sensor for evaluation may not be appropriate. We have considered manually annotating the 3D location of freehand images in the atlas and evaluating their distance with our prediction, but the annotation is very laborious. We hope to include that in our future work.
Table 1 (R1, R3) The voxel size (ED) is 0.6mm. The p values are orders of magnitude lower than 0.05, and remained significant after Bonferroni correction.
NSTD (R1, R4) NSTD is normalized by the mean of delta c. As this metric was designed to quantify the smoothness of transition between frames, it may not have an intuitive clinical interpretation, but is useful for inter-method comparison. We used it to supplement the qualitative results (Fig. 1 and GIF files in Supp. Materials).
Response to R4 (1) We used “unsupervised” in accordance with terminology used in unsupervised domain adaptation that no manual labels are used for the target domain data but can be used in the source domain data (i.e. our training set). (2) Predicting Dik is necessary as an auxiliary task to facilitate our proposed cycle consistency fine-tuning. If only L is predicted and used to obtain/constrain D (instead of being predicted individually as we propose), during fine-tuning, L.H.S and R.H.S of Eq. 5 will always be exactly the same and no retraining is possible. (3) Scaling and in-plane translation will not change Li as the position of the corresponding cross-sectional plane in the atlas is unchanged. We want AdLocUI to be invariant to them as they are irrelevant to our task (i.e. freehand scanning guidance). (4) During fine-tuning, S is sampled randomly and their order is permuted to make AdLocUI invariant to non-smoothness. (5) The difference of Yeung et al w/o fine-tuning under 2 settings in Table 1 is due to different testing data. We list both to avoid ambiguity if emptying either from the table. We will clarify in the paper. (6) We agree our claim is not fully justified and will edit it to “extension to other structures will be further studied in the future work”
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
All reviewers except for one were on the accept side before rebuttal. The most critical reviewer was concerned about application and validation. Authors addressed both in their rebuttal, and that reviewer changed his recommendation to weak accept. All reviewers now agree on acceptatnce.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
I would like to thank the authors for clarifying the issues raised by the reviewers. The idea of using unsupervised cycle consistency is interesting and will facilitate the use of the network on different ultrasound machines. I suggest that the authors add a few lines to the revised paper to incorporate their response on these two topics: (1) external tracking sensor and (2) validation. In addition, there are two MICCAI papers that used unsupervised cycle consistency in ultrasound images that the authors may consider to cite: Tehrani, A, et al. “Semi-supervised training of optical flow convolutional neural networks in ultrasound elastography.” MICCAI 2020 Delaunay, R, et al. “An unsupervised approach to ultrasound elastography with end-to-end strain regularisation.” MICCAI 2020
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper presents a method or ultrasound standard plane localization in freehand ultrasound seuquences relying on a deep learning approach and an atlas. Despite some initial questioning on the clinical motivation and the validation, after rebuttal there seems to be agreement among the reviewers for acceptance. The lower ranking in my pile is because I cannot clearly identify the novel aspects from reviews and rebuttal.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
8