Authors

Qianhui Men, Clare Teng, Lior Drukker, Aris T. Papageorghiou, J. Alison Noble

Abstract

Eye trackers can provide visual guidance to sonographers during ultrasound (US) scanning. Such guidance is potentially valuable for less experienced operators to improve their scanning skills on how to manipulate the probe to achieve the desired plane. In this paper, a multimodal guidance approach (Multimodal-GuideNet) is proposed to capture the stepwise dependency between a real-world US video signal, synchronized gaze, and probe motion within a unified framework. To understand the causal relationship between gaze movement and probe motion, our model exploits multitask learning to jointly learn two related tasks: predicting gaze movements and probe signals that an experienced sonographer would perform in routine obstetric scanning. The two tasks are associated by a modality-aware spatial graph to detect the co-occurrence among the multi-modality inputs and share useful cross-modal information. Instead of a deterministic scanning path, Multimodal-GuideNet allows for scanning diversity by estimating the probability distribution of real scans. Experiments performed with three typical obstetric scanning examinations show that the new approach outperforms single-task learning for both probe motion guidance and gaze movement prediction. The prediction can also provide a visual guidance signal with an error rate of less than 10 pixels for a 224x288 US image.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16449-1_10

SharedIt: https://rdcu.be/cVRUQ

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The authors propose a multimodal neural network to jointly predict the change in gaze position on the ultrasound image and the movement applied on the probe by a human operator to reach each of the many standard planes during a routine obstetric sonographic scan. The aim is to use gaze information to suggest probe motion, and viceversa, during training of new specialists. The performance improvement of the method over single-task networks is consistently shown by the experimental evaluation, which was performed on a dataset which is not mentioned to be public. The methodology used for data collection is however very well explained, such that replication should be feasible.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

In my estimation, this is a great paper overall. Well written, informative and innovative. The proposed method can be useful in the real world, shows improved performance over the state of the art and may inspire further innovative research.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

It is not completely clear to me if the paragraphs in the Methods section explain existing ideas that have been incorporated into the proposed system, or if they are also contributions. It would be nice if this distinction were made more clear.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The dataset is not mentioned to be publicly available, but the methodology for data collection and for evaluation are very clearly documented.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

Hard to find room for improvement here, at least for me. I am not sure if there are any theoretical contributions (see previous comments), but the innovative approach already makes for a good paper on its own.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This work presents an innovative problem formulation, with measurable real world implications and that could be applied to further scenarios. I believe this idea deserves being advertised within our community.
Number of papers in your stack

4
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

This paper explores a completely new paradigm to use Ai techniques to assist less experienced operators of obstetric ultrasound to obtain high quality results. It achieves its goal through the novel combination of probe and the gaze of the sonographer, and results demonstrate that such a joint approach outperforms single-task learning for both probe motion guidance and gaze movement prediction. This is a very tangible and practical first step towards making ultrasound more universally available as a diagnostic imaging tool, even in the hands of non experts. For me, this paper embodies the spirit and raison d’etre of the MICCAI Society perfectly.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This approach is entirely novel – and as far as I am aware, this is the only group pursuing these ideas. While the use of eye-gaze has in the past been used as part of skills assessment for sonographers, this is the first time it has been used as part of a dataset to train a network. The key point is the use of gaze and probe movements as random variables to account for inter- and intra-sonographer variation. The underlying assumption is that a sonographer will react to the next image inferred from their hand movement on the probe, and conversely that the probe motion can guide gaze. To this end, the authors have developed a platform “Multimodal-GuideNet” that observes scanning patterns from a large number of actual obstetric scanning studies where probe motion and gaze trajectory data have been collected along with the US images.

The authors have shown convincingly that the model can generate real-time predictions to guide the operators, based on the probe motion and gaze trajectory signals.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

In figure 4, Multimodal-GuideNet* reports the error of the best generated gaze pointthat is closest to ground truth for the Multimodal-GuideNet procedure. It would be helpful to see this parameter reported for the Gaze-GuideNet procedure as well.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

No particular issues
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

Can the authors make some comment on the practical significance in terns of diagnostic accuracy of the improved error rate of Gaze vs Multi-modality versions of the network? It is not clear what the significance of the statement that “the error of Multimodal-GuideNet* is within 10 pixels” is.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

8
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

THis is a ground-breaking example of the effective use of AI to facilitate the use of Ultrasound in under-resources communities. Very exciting work with an immediate practical application
Number of papers in your stack

4
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

This paper presents a multi-task learning framework to predict gaze position and US probe rotation for guiding obstetric US scanning, which is a novel approach to tackle this kind of problems.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This paper explores a novel idea to leverage the gaze information to guide the US probe. The idea is nicely formulated in a multi-task learning framework where graph convolutional network is used. The paper is scientifically sounding and also well structured and written. The experiments were thoughtfully designed with results clearly presented.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Not much weakness from my perspective, though the paper is a bit compact with a lot of details not well elaborated, but I guess mostly due to the page limit of the paper.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The experiments design and data generation are clearly described in the paper, which matches what the authors claimed in the reproducibility checklist.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

In the experiment, all data were downsampled to 6 Hz. What is the rationale for it? Is it due to the computational constraints?
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper presents a novel idea with scientifically sounding formulation of the problem. The paper is also very well written and structured.
Number of papers in your stack

4
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Somewhat Confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The authors propose a neural network to jointly predict the change in gaze position on the ultrasound image and the movement applied on the probe by a human operator to reach each of the standard planes during obstetric scans. The aim is to use gaze information to suggest probe motion, and vice-versa, during training of new specialists. The paper is well-written and easy to understand.

However, there are still issues with the paper that should be addressed before publication. The issues are outlined in detail in the reviewers’ comments, but here is a summary. First, it is not clear in few places if the authors are explaining their contributions or referring to previous work. It would be nice to add to figure 4 the results of Gaze-GuideNet procedure. Finally, improve the clarity of some noted parts, even though it is challenging given the page limit.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

1

Author Feedback

We appreciate the reviewers and area chairs for their positive assessment of this work. Here are the responses related to your comments: • “it is not clear in few places if the authors are explaining their contributions or referring to previous work”: Spatially, a multi-modality graph is newly proposed that sharing information among US video, probe motion, and gaze trajectory. Temporally, a bidirectional pathway is newly proposed to model the temporal correlation between probe and gaze movements within the standard GRU cell. The network backbone is formed by graph convolutional GRU [1] to allocate useful dependencies in the designed multi-modality graph. The contributions and notions will be clarified in Methodology.

• “It would be helpful to see this parameter reported for Gaze-GuideNet…”: The result of Gaze-GuideNet* is supplemented as follows. |Error in Pixel—————-| ——- TVP ——- | ——- ACP——- | ——- FSP ——- | —– Overall ——| |———————————| Coarse —- Fine | Coarse —- Fine | Coarse —- Fine | Coarse —- Fine |
|Gaze-GuideNet* ———-| 6.36 ——– 7.81 | 5.21 ——– 6.53 | 6.18 ——– 5.52 | 5.92 ——– 6.62 | |Multimodal-GuideNet* —| 5.62 ——– 7.12 | 4.42 ——– 6.41 | 5.57 ——– 5.79 | 5.20 ——– 6.44 |

• “… the practical significance of diagnostic accuracy…”: Practically, Multimodal-GuideNet* would be useful when a precise gaze is needed such as when the sonographer focuses over a small range of underlying anatomical structure (mainly during coarse adjustment), and its improvement over Gaze-GuideNet* indicates probe guidance would potentially help locate such a fixation point. The results and justifications will also be reported in the experiments.

• “In the experiments, all data were downsampled to 6Hz..”: The sampling rate is chosen empirically based on the US video frame rate while preserving temporal properties of scanning.

[1] Li et al. Gated Graph Sequence Neural Networks. ICLR’16

back to top

Multimodal-GuideNet: Gaze-Probe Bidirectional Guidance in Obstetric Ultrasound Scanning