Authors

Chong Wang, Daoqiang Zhang, Rongjun Ge

Abstract

Multi-organ segmentation of the abdominal region plays a vital role in clinical such as organ quantification, surgical planning, and disease diagnosis. Due to the dense distribution of abdominal organs and the close connection between each organ, the accuracy of the label is highly required. However, the dense and complex structure of abdominal organs necessitates highly professional medical expertise to manually annotate the organs, leading to significant costs in terms of time and effort. We found a cheap and easily accessible form of supervised information. Recording the areas by the eye tracker where the radiologist focuses while reading abdominal images, gaze information is able to force the network model to focus on relevant objects or features required for the segmentation task. Therefore how to effectively integrate image information with gaze information is a problem to be solved. To address this issue, we propose a novel network for abdominal multi-organ segmentation, which incorporates radiologists’ gaze information to boost high-precision segmentation and weaken the demand for high-cost manual labels. Our network includes three special designs: 1) a dual-path encoder to further integrate gaze information; 2) a cross-attention transformer module (CATM) that embeds human cognitive information about the image into the network model; and 3) multi-feature skip connection (MSC), which combines spatial information during down-sampling to offset the internal details of segmentation. Additionally, our network utilizes discrete wavelet transform (DWT) to further provide information on organ location and edge in different directions. Extensive experiments performed on the publicly available Synapse dataset demonstrate that our proposed method can integrate effectively gaze information and achieves Dice similarity coefficient (DSC) up to 81.87% and Hausdorff distance(HD) reduction to 11.96%, as well as gain high-quality readable visualizations.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43990-2_3

SharedIt: https://rdcu.be/dnwLd

Link to the code repository

https://github.com/code-Porunacabeza/gaze_seg/

Link to the dataset(s)

N/A

Reviews

Review #2

Please describe the contribution of the paper

The paper presents a novel network for abdominal multi-organ segmentation that leverages radiologists’ gaze information to improve segmentation precision and reduce the need for expensive manual labeling. The network incorporates a dual-path encoder, a cross-attention transformer module (CATM), and multi-feature skip connection (MSC). The proposed method was evaluated on the Synapse multi-organ segmentation dataset and demonstrated good performance.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The paper presents a novel eye-guided multi-organ segmentation network, which outperforms baselines. The paper is clearly written and easy to follow.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

I couldn’t find a significant weakness.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper is written clearly and seems reproducible. I will suggest the paper release the code.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

The paper is well-written and easy to follow. It might be helpful to experiment on more than dataset. I will suggest the paper release the code.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Technical novelty and result achieved.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

The authors propose a novel method to introduce human gaze as a supervisory signal for abdominal organs segmentation. The method consists of three novel modules: a dual path encoder to integrate the cognitive signal into the network, a cross attention module to further combine the supervisory signal and a multi-feature skip connection.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The main strengths of this paper are (1)introducing human gaze information as a supervisory signal (2) a novel method consisting of three special blocks to integrate this special supervisory signal into a segmentation framework - a dual encoder extracts information from the image data and gaze data separately, multi-feature skip connections which helps in combining spatial information from both the branches with the decoder features, cross attention module which interacts with both the image attention path and gaze attention path and captures the relevant features which are then used by the decoder. (3) Utilizing discrete wavelet transform (DWT) to provide additional information on organ location and edge in different directions. (4) Experimentation with the public dataset Synapse and showing that the method outperforms other methods with a significant decrease in Hausdorff Distance.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Lacks some implementation details such as hardware used, batch size, training epochs, hyperparameters etc. But since the reproducibility list mentions that authors will be releasing the code, the code release should be helpful to resolve this.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

According to the reproducibility list, the code will be released. The authors describe the network architecture in a detailed way. It seems that the paper would be reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

(1)Page 7: typo “Comparision with Existing Methods Performance.” (2)Discussion on why the method cannot outperform on gallbladder and spleen would be helpful. (3)explaining the abbreviation MSA on page 6 should be helpful
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The introduction of eye gaze as a supervisory signal and the method to integrate eye gaze and image information have good novelty.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #1

Please describe the contribution of the paper

The main contribution of the paper is the integration of gaze data in the segmentation of abdominal structures. A novel network structure that includes gaze data is introduced.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The main strengths of the paper is the use of gaze tracking data in the segmentation. The methodology seems to be a robust use of previously described approaches.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The main weaknesses is the description of the used data, how they were used and how the evaluation was performed. Also the amount of data used to train and test the model is limited.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

It might be possible to reproduce the methods described in the papers. However, the training, validation and testing is not described in sufficient details to be reproduced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

The introduction gives an ok introduction to the use of gaze tracking in segmentation tasks. Some references to state-of-the-art methods in abdominal organ segmentation would lift the paper. One example is the TotalSegmentator tool that is also offered as a 3D slicer plugin.

The article goes directly from the introduction to the methodology. The common reader would be greatly helped by a brief introduction to the data. Is it a 2D or 3D segmentation. How where the eye-tracking data acquired and was a slice-by-slice eye tracking?

Section 2.1 is difficult to understand. A wavelet transform is apparently applied to gaze tracking heat maps. Why is this necessary - why not just use the heat maps directly?

Section 3.1 describes the data but it is not clear how it was split into training, validation and testing?

Was the evaluation metrics computed on the full resolution of the original scans or on a downsampled version?

It is not clear what split of the data the method were tested on. The total data set only includes 30 CT scans which is very few compared to recent publications. For example:

Wasserthal, Jakob, et al. “TotalSegmentator: robust segmentation of 104 anatomical structures in CT images.” arXiv preprint arXiv:2208.05868 (2022).

In conclusion, an interesting approach and a good combination of previously described method. However, the used data and the final evaluation is not satisfactory.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

3
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I do believe that a more thorough description of the data and the test procedure is needed.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper proposes eye-guided network which contains dual-path encoder that integrates human cognitive information and cross-attention transformer module for multi-organ segmentation network using diverse abdominal organ images. It is an interesting work.

Author Feedback

N/A

back to top

Eye-Guided Dual-Path Network for Multi-organ Segmentation of Abdomen