Authors

Ziqiao Weng, Jiancheng Yang, Dongnan Liu, Weidong Cai

Abstract

Accurate segmentation of pulmonary airways and vessels is crucial for the diagnosis and treatment of pulmonary diseases. However, current deep learning approaches suffer from disconnectivity issues that hinder their clinical usefulness. To address this challenge, we propose a post-processing approach that leverages a data-driven method to repair the topology of disconnected pulmonary tubular structures. Our approach formulates the problem as a keypoint detection task, where a neural network is trained to predict keypoints that can bridge disconnected components. We use a training data synthesis pipeline that generates disconnected data from complete pulmonary structures. Moreover, the new Pulmonary Tree Repairing (PTR) dataset is publicly available, which comprises 800 complete 3D models of pulmonary airways, arteries, and veins, as well as the synthetic disconnected data. Our code and data are available at \url{https://github.com/M3DV/pulmonary-tree-repairing}.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43990-2_36

SharedIt: https://rdcu.be/dnwLR

Link to the code repository

https://github.com/M3DV/pulmonary-tree-repairing

Link to the dataset(s)

https://onedrive.live.com/?authkey=%21AEq1v5hZHJORzRA&id=66346B2D10575CA6%21252787&cid=66346B2D10575CA6

Reviews

Review #3

Please describe the contribution of the paper

Authors present a deep learning approach to repair the topology of disconnected pulmonary tubular structures. They do so by creating a dataset by deliberately creating disconnections and the topology and then predicting disconnected points as key points using 3D-UNet.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

1) Authors target a novel and a very important clinical application. 2) Authors create a novel dataset which will be useful for a lot of applications requiring accurate segmentations of vessels and airways. 3) The dataset used in this study would be made public and hence would be helpful to drive the research forward in obtaining most accurate segmentations of pulmonary tubular structures.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

1) The approach used in the study to solve a novel application is very preliminary 2) As authors state in the paper a lot of assumptions have been made
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Authors have agreed to make the code and data public.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

Although the approach used in the paper sets a baseline, many improvements could have been made.

1) Although a baseline studies, ablation studies have to be conducted. Authors could have compared the sensitivity of different network architectures. How does some of the state of the network architectures such vision transformers compare ? Sensitivity towards patch size ? Optimal size of gaussian kernel to depict key points ?

2) Authors should provide more details of the dataset. What population of dataset was used ? All healthy patients ? Or patients with abnormalities ? Resolution of the dataset in x,y,z plane ? Were the scans converted to being isotropic ?
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Novel application and dataset. However critical details of the dataset is missing and approach used is very preliminary.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #1

Please describe the contribution of the paper

Disconnected tree segmentations from input medical images are commonplace. The key contributions of the paper are a) a promised large database of pulmonary CT scans with manually segmented arteries, veins and airways and b) a method to connect disconnected binary segmentations by detecting pairs of key-points, where key-points are defined as the centreline of the tree where the disconnection happened.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The main strength of the paper is the novel formulation of the problem. Standard methods for tree completion are often based in graph theory or segmentation completions using deep learning. The proposed strategy of key-point detection is novel, and worth a thought. The large dataset, of 800 CT scans with segmented pulmonary airways, arteries and veins, if made publicly available, has a lot of value.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

As the authors acknowledge, the deep learning models trained “may not be applicable to real-world data”. The input to the deep learning method are paired volumes, where one volume contains the location of the key-point of the disconnected component and the second one contains the key-point of the main connected component. How are those volumes selected in a real case? The authors propose a sampling method in the appendix, but its evaluation is not clearly performed. How would the authors handle a plurality of disconnected components? It may be the case that small component 1 is connected to small component 2, and the joint component to the main one. With their current algorithm, they will always connect small component 1 to the main component and small component 2 to the main component. Further, the problem at hand is not a single tree reconstruction, but three parallel reconstructions. Disconnected components may belong to veins, arteries or airways, being veins and arteries easily mistaken. Given a disconnected component, should it be paired with the component of arteries or with that of the veins? It is unclear from the text how to handle such situation. Several typos appear on the text, as two equal signs in equation (1) or “detailed”. I suggest a thorough review.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

If the data is made available, I see no problem with the reproducibility of the method.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

The proposed method answers a key question: is this sub-volume connected with this other sub-volume, and if so, where is the connection? It is a great building block for further processing of tree structures.

It would be interesting to have metrics in how well the tree is reconstructed after key-point detection and how would that compare with, let’s say, a method that outputs a reconstructed mask. The reconstructions in Fig. A1 are impressive, as the method seems to connect the disconnected component in a geometrically consistent manner.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The main factors that led to my overall score are: • The immense value of the open dataset • The originality of the method, that can serve as a building block of more comprehensive tree reconstruction methods.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

This paper proposes a method to detect keypoint for the topology recovery in pulmonary vessel trees. Also, a dataset with a variety of real and synthetic data is created and will be made publicly available.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This paper addresses an important but under-studied problem for medical image processing: Reconstruction of topologically correct tree structures. Especially the dataset is useful.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Description of the method is unclear at several critical places.
- The proposed segmenation method is not compared with other methods, thus the algorithmic contribution is hard to judge.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors plan to release their dataset. Before that, the results are not reproducible, since both original data and the method for adding discontinuities into the data are not yet available.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- Some description is unclear. For example, in Sec. 2.2 the authors state “The keypoints were then subjected to morphological operations”. This is much too vague. Is it erosion around the key points?
- The description of the method in Sec. 2.3 is based on the notion of KP_1 and KP_2, which are introduce at the end of Sec. 2.2. However, it is still quite unclear what KP_1 and KP_2 are. This makes the understanding of the subsequent sections very difficult.
- Although discontinuities in vessel trees are a common problem, in this paper discontinuities are generated artificially. How severe are the discontinuities in real scenarios, and how realistic are the artificially added discontinuities? For example, in Fig. 2b there is a huge discontinuity in the main pulmonary artery (sub-figure under the caption “disconnected patch, R=9.3”). What would cause such discontinuities in reality?
- It is hard to recognize how the description in Sec. 2.4 correlates with the pseudo-code in the supplementary material. At least the pseudo-code should be in the paper itself.
- Although the results in the supplementary material look promising, the experiments are still too limited, as there is no comparison with other methods. Also, it would be better to put Fig. A1 into the paper itself, not the supplementary material.
- In Fig. 1 there are pre-extracted centerlines. Are these centerlines part of the manual annotation that comes with the dataset, or are they computed from the 3D volume?
- Are you going to add more information (such as location of bifurcations) which is valuable for the study of topology? That would make the dataset even more useful.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The dataset is valuable, as it deals explicitly with the common problem of discontinuities in vessel / airway trees. However, the proposed algorithm is not promising, since the description is unclear, and the experimental comparison with other methods is missing.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This work proposes a UNet based network to repair disconnected airway and vessel trees from thorax CT segmentations, as well as a large manually annotated dataset of such structures that was used to synthesize disconnections, such that the UNet could be trained to connect branching points (or any points along the skeletonization). The network performs heatmap regression to implement this connection. The two major strengths of this work are: (1) a novel idea how to think about the problem of disconnected tubular structures, as frequently encountered in machine learning based segmentation approaches, and (2) a valuable dataset of 3D models of these structures which is promised to be made available to the public.

An important weakness of this work is that it is unclear how to use the trained connection model in real world scenarios. Since in practice, it is not known if segmentation results are false positives or not, connecting those to main branches may be a problem. Furthermore, specifically in this application of airway, artery and vein reconnection, reconnection candidates might come from the wrong vascular tree, especially when thinking about arteries and veins which are not distinguishable from grey value alone. Thus, as the authors also mention, it will be hard to use this baseline in practice. However, I see the possibility, that it might be used as a building block in a higher level reconnection algorithm.

Another shortcoming is the limited discussion of heatmap based keypoint prediction methods in medical image analysis, all of the cited works are from the computer vision literature, although there is a large body of work in medical applications using heatmap regression of keypoints and landmarks, which should be mentioned as well in a revised version of the paper.

Overall, I see potential in this proposed method such that others could build upon it, and I find the published dataset extremely valuable. To be valuable for the community I would suppose that the published data is the 3D models of airways, arteries and veins, including its corresponding CT datasets, however, this is not entirely clear from the description in the paper and should be refined.

Author Feedback

Dear reviewers and meta-reviewers,

We appreciate the invaluable and constructive feedback provided by all the reviewers. Rest assured that we will duly address all the concerns raised in the camera ready of the paper. This entails adding further explanations, correcting any typographical errors, and carefully reviewing the paper’s structure based on the reviewers’ comments. Additionally, we will provide detailed clarifications on specific significant concerns, as outlined below:

Dataset: Our final dataset will encompass the 3D models (binarized ground truth segmentation masks), centerlines, disconnected volumes, and a corresponding graphml file for each subject. Each graphml file will provide comprehensive information, including the coordinates of all bifurcations and endpoints, as well as the coordinates of all points along each branch, capturing a diverse range of characteristics specific to each blood vessel. The specifics of the dataset will be disclosed upon its release as an open-source resource.

Practicality of our method: Regarding investigating more complex scenarios, such as tackling the challenge of multiple disconnected components, differentiating between arteries and veins, and implementing our method in real-world settings, we will leave these aspects for our future investigation. Additionally, our future work will encompass a thorough exploration of ablation studies involving a variety of network architectures and hyperparameters.

back to top

Topology Repairing of Disconnected Pulmonary Airways and Vessels: Baselines and a Dataset