List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Kaiming Kuang, Li Zhang, Jingyu Li, Hongwei Li, Jiajun Chen, Bo Du, Jiancheng Yang
Abstract
3D reconstruction of pulmonary segments plays an important role in surgical treatment planning of lung cancer, which facilitates preservation of pulmonary function and helps ensure low recurrence rates. However, automatic reconstruction of pulmonary segments remains unexplored in the era of deep learning. In this paper, we investigate what makes for automatic reconstruction of pulmonary segments. First and foremost, we formulate, clinically and geometrically, the anatomical definitions of pulmonary segments, and propose evaluation metrics adhering to these definitions. Second, we propose ImPulSe (Implicit Pulmonary Segment), a deep implicit surface model designed for pulmonary segment reconstruction. The automatic reconstruction of pulmonary segments by ImPulSe is accurate in metrics and visually appealing. Compared with canonical segmentation methods, ImPulSe outputs continuous predictions of arbitrary resolutions with higher training efficiency and fewer parameters. Lastly, we experiment with different network inputs to analyze what matters in the task of pulmonary segment reconstruction. Our code is available at https://github.com/M3DV/ImPulSe.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16431-6_47
SharedIt: https://rdcu.be/cVD63
Link to the code repository
https://github.com/M3DV/ImPulSe
Link to the dataset(s)
N/A
Reviews
Review #2
- Please describe the contribution of the paper
This paper describes the definitions of the anatomy of pulmonary segments and the definition of dice score in segmentation metric. This paper also provides a convolutional deep network called ImPulse for the segmentation task
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The topic is interesting and can provided useful info for the surgical treatment
The ImPulSe network have the potential to generate high resolution output when the input resolution is low.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The contribution is consider limited. The introduced definition is not novel.
The efficiency of the ImPulSe network is very convincing. Using the dice score is not very sufficient for the reconstruction task.
Labeling the pulmonary segments should be a challenging part of the task. A induction of the labeling procedure and ground truth qualify will be preferred.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Without the medical data and label, this work is hard to be reproduced.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Introducing the definition of anatomy of pulmonary segments and dice score is not considered as novelty. There are several MICCAI papers introduced the pulmonary structure in the last two years.
The experiments need to include more comparison with more recent methods.
The ImPulSe network uses less parameters comparing to UNet. but it might have a bigger feature space. I am a little concerned about the efficiency.
The boundary of the pulmonary segments should have a clear anatomical definition. A geometry reconstruction is needed before the segmentation output can be considered in the surgical treatment.
The labeling the pulmonary segments should be a important part of the task, since there is not a lot open resource. The labeling procedure and the quality of the labels are expect to be introduced in detail.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Even though the topic is interesting, but the novelty of contribution is limited.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
The authors present a method for the automatic reconstruction of lung segments using deep learning. First the authors formulate the problem in a concise manner. Second they propose a deep-learning method for lung segment reconstruction that is not based on pixel-wise assessment. Instead, they authors probe different locations of the lung segment border and report a class for such continuous locations.
The authors show the performance of the method in a database of 800 CT scans from multiple medical centres with a data split 7:1:2. The proposed implicit method achieves better performance than end-to-end networks. Further experiments show the performance of the method when detecting bronchi and arteries. - Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The strengths of the paper are:
- Extremely well written.
- Very good problem statement.
- Thorough analysis of the problem.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
While the introduction and the title refer to pulmonary segment detection, the authors go beyond that, and also detect arteries and veins. This should be stated from the beginning. It comes as a surprise in the result section. The main justification of the method is the “ability to output reconstruction at arbitrary resolutions”, with the side nice features of having less parameters and faster training time. However, I should be critical with the method. The authors have a continuous function that they use to define the surface of lung segments. However, the features used as input of such continuous function are obtained by interpolating the feature space, which has the same resolution as the input image (and therefore the output image of standard methods, such as the u-net). It seems as if the authors are doing interpolation in feature space rather than interpolation in the resulting output image. Following in that train of thought, Figure 2 is not a fair comparison between voxel-based methods and implicit functions, since the gaps between voxels makes them look artificially bad. Also, in the illustrative 2D example, a simple interpolation in the voxel space would have given a soft border. The authors up sample the reconstructions using nearest-neighbour interpolation. What would have happened if the authors used other interpolation methods (as trilinear, as used for the features)?
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
This paper can only be reproduced with the open dataset. The method seems easy to be reproduced.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The idea of doing interpolation on feature space instead of in the segmentation space is interesting and has value. However, the experimentation performed are not a fair comparison between the two methods due to the nearest neighbour interpolation in output space and trilinear in the feature space.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I really like the rigorous problem statement and the idea of the continuous border assessment.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #4
- Please describe the contribution of the paper
The authors propose an implicit-function-based model for the pulmonary segment reconstruction, that makes the pulmonary anatomical lobe segmentation further. The anatomy of pulmonary segments are defined. Then, the implicit pulmonary segmentation (ImPulSe) is proposed. In the experiments, the ImPulSe achieves better performance and uses less training time, comparing to the fully-convolutional methods like UNet.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper makes the pulmonary anatomical segmentation further, from lobes to segments. The method will benefits for clinical applications, like surgical planning for patients with lung cancer. The method ImPulSe achieves better performance comparing to fully-convolutional methods.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) In section 2.3, it states that the decoder predicts 19-class (including 18 segments and background). However, the Dice of pulmonary segments, bronchi and arteries are shown in the evaluation results. Does the model predict bronchi and arteries as well? It’s confusing, please clarify it. 2) In section 3.3, the authors evaluate the effect of different inputs in reconstruction of pulmonary segments. It’s confusing by when given inputs of bronchi and arteries, the validating the predicting of bronchi and arteries. 3) In Table 2, predicting pulmonary segments from the inputs of lobes mask (or binary mask of bronchi and vessels) doesn’t make sense. I suggest to try with I, I+L, I+B, … 4) During inference, ImPulSe works with the original resolution of CT images, however, the fully convolutional models work with down-sampled images (128^3) and up-sample the predictions back to the original resolution. Will the down-sampling and up-sampling operation affect the results? Why do the authors not try with sliding-window strategy, for fully convolutional models?
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The reproducibility of the paper is OK
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
In this study, the authors annotated pulmonary segments, bronchi, arteries and veins for 800 CT scans. Making the data open access or organizing a challenge will definitely enhance the impact of this study.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The study has clearly clinical impacts, however, some details need to be clarified
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This work presents a deep learning approach for pulmonary lobe segmentation. This is a challenging task that has not yet been dealt with a lot using convolutional neural network based approaches. However, the current work does neither discuss existing non deep learning based methods, nor does it do a thorough job identifying CNN methods for segmenting lobes (e.g. Gerard and Reinhardt, ISBI 2019) or the similar problem of detecting fissures (Gerard et al., TMI 2018), which could be a foundation for lobe segmentation.
Nevertheless, reviewers agree that the work is well written with a thorough problem statement, which raises its interest for the MICCAI community. There are a number of weaknesses mentioned in the reviews: (1) A lack of comparison with recent methods, but also with traditional (i.e. non deep learning based) lung lobe and artery/vein/bronchi segmentation methods. (2) There is confusion about the problem setup being tailored to lung lobes, but then experiments include arteries and veins and bronchi as well. Related to this, there is confusion regarding a 19 class loss, which is then not used with the Dice scores. (3) Qualitative results show favorable outcomes of ImPulSe compared with other methods like UNet, however this is also due to a nearest neighbour upsampling of UNet results to the corresponding resolution of the ImPulSe result. This is an unfair comparison as stated by two reviewers.
Overall, the merits of this work seem to slightly outweigh the weaknesses, and it seems clarification may be possible through a rebuttal and a revision of the paper. Please carefully address the reviewer comments in your rebuttal and especially focus on the issues raised above.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5
Author Feedback
We appreciate high-quality reviews from the (meta-)reviewers (R2, R3, R4, MR1). The primary concerns can be summarized as follows.
For short, PS=pulmonary segment(s).
Problem settings (MR1&R4) There are misunderstandings about our problem and experiment settings. First and foremost, our paper deals with the automatic reconstruction of pulmonary segments, NOT pulmonary lobes (MR1). We will discuss more about previous works (Gerard et al., TMI 2018, Gerard and Reinhardt, ISBI 2019) of pulmonary structures in our revision. ImPulSe only predicts PS. We train extra models to reconstruct bronchi, arteries and veins so that ImPulSe can take them as inputs, while these models are not the primary focus of our study.
Evaluation metrics (MR1&R2&R4) Reconstruction of PS cannot be regarded as standard voxel-to-voxel segmentation, because the segmental boundaries are valid as long as they separate correctly the segmental branches of bronchus and artery tree (Sec 2.1). One of our contributions is to define how to compute the dice that can represent the reconstruction quality of PS, rather than the dice itself (Sec 2.2). A high-quality PS reconstruction leads to 1) high dice between human annotations of PS (Dice_o); 2) bronchus/artery voxels are separated into correct PS (Dice_b & Dice_a), where we calculate dice only on voxels labeled as bronchi/arteries instead of the entire volume, i.e., whether each bronchus/artery voxel is correctly labeled into any of the 18 segments.
Comparison (MR1&R2) There are few deep learning studies investigating reconstruction of PS. Therefore, we have compared with popular segmentation methods (FCN, UNet, DeepLabv3) in Tab 1. As for non DL methods to reconstruct PS, we tried to reimplement the core idea of Rikxoort et al., TMI 2009, which uses relative coordinates in the lung to reconstruct PS. Due to the lack of official open-source code, we use an ImPulSe-like neural network instead of LDC in Rikxoort et al. The performance is much lower than ImPulSe: Dice_o | Dice_b | Dice_a 60.63 | 66.01 | 67.54
Unfair comparison due to nearest-neighbor interpolation (MR1&R3&R4) The evaluation based on bronchi/arteries is not dependent on pulmonary segment boundaries, therefore the interpolation is not a key. Nevertheless, we tried trilinear interpolation on FCN, UNet, DeepLabv3, whose performance is improved but still lower than ImPulSe: Model | Dice_o | Dice_b | Dice_a FCN | 81.70 | 83.67 | 84.92 U-Net | 83.25 | 84.71 | 86.07 DeepLab | 82.44 | 84.48 | 85.64 ImPulSe | 83.54 | 85.14 | 86.26 Please note that these are fair comparison as all models take the same inputs.
Open access of dataset (R2, R3, R4) We agree that the annotation of PS, bronchi, arteries and veins for 800 CTs is challenging (~2h per sample), and the open access of the dataset would increase the impact. However, restrictions apply to the dataset. We will try to make part of the dataset publicly available in the future extension.
- Method novelty (R2) Our major contributions are two-fold:
- “Problem statement and thorough analysis of pulmonary segment reconstruction” (R3), and appropriate evaluation metrics based on dice to represent the reconstruction quality, rather than dice itself (more in answer #2).
- A deep implicit surface model (ImPulSe) introduced in the medical imaging domain with proven effectiveness.
- Sliding window (R4) As discussed, the reconstruction of PS needs global information in the lung, thus the sliding-window approach is not expected to work. To address the concern, we trained a U-Net on cropped cubes (64^3) with an nnUNet-like pipeline. This sliding-window model totally collapsed. Dice o | Dice b | Dice a 0.11 | 0.05 | 0.02
Other issues: a) Dataset labeling (R2): Dataset annotations are made with Materialise Mimics 19 following [16,5]. b) Figure 2 (R3). It is not a comparison but a visualization to illustrate the difference between voxel and implicit functions. We will clarify it in the revision.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Authors clarified several misunderstandings of the reviewers and the meta reviewer. The key strengths of the work are an approach for pulmonary segment extraction, an evaluation metric that takes into account the definition of pulmonary segments, which differs from segmentation metrics for more precisely defined structures like pulmonary lobes, and the thorough evaluation on a tediously annotated dataset. Given the clarifications, the merits of this work strongly outweigh the weaknesses. It is suggested that in case of acceptance, the comparison to the better performing comparison methods (involving trilinear interpolation) should be added to the revised paper.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
I found this an interesting paper to read. I agree with reviewer concerns as to the “surprise” of segmenting bronchi, so this warrants some effort by authors in clarity. However, the authors here make some valuable contributions as to (1) the evaluation of PS segmentation and (2) a refreshing approach based on implicit surfaces that tackles the problem in a new way, that is much more novel than just proposing a “new” FCN. For this reason, I did not agree with reviewer concerns as to novelty.
As to (2) above, the performance is better than FCNs, but only by small margins in terms of mean metrics (~1% DSC over DeepLab in the rebuttal, which may be statistically significant, but is likely not clinically significant ). I highly recommend authors use box and whisker plots or some other way to present performance that shows worst-case performance/robustness, as this might better reveal improvements than mean metrics. Even with this caveat, I view this paper as a clear accept, given the novelty of the approach, the ability to naturally produce high-quality surfaces, and the discussion of evaluation metrics.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
1
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors propose an implicit-function-based model for the pulmonary segment reconstruction. Reviewers agree that the paper is well written, clearly motivated, but raised a number of questions including novelty and experiments comparison. The authors have clarified most of the points in rebuttal.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5