Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews Back to top
List of Papers By topics Author List
Paper Info Reviews Meta-review Author Feedback Post-Rebuttal Meta-reviews

Authors

Alina F. Dima, Veronika A. Zimmer, Martin J. Menten, Hongwei Bran Li, Markus Graf, Tristan Lemke, Philipp Raffler, Robert Graf, Jan S. Kirschke, Rickmer Braren, Daniel Rueckert

Abstract

Automated segmentation of the blood vessels in 3D volumes is an essential step for the quantitative diagnosis and treatment of many vascular diseases. 3D vessel segmentation is being actively investigated in existing works, mostly in deep learning approaches. However, training 3D deep networks requires large amounts of manual 3D annotations from experts, which are laborious to obtain. This is especially the case for 3D vessel segmentation, as vessels are sparse yet spread out over many slices and disconnected when visualized in 2D slices. In this work, we propose a novel method to segment the 3D peripancreatic arteries solely from one annotated 2D projection per training image with depth supervision. We perform extensive experiments on the segmentation of peripancreatic arteries on 3D contrast-enhanced CT images and demonstrate how well we capture the rich depth information from 2D projections. We demonstrate that by annotating a single, randomly chosen projection for each training sample, we obtain comparable performance to annotating multiple 2D projections, thereby reducing the annotation effort. Furthermore, by mapping the 2D labels to the 3D space using depth information and incorporating this into training, we almost close the performance gap between 3D supervision and 2D supervision. Our code is available at: https://github.com/alinafdima/3Dseg-mip-depth.



Link to paper

DOI: https://doi.org/10.1007/978-3-031-43907-0_14

SharedIt: https://rdcu.be/dnwcc

Link to the code repository

https://github.com/alinafdima/3Dseg-mip-depth

Link to the dataset(s)

N/A

Reviews

Review #1

  • Please describe the contribution of the paper

    Paper introduces a methodology to segment 3D vessels. The novelty part of this methodology is optimizing the process of getting expert annotation, i.e. taking advantage of depth information with the aim of requiring expert annotation from a single 2D slice instead of the annotation of the whole volume for model training. Specifically, expert annotation of a single randomly selected 2D slice is mapping back to the 3D space via the depth of the maximum intensity projections (MIP), which becomes the input of a 3D U-Net. Application of this methodology is demonstrated on the task of peripancreatic vessel segmentation on CT images (n=141). Performance is computed and compared with baseline (input images to the U-Net: 3D annotations), obtaining competitive performance. Authors demonstrated statistical significance between using and not using depth information.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • It is interesting the novelty strategy to reduce the annotation effort to get vessel segmentation by using a single 2D slice and depth information.
    • A independent subset of data is used to test the proposed methodology It is notable the exhaustive experimentation by varying the input information to the 3D U-Net (using 2D annotations, using 3D annotations, not including depth information, including depth information, different fixed viewpoints, different data size).
    • Authors include a good background review, taking into account two main points: Learning from weak annotations and incorporating depth information
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    • Lack of clarity: authors described that the dataset contains binary 3D annotations of the peripancreatic arteries carried out by two radiologists. However, it is not clear if authors used one or the two expert annotations. In case both were used, the next question is how did authors use both annotations to compute the dice score to evaluate the obtained arterial segmentation?
    • Not clear how authors probed random selection of 2D slices was carried out, i.e, How was the 2D slice selected? authors said randomly, but ¿did they test the use of different slices and obtain the same results?
    • Abstract does not mention the used type of medical imaging and it falls short in describing methodology (just one sentence).
  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    Within supplementary material authors provide information about preprocessing steps and network architecture.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
    • Be careful about the use of expressions like “excellent” segmentation performance.
    • I suggest adding references when authors describe the importance of blood vessels segmentation in medical imaging.
    • Make the correction of the year for reference 5.
    • I suggest to not only run 1 iteration of 5-fold, at least 5 iterations, to avoid a bias due to data partitioning.
    • Compute Dice score between the two expert annotations could provide valuable information about the inter-reader variability and the relevance of making this task automatic.
    • It would be interesting if authors compare the obtained results with the results obtained by Koziński et al., the unique reported publication (conference paper) which has been reported results on the same task (peripancreatic vessel segmentation).
    • It would be useful if authors are more specific in the main paper to reference a particular table/figure in the Supplementary Material instead of referencing the whole document as “can be found in the supplementary material”.
  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    6

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Although the paper has some weaknesses, these are mainly related to lack of clarity and some experiments suggestions to be carried out to make the work stronger. In spite of these limitations, paper introduces a novelty strategy to optimize the process of getting expert annotation to automatically segment 3D arterial structures, i.e., taking advantage of depth information with the aim of requiring expert annotation from a single 2D slice instead of the annotation of whole volume for model training.

  • Reviewer confidence

    Very confident

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    6

  • [Post rebuttal] Please justify your decision

    First of all, thanks to the authors for your response. I’ll discuss your answers based on each of the review’s comments:

    1. Lack of clarity about dataset description and expert annotations. Regarding these two related comments, authors made a clarification by saying that althought two radiologist partifcipate in the study, ecah one only annotated a half of the dataset, reason why i) there are not two annotations per case and therefore ii) there is not possible to compute inter-reader variability.
    2. Lack of clarity with respect to the strategy of random selection of 2D slices. About this point, authors clarify that they tested 3 different strategies to select the 2D slices, and the best (randomly sampled orthogonal VPs) is the one reported in the manuscript.
    3. Suggestion to run not only 1 iteration of 5-fold (at least 5) to avoid a bias due to data partition. Authors said they performed 5-fold cross-validation paired with seed re-initialization to account for data and initialization variance. They also mentioned multiple cross-validation iterations would further reduce partitioning bias, but also increase the computational cost. Regarding this point, I consider authors should include these results since dispite of the computational cost, these results make the work stronger
    4. It would be interesting if authors compare the obtained results with the results obtained by Koziński et al., the unique reported publication (conference paper) which has been reported results on the same task (peripancreatic vessel segmentation). About this point, authors provide arguments and clarifications that should be placed in the paper.
    5. There were minor details that I hope authors address in the final version in case it is accepted: use of terms like “excelent”, reference to describe the importance of blood vessels segmentation in medical imaging, reference correction, include the type of medical imaging used in the study in the abstract, appropiate reference SM.



Review #2

  • Please describe the contribution of the paper

    The paper presents a novel approach to the automated segmentation of blood vessels in 3D medical images, addressing the challenges of manual annotation in 3D vessel segmentation. The authors demonstrate that it is possible to achieve accurate 3D vessel segmentation by annotating a single MIP per training image. This method substantially reduces the annotation effort required. They also show that the best annotation strategy is to label randomly chosen viewpoints, which minimizes the annotation cost. Additionally, by incorporating depth information obtained from 2D annotations, the authors almost close the gap between 3D supervision and 2D supervision.

    The potential impact of this work in the clinical environment is significant, as it can facilitate better visualization, assist in surgery planning, and enable the computation of biomarkers and other downstream tasks. This can ultimately lead to improved diagnosis and treatment of various diseases.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The main strength and novelty of this paper lies in the fact that, despite the time-consuming and economically burdensome nature of manually annotating 3D vessels, the study demonstrates the possibility of achieving performance comparable to 3D annotation-based algorithms by annotating single MIP views in 2D.

    The research demonstrates that it is feasible to segment the full extent of vessels in 3D images with high accuracy using only a single randomly selected 2D projection per training image. This approach significantly reduces the annotation effort and provides additional depth information from 2D annotations, exhibiting performance that is almost comparable to a 3D supervision-based approach.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    Although the author contends that by incorporating additional depth information obtained from 2D annotations at “no extra cost,” obtaining additional depth information appears to require a considerable amount of manual or semi-automatic effort, such as:

    • Automatically or semi-automatically extracting objects (like vertebrae and ribs) with similar intensities as arteries, which occlude the vessels along the depth direction.
    • Deriving vessel boundaries in the depth direction based on image intensity.

    Decomposing contrast materials in the depth direction based on intensity in contrast-enhanced CT scans is expected to be quite challenging, making it difficult to ensure reproducibility. Moreover, it is necessary to verify whether performance can be guaranteed when noisy depth estimation is present.

    Obtaining such depth information presents challenges, including the additional computational load required for forward-projection. Moreover, according to Table 1, the performance difference between 1VP+D vs random 1VP-D in terms of Dice is not as significant as the performance difference between 3D vs. 1VP+D, which limits its clinical feasibility.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    This study suggests that the reproducibility of performance can be ensured when the following two accurate annotations are secured:

    • Depth information: Even in spectral CT, decomposing contrast material in contrast-enhanced CT is a challenging task, and the intensity of the contrast becomes blurred and faint over time. Therefore, it is expected that many voxels with similar intensity to the contrast material will be encountered in noisy contrast-enhanced CT images depending on the image, and substantial manual work will be required to remove them, which may significantly impair reproducibility.

    • 2D annotations: While this study obtained 2D annotations by performing forward-projection from 3D annotations, securing the quality of 2D annotations in this study might not be easy if 2D annotations were performed from the beginning. For instance, as shown in Fig. 2 (g), directly performing 2D annotation would make it difficult to accurately segment small, separated contrast voxels. If the 2D annotations are noisy, the reproducibility is likely to be compromised as well.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

    1. Considering the complex 3D structure of arteries, it is possible that some arteries may be occluded by others in the depth direction. In estimating the depth, if a significant portion of arteries is obscured by other arteries or objects with similar intensity, please comment on the feasibility of ensuring segmentation accuracy for the obscured arteries (i.e., those without front surface information).

    2. It would be beneficial to have an analysis of performance variations depending on the sparsity or noise level of depth information.

    3. The authors have successfully demonstrated that the algorithm’s performance is maintained even when randomly selecting a VP. However, it is expected that there would be performance differences depending on the VP angle. Suggesting an optimal VP angle could be beneficial for ensuring reproducibility and practicality.

    [Minor]

    1. In Table 1, it is difficult to interpret the table contents based solely on the caption; thus, it would be helpful to provide additional clarification in the caption description. For example, it is not clear how the ‘3D’ experiment and ‘3D’ model differ from each other.
  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    5

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    As mentioned above, the method demonstrates originality; however, the performance improvement (+D vs -D) is not substantial, and the implementation appears to require non-negligible preprocessing or manual image processing efforts. Reproducibility and the assumption of uniform intensity in contrast may be difficult to generalize, leading to a recommendation of a weak accept.

  • Reviewer confidence

    Very confident

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    5

  • [Post rebuttal] Please justify your decision

    The author has provided responses diligently, however, it is still difficult to find their explanations on the manual preprocessing and computational cost of depth maps convincingly comprehensive. Moreover, there seems to be an insufficient benchmarking study against other methods, which led to this decision.



Review #3

  • Please describe the contribution of the paper

    The authors present a novel approach for 3D segmentation of peripancreatic arteries using sparse 2D annotations. Their method, which leverages single random orthogonal 2D annotations for each training sample along with additional depth information acquired at no extra cost, achieves nearly the same accuracy as fully supervised models trained on 3D data while significantly reducing annotation costs.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • Interesting idea that use only one annotated 2D projection per image for segmentation task
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    The clarity of this manuscript could be further improved as it is a bit difficult to connect Figure 1 and the main content in Section 3. Figure 1 is also confusing, e.g. I see two MIP projection blocks and arrows pointing to different blocks and not sure I understand what that means.

    Some expressions need to be further corrected, e.g., ‘MIP projections’ should be just ‘MIP’.

    I was expecting to see results comparing the proposed method with other competing methods, but only the same method with different settings were reported.

  • Please rate the clarity and organization of this paper

    Poor

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    Didn’t see the code or link contained in the submission, not sure how to comment on reproducibility.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

    It would be good to compare this method with other benchmark methods and improve the clarity of the descriptions, might be good to update a clear diagram that connect Section 3 main components with the current Figure 1.

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    3

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The clarity and the results may need some further major changes before the work can be published, which I don’t think can be done within the rebuttal period.

  • Reviewer confidence

    Confident but not absolutely certain

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Primary Meta-Review

  • Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The paper presented a novel approach for 3D segmentation of peripancreatic arteries using sparse 2D annotations. The method, which leverages single random orthogonal 2D annotations for each training sample along with additional depth information acquired at no extra cost, achieves nearly the same accuracy as fully supervised models trained on 3D data while significantly reducing annotation costs. The three reviewers also affirmed this is an interesting work. The paper will furtherly be improved. The issues include comparison ith other benchmark methods, improving the clarity of the descriptions, goodness to update a clear diagram that connects Section 3 main components with the current Figure 1, adding an analysis of performance variations depending on the sparsity or noise level of depth information, adding references when authors describe the importance of blood vessels segmentation in medical imaging, and some other details mentioned by reviewers. Please address these concerns in the final version.



Author Feedback

We appreciate the reviewers’ comments and insightful suggestions. They consider our proposed idea of training 3D segmentation models on single random 2D projections and incorporating depth information methodologically novel (R1, R2, R3). We show the possibility of achieving accuracy on par with models trained with full 3D supervision using a single annotated max intensity projection (MIP) (R2), with significant potential clinical impact (R2). Our exhaustive experimentation is appreciated (R1). Below we address the reviewer’s concerns.

Baselines (R1, R3) R3 criticized the lack of comparison to other works, while R1 wished for a comparison with [13]. Our method is validated under many supervision regimes: 3D annotations vs. various amounts of 2D annotations: 1, 2, or 3 projections from fixed orthogonal viewpoints (VPs), and 1 projection from random orthogonal VPs. We actually included [13] as a baseline - the setting with 1, 2, or 3 fixed VPs - with our own implementation due to a dataset change and necessary hyperparameter tuning, which we now better emphasize in the paper. To the best of our knowledge, other supervised or semi-supervised methods employing the same 2D annotations do not exist in the literature. Unsupervised methods for vessel segmentation are known to be inferior to supervised methods, as found e.g., by Galdran et al., Sci Rep 12, 6174 (2022), and thus unsuitable for comparison.

Methodology (R2) Preprocessing and computational cost of the depth maps The former is required to produce 2D projections regardless of the depth maps (no extra cost), while the latter is negligible (1s/img). Overall, the depth incurs negligible costs compared to 2D projections without depth, which we clarify in the manuscript.

VP selection strategy In our work we propose a simple yet very effective VP selection strategy: randomly sampled orthogonal VPs, which have a regularizing effect and allow for a reduction in annotation cost. We also investigated (not reported) non-orthogonal VPs (worse) and multiple randomly sampled VPs (no benefit over 1VP), and we will continue to investigate further.

Discussion on occluded vessels Information sparsity is indeed a challenge; segmenting 3D vessels from 2D annotations is ill-posed. Too many overlapping vessels in 2D projections would provide insufficient supervision; however we find vessel trees to be typically spread-out and relatively sparse in 3D, so the image contrast is sufficient to segment from very sparse annotations. The heterogeneity in vessel size is an even bigger challenge induced by sparsity, and our key insight is that randomly sampled VPs are very effective at mitigating this.

Analysis of performance variation depending on the noise in the depth maps While we agree that such an analysis would be interesting, it is difficult to execute on our data, where the depth maps are already very robust, with a precision relative to gt of 0.94+-0.08. Our assumption of consistent contrast is realistic when using contrast-enhanced images without depth intensity variation, as it can be achieved through simple intensity clipping. We rework the manuscript for further clarification.

Evaluation (R1) We performed 5-fold cross-validation paired with seed re-initialization to account for data and initialization variance. Multiple cross-validation iterations would further reduce partitioning bias (R1), but also increase the computational cost 5-fold (915h only for Table 1).

Clarity (R1, R2, R3) We improve Fig 1 (R3), captions (R2), the abstract (R1), references (R1), supp. material references (R1), vague formulations (R1), MIP terminology (R3). We expand the dataset description (R1) to address: 1) annotations: each annotator annotated half of the dataset 2) inter-rater agreement: we have single annotations per scan, therefore an inter-rater study is not feasible. We further clarify the use of MIPs instead of 2D slices, which require no selection (R1) since they are unique given a projection axis.



Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    This paper introduces a novel approach that enables 3D vessel segmentation using only 2D supervision from projections. However, the main issue with this paper is the lack of comparison with similar existing methods, which undermines the persuasiveness of the experimental results. In their response, the authors mentioned that ‘other supervised or semi-supervised methods employing the same 2D annotations do not exist in the literature.’ However, there are relevant works in natural scenes, such as ‘Semi-Supervised Video Semantic Segmentation with Inter-Frame Feature Reconstruction.’ During their research, the authors should strive to identify potential comparative methods to demonstrate the effectiveness of their approach. Even if such methods are not available, the authors can still apply their proposed method to different baseline models to showcase its effectiveness. In conclusion, this paper showcases some novelty, and I consider it to be a borderline accept paper.



Meta-review #2

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    The paper presents a method to segment the full 3D peripancreatic arteries solely from one annotated 2D projection per training image with depth supervision. All the reviewers highlighted that this work is interesting however, they raised some questions with respect to missing comparisons, the evaluation and some methodology components. The authors submitted a rebuttal to address these points. The two reviewers appreciated the provided answers however some points still are not very clear. Overall, the metareviewer agrees with the raised points, however he/ she finds the answer of the authors convincing, the method interesting and he/ she thinks that it will be a nice contribution for MICCAI 2023.



Meta-review #3

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    The authors diligently addressed the reviewers’ concerns, although some concerns remain – I feel publishable MICCAI



back to top