Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews Back to top
List of Papers By topics Author List
Paper Info Reviews Meta-review Author Feedback Post-Rebuttal Meta-reviews

Authors

Martin J. Menten, Johannes C. Paetzold, Alina Dima, Bjoern H. Menze, Benjamin Knier, Daniel Rueckert

Abstract

Optical coherence tomography angiography (OCTA) can non-invasively image the eye’s circulatory system. In order to reliably characterize the retinal vasculature, there is a need to automatically extract quantitative metrics from these images. The calculation of such biomarkers requires a precise semantic segmentation of the blood vessels. However, deep-learning-based methods for segmentation mostly rely on supervised training with voxel-level annotations, which are costly to obtain.

In this work, we present a pipeline to synthesize large amounts of realistic OCTA images with intrinsically matching ground truth labels; thereby obviating the need for manual annotation of training data. Our proposed method is based on two novel components: 1) a physiology-based simulation that models the various retinal vascular plexuses and 2) a suite of physics-based image augmentations that emulate the OCTA image acquisition process including typical artifacts.

In extensive benchmarking experiments, we demonstrate the utility of our synthetic data by successfully training retinal vessel segmentation algorithms. Encouraged by our method’s competitive quantitative and superior qualitative performance, we believe that it constitutes a versatile tool to advance the quantitative analysis of OCTA images.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16452-1_32

SharedIt: https://rdcu.be/cVVpE

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #2

  • Please describe the contribution of the paper

    This paper proposes a novel method for the synthesis of OCTA 3D tomographies with associated vascular segmentation ground truths, that are used to pre-train segmentation networks over OCTA images. The experiments presents promising results.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1. The proposed formulation is novel, to the best of my knowledge.

    2. The preliminary results shows advantages in the use of the synthetic data for pretraining + refinement on real data, improving the performance of the 2D segmentation achieved with only training on real data.

    3. Preliminary results on 3D segmentation of real OCTA volumes using the synthetic data only.

    4. Ablation study.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    1. Lack of specification of the training details, and network architectures used.
  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    Despite github links have been provided for third party implementation of the networks and experimental settings, these details are not described in the paper. This is important, as repositories evolve, while reported results are fixed to a given version. Should the authors cite the paper (i.e. [16]), and ensure that the repository version used is preserving the settings reported, and otherwise tell the difference.

    Moreover, the cited paper/repository from Ma et al. IEEE TMI:40(3) 2020 [16], reports a fixed number of epochs with exponential learning rate decay (which requires a fixed number of epochs). Instead, the authors report that they “introduce a validation split for model selection”, which indicates the use of an stopping criterion that is not specified. This is a relevant detail for reproducibility that is not sufficiently described.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

    Overall, this is a novel and original paper. The proposal is interesting and the use of synthetic data for training demonstrates advantages on pre-training, as well as providing a way to train vascular segmentation on 3D OCTA.

    However, the description of the experimental setting is lacking details. Specifically, the reproducibility issue described above is important, in my opinion, because of the following reasons:

    1. Adding an stopping criteria significantly changes the settings with respect to those reported in [16]. The authors should clearly report what this criteria is, otherwise the amount of training is difficult to evaluate, and, as a more relevant issue, depending on the specific setting, some networks may have stopped way earlier than others, inducing biases in the comparison.
    2. Considering a fixed number of epochs, and comparing networks trained with different dataset sizes, induces a bias regarding a different level of overall training of the networks. This is because the number of network updates depends on the total number of minibatches, while the decaying learning rates (i.e. the strength of updates) depends on the epoch number (using the poly rule reported in [16]). Thus, larger datasets imply a larger number of updates with larger learning rates, i.e. more training. If the networks have not reached their full potential, the comparison may be biased.

    Related with this topic, there is a potential issue with the results reported in table 2. On the one hand, the trainings with 32 vs 320 vs 3200 synthetic datasets imply completely different training settings. While larger datasets could imply larger diversity, the question arises on what would happen if the 32 images where presented the exact same number of times, with the exact proportion of learning rates as the for the 3200 images dataset (i.e 3200 over 200 epochs vs 32 images over 20000 epochs with decaying learning rates accordingly). Moreover, the case of the synthetic + finetunning (320 images over 200 epochs + 32 images over 200 epochs) vs real (32 images over 200 epochs) may also be imbalanced wrt the refinement level. This is something that worth look into, as it is not guaranteed that the networks have reached their maximum refinement (and potential of the data) when the training stopped, nor any note has been provided regarding this limitation, nor regarding any actions taken to prevent this potential bias in the comparison.

    It is important to explicitly report if augmentation was used, and their details.

    As a minor detail, the authors should add the cite number after Liu et al. in page 3 (i.e. [14]), and after Ma et al. in page 6 (i.e. [16]). Otherwise, the citing style is not coherent.

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    6

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Overall this paper is novel and with fair contributions. The weaknesses are lack of details on the specific trainings in the paper, which are moderate at most.

  • Number of papers in your stack

    4

  • What is the ranking of this paper in your review stack?

    1

  • Reviewer confidence

    Confident but not absolutely certain

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #4

  • Please describe the contribution of the paper

    Segmentation of vessels from OCTA images is a clinically relevant problem, but a large amount of publicly available datasets is not available to train a deep learning based segmentation method. The paper proposed a pipeline for physics-based simulation of OCTA images with corresponding ground truth labels for segmentation. Low amount of annotated data is a typical problem in medical imaging, this paper tries to address this issue by generating synthetic images. It also followed physics-based methods for augmenting datasets and introducing several artifacts generally introduced in OCTA image acquisition.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. The paper is very well written.
    2. The paper addresses an extremely important and clinically relevant medical imaging problem.
    3. The results are promising and authors did a lot of experiments.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    he paper used one synthetic dataset but as it did not have ground truth annotation, the quantitative performance measure was not reported. This is a crucial information as the motivation of training only with synthetic examples can only be validated if the trained model performs similarly well in real experimental data.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    The notes on reproducibility seems satisfactory

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

    This is overall well-wrritten and holds a lot of potentials to tackle an important clinically relevant problem. There are a few statements like “Qualitatively, we find our segmentations to be superior (see figure 4). “ Can the authors comment on this and how generalized this superiority of results hold for different example images.

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    6

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper addresses an important clinical problem, the physics-based simulation algorithm used here, is novel and the synthetic dataset performed similarly like real data. It can be a useful technique in the long run as data availability is an issue in medical imaging.

  • Number of papers in your stack

    4

  • What is the ranking of this paper in your review stack?

    1

  • Reviewer confidence

    Confident but not absolutely certain

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #3

  • Please describe the contribution of the paper

    In this work, the authors present a method to generate highly realistic, synthetic OCTA images with intrinsically matched ground truth labels. To some extent, it solves the problem that deep learning methods need time-consuming and labor-intensive manual annotations to train blood vessel segmentation models on OCTA images. The quantitative and qualitative performance show that this method could be a versatile tool to advance OCTA analysis. In addition, they quantify the intrinsic scalability of the proposed approach and investigate how it can facilitate segmentation of the retinal vasculature in three-dimensional OCTA images.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. The simulation of OCTA images and their corresponding labels are based on two novel components: a physiology-based simulation and a suite of physics-based image augmentations.
    2. The authors demonstrate the feasibility of the proposed method by successfully training several segmentation algorithms, providing an effective tool for solving manual annotation.
    3. It appears that the proposed method holds considerable promise for expansion beyond vessel segmentation and ultimately to advance the quantitative analysis of OCTA in clinical practice.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    1. The citation and explanation of some methods are not intuitive, which bring concerns to the reproducibility of the method.
    2. The description of the manipulation of 3D-level deformation of vessels is too general and needs more details.
    3. In the experimental part, the description of the evaluation dataset is vague.
  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    It is still necessary to further explain the implementation details, such as the selection of three-dimensional deformation-related parameters, node positioning when simulating a blood vessel tree, etc., to ensure the repeatable implementation of the method.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

    1. It is necessary for the author to explain in detail the number of samples, size, sampling method, etc. contained in the dataset, which is helpful for other researchers to conduct repeated research.

    2. In section 3.1, the authors only stated that they extended a method by Schneider et al. However, for a vessel tree, how to determine the parent or child tree and how to select the root node, I think it is necessary to give an example to explain in detail, which is a key step in the simulation.

    3. In Section 3, the authors describe that they have performed image deformation to simulate the typical curved shape of the retina. However, the details (e.g., method and setting) of how you exactly perform the deformation are not given.

    4. I have doubts about the operation of three-dimensional deformation of blood vessels, including how to confirm the connection point of SVC and DVC, and whether this bending deformation conforms to the actual retinal anatomy, all of which need to be verified.

    5. By observing the examples shown in Fig. 3, I feel that there still have room for improving the model. The synthetic data is still different from the realistic data in terms of the noise level and vessel intensity distribution.

    6. Since the authors use the ROSE data for evaluation, it would be more intuitive to also compare with the OCTA-Net developed for ROSE data.

    7. In the introduction part and Fig. 1, the authors declare that their method is the “proof-of-concept of 3D segmentation of OCTA images for the first time”. As far as I understood, this is an inaccurate statement. The authors should better review the literatures published on IEEE-TMI, IEEE-JBHI, MICCAI and ISBI for the methods developed for 3D OCTA segmentation and analysis.

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    5

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper presents ideas for OCTA simulation. There are still room for improving the model with better performance.

  • Number of papers in your stack

    3

  • What is the ranking of this paper in your review stack?

    2

  • Reviewer confidence

    Very confident

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #5

  • Please describe the contribution of the paper

    The authors proposed a method to synthesize large amounts of OCTA images that have matching vessel segmentation labels, which can avoid the dependence on a large number of manual annotation training data. To improve the quality of synthesis images, they presented two novel components, i.e., a physiology-based simulation and a suite of physics-based image augmentations. Extensive experiments shown that proposed method was highly effective.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    -The paper is well writing and organized. -The proposed method constitutes a versatile tool to advance the quantitative analysis of OCTA images.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    Although experiments have shown that the synthetic images are useful for segmentation, the quality of the images have not been directly evaluated, and there is a significant difference between the synthetic images and the real images in terms of visualization.

  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    Good.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

    -The proposed method can generate vascular network, however, the synthesize retinal vasculature do not accord with anatomy. For example, the density of small blood vessels increases as they approach the center of the macula, and the foveal avascular zone is surrounded by small blood vessels, which the authors did not consider. -Although the main contribution of the paper is to propose a method to generate OCTA images, there is a lack of detailed description of the training details for vessel segmentation model. -What is the performance of the segmentation model when using synthetic data and real data for joint training？

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    6

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Contribution of the proposed approach to the community.

  • Number of papers in your stack

    5

  • What is the ranking of this paper in your review stack?

    1

  • Reviewer confidence

    Confident but not absolutely certain

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Primary Meta-Review

  • Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    This paper proposes a retinal vessel segmentation framework for OCTA image by using synthetic 3D labels. Four reviewers recommend accepting this paper, by concerning its high novelty, good quality on writing, and promising experimental results. AC also suggests accepting this paper, however, two issues are also raised: 1) The synthetic 3D label looks not in common with the anatomical structure of retina (last subfigure of Fig 2); the frame of retinal vasculature is bended in a wrong direction. 2) Missing an important reference: S. Yu et al., Cross-Domain Depth Estimation Network for 3D Vessel Reconstruction in OCT Angiography, in MICCAI, 2021.

  • What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    1



Author Feedback

We would like to thank the area chair and reviewers for their careful review, positive comments and helpful feedback. In the following, we have briefly summarized the main concerns of the referees, provided answers to each of their points and outlined the corresponding changes that we have made to the paper:

  • We have provided additional details about the training strategy of the segmentation algorithms in the experiments and results section (as requested by reviewers #2, #3 and #5). In particular, we added information on our network training stopping criterion and data augmentation. We agree with the raised point that interactions between the dataset size, training batch size, number of total weight updates and learning rate schedule impact the ultimate performance of the segmentation algorithms. Consequently, the absolute performance and relative rankings of the different algorithms may differ from the work by Ma et al. The page limitations keep us from fully exploring this issue, but we have added additional passages that discuss and acknowledge this as a limitation of our study. However, we would like to emphasize that the main focus of our work is the generation of synthetic OCTA images and its utility. We are confident that our approach, which uses a validation dataset split and independent testing split as well as five-fold cross validation, is well suited for our experiments.

  • We have included more information on the physiology-based algorithm for vessel tree generation by Schneider et al. (reviewer #3). In particular, we now explain how the algorithms selects the location at which the vessel tree is grown.

  • Our strategy to deform the vessel complexes according to the retinal curvature is now described in more detail (reviewer #3). In the past, we accidently extracted the retinal curvature from OCT images that were previously “flattened” to the Bruch’s membrane. This resulted in the synthetic vessel trees being deformed unnaturally in a convex shape (as noted by the area chair). We have since corrected this mistake in our workflow. We find that this change does not affect the 2D segmentation results as the shearing of the vessel graphs occurs perpendicular to the en-face projection direction. On the other hand, it appears to improve the qualitative 3D segmentation results.

  • We have added and briefly discussed additional references (AC and reviewer #3). We have also toned down the statement of first proof-of-concept of 3D OCTA segmentation in figure 1 (reviewer #3).

  • We now explicitly highlight that all presented vessel graphs, synthetic images and qualitative segmentation results have been randomly selected from our data and are representative of our findings (reviewer #4).

  • We agree that our synthetic OCTA images are still visually different from real images (reviewers #3 and #5). We have expanded the discussion section to acknowledge this limitation, while pointing out the main aim of this work was not the generation of picture-perfect medical images, but rather the creation of a dataset with utility for training of neural networks for downstream segmentation tasks.



back to top