List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Michael Sommersperger, Shervin Dehghani, Philipp Matten, Kristina Mach, M. Ali Nasseri, Hessam Roodaki, Ulrich Eck, Nassir Navab
Abstract
Swept-Source Optical Coherence Tomography (SS-OCT) integrated with surgical microscopes has enabled fast, high-resolution, and volumetric visualization of delicate tissue-instrument interactions. However, some visual features, which provide essential perceptual information in microscopic surgery, are not present in 4D OCT. Such a feature is the shadow of the surgical instruments cast onto the retina by the endo-illumination probe, which is among the most important cognitive cues for perceptual distance estimation. In this work, we propose Semantic Virtual Shadows (SVS), a novel concept to artificially generate instrument-specific shadows in OCT volumes, enabling naturally non-existent but important perceptual cues that are present in microscopic surgery. Semantic scene information is leveraged by considering only voxels associated with shadow-casting and shadow-receiving objects, identified using a learning-based approach and efficient volume processing, respectively. Real-time performance is achieved by a precomputed semantic shadow volume texture that assigns a shadowing factor to each voxel associated with a shadow-receiving object. The novelty of the method includes not only instrument-specific shadowing on the surface anatomy but also exclusively on deep-seated subsurface structures, providing advantages for various vitreoretinal procedures. Our user study indicates the benefits of the method for 4D OCT-guided surgery in several cognitive and performance-specific aspects.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43996-4_39
SharedIt: https://rdcu.be/dnwPk
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This article proposes a semantic virtual shadows method to simulate the presence of shadows on surgical instruments during ophthalmic surgery and claims that it can improve the perception of the surgeon.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The background of this paper is interesting, especially in generating shadows and enhancing physicians’ perception of space in 4D OCT images. The proposed approach is of potential application to the clinic.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The major weakness of this paper lies in the motivation. The authors claim that shadow simulation in 4D OCT can effectively enhance physician perception and attempt to demonstrate this through user experiments. But this conclusion appears to be subjective. First, shading is an indirect information for physicians with spatial perception in 2D images, and why do we need such an undirected method when we already have spatial data? Second, if the authors want to provide richer information guidance for physicians in 4D OCT, why not choose a conventional segmentation and guidance mode? Since the authors already have access to accurate OCT volume data, the conventional guidance method is perfectly achievable, what is the need for shadow simulation in contrast? The authors need to clarify this point.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
From the technical aspect, this paper is highly reproducible. However, the results of this paper are mainly user experiments, which are subjective and difficult to be reproduced.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- The authors should strengthen the explanation of the principles of their work, after all the shadows do not seem to be scientific information, and the perception mentioned by the authors is more dependent on the habits of the doctors than on the access to more information. Clarifying the importance of this not direct information is necessary.
- The speed of the proposed method needs to be evaluated in relation to the feasibility of its application in surgical procedures. After all, the processing of volume consumes more computational resources than 2D images, and the application is the major motivation of this work.
- As mentioned before, the authors wanted to improve the perception of the doctors. They chose a gray shadow, which is common sense. But since we already have accurate information, why didn’t the authors change more obvious colors and displays to enhance more obvious information?
- The standard of the Depth-specific error needs to be clarified and what level of error is acceptable.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The principle of using virtual shadows to improve physician perception is not clear enough, and it appears that the authors could have used 4D OCT to realize a more precise surgical guidance process.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The authors address an inherent problem of volumetric visualization in Optical Coherence Tomography (OCT)-based microscopic surgery, namely the lack of important perceptual information when it comes to interactions between tissue and surgical instruments. Such important but missing perceptual information may hinder widespread adoption of 4D-OCT technology and is identified as the surgical instrument’s shadow on the retina, which allows for perceptual distance estimation. In order to address this issue, the authors propose a novel concept for the artificial generation of surgical instrument-specific shadows in OCT volumes called “Semantic Virtual Shadows (SVS)”. A quantitative comparison to other shadow generation approaches via inference time profiling and a user study with biomedical experts confirm the benefits of the proposed method.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Overall, a very good paper with well-explained motivation and contribution, well thought out document structure and convincing results. It is commendable that the authors used a separate ‘Related work’ section instead of describing the related work in the introduction section.
- The proposed novel method of Semantic Virtual Shadows seems to have obvious benefits when it comes to the widespread clinical acceptance of 4D OCT-guided surgery.
- Good experimental design: The advantage of the proposed method is shown via a comparative time profiling and a user study.
- Excellent supplementary video that explains the proposed method and experimental results.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The fact that no source code is provided reduces the likelihood of this work being reproduced by other researchers.
- Neural network related hyper-parameters are not described which further decreases reproducibility. Since a U-net architecture and a ResNet backbone are mentioned in the paper, there are certainly network training details that could be mentioned here. If such hyper-parameter details are not applicable - as stated in the reproducibility response - the authors should mention this in the paper.
- Overall, the Methodology sections doesn’t seem to contain enough details to ensure an accurate reproducibility of the propsed method (I’ve added more details under ‘reproducibility of the paper’).
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Overall, the authors list several details that facilitate a potential reproduction of the proposed concept. On a positive note, it is worth mentioning that the provided mathematical equations are well-described and increase reproducibility of important algorithmic components. However, there are a couple of this that could be improved in order to increase reproducibility: 1.) Availability of source code: While there is no need to provide source code, it would have greatly increased reproducibility. In addition, as already mentioned under the main weaknesses of this paper, providing more details regarding hyper-parameters (and maybe training checkpoints) would have increased reproducibility.
2.) Implementation details: The used graphics card is mentioned, as well as software library details such as Pytorch, TensorRT and OpenGL incl. their respective versions. However, The C++ version is missing and the used OS incl. version.
3.) Level of descriptive detail in the Methodology section: Some parts of the proposed methods are not described accurately enough to ensure reproducibility. Example: Page 6, text lines 4 and 5: “…while preserving surface highlights similar to [12].”: The word “similar” in this sentence leaves a lot of room for interpretation and reduces reproducbility. This should be described in more detail.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
There are a few minor things that could be improved:
1.) Section 1, page 2: This page should be split into more paragraphs in order to make it easier to read.
2.) Sections 3 and 4: Using sub-sections rather than paragraphs with highlighted text would increase readability.
3.) Section 1, page 2: the following sentence: “The absence of such perceptual cues increases the burden for surgeons to adapt to 4D OCT.”: Please add a reference that supports this statement, if possible.
4.) Section 3, page 5, equation 3: In order to increase reproducibility it would be better to mention some concrete values of t_s rather than just saying “where t_s is an empirically chosen threshold.”
5.) Section 4, page 6: Please mention the concrete C++ version as well as OS incl. Version. This increases reproducibility.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Overall a very good paper with strong motivational and novelty aspects combined with a well thought out document structure, a very good experimental design and results demonstrating the concept’s utility. However, there are some things that could improve reproducibility and readability as mentioned in my comments to the authors.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This paper points out the lack of adequate depth cues associated with micro ophthalmic OCT when used to guide retinal and sub-retinal procedures, and proposes a novel method of introducing artificially generated “shadows” of the surgical instrument into the visualization environment. This approach permits the use of the instrument shadow as a depth cue (i.e. when shadow and instrument tips coincide the tip is on the surface). Moreover, since there is more than one surface in the retina to consider, this approach has the provision for the proximal surface to be rendered semi-transparent and for the shadow to be interpreted on the more distal surface. This addition to standard use of OCT in ophthalmic surgery has the potential to make OCT-guided eye surgery more precise and efficient.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper builds on the concept of efficient artificial shadow generation as a useful depth cue that has been around for a while. What makes this work novel however is it’s adaptation to retinal OCT. As pointed out by the authors, retinal OCT intrinsically casts a shadow of the surgical instrument, but it is constrained to be in the path of the scanned laser illumination, and may not be as intuitive as a shadow cast obliquely. Another aspect of this work is that the instrument is recognized automatically from the ICT image and the artificial shadow is only generated to reflect this instrument (hence semantic segmentation), while ignoring other structures that might be in the field of view. The overall strength of this paper is enhanced by the fact that the authors recognize that the sub-retinal Retinal Pigment Epithelium surface is also often the target of procedures and that the approach can be tailored to cast shadows on this surface while rendering the more proximal retina semi-transparent. This work has clear potential for clinical translation, with the described method being readily adapted into existing workflow. A comprehensive user study indicates that the use of this approach significantly reduces the targeting error, as well as improving scores in most TLX metrics. The paper is well written, laid out logically and easy to follow.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
There are only a few minor weaknesses in the paper. These arfe all captured in the detailed comments below.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors have provided comprehensive data to allow this work to be preproduced.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
It would have been helpful if one of more of the participants has been ophthalmological surgeons. While the trial participants were “biomedical experts” their expertise is not identified. If they are novices with respect to the actual procedure, does their performance reflect that of an experienced surgeon? Even comments from a single surgeon, relating both to the realism of the virtual testing environment and the degree of consistency with standard clinical workflow, would add tremendous value to this paper, I may be missing something here, In the intro you state that “ This OCT shadow is fixed in perspective and always occurs directly beneath the instrument. Hence, it does not provide the same intuitive cognitive cues, especially when viewing the volume from a top view, which is the usual view during ophthalmic surgery.” While I understand this to be the usual case for surgery, can the reconstructed 3D OCT volume nevertheless be visualized from any arbitrary pose? Would this be a valid approach, or would it necessitate deviation too far from the standard workflow. A comment here would be appreciated. I note that you randomly varied the direction of the artificial illumination during the trial. Can you comment on how the artificial illumination direction would be chosen in practice? Table 1 caption and table are inconsistent. I think the “3D shadow volume buffer without semantic information (SV)” should refer to SB? In line 12 of P 8, do you mean ”faster convergence”? P 5 6 lines from the end. “past” instead of “passed” Should the phrase in para 3 on p 5 read: that refers to Fig 3. Read: “while in the bottom row the same volumes are rendered with both Phong shading and SVS?” The shadows cast in Fig. 3 are quite difficult to see. Perhaps outlining them in a contrasting colour would help?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
8
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Noverl application Well written Direct clinical relevance to a real problem
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The authors propose a method to use artificially generated shadows of surgical instruments in the context of ophthalmic surgery (4D OCT) in order to improve surgeon perception and in turn make OCT guided surgery more precise. A user study with novice participants showed that the method significantly reduces targeting error, and improves scores in most NASA TLX metrics.
The reviewers agree that this is a very good paper with novel aspects that would have clinical applications. Some details are missing as mentioned by R3 to make the method reproducible, clearer motivation (R1), and if possible input from a clinician would be ideal (even if not part of the study) (R2).
Author Feedback
We are delighted that the reviewers found our work to have “strong motivational and novelty aspects” (R3) with “direct clinical relevance” (R2) and “a very good experimental design and results” (R3) that have been integrated in a “well thought out document structure” (R3). We are thankful that the reviewers appreciated the high generalizability of our proposed concept, as it can be tailored to various procedures at or above the retinal surface as well as to subretinal interventions close to the RPE layer. In the following, we would like to address some of the reviewers’ comments that were of great value for improving the quality of our manuscript.
We thank R1 for finding our approach “of potential application to the clinic”. With respect to the reviewer’s comments, we would like to discuss further motivation of our work. 4D OCT can be considered as a novel imaging modality, while commonly ophthalmic procedures are performed under visualization through a stereo microscope, where the instrument shadow generated by the endoillumnation is a main cue for distance perception. Integrating such familiar cues artificially in 4D OCT can help surgeons to familiarize themselves with 4D OCT faster and improve its acceptance.
We agree with the reviewers on the importance of the clinicians’ acceptance. The simulation method from which our virtual environment was generated, has been evaluated by clinical experts in a previous publication. The reference will be added in the final version of the paper to preserve anonymity throughout the review process. In response to R2, from our point of view the best positions of the virtual light source could be the ones learned from real ophthalmic surgery, in which the surgeons are adjusting the light source to optimize their interaction in microscopic imaging. The idea would be to transfer this into the OCT space. In the current version, we are doing this intuitively, however this could be nicely learned by observing current microscopic surgeries.
We appreciate the valuable comments of all reviewers, which helped to further improve the quality, reproduciblity and readability of our paper. We hope that this work will further advance the adaptation and integration of 4D OCT systems.