List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Jan Sellner, Silvia Seidlitz, Alexander Studier-Fischer, Alessandro Motta, Berkin Özdemir, Beat Peter Müller-Stich, Felix Nickel, Lena Maier-Hein
Abstract
Robust semantic segmentation of intraoperative image data could pave the way for automatic surgical scene understanding and autonomous robotic surgery. Geometric domain shifts, however – although common in real-world open surgeries due to variations in surgical procedures or situs occlusions – remain a topic largely unaddressed in the field. To address this gap in the literature, we (1) present the first analysis of state-of-the-art (SOA) semantic segmentation networks in the presence of geometric out-of-distribution (OOD) data, and (2) address generalizability with a dedicated augmentation technique termed ’Organ Transplantation’ that we adapted from the general computer vision community. According to a comprehensive validation on six different OOD data sets comprising 600 RGB and hyperspectral imaging (HSI) cubes from 33 pigs semantically annotated with 19 classes, we demonstrate a large performance drop of SOA organ segmentation networks applied to geometric OOD data. Surprisingly, this holds true not only for conventional RGB data (drop of Dice similarity coefficient (DSC) by 46 %) but also for HSI data (drop by 45 %), despite the latter’s rich information content per pixel. Using our augmentation scheme improves on the SOA DSC by up to 67 % (RGB) and 90 % (HSI) and renders performance on par with in-distribution performance on real OOD test data. The simplicity and effectiveness of our augmentation scheme makes it a valuable network-independent tool for addressing geometric domain shifts in semantic scene segmentation of intraoperative data. Our code and pre-trained models are available at https://github.com/IMSY-DKFZ/htc.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43996-4_59
SharedIt: https://rdcu.be/dnwP3
Link to the code repository
https://github.com/IMSY-DKFZ/htc
Link to the dataset(s)
N/A
Reviews
Review #3
- Please describe the contribution of the paper
The authors identify an issue with semantic segmentation performance underr certain image modifications, they propose an augmentation strategy to handle this. They use a large dataset of RGB + hyperspectral images to validate this idea and show that, in some cases, a large improvement over the baseline case is seen.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper clearly identifies a problem which is relevant to the community. Hyperspectral data is very interesting and the ability to reliably semantic segmentation of this type of data would be a very useful contribution.
The datasets, training strategy and validation process are all very well explained.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The results show only marginal improvement in the datasets where ‘realistic’ situations are created. Datasets such as the ‘isolation’ and ‘removal’ datasets (with the exception of the isolation_real) have clearly unrealistic regions which appears to affect the network far more than realistic modifications.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The code will be released but the data is not public (as far as I understand) so the reproducibility is weak.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
I believe that validating on the ‘non-realistic’ datasets harms the paper since they distract from the main message of the paper. Although the improvement is much more modest on the realistic datasets, this is where the clinical value is more apparent to me. I would have suggested a greater level of analysis into these datasets.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I think that the results are not hugely significant, since in the realistic datasets the results are only slightly better than the baseline. The improvement seems to be more heavily weighted towards the cases where the image does not look realistic.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The manuscript performs an analysis of semantic segmentation networks under the presence of OOD data for surgical scene understanding. The authors also propose a data augmentation technique called “Organ Transplantation” for addressing geometric data shifts. The performance was evaluated on dataset collected by the authors through a study with 33 pigs.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The abstract and the overall paper are very clear and well written.
- The results are interesting and will be valuable to the surgical data science community. The authors show systematically that state-of-the-art surgical scene segmentation networks fail under geometric domain shifts.
- The evaluation of performance on both RGB and HSI data is worth studying in this context.
- The OOD scenarios described are clinically relevant and the approach is straightforward
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The authors mention that they are the first to the best of their knowledge, they are the first to contribute to geometric domain shifts. The paper “Rethinking Surgical Instrument Segmentation: A Background Image Can Be All You Need” from MICCAI 2022 feels very relevant to the approach proposed by the authors, albeit for instrument segmentation. This is a good reference to include and compare performance with (report on codebase, not necessary for rebuttal).
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors provide a very clear description for reproducing the paper. They will include information about running experiments and tuning hyperparameters in their code base that will be publicly released.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
See weaknesses
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper provides a detailed analysis of performance under geometric domain shifts. Though the authors missed an important prior work in the surgical domain, this is still the first work of its nature within surgical scene understanding.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #1
- Please describe the contribution of the paper
The manuscript investigates the problem of semantic segmentation of endoscopic surgical images under geometric domain shifts.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Really nice paper
- Well written and easy to follow
- Nice experiments
- Detailed results
- Nice images
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The benefit of using HSI vs RGB images in surgery may be detailed better (not just through the quantitative comparison of the results).
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Relevant information for reproducibility is included.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
I congratulate with the authors for the nice work. Maybe more space could be given to describe the complementary transformations listed at the end of page 2. The clarify of Fig. 1 may be improved. E.g., what is the meaning of the question mark?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
8
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Nice paper, nice idea.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
All three reviewers recommend acceptance of the work and commend the interesting idea, well thought out and detailed experimental protocol, and clear presentation of the work.
Overall, there seem to be few weaknesses, and the ones mentioned include 1) only marginal improvements on “realistic” cases (where more emphasis could have been put into analysis of those cases as they are most relevant), and 2) greater detail on the benefits of hyperspectral imaging over RGB.
Author Feedback
We thank the reviewers for unanimously suggesting acceptance of our paper and would like to comment on some suggestions:
- More emphasis on “realistic” cases (R3, MR1): We evaluated the organ segmentation performance under geometric domain shifts for three scenarios, namely organs in isolation, organ removals, and situs occlusions, and found our Organ Transplantation augmentation to be beneficial in all cases (cf. Fig. 3). We agree that the pure number of manipulated out-of-distribution (OOD) datasets (4) is larger than the number of real OOD datasets (2). However, for the isolation scenario the improvement in segmentation performance on manipulated and real data is comparable (Dice Similarity Coefficient (DSC) improvement of 57 % vs. 50 %). The overall benefit in the occlusion scenario is smaller than for the isolation and removal cases, but especially the outlier performance increases (e.g., DSC improvement for pancreas 283 % and stomach 69 %). Although situs occlusions might be the most frequent domain shifts in real-world surgeries, the other two scenarios also occur and are thus important. We updated the manuscript to reflect these important points and provide a more detailed analysis of the real OOD datasets.
- Benefit of hyperspectral imaging (HSI) over RGB data (R1, MR1): We have shown that RGB-based organ segmentation networks are more susceptible to geometric domain shifts and HSI-based segmentation yields consistently better performance. Apart from improving organ segmentation, HSI offers many additional benefits such as the ability to recover functional tissue properties (e.g., perfusion state of an organ). We added this important additional benefit in the manuscript.
- More detailed description of complementary transformations (R1): We added a short description of each complementary transformation in the manuscript.
- Clarity of Fig. 1 (R1): We modified Fig. 1 to make it more self-explanatory.
- Related work “Rethinking Surgical Instrument Segmentation: A Background Image Can Be All You Need” (R2): The authors address the lack of large annotated datasets for instrument segmentation by synthesizing datasets from a small set of background and instrument images. Among others, they mix images by pasting instruments onto background images. Despite the focus of the paper being different from ours, we included a reference to this work in our manuscript, as the idea is indeed related to ours.