List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Adeleh Bitarafan, Mohammad Farid Azampour, Kian Bakhtari, Mahdieh Soleymani Baghshah, Matthias Keicher, Nassir Navab
Abstract
This work proposes a self-supervised algorithm to segment each arbitrary anatomical structure in a 3D medical image produced under various acquisition conditions, dealing with domain shift problems and generalizability. Furthermore, we advocate an interactive setting in the inference time, where the self-supervised model trained on unlabeled volumes should be directly applicable to segment each test volume given the user-provided single slice annotation. To this end, we learn a novel 3D registration network, namely Vol2Flow, from the perspective of image sequence registration to find 2D displacement fields between all adjacent slices within a 3D medical volume together. Specifically, we present a novel 3D CNN-based architecture that finds a series of registration flows between consecutive slices within a whole volume, resulting in a dense displacement field. A new self-supervised algorithm is proposed to learn the transformations or registration fields between the series of 2D images of a 3D volume. Consequently, we enable gradually propagating the user-provided single slice annotation to other slices of a volume in the inference time. We demonstrate that our model substantially outperforms related methods on various medical image segmentation tasks through several experiments on different medical image segmentation datasets.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16440-8_58
SharedIt: https://rdcu.be/cVRwJ
Link to the code repository
https://github.com/AdelehBitarafan/Vol2Flow
Link to the dataset(s)
https://sliver07.grand-challenge.org/
https://chaos.grand-challenge.org/
https://www.ircad.fr/research/3dircadb/
https://kits21.kits-challenge.org/
https://www.synapse.org/#!Synapse:syn3193805/wiki/217789
https://www.cancerimagingarchive.net/
Reviews
Review #1
- Please describe the contribution of the paper
This paper presented a self-supervised algorithm for 3D image registration to find 2D displacement fields between all adjacent slices within a whole volume together. The output of Vol2Flow is employed to segment each arbitrary anatomical structure in a 3D medical image by gradually propagating the 2D segmentation mask provided by a user between other slices.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is well-written and organized. The experiments is sufficient.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Many important method and experiment details are missing in the paper.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
I believe that the obtained results can be reproduced.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
More details about the user-provided single slice annotation should be added for clarity. Additionally, the influence on the segmentation performance from the slice annotation should be investigated.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The idea is novel and useful to improve the effective utilization of medical datasets.
- Number of papers in your stack
2
- What is the ranking of this paper in your review stack?
5
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
The paper presents an image segmentation method via slice-to-volume label propagation using image registration. It is overall an interesting and novel approach and it has shown to outperform a few other methods in the literature (e.g. fully supervised, other registration methods, etc.).
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The volume segmentation problem is addressed busing slice-wise image registration. Only one slice needs to be annotated to segment the whole volume in test time.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The actual applicability in real scenario worth an in-depth discussion. Additionally, the method is suffered from severe error propagation problem due to slice-wise registration.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
It is difficult to re-implement the exact method and reproduce the results due to missing implementation details.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The paper presents an image segmentation method via slice-to-volume label propagation using image registration. It is overall an interesting approach and it has shown to outperform a few other methods in the literature (e.g. fully supervised, other registration methods, etc.). The actual applicability in real scenario worth an in-depth discussion. A few comments are provided as below.
- From Fig.1 and the corresponding description, it seems the warped slice is used as the input to produce the next warped slice. Error will propagate rapidly.
- The method requires a SVM training process to post-process the label, which seems quite inconvenient. In a multi-label scenario, do you need to train several SVM classifier? Please clarify.
- Different organs may require to annotate different slices, if they’re not appeared in a single slice simultaneously. It is desirable to investigate the effect of the location of the annotated slice to the segmentation performance. Is it better to annotate the central slice of the target organ? How to select which slice to annotate?
- One drawback of registration based method is computational time. It is desirable to report the training time and inference time when comparing different methods.
- It is an interesting approach to propagate the label from slice to volume. However, it is still an intensive workload to annotate one slice (or potentially a few slices) for each unseen volume accurately, especially in a multiple organ case. How does it compare with few shot learning/semi-supervised learning?
- Could the author comment on how well the registration method work on slices with organ transitions or slices with larger thickness? When does the registration fail?
- Worth considering applying the method to an interactive image segmentation scenario, where the user can interactively improve the segmentation result.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper has certain novelty and the performance is good and convincing.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
The paper proposes a method trained to create a 3D segmentation given a single slice manual 2D segmentation. The 2D segmentation is propagated sequentially to neighboring 2D slices using the inter-slice 2D displacement field (flow). The Flow between all pair of 2D slices is generated by the network, which was trained (self supervised) to align neighboring slides of the train images.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper presents an approach for semi-automatic segmentation, combining ideas from self-supervised learning and registration. It builds up upon Sli2vol algorithms to take advantage of a global 3D context.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) the displacement field between the neighboring slices may not exist. For instance, Z-spacing of CT images could be 5mm, and the object boundary changes significantly from slice to slice. it seems the algorithm most likely will fail in such cases 2) it unclear how the algorithm propagates the segmentation when the organ boundary is reached and it disappears from the next 2D slice. It seems to be another limitation. 3) Comparisons are insufficient. Optical flow algorithm chosen is very basic and old, its results looks artificially weak. A better deformable image registration method (based on Mutual Information) should be tried for a fair comparison. Another comparison lacking is to other semi-automatic segmentation methods, such as segmentation based on extreme points (e.g. in 3D) , at least a discussion about such methods would be nice.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
seems reasonable to reproduce.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
the method is consequentially novel and interesting and it deviates from a common algorithms for the task. Nevertheless the approach seems not very practical, and the comparisons are limited.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The method is limited in applicability since it is based on the assumption that a displacement field exists between the 2D slices, which is rarely true. It also seems to over-complicate the task of producing registration if displacement field is desired. A fair comparison with a better deformable image registration algorithms (non-deep learning) should be able to accomplish the same task without any training, which would be much simpler.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
A 3D segmentation approach derived from slice-to-slice registration is proposed. In test time, only one slice needs to be annotated manually. Reviewers agree that the idea is interesting. The major concern is on the practical value. As pointed out by Reviewer #3, the basic assumption that a displacement field exists between slices is rarely true in practice. Reviewer #3’s detailed comments are also worth attention. In addition, more details about the user-provided slice annotation should be discussed. Further, the final segmentation accuracy seems to be much worse than SOTA methods’ performance.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
7
Author Feedback
We thank all reviewers for their constructive comments and suggestions. We appreciate that reviewers recognize the novelty of our method (R1, R2, and R3), the clarity and reproducibility (R1, R3), and find it interesting (R1, R2, R3, MR).
Our experimental setup is based on the most related study [26] (Sli2Vol), and thus we follow their approach of choosing a slice with the largest annotation as the source (R1, R2, MR). However, we found selecting the central slice, including the organ, can also provide comparable results (e.g., 92.3 DSC on Sliver07). We will include this information in the camera-ready version.
To tackle the error propagation problem (R2), we introduced a refinement function to amend pseudo labels after each step of mask propagation (Section 2.2). We train an SVM using foreground pixels of the source slice (according to a ground truth mask) as positive class samples, which can extremely reduce false-positive errors in the mask propagation. This training process is fast, even in a multi-class scenario.
One of the main advantages of Vol2Flow is using the global context information within the whole volume when inferring 2D displacement fields, which helps in modeling the inter-slice information. It greatly helps in a scenario where the slice thickness might affect registration results (R2, R3, MR). Please note that experiments show Vol2Flow is practical, even on volumes with a large slice thickness (see results on Sliver07, which has a slice thickness up to 5 mm). Also, displacement fields inferred using the global 3D context can handle when an organ is suddenly shrinking (R2, R3, MR) or completely disappearing with the help of the refinement module. Mainly, 3D spatial information leads to a better understanding of how an organ is expanding or shrinking and helps us better map the boundaries of organs in consecutive slices. Based on reviewers’ suggestions, we investigated the performance of non-registration-based methods (R2, R3) and traditional registration methods (R3). For non-registration-based methods, the main solution is pre-training using many unlabeled train samples and then fine-tuning the pre-trained model at inference time using the annotated slice by few-shot or semi-supervised learning methods. However, these methods are slow in inference (due to their test-time training requirement), while Vol2Flow keeps the model fixed after the self-supervised training and propagates masks without fine-tuning. To evaluate the performance of a type of semi-supervised method, we fine-tuned the pre-trained 2D-UNet using a self-training scenario, on the annotated slice of the test volume to predict the mask of slices in that volume. This approach did not achieve good results, especially on small organs (e.g., 59.7 DSC on the kidney), and the testing time is almost 200 seconds per slice (s/sl), where the model is fine-tuned for 100 epochs. However, inference time of Vol2Flow and Sli2Vol is about 0.83 and 0.58 s/sl, respectively. The shortcoming of traditional registration methods is ignoring global 3D information. In contrast, we take 3D information beyond two slices, showing superior performance. We compared Vol2Flow with a conventional deformable registration method using Mattes Mutual Information as a similarity metric and Gradient Descent as an optimizer to measure this effect. The DSC result on Sliver07 was 66.6 before refinement and 78.8 after that, with a testing time of 34 s/sl giving inferior results compared to Vol2Flow.
Given a large set of annotated samples, fully supervised SOTA methods show upper bounds in segmenting an organ in the same domain as training data. However, they lack generalizability to unseen organs. The practical use of our method is when no such annotated data exists. Here our method can produce an initial annotation for each arbitrary organ without any fine-tuning that can be further refined manually. Overall, we believe that this work offers new paths to the MICCAI community.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The rebuttal addressed most of the raised questions. Although the proposed method’s practical value is still questionable, the idea is interesting, and the paper is of overall good quality. I recommend acceptance.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
19
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This work proposes an interesting take on segmentation that is of interest to the community. The general idea of annotating one slice and then propagating it to the rest of the volume is interesting, and optimizing an SVM for each test instance to help the propagation is a good way to avoid the domain shift problem altogether. So overall, I think this work has enough merit to meet the MICCAI bar.
Having said that, I do agree with certain of R3’s criticisms. In particular, other registration approaches should have been tried. Rebuttal discussed one, but ideally more would be tried. Additionally, extreme point segmentations or other user-interactive annotation approaches should have been compared - these can often overcome many domain shift hurdles. Despite these caveats, the idea is interesting enough and the experiments convincing enough that I am happy to recommend an accept here.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
4
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper presents a strategy to segment an image volume using only one annotated slice. This strategy has been employed previously by multiple works some of which have been cited and others that have been omitted [1]. These previous works, in particular Slic2Vol or [1] demonstrate that the misalignment between consecutive slides is minumum, thus registration seems unnecessary, as it is pointed by R3. This a flawed assumption that is central to the method and the main motivation of my recommendation.
Minor remarks
- The authors claim that a linear model does not work because “ the supposed classification problem is not linearly separable since pixels in the N region bring up the different distributions” -> Can the authors proof this statement?
[1] Wang, G., et al. Slic-Seg: slice-by-slice segmentation propagation of the placenta in fetal MRI using one-plane scribbles and online learning. In International Conference on Medical Image Computing and Computer-Assisted Intervention 2015
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
NR