List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Robin Peretzke, Klaus H. Maier-Hein, Jonas Bohn, Yannick Kirchhoff, Saikat Roy, Sabrina Oberli-Palma, Daniela Becker, Pavlina Lenga, Peter Neher
Abstract
Accurately identifying white matter tracts in medical images is essential for various applications, including surgery planning and tract-specific analysis. Supervised machine learning models have reached state-of-the-art solving this task automatically. However, these models are primarily trained on healthy subjects and struggle with strong anatomical aberrations, e.g. caused by brain tumors. This limitation makes them unsuitable for tasks such as preoperative planning, wherefore time-consuming and challenging manual delineation of the target tract is typically employed. We propose semi-automatic entropy-based active learning for quick and intuitive segmentation of white matter tracts from whole-brain tractography consisting of millions of streamlines. The method is evaluated on 21 openly available healthy subjects from the Human Connectome Project and an internal dataset of ten neurosurgical cases. With only a few annotations, the proposed approach enables segmenting tracts on tumor cases comparable to healthy subjects (dice = 0.71), while the performance of automatic methods, like TractSeg dropped substantially (dice = 0.34) in comparison to healthy subjects. The method is implemented as a prototype named atTRACTive in the freely available software anonymous. Manual experiments on tumor data showed higher efficiency due to lower segmentation times compared to traditional ROI-based segmentation.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43993-3_23
SharedIt: https://rdcu.be/dnwNo
Link to the code repository
https://github.com/MIC-DKFZ/atTRACTive_simulations
https://github.com/MIC-DKFZ/MITK-Diffusion
Link to the dataset(s)
https://zenodo.org/record/1477956
Reviews
Review #2
- Please describe the contribution of the paper
The paper proposes a semi-automatic entropy-based active learning for the segmentation of white matter tracts from whole-brain tractography. The paper evaluates the approach for healthy and patients with tumor where tracts were distorted.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Active learning for tract segmentation enabling feedback loop with manual check during classifier training.
Use of entropy based sampling to select streamlines to be manually annotated during the training.
Optimal performance on patients with tumors that distort tracts.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Time consuming approach. (~5-7 min to delineate tracts).
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper has the source code available for the reproducibility of the results.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
In Fig. 4 is showed the results of the proposed method and tractSeg. It would be nice to see that performance of the Classifyber too.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
One of the major issues in tract segmentation is to be able to delineate well tracts in the presence of tumors that distort their overall shape. In this paper, the authors proposed a semi-supervised approach that results in optimal delineation of distorted tracts in patients with tumors. The initial results are promising and they can lead to a clinical impact ruing preoperative planing.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #1
- Please describe the contribution of the paper
The paper presents a novel streamline-based method to semi-automatically segment white matter bundle tracts with active learning. It is the first paper proposing an active learning framework to segment white matter tracts. The method may speed the segmentation of the white matter tracts in clinics. Interestingly, it showed promising results in a clinical case where a tumor was adjacent to the segmented tract, that is a complicated framework. In this method is still fundamental the contribution of a human expert that provides iteratively the labels of some streamlines to create the training dataset and that stops the iterative learning when the segmentation is evaluated as satisfactory.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The main strength of the paper is to present a semi-automatic method to segment white matter bundles, that may be used in glioma patients, where the tumor can disrupt and deformate the neuroanatomical pathways. The segmentation of white matter bundles in patients with brain-tumor in clinics is now performed manually by experts. Fully automatic methods are prone to failure in segmentation in the glioma patients. This method is not fully automatic and requires human expert intervention during the training. However, it may speed up the segmentation of white matter bundles in clinics while preserving the accuracy of the results.
The challenge of white matter bundles segmentation in patients with brain-tumor is that each tumor can deformate the brain in a particular and unique way, according to its dimensions and position. An important novelty of the presented method is that, in the training, streamlines of the patient are used, providing information about the possible patient-specific anatomy deformation. On the other end, classic fully automatic algorithms for white matter segmentation rely on a dataset regarding healthy population. Hence, they are not by definition patient specific and their generalizability is limited to the healthy subjects.
In the method presented, the human expert is asked iteratively to label the streamlines whose classification is more uncertain. In this way the training dataset is updated with relevant information at each iteration. The segmentation is stopped when the human expert is satisfied with the segmentation. This framework where a human expert interacts with the algorithm may probably increase the trustability of the segmentation that is fundamental in clinics.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The proposed strategy to support the white matter bundle segmentation in a clinical setting is still at the initial stage and the results are preliminary and partial: poor characterization of data used for empirical analyses poor quantitative comparison with random strategy in the simulation poor description of the trials with real users
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors have committed themselves to sustain a good reproducibility of the paper, despite a portion of empirical investigation based on trials with human operators. Some of the data used in the analysis belongs to the Human Connectome Project, a public dataset. In the paper, it is indicated that the code used for the paper’s analysis is publicly available. Moreover the authors provide the proposed method, atTRACTive, as a prototype freely available. The major issue with the reproducibility of the paper is the clinical dataset. Both data tractography and annotated bundles are not distributed as open data.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The proposed strategy to support a combination of human-machine white matter bundle segmentation is really interesting and promising. This study would deserve to be extended because the results may have an impact on the scientific community and best practice.
First of all it would be helpful to characterize the data. There is no information on bundle statistics between healthy individuals and patients. Since the data annotation has been carried out by different operators, it is important to investigate how a different bias on neuroanatomical definition of the bundle is affecting the empirical study. In addition, to properly support the interpretation of results it would be required to describe in detail the clinical dataset: how the single bundle differs from the normative model? The quantitative analysis can meaningfully change whether the alterations are moderate or there are only few critical examples.
Simulation of the “active” segmentation method and hands-on trials of the method with real operators are completely different and they require a different experimental design and evaluation metrics. The manual intervention of the operator is meaningfully related to the type of interaction supported by the graphical user interface. For this purpose it is necessary to provide a detailed description of the interaction model implemented in the tool. Method can be sound while the tool doesn’t. This distinction is not properly remarked in the manuscript.
The comparison in the simulation experiments with the random strategy is trivial and not meaningful. In Porro-Munoz et al. [13] the iterative strategy includes a step of nearest neighbour. After the selection of a set of candidates, the operator can ask for a set of nearest neighbour streamlines and refine the selection accordingly to the neuroanatomical definition pursued by the operator.
The bootstrap of the method is operator dependent. For the simulation and the empirical analyses it is a weak point. For a known set of bundles there is an atlas that defines the waypoint ROI. Usually such ROI are quite conservative and can be used as sampler for the seeding of the streamlines. Warrington S, Bryant K, Khrapitchev A, Sallet J, Charquero-Ballester M, Douaud G, Jbabdi S, Mars R, Sotiropoulos SN* (2020) XTRACT - Standardised protocols for automated tractography and connectivity blueprints in the human and macaque brain. NeuroImage, 217(116923). DOI: 10.1016/j.neuroimage.2020.116923
The size of a white matter bundle can be of the order of 10^3-4. It is not clear how the user interaction can deal with such a huge amount of streamlines providing a fine grain annotation for each streamline. On the other hand, if we limit the iterative process to present only a small set of streamlines how the process can converge to the final size of the bundle. This aspect is not well illustrated in the manuscript. Partially related to this aspect is the lack of information on what portion of the streamlines are retained or rejected at each stage of the iterative process.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The problem is of interest. The proposed solution is sound. The experimental design is poor and still at the preliminary stage. The manuscript is neglecting relevant information to properly support the interpretation of results.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The paper proposed a semi-automatic entropy-based active learning method for white matter segmentation, which shows good performance on tumor data.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The motivation of this paper, improving tract segmentation performance on subjects with strong anatomical aberrations, is meaningful.
- The method adopted active learning to involve expert interaction, which benefits tract segmentation performance.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The tract segmentation results is not properly evaluated. First, only limited SOTA methods are compared and advanced data-driven segmentation methods are ignored. For example: Zhang, Fan, et al. “Deep white matter analysis (DeepWMA): Fast and consistent tractography segmentation.” Medical Image Analysis 65 (2020): 101761. Second, more evaluation metrics should be adopted to evaluate the performance. Third, the paper claims the method to be quick but the computation time is not compared. Fourth, the method could be evaluated on public tumor dataset.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
It might be difficult to reproduce because it includes in-house data.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Give more comprehensive evaluations of the performance of the method.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The motivation of this task is meaningful but the performance results are not convincing.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This work proposes an active learning framework for white matter tract segmentation. All reviewers agree on the novelty of the paper and see its potential in dealing with clinical applications. Overall this is a solid paper to be accepted at MICCAI this year. The reviewers also provide constructive feedback to improve, e.g., experimental comparisons and characterizing the data. The authors please make proper updates accordingly.
Author Feedback
We are very grateful to all reviewers for their constructive comments and appreciation of our work.
- Reviewer 1 and reviewer 3 criticized that the tumor data set is not public. We plan to release the data, but unfortunately this is a rather long and complicated process related to data protection regulation and will take more time.
- To address reviewer 1’s comment to better characterize the data, we will add more information of the shape characteristics of the bundles of the tumor patients between healthy subjects and patients.
- Reviewer 1 expects a more detailed description of the interactive model. In this work, we have focused more on the methodological aspects of the active learning approach to the tract segmentation, and due to space constraints, we have not added a detailed description of the interactive model with all its aspects. However, we agree that this is an important aspect and future work will describe this interaction in more detail and include further experiments to better characterize this approach.
- Reviewer 1 suggested an interesting approach of using an atlas as an initializer for the proposed approach. In principle we agree with the reviewer that this could be a promising approach. The main use-cases of atTRACTive, however, are the ones where atlas-based approaches do not yield good results, such as tumor datasets, or where no atlas exists, e.g. for certain animals or tractography of the muscle.
- Reviewer 1 is concerned how feasible the approach is for bigger bundles. The actual purpose of atTRACTive is to avoid the annotation of all streamlines by annotating only the most informative ones. Therefore, only a small number of annotations is required, even if the bundle contains thousands of streamlines.
- Reviewer 2 suggests adding the performance of Classifyber for tumor data to Figure 4. Regrettably, Classifyber does not provide the functionality of OR segmentation (the limitation to fixed sets of tracts is one of the drawbacks of fully automated methods based on supervised machine learning).
- We agree with Reviewer 3, that the inclusion of additional Sota methods is important to obtain a comprehensive picture. Although this is beyond the scope of this preliminary study, we plan to include additional approaches, including DeepWMA, in our upcoming experiments.
- Reviewer 3: The computation time is highly dependent on the number of streamlines in the data, which makes it difficult to be precise on exact computational times. The experiments proposed in this work have shown overall segmentation times (including the time to compute) to be shorter than manual ROI based methods. Further experiments will more extensively investigate the required user interaction and effort.