List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Meng Jia, Matthew Kyan
Abstract
We propose a novel framework that applies atlas-based whole-brain segmentation methods to tumor-bearing MR images. Given a patient brain MR image where the tumor is initially segmented, we use a point-cloud deep learning method to predict a displacement field, which is meant to be the deformation (inverse) caused by the growth (mass-effect and cell-infiltration) of the tumor. It’s then used to warp and modify the brain atlas to represent the change so that existing atlas-based healthy-brain segmentation methods could be applied to these pathological images. To show the practicality of our method, we implement a pipeline with nnU-Net MRI tumor initial segmentation and SAMSEG, an atlas-based whole-brain segmentation method. To train and validate the deformation network, we synthesize pathological ground truth by simulating artificial tumors in healthy images with TumorSim. This method is evaluated with both real and synthesized data. These experiments show that segmentation accuracy can be improved by learning tumor-induced deformation before applying standard full brain segmentation. Our code is available at https://github.com/jiameng1010/Brain_MRI_Tumor.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16443-9_24
SharedIt: https://rdcu.be/cVRyC
Link to the code repository
https://github.com/jiameng1010/Brain_MRI_Tumor
Link to the dataset(s)
https://www.med.upenn.edu/cbica/brats2020/data.html
https://www.nitrc.org/frs/?group_id=546
Reviews
Review #1
- Please describe the contribution of the paper
The authors aim to segment the brain despite the presence of large tumors. Given a brain tumor segmentation, the proposed method uses synthesized images with accompanying ground truth deformation fields of mass-effect and infiltration to learn the tumor-induced deformation to the rest of the brain with a point cloud network. During inference, a tumor is injected into an atlas (including both a new tumor label plus deformation of the rest of the brain), and this new patient-specific atlas is used in atlas-based segmentation. The main experimental results include an improvement of about ~4 Dice over SAMSEG for subcortical labels on the tumor side in simulated images, plus qualitative results on real images from BraTS.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Very interesting problem to segment brain structures despite very large tumors
- Method is sound, interesting to smartly inject the tumor into the atlas including deformation of other brain structures in the atlas.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- A big caveat of this paper is that the tumor must already be segmented.
- Validation is missing baselines, which could include simple ones like basic tumor synthesis, atlas registration with tumor masking, etc. The main comparison is against SAMSEG. If I’m understanding correctly, the second baseline (called Our(tau=0)) inserts the segmented tumor into the atlas but never actually deforms the atlas to the new image. In my opinion, this is an irrelevant baseline since no one would reasonably do this.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- Several aspects missing, e.g., the number of training and evaluation runs, details of baseline methods, details of train/validation/test splits,
- No standard deviations or statistical significance tests for the results of synthesized images, clinical implications
- The parameters for the point cloud network and its training (architecture, batch size, learning rate, etc) are not in the main text nor the supplemental material
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- I’m not convinced that the inter-hemisphere symmetry validation is very insightful. For such big tumours it seems to be near impossible to attain true symmetry (since one would have to majorly deform the image to eliminate the tumor). The reported symmetry improvement is very small. Is this meant to capture the effect of tumor-induced deformation on the rest of the brain only?
Small comments:
-
Please clarify the use of the parameter tau when defining the displacement field and stationary vector field. I don’t see why this scaling factor is needed when def(q P) is given by the network, which would hopefully just return the correct deformations directly. Why does it make sense to have a user adjust the degree of deformation? - In the introduction, consider rephrasing “grows a tumor in the atlas is still highly complex and challenging” and “warping (grow a tumor in) the brain atlas to reflect the tumor and spatial changes in the patient image” because these phrases can sound very similar to what you’re doing. Also avoid criticizing “these approaches require a good tumor segmentation” because your method does too.
-
Let the tuple… before eqn (1): there are two errors, it should be D_i(q_j^(i) j= [1…J] - i.e., subscript (i) and the … between 1 and J. - Section 3.2, “Our method, including the baseline, significantly outperforms plain SAMSEG” - I wouldn’t use terms like “significantly” if you’re not doing a statistical test.
- Figure 5 is hard to understand without ground truth segmentations, which I understand may not be available. Can you use arrows or boxes to highlight the main areas the reader should look at and give more information about these areas in the caption or main text?
- Some typographical errors, paper needs a grammar and spelling check
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
It’s a good problem and the method is well-formulated and looks as if it should work, but the lack of suitable baselines in the experimental results is concerning.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
I’m keeping my score as is - overall I like the idea, but the validation is a bit lacking (as the authors say though, there is no ground truth whole brain segmentation for images with tumours, so it’s a tough problem).
Review #2
- Please describe the contribution of the paper
The authors have proposed a method to learn brain deformation induced by tumors to improve brain MR segmentation. To this end, they have trained a point-cloud deep learning network to learn deformation caused by the growth of the tumor and used it to warp a healthy brain atlas so that it can be used for pathological images. They have mainly used synthetic data for the validation of the proposed method and real dataset is only used for qualitative evaluation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper is very well-written and clear.
- The use of synthetic data for the evaluation of the method is valuable.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The novelty of the work is very limited.
- Lack of quantitative validation of the model using real data is the most important weakness of the method. Without evaluation using a real dataset, it is not possible to correctly evaluate the performance of the method.
- The method relies on an initial automatic tumor segmentation, and it is not clear how sensitive the method is to the quality of the initial segmentation.
- Training a network using only synthetic data especially
- There is no comparison with the state-of-the-art methods.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors have not provided any code. The description of the model is clear and well-written.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- The method relies on initial tumor segmentation. How sensitive the method is to this initial segmentation?
- Please provide a measure of variability for the provided Dice scores.
- What does the author mean by weighted-mean-Dice? How was it calculated?
- The authors should quantitatively evaluate their method on real data and compare it with some of the state-of-the-art methods.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- There is no validation using real data.
- Technical novelty is very limited. The proposed point-cloud deep learning network has already been used in the literature.
- Number of papers in your stack
3
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
3
- [Post rebuttal] Please justify your decision
The paper does not have a real-world evaluation, and the explanation regarding the novelty of the work is not convincing.
Review #3
- Please describe the contribution of the paper
The paper proposes a novel whole brain segmentation method in the presence of pathology (tumours). The method first predicts the displacement field induced by the tumour using a point based deep network trained from synthetic data. Next, an atlas is deformed with the predicted deformation field to account for the tumour growth. Finally this atlas is matched with the current image to produce the final brain segmentation.
The regression deformation network uses two models - (1) a direct displacement field and (2) a diffeomorphism represented using a SVF. The network is trained using synthetic data - simulated tumours using TumorSim [ref 26]
The method is evaluated in 3 ways: (1) by testing the brain symmetry after reversing the computed deformation field; (2) using data that synthesize pathological ground-truth by fusing well-labelled health brain images (MindBoggle-101) and tumor scans (BraTS) (3) using real tumour data from BRATs (only qualitative evaluation)
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The proposed point-based deep regressor used to learn the tumour deformations is interesting. I like the idea of improving on traditional methods with deep learning components rather than blindly training deep models to perform everything. This particular task is also lacking ground truth segmentation data that would be required by a deep learning method.
The paper is well written and clear.
Despite the lack of ground truth, the author proposed few ways to evaluate the method.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Evaluation:
- is there any explanation on why the SVF-based approach performs worse than the plain deformation-based approach
- I agree that the method shows clear improvement if compared with the traditional atlas based method. I am wondering how would it compare with a pure segmentation method trained on the same type of synthetic data the evaluation is done on
- also, the method could have been compared with a traditional model-based approach [ex. ref 1,11, 29] where the tumour growth is simulated using a reaction-diffusion equation
Synthetic data used in testing - how are the tumour deformations simulated ? are the author using the same TumorSim method ? I would be a bit concerned that both training and testing data are simulated using the same method.
In terms of references, there are related works on tumour growth simulation using deep learning. The authors could mention the connection.
In conclusion, the method is sound, but I have some concerns with the evaluation.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper is clear and detailed. Precise parameters of the regression network are not given. There are also missing some details in how the synthetic data is generated.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
While a realized that adding the comparison with another method would be a lot of work, I would recommend at least adding to the discussion of the results (ex if there are any insights into why SVF representation performs worse in experiments 3.2; SVF guarantees a valid diffeomorphic field and adds implicit regularization ; would a PDE model-based method would perform similarly ?)
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The method proves an improvement over the traditional atlas-based segmentation. I like the idea of combining deep-learning and modeling approaches. The regression network on its own is well formulated.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper presents a point cloud network for learning deformation of brain tissues due to a tumor. Author feedback should in particular refer to the comment on brain symmetry, the generation of synthetic data, the sensitivity to the accuracy of the initial tumor segmentation. A discussion on the advantages of the proposed contribution with respect to alternative methods should be added.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5
Author Feedback
We appreciate the reviewers’ time and their helpful opinions on our work. As consistently pointed out by all reviewers, our results are limited in terms of real-world evaluation. The main reason is the lack of a real tumor dataset with whole-brain ground truth (GT). We agree with R#3 that using deep-learning to boost traditional methods is desirable. We consider the novel contribution here (R#2’s concern), to be the boosting of traditional atlas-based methods (e.g., SAMSEG), with abilities learned from synthetic deformation. For this, our experimental validation is quite convincing. Meanwhile, our evaluation using symmetry and method for synthesizing labeled image data for evaluation can also be considered contributions. 1) Directly assessing non-rigid deformation is fundamentally hard. Our symmetry metric is inspired by a common strategy of evaluating non-rigid registration [8], which assesses inter-subject registration by looking at the degree of overlap of image segments after applying a predicted displacement. The brain hemisphere symmetry score is meant to validate the deformations learned by our network. Notice that the network is trained purely by regressing the deformation without any target on overall symmetry. Most tumor sizes in BraTS are not large (Fig3 is perhaps misleading as it shows an extreme sample). When implementing our TumorSim, the tumor size distribution appears Poisson (with Lambda 0.5~1.0 and mean size~=40cm^3). Symmetry improvement (R#1’s concern) is small because tumor-induced symmetry decay is predominantly small in this dataset. We concur with R#1 that symmetry primarily monitors the effect of the tumor deformation on the rest of the brain. 2) Regarding lack of comparisons: we agree with R#1 that this work could benefit from an extension to other baseline comparisons. SAMSEG has been extended to deal with segmenting tumors that induce deformation [24]. We consider an implementation of this as our baseline, as it permits us to directly assess the boost achieved with our learned deformation model. Ideally, more quantitative evaluation with against [1,11] is preferred and is intended for future work. The challenge is the lack of real-world datasets that include full brain GT, against which such comparison should be made. Currently [1,11] perform joint segmentation and registration, and so to provide a fair comparison, an analysis of automated input segmentations should also be considered. Again, we are eager to conduct this analysis given the boost we appear to get from our deformation learning reported here. R#3’s question about why SVF performs worse than direct deformation: The SVF training data is approximately computed and may be over-smoothed. When applying the deformation to the atlas, transformation to displacement is performed again and thus smoothed twice. Regarding R#3’s concern about using TumorSim for generating and synthesizing test data: TumorSim functions a brain model (tetrahedrons with stiffness properties) and a set of reaction-diffusion equations. It produces varied displacement fields by varying initial seed positions with additional randomness (using a half-normal distribution). The test set is quite different from training as they are produced from different seeds. 3) Our method requires an initial segmentation. The deformation field produced by the network will be affected by any inaccuracy here. However, the final whole-brain segmentation may not be that directly sensitive due to the generative modeling and Bayesian decision-making in SAMSEG. Roughly speaking, the scale factor \tau can be used to compensate for the incorrect overall size of the tumor and edema. And the “tumorous” probability \alpha appended to the probabilistic atlas can be adjusted based on how confident the initial segmentation is. For example, if it is a rare type of glioma that has little representation in the training set. The user may try a dilation on the tumor mask but assign a smaller probability.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
-
Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
-
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
-
What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors proposed a method to learn brain deformation induced by tumors to improve brain MR segmentation. The rebuttal addressed the concerns of no validation on real data, symmetry metric and insufficient comparison. Overall, this is an interesting paper where merits slightly weigh over weakness.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
7
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The main concern raised by the reviewers is on evaluation. The paper did not include comparison with state of the art methods and there is no validation on real data. The authors provided logical explanation in the rebuttal, which is the lack of real-world datasets that include brain tumor and full brain annotation. The clarification on using symmetry metric as a validation method also makes good sense. Although one reviewer raised novelty concerns, I tend to agree with the authors and the other reviewers that using learned synthetic deformation to improve atlas-based methods seems interesting and new.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
8