List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Balint Kovacs, Nils Netzer, Michael Baumgartner, Carolin Eith, Dimitrios Bounias, Clara Meinzer, Paul F. Jäger, Kevin S. Zhang, Ralf Floca, Adrian Schrader, Fabian Isensee, Regula Gnirs, Magdalena Görtz, Viktoria Schütz, Albrecht Stenzinger, Markus Hohenfellner, Heinz-Peter Schlemmer, Ivo Wolf, David Bonekamp, Klaus H. Maier-Hein
Abstract
Data augmentation (DA) is a key factor in medical image analysis, such as in prostate cancer (PCa) detection on magnetic resonance images. State-of-the-art computer-aided diagnosis systems still rely on simplistic spatial transformations to preserve the pathological label post transformation. However, such augmentations do not substantially increase the organ as well as tumor shape variability in the training set, limiting the model’s ability to generalize to unseen cases with more diverse localized soft-tissue deformations. We propose a new anatomy-informed transformation that leverages information from adjacent organs to simulate typical physiological deformations of the prostate and generates unique lesion shapes without altering their label. Due to its lightweight computational requirements, it can be easily integrated into common DA frameworks. We demonstrate the effectiveness of our augmentation on a dataset of 774 biopsy-confirmed examinations, by evaluating a state-of-the-art method for PCa detection with different augmentation settings.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43990-2_50
SharedIt: https://rdcu.be/dnwL5
Link to the code repository
https://github.com/MIC-DKFZ/anatomy_informed_DA
https://github.com/MIC-DKFZ/batchgenerators
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
A study into the use of Anatomically realistic Data augmentation applied to training of a deep learning model for the detection of prostate cancer on MRI. A realistic deformation model is presented that is fast to allow deployment in training a deep learning model. It extents the simple nnUNet augmentation strategies by a smart anatomically driven strategy. Examples are show of realistic augmentations and a small improvement in model performance was shown.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Data augmentation is an important tool in optimizing the use of limited medical imaging data. A novel augmentation strategy that makes sense, produces realistic results and shows a small but clear improvement in diagnostic performance is a great contribution to the deep learning for medical imaging field. The paper is very well written, a vert modest claim perfectly demonstrated with a clear method and associated experimental data. After reviewing a series of abstracts that all ‘‘significantly outperform SOTA’’ and ‘‘solve all problems’’ this paper is a breeze to read. Appreciate the solid scientific approach, an example to MICCAI.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
To really prove the point it would be rock solid to demonstrate this on a couple of other clinical tasks. Adding a few fine-tunings the performance gain is likely to be significant and a great high impact paper should be feasible. I assume you’re in contact with the nnUNet team and are probably integrating this. Cannot wait to use it. Another weakness is a significant increase in annotation effort. It would great to apply this to a anatomy segmentation model. I don’t think it needs to be all that accurate to still achieve the gain.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
This is a 100% reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
See my comment 6. Maybe the pAUROC is a bit confusing. Why not AUC? The choice of threshold for pAUCROC is also unclear.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
8
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Reads well, convinces me, and backs up with solid evidence. Nice. Great MICCAI abstract.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
Proposed an anatomy-informed transformation that leverages information from adjacent organs to simulate typical physiological deformations of the prostate and generates unique lesion shapes without altering their label.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The inclusion of information from adjacent organs to simulate prostate deformations is a bright idea. Alleviates limitations in common model’s ability to generalize to unseen cases with more diverse localized soft-tissue deformations. The proposed model requirements are computationally more efficient, which can be easily integrated into common DA frameworks.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) Using adjacent organ information to extrapolate target organ deformations are not uncommon. Applications in the prostate might be few, but we need more reviews on related methods not only limited to prostate application. 2) The training dataset used in this paper has very limited size so it is difficult to decide potential improvements on any (and more complex) real scenario. 3) It seems the model depend heavily on existing good segmentations. So applications under more realistic settings are more difficult to achieve than currently presented. Inclusion of an automated segmentation process or any evaluation on its impact in the pipeline would make the method more complete.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Reproducible. Method details described well.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
This is a very interesting work with some good innovation. The paper is also well written. It is worth being published in some form. Though considering the novelty required in MICCAI, I am not totally convinced to recommend a full acceptance. The hardest job for the authors is to show the value of this new method and justify its improvements in a more realistic scenario. But currently it feels weak for me.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1) This is an overall interesting work. Very good innovation. Manuscript well written. Has scientific value. The novelty is good enough for a weak accept. 2) The weaknesses of the paper was listed above. Addressing any of those points would increase the presentation value of this work.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
No extra information from rebuttal to change my initial impression
Review #3
- Please describe the contribution of the paper
The paper proposes an anatomy-informed data augmentation technique for prostate cancer detection on MRI. The technique leverages information from adjacent organs to simulate typical physiological deformations of the prostate using a mathematical FEM model, resulting in a wider range of organ and lesion shapes in the training set. The authors evaluated effectiveness of the proposed approach on a dataset of 774 biopsy-confirmed examinations, leading to improved model performance in PCa detection on an independent test set.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper’s concept of using anatomy-informed data augmentation for prostate cancer detection on MRI was found to be interesting and valuable. The proposed method leverages information from adjacent organs to simulate typical physiological deformations of the prostate, resulting in a wider range of organ and lesion shapes in the training set. This could potentially improve the performance of computer-aided diagnosis systems for prostate cancer detection. Additionally, the paper is well-written, well organized, and easy to understand. The data collection and experimental design are rigorous.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
As elaborated below in comments to the athors, the paper’s proposed method was found to be inadequately explained, which limits its replicability. The evaluation of the proposed method was also deemed insufficient as it was tested on only two clinicians who are still in training and used only one backbone segmentation model. Furthermore, the reported performance improvements were highly variable and lacked statistical analysis to confirm their significance.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper is unlikely to be reproduccible with the current information provided.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The paper’s concept of using anatomy-informed data augmentation for prostate cancer detection on MRI was found to be interesting and valuable. The proposed method leverages information from adjacent organs to simulate typical physiological deformations of the prostate, resulting in a wider range of organ and lesion shapes in the training set. This could potentially improve the performance of computer-aided diagnosis systems for prostate cancer detection.
However, it was noted that the method has not been adequately explained in the paper and is not reproducible based on the information given. Insufficient detail was provided on the implementation of the proposed method, which limits the potential for others to replicate the approach. For instance, there is no explanation of the approach for obtaining vector fields from adjacent MRI images. As a result, the effectiveness of the method cannot be properly evaluated.
The evaluation of the proposed method was found to be insufficient. The proposed method was only tested on two clinicians/radiologists who are still in training. It would be more appropriate to include practicing radiologists in the Turing test as the final medical diagnosis is always reviewed by an attending physician. It would also be relevant to present augmented images of both the proposed approach and random deformable transformations and compare the sensitivity of radiologists to their realness. Additionally, the proposed approach was evaluated using only one backbone segmentation model. A more agnostic approach to the backbone model used is required to ensure that the method can be applied in various contexts and using various backbone models. Furthermore, the reported performance improvements in Table 1 were highly variable. It is advisable that the authors should perform statistical analysis to confirm the significance of their results.
In summary, this review acknowledges the interesting and valuable concept of using anatomy-informed data augmentation for prostate cancer detection on MRI. However, the lack of detail for replication of the proposed method and insufficient evaluation due to the limited number of radiologists and backbone segmentation model used hinders the proper evaluation of the proposed method.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I acknowledge the interesting and valuable concept of using anatomy-informed data augmentation for prostate cancer detection on MRI. However, the lack of detail for replication of the proposed method and insufficient evaluation due to the limited number of radiologists and backbone segmentation model used hinders the proper evaluation of the proposed method.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
4
- [Post rebuttal] Please justify your decision
After carefully examining the authors’ response to my comments and those of other reviewers, I have found that some of my concerns have been adequately addressed. As a result, I am pleased to revise my rating by increasing it by one point. However, it is essential to note that a significant limitation remains unaddressed by the authors, namely the potential dependency of the proposed augmentation approach on the choice of backbone model and datasets utilized.
Although the study shows a slight improvement in cancer detection performance using an nnU-Net model on a specific dataset, the value of an augmentation approach lies in its ability to enhance performance across diverse datasets and model backbones. Consequently, it is imperative for this augmentation approach to demonstrate consistent performance gains that are independent of these factors.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This study focuses on using 3D biomechanical models to simulate plausible changes in the anatomy of the prostate, and use it as an augmentation strategy.
The strength of the study includes:
- simulating anantomic changes in the bladder and rectum to create more realistic augmentations.
- Addressing the slow nature of FEM to induce physically plausible changes in the prostate.
- Great study design, including both clinicians to assess how plausible the deformation is and automated detection methods to study the improvements in performance.
- Visually, the deformations look very realistic, and I can see how this approach actual helps training.
No major weaknesses were found in this study. It would be great to see how these types of augmentations work in other organs. Also, provide some guidance regarding whether all 700 or so cases had the bladder and rectum segmentated. How many more lesions were detected using the proposed approach (vs classic augmentation), and as such can you comment whether the extra effort in annotation was worth it.
Please address the aforementioned weaknesses and those mentioned by the reviewers.
Author Feedback
We thank the Meta Reviewer (MR) and all Reviewers (R) for their time and valuable feedback. We greatly appreciate their motivation to extend our approach to other organs and clinical tasks. We are committed to applying our approach as a blueprint in the future for addressing similar challenges in different soft tissues. Furthermore, we provide explanations for their raised questions and concerns, hoping that they will increase their rating:
- MR, R1, R2 were interested in the details of the organ segmentation and whether an automated setting can be achieved. All organ segmentations were automatically predicted by a model built upon nnU-Net (Isensee 2021) trained iteratively on an in-house cohort initially containing a small portion of our cohort. Multiple radiologists supported the quality of the predicted segmentations. We extended our manuscript with this information. This extra effort enables us to automate our process for future studies by easily extending our training cohort upon newly available clinical exams.
- MR, R2, R3 raised questions about the significance of results and how many more lesions were detected. At the chosen working point of 0.32 average number of false positives per scan, our method helped to identify 4 more lesions (5.3%) from the 76 lesions in our test set. The working point was selected by calculating the radiologists’ lesion level performance at PI-RADS 4 threshold. Using our bootstrapping results, we calculated p-values using t-test, and we found this improvement significant. Similarly, we got significant improvement for the patient-level working point used for the F1 score calculation. We add this information to our manuscript.
- R3 had concerns about the reproducibility. Our proposed method will be made publicly available in a medical data augmentation and nnU-Net framework with all implementation details to enable easy access and replication, as already stated in the manuscript. Furthermore, we provided a mathematical formula following standard notation with detailed explanation and hyperparameters for the proposed transformation, including the operator for obtaining the necessary vector field. To enhance further clarity, we extend the appendix with more implementation details in a structured manner.
- We agree with R3, including the random elastic deformation into the Turing test would provide a more complete picture. We have done so and the radiologists easily and immediately detected transformation irregularities, in contrast to our method being challenging to detect its artificial nature. We extend our results with this finding.
- R3 noticed high standard deviations in the results. We corrected minor mistakes in the clinical information and the evaluation algorithm, resulting also in lower standard deviations. The findings of the main values remained the same.
- R2: To the best of our knowledge, the mentioned organ deformation models are based on complex networks e.g. with population statistics (Romaguera 2021) or FEM simulations (Pfeiffer 2019), none of them have successfully integrated such information into model training as an online augmentation in an efficient manner. We extend our introduction with these examples, further supporting the novelty of our lightweight method.
- R2 finds the training heavily dependent on good segmentations. The Gaussian kernel in our transformation allows certain errors in the organ segmentations as we discussed in the manuscript. To support this claim, we shifted the organ segmentations randomly during training in the range of 2 voxels and we got similar results in the selected working points.
- We agree with R2 that the training dataset is limited in size. However, the ground-truth assessment of extended systematic and targeted biopsies provides sensitivity for PCa comparable to radical prostatectomy which is complemented by expert centre radiologist assessment. The dataset represents high-standard clinical practice well-suited for method developement.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors addressed the metareviewers questions and the points raised by the reviewers. I recommend acceptance of this higliy interesting study.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Development of biomechanical framework for anatomically realistic data augmentation applied to training of a deep learning model for the detection of prostate cancer on MRI. Strengths include intuitive, novel approach to augmentation, well presented results, and reasonable validation. Rebuttal addresses question about segmentation, significance of results (which exists, even if marginal), reproducibility. Additional clarification provided on using elastic deformation, std deviations, previous work, and dependence on segmentations. Overall, it appears the paper is well suited to MICCAI.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The work is interesting and the authors have provided a good rebuttal to clear reviewers concerns. I agree with R3 that a limitation of the method should be added to the camera ready related to “The potential dependency of the proposed augmentation approach on the choice of backbone model and datasets utilized.