List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Bin Wang, Lin Teng, Lanzhuju Mei, Zhiming Cui, Xuanang Xu, Qianjin Feng, Dinggang Shen
Abstract
Accurate dose map prediction is key to external radiotherapy. Previous methods have achieved promising results; however, most of these methods learn the dose map as a black box without considering the beam-shaped radiation for treatment delivery in clinical practice. The accuracy is usually limited, especially on beam paths. To address this problem, this paper describes a novel “disassembling-then-assembling” strategy to consider the dose prediction task from the nature of radiotherapy. Specifically, a global-to-beam network is designed to first predict dose values of the whole image space and then utilize the proposed innovative beam masks to decompose the dose map into multiple beam-based sub-fractions in a beam-wise manner. This can disassemble the difficult task to a few easy-to-learn tasks. Furthermore, to better capture the dose distribution in region-of-interest (ROI), we introduce two novel value-based and criteria-based dose volume histogram (DVH) losses to supervise the framework. Experimental results on the public OpenKBP challenge dataset show that our method outperforms the state-of-the-art methods, especially on beam paths, creating a trustable and interpretable AI solution for radiotherapy treatment planning.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16449-1_55
SharedIt: https://rdcu.be/cVRXt
Link to the code repository
https://github.com/ukaukaaaa/BeamDosePrediction
Link to the dataset(s)
https://competitions.codalab.org/competitions/23428#learn_the_details-overview
Reviews
Review #1
- Please describe the contribution of the paper
The presented work introduces a new strategy for deep-learning-based radiotherapy dose prediction. The proposed method includes two main innovations: 1) A neural network is used to predict the radiation dose along the paths of the individual radiation beams before averaging the obtained dose maps. Explicitly encoding the prior knowledge of beam-wise radiation delivery has the goal of improving accuracy along the beam paths. 2) The authors introduce two loss functions, value-based DVH loss and criteria-based DVH loss, with the aim of placing more emphasis on the dose prediction in the regions containing the tumor and critical organs. In extensive benchmarking experiments using the publicly available OpenKBP dataset, the authors show that their method outperforms seven other algorithms.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
The newly introduced strategy to include the beam directions as prior knowledge makes intuitive sense. It is elegantly implemented in form of the beam masks, which can be efficiently calculated and processed.
-
The proposed method has been extensively compared to seven relevant baseline methods. Additionally, ablation experiments were conducted to quantify the benefit of the newly introduced components. All benchmarking experiments were based on the publicly available OpenKBP dataset.
-
The manuscript is clearly structured and nicely illustrated, making it easy to understand the main ideas and conducted experiments.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
I do not understand the reasoning behind the value-based DVH loss. In order to calculate a dose volume histogram, voxels receiving a similar amount of radiation dose have to be binned together (see Nguyen et al., “Incorporating human and learned domain knowledge into training deep neural networks: A differentiable dose-volume histogram and adversarial inspired framework for generating Pareto optimal dose distributions in radiation therapy”, Medical Physics 47.3 (2020): 837-849, reference 10). Without such thresholding or binning operations, the inner part of equation 3 simply becomes the average error - the sorting operation does not change this.
-
Radiotherapy dose prediction is one important step for radiotherapy treatment planning and has been arguably the most researched one in the recent deep learning era (Ge and Wu, “Knowledge‐based planning for intensity‐modulated radiation therapy: a review of data‐driven approaches.” Medical physics 46.6 (2019): 2760-2775). However, in order to generate an actual treatment plan, a set of radiation beam arrangements, shapes and intensities that delivers the desired dose distribution has to be found. While experiments demonstrating the utility of the proposed method as starting point for such an optimization may exceed the scope of this study, the authors should at least discuss how exactly they envision their method to fit into the clinical workflow and acknowledge that the proposed method offers no guarantee to generate a radiation dose distribution that is physically deliverable.
-
One aim of the work was to specifically improve the prediction accuracy increase along the radiation beam paths. However, dedicated experiments that quantify the obtained benefit have not been included.
-
Currently, several baseline methods are not included in the result figures 4 and 5.
-
In several passages of the paper, the authors use the terms “statistic” and “significant” without having performed any statistical analysis. The authors should either include a statistical test to determine whether the observed improvements are significant or refrain from using these terms altogether.
-
There are a few grammar and wording mistakes throughout the manuscript that should be corrected.
-
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors have essentially answered “yes” to every question in the reproducibility checklist. However, in reality most of the relevant points have not been included in the manuscript. While not all raised questions can be addressed in the manuscript, the checklist should be filled out correctly. I believe the most interesting information to add to the manuscript would be how the proposed method’s hyperparameters were tuned (especially the loss function hyperparameters alpha and beta), how the baselines were implemented and tuned and additional discussion of the clinical significance (see comments above).
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
I believe the authors should aim to address the main weaknesses of the paper outlined above. In particular, the value-based DVH loss function should be re-evaluated and checked whether it is indeed equal to the average error. In case I have have misunderstood the corresponding paragraph, the authors should consider reworking it to improve its clarity. The experiments and results section should be updated to include all baseline methods. The authors should consider including dedicated experiments that evaluate the quality of the dose map predictions along the beam paths. Finally, I believe the authors should aim to more carefully discuss how their work fits into their vision of an “AI solution for radiotherapy”.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I believe the core idea of this work - decomposing the dose prediction utilizing beam path masks - is an interesting concept that has been well implemented via the proposed method. However, I fail to see the rationale behind the value-based DVH loss, the other main contributions claimed by the authors. Additionally, I have concerns regarding the incomplete experiments and the lack of discussion how this work fits into the full clinical radiotherapy pipeline.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
The authors have put in considerable efforts addressing the referees’ comments and have promised to substantially expand the paper. As a result, I believe the paper’s quality and impact has markedly improved and most of my concerns have been resolved. However, I have two minor comments I believe the authors should aim to address:
-
After understanding the authors’ sensible explanation regarding the value-based DVH loss, I reread the corresponding section in the paper and still found it hard to understand. In particular, I am missing an explicit description of the binning operation and an explanation how the DVH calculation exactly differs from reference 10. The description of the value-based DVH loss should be further improved and potentially expanded.
-
Please clearly indicate which values in table 1 have been taken from the respective publications.
-
Review #2
- Please describe the contribution of the paper
This paper proposes a deep-learning based dose prediction method. It generates beam mask to represent an irradiation boundary as a prior knowledge, and applies “disassembling-then-assembling” strategy for the dose prediction.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This paper proposes a novel deep-learning based dose prediction method, which generates beam mask to represent an irradiation boundary as a prior knowledge.
- This paper evaluates the estimation accuracy using a grand challenge dataset and showed the proposed method outperformed the state-of-the-art methods.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- No statistical analysis has been made for evaluation, and then it is unclear that the result can be confirmed. The authors should show the variance of the resultant scores. In addition, statistically significant difference should be discussed.
- Improvement from the existing studies seems to be small.
- Each radiotherapy beam might have a machine-specific dose distribution. However, it does not assume it.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- This paper describes the experimental conditions for evaluation well.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- This paper seems to take a promising approach.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This study propose a novel dose estimation method and showed experimental results using a grand challenge dataset. The reviewer found that this study is interesting. However, this paper has a room for improvement.
Major concerns:
- No statistical analysis has been made for evaluation, and then it is unclear that the result can be confirmed. The authors should show the variance of the resultant scores. In addition, statistically significant difference should be discussed.
- Each radiotherapy beam might have a machine-specific dose distribution. However, it does not assume it. Please discuss the validity for the proposed method that does not assume a machine-specific dose distribution.
- Improvement from the existing studies seems to be small.
Minor comments:
- Table 2 appears earlier than Table 1. Table 2 should be placed later in the paper.
- Number of papers in your stack
3
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #4
- Please describe the contribution of the paper
This paper presents a framework for head and neck radiotherapy planning dose prediction, including global coarse dose prediction and beam-wise dose prediction based on decomposition and multibeam voting mechanism. Some experimental results on the public OpenKBP challenge dataset are provided.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
The general idea of “disassembling-then-assembling” strategy to consider the dose prediction task seems interesting.
-
The proposed architecture looks workable to this specific task.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
Some of the key technical components are introduced without sufficient details.
-
The evaluation and comparison are not sufficient.
-
A fair comparison description should be provided in the experiments.
More detailed comments are given in the following Sec. 8.
-
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Seems okay since author promised to provide the data and code in the reproducibility checklist. Otherwise, it is unclear to say since some key network architecture details are not provided in the paper itself.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
There are some concerns of this paper:
-
In recent years, many deep learning-based methods have been proposed to handle this dose prediction task, what are the limitations of the previous methods? Authors lack to discuss them in the introduction section.
-
There is no clear information about what network architectures are used in Global Dose Network and Beam-wise Dose Network.
-
It is unclear how they realize the “disassembling-then-assembling” strategy at the network design level.
-
The experiment evaluations are a bit insufficient:
-
In Fig. 4, how about other views in the visualization results?
-
It seems that there is no clinical expert involved in the evaluations.
-
All comparison methods should be provided in the visualization of DVH curves (Fig. 5).
-
How did you implement the methods, which do not have the source codes in public, such as Lin et al. [5], Gronberg et al. [3], etc.? More information should be provided for this in order to guarantee a fair comparison.
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper proposes a global-to-beam network to conduct the dose prediction task. Meanwhile, there are some concerns on unclear technical/network description, insufficient comparison and evaluation, esp. to justify that it is really useful and practical in the clinical studies.
- Number of papers in your stack
6
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
4
- [Post rebuttal] Please justify your decision
Thanks for the authors’ feedback. However, the major concerns of the technical novelty and insufficient (clinical) evaluation are still not well explained/addressed.
Finally, I am giving a “weak reject” based on its overall quality comparing to other (published) MICCAI papers.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The manuscript presents a deep learning approach for radiotherapy dose prediction. This is performed by predicting dose along individual radiation beams followed by a beam averaging. Experimental results are shown for the OpenKBP dataset.
Strengths of this method are an intuitive and interesting strategy of disassembling then assembling strategy making use of beam masks as prior knowledge.
However, there are notable weaknesses. First, all reviewers mention that the evaluation is not sufficient. Critiques include how comparison methods were implemented, the statistical analysis of the method (noted that the authors use significant without presenting statistical tests showing significance), variance of scores within the dataset, and baseline methods not being included in the qualitative results. Also if any experts evaluated the qualitative results or where all comparisons quantiative.
There were also some questions/concerns about how the dose predictions would be used within the clinical pipeline, for instance is there a guarantee the doses can be implemented within a clinical workflow.
One reviewer had serious questions about the value-based DVH loss and how it was implemented, how is this different from average error? And another reviewer generally found the technical components presented without sufficient detail for understanding/implementation.
I recommend this manuscript for rebuttal as there is a clear and interesting technical contribution, however the authors need to clarify their technical implementation and justify the evaluation of their method to demonstrate they are improving over the state of the art.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
9
Author Feedback
We thank all the reviewers for their acknowledgment of the technical novelty in our paper. Below, we will clarify confusions about our method mentioned by the reviewers.
Statistical analysis (AC & R1 & R2) We update our results with p-value (t-test) and standard deviation (SD) for all methods. 1) All the p-values are <0.05, showing the statistically significant improvement of our method. 2) Our method also got low SD on both Dose score (±1.013) and DVH score (±1.163), indicating that our method is more stable than other methods.
Fair and consistent comparison with SOTA (AC & R1 & R4) 1) We compared our method with 7 SOTAs. 4 of them (including 1st and 2nd of OpenKBP challenge) have provided code or network details for us to visualize their results. However, the rest of 3 methods did not release code and it is hard to achieve the performance reported in their papers. Hence, we only report their quantitative results without doing qualitative comparisons. 2) All the results were yielded from the same training/testing data of OpenKBP challenge using same evaluation metrics and thus can be compared directly and fairly.
Value-based DVH loss vs. Average error (AC & R1) 1) The sorting in Value-based DVH loss ranks the voxels with similar dose values, which is equivalent to the binning operation mentioned by R1. Hence, our DVH loss is a variant of average error with extra binning operations. 2) Compared with using average error loss, Value-based DVH loss brought a significant performance gain (DVH score from 1.567 to 1.257, p-value<0.05). 3) Value-based DVH loss simplifies the DVH calculation and accelerates training speed by 3 times compared with vanilla DVH loss [10] mentioned by R1.
Dose prediction in the clinical workflow (AC & R1) Treatment planning is the most labor-intensive and time-consuming step in radiotherapy, involving massive hyper-parameter tuning via a trial–and–error manner and severe inter-observer variations. 1) Our method can provide fast and reproducible dose prediction on unseen cases based on the knowledge learned from abundant real plans. The predicted dose has been demonstrated very close to the physically deliverable one and thus can be used as a good starting point in treatment planning, substantially reducing the time and inter-observer variations in clinical workflow [1]. 2) Also, as treatment is delivered through beam-shaped radiations, the accurate prediction on beam paths can provide physicians with intuitive guidance about the final treatment plan. 3) We agree that our method offers no guarantee to generate physically deliverable dose. But our result is more interpretable and closer to the final deliverable dose than other methods as shown in the paper, and also provides good initialization for subsequent dose optimization.
Experts evaluation (AC & R4) All the visual results have been approved by the dosimetrists and have guidance to the later clinical steps.
Technical details (AC & R4) Both Global Dose Net and Beam-wise Dose Net have a U-Net structure. We will refine more technical details and release our code link in the final paper for better reproducibility.
Performance gain along beam paths (R1) As suggested, we had reported the prediction accuracy along individual beam paths in Tab.1 as a new column. Our method outperforms other methods by statistically significant margins (average MSE on all beam paths decreased by 6.6%, 13.5%, 25.6%, and 28.9% compared with [6], [9], [8], and [3], respectively, indicating effectiveness of our beam decomposition design for dose prediction).
Machine-specific dose distribution (R2) Currently, due to lack of machine-specific info, we mainly focus on the prototype study aiming to predict dose along decomposed beam doses. We agree that machine-specific dose distribution is one important constraint in the final treatment plan and our framework is generalizable enough to deal with the case when we have machine-specific parameters.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
After rebuttal, reviewers revised there scores up because overall the authors did a very good job of clarifying ambiguities and addressing concerns. There are still some concerns around the strength of evaluation, however I think in general this paper is at the level of acceptance in MICCAI.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
7
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The main contribution of this work is a new deep learning-based framework for radiotherapy dose prediction that is trustable and interpretable for treatment planning.
Key strengths:
- Technically sound approach using a “disassembling-assembling” idea that includes the beam directions
- Strong comparison study including relevant baseline methods and ablation experiments using a public dataset
Key weaknesses:
- It is unclear how this method fits in the workflow.
- No statistical analysis included in the original paper (but added in the rebuttal).
- Marginal improvement.
Review comments & Scores: MR1 and R1 raised some concerns regarding the integration of the method in the workflow. MR1 also suggested to clarify the technical implementation and justify the evaluation of the method over SOTA methods.
Rebuttal: A summary of the statistical analysis is provided in the rebuttal, which demonstrates the improvement. I agree with the authors that treatment planning is labour-intensive and time consuming, however, the authors have failed to explain how the proposed method integrates in the current clinical workflow and have limited the discussion on the motivation of the method.
Evaluation & Justification: Despite the lack of discussion on the integration of the clinical workflow, I believe the authors have responded to the MR and reviewers concerns satisfactorily.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
7
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have put considerable effort into the rebuttal and addressed the concerns from reviewers and MR. I would say the strength outweigh the limitations. Therefore, I recommend acceptance of the paper.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
8