Authors

Jiansheng Fang, Anwei Li, Pu-Yun OuYang, Jiajian Li, Jingwen Wang, Hongbo Liu, Fang-Yun Xie, Jiang Liu

Abstract

Radiation encephalopathy (REP) is the most common complication for nasopharyngeal carcinoma (NPC) radiotherapy. It is highly desirable to assist clinicians in optimizing the NPC radiotherapy regimen to reduce radiotherapy-induced temporal lobe injury (RTLI) according to the probability of REP onset. To the best of our knowledge, it is the first exploration of predicting radiotherapy-induced REP by jointly exploiting image and non-image data in NPC radiotherapy regimen. We cast REP prediction as a survival analysis task and evaluate the predictive accuracy in terms of the concordance index (CI). We design a deep multimodal survival network (MSN) with two feature extractors to learn discriminative features from multimodal data. One feature extractor imposes feature selection on non-image data, and the other learns visual features from images. Because the priorly balanced CI (BCI) loss function directly maximizing the CI is sensitive to uneven sampling per batch. Hence, we propose a novel weighted CI (WCI) loss function to leverage all REP samples effectively by assigning their different weights with a dual average operation. We further introduce a temperature hyper-parameter for our WCI to sharpen the risk difference of sample pairs to help model convergence. We extensively evaluate our WCI on a private dataset to demonstrate its favourability against its counterparts. The experimental results also show multimodal data of NPC radiotherapy can bring more gains for REP risk prediction.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16449-1_19

SharedIt: https://rdcu.be/cVRUZ

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #4

Please describe the contribution of the paper

(1)To assess the rationality of nasopharyngeal carcinoma radiotherapy regimen, the author proposed a deep multimodal survival network (MSN) equipped with two feature extractors to learn discriminative features from image and non-image data; (2)A new WCI loss function is proposed for MSN training, which has temperature hyperparameters that can effectively utilize the REP samples of each batch to help the model converge.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

For the first time, both image and non-image data were used in an NPC radiotherapy regimen to predict radiotherapy-induced REP. A deep multimodal survival network (MSN) with two feature extractors is designed to learn to identify features from multimodal data. One feature extractor performs feature selection on non-image data, and the other learns visual features from images. A new weighted CI (WCI) loss function is proposed, which allocates different weights to all representative samples through a double averaging operation, thus effectively utilizing all representative samples. A temperature hyperparameter is introduced to sharpen the risk differences in sample pairs and help the model converge.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

(1) The author conducted ablation experiments on their own models on the two modal data and did not use other current models for comparison; (2) Regarding the processing of image data, I hope that the author will introduce more details, such as: how does the author obtain the ROI region, manual or automatic detection? What is the ROI area size? Is the number of CT, RS, and RD pictures the same per patient? Three kinds of picture information can be directly controlled by the operator? (3) The author’s model is evaluated on a private dataset, is the public dataset used for experiments?
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The author used the proposed method on the private data set, which is not conducive to the reproduction of the method.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

(1) It is recommended to supplement and compare with other current models; (2) Regarding the processing of image data, I hope that the author will introduce more details; (3) Experimental comparison in public datasets using the model proposed by the authors.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The author had too little comparison in the experimental part, which is not conducive to comparison with the current methods in this field at home and abroad.
Number of papers in your stack

5
What is the ranking of this paper in your review stack?

2
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #5

Please describe the contribution of the paper

This paper proposes a new multi modality model for predicting radiotherapy-induced radiation encephalopathy (REP). The author proposes a multimodal survival network and a new weighted CI loss function to improve the accuracy of the prediction.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The author propose way to use multi-modality data from both images and non-image data is new to this task.
- The proposed WCI loss is helpful to alleviate data imbalance issue and improve the accuracy.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The dataset is highly imbalanced (1:10), a confusion metric or similar might be worth adding to show what kind of mistakes the model is more likely to make.
- I am not sure if these improvement on CI is statistically important. The authors may want to add some analysis on the statistic significance.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The dataset is private, which I assume will not be made public. The authors answered the codes for this study will be released.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- Is the CE in Table 1 weighted or not? What’s performance if simply use weighted cross entropy for the data imbalance?
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper looks solid: proper review of related study and sound experiment result. I am not familiar with the task of radiation encephalopathy assessment and survival model. But based on the presentation of method and result, I vote for weak accept.
Number of papers in your stack

7
What is the ranking of this paper in your review stack?

2
Reviewer confidence

Somewhat Confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #6

Please describe the contribution of the paper

This paper describes a model to predict the risk at 36 months for developing radiation encephalopathy (REP) in patients diagnosed with nasopharyngeal carcinoma (NPC) and treated with radiotherapy. The model was trained using image data (CT and RT) fed to a multimodal survival network and non-image diagnostic data fed to a multilayer perceptron model. The outputs from the two networks were weighted and summed to provide a final risk value. The data used to train, test, and validate the model was used from the time of treatment for a pool of n=4,816 patients. The model was implemented and tested using multiple different loss functions, including their own novel weighted concordance index loss function. The author’s novel WCI loss function resulted in the trained model that had the highest measured CI and AUC values compared to the same model trained with 5 other loss functions.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The strengths of this paper include:
- Large number of subjects available for training and testing.
- Combined use of image and non-image data.
- Use of survival analysis methodology and loss functions to train model.
- Inclusion of ablation to examine empirically derived parameters such as weights for image versus non-image risk outputs to final output and temperature parameter used for WCI loss function.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The weaknesses of this paper include:
- Claim of “best accuracy” despite no statistical testing and/or analysis between ROC AUC or CI values for significant differences.
- The weights may have been made parameters of the model to train (w_nv, w_v).
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors state they will release the trained model if accepted. But the dataset will be private, so reproducing the same results would be impossible. And training a model implemented independently based on the description would also be impossible. Something similar could be done, but not exact.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
This is a very well written and organized paper. Some suggestions to make it even better:
- Replace “synthesizing” in section 3.2 with “using all” as you are not synthesizing data but combining actual data.
- Possibly add more information about how the image data is pre-processed before it is input to the model.
- Add methodology to test for significant differences between the measures for the different loss functions.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The authors presented implemented a model with to predict the future risk of an unwanted injury due to radiotherapy. Having knowledge of this risk prior to treatment would allow clinicians to modify their plans to reduce the risk. The paper was well organized and has many strengths as described above. The weaknesses are minor.
Number of papers in your stack

5
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper describes a model to predict the risk for developing radiation encephalopathy (REP) in patients diagnosed with nasopharyngeal carcinoma (NPC) and treated with radiotherapy. The model uses both image data (CT and RT) and non-image diagnostic data. The model was trained and tested on a large private data set. The author also proposed a novel WCI loss function. The reviewers agreed on the novelty of the method, the use of large dataset and the ablation study. They also raised a few concerns that should be addressed, including 1) no comparison with SOTA method; 2) evaluation on private data set; 3) implementation details; and 4) statistical analysis;.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

5

Author Feedback

We would like to thank all the reviewers for acknowledging the contributions of our work to predict radiotherapy-induced REP in NPC, as well as the insightful comments.

There are two main contributions of this work. (1) We utilize the MSN model to confirm that NPC radiotherapy’s image and non-image data positively relate to REP risk. (2) We propose a novel WCI loss function to help the MSN model optimize and converge by casting REP prediction as a survival ranking problem. Our experiments correspondingly demonstrate the significance of multi-modal data in predicting REP onset and the superiority of our WCI in survival analysis. Based on briefly reviewing the intentions of this work, we will address four general concerns as follows:

Q1: Comparison with SOTA method A1: In order to verify the achievements of our intentions, we only need to conduct comparison and ablation experiments for WCI loss function, not MSN. In future work, we will improve our MSN to boost the performance of REP prediction in NPC and compare it with SOTA multi-modal learning framework.

Q2: Evaluation on private dataset A2: Due to the extreme difficulty in acquiring NPC datasets for REP prediction, there are no publicly available datasets currently. Hence, we only make evaluations on our private dataset. We hope this work can help foster the research in this community. On the other hand, in terms of our proposed WCI loss function, we will validate its effectiveness on public datasets of survival prediction tasks in future work.

Q3: Implementation details A3: RD images show the radiation dose distribution to the human body during radiation therapy. Clinicians sketch the tumor outline in closed curve coordinate form to yield the RS image. Then we apply the mask of RS images to generate the input ROIs of RD and CT images. Regarding the pre-processing of image data, we will supplement those details in Sec2.1.

Q4: Statistical analysis A4: We conduct McNemar tests on CI between our WCI and other loss functions (including CE, Cox, BCI, CCE, and WCI w/o τ). All comparisons in the p-value of less than 0.01 can support that our WCI brings significant improvement to CI. We will add the result of statistical significance in Sec3.2. In addition, in future work, we will validate the effectiveness of our WCI on public datasets of survival prediction tasks and analyze significant differences with other loss functions.

Thank all the reviewers again for giving us valuable suggestions to improve our work. We will revise the paper according to the above responses.

back to top

Weighted Concordance Index Loss-based Multimodal Survival Modeling for Radiation Encephalopathy Assessment in Nasopharyngeal Carcinoma Radiotherapy