List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Ruby Wood, Enric Domingo, Korsuk Sirinukunwattana, Maxime W. Lafarge, Viktor H. Koelzer, Timothy S. Maughan, Jens Rittscher
Abstract
Existing methods for interpretability of model predictions are largely based on technical insights and are not linked to clinical context. We use the question of predicting response to radiotherapy in colorectal cancer patients as an exemplar for developing prediction models that do provide such contextual information and therefore can effectively support clinical decision making. There is a growing body of evidence that about 30% of colorectal cancer patients do not respond to radiotherapy and will need alternative treatment. The consensus molecular subtypes for colorectal cancer (CMS) provide one such approach to categorising patients based on their disease biology. Here we select the CMS4 subtype as a proxy for stromal infiltration. By jointly predicting a patient’s response to radiotherapy, the presence of CMS4, and the epithelial tissue map from morphological features extracted from standard H&E slides we provide a comprehensive clinically relevant assessment of a biopsy. A graph neural network is trained to achieve this joint prediction task, which subsequently provides novel interpretability maps to aid clinicians in their cancer treatment decision making process. Our model is trained and validated on two private rectal cancer datasets.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43904-9_73
SharedIt: https://rdcu.be/dnwIl
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #2
- Please describe the contribution of the paper
The paper presents a novel method for predicting a patient’s response to radiotherapy in colorectal cancer by jointly predicting a complete response to radiotherapy, CMS4 classification, and epithelial tissue. The model uses a graph neural network to provide a clinically relevant assessment of a biopsy. The method addresses the lack of interpretability of existing prediction methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors use a graph neural network to achieve a joint prediction task, which is a novel approach that has not been widely used in this context. Furthermore, the interpretability maps generated by the model can help clinicians understand why the model made a particular prediction and can also identify features that are important in determining a patient’s response.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
a. This method appears to be a combination of existing methods (self-supervised learning and graph neural network) and lacks significant innovation in terms of methodology. b. All the details of the method are described using text, and using formulas would make it more professional and easier to understand. c. This paper lacks any comparison with other methods, and it is not clear what advantages this method provides in comparison to others. d. This paper has no ablation studies on the modules of the method and does not explain the reasons and advantages for selecting each module. Furthermore, 1) replacing the GNN with another CNN and then calculating the classification activation map could also show the response. 2) It is unclear why the authors used superpixels instead of grid features to be the node of the graph. e. The author mentions in the paper that their approach is semi-supervised, but they do not explicitly state this issue. The author should specify their problem settings in the methods section.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
the reproducibility of the paper is fine.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
please refer to the strength and weakness.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
the novelty of the method, and the experiments.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper focuses on developing prediction models that provide contextual information for clinical decision making in the context of colorectal cancer biopsies. The authors use the consensus molecular subtypes (CMS) as proxy data and propose a graph neural network that jointly predicts a patient’s response to radiotherapy, the presence of CMS4, and the epithelial tissue map. The model is trained and validated on two private rectal cancer datasets, and the resulting interpretability maps are expected to aid clinicians in making informed treatment decisions.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
(1) The use of DINO for extracting contextual information and the SLIC superpixel algorithm for building a superpixel-based graph neural network is a novel pipeline that provides valuable insights to help doctors in clinical decision making. (2) The generation of novel node activation maps from the three prediction branches, with nodes colored by their predictions, has real-world impact for clinical workflows and evaluations, enhancing the understanding of the model’s predictions in a clinical context.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
(1) The paper should provide justification for the different magnification levels used in the data, such as ViT extracting features at 20x magnification, SLIC superpixel at 5x, and label generation using U-Net at 10x. This would help to understand the rationale behind these choices and their implications on data volume and feature levels. (2) The paper should explain any special considerations or reasoning behind setting the SLIC algorithm’s compactness to 20, which results in superpixels that are more regularly shaped, https://www.mathworks.com/help/images/ref/superpixels.html. Additionally, it should address how the hyperparameters were chosen, including whether they were based on the validation dataset experimentation or other methods. (3) The paper should provide detailed information on how to calculate the Node activation maps in Figure 2, as this is a key highlight of the work and would be useful for doctors to analyze the classification results.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Good
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The paper mentions the use of different magnification levels for feature extraction, such as ViT at 20x, SLIC superpixel at 5x, and label generation using U-Net at 10x. This could potentially lead to variations in data volume and scales of features, which may affect the interpretation of results. It would be helpful to provide a justification for using different magnification levels and discuss the potential impact on the findings. Alternatively, considering the manual cut-off created by different magnification levels, is it possible to use the same magnification levels and automatically organize them? This could help eliminate potential biases and provide more robust results.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The utilization of a graph neural network based on superpixels and ViT feature extractor in this paper is novel and promising for clinical decision making in colorectal cancer patients. The validation of the proposed approach through node activation maps provides valuable insights for real-world clinical trials.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #4
- Please describe the contribution of the paper
This study proposes to jointly predict radiotherapy response, CMS4 tumoral gene expression which is indicative of recurrence risk, and whether images belong to epithelial tissue from H&E Whole Slide Image (WSI) using a graph neural network (GNN) on a private dataset (n=249) of colorectal cancer. A vision transformer (ViT) with knowledge distillation is used to extract patch level features, and the superpixel algorithm is used to split the 20x magnification image into patches/nodes to the GNN. The model achieves 82%, 82% and 76% mean ROC AUC respectively in the 4-fold cross-validation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This study proposes a novel way of predicting radiotherapy (RT) response using a graph neural network (GNN) to model semantically connected patches of a Whole Slide Image (WSI) of colorectal cancer biopsy.
The proposed framework also incorporates vision transformer (ViT) and self-distillation (DINO framework) to extract image features as inputs to the GNN.
The authors propose to jointly predict RT response, tumor genetic biomarker and epithelial tissue class trading performance for better interpretability.
An ablation study was performed to obtain the final hyperparameters to ensure robustness of the chosen models.
Activation maps are presented which help interpretability and allows clinicians to better visualize affected tissues.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Lack of a comparison with existing methods, such as the mentioned SlideGraph. No baseline model was presented to predict the targeted outcomes to be able to judge whether the performance of the proposed model is significant.
No discussion of the performance of the other components of the proposed framework, such as the self-distillation ViT or the superpixel segmentation/clustering algorithm.
No presentation of the ablation study results, with statistical testing to show that the proposed configuration truly is the most performant.
The study mentions that the data imbalance (23% slides with complete RT response, 12% with CMS4) yet no discussion is given of the results with regards to this topic.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The use of two private datasets makes this study non-reproducible.
The authors promise all code available upon request.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Including baseline and comparative models, as well as the performance of the ViT/superpixel algorithm and ablation study would strengthen the validity of the study, which has otherwise good results.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Novel methodology and adequate sample size with strong interpretability, but poor reproducibility and external validation.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Proposes a method for predicting a patient’s response to radiotherapy in colorectal cancer by jointly predicting a complete response to radiotherapy, CMS4 classification, and epithelial tissue.
- On the face of it, use of graph neural network to achieve a joint prediction task is could be considered novel, but overall framework appears to be an amalgamation of existing techniques without much consideration for why they should be put together
- Differences in magnification between modules is not well justified’
- Parameter choices for SLIC and hyperparameters need better explanation
- Cross validation results presented, no statistical analysis despite relatively large cohort overall
- Results are generally poorly presented (text only), no comparison to SOTA provided
- Highly imbalanced data, making it hard to rely on AUC as a metric of performance.
The biggest shortcoming here is that the method is being used to assess complete response after treatment (as defined by a pathologist evaluation?) while using POST-TREATMENT pathology. It is unclear what the point of the exercise is - simply identify the amount of tumor remaining, and get an assessment? The so-called insights such as “the overlap between CMS4 and specific pathologic phenotypes” does not add anything here since it is all on post-treatment path - so all of these could simply be the result of treatment.
Author Feedback
Today there are no established predictive biomarkers for radiotherapy response in rectal cancer patients. We address this unmet clinical need by extending the state of the art to provide a novel interpretability method. All three reviewers provide supportive and positive statements: R2 - novel approach by joint prediction, interpretability maps can help clinicians understand predictions and identify important features; R3 - novel pipeline, provides valuable insights to help doctors, novel node activation maps have real-world impact, excellent clarity and organisation, promising for clinical decision making; R4 - novel methodology, strong interpretability, activation maps allow better visualisation.
We provide a critical clarification and response to comments:
MR: Method is using post-treatment pathology Clarification: This apparent “biggest shortcoming” is a misunderstanding. The only images used in the development of this paper and seen by our model are pre-treatment biopsy slides, not post-treatment. For this work, we have access to two well-characterised rectal cancer cohorts, consisting of pre-treatment biopsies and matched information on pathological response to radiotherapy treatment (tumour regression) for each single patient under study. This work is uniquely positioned to identify image features predictive of treatment response before radiotherapy is applied and addresses an important clinical problem with a new technological approach.
Responses:
R2, MR: Lacks innovation We address an unmet need by combining three prediction branches for three carefully considered elements of CRC, which will help clinicians to interpret our predictions in context of their existing knowledge. Supporting review comments: R2 - approach using a GNN for joint prediction tasks is novel in this context; R3 - network design and resulting node activation maps are novel.
R2, R4: No ablation studies on modules Our primary contribution is the three prediction branches and their corresponding node activation maps. Fitting other modules into this framework is not trivial and would take considerable effort, whilst not impacting the main focus of this work.
R2, R3, MR: Use of superpixels, choice of SLIC compactness and other hyperparameters (HPs) Superpixels give a more natural segmentation of the tissue instead of the arbitrary square tile which has no biological meaning, and this is useful in the node activation visualisations since the nodes correspond to more meaningful tissue sections than tiles would provide. A CNN would not work as well with superpixels as a GNN does. SLIC parameters chosen to provide meaningful segmentations confirmed by clinical experts. Graph HPs chosen from visualisation and model HPs determined by ablation studies on validation set.
R3, MR: Justify magnification levels 5X to calculate superpixels due to memory constraints; 20X to train feature extractor to capture fine-grained information in image; 10X for epithelial segmentations chosen in previous work.
R2, R4: Lack of comparison with other methods No similar interpretability-focused methods exist for CRC.
MR: No comparison to SOTA Main contribution is our node activation maps, not the performance metrics. There is no SOTA metric for this deep learning RT prediction task apart from that generated by the first author in previous work.
R4, MR: No discussion of data imbalance in results/hard to rely on AUC We provided six different metrics including both balanced and unbalanced metrics.
R4: Poor reproducibility No public CRC histology datasets exist which have the patient’s recorded response to RT, hence impossible to perform this task on public data.
Minor: R3 - node activation calculation clear in code R2 - lack of formulas due to lack of space R2 - not explicitly explained semi-supervised but clear from loss function R4, MR - lack of statistical tests due to lack of space MR - results are not text only; table of metrics provided
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Proposes a method for predicting a patient’s response to radiotherapy in colorectal cancer by jointly predicting a complete response to radiotherapy, CMS4 classification, and epithelial tissue. Major comment regarding use of “post-treatment” has now been clarified in the rebuttal, suggesting a more predictive model than originally considered. Based on this, I re-reviewed all the material provided but still have major concerns. Concerns regarding innovation aren’t really discussed - what shortcoming from previous work is being addressed through the approach being designed here? SOTA/other methods have been cited in the introduction, but the proposed approach has worse performance reported. SLIC and magnification parameters could significantly impact the results, and is not really discussed in the rebuttal or the paper. Issue regarding imbalanced dataset is not acknowledged, despite only 23% complete response rate and only 12% CMS rate - there is a mention of “weighted metrics” but not clear what this means. The biological insights about overlap between CMS4 and response are more interesting with the aforesaid clarification, and suggest some level of interpretability. However, unclear how significant this trend is over the cohort (only 2 slides presented). The ablation study is not discussed in the rebuttal, supplementary, and has only cursory text in the main paper. As such, the method may have promise, but I’m unconvinced about its performance or the study design
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
On the one hand, I find the paper is very interesting and believe it to be novel (predicting a patient’s response to radiotherapy in colorectal cancer) on the other I am not sure the authors adequately responded to the reviewers concerns (e.g. about ablation studies)…. thus I’m on the fence but given the novelty of the paper lean towards acceptance.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This work presents a new method to jointly predict treatment response, CMS4 subtype, and epithelial tissue map from colorectal cancer biopsy H&E images. The pipeline uses for the first time self-supervised VIT DINO approach to extract morphological features, superpixel approach to construct graph, and GNNs for final prediction. The resultant node activation maps are used for interpretability. This is an interesting paper with merit for publication, especially following the rebuttal.