List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Xiaofei Wang, Stephen Price, Chao Li
Abstract
Most recently, the histopathology diagnosis of cancer is shifting to integrating genomic makers with histology. It is a urgent need for digital pathology methods to effectively integrate genomics with histology, which could lead to more accurate diagnosis in the real world scenarios. This paper is a first attempt to integrate genomics with histoimics and model their interactions for classifying diffuse glioma bases on whole slide images. Specifically, we propose a hierarchical multi-task multi-instance learning framework based on histopathological data to jointly predict histology and genomics. Moreover, we propose a co-occurrence probability-based label correction graph network to the co-occurrence of genomic markers. Lastly, we design an inter-omic interaction strategy with the dynamical confidence constraint loss to model the interactions of histomics and genomics. Our experiments show that our method outperforms other state-of-the-art methods in classifying diffuse glioma, as well as related genomics and histology on a multi-institutional dataset.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43990-2_52
SharedIt: https://rdcu.be/dnwL7
Link to the code repository
N/A
Link to the dataset(s)
https://portal.gdc.cancer.gov/
Reviews
Review #3
- Please describe the contribution of the paper
The authors develop a framework to predict the molecular sub-types and/or molecular markers of gliomas using digitized, Whole Slide Imaging data as input. To achieve the predictions, the authors use a MulitTask and Multi Instance Learning framework to provide multi-label classification. The novelty of the work lays in the ability to estimate molecular subtyping either ahead of or (eventually) in-lieu of sequencing and/or immunoStaining which are more expansive than histology and not as accessible in all regions.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
*Important problem and well articulated. *HMT-MIL using only digitized histopath input data - not including molecular biomarkers as in other work. *Addressing the co-occurance of molecular markers in the framework and its output *Classification of diffuse gliomas considering the interaction of histology and molecular markers. *Ability of framework to extract information from many patches
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
I believe the main weakness of the work is the structure of the Methodology section and the matching to Figures 1 and 2. I think a more interpretable structure would to setup the subsections according to the 4 functional blocks of the framework (stem, histology prediction, etc). There is some confusion in Figure 2 regarding what outputs flow from one subsystem to another (eg: the cross-omics interaction module - is the confidence weight vector output from the genomics predictor for all subtypes?)
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
I rate the reproducibility of this work high. It is the highest of the 5 papers I’ve reviewed. The supplemental material (including code) is complete. Some of the weaknesses in the presentation of the methods could hinder some attempts at reproducing.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
As mentioned, consider reordering or structuring Section 3. it is difficult to match the logical flow and elements with Figures 1 and 2.
A discrete listing of outputs would be beneficial for interpretation ; the training labels don’t readily match to the DeepMO-Glioma outputs enumerated in the first paragaph of Section 3.
Section 3.3 is confusing and rewording will help understanding. The 2nd paragraph, in particular, is challenging. I’m not clear how the DCC loss doesn’t penalize histology and molecular markers (eq 3).
It is unclear how the Graph is structured and is ‘working’ within the training paradigm.
4.2 - you should mention if those other methods and work are all factoring into their decision making the molecular marker data (subtypes etc). Difficult to make an objective comparison.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This is a comprehensive piece of work and has very interesting clinical implications and utility.
- Reviewer confidence
Somewhat confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #5
- Please describe the contribution of the paper
This paper presents a method to jointly predict molecular markers and histologic features of diffuse gliomas from whole slide images. The classification model included several modules to account for the interaction between the molecular and histologic features and achieved state-of-the-art performance when compared to other existing methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Based on the consideration of the interaction between the molecular and histological features, the paper included several elaborated modules, such as CPLC graph network, LC loss, and DCC strategy.
- The rationale for each module was well written and sounded reasonable.
- The contribution of the modules to the classification performance was confirmed by the ablation studies.
- The final model achieved state-of-the-art classification performance for diffuse glioma compared to other existing methods.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The ratio of the training labels needs to be described.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The reproducibility of the paper seems to be guaranteed due to the availability of the source codes and datasets.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
It will be interesting to see if the proposed method is effective for diseases other than gliomas, since updating the diagnostic system from histologic features to molecular markers is also true for other diseases.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I think the paper is good for acceptance because of its technically sound methodology with extended ablation studies, state-of-the-art performance with decent comparison, and guaranteed reproducibility.
- Reviewer confidence
Somewhat confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #4
- Please describe the contribution of the paper
This paper proposes a novel approach to integrating molecular markers with histology features for cancer diagnosis. Specifically, the authors explore a hierarchical multi-task multi-instance learning framework to jointly predict histology and molecular markers, as well as a co-occurrence probability-based label correction graph network to model the co-occurrence of molecular markers. The proposed model achieves superior performance over other state-of-the-art methods and serves as a potentially useful tool for digital pathology based on whole slide images in the era of molecular pathology.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This article proposes a multi-task, multi-instance learning framework that utilizes HE-stained WSI images to simultaneously predict four molecular markers and the clinical outcome of Diffuse Glioma. The authors promote the prediction of Diffuse Glioma by leveraging the correlation between the prediction of molecular markers and their labels, which is an interesting idea. Additionally, the writing in the article is clear and complete code is provided as an attachment.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
However, I have several concerns as follows:
- The experimental setup may have issues. The authors should clarify the following points through the experiment:
1) Compared to the model without the addition of 4 molecular markers, the model with the addition of these markers should significantly improve the predictive performance of Diffuse Glioma. However, I did not find clear results in Table 1, and I am not sure what ResNet, DenseNet, and VGG-16* represent. Why did the authors compare these different backbone models, and why are they not all in the MIL setting? The authors should provide a clear response to these questions in their rebuttal.
2) The proposed method (multitask architecture) should outperform the current state-of-the-art (SOTA) method (directly predicting Diffuse Glioma) in terms of predictive performance. I found the comparison results in the left column of Table 1. Are the results for CLAM and TransMIL based on direct predictions of Diffuse Glioma, or do they also incorporate the multitask architecture?
3) Based on points 1) and 2), the proposed method (multitask architecture) should outperform the method of directly predicting Diffuse Glioma since it uses multiple molecular markers as additional label information. However, this raises new questions, as follows:
3.1) Does the proposed method fall under the category of multimodal MIL techniques with other data and pathological WSI? Many studies [1,2] have proposed joint frameworks for multimodal pathological MIL, including clinical information + pathological WSI, molecular subtyping information + pathological WSI, etc. Since this study introduces additional molecular prediction information on the basis of pathological images, can the proposed paradigm also be understood as a multimodal pathological MIL paradigm? If so, the current comparison method may seem unfair, and the authors should compare the proposed method with MIL methods that combine multimodal data.
[1] Chen R J, Lu M Y, Weng W H, et al. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. ICCV. 2021: 4015-4025.
[2] Li H, Yang F, Xing X, et al. Multi-modal multi-instance learning using weakly correlated histopathological images and tabular clinical information. MICCAI.2021: 529-539.
3.2) Can the label information of these molecular markers be used to directly predict Diffuse Glioma? How’s the performance? This should be experimentally demonstrated.
3.3) Is it practical to obtain both WSI and molecular marker information in clinical applications? One concern is whether patients who undergo molecular marker testing also undergo testing for other indicators of Diffuse Glioma. If so, this multitask model may not be very convenient for future applications.
- AUC should be added to the experimental results in Table 1 because other indicators may be affected by threshold selection.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The author submitted the code as a separate attachment, ensuring the good reproducibility of the article.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Additional weakness.
- The presentation of multiple results in Fig.3 is not clear, and there are two issues:
1) This figure does not clearly demonstrate the superiority of the proposed method compared to other state-of-the-art (SOTA) methods on the prediction of the four markers.
2) The comparison of the ablation experiments is even less clear. The colors are too messy, making it difficult to distinguish between them.
Please refer to all the weaknesses.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
In general, I am very interested in this article, but I have many questions and concerns that have not been answered. In the first round of review, I would like to give a score of weak accept, but I would like the authors to carefully address all of my concerns. After the rebuttal, I will carefully revise my score.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
In this paper, authors introduce a pioneering approach to classify diffuse glioma based on whole slide images by jointly predicting molecular markers and histology features, and modeling their interactions. The proposed method is a hierarchical multi-task multi-instance learning framework, which effectively combines both histology and molecular markers in a joint prediction task.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
(1) The paper is overall very well organized and presented. (2) Both the technical novelty and clinical meaning of this study are sufficient for the conference (3) This is the only submission that has code attached, which is very helpful to reproducibility.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
I don’t notice significant weakness of this paper
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Very good.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
This paper is well organized and presented. Particularly, the proposed method and evaluation are detailed in a very decent way.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Technical novelty, organization of paper and presentation of results.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposes a novel approach to integrating molecular markers with histology features for cancer diagnosis.
Key strengths:
- Good reproducibility with attached codes and data
- Novel idea of integrating histology and molecular markers
- Extensive validation experiments
Key weaknesses:
- A few unclear points in method description.
- A few unclear points in experimental presentation.
Author Feedback
We thank the reviewers and the AC for their valuable feedback. [R1] We would thank the reviewer for acknowledging our contribution.
[R2] Thanks for the insightful comments. We will follow the reviewer’s suggestions to reorganise the method and experiments sections in our final version. We will describe different subnets following the processing chain. Besides, we will describe Section 3.3 and eq3 more clearly, as well as add a discrete list of outputs for better interpretation. In addition, Fig. 2 is the pipeline of CPLC-Graph network, used for intra-omic modelling of the molecular biomarkers. Following the Reviewer’s suggestions, we will add a more detailed description of the Graph structure and its workflow.
[R4]
Ablating molecular markers. As described in Section 2, the ground-truth label for glioma classification is generated based on integrating molecular markers (IDH, 1p/19q and CDKN) and histology features (NMP), according to the latest WHO criteria. Therefore, the status information of the molecular markers is intrinsically embedded in the taxonomy of glioma. Hence, it is not possible to completely ablate the molecular information from the classification task.
Fairness in experiments. All the compared methods, i.e, CLAM, TransMIL and ResNet, are trained directly by the final labels (glioma classification), which are defined by integrating molecular markers with histology, as explained above. Therefore, all the tested models are provided with the molecular and histology information, demonstrating the fairness of our comparison. The superiority of our method over other SOTA methods may be due to our model design. Specifically, instead of directly predicting glioma labels from the WSI, our method disentangles the prediction task into two joint learning tasks (molecular markers and histology prediction) and finally classifies glioma based on the outputs of these two tasks.
Baselines. We utilise three commonly-used image classification methods, i.e, ResNet, DenseNet, and VGG-16, as the baselines. Since these methods are not originally designed for MIL, we slightly modified them for the MIL setting. Specifically, for each input WSI, we converted it to a bag of image patches and then tailored the baseline models to take the input.
Multimodal MIL. Our method is distinct from the traditional multimodal learning methods, e.g., [1,2] as listed by the reviewer. The major difference is that, we treat the molecular markers as the output, instead of the input. However, in the above traditional multimodal learning methods, molecular markers are taken as input, which are not predicted from the WSI. Therefore, in practice, these multimodal learning methods rely on the molecular markers, while our model does not need the additional input of molecular information.
Evaluation metrics. Following the reviewer’s suggestion, we will use more metrics to evaluate the models.
Fig.3. Following the reviewer’s advice, we will use a clearer colour setting and also add a zoom-in window of the upper left of the ROCs to better show the results.
[R5]
- Ratio of training labels. The ratio of the training set is given in Section 2 of the original paper.
- Use in other diseases. Following the reviewer’s inspiring advice, our future work will explore other diseases to generalise our model.