Authors

Leo Fillioux, Joseph Boyd, Maria Vakalopoulou, Paul-Henry Cournède, Stergios Christodoulidis

Abstract

Multiple instance learning is an ideal mode of analysis for histopathology data, where vast whole slide images are typically annotated with a single global label. In such cases, a whole slide image is modeled as a collection of tissue patches to be aggregated and classified. Common models for performing this classification include recurrent neural networks and transformers. Although powerful compression algorithms, such as deep pre-trained neural networks, are used to reduce the dimensionality of each patch, the sequences arising from whole slide images remain excessively long, routinely containing tens of thousands of patches. Structured state space models are an emerging alternative for sequence modelling, specifically designed for the efficient modelling of long sequences. These models invoke an optimal projection of an input sequence into memory units that compress the entire sequence. In this paper, we propose the use of state space models as a multiple instance learner to a variety of problems in digital pathology. Across experiments in metastasis detection, cancer subtyping, mutation classification, and multitask learning, we demonstrate the competitiveness of this new class of models with existing state of the art approaches. Our code is available at [this URL].

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43907-0_57

SharedIt: https://rdcu.be/dnwdE

Link to the code repository

https://github.com/MICS-Lab/s4_digital_pathology

Link to the dataset(s)

N/A

Reviews

Review #3

Please describe the contribution of the paper

The authors report on the development of a network for histopathology data classification. GIven the large size whole side images (WSIs), one typically models the entire WSI with a collection of small patches. Therefore, this work categorizes WSI classification as a multiple-instance learning task. Beside, to efficiently model all patches, the authors propose to use structured state space models.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The method of using structured state models is novel, well justified, and presented in detail.
- The experiments are well done and the results are valuable.
- The authors offer their code to the scientific community.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

I do not see a major weakness. Maybe, more comprehensive evaluation of the proposed method would be valuable.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

ok
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

Please add more explanation of why sequence modeling using structured state models is of high relevance to multiple-instance learning. Besides, it would be beneficial to test the model on other metrics such as precision, recall, F1 values in the supplement.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This paper has a clear motive and structure. It creatively maps the disease classification on WSIs to multiple-instance learning. In addition, I found it’s novel and appealing to model the large number of patches from a WSI using structured space state models.
Reviewer confidence

Somewhat confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #1

Please describe the contribution of the paper

This paper proposes the use of state space models as a multiple instance learner to a variety of problems in digital pathology. The paper demonstrates the competitiveness of this new class of models with existing state-of-the-art approaches across experiments in metastasis detection, cancer subtyping, mutation classification, and multitask learning.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The use of structured state space models as a multiple instance learner is a novel approach to sequence modeling in digital pathology.
2. The experiments are comprehensive, including comparison with existing MIL baselines, analysis of model and time complexity, ablation study, effectiveness on long sequence modeling, etc.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The results shown in Table 1 indicates that the proposed method is less convincing to beat the sota methods.
2. The key contribution of this paper is the using of Structured State Space Models. However, in introduction, why Structured State Space Models are used and why they can benefit the WSI classification are not well-explained. Also in Method, the structure of Structured State Space Models is not detailed. Especially for Fig. 1, I cannot obtain much information of Structured State Space Models.
3. It seems that according to Fig.2, the proposed method achieved less desired visualization than TransMIL.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors state that their code is available at a specific URL, which suggests that their work is reproducible to some extent.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
1. This paper novelly proposes to use Structured State Space Models for modeling the long sequence problem of MIL-based WSI classification. It would be more understandable if the motivation, advantage, method details of Structured State Space Models are specified. In-depth analysis would be desirable.
2. Current results do not significantly show the advantages of using Structured State Space Models over existing SoTA methods.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

4
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Given the writing and performance, I would rate the paper accordingly.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

This paper proposed a novel MIL method for WSI classification, especially the use of the Structured state space model to alleviate the burden of long sequence modeling in WSI analysis. The experimental results on three publicly available datasets proved that the state space model is an alternative solution for the aggregation part of MIL in WSI analysis.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. This paper is well written and easy to follow.
2. The first time to use state space models for WSI analysis, which can address the limitation of current MIL method for long sequence modeling.
3. It is interesting to see an alternative MIL solution for WSI classification, which is totally different from the current attention-based or other MIL methods.
4. Extensive experiments are conducted, and the efficiency of proposed method is proved.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The introduction of existing methods for WSI analysis is limited. First, there are other learning paradigms for WSI analysis, i.e., fully-supervised, semi-supervised, and the “Neural Image Compression”. And also, some other MIL-based methods. [1] Coudray N, Ocampo P S, Sakellaropoulos T, et al. Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning[J]. Nature medicine, 2018, 24(10): 1559-1567. [2] Gao Z, Hong B, Li Y, et al. A semi-supervised multi-task learning framework for cancer classification with weak annotation in whole-slide images[J]. Medical Image Analysis, 2023, 83: 102652. [3] Tellez D, Litjens G, van der Laak J, et al. Neural image compression for gigapixel histopathology image analysis[J]. IEEE transactions on pattern analysis and machine intelligence, 2019, 43(2): 567-578. [4] Chen R J, Chen C, Li Y, et al. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 16144-16155.
2. What is the difference between structured state space model and state space model? More explanation should be added.
3. More details about the proposed method should be added to Section 3. For instance, how to map the patch embeddings “{x1, …, XL}” to the corresponding softmax probability “y” with equation (1)?
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper’s code is publicly available on Github, and the datasets used in this study are already publicly accessible, indicating a high level of reproducibility for this paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
1. More details should be added to Fig 1. From the current figure, it is hard to understand the role of eq1. in MIL aggregation.
2. The definitions of “M” and “c” in Section III are missed.
3. The heatmap visualization in Fig 2 is meaningless, the results of the proposed method show no superior to the TransMIL.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

An alternative and novel MIL solution for WSI classification. Extensive Experiments are conducted to evaluate the effectiveness of proposed method.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposes to use the state space models as a multiple instance learner for whole slide pathology image analysis. The experimental results demonstrate competitiveness over the existing multiple instance learning methods. Overall, the idea is interesting and two of three reviewers recommended accept. Thus, a decision of Provisional Accept is recommended. However, the concerns, including more explanation and in-depth analysis of sequence modelling using structured state models, justification of performance advantage over SOTA methods, etc., should be fully addressed in the final version.

Author Feedback

We greatly appreciate the feedback received by all reviewers as well as the meta-reviewer. The reviewers have noted the quality of our experiments, as well as the novelty of the approach, but we would like to address some of the comments regarding the weaknesses of our work.

R1 rightfully mentioned that our method does not beat SoTA on all benchmark tasks. However, as shown in Tables 1 and 2, our method is quite competitive compared to the current best methods, both in terms of metrics and inference time. Most importantly, as shown in Table 5, our method clearly outperforms other methods when applied on long sequences, which is a major point for whole slide images.

R1 requested more detail on the method and its motivation while R2 requested more details on the proposed method and clarifications between the variants of SSMs, and both reviewers criticised Figure 1. We therefore updated the methods section with a subsection explaining SSM models in more detail. The content of the original Methods section, including Figure 1, has also been updated to cohere better with this and ensure consistent notation. The Related Work section has been updated to clarify the differences between SSM variants.

R1 and R2 criticised the relevance of Figure 2. This has been moved to the supplementary section. We would like to stress that this visualisation was originally included to help conceptualise multitask learning, rather than to serve as a qualitative comparison. By contrast, Table 4 shows that the proposed method quantitatively improves on TransMIL.

R2 identified missing literature, the provided references have been added to the Related Work section, which has been slightly modified for readability. The TransMIL and CLAM references have been transferred from the introduction to join these new references.

R2 identified mistakes in the Methods section formulation, which have now been corrected.

All reviewers commented on the availability of the code and datasets for the purposes of reproducibility. Respecting the principle of reproducibility we have included links to our codebase in the revised manuscript.

back to top

Structured State Space Models for Multiple Instance Learning in Digital Pathology