Authors

Jun Li, Yushan Zheng, Kun Wu, Jun Shi, Fengying Xie, Zhiguo Jiang

Abstract

Image representation learning has been a key challenge to promote the performance of the histopathological whole slide images analysis. The previous representation learning methods followed the supervised learning paradigm. However, manual annotation for large-scale WSIs is time-consuming and labor-intensive. Hence, the self-supervised contrastive learning has recently attracted intensive attention. In this paper, we proposed a novel contrastive representation learning framework named Lesion-Aware Contrastive Learning (LACL) for histopathology whole slide image analysis. We built a lesion queue based on the memory bank structure to store the representations of different classes of WSIs, which allowed the contrastive model to selectively define the negative pairs during the training. Moreover, We designed a queue refinement strategy to purify the representations stored in the lesion queue. The experimental results demonstrate that LACL achieves the best performance in histopathology image representation learning on different datasets, and outperforms state-of-the-art methods under different WSI classification benchmarks.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16434-7_27

SharedIt: https://rdcu.be/cVRrJ

Link to the code repository

https://github.com/junl21/lacl

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The paper proposed a novel weakly supervised representations learning method with a designed lesion queue under the setting of contrastive learning.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The paper is well organized and easy to follow.
2. The proposed contrastive learning method which can guide the negative samples selected from different classes is novel and interesting.
3. Experiments conducted on different datasets with CLAM and TransMIL demonstrate the effectiveness of the proposed pretraining strategy.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The paper state that a lesion-aware contrastive learning method that is specially designed for pathological images is proposed. This seems to be over-sale. This method can be performed on any other type of image with a corresponding label. The authors need to clarify this in the introduction.
2. Why a dynamic queue is needed (The lesion queue). As the label of each WSI is integrated into this framework, what if selecting negative samples from WSIs with different labels directly. More descriptions and experiments are needed.
3. The datasets seem to be private. If they will not be released, more details about them need to be listed, like the patients/WSIs of each class, and the magnifications of WSIs.
4. Please clarify the metrics reported in Table1-3 are in which level. Patient-level? WSI-level? Or patch-level.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Reasonable level of implementation details has been provided in the paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

See above.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

There is some novelty in the method design, especially for the member back queue part. The experiment is solid and proves the effectiveness of this method.
Number of papers in your stack

5
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

The authors propose a weakly supervised contrastive learning framework called LACL, where WSI labels are introduced to relief the class collision problem. Specifically, LACL builds a queue for each WSI class, thus negative samples are selected from different classes. To effectively update the queue, the authors select the representative samples based on the similarity distribution. The authors validate LACL on two WSI benchmarks and achieves incremental performance gain.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The organization of the paper is clear and easy to follow. The motivation and method are clearly demonstrated.
2. The experiment is basically complete. The proposed method achieves performance gain on different benchmarks.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The improvements are expected since more supervisions ( i.e., WSI labels) are used, which makes it an unfair comparison with other SSL methods.
- LACL is motivated by relieving the class collision problem in SSL. However, WSI classification is a typically multi-instance learning problem, where the tile labels are usually different from WSI labels. Though the WSI classification performance is improved, to what degree the class collision problem is mitigated, is not discussed and analyzed.
- Some settings seem unreasonable, e.g., the expected distribution is queue refinement strategy, where the similarity is 1 and 0 for same and different pseudo-labels, respectively. However, the ideal similarity between instances with different labels should be -1 under cosine similarity. More explanation should be provided, as well as the setting of threshold in eq.5 and the same length of each queue, etc.
- Experiment analysis in section 4.3 is not convincing.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The code will be released for reproducibility.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- As the WSI labels are weak supervision, it would help to provide more analysis about the risk of overfitting the noisy pseudo labels.
- To what degree the class collision problem is mitigated should be discussed and analyzed, e.g., the T-sne visualization.
- More explanations of settings: (a) The expected distribution is queue refinement strategy, where the similarity is 1 and 0 for same and different pseudo-labels, respectively. However, the ideal similarity between instances with different labels should be -1 under cosine similarity. (b) The threshold in eq.5: As the instances in mini-batch are randomly samples, how to understand the average KL divergence? (c) The same length of lesion queue for each class: As the class imbalanced problem (metioned in section 4.3), the update of each queue may be inconsistent under such settings.
- More details of experiments: (a) As the authors also split a validation set, it would help to explain how to use it in self-supervised learning for early stop? (b) The authors should clarity the evalution protocal, i.e., fixed or fine-tuning. (c) The authors are encouraged to explain why the MLP is of fc-BN-ReLU-fc (BYOL style) instead of fc-ReLu-fc (MoCo v2 style), as the MoCo v2 is chosen as baseline.
- Better experiment analysis in section 4.3.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The proposed method is well-motivated but not well-explained. There are some settings are unclear and more details should be provided.
Number of papers in your stack

7
What is the ranking of this paper in your review stack?

3
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

Authors propose a novel class-aware semi-supervised contrastive learning framework LACL for histopathological images. LACL replaces the memory bank of MoCo with several class-specific queues. Negative samples are generated from these queues to alleviate class collision problem. To guarantee the queue purity, a queue refinement strategy is proposed for queue updating. Authors validate the proposed method on two WSI-level classification datasets.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The main strength of this paper is the novel class-aware semi-supervised contrastive learning framework LACL. Unlike MoCo, LACL maintains several class-specific queues, so it can alleviate the class collision and provide reliable negative samples for contrastive learning. Besides, a queue refinement strategy is proposed to guarantee the queue purity. Ablation studies show the contributions of class-specific queues and queue refinement.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The method comparison is insufficient/unfair. The proposed LACL is a semi-supervised learning method, it utilizes the labels of WSI during contrastive representation. However, authors compare their method only with self-supervised/unsupervised learning methods.
2. Authors abuse the mathematic notation for i and y in Eq. (3) and Eq. (4), which may confuse readers.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Authors claim they will release the code. If so, the study could be reproduced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1. For the title, “class-aware” is more suitable than “lesion-aware”.
2. For fairness, author should at least compare the representation capacity of the proposed method with a ResNet50 trained with WSI-level labels.
3. Authors can discuss the effect of queue initialization on the performance.
4. Eq. (3) and Eq. (4) should be reformulated.
5. Typos like “MoVo” and “z_k = f_q(v_k)” should be corrected.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Novel class-aware contrastive learning framework, insufficient/unfair method comparison.
Number of papers in your stack

5
What is the ranking of this paper in your review stack?

2
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper proposed a novel weakly supervised representations learning method. The innovation is appreciated by all reviewers. Moreover, the paper is well written and easy to follow. I think it will be an interesting topic for MICCAI community. Even fundamental changes and more experiments are not suggested in MICCAI review process, it would be helpful if the ambiguities are clarified.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

1

Author Feedback

We thanks the reviewers and meta-reviewer for their efforts in our work. We realized it would be helpful to clarify some points based on the reviewer’s comments, which are listed below.

The motivation of the proposed method. Our motivation for proposing this method is to address the problem of noisy supervision of histopathological representation learning when only the WSI label is available for training. In this case, there are many samples whose actual class does not match the WSI label. Therefore, we proposed this method to learn lesion-specific representations based on the noisy labels for histopathology WSI analysis. Certainly, the method can indeed be applied in various labeled contexts. However, supervised learning may be the better choice when the samples can be correctly labeled.

The necessity of the dynamic queue updating. As we only use WSI-level label, there are many false positive labels (noisy labels) in the initialized pseudo labels. Firstly, the dynamic queues can purify each lesion queues by pushing more samples with true labels into the queues based on the proposed update strategy. Secondly, this dynamic update ensures that the representations in the queue keep consistent with the current model, just like MoCo did.

Further comparison with weak/noisy supervised methods. We realized the comparison with weak/noisy supervised methods are also necessary. Therefore, we conducted another two experiments involving weakly supervised representation learning methods on the EGFR dataset under CLAM benchmark, multi-instance representation learning (Lerousseau, M. et al., in MICCAI 2020. pp. 470–479.) and pseudo-labels supervision (as suggested by Reviewer #3), which had achieved ACC/AUC/F1-Score of 0.466/0.804/0.431 and 0.426/0.776/0.403, respectively. These results have significant gaps to those achieved by our method (0.525/0.826/0.510). We will add these results on Table 3, as will the results by TransMIL and on Endometrial dataset.

Other clarification. –A small KL value indicates a good representation that matches to its WSI label. The setting of average KL divergence can be regarded as a threshold operation. It figures out half of the representations from the mini-bath that are more representative to their categories and updates these representations to the queues. –The method is not strict for class-balanced mini-batch. We alleviate the problem of class imbalance by building equal-length queues for each class. Certainly, we suppose doing category balancing in mini-batch and doing class-specific queue update, as the reviewer suggesting, might further improve the performance of the method. We will consider it in our future work. –In the pre-training stage, i.e., representation learning stage, we trained the models with a same number of epochs for all methods (except the ImageNet transfer learning). In the second stage, we extracted representations using the trained feature extractors and fixed the representations in the WSI classification stage. –Early stop was only used in the second stage where we evaluated the representations using the benchmark for WSI classification. –We normalized the cosine similarity using the softmax function to make the similarity values range from 0 to 1. –We randomly initialized the queue referring to MoCov2 to verify the effectiveness of the main innovation of our method. –All the WSIs we used were scanned under 20x lenses. –All metrics are reported at the WSI level. –We checked the code for experiments and found that we used MoCo style MLP (MoCo v2 style) rather than BYOL style in our method. We misrepresented it as BYOL style in the original submission. We will revise it in the next version. Thanks for the reviewer’s reminder.

We will correct the formulas throughout the paper and try our best to polish our writings to clarify the possible ambiguities.

back to top

Lesion-Aware Contrastive Representation Learning for Histopathology Whole Slide Images Analysis