Authors

Lulin Shi, Yan Zhang, Ivy H. M. Wong, Claudia T. K. Lo, Terence T. W. Wong

Abstract

The conventional histopathology paradigm can provide the gold standard for clinical diagnosis, which, however, suffers from lengthy processing time and requires costly laboratory equipment. Recent advancements made in deep learning for computational histopathology have sparked lots of efforts in achieving a rapid chemical-free staining technique. Yet, existing approaches are limited to well-prepared thin sections, and invalid in handling more than one stain. In this paper, we present a multiple histological staining model for thick tissues (MulHiST), without any laborious sample preparation, sectioning, and staining process. We use the grey-scale light-sheet microscopy image of thick tissues as model input and transfer it into different histologically stained versions, including hematoxylin and eosin (H&E), Masson’s trichrome (MT), and periodic acid-Schiff (PAS). This is the first work that enables the automatic and simultaneous generation of multiple histological staining for thick biological samples. Moreover, we empirically demonstrate that the AdaIN-based generator offers an advantage over other configurations to achieve higher-quality multi-style image generation. Extensive experiments also indicated that multi-domain data fusion is conducive to the model capturing shared pathological features. We believe that the proposed MulHiST can potentially be applied in clinical rapid pathology and will significantly improve the current histological workflow.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43987-2_71

SharedIt: https://rdcu.be/dnwKu

Link to the code repository

https://github.com/TABLAB-HKUST/MulHiST

Link to the dataset(s)

N/A

Reviews

Review #2

Please describe the contribution of the paper

This paper presents deep learning-based a multiple staining model for thick tissues sections (MulHiST) by using grey-scale light sheet microscopy image as an input. The image is transferred into different histologically stained version, including hematoxylin and eosin (H&E), Masson’s trichrome (MT), and periodic acid Schiff (PAS). To the best of knowledge this appears to be the first contribution that generates multiple stained images from one thick section of the tissue.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The main novelty of the paper is architecture of AdaIN-based generator for creating multiple stained versions of the thick tissue sections. It is empirically demonstrated in the paper to be superior to other architectures for the same problem. The concept uses domain-specific-attribute vector in a form of one hot vector to indicate unstained and various stained domains. The generator aims to transfer the image style input to desired style domain while preserving the context of the input. The generator learns mappings between any two domains.

The main strength of proposed method is that no or very little sample preparation is required in generation of multiple stained versions of the thick tissue sections. It has potential to be applied in clinical rapid pathology and significantly improve current histological workflow.

Unlike SOTA methods that require pixel-wise registration source-unstained and target-stained image pairs for supervised model training, propose MulHiST method generates multiple stained versions from unstained input in a fully unsupervised way. That distinction is crucially important, since pixel-wise registered image can only be provided for thin tissue sections. Thus, multiple stains of the same tissue sections are not possible to generate with such concepts. The current SOTA methods for thick tissue sections can only produce virtual H&E-stained images. The current SOTA methods for multiple stain transfers require well-stained H&E stained image as an input. That requires laborious section preparation and chemical staining process.

Proposed model was verified on whole slide images obtained from six thick tissue slabs working with 256x256 patches. One tissue section was used for model training, and five for testing. The model is quantitatively evaluated in terms of Kernel Inception Distance (KID) and Frechet Inception Distance (FID). To validate clinical values of generated images the model trained on thick tissue section was used to predict the cell nuclei on thin tissue section so it can be compared with fluorescent images of the same thin section. The baseline models were cycleGAN, MUNIT and starGAN. Proposed MulHiST model outperformed other methods significantly. The MulHiST model was further verified on 2 sets of thin mouse kidney sections. As shown in Figure 3, the model generated correct pathological features. Table 1 indicated that proposed MulHiST model outperformed baseline models in several image quality metrics.

Ablation study with the results displayed in Figures 4 and 5 confirmed the two main hypothesis upon which the MulHiST model was built: (i) one model can benefit from data fusion from multiple domains due to domain-invariant features that are shared by multiple domains; (ii) the AdaIN-based style transfer is better in source image feature extraction than channel-wise style concatenation.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The main weakness of proposed MulHiST concept is lack of clarity in explaining how the training stage of the GAN network is performed. The authors emphasized several times that their method is optimized in an unsupervised manner, does not require pixel-wise registration source-unstained and target-stained image pairs for supervised model training, or as the current SOTA methods for multiple stain transfers does not require well-stained H&E stained image as an input. However, based on section 2 it appears that N-to-N manner is used to train generator that learns the mapping between the every two domains. To that end, multiple domain images are used during the training. That is said prior to eq.(1) and visible on Figure 1. It appears that images from different domains do not have to be registered. It is unclear what other specific requirements (if any) have to be met in preparation of such training ensemble. At the beginning of section 3.1, authors say that standard protocols are used to obtain different-style images for training. The authors should clarify that earlier in the paper. In particular that have to be more specific on what does it mean when they say that model is optimized in an unsupervised manner.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors promised that PyTorch code of proposed MulHiST method will be available upon acceptance. So far, reproducibility is based on description of the method in Section 2. It is very elementary as limited by the format of the paper. On descriptive level I can only judge meaningfulness and novelty of their proposal. Up to certain extent described results on performed experiments back up the statements and suggest that presented results will be reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

The main weakness of proposed MulHiST concept is lack of clarity in explaining how the training stage of the GAN network is performed. The authors emphasized several times that their method is optimized in an unsupervised manner, does not require pixel-wise registration source-unstained and target-stained image pairs for supervised model training, or as the current SOTA methods for multiple stain transfers does not require well-stained H&E stained image as an input. However, based on section 2 it appears that N-to-N manner is used to train generator that learns the mapping between the every two domains. To that end, multiple domain images are used during the training. That is said prior to eq.(1) and visible on Figure 1. It appears that images from different domains do not have to be registered. It is unclear what other specific requirements (if any) have to be met in preparation of such training ensemble. At the beginning of section 3.1, authors say that standard protocols are used to obtain different-style images for training. The authors should clarify that earlier in the paper. In particular that have to be more specific on what does it mean when they say that model is optimized in an unsupervised manner.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This paper presents deep learning-based a multiple staining model for thick tissues sections (MulHiST) by using grey-scale light sheet microscopy image as an input. The image is transferred into different histologically stained version, including hematoxylin and eosin (H&E), Masson’s trichrome (MT), and periodic acid Schiff (PAS). To the best of knowledge this appears to be the first contribution that generates multiple stained images from one thick section of the tissue. The main strength of proposed method is that no or very little sample preparation is required in generation of multiple stained versions of the thick tissue sections. It has potential to be applied in clinical rapid pathology and significantly improve current histological workflow.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #4

Please describe the contribution of the paper

This work leveraged the existed StarGAN with an innovative idea by adding the adaptive instance normalization, referred as AdaIN layer. The authors applied the MulHiST to generate virtual staining images including H&E, MT, and PAS.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The proposed MulHiST architecture have a significant potential in clinical practice to generate high-quality virtual staining images rather than the conventional clinical staining process. Compared with the existed AI-based virtual staining imaging generation approaches, the proposed MultiHiST model could not only generate the synthetic H&E images, but also the photorealistic MT, and PAS images.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The qualitative assessment by comparing the generated images using the proposed MulHiST with the conventional approaches, including CycleGAN, MUNIT, and StarGAN is lacking ground truth, and no sufficient pathologists insights provided on these assessments. These visualization results are hard for the general audience to determine whether the proposed MulHiST outperform the conventional approaches.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors did provide sufficient information with details in data preprocessing and model training. The computing infrastructures, model training hyper-parameters are clearly stated for experiment reproducibility purposes.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

The proposed MulHiST architecture have a significant potential in clinical practice to generate high-quality virtual staining images rather than the conventional clinical staining process. Compared with the existed AI-based virtual staining imaging generation approaches, the proposed MultiHiST model could not only generate the synthetic H&E images, but also the photorealistic MT, and PAS images. However, the qualitative assessment by comparing the generated images using the proposed MulHiST with the conventional approaches, including CycleGAN, MUNIT, and StarGAN is lacking ground truth, and no sufficient pathologists insights provided on these assessments. These visualization results are hard for the general audience to determine whether the proposed MulHiST outperform the conventional approaches.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The innovation of the proposed architecture, and the potential clinical use case and significance of the proposed work.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #1

Please describe the contribution of the paper

This paper proposes a method for automatically and simultaneously generating multi-histological staining of thick biological samples, which maybe potentially be applied in clinical rapid pathology.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The paper is the first attempt to achieve multiple histological staining generations for thick tissues.
2. The authors incoporate multi-domain translation into MulHiST.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. No cross validation is conducted in the experimental part.
2. The expression of the paper is not very clear.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

not good. Since the in-house dataset were used, more details on the data acquisition should be given
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
1.The authors should add citations for the comparison methods. 2.The author should compare some more recent methods, as the latest contrast methos was proposed in 2018.
1. The method section should be divided into multiple subsections
2. Does the method proposed in this paper also require stained images for training the model? If so, what are the advantages of the proposed method compared to those in references [9] and [10]?
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. The paper is the first attempt to achieve multiple histological staining generations for thick tissues, which maybe potentially be applied in clinical rapid pathology.
Reviewer confidence

Somewhat confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper proposed the first attempt to achieve multiple histological staining generations for thick tissues. It has significant potential in clinical practice to generate high-quality virtual staining images rather than the conventional clinical staining process. The innovation is appreciated by all reviewers. Moreover, the paper is well written and easy to follow. I think it will be an interesting topic for MICCAI. Even fundamental changes and more experiments are not suggested in MICCAI review process, it would be helpful if the ambiguities are clarified.

Author Feedback

N/A

back to top

MulHiST: Multiple Histological Staining for Thick Biological Samples via Unsupervised Image-to-Image Translation