List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Parmida Ghahremani, Joseph Marino, Juan Hernandez-Prera, Janis V. de la Iglesia, Robbert J. C. Slebos, Christine H. Chung, Saad Nadeem
Abstract
We introduce a new AI-ready computational pathology dataset containing restained and co-registered digitized images from eight head-and-neck squamous cell carcinoma patients. Specifically, the same tumor sections were stained with the expensive multiplex immunofluorescence (mIF) assay first and then restained with cheaper multiplex immunohistochemistry (mIHC). This is a first public dataset that demonstrates the equivalence of these two staining methods which in turn allows several use cases; due to the equivalence, our cheaper mIHC staining protocol can offset the need for expensive mIF staining/scanning which requires highly-skilled lab technicians. As opposed to subjective and error-prone immune cell annotations from individual pathologists (disagreement > 50%) to drive SOTA deep learning approaches, this dataset provides objective immune and tumor cell annotations via mIF/mIHC restaining for more reproducible and accurate characterization of tumor immune microenvironment (e.g. for immunotherapy). We demonstrate the effectiveness of this dataset in three use cases: (1) IHC quantification of CD3/CD8 tumor-infiltrating lymphocytes via style transfer, (2) virtual translation of cheap mIHC stains to more expensive mIF stains, and (3) virtual tumor/immune cellular phenotyping on standard hematoxylin images. The dataset is available at https://github.com/nadeemlab/DeepLIIF.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43987-2_68
SharedIt: https://rdcu.be/dnwKr
Link to the code repository
https://github.com/nadeemlab/DeepLIIF
Link to the dataset(s)
https://github.com/nadeemlab/DeepLIIF
Reviews
Review #1
- Please describe the contribution of the paper
This paper presents a new dataset that will be publicly released, containing histopathological images from the same tumor sections in both multiplex immunofluorescence and multiplex immunohistochemistry. The dataset contains images from 8 different patients suffering from head&neck cancer.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- new public dataset in histopatholoy with multiplex immunofluorescence images registered with immunochemistry is really usefull for the community
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- no information on the size (number of images) of the dataset
- results for one usecase over evaluated
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
No relevant
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The main concerns on this paper are first of all the presentation of the dataset which could be more detailed and clearer. A table could for example summarise all the images that will be available for each patient, with the number of cells, etc. Secondly, on the first use case presented (IHC CD3/CD8 scoring using mIF style transfer), the authors claim that the use of their dataset outperforms the use of NuClick, which is not statistically correct in view of the results and standard deviations presented in table 2.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Having a new public dataset is very interesting for the field. However, the description of the data and the experiments showing its interest should be more precise and acurate.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The authors propose a dataset consisting of co-registered images using different staining techniques to visualize cell properties (mIHC and mIF). They support the dataset by including some experiments they conduct using the dataset such as style transfer and cell phenotyping.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Co-registered datasets are rare, especially between immunofluorescence and IHC stainings. This dataset presents a contribution to the community of biomedical image analysis by providing 1. insights into the quality of the different stainings (but using the same biological markers) and 2. the data to train style transfer networks, relying on cheap and easy-to-use staining techniques, and 3. applying pretrained networks on style-transformed data that would otherwise not be applicable as of a different domain.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The paper lacks a discussion, leaving the reader with some speculation of how the dataset could be applied to other use cases then that presented in section 3. In general, the dataset is interesting because the direct comparison between staining techniques is investigated. To the best of my knowledge, this depicts a novelty. However, it is difficult to estimate the impact of the proposed dataset to the community. First, the images stem from only one disease, which limits the potential application when training deep learning architectures on arbitrary tasks. In addition, the size of the dataset seems to be limited in general, as it consists of 9 ROIs from 8 cases with different cancer sites, so 72 images in total and 36 images per cancer site. This is a rather low number if it comes to training of approaches such as style transfer (although results of section 2 seem promising, but it is unclear if the domain shift between the stainings are important w.r.t. segmentation). Another issue of the manuscript is that the dataset could have been described in more clarity. A table is missing describing how many images from which samples, stained with a certain type of marker, are available. This information can be found in the text, but is somehow hard to obtain. The description of the experiments conducted to show the usability of the dataset is rather confusing. The motivation to conduct CD3/CD8 scoring using mIF style transfer is not obvious - but is not given in the section (although the qualitative results look convincing). The architecture of the networks contributing to the style transfer are shown in the supplementary file, but could have been presented better - which are the input images, what are intermediate results, which images of the dataset were used (and which channels), are images from public datasets used? I have to state that i have a strong background on computer vision especially working on fluorescence microscopy images including multiplex immunofluorescence assays, but struggled to decipher the exact workflow.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The dataset does not seem to be available for download.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
-) The calculation of the concordance between the marker intensities is confusing and could have been described better. What is meant by “Each square represents an image in the dataset and the top half of each square shows the mean color value of the positive cells, extracted from mIHC-AEC using its corresponding mIF image”? Does it mean that based on thresholding the mIF image a mask was generated and applied to the mIHC image to calculate the intensity of foreground pixels upon masking? -) experiments conducted to show the usability of the dataset could have been described with more clarity. In section 3.1, the description of the worfklow to create the mIF style transfer is confusing, it would have helped to have a legend in Suppl. Fig. 1 (which should be placed in the main manuscript). In section 3.2 and 3.3, the DeepLIIF translation module could have been described, and some details of the training procedure would have helped to understand if the data is sufficient, and if the dataset could be applied to other use cases as well.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The manuscript presents a dataset and supporting experiments that could be useful for the biomedical image analysis community, especially for groups who work with flourescence images and multiplex assays, as is often the case in cancer research. The comparison between the different staining techniques, applied on the very same slides, is interesting, potentially opening the possibility to train style transfer networks (although the manuscript lacks some details w.r.t. this use case). However, it is hard to guess the overall impact of the dataset, as its size is limited and it only contains images of patientes suffering from head-and-neck squamous cell carcinomas - including images of biopsies from patients diseased with different cancers could have helped to achieve a much broader impact.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The paper tries to establish proof for equivalence between mIF and mIHC.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- establishing of dataset
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- no statistical analysis
- no conclusion
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper focuses on dataset that is strictly related to the data, so it is reproducible for someone possessing the base data.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- is 20x magnification typical and enough for such study? please provide proper reference for justification
- when equivalence is considered - the lack of difference should be proved with statistical analysis - test such as McNemar’s or 5x2cv could be used
- conclusion should be expanded
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
establishing new datasets is very important
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
All the three reviewers appreciate that a new dataset is introduced. It is very valuable to have a public co-registered image data that uses different staining techniques, such as multiplex immunofluorescence (mIF) and multiplex immunohistochemistry (mIHC), for the microscopy image analysis community. The manuscript also presents three use cases to demonstrate the values of the dataset, including IHC CD3/CD8 scoring and image style transfer.
However, the reviewers also raised some concerns: the description of the dataset is not clear and more details are needed (Reviewers #1 and #3), the experimental results are overstated by considering the standard deviation values in Table 2 (Reviewer #1), lack of statistical analysis/tests to verify the effectiveness of the dataset (Reviewer #2), and the discussion or description of the usability of the dataset is not clear (Reviewer #3). In addition, Reviewer #3 mentioned that the dataset is small (only from 8 subjects) and the images are acquired from one type of disease, and thus the impact of the dataset might be questionable. Please consider addressing these comments and improve the manuscript when preparing the final version.
Author Feedback
Thank you. We will address these concerns in the final version.