List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Yeman Brhane Hagos, Ayse U Akarca, Alan Ramsay, Riccardo L Rossi, Sabine Pomplun, Alessia Moioli, Andrea Gianatti, Christopher Mcnamara, Alessandro Rambaldi, Sergio A. Quezada, David Linch, Giuseppe Gritti, Teresa Marafioti, Yinyin Yuan
Abstract
Multispectral immunofluorescence (M-IF) analysis is used to investigate the cellular landscape of tissue sections and spatial interaction of cells. However, complex makeup of markers in the images hinders the accurate quantification of cell phenotypes. We developed DeepMIF, a new deep learning (DL) based tool with a graphical user interface (GUI) to detect and quantify cell phenotypes on M-IF images, and visualize whole slide image (WSI) and cell phenotypes. To identify cell phenotypes, we detected cells on the deconvoluted images followed by co-expression analysis to classify cells expressing single or multiple markers. We trained, tested and validated our model on > 50k expert single-cell annotations from multiple immune panels on 15 samples of follicular lymphoma patients. Our algorithm obtained a cell classification accuracy and area under the curve (AUC) ≥ 0.98 on an independent validation panel. The cell phenotype identification took on average 27.5 minutes per WSI, and rendering of the WSI took on average 0.07 minutes. DeepMIF is optimized to run on local computers or high-performance clusters independent of the host platform. These suggest that the DeepMIF is an accurate and efficient tool for the analysis and visualization of M-IF images, leading to the identification of novel prognostic cell phenotypes in tumours.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16440-8_14
SharedIt: https://rdcu.be/cVRvy
Link to the code repository
https://github.com/YemanBrhane/DeepMIF
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
- The authors develop a deep learning based framework to identify cell phenotypes for M-IF images.
- The authors develop a whole slide M-IF viewer to visualize the M-IF images as well as cell detection and phenotytes.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- A deep learning based framework is proposed for the cell detection and phenotyping for M-IF analysis.
- A GUI is developed for the M-IF visualization.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Cell detection is proposed on M-IF images, though the authors show promising classification performance based on the detection results. While cell segmentation is more desired from the biology point of view. Besides, there are existing deep learning based cell segmentation algorithm (deepcell).
- With the detected cells, a fixed size (20x20x3 for nuclear, and 28x28x3 for non-nuclear) is used for classify marker positivity. While this size lead to the inaccuracy for the marker positivity evaluation since different cell can have different size.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Highly reproducible. Also the authors suggest it would be publicly available .
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- Although cell detection is one manner for the cell-based analysis, while segmentaion based manner should be the preferred manner. Firstly, there is already model (deepcell) with promising performance, and secondly downstream analysis based on cell segmentation is far more interpreable compared with detection based manner.
- Secondly, the classification evaluated in this manuscript is confusing to me. Is the NK/T deemed as positive, and macrophage as negative in the setting? Then how about other cell phenotypes?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This study has great promising in the future, but the current framework needs further refinement. Cell detection is suggested to be replaced with cell segemntation. Secondly, the classification evaluation need to be further elaborated.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The author proposed DeepMIF, a novel deep learning method for detecting and quantifying cell phenotypes on M-IF images. DeepMIF also includes a GUI, making it accessible to researchers with less programming background. The authors also demonstrated that DeepMIF was able to achieve effective cell classification performance.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Building accessible computation tools is essential for better biomedical research. DeepMIF enables effective cell phenotype identification in M-IF images. It also shows great accessibility with a GUI and generalizability to different panels.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
While the paper is clearly written and the authors provided detailed results to demonstrate the effectiveness of the proposed method, some parts of the evaluation could be further improved. Moreover, as DeepMIF is proposed as an accessible tool, it would be great to see its application on different diseases. This is also mentioned by the authors in the limitations.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The reproducibility of the paper is good. The authors provided necessary details from the reproducibility checklist.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- For Table 1, could the authors also list the positive and negative distributions of each DI? It would be better to also include which markers are nuclear and which are non-nuclear.
- Since cell detection and classification are two stages in DeepMIF, it would be better to also show the overall combined evaluation metrics to show how effective cell types have been identified from detection to classification.
- For the external validation dataset, what’s the performance for cell detection? Could we perform cell detection for the external validation dataset based on the trained model from the Immune T cell panel? As DeepMIF cell identification is a two staged process, only reporting the classification performance could not provide a comprehensive view of the performance of DeepMIF.
- As performance between VGG16 and DeepMIF is very close, it might be better to perform cross validation to see if the performance of DeepMIF is statistically significantly better. Moreover, the authors mentioned that model 2 contains much fewer parameters than VGG16. What about the training time between model 2 and VGG16?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The problem addressed in the paper is quite important as accessible computational tools are essential for more efficient biomedical research. The paper is well-written and the results presented demonstrated the effectiveness of the proposed method.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #4
- Please describe the contribution of the paper
This paper proposes a graphical user interface based on deep learning for the profiling of cells in multispectral immunofluorescence. The paper is well writen and described and a graphical user interface would be attractive for researchers who are not interested in running scripts from the command line. As such it is an interesting contribution.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is well written, offer a standalone GUI capable of analysing multispectral data (up to 25 channels) and provided a very high output, higher than Resnet, Inception and other architectures. The improvement over VGG is marginal but still it is better, which is good, and at the levels they are working, would be difficult to have a higher improvement.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
My single criticism of the paper is that I could not find that the data is available. The code will be released through GitHub upon acceptance, and that is fine, but there was no mention (or I missed it) of where the data is and if it is available for comparison. The other limitations (small sample size, etc.) are acknowledged by the authors and is understandable at this stage of their work.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The software will be made available through github upon acceptance. This is reasonable and is not public yet, perhaps to keep the process double blind. The data on the other hand is not public or I could not see where this is available.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
This is a good paper bar the availability of the data.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I think this is the best paper of my stack.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The author proposed DeepMIF, a novel deep learning method for detecting and quantifying cell phenotypes on M-IF images. DeepMIF also includes a GUI, making it accessible to researchers with less programming background. The authors also demonstrated that DeepMIF was able to achieve effective cell classification performance. All reviewers consider the paper well-written and would make an important contribution to analyse multispectral data. Yet R1 raise some concern why authors choose for cell detection rather than cell segmentation, and R4 asks for possible publication of the dataset. Authors are encouraged to consider review comments in their final submission.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
1
Author Feedback
N/A