List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Kazuya Nishimura, Ami Katanaya, Shinichiro Chuma, Ryoma Bise
Abstract
Detection of mitosis events plays an important role in biomedical research. Deep-learning-based mitosis detection methods achieve outstanding performance with a certain amount of labeled data. However, these methods require annotation for each imaging condition. Collecting a certain amount of labeled data is time-consuming and human labor. In this paper, we propose a mitosis detection method that can be trained with partially annotated sequences. The base idea is to generate a fully labeled dataset from the partially annotated sequences. We first generate an image pair which not contain mitosis events by frame-order-flipping. Then, we paste mitosis events to the image pair by alpha-blending-pasting and generate a fully labeled dataset. We demonstrate the performance of our method on four datasets, and we confirm that our method outperforms other comparisons which use partially labeled sequences.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43993-3_47
SharedIt: https://rdcu.be/dnwNS
Link to the code repository
https://github.com/naivete5656/MDPAFOF
Link to the dataset(s)
https://github.com/naivete5656/MDPAFOF
Reviews
Review #1
- Please describe the contribution of the paper
The paper presents a method for mitosis detection trained with partially annotated data. The method flips the order of video frames to ensure that there are no mitosis events in the unlabelled regions and then uses copy pasting of annotated mitosis cells using alpha blending to add mitosis events to the flipped images. The flipping of the order of frames has no impact on other cells as only their movement direction is reversed. This augmented/generated dataset is then used to train a mitosis detector.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper is well written and easy to follow
- The idea of flipping frames is really simple but appropriate for mitosis detection.
- The synthesized images using alpha blending look realistic.
- The experiments are well designed and thorough, i.e. 4-fold cross validation and the N-samples used for N-shot learning were selected using 5 different seeds.
- The proposed method improves performance on 4 public datasets in the 1/5-shot setting.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The main contributions of the proposed method (frame order flipping and alpha blending) are relatively simple, which can be considered a weakness.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper is quite clear and should be reproducible with some effort. However, it would be nice if the authors publish their code.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- Table 1 in supplementary material probably should be in the main paper as it is good to know how the method performs when given a larger number of annotations.
- It would have been nice to include larger images in the supplementary material.
- It would be good to include standard deviation (over the folds and different seeds) in table 1.
- page 7: It is unclear why data augmentation is considered as an alternative to FOF. Was the data augmentation not used in the proposed method?
- Minor language issue, e.g. on page3: “unlabeled learning is not work on mitosis detection.” -> “unlabeled learning does not work on mitosis detection.”
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper presents a simple but relevant techniques for improving mitosis detection performance. It performs thorough experiments on 4 public datasets and the results demonstarte the merits of their proposed techniques.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The paper deals with the problem of mitosis detection from consecutive 2D frames of microscopy movies of fluorescently labeled cells. Specifically they address the problem of limited training data/partially labeled images, i.e. movies where only a subset of all true mitosis events are annotated. When naively used as training data, this would result in potentially many false negatives as there are potentially many frame pairs containing an unlabeled division. To remedy this, they propose to i) temporally flip all frame pairs that do not contain annotated division event, thus making sure that no temporally consistent division event will be present anymore, and ii) augmenting the training data by pasting images containing a vision into background regions using a specific alpha blending process. They finally show that this improves mitosis detection results when compared to baselines.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Practically relevant problem
- A simple idea that seems to work well in practice
- Improvements are quite large compared to all compared methods
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- paper is hard to read and contains many typos and even missing text (“unlabeled learning is not work on mitosis detection. the appearance of the mitosis”)
- many details are missing:
- what is the 1-shot/5-shot setting? Using only 1/5 labeled divisions?
- For the F1 metric, what distance was used in time (like 15 pixels in space)?
- no code
- Experimental results are relatively scarce. Table 1 only compares 1/5 shot setting (which is rather unrealistic). Additionally, Fig 7 only reports 0-30% missing, why not up to 100%? Furthermore, all compared baselines work substantially less well than the proposed method. E.g. PU, which aim to address the same problem, is basically failing completely. Why is that? Was PU applied correctly?
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Seems to be fine (but no code provided)
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The paper needs to be proof-read and the clarity improved. Apart from that I think its a simple idea that seems to work well in practice, albeit the experimental demonstration is not entirely convincing (see above).
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
As before.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This manuscript presents a method for generating a fully-labelled data set for mitosis detection. The method is validated against several methods designed to improve partial annotations for cell detection, which turns out to be a rather different task as all these methods perform poorly when applied to mitosis detection. Some additional experiments shed a bit more light on the performance, but many questions remain unanswered. Moreover, the need for developing this approach is not convincingly presented.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
• Novel method for simulating cell mitosis events.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
• Questionable motivation. • Questionable validation (against infeasible methods). • Lack of detail and structure.
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Some parts of the applied methodology are not described with sufficiently level of detail; see, in particular, my comment 4.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
-
The authors approach development of this methods from a similar perspective as for the problem of cell detection. However, these two problems, in my opinion, seem to be rather different as the former is much less prevalent than the latter. Among four validation sequences, the highest average division number is 141, which requires virtually the same number of clicks to annotate all of the mitotic events and should not take much time. Moreover, the authors validate their method against the method developed for dealing with partial cell labels and perform (very) poorly when applied to the task at hand as in vast majority of the cases it is much better to do nothing at all (‘Baseline’) than apply these methods.
-
At the same time, at several places in the manuscript the authors identify problem op “potentially missed mitosis events” as a major problem. Which means that they should have trained a network on the fully annotated set, (this would not take so much time as I have already pointed out) and used these results (including the annotation time) as a benchmark. Results of such an experiment are indeed presented later, in particular in Figure 7. This figure illustrates that the proposed method improves performance. However, even for the 30% of the missing annotations the performance still remains reasonable. Moreover, it is entirely unclear on what sequence were these results obtained.
-
Later, from Table S1 we can deduce that results in Figure 7 were obtained on the ES sequence. At the same time, it appears that, when applied to a fully-annotated sequence having reasonable amount of mitotic events, the proposed method actually deteriorates. And for the remaining sequences the improvement remains marginal, with the exception of the sequence that has a very low amount of such events. I find this an important result, which should definitely be moved to the main body of the manuscript. I am also struggling to see the added value of the “Average” statistics, as the sequences differ so strongly in the number of mitotic events.
-
The manuscript is missing many important detail: how the data was split for training and testing, on how many events was the method effectively trained (as also additional augmentation was applied), which sequences were used for obtaining the results in Table 2 and Figure 7, etc. I personally also find the method description in the paragraph starting with the words: “Table 1 shows the quantitative…”very vague and unclear.
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Major weaknesses in all core parts of the manuscript: motivation, validation, presentation, structure
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
4
- [Post rebuttal] Please justify your decision
I do agree that this method is based on an interesting idea of flipping the frame order. Generation of this kind of synthetic images might indeed be useful in certain scenarios; in particular, for the phase-contrast or histopathology data. However, in my opinion, this is much less the case for the fluorescence microscopy data, including the sequences chosen for the method validation. Hence, my overall assessment of this submission remains unchanged, although I did increase my grade based on the provided rebuttal.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper presents a simple yet effective method for mitosis detection in time-lapse fluorescence microscopy images, by using frame-order flipping and alpha-blending pasting. The experiments show that the method outperforms some other competitor approaches on multiple datasets. The reviewers appreciated the simple idea to generate a fully labeled dataset from partially annotated data, so that a data-hungry deep model can be successfully trained. R1 also gave positive comments on the performance improvement compared with other approaches. However, the reviewers also pointed out several concerns as follows:
-
Presentation of the method and the experiments is not clear, and many details are missing, such as why data augmentation can be considered as an alternative to the frame-order flipping (R1), what 1-shot and 5-shot settings are (R2), what distance is used in time to calculate F1 (R2), which sequences are used for Table 2 and Figure 7 (R3), and many others.
-
The experimental results in the supplementary document (e.g., Table 1) should be moved to the main text, and more explanation/discussion should be provided (R1 and R3).
-
Experiments are not convincing (R2 and R3). In particular, the competitor methods are specifically designed to handle imperfect labels, but they provide much worse performance than the baseline [13]. More discussion/analysis is needed. Additionally, why are only experiments on 0-30% missing labels provided?
-
The manuscript is difficult to follow due to typos and missing text (R2), e.g., “… unlabeled learning is not work on mitosis detection. the appearance of the mitosis …”. R3 also commented that the clarity and organization of the paper should be improved.
-
The motivation of the study is not well verified, either in the method description or in the experiments (R3). For instance, it may not take much time to annotate all of the mitotic events in the sequences used in the experiments, given that the numbers of mitotic events are low.
Please consider addressing the comments above in the rebuttal.
-
Author Feedback
We thank the reviewers for their insightful comments, where all reviewers recognized the novelty of our method. We are encouraged that they find our method to be appropriate (R1), practically well (R2), and novel (R3), and experiments are well designed (R1). We will address the main concerns as follows.
Motivation of the study (R3) Our method can reduce the annotation cost in mitosis event analysis. The annotation of mitotic cells is more difficult than expected because the appearance is similar to the normal cells, as stated in the 3rd paragraph of Section 1. Please imagine exhaustively finding all mitosis events from more than hundreds of cells that appear similar to mitotic cells in a video. We have to carefully check the detailed motion of each cell in all frames to avoid missing annotations. It takes time. Our method could detect mitosis events with only five annotations, whose cost is extremely less than the supervised method. In addition, biologists often want to extend the analysis to various cell types and cell culture conditions. In this case, it is necessary to create training data for various conditions since deep learning methods do not work well in the different conditions from the supervised data due to domain shift problems, e.g., different image appearances by cell types and imaging setups. Therefore, we consider our method useful to deal with multiple conditions.
Detail of experiments settings R2, R3: Reasons for the poor performance of comparisons. As described in Sections 1 and 3, the comparative methods are designed for cell detection problems and assume flat backgrounds (the area there are no cells), i.e., the appearance of the background is different from the cell regions. However, in mitosis event detection tasks, the background contains cells that appear similar to mitosis cells. Because the comparative methods do not consider such backgrounds, these methods could not work. In contrast, our frame-order-flipping can easily generate background images. Therefore, our method outperformed comparative methods.
R1: Why use data augmentation as an alternative way of frame-order-flipping? This is another simple method to make a ‘background image’ synthetically. Since the augmented data has the same number of cells as the original image, the pair of the original and augmented images do not contain mitosis, i.e., background. To perform an ablation study of frame-order-flipping, we used this method.
R2: Why are only experiments on 0-30% missing labels provided? The reason is we considered the actual missing label rate when creating supervised annotation is at most 30%. Since Baseline does not consider missing labels, our method is naturally considered superior to Baseline on a large missing rate. In fact, we evaluated our method on more high missing rate conditions ranging from 40% to 90% and confirmed that our method is superior to Baseline.
R1, R3: The supplementary document (e.g., Table 1) should be moved to the main text. Since the main target of our method is addressing learning from partial labels, we consider that the priority of the effectiveness of the proposed method in supervised settings is lower than the analysis described in the main paper. Due to the page limit, we keep the current organization; the experiments are provided in the supplemental document.
- Clarity and organization of the paper R1: What are 1 or 5-shot settings? As described in the 5th paragraph of Section 3, it indicates the number of annotated mitosis events used for training data. So, in the N-shot setting, we generated synthetic training data based on only the N mitosis events.
R2: What distance is used in time to calculate F1 We used 6 for time distance, which is the same as the CVPR mitosis detection contest.
R3: Which datasets are used for Table 2 and Figure 7? We used the HeLa dataset for Table 2 and the ES dataset for Figure 7.
We will clarify these descriptions in our paper.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This manuscript introduces an interesting method to generate fully-labeled datasets from partial annotations for mitosis detection in fluorescence microscopy images, mainly based on frame-order flipping and alpha-blending pasting. The reviewers raised some concerns regarding the presentation, especially for the motivation and the experiments. The rebuttal clarified the motivation of the study and also addressed some of the reviewers’ concerns about the experimental settings and result interpretation. In addition, the reviewers gave positive comments on the idea of data generation for deep learning-based mitosis detection applications. Thus, although the clarity of the manuscript needs to be further improved (the authors are encouraged to do so), an acceptance is recommended.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
strengths; a simple method in simulating cell mitosis to address the important problem of mitosis detection, with noticeable performance improvement weaknesses; code is not publicly available; empirical evaluations are not sufficient; presentation could be improved how the rebuttal informed your decision: not in a major way
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper presents a method for mitosis detection in time series videos which uses a simple yet elegant way of generating a negative pair of images by flipping the frames and then pasting annotated mitotic figures into the frames. This is appreciated by all reviewers, who also emphasize the value of the evaluation on multiple datasets. The weaknesses of this paper include the presentation of the method and the structure of the paper in general, the design and choice of the baseline methods, and the motivation of the study setup.
The authors addressed these points in their rebuttal; however, also after the rebuttal the motivation of the general setup and the choice of comparison methods does not convince me. As the authors correctly mention, the comparison methods expect a uniform background and don’t have negative “structures” as such. It did not become clear to me why the annotation task was not modelled to include also sparse negative events (no mitotic event on a cell), which should be extremely easy to obtain. Reviewer #1, who evaluated the paper most positively, raised several strength but did not question the general motivation of the approach. Together with the presentation, I believe the current paper is not at the level for publication as a MICCAI paper.