List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Yudong Zhang, Ge Yang
Abstract
Single particle tracking is an important image analysis technique widely used in biomedical sciences to follow the movement of subcellular structures, which typically appear as individual particles in fluorescence microscopy images. In practice, the low signal-to-noise ratio (SNR) of fluorescence microscopy images as well as the high density and complex movement of subcellular structures pose substantial technical challenges for accurate and robust tracking. In this paper, we propose a novel Transformer-based single particle tracking method called Motion Transformer Tracker (MoTT). By using its attention mechanism to learn complex particle behaviors from past and hypothetical future tracklets (i.e., fragments of trajectories), MoTT estimates the matching probabilities between each live/established tracklet and its multiple hypothesis tracklets simultaneously, as well as the existence probability and position of each live tracklet. Global optimization is then used to find the overall best matching for all live tracklets. For those tracklets with high existence probabilities but missing detections due to e.g., low SNRs, MoTT utilizes its estimated particle positions to substitute for the missed detections, a strategy we refer to as relinking in this study. Experiments have confirmed that this strategy substantially alleviates the impact of missed detections and enhances the robustness of our tracking method. Overall, our method substantially outperforms competing state-of-the-art methods on the ISBI Particle Tracking Challenge datasets. It provides a powerful tool for studying the complex spatiotemporal behavior of subcellular structures. The source code is publicly available at https://github.com/imzhangyd/MoTT.git.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43993-3_49
SharedIt: https://rdcu.be/dnwNU
Link to the code repository
https://github.com/imzhangyd/MoTT.git
Link to the dataset(s)
http://bioimageanalysis.org/track/
Reviews
Review #1
- Please describe the contribution of the paper
The paper proposes a transformer based multiple object tracking algorithm where the linking step of the tracking is performed via running a global optimization on probabilities given by a transformer trained on choosing the right particles corresponding to individual tracklets among a tree of hypotheses. Results on ISBI PTC datasets are shown where the method outperforms some existing ones.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
Usually, transformers are used in the context of language where the network is finding complex patterns that help with predicting the next token/word. The application of transformers to tracklets and tracking is interesting. Although I doubt if using transformers in the context of tracking is completely novel. Thinking of particles as tokens that will then be connected together to form a tracklet (sentence) is an interesting idea.
-
The presented empirical results and robustness analysis is helpful for determining the strengths and pitfalls of the proposed method.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
The literature isn’t fully discussed, there are many multiple cell tracking methods that are developed in the field addressing different challenges associated with tracking cells and biological compartments. See below for an incomplete list [1-6].
-
The time complexity and run time of different algorithms aren’t discussed in detail. Specifically, for the compared methods there’s no information on their algorithmic differences. This information is critical for bringing the comparisons to the same footing.
-
There’s no mention of the amount of training data required to achieve this level of accuracy and how it compares to other methods. I assume that some of the presented methods are off-the-shelf unsupervised methods not requiring any training data. This will help with generalizing across different domains whereas the presented algorithm may depend on the statistics of motion in different datasets.
-
The results are quite minimal, more results on complex biological datasets such as http://celltrackingchallenge.net/ would help the reader to decide if this is the right technique for their dataset.
-
I don’t find the results in Table 1 to be quite convincing for a few reasons. (1) There are other metrics developed in the field for assessing multi-object tracking (MOT) results such as CLEAR metrics which aren’t used in this table. (2) There are tracking algorithms that are one-shot (as opposed to detect-then-link). Comparisons against those would require a different strategy. (3) It’s unclear of the results reported are due to stronger linking strategy or better detection (this is partially addressed in the Table 2, but only against naive linkers such as KF and LAP. Also I’m not sure if they use any training data, so that brings up another issue with the comparisons. (4) Even taking all of these points for granted, still other methods outperform the proposed one in different settings. (5) Compared methods are mostly old (2021 is the most recent, and the one after that is 2019), there are many other algorithms developed since then for multiple cell tracking.
-
Correct me if I’m wrong but the idea of using transformers for tracking existed before, see [7] and the references therein. What’s the main novelty and contribution of this paper?
-
The relinking part is not properly and clearly described, I still don’t know how it exactly works.
[1] Rapid detection and recognition of whole brain activity in a freely behaving Caenorhabditis elegans [2] Fast deep neural correspondence for tracking and identifying neurons in c. legans using semi-synthetic training [3] Automatically tracking neurons in a moving and deforming brain [4] Robust single neuron tracking of calcium imaging in behaving Hydra [5] 3DeeCellTracker, a deep learning-based pipeline for segmenting and tracking cells in 3D time lapse image [6] Versatile Multiple Object Tracking in Sparse 2D/3D Videos Via Diffeomorphic Image Registration [7] Transformer Tracking
-
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Except for the training data the training strategy is described and reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Comparison against more recent methods in more benchmarks and controlling for the benefits given by detecting vs. linking (this is the main contribution) would help us assess the significance of the contribution.
Simulation studies in varying levels of background noise, motion smoothness and frame rate, object size and density, occlusions, etc. would be very helpful.
A more thorough literature review should be added to the paper.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
More complete benchmarking and literature review is needed for acceptance of this paper.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
I thank the authors for addressing most of my comments. The literature review and empirical results sections look stronger now. I’m going to increase my score to acceptance but I still have some concerns that I’ll right below.
-
Run time comparison of different methods. The authors mentioned that the algorithmic complexity is O(n^2). But that should be added with the time that it takes for the transformer to be trained and the inference time and solving the global optimization problem. A comparison of the wall clock time of various algorithms will be very helpful. Specifically given than some methods solve the tracking algorithm in one shot, inputing a video and outputting the tracks.
-
The relinking part of the paper is not properly described (it’s my only comment that’s not discussed by the authors in their response at all) and it remains the most unclear part of the paper.
-
The generalization properties of the methods is not known and this can hugely reduce the applicability of the method undermining the original motivation of developing these tools. If someone hands me a new dataset where there are very few training examples (which is very common given that hand annotating training data is very labor some), do I need to train a transformer from scratch or can I reuse an already trained network?
-
Review #2
- Please describe the contribution of the paper
This work proposes a novel Transformer-based single particle tracking method it estimates the matching probabilties between each live/established tracklet and its multiple hypothesis tracklets simultaneously, and perform global optimization to find the overall best matching for all live tracklets.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
By introducing transformer, the new method can learning more complex and long-term relations can thus can effectively improve the performances. The paper also provide abundant results and movies in the supplementary.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
They did not provide the source code, and some recent works are not included in the comparisons, for example: Graph Neural Network for Cell Tracking in Microscopy Videos, ECCV, 2022
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
They did not provide the source code, which is the main weakness. However, the provided avi give me the confident that the results are reproducible at some extend.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
I suggest the authors to include the new published work for comparison, they should also include the Association Accuracy and Target Effectiveness as the metrics for comparison.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
They provide a novel way for particle tracking, and the results can support the effectiveness for their works.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The paper introduces a novel method called Motion Transformer Tracker (MoTT) for single particle tracking in fluorescence microscopy images. MoTT utilizes the attention mechanism of the Transformer architecture to learn complex particle behaviors from past and hypothetical future tracklets, estimating matching probabilities, existence probabilities, and positions of live tracklets. Global optimization is employed to find the overall best matching for all live tracklets, and a relinking strategy is used to substitute for missed detections in tracklets with high existence probabilities. Experimental results show that MoTT outperforms competing methods on the ISBI Particle Tracking Challenge datasets, making it a powerful and robust tool for studying subcellular structures’ spatiotemporal behavior.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
(1) Motion Transformer Tracker (MoTT) presents a novel formulation to effectively address tracking problems. (2) Single particle tracking in fluorescence microscopy images demonstrates an innovative approach to leveraging data. (3) The robustness of the evaluation is reinforced by comprehensive testing in various scenarios, including high and low density situations, as well as videos of microtubules, vesicles, and receptors.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
(1) It would be beneficial to have a comparison of the proposed model with other commonly used methods in the community, such as Trackpy (http://soft-matter.github.io/trackpy/v0.6.1/), as the main idea of the proposed model involves establishing links between particles and Trackpy does the same and does more with the incorporation of uncertainty estimation and adaptive search for looking back, while also considering the history of feature coordinates in a routine that combines linking and feature-finding. (2) It would be informative to have more comparisons with previous methods like DeepSORT or other approaches that combine YOLO with Kalman filters, etc., to better understand the performance of the proposed method in relation to these existing techniques. (3) It would be interesting to learn how the authors determined the optimal values of parameters such as m and d for tree construction, as these are critical components for generating tracklets. (4) As DeepBlink was used for object segmentation and requires training, it may have limitations in terms of applicability, which could be further discussed in the paper.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Good
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
As mentioned in the weaknesses section, further experiments are needed to validate the effectiveness of the proposed methods. It would be beneficial to not only compare them with different conditions or object types, but also to include comparisons with current and existing methods commonly used in the field, such as Trackpy and DeepSORT. This would provide valuable insights and help establish the superiority of the proposed methods in a more comprehensive manner.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The novelty of the data and the uniqueness of the proposed pipeline are key strengths of this study. The authors have developed a new approach that brings a fresh perspective to the field, which contributes to the originality and innovation of the research.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Strengths: the use of transformer in cell tracking; Weaknesses: need more systematic empirical evaluation, with comparison & analysis of the current SOTA methods; lack of sufficient literature review
Author Feedback
We thank the reviewers for their thoughtful and constructive comments. Here we use Rm.n to mark comment #n raised by reviewer #m, and Am.n to mark our answer/response to Rm.n.
R1.4&R1.5&R2.2&R3.1&R3.2: Need fair comparison with current methods as well as additional metrics. A1.4&A1.5&A2.2&A3.1&A3.2: We have now compared our methods with Trackpy, DeepSORT and Bytetrack and have added performance metrics CLEAR (MOTA, MOTP), Association Accuracy (AA) and Target Effectiveness (TE). For fairness, the same detection results are used for all the methods. Due to space constraints, we only show the results on the microtubule dataset (SNR=4, medium density) of ISBI Particle Tracking Challenge (ISBI PTC). Our method outperforms the other methods in most metrics. method______α__βJSCθ__JSCMOTA_MOTP__AATE trackpy__0.535__0.432__0.667__0.459__0.809__0.847__0.887__0.629 DeepSORT_ 0.5440.478__0.733__0.528__0.691__0.847__0.888__0.635 BYTETRACK__0.555__0.495__0.717__0.552__0.719__0.848__0.788__0.671 ours___ 0.814__0.719__0.760__0.769__0.837__0.846__0.995__0.876
We have also compared our methods with (Ben-Haim, ECCV, 2022) and DeepSORT on the ISBI Cell Tracking Challenge Dataset using the CLEAR performance metric. Here we show the results on the Fluo-N2DH-SIM+02 data. Our method outperforms the other methods. method__ MOTA__MOTP__MODA DeepSORT____ 0.8941.000_ 0.926 (Ben-Haim)__ 0.921__1.000_ 0.947 ours____0.964__1.000_ 0.998
R1.6: The novelty of using Transformer and contributions of the proposed method. A1.6: As pointed out by the reviewer, Vision Transformer has been utilized for pedestrian tracking before. However, our method is based on Text Transformer, which differs substantially from Vision Transformer in data processing and functionality. Vision Transformer focuses on matching appearance cues in tracking and therefore is not suitable for tracking particles because they lack appearance cues. By using Text Transformer, our approach introduces a new idea by treating hypothetical particle tracklets as tokens that will be connected with established tracklets to form complete tracks. Additionally, we have developed an effective relinking strategy to handle disconnected trajectories caused by missed detections. It substantially enhances the robustness of our method.
R1.1: Need more literature review of cell tracking methods. A1.1: We have expanded our literature review. Cell tracking is closely related to particle tracking. There are different classes of cell tracking methods. An important class is tracking-by-evolution, such as (Maška, TIP, 2013; Dufour, TMI, 2011), which assumes spatiotemporal overlap between corresponding cell regions. It is not applicable for tracking particle because they generally do not overlap between frames. Another important class is tracking-by-detection, such as (Wen, Elife, 2021; Nguyen, PLoS Comput. Biol., 2017; Lagache, HAL, 2020), which assumes coherence in motion of adjacent cells. It is not suitable for tracking particles that particles move independently of each other.
R1.3: There is no description on the amount of training data required. R1.3: We have only used the training data provided by the ISBI PTC. No additional training data is used.
R1.2: The computational complexity is not discussed in detail. A1.2: Computational complexity of our method is determined primarily by the construction of the multiple hypothesis tree, which has a depth of two. Its computational complexity is on the order of O(n^2) where n is the number of particles.
R2.1: Source code was not provided. A2.1: As we have written in the abstract of our original manuscript, the code will be made publicly available on GitHub. We did not upload the source code because we thought that the .py format files were not permitted in the supplementary material. We regret that we might have had a misunderstanding of MICCAI submission guideline.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Strengths: the use of transformer in cell tracking, and the results are encouraging;
Weaknesses: yet to make systematic empirical evaluation & comparison wrt the SOTA methods (esp. those mentioned by the reviewers); runtime comparison with SOTA methods; code to be made available
How the rebuttal informed your decision: the authors mention that they will make more thorough literature review, as well as more complete empirical evaluation and analysis with the SOTA methods
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This manuscript introduces a Transformer-based particle tracking method for fluorescence microscopy image data. It also designs a detection relinking strategy to alleviate the effects of missing detections. The experiments show that the method outperforms some other competitors in a public multiple object tracking dataset, i.e., the ISBI 2012 Particle Tracking Challenge. The rebuttal has addressed most of the reviewers’ comments, including adding more literature review, comparison results with recent methods and the technical contributions. All reviewers recommended acceptance of the manuscript.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Since the author response period, the authors have improved the literature review and empirical results, satisfying R1’s concerns. The paper rating has been improved to an accept, followed by my recommendation to present the work as an oral presentation.