Authors

Yi Liu, Su Peng, Jeffrey Caplan, Chandra Kambhamettu

Abstract

Filamentous objects are ubiquitous in biomedical images, and segmenting individual filaments is fundamental for biomedical research. Unlike common objects with well-defined boundaries and centers, filaments are thin, non-rigid, varying in shape, and often densely overlapping. These properties make it extremely challenging to extract individual filaments. This paper proposes a novel approach to extract filamentous objects by transforming an instance segmentation problem into a sequence modeling problem. Our approach first identifies filaments’ tip points, and we segment each instance by tracing them from each tip with a sequential encoder-decoder framework. The proposed method simulates the process of humans extracting filaments: pick a tip and trace the filament. As few datasets contain instance labels of filaments, we first generate synthetic filament datasets for training and evaluation. Then, we collected a dataset of 15 microscopic images of microtubules with instance labels for evaluation. Our proposed method can alleviate the data shortage problem since our proposed model can be trained with synthetic data and achieve state-of-art results when directly evaluated on the microtubule dataset and P. rubescens dataset. We also demonstrate our approaches’ capabilities in extracting short and thick elongated objects by evaluating on the C. elegans dataset. Our method achieves a comparable result compared to the state-of-art method with faster processing time. Our code is available at https://github.com/VimsLab/DRIFT.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43993-3_61

SharedIt: https://rdcu.be/dnwN6

Link to the code repository

https://github.com/VimsLab/DRIFT

Link to the dataset(s)

https://github.com/VimsLab/DRIFT

Reviews

Review #3

Please describe the contribution of the paper

This paper proposes a novel approach for instance segmentation of filamentous objects. It involves creation of a synthetic dataset containing instance labels of filamentous objects . The method is validated on a microtubule dataset and it successfully extracts short and thick elongated objects.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

1) The approach of using human action analogy for extraction of filamentous objects is interesting. 2)The creation of synthetic datasets for training alleviates the training data shortage problem typically associated with filament segmentation tasks. 3) The authors have provided demo videos in supplementary materials showing the operations of their proposed method. 4) The RNN based tracing results as seen in the demo video looks promising.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The results shown in the demo videos are good in tracing filaments that are simple in structure. However, in reality, filamentous objects like microtubules may have higher geometrical complexities and the images could be perturbed by noise and blur. The authors can test their method on such datasets and study the performance of their method.
2. The code for reproducing the paper’s methodology is not found in the supplementary materials.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors mention that the they provide collected data once paper accepted. The coding resources are not found within the provided supplemental files.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

The authors present an approach that mimics human action for extraction of filamentous objects . The concept of using a recurrent network for filament tracing. The creation of synthetic datasets is advantageous as obtaining training data for filament segmentation tasks is difficult. Releasing the synthetic dataset and associated code in the future will be beneficial to the research community. The method is seen to perform well on the provided demo videos , but the use cases show short simple shapes of filaments that are easy to segment. Authors could test some examples with noise or blur effects or longer filaments. Also the metrics (such as AP) used should be defined clearly in the paper or supplementary material rather than citing references of them.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The authors provide demo videos which shows the working of the proposed method in the paper. The performance of the proposed method on more realistic datasets with artifacts like noise could be studied to know more about the efficiency of the method.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #1

Please describe the contribution of the paper

The paper introduces a new method for tracing filaments in microscopy images. The method is based on a recurrent network that picks and traces individual filaments one strand at a time. The performance of this method is demonstrated on a synthetic as well as on three real datasets.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The method introduced by the paper is novel and introduces a new formulation for the filament tracing problem based on sequence modeling with a recurrent neural network. It addresses an important problem in microscopy image analysis where overlapping filament objects have to be segmented, a task for which proposal based segmentation methods are not very well suited. The paper is very well written and easy to understand, the experiments show the effectiveness of the proposed method (though the comparison to baseline methods could be improved, see weaknesses).
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The comparison to baseline methods could be expanded and explained in more detail:
- The main baseline compared to is Liu et al. [17]. From my understanding this method is trained on the same data splits for the results in Table 1. Is this correct? (This is not quite clear from the experiment description).
- How do you use Liu et al. for the results reported in Table 2 and 3? Is this using the method you have retrained yourself on your own synthetic dataset, or is it using the method as published in [17].
- Further baselines for the results in Table 1, 2, 3 would help to better place the contribution, for example the embedding based method of De Brabendere et al. [5], which is in principle suited to solve the problem of instance segmentation with overlaps addressed here.
The dataset contribution is a bit unclear:
- There are no details given on how the synthetic dataset is produced, i.e. what is the algorithm for this.
- Liu et al. [17] have already introduced a synthetic dataset for this problem. How is your dataset different from it and why could you not just use the one from [17]?
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The method is described well enough to be reproduced. The dataset is not described in enough detail. The authors do not provide any code for method or dataset. I would advise to publish these upon acceptance and link to them in the paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

To improve the paper I would improve the description of the baseline experiments and synthetic dataset, and if time allows add a new baseline experiment to evaluate the performance of embedding based instance segmentation from [5]. See weakness for details on these points. Furthermore, I think the limitations of the method should be addressed: another common use-case for filament segmentation in microscopy are branching structures (e.g. neuron segmentation in light microscopy: https://www.nature.com/articles/s41592-023-01848-5). The current formulation of the method is not applicable to such problems because it cannot deal with branching points. It would be good to mention this limitation and discuss potential approaches to extend the method to branching filament structures. To be clear, I think it is totally valid to propose a method with this limitation, but I think it would be good to address it head on and also think about how to mitigate it since the potential extension to branching structures would make the contribution more impactful. Finally a minor comment: It seems like you are using BatchNorm and weight decay (= L2 regularization). Common wisdom is that this is not a good design choice, see for example https://arxiv.org/abs/1706.05350. Did you mitigate the effects from this somehow or did this turn out not to be an issue in training your model in practice?
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This is a very well written and easy to follow paper that describes a novel method for an important problem in microscopy image analysis. Most of the weaknesses in the experimental evaluation and description should be easy to address in the rebuttal (see above for details).
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

The authors propose a new method (Deep Recurrent Instance Filament Tracer (DRIFT)), for instance segmentation of filament-like structures in microscopy images. The method identifies the starting points of every filament in the field of view and from each point, delineates dynamically the filaments using a sequential encoder-decoder.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Instance segmentation of filamentous like structures is one the big challenges in microscopy image analysis, which haven’t been resolved yet. Enabling the automatic analysis of such geometries would enable further research involving, among many others, collagen structures, microtubules or actin which are incredibly important for further understanding cell migration, molecular biology, mechanotransduction or tissue mechanics.

The proposal made by the authors has shown to be quite accurate and feasible to reproduce with new data**. Morover, the authors prove the potential of the method on different datasets (simulated and real) and compare their performance with other existing algorithms.

** The authors stated that data and code will be made available with the acceptance of the manuscript.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

In terms of computation, seems that in inference, it is not possible to feed the network with batches. Is this one correct?

Is there a reason why the authors decided to use BCE instead of L1 or L2 for the tip prediction?

The current method requires a preliminary segmentation. Would there be any way to skip this step to avoid error propagation?

The evaluation of the method has been done using the average precision from COCO. While this is not wrong, it is highly sensitive to the pixel resolution and the size of the structures. I would recommend the authors providing the distribution of infered lengths and comparing it with the real ones.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper is well written and the method explained in detail, which facilitates the reproducibility of the approach. Additionally, the authors stated their will to provide the data and the code as soon as the paper is accepted. On the other hand, the proposed method seems easily transferable across specimens and biological structures as long as they are filaments with a starting and an ending point.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

Please cite Hough Voting method.

In section 2.3, when the authors explain how training data was generated, it is not clear how each of the patches or “the next patch” is computed for each filament.

When generating the final instances, how are the authors solving the problem of fibers overlapping each other?

When describing the synthetic data, it would be convenient to provide also the ration between the microtubules’ diameter and the pixel.

Section 3.2., the authors say “As the average width of microtubules is five”, five pixels? I see that the performance of the method can vary according to the input shape chosen, the ratio between the structure’s shape and the pixel size, and even the fiber density. It would be convenient if the authors could comment on it or provide some technical recommendations to consider when reusing the method.

As stated in the previous lines, this reviewer is quite confident of the positive impact that this work can have and on the highly transferable/reproducible potential. Thus, if the authors would like their method to be exploited in real applications, I would highly recommend them to provide a notebook together with the code in which medium experts or bioimage analysts can retrain and use their approach.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

8
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

As stated before, segmentation of filamentous structres is one the challenges currently in bioimage data analysis. Accessibility to accurate and reproducible methods is very much needed. While one could argue about the technical novelty of the neural network architecture, it gathers a creative solution that can impact positively the field of microscopy data analysis and become an inspiration for more sophisticated approaches.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #4

Please describe the contribution of the paper

This work proposed a deep learning based method for filamentous detection. The start points are detected and a tracking model is applied for line tracing detection. According to the experiment, the proposed model achieved better results and running speed.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This work proposed a filamentous detection model, and achieved the state-of-the-art performance. The method is concise and easy to understand, and it is evaluated on several datasets.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Hyperlinks are missing in this pdf document. Line 3 in the introduction, missing the word after “and”. A large amount of typos are found in this paper. To be acceptable, it should be further checked. Detailed analysis is missing. For example, how each component influences the final performance, and what is this model good at and not.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

It should be able to reproduce, but it is hard if the codes and datasets are not accessible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
This work is well-done, but I have several questions beyond the previous comments.
- Why an LSTM is applied but not a basic RNN or a powerful transformer.
- The tracking is step-by-step, how to make sure it would not stop at a corner of a right-angled.
- In Fig.2, the right-up part of the input image, how this model distinguishes a ‘k’ style cross.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The writing is poor, but the method is simple but effective. The performance is higher than several previous methods. Even though several experiments are done but lack detailed analysis.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

strengths: a new way for the filament tracing with more accurate performance; important problem with good empirical results; own syn. dataset; demo video weaknesses: comparison methods and more detailed analyses; unclear how to deal with complex filament structures;

Author Feedback

We appreciate the reviewers’ feedback and constructive comments on our paper. We will address reviewers’ comments point by point as below.

Reviewer 1: 1.1 Experiments-related questions: 1.1.1 The methodology of [17] is different from our proposed method. [17]’s model aims to sort the filaments into different bins at intersections, and the synthetic training data in [17] contains six channels for each direction. Therefore our synthetic instance label can’t be used in [17] for training. However, our synthetic filaments are similar to [17], with similar curvatures and thicknesses, so we can use our dataset to evaluate the method fairly.
1.1.2. Results in Table 2a are replicated from [17]; the performance matches the results reported in [17]; we use the AP metric to keep the reporting results consistent with other experiments. Table 2b uses the method as published in [17]. 1.2 Data set contribution: 1.2.1 We will publish synthetic dataset generation code. It is a random curve generation algorithm, and we combine different curves into images to form instance labels. 1.2.2 The synthetic dataset in [17] is entirely different from ours; in [17], the labels are six-channel images, and each channel contains filament with a certain direction. 1.2.3 For Y-shape junctions, it would be challenging to determine which way to go next with binary input and little domain knowledge. We will address these as limitations and future steps.

Reviewer 2: 2.1 Batch inference implementation would be our future research steps. 2.2 We use binary segmentation to get a rough area of the tip where BCE loss is used; then, hough voting is used to get the precise location. 2.3 We can directly train our model on raw images without preliminary segmentation if there are instance labels. The advantage of using segmentation is that researchers don’t need to annotate each filament for training, which is costly and laborious. 2.4 Thank reviewer 2 for the constructive comments. We will make the edits accordingly. 2.4.1 Sequenced patches are generated by cropping around the sequenced tracing points. 2.4.2 If two fibers overlap entirely, it would be difficult to tell them apart.

Reviewer 3 3.1 Our collected data, and synthetic data generation code will be released. 3.2 Due to lack of instance-labeled microtubule data, we could not directly train our model on the microtubule dataset. Instead, we take a two-step approach, where we first segment the binary mask of microtubules and then use our methods to extract individual filaments by tracing them. In this case, image noise or blur effects are handled during the binary segmentation stage.

Reviewer 4: 4.1 The key contribution of this work is transforming an instance segmentation problem into a sequence modeling problem. We adopt LSTM due to its advantage in dealing with longer sequences compared to basic RNN. In our approach, we model each filament as a sequence of patches, leveraging encoded embeddings features at each patch to aid in predicting the subsequent points. By utilizing LSTM, the model can retain and utilize information from previous steps to forecast the location of the next patch. This design fundamentally differs from popular vision transformers like ViT, which divide an entire image into fixed patches. However, we could extract each patch sequence as a training sample and employ a transformer to learn the “sequence” features. Nonetheless, this modification would require substantial alterations to our current method and could lead to a substantially different study. 4.2 Essentially our model is learning the propagation features of curves. The model’s stop flag should learn when to stop if many filaments show certain features (such as right-angled). 4.3 Thank reviewer 4 for the comments; we will fix the typos and add more details to our experiment analysis.

back to top

Pick and Trace: Instance Segmentation for Filamentous Objects with a Recurrent Neural Network