Authors

David Wiesner, Julian Suk, Sven Dummer, David Svoboda, Jelmer M. Wolterink

Abstract

Methods allowing the synthesis of realistic cell shapes could help generate training data sets to improve cell tracking and segmentation in biomedical images. Deep generative models for cell shape synthesis require a light-weight and flexible representation of the cell shape. However, commonly used voxel-based representations are unsuitable for high-resolution shape synthesis, and polygon meshes have limitations when modeling topology changes such as cell growth or mitosis. In this work, we propose to use level sets of signed distance functions (SDFs) to represent cell shapes. We optimize a neural network as an implicit neural representation of the SDF value at any point in a 3D+time domain. The model is conditioned on a latent code, thus allowing the synthesis of new and unseen shape sequences. We validate our approach quantitatively and qualitatively on C. elegans cells that grow and divide, and lung cancer cells with growing complex filopodial protrusions. Our results show that shape descriptors of synthetic cells resemble those of real cells, and that our model is able to generate topologically plausible sequences of complex cell shapes in 3D+time.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16440-8_6

SharedIt: https://rdcu.be/cVRvq

Link to the code repository

https://cbia.fi.muni.cz/research/simulations/implicit_shapes.html

Link to the dataset(s)

https://cbia.fi.muni.cz/research/simulations/implicit_shapes.html

Reviews

Review #1

Please describe the contribution of the paper
1. The paper proposes a new cell shape representation method.
2. The paper proposes a deep learning framework for spatio-temporal cell representation and synthesis.
3. The proposed method allows for shape synthesis with virtually unlimited spatial and temporal resolution.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

the work proposes a novel and effective cell shape representation method which is able to generate sequences of cell shapes in 3D+time
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The references are cited randomly in a disturbed order.
2. The best results in Table 1 are not annotated.
3. Section 2 only describe the proposed method.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The work could be reproduced
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1. The references are cited randomly in a disturbed order.
2. The best results in Table 1 are not annotated.
3. Section 2 only describe the proposed method.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The proposed method is novel and effective in spatio-temporal cell representation ad synthesis.
Number of papers in your stack

5
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

The paper addresses the use-case of generating synthetic datasets of cell shapes. It trains a NN to generate sequences of cell shapes from a signed distance function.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This paper is creative and interesting. It is well-written and a pleasure to read. I regret that due to the hour of the day I will not do it justice in these comments.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

No non-trivial weaknesses evident to me.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper indicates that source code will be available.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

Discussion: it would be good to hear about the dynamics of new protrusions in the cancer cell case. This looks like a bifurcation or other non-linear state change. How does the model generate this effect?

Tables: uncertainty measures are good, thank you. I wonder if in the cancer cell case there is a better measure than sphericity, one that quantifies the existence and characteristics of protrusions.

(2.1): Re sigma in L_code: is this sigma related to the covariance matrix of the multivariate gaussian? If yes, how is it extracted from the covariance matrix? If not, a different notation would be clearer, since sigma routinely refers to std dev. Does it correspond to the std dev of the initializations?

(2.2): Did the sine activation work any better than ReLU? Is the “low frequency bias” of ReLU an issue in this context?

(2.3): “70%”: Is 100% preferable? Was 70% chosen solely based on particular RAM restrictions? Does 3 GB max out the GPU’s memory?
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper is well-written, the method is creative, and the results are compelling.
Number of papers in your stack

5
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

This paper proposes a approach to segmentation of living cells in 3d+t using a generative neural architecture. The paper is based on a level-set represented with a MLP that learns an implicit shape model from annotated images and is able to generate accurate boundary models of cells in fluorescence images.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The method is simple and appears to be quite efficient with quite modest memory requirements. The integration of time as equivalent to the three spatial dimensions is important. The paper is written extremely well. The model seems to be able to cope well with anisotropic resolution (ec C.elegans dataset).
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

There are no comparisons with other methods, therefore making it difficult to assess performance. The integration of time by normalisation of all coordinates onto [-1,1] is practical, but seems a little unsatisfactory as it does not fundamentally address the different scales in space and time (eg whether the processes are fast or slow). The same applies to the spatial dimensions. Some of the design choices are not well explained, eg section 2.2 why are the code vectors inserted into laters 1,5,8. I did not find the synthetic textures in figure 4 to be very convincing because they treat the segmentation mask as a hard boundary and the behaviour near to the edge does not appear to be very realistic.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Sufficient information has been provided to allow the paper to be reproduced. The data appears to already be in the public domain and the methods are sufficiently well-described to allow me to believe that I could implement this method from the information provided.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

It is unlikely that it will be possible for a more thorough evaluation to be completed against other methods during the rebuttal period but I would encourage the authors to consider this for any further journal paper. I would suggest considering at minimum whether this approach performs well against a classical level set approach, and how suitable the generated samples are for downstream tasks.

Further discussion of how architectural choices were made should also be provided (section 2.2). What process was followed to arrive at this architecture?
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper makes a relatively modest technical contribution and the evaluation could be stronger. These factors are balanced against the inclusion of time as a dimension, which is an interesting contribution.
Number of papers in your stack

4
What is the ranking of this paper in your review stack?

2
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper presents a cell shape representation method based on deep learning which is able to generate sequences of cell shapes in 3D+time. A well written, interesting and innovative paper
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

1

Author Feedback

We are very thankful to the reviewers for their supportive comments and constructive criticism. Unfortunately, we had to skip replies to some of the comments due to the limited length of the response.

Reviewer 1

The references are cited randomly in a disturbed order.

The best results in Table 1 are not annotated.

Section 2 only describe the proposed method.

RE-1. The references in the submitted version were sorted alphabetically. We will sort the references in the order of appearance in the revised paper to improve readability. RE-2. Table 1 lists means and standard deviations of shape descriptors computed on real and synthetic data and shows their similarity. Consequently, there is no best result to be highlighted in this case. RE-3. We agree with this remark and will change the name of the respective section to better reflect its contents.

Reviewer 2
1. Discussion: it would be good to hear about the dynamics of new protrusions in the cancer cell case. This looks like a bifurcation or other non-linear state change. How does the model generate this effect?
RE-1. The inherent feature of the proposed model is that we do not need to input any apriori knowledge about the modeled specimen and its growth dynamics. This is a different approach to parametric models, where the specimen behavior is defined using prior biological knowledge.
2. I wonder if in the cancer cell case there is a better measure than sphericity, one that quantifies the existence and characteristics of protrusions.
RE-2. We appreciate this suggestion and will focus on more extensive evaluation in our future work. The Hausdorff distance could be used for comparison of whole shapes. For characterizing the filopodia specifically, the commonly used metrics include traveled distance, angle of growth, length, radius, growth rate, and branching rate.
3. (2.1): Re sigma in L_code: is this sigma related to the covariance matrix of the multivariate gaussian? If yes, how is it extracted from the covariance matrix? If not, a different notation would be clearer, since sigma routinely refers to std dev. Does it correspond to the std dev of the initializations?
RE-3. We concur that this information was not well explained in the paper and we will revise it accordingly. We assume the prior distribution over latent codes p(z_i) to be a zero-mean multivariate-Gaussian with a spherical covariance sigma^2 I (I stands for matrix diagonal). This sigma is used for code initialization and for the regularization in L_code during the training procedure.

Reviewer 3
1. I did not find the synthetic textures in figure 4 to be very convincing because they treat the segmentation mask as a hard boundary and the behaviour near to the edge does not appear to be very realistic.
RE-1. We acknowledge that the method used for generating the textures can be further enhanced. As the focus of this paper was generating the shape sequences, we intended this to be simply a demonstration of the possible application of the method. However, the synthesis of plausible looking time-developing textures is among our future research goals.
2. Some of the design choices are not well explained, eg section 2.2 why are the code vectors inserted into layers 1,5,8. Further discussion of how architectural choices were made should also be provided (section 2.2). What process was followed to arrive at this architecture?
RE-2. We recognize that the paper would benefit from more detailed explanation of the architectural choices and will include additional information in the revised version. The architectural decisions were motivated by the work of Park et al. (DeepSDF), where the authors found that inserting the latent vector again in the middle layers significantly improves the learning. Our model exhibits similar behavior, where the network would not converge on long spatio-temporal sequences without this additional inserted information.

back to top

Implicit Neural Representations for Generative Modeling of Living Cell Shapes