Authors

Ziquan Wei, Tingting Dan, Jiaqi Ding, Mustafa Dere, Guorong Wu

Abstract

High-throughput 3D nuclei instance segmentation (NIS) is critical to understanding the complex structure and function of individual cells and their interactions within the larger tissue environment in the brain. Despite the significant progress in achieving accurate NIS within small image stacks using cutting-edge machine learning techniques, there has been a lack of effort to extend this approach towards whole-brain NIS. This critical area of research has been largely overlooked, despite its importance in the neuroscience field. To address this challenge, we propose an efficient deep stitching neural network built upon a knowledge graph model characterizing 3D contextual relationships between nuclei. Our deep stitching model is designed to be agnostic, enabling existing limited methods (optimized for image stack only) to overcome the challenges of whole-brain NIS, particularly in addressing the issue of inter- and intra-slice gaps. We have evaluated the NIS accuracy on top of three state-of-the-art deep models with $128\times 128\times 64$ image stacks, and visualized results in both inter- and intra-slice gaps of whole brain. With resolved gap issues, our deep stitching model enables the whole-brain NIS (gigapixel-level) on entry-level GPU servers within 27 hours.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43901-8_5

SharedIt: https://rdcu.be/dnwCF

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

Submission 286 addresses the problem of stitching artefacts which arise when large volumetric datasets are segmented block-wise. The proposed solution directly stitches intermediate instance segmentation results rather than stitching probability maps. The stitching in-plane happens by the usual overlap approach, while the stitching across Z is performed by training a graph neural network to predict correspondences between nuclei detected in individual blocks.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Strengths:
- the paper addresses an important problem in large volume analysis
- as far as I can judge, the approach of using a GNN for such matching is novel and elegantly simple
- the method will likely be applicable to instance segmentation of other objects
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Weaknesses:
- nothing major, see section 9 for more detailed suggestions
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

looks ok, everything made available
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Questions:
- My biggest question is why do the authors choose to work in 2D? It has been shown multiple times that 3D segmentation is better and more reliable. The algorithm is based on graphs, so it shouldn’t make any difference how the original detection is done and the graph problem would also get smaller.
- Supplementary Fig. 1, what are the diagonal lines, are these the remaining blocking artefacts? Are they caused by sub-sampling?
- Supplementary Fig. 2, I don’t see any “obvious issues” without the ground truth (unlike the diagonal lines in Fig. 1 which you don’t comment on). Can you zoom in or mark them in a more obvious way?
Suggestions:
- Cite previous work on 3D segmentation as learned stitching [Funke et al, Efficient automatic 3D reconstruction of branching neutrons from EM data, CVPR 2012] and blockwise instance segmentation merging through optimisation on graphs [Pape et al, Solving large multicut problems for connectomics via domain decomposition, ICCV Workshops, 2017].
- State the modality you are working on in the abstract
- Remove “alternatively” from the 2nd sentence, tissue clearing is not an alternative to light-sheet microscopy.
- Fig. 1 caption: Top right is actually top left?
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The authors address an important problem that arises not only at the whole brain scale, but whenever an experimentalist needs to image more than a toy benchmark dataset. While stitching through graph optimization has been proposed previously, that was before the age of deep learning. It’s good to see a modern and efficient solution to this old problem.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

This manuscript presents a method for instance nuclei segmentation from large microscopy images. In general, the methodology is clearly presented and the validation is simple yet convincing.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

• Interesting training-based approach for finding corresponding nuclei instances in the overlapping images • Convincing validation
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

• Presentation of the results needs improvement
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

This work is reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
1. The authors present an interesting training-based method for detecting the same nucleus instance in the overlapping images, which proves to be efficient when applied to results obtained by several advanced segmentation methods.
2. I find Figure 4 too small. However, the results presented in it are confusing as the images before and stitching differ not only in the stitched area. For example, on the image from the second row, top-right corner, second cell when counting from the right is segmented before but not segmented after. There also a few similar cases on the other images. This needs to be explained.
3. Page 2: “However, fusing categorical indexes of NIS results in the complex arrangements of nuclei in close proximity poses a challenge for achieving whole-brain NIS.” I was not able to understand this sentence as it does not make much sense in its current form; please rewrite.
4. Page 4: it is not clear why the set of nodes V consists of the points located further than the predefined threshold; I would expect otherwise.
5. Page 7: it is not clear to me how many whole-brain images where used for validation.
6. Supplementary material is somewhat loose from the main manuscript as the latter does not contain any references to the former. Moreover, “obvious intra-slice gaps” mentioned by the authors are quite difficult to spot on the Figure S2.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This is a good conference paper with a simple yet convincing validation.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

This paper a graph learning approach for stitching segmentations of nuclei in whole brain lightsheet data. This is a huge problem in current day lighthsheet image processing and therefore I think this problem is very interesting.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Novel formulation for using existing state of hte art methods of segmentation on scaled up datasets
- Clearly written
- Experimental validation done with state of the art methods and good measurements
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Some parts of the paper are densely written and figures could help there such as section 2.2
- Whole brain experiments with partly annotated data on multiple brains would provide a better picture of generalizabilty and complete the paper
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper is reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- Figure 2 may not be very clear for someone who is not familiar with the problem. I would suggest separating the bottom from the top and having more detailed explanations of the method
- Section 2.2 is a little densely written. A picture outlining the graph neural net with message passing, etc will make that easier to read.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Scaling systems to perform segmentation on whole brains is a very relevant topic today in neuroscience and biological imaging. With image resolutions getting larger in all fields, this can have high impact even in other medical systems. I think this paper does a good job of dealing with this problem and can have immediate high impact in the realm of microscopy.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposed a novel solution for 3D nuclei instance segmentation in microscopy images. Overall this is a well-written paper and the proposed graph-learning-based method has technical novelty, and the experimental results are promising. All reviewers gave positive ratings and recommended acceptance of this paper. However, some minor concerns have also been raised, such as improving the quality of some figures, and including more details on some illustrations. These need be addressed in the final version.

Author Feedback

Major questions: ** Reviewer #1: Why the authors choose to work in 2D instead of 3D, considering the known advantages of 3D segmentation. The proposed approach can indeed work for both 2D and 3D segmentation. However, we haven’t run in 3D due to two reasons. Firstly, the anisotropic resolution in the Z-dimension of the 3D lightsheet microscopy image makes manual labeling of nuclei instances in the X-Y plane more practical. This leads to the training of a 3D model that is not compatible with images from different resolution in Z-dimension. Secondly, there is currently no 3D method that can handle this anisotropic issue. The proposed stitching solution addresses this problem by using the 2D output of the deep learning model to recover the 3D instance segmentation. Overall, the proposed stitching solution is motivated by such anisotropic resolution challenge in 3D lightsheet microscopy.

** Reviewer #2: Confusion regarding Figure 4, there are discrepancies that need to be explained in the segmented areas before and after stitching. The presence of a design limitation in Convolutional Neural Networks (CNNs) is responsible for this situation. Despite the translation invariance property of CNNs, the edge portion of the image is theoretically not invariant due to the use of zero-padding in modern CNN models. In our particular case, the outputs differ when applying our approach, which involves a subtle motion for input images to let our graph model can stitch nuclei beyond the patch image. But for the records, this difference is so slight that researchers in the computer vision field barely mentioned it. Our stitching solution can’t address this issue for now, since it is caused by the RCNN-based model. RCNN sort each instance by one score, which leads to the absence/presence of nuclei instance being very sensitive. By our further ablation experiments, the Unet-based model barely has such downsides by the edge issue. But there was no space to put them on the paper.

Others:

Reviewer #1: Questioning Fig. S1’s diagonal lines. Fig. S2 needs to zoom-in The visual bias observed in Fig. S1 is caused by sub-sampling for visualization purposes. In the final version, we will add several zoom-in areas in Fig. S2 and stating the modality in Abstract will be fulfilled. Thanks for providing previous works about stitching method and graph based instance segmentation. They will be cited. About Fig. 1 caption, we will revise the sentence starting with “Furthermore”, more clearly. It explains the “Top left” part.

Reviewer #2: the number of whole-brain images used for validation. There is one whole brain volume for validation, which consists of 89,424 small chunk images (128x128x64). These are unlabeled. There is also another labeled set of 16 chunk images to validate our method.

Reviewer #2 confused with one unclear sentence We rewrite it more clearly as “However, when nuclei are in close proximity and represent the same entity, it becomes crucial to accurately match the indexes of nuclei instances, which refer to the segmentation labels. We call this the nuclei stitching issue. This issue presents a significant challenge in the pursuit of achieving whole-brain NIS”

Reviewer #3 raises concerns about the organization and clarity of Figure 2, as well as Section 2.2 is densely written, and the multiple whole-brain results. We agree with you to split the two stages of Figure 2 into separate figures with more detailed illustrations. This change will address the concern about Section 2.2 is densely written. We have made a pipeline for multiple whole brain computation. We only show one whole brain result due to space constraints. The multiple whole brain results will be available in a future journal paper.

back to top

A General Stitching Solution for Whole-Brain 3D Nuclei Instance Segmentation from Microscopy Images