Authors

John Kalkhof, Anirban Mukhopadhyay

Abstract

Medical image segmentation relies heavily on large-scale deep learning models, such as UNet-based architectures. However, the real-world utility of such models is limited by their high computational requirements, which makes them impractical for resource-constrained environments such as primary care facilities and conflict zones. Furthermore, shifts in the imaging domain can render these models ineffective and even compromise patient safety if such errors go undetected. To address these challenges, we propose M3D-NCA, a novel methodology that leverages Neural Cellular Automata (NCA) segmentation for 3D medical images using n-level patchification. Moreover, we exploit the variance in M3D-NCA to develop a novel quality metric which can automatically detect errors in the segmentation process of NCAs. M3D-NCA outperforms the two magnitudes larger UNet models in hippocampus and prostate segmentation by 2% Dice and can be run on a Raspberry Pi 4 Model B (2GB RAM). This highlights the potential of M3D-NCA as an effective and efficient alternative for medical image segmentation in resource-constrained environments.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43898-1_17

SharedIt: https://rdcu.be/dnwAO

Link to the code repository

github.com/MECLabTUDA/M3D-NCA

Link to the dataset(s)

medicaldecathlon.com

Reviews

Review #4

Please describe the contribution of the paper

The paper propose the M3D-NCA, a novel method based on Neural Cellular Automata for 3D medical image segmentations. M3D-NCA achieves comparable performance compared to state-of-the-art neural networks with significantly less parameters, which is effective and efficient and can provide build-in quality control.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The paper is generally well-written and easy to follow.
2. The paper propose a novel algorithm for using Neural Cellular Automata in 3D medical image segmentation, with the n-level architecture design and batch duplication scheme that reduces memory usage and stabilizes the training processes.
3. Extenstive experiments demonstrates the effectiveness and efficiency of the proposed method. In addition, M3D-NCA can be run on a Raspberry Pi 4 Model B, which can be used for improving healthcare in resource-constrained environments.
4. M3D-NCA also provides automatic quality control by computing the variance of multiple inference outputs, which is important for clinical deployment.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Paper writing: a more detailed introduction on neural cellular automata can help readers to understand this paper easier.
2. In Eq. (1), the authors chose N=10 without justification. Have the authors tried other N values?
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

This paper uses public datasets and the authors agree to make the codes and trained model public upon acceptance, which will make this work easy to reproduce.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

The authors are encouraged to address the comments in the weakness section.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper proposes an efficient and effective algorithm for 3D medical image segmentation. I think this work is solid and ready to be published.
Reviewer confidence

Somewhat confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #1

Please describe the contribution of the paper

This paper describes a light-weight segmentation tool using neural cellular automata. The method can run with a very limited memory foot print and the results are convincing.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The main strengths of the paper is the implementation and testing of a novel approach to segmentation that can run on limited hardware resources. The method compares favorably with current state-of-the-art methods that relies on much larger GPUs.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The paper could have been tested on a larger dataset.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

I do believe that it is possible to reproduce the methods and the results based on the paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

In general, it is a well written paper describing an interesting take on low-resource medical image segmentation.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The method and the paper is interesting and well written. It could have been tested on a larger dataset.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

The authors present a very promissing learning-based approach for medical image segmentation based on neural cellular automata and evaluate their approach on two publicly available datasets for hippocampus and prostate segmentation, showing that their method can achieve good accuracy compared to the state-of-the-art while requiring orders of magnitute fewer learnable parameters.

Although some preliminary work exists in the area of neural cellular automata, this paper provides further evidence for the usefulness of the approach for medical image segmentation. It also further generalizes previous work on patch-wise training to scale the method to larger 3D volumes.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The paper is well written and well structured. It also provides relevant references to related neural cellular automata work in case where the description in the paper is kept very brief.

The authors present a very novel approach for medical image segmentation based on neural cellular automata. Similar to the very popular U-Net-like architectures, the method defines a parametric model where the parameters are learned during training. Unlike the U-net, the model itself only contains very few layers and only one convolutional layer. This is in contrast to current research that typically shows that larger and deeper models outperform smaller models. To retain a large receptive field and to compute more abstract and non-linear representations, same model is applied multiple times to the ouput of a previous run, thereby forming a hierarchy of inference layers that all share the same weights, similar to the unrolling operation of a recurrent neural network. Consequently, the network as many orders of magnitute fewer parameters than U-net-like architectures, while still maintaining a large receptive field. (typically, the size of the receptive field increases with the number of layers, which also increases the number of parameters. The subsequent application of the same network also increases the receptive field without increasing the number of parameters).

The authors present an extension of previous work to use a hierarchy of cell arrays coresponding to different resolution levels, which further increases the receptive field and allowing high and low-level features to be learned. They combine it with a novel patch-based learning scheme, which, unlike U-net-like patch-based training, does not limit the receptive field, because coarser layers corresponding to larger receptive fields are trained on larger (in physical dimensions) patches then fine layers.

The stochastic nature of NCAs allow the calculation of quality metrics similar to Monte Carlo dropout-based methods. The authors show that the quality metrics are suitable to detect a degradation of segmentation quality due to artifically introduced imaging artifacts, which gives some evidence for the usefulness of the quality metrics to detect artifact-induced segmentation errors.

The evaluation is clear and understandable. The method was evaluated on a publicly available dataset and compared against publicly available methods, with clear references to the corresponding implementations of the methods and source code that was used. Also the choice of methods it was compared against seems reasonable and representative of the current state-of-the-art.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The description of the method is limited to some main aspects of the work, but some details about the implementation of NCAs are omitted. E.g., it is mentioned that NCAs are stochastic but the description of the method omits the reset gates described in the referenced literature, which makes the model stochastic. All missing details can be found in the referenced literature.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The network architecture and training parameters are not all provided in detail. The number of parameters was mentioned in the paper, but I was not able to derive the same number of the description of the paper. E.g., the 2 level architecture was described as having 12480 parameters, but I could only identify 12064 from the description in the paper. A description of the architecture in a table with the exact size of the input, output and parameters tensors of each layer would be helpful. Other missing information are how weights are initalized, how the first state of the cell array is initalized, how sampling of the patches are performed and many more. However, the complexity of learning-based methods in general make it almost impossible to provide all the details without losing focus of the main contribution of the paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- I didn’t find figure 1 useful or needed ot understand the paper. It’s merely a visual representation of the claims but gives no information about how it works. Could be omitted.
- It is often stated that the number of parameters is a limitting factor of U-net-based approachs for low-cost hardware. However, I would argue that it is not the number of parameters. It’s temporary memory requirements (number of hidden units, neurons) and number of computations (e.g., multiplications). A simply matrix multiplication in dense layers might use a lot more parameters than a convolution operations, albeit requiring much less memory and being faster.
- Since efficiency was mentioned a couble of times, how does the method compare to U-net-like method (e.g., nnU-Net) in terms of temporary memory requirements and runtime? It is mentioned that training is very expensive. Having a comparison of memory and runtime would strenthen the claim.
- Figure 2 would be easier to understand when illustrated in 2D with proper dimensions added to the different results of the computations.
- “Once the model is trained, inference can be performed directly on the fullscale image. This is possible due to the one-cell architecture of NCAs, which allows them to be replicated across any image size, even after training.” Doesn’t the model still require at least x x 4 bytes of memory? With large CT scans of 512x512x700 and h = 64, this would require at least in the order of 40GB for inference unless the input is divided into patches. How would the patch calculation be done during inference?
- Use 1x1 convolutional layer instead of dense layer. Dense layer is sometimes used to describe a layer of a multi-layer perceptron (e.g., just a matrix-vector multiplication).
- Clarify if s is recalculated per resolution level or of the equation in 2.2 is with respect to the original image dimensions and the same s is used for all levels.
- Why does the 4 level model has fewer parameters than the 3 level model? Does it use different values for c and h?
- “Increasing the number of layers for larger datasets is a trade-off with each additional layer reducing segmentation performance by 2.7-5.5%.” Can you explain what the trade-off is? It seems that adding more layers reduces accuracy and increases the number of parameters. It also potentially increases memory requirements for training and inference time, so it seems to have only negative impact.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I think the presented method could be a real alternative for U-Net and transformer-based approaches and the MICCAI community would be very interested in learning about this class of approaches. It is not necessarily noval compared to related work. It does provide some incremental improvements, which can be seen as small compared to Med-NCA. However, Med-NCA is does not seem to be a peer-reviewed publication, so I didn’t use it in my judgement of novalty. Compared to Seg-NCA it shows how to scale it to 3D problems, it shows the potential for medical image segmentation and might inspire new directions for research in a field that is very dominated by U-Net and transformer-like architectures.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper proposes a novel and effective algorithm for lightweight 3D medical image segmentation. It is well-written and interesting and the evaluation is clear and convincing. Please consider taking care of the minor questions/points of the reviewers (especially R2) for the camera-ready version.

Author Feedback

We sincerely thank you for the positive feedback and high ranking. We will incorporate the minor changes suggested by the reviewers into the final manuscript.

back to top

M3D-NCA: Robust 3D Segmentation with Built-in Quality Control