List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Dominik J. E. Waibel, Scott Atwell, Matthias Meier, Carsten Marr, Bastian Rieck
Abstract
Reconstructing 3D objects from 2D images is both challenging for our brains and machine learning algorithms. To support this spatial reasoning task, contextual information about the overall shape of an object is critical. However, such information is not captured by established loss terms (e.g. Dice loss). We propose to complement geometrical shape information by including multi-scale topological features, such as connected components, cycles, and voids, in the reconstruction loss. Our method uses cubical complexes to calculate topological features of 3D volume data and employs an optimal transport distance to guide the reconstruction process. This topology-aware loss is fully differentiable, computationally efficient, and can be added to any neural network. We demonstrate the utility of our loss by incorporating it into SHAPR, a model for predicting the 3D cell shape of individual cells based on 2D microscopy images. Using a hybrid loss that leverages both geometrical and topological information of single objects to assess their shape, we find that topological information substantially improves the quality of reconstructions, thus highlighting its ability to extract more relevant features from image datasets.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16440-8_15
SharedIt: https://rdcu.be/cVRvF
Link to the code repository
https://github.com/marrlab/SHAPR_torch
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper uses persistent homology, a popular approach in topological machine learning, to design a loss function for 3D reconstruction of 2D image slices. The proposed loss function is suitable for neural networks (fully differentiable), and assesses the topological persistence (via persistence diagrams of image cubical complexes) of a predicted likelihood function measuring the probability that image voxels are part of the 3D object’s shape. This multi-scale loss incorporates novel shape information to help improve reconstruction, and is compared to other reconstruction method loss functions on cell shape prediction from 2D microscopy images.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper concisely explains topological data analysis and persistent homology in a clear way, and makes the reader aware that not only is this loss function more suitable for reconstruction, but it can be implemented in pre-existing modeling architectures due to its desirable properties. TDA methods are also stable under noise perturbations, which is useful for image data.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Improvement seems fairly marginal for the nuclei dataset, as compared to red blood cells, and there is not much explanation for why this may be the case. This would be relevant, as it is discussed that computation can be a bottleneck (though speed-ups are proposed) in optimization with respect to this topological loss function, and so providing suggestions, based on dataset, on when this extra computation may not be “worthwhile” would be helpful for a practitioner.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Code does not appear to be provided, though the authors do use a pre-existing deep learning method for 3D image reconstruction with its corresponding datasets which have been made publicly available. Some of the TDA methods (computing persistent homology, along with the optimal transport algorithm) would be difficult to reproduce from scratch.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
It would be interesting to see how things might compare to other persistent homology representations (e.g., persistence landscapes), or even simpler topological descriptions.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This contribution could be deemed innovative, and incorporating topological/shape information into reconstruction appears valuable, but would need to do a more exhaustive comparison and discuss implementation/computation further.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This paper introduces a topological loss into a deep neural network for 3d reconstruction from 2d images. The purpose of introducing this additional loss term is to capture multi scale information about the general shape of an object in addition to the more standard geometric information. The new loss is fully differentiable and is added to the SHAPR model for image reconstruction. It is shown that this improves the accuracy of image reconstruction.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The topological loss, formulated as the Wasserstein distance between persistence diagrams plus the total persistence, is well justified. That it is also differentiable makes it especially valuable. This is a potentially significant advance for this field that provides a general mechanism for the inclusion of topological information which has been neglected by the field.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The construction of the cubical complex is not adequately explained. What do the nodes of the complex correspond to?
The description of topology and persistent homology may not be sufficient for the majority of readers who are likely to be unfamiliar with the approach.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
I do not think I would be able to reproduce this paper from the information provided. Some key details are omitted that would be needed to reproduce the work. There are no statements about code availability.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Please provide full details about the construction of the cubical complex from the image data. The choice of cubical complex should be explained (I assume this is to align with the voxel basis but this should be made explicit).
A brief introduction to the key aspects of persistent homology could usefully be included for the majority of readers.
On P2, microscopy images are referred to as having a multiscale nature. It was unclear what this meant, especially since microscopy covers a very broad range of techniques. This should be clarified.
Some small corrections to the text are required: P1: “permit to draw conclusions” should be “permit conclusions to be drawn” P2: “likelihood of a voxel x to be part of” should be “likelihood that a voxel x is part of” P7: the panels in Figure 2 are not aligned (the graphs for the nuclei are vertically offset)
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper makes a valuable technical contribution - a differentiable topological loss function - that could enable significant further work in this area if the descriptions of certain aspects of the process are clarified to support reproducibility of the work. Although other attempts have been made to do this in the past, this is the most convincing I have seen.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #4
- Please describe the contribution of the paper
The submission focuses on a challenging task that aims to reconstruct the 3D objects from 2D images and masks. Upon the existing SHAPR framework that was optimized with a combination of Dice and BCE loss, this submission proposes a topology-aware loss (L_T) consisting of two terms: a Wasserstein distance between two persistence diagrams and a noise reduction term that incentivizes the model to reduce overall topological activity. The experiments are conducted on reconstructing 3D red bed cells and nuclei using the SHAPR framework with or without the proposed topology-aware loss. The topology-aware loss improves predictions in relevant metrics.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
To my best knowledge, the proposed topology-aware loss is novel. As discussed in related works, the idea of topology-aware loss functions has been explored for image segmentation tasks, but the application of the 3D reconstruction from 2D images is novel. Besides, the method is supported by solid mathematical analysis, including the error bounds in loss calculation.
This submission is generally well-organized, with informative illustrations. Although the method section is a bit mathematically heavy, it’s easy to follow with intuitive explanations.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The main weaknesses are in the experiments. First, Sec. 4.2 mentioned that the regularization strength parameter \lambda for L_T is optimized in [10^-3, 1x10^2], which is a very wide range. \lambda = 0.1 is used for all experiments after the hyper-parameter search. This implies that performance degradation should happen at some point when \lambda > 0.1. The problems are:
- There are no results on the robustness of L_T for different values.
- This is no explanation for why it starts to decrease the performance at some point.
- Does L_T directly work without BCE or Dice losses (L=L_T instead of L=L_G+\lambda L_T)? If yes, what’s the performance of using only L_T? If not, why does L_T itself not achieve better or comparable performance than BCE or Dice?
Besides, although the loss is topology-aware, the 3D masks shown in the submission all have a Betti number of 0, which means a single connected component. It is not very clear if the L_T loss can improve the reconstruction of masks with a more complex topology.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The proposed method is mathematically sound, and the baseline model SHAPR has an open-source implementation. Experimental settings are clearly described. Generally, I have a positive impression of reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
As described in the weaknesses, providing more experimental results on the robustness of the proposed L_T under different strength parameters and analyzing its impact on the final reconstruction performance will make the submission stronger.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The proposed method is novel with solid mathematical analysis. The submission is generally well-written and clear to follow. The main weakness is the lack of experimental verification on the robustness of L_T with different weights, which is of practical value. Thus I recommend a weak acceptance before rebuttal.
- Number of papers in your stack
8
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper uses persistent homology to design a multi-scale loss function for 3D reconstruction of 2D image slices. All three reviewers showcased enthusiasm on the paper’s technical novelty and justification of specific design. Therefore, I recommend acceptance of this work.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
”
Author Feedback
Dear Area Chair,
We thank you and the reviewers for constructive feedback on our manuscript and for giving us the opportunity to clarify the points raised.
Reviewers R1, R2, and R4 acknowledged our work, the clear organization and structure of the manuscript, and its benefit for the field.
R1 and R2 have raised the question if code will be made available. We are happy to provide the main model plus supporting library on our GitHub repositories ( under an open-source license (BSD-3-clause), along with all data to reproduce the experiments. In the initial submission, we censored the link to all repositories because it would have compromised anonymity, but will now provide all links in the revised version of our manuscript, making our study fully reproducible. R3 raised the question of how robust our experiments were and how we came to choose the hyperparameters. Accordingly, we will publish our experiments, tracked with “Weights & Biases”, along with the code on our project’s GitHub repository.
Finally, we extended the background on cubical complexes and persistent homology in the revised camera-ready version of our manuscript, as suggested by R2 and R3.