Authors

Yu Liu, Kurt Weiss, Nassir Navab, Carsten Marr, Jan Huisken, Tingying Peng

Abstract

Light-sheet fluorescence microscopy (LSFM) is a cuttingedge volumetric imaging technique that allows for three-dimensional imaging of mesoscopic samples with decoupled illumination and detection paths. Although the selective excitation scheme of such a microscope provides intrinsic optical sectioning that minimizes out-of-focus fluorescence background and sample photodamage, it is prone to light absorption and scattering effects, which results in uneven illumination and striping artifacts in the images adversely. To tackle this issue, in this paper, we propose a blind stripe artifact removal algorithm in LSFM, called DeStripe, which combines a self-supervised spatio-spectral graph neural network with unfolded Hessian prior. Specifically, inspired by the desirable properties of Fourier transform in condensing striping information into isolated values in the frequency domain, DeStripe firstly localizes the potentially corrupted Fourier coefficients by exploiting the structural difference between unidirectional stripe artifacts and more isotropic foreground images. Affected Fourier coefficients can then be fed into a graph neural network for recovery, with a Hessian regularization unrolled to further ensure structures in the standard image space are well preserved. Since in realistic, stripe-free LSFM barely exits with a standard image acquisition protocol, DeStripe is equipped with a Self2Self denoising loss term, enabling artifact elimination without access to stripe free ground truth images. Competitive experimental results demonstrate the efficacy of DeStripe in recovering corrupted biomarkers in LSFMs with both synthetic and real stripe artifacts.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16440-8_10

SharedIt: https://rdcu.be/cVRvu

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The proposed method removes stripe artifacts from LSFM images. It leverages domain specific information including fourier domain behaviors, and uses a graph NN to fill in suspected corrupted datapoints in the fourier domain. It combines this with an optimization in the spatial domain to encourage spatial continuity.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The motivation (problem to be solved and why) is compelling. The blending of optimization in both spatial and spectral domains is ingenious, and leverages the specifics of the use-case. The proposed method appears to give better results than the baselines used (though there is little visible difference in the output images). Uncertainty measures are provided.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Some of the exposition was insufficiently clear (examples given in Detailed Comments). I was unclear why a GNN was desireable, and why a more straight-forward method might not work (see Detailed Comments).
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

There is no mention in the main text about code being available.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Thank you for providing uncertainty measures - highly useful. p-values are not so useful. See Wasserstein, ASA statement on P-values 2016.

“within a wedge in the Fourier space”: maybe clarify here that stripes are always perpendicular to the edge in LFSM images (not diagonals), assuming this is the case. Fig 4 does not show any difference between filling the wedge and DeStripe - would a differently-zoomed image show a difference?

(2.1) “survival function”: what is this? “Rayleigh distribution”: why Rayleigh, and what is its relevance here? “this concentric annulus”: how thin or thick? Does it matter?

Fig 1e: It looks like all coefficients in the wedge are labeled as corrupted. Is this typical, or would the actual mask be more pixelated, with clean and corrupt labels intermingled? Perhaps clarify that at the end of 2.1 where the wedge mask is discussed. If pixelated, perhaps modify the figure to make this clear.

(2.3) This section leaves me wondering why a more straightforward method, such as filling in the fourier coefficients with the MAP based on the annulus distribution, would not work. Can you explain why the complex GNN system proposed is necessary? “also corresponding frequencies into account”: what does “taking into account” mean concretely?

(2.4) How are lambda_x, etc determined? Presumably they are image-specific, based on the number of edges and steps in the image.

(2.5) First term of eqn 7 (ie |Y-X|): Why do you strive to match corrupted sections of Y (the stripes), if these are what you are trying to remove, ie why is the first term used at all?

Fig 1, 2,: Sideways lettering is basically impossible to read. If it is important, perhaps the figure can be rearranged so that the lettering is horizontal.

Miscellaneous: “prior one”: what does this mean?

Typos/grammar: In general, an additional thorough proofing would improve the text.

-holds true -> hold true
- barely exits -> exists. Also the grammar of the sentence wants correction -LSFM once excites -> leave out ‘once’ “B is the Bregman” -> V (I think; there is no B in eqn 2) -(in 2.5) are X and X_tilda the same thing?
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The method is interesting, creative, and well-motivated by the problem statement. The results appear to be strong. The writing (clarity, grammar) is somewhat weak, so the content is harder to absorb than would be ideal. One piece of the architecture (the GNN) is not well-motivated. The paper’s rank in my stack could have been higher; I had to pick an order. The paper rating could have been higher; I had to pick a number.
Number of papers in your stack

5
What is the ranking of this paper in your review stack?

3
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

This paper tackles the problem of removing stripe artifacts in light-sheet fluorescence microscope images. The method models stripes as a combination of a graph neural network on the Fourier coefficients connected in a polar coordinate structure, a spatial domain Hessian based regularization scheme. and an energy minimization scheme to seek the optimal noise reduced image. The method is analyzed on a single noise-free image-volume with repeatedly artificially added stripe noise and compared with several existing algorithms. The results are assessed wrt. peak signal to noise ratio and structure similarity measure and performs better than the existing methods. The method is also applied to a real image with favorable result.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The paper proposes a novel extension of existing works by applying a graph neural network as a model in Fourier space together with the Bregman algorithm to optimize an energy formulation. The paper is well written, and it is compared with a wide range of existing techniques. The method appears to perform as state-of-the-art
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The model is very complex, has several layers, and it is unclear how many parameters, the method has, what the computation resources need to run the method, and the method has only been compared against the state-of-the-art on synthetic images. Since SSIM is linked to human perception, which seems a less obvious choice in this case, and it is unclear how the PSNR between two images has been calculated, and thus the results in Table 1, can be difficult to interpret in light of the noise removal task.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

My mental test is always, whether I believe I would be able to make a student reproduce the results, and in this case, the answer is Yes.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

See weaknesses for useful improvements.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Although modelling the stripes in the Fourier domain is not novel, e.g., [8] in the paper’s list of references, the gnn, optimization, and the comparison with existing methods makes it an acceptable paper.
Number of papers in your stack

1
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

This paper contributes a new method to remove stripe artifacts from light-sheet microscopy data. The approach is based on self-supervised learning with a graph neural network and a Hessian regularization and thus, combines both frequency and spatial domain features.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The paper introduces a novel method to solve the stripe artifact removal which is an important problem in light-sheet microscopy. The results are very convincing.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

I don’t see many weaknesses. If I have to name one, I would say the paper could benefit from a few more explanations to lead the reader to better understand the technical details of the methods.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors state that the code will be made available for biologists for academic use. Can you clarify how restrictive this is? What about scientists from other fields? Maybe you could comment on why you decide against open-sourcing the code.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- p.4. “which fall within the same thin concentric annulus A^r_k” Maybe you could annotate this annulus (or an example) in Figure 1 to guide the reader.
- p.4. after equations 2a - c) you mentioned “B is the Bergman variable” but there is no “B” in the formula. I suppose this is a typo
- p.5. “ Note that we use complex-valued building blocks [20] for W1 and W2”. Just out of curiosity: What framework did you use? I remember that there were some issues with PyTorch complex numbers implementations. But maybe this has been solved by now.
- p.5 “s a result, by explicitly modeling the sample-only spectral response as a weighted combination of its uncorrupted neighbors on a polar coordinate, stripe-only Fourier projection is exclusively re-served as activation M ⊙ H(L+1), which can then be subtracted from the input stripe-sample mixture Y for striping removal.” I think I get the point, but I find this hard to understand when reading it. Maybe this sentence should be rewritten.
- p.6 “shrink(•) is the scalar shrinkage operator” Maybe a naive question: I am not sure how standard this operator is but maybe you could add a reference here just in case.
- p.6 “Eq. (7) is to prevent the model from learning an identical mapping by quantifying isotropic properties of recovered X, where Pk is the corrupted subset of Ark indicated by corruption mask M, and Qrk = {x ̃ |x ̃ ∈ Ark , x ̃ ∈/ Prk }.” I think here a short explanation might be nice to help the reader understanding the general idea here.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I think it is a strong and interesting method, the results are very convincing.
Number of papers in your stack

5
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper introduces a method for destriping light sheet data with a novel combination of frequency and spatial domain features. All the reviewers agree on the novelty and contribution of the work. However, please respond to all their comments about clarifications in writing for the final manuscript submission.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

3

Author Feedback

Dear Area Chair,

We thank you and the reviewers for their constructive feedback on our manuscript and for giving us the opportunity to clarify the points raised.

Our method, a blind stripe artifact remover for light-sheet microscopy called DeStripe, is “a novel extension of existing works by applying a graph neural network as a model in Fourier space together with the Bregman algorithm to optimize an energy formulation” (R2). All reviewers acknowledge the novelty of our proposed method and find that “the blending of optimization in both spatial and spectral domains is ingenious, and leverages the specifics of the use-case” (R1). They further agree that the experimental evaluations are “strong” (R1), “state-of-the-art” (R2), and “very convincing” (R3). Yet they also have concerns about i) the complexity and reproducibility of DeStripe (R1, R2, R3), (ii) motivation for the uses of a graph neural network (R1), and (iii) clarifications in writing for the final manuscript submission (R1, R3).

i) As stated in the conclusion section of our manuscript, DeStripe code will be made accessible for biologists for academic usage. Regarding the concerns raised by R1 and R3, we will make DeStripe code open-source on GitHub and are very happy to see DeStripe help scientists from other fields for their research. Along with the Pytorch code for DeStripe, we will also make our code for calculating measurements, e.g., PSNR, available, to “allow the results in Table 1 more interpretable in light of the noise removal task” (R2).

ii) To address the concerns of specific motivation for using a graph neural network raised by R1, we include the following explanation in our final submission: Inspired by the homogeneous Fourier projection of a biological sample against highly directional one for stripings, sample-only spectral response within the corruption mask M is modeled as a weighted combination of their uncorrupted neighbors on a polar coordinate. To this end, we adopt a graph neural network, which is able to vary the neighborhood size by constructing the receptive fields.

iii) With respect to concerns in writing for the final manuscript submission, we check the detailed comments and update the paper accordingly: iii. a) we clarify the perpendicular of stripes to the edge in LSFM images (R1), add a differently-zoomed image in Fig. 4 to “show difference between filling the wedge and DeStripe” (R1), and re-organize Fig. 1 with the pixelated corruption mask clarified (R1) and the annulus in Section 2.1 annotated (R3). iii. b) Reference “Sparse deconvolution improves the resolution of live-cell superresolution fluorescence microscopy” is included to specify the scalar shrinkage operator (R3). iii. c) we clarify that we borrow the design of complex-valued building blocks from [20] for W1 and W2, which simulates complex arithmetic using two real-valued entities (R3). iii. d) we include our motivation for using Rayleigh distribution as: for every slice, its Fourier coefficients except stripe-corrupted ones, which fall within the same thin concentric annulus, mathematically follow a two-dimensional Gaussian distribution [8], which in turn leads to the Rayleigh distribution as the amplitude distribution model (R1). iii. e) we design the first term of Eq. 7 to encourage the agreement between the prediction and input image, corresponding to the data fidelity term in Eq. (1) in a model-based optimization framework (R1).

Finally, we thank the reviewers for identifying some typos and misleading sentences in the paper that we have already corrected. We hope that we could clarify the concerns of the reviewers and provide a revised manuscript that is “interesting, creative, and well-motivated by the problem statement” (R1), for “an important problem in light-sheet microscopy” (R3).Dear Area Chair,

We thank you and the reviewers for their constructive feedback on our manuscript and for giving us the opportunity to clarify the points raised.

Our method, a blind stripe artifact remover for light-sheet microscopy called DeStripe, is “a novel extension of existing works by applying a graph neural network as a model in Fourier space together with the Bregman algorithm to optimize an energy formulation” (R2). All reviewers acknowledge the novelty of our proposed method and find that “the blending of optimization in both spatial and spectral domains is ingenious, and leverages the specifics of the use-case” (R1). They further agree that the experimental evaluations are “strong” (R1), “state-of-the-art” (R2), and “very convincing” (R3). Yet they also have concerns about i) the complexity and reproducibility of DeStripe (R1, R2, R3), (ii) motivation for the uses of a graph neural network (R1), and (iii) clarifications in writing for the final manuscript submission (R1, R3).

i) As stated in the conclusion section of our manuscript, DeStripe code will be made accessible for biologists for academic usage. Regarding the concerns raised by R1 and R3, we will make DeStripe code open-source on GitHub and are very happy to see DeStripe help scientists from other fields for their research. Along with the Pytorch code for DeStripe, we will also make our code for calculating measurements, e.g., PSNR, available, to “allow the results in Table 1 more interpretable in light of the noise removal task” (R2).

ii) To address the concerns of specific motivation for using a graph neural network raised by R1, we include the following explanation in our final submission: Inspired by the homogeneous Fourier projection of a biological sample against highly directional one for stripings, sample-only spectral response within the corruption mask M is modeled as a weighted combination of their uncorrupted neighbors on a polar coordinate. To this end, we adopt a graph neural network, which is able to vary the neighborhood size by constructing the receptive fields.

iii) With respect to concerns in writing for the final manuscript submission, we check the detailed comments and update the paper accordingly: iii. a) we clarify the perpendicular of stripes to the edge in LSFM images (R1), add a differently-zoomed image in Fig. 4 to “show difference between filling the wedge and DeStripe” (R1), and re-organize Fig. 1 with the pixelated corruption mask clarified (R1) and the annulus in Section 2.1 annotated (R3). iii. b) Reference “Sparse deconvolution improves the resolution of live-cell superresolution fluorescence microscopy” is included to specify the scalar shrinkage operator (R3). iii. c) we clarify that we borrow the design of complex-valued building blocks from [20] for W1 and W2, which simulates complex arithmetic using two real-valued entities (R3). iii. d) we include our motivation for using Rayleigh distribution as: for every slice, its Fourier coefficients except stripe-corrupted ones, which fall within the same thin concentric annulus, mathematically follow a two-dimensional Gaussian distribution [8], which in turn leads to the Rayleigh distribution as the amplitude distribution model (R1). iii. e) we design the first term of Eq. 7 to encourage the agreement between the prediction and input image, corresponding to the data fidelity term in Eq. (1) in a model-based optimization framework (R1).

Finally, we thank the reviewers for identifying some typos and misleading sentences in the paper that we have already corrected. We hope that we could clarify the concerns of the reviewers and provide a revised manuscript that is “interesting, creative, and well-motivated by the problem statement” (R1), for “an important problem in light-sheet microscopy” (R3).

back to top

DeStripe: A Self2Self Spatio-Spectral Graph Neural Network with Unfolded Hessian for Stripe Artifact Removal in Light-sheet Microscopy