List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Xuanyu Tian, Qing Wu, Hongjiang Wei, Yuyao Zhang
Abstract
Fluorescence microscopy is a key driver to promote discoveries of biomedical research. However, with the limitation of microscope hardware and characteristics of the observed samples, the fluorescence microscopy images are susceptible to noise. Recently, a few self-supervised deep learning (DL) denoising methods have been proposed. However, the training efficiency and denoising performance of existing methods are relatively low in real scene noise removal. To address this issue, this paper proposed self-supervised image denoising method Noise2SR (N2SR) to train a simple and effective image denoising model based on single noisy observation. Our Noise2SR denoising model is designed for training with
paired noisy images of different dimensions. Benefiting from this training strategy, Noise2SR is more efficiently self-supervised and able to restore more image details from a single noisy observation. Experimental results of simulated noise and real microscopy noise removal show that Noise2SR outperforms two blind-spot based self-supervised deep learning image denoising methods.We envision that Noise2SR has the potential to improve more other kind of scientific imaging quality.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16446-0_32
SharedIt: https://rdcu.be/cVRTr
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
- Authors proposed a self-supervised image denoising method to train a image denoising model based on single noisy fluorescence image.
- The propose technique consists of a sub-sampler module that generates sub-sampled noisy images from the original one; and an image SR module that improves the sub-sampled noisy image resolution to that of the original one.
- Quantitative and qualitative experiments are presented comparing the proposed technique and some state-of-the-art methods.
- The results show a good performance (PSNR, SSIM).
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Novelty: method does not require multiple noisy observations and external noise distribution assumptions.
- The method shows a good performance in terms of PSNR and SSIM when compared to some state-of-the-art methods.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Execution time is not presented.
- Limited comparison to state-of-the-art: Only PSNR and SSIM mesures are studied (but, it’s enough for a conference paper). A computational time comparison is missing.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The author provides the code and all the information necessary to reproduce the results.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Some suggestions:
- It would be interesting that authors gave some information about the computational time for proposed method.
- Summarize the research limitations and future research directions.
- Describe the computing system (hardware) used in experiments.
- Extend the Conclusion with details on the method performance when compared to other tested techniques (in terms of PSNR improvement and SSIM).
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The topic of the paper is relevant and interesting to the MICCAI community .
- It presents an innovative idea for Fluorescence image denoising based on single noisy observation. Authors propose a sub-sampler module that generates sub-sampled noisy images from the original one; and an image SR module that improves the sub-sampled noisy image resolution to that of the original one.
- Although some information is missing in the results section (computational time and hardware), the results are good in terms of PSNR and SSIM.
- Number of papers in your stack
6
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
7
- [Post rebuttal] Please justify your decision
The paper is original enough and of sufficient interest to the MICCAI community to merit publication.
Review #2
- Please describe the contribution of the paper
This paper aims to denoise a set of noisy images with the same noise distribution with only one observation per image. It cleverly improves Noise2Self idea by creating training image pairs through image subsampling and randomizing subsampling mask. In experiments, the proposed method achieves competitive results and outperforms Noise2Self.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
Novelty. It’s a neat idea to make use of image subsampling to generate the image pairs for self-supervision. (1) Compared to the previous Noise2Self work, the sampling of pixels for prediction is straightforward in this work. (2) Instead of simply predicting masked pixel intensities, this work trains a super-resolution network to effectively regress with 1/4 pixels to the rest pixels.
-
Convincing experiments. On both simulated and two types of microscopy images, the proposed method significantly outperforms (0.2-0.4 dB) its counterpart Noise2Self and achieves competitive results with the SOTA N2N that requires multiple observations.
-
The paper is well-written and easy to follow. The method is described in detail and straightforward to reproduce.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Lack of extensive evaluation: The experiments are done only on fluorescence images. It’s unclear how applicable the method is for general biomedical images. E.g. N2N is evaluated on MRI.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Yes. The paper provides enough details and the algorithm itself is easy to implement.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
-
Model: The paper can be viewed as an extension of N2S, predicting a random set of pixels. It’ll be great if the paper can plot the performance vs. number of pixels to predict, showing the interpolated results between two methods.
-
Experiments: The proposed method will be more convincing if evaluated on different image modalities.
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper proposes a neat idea improving upon previous method significantly. I don’t give higher score due to the lack of thorough evaluation on different image modalities. In general, this paper is solid in terms of both idea and experiments.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
The rebuttal clarifies that (1) N2N achieves better result but with extra data, (2) its advantage over NB2NB. Although at a first glance the proposed method looks like an incremental improvement over NB2NB, the proposed method naturally resolve the intensity gap between NB pixels issue and has a much simpler pipeline (e.g., reconstruction loss only). The upsampling module initially looks like redundant, but actually resolves the coordinate mismatch problem in NB2NB.
I agree with meta that the paper needs to be revised to explain its difference from NB2NB clearly.
Review #3
- Please describe the contribution of the paper
The authors propose a self-supervised image demonising model using U-Net. Here, the authors used a single noisy observation for self supervision. During the main task the authors used paired noisy images of different dimensions. While the model takes advantage of well-known noise2noise, unlike this method the use of sub-sampling module allows to authors to train image pairs at different resolutions. The experimental results showed competitive results on fluorescence microscopy datasets.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Simple idea of using self-supervision
- Easy to follow the paper
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The paper lacks enough rationale behind self-supervision and subsampling technique.
- The paper requires ablation to provide evidence on each stage improving results. Currently the overall gain is marginal and its hard to say where this is coming from
- N2N and propose N2SR have competitive results so why the proposed is better or effective? This is Not well established.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors have provided a link to the dataset and have agreed to provide train-test dataset splits.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- What does super-resolved mean? Authors need to define SR in the text.
- In N2N, one can use patches to use training, i.e. can be trained using less samples so how does using sub-sampling module help boost performance?
- As commented in the weakness above, authors could benefit from providing more ablation details.
- Please add time comparisons to better understand more clearly if there is any added value to this method
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The presented method is very close to noise2noise method with small changes such as able to use sub-sampling module. However, the results are very competitive to the former N2N method. The paper does not show any comprehensive analysis on why the presented method is of advantage over other methods. In this light ablation studies are not provided which makes it hard to understand whether the sub-sampling module is helping of the pretext assignment is giving an advantage. It requires more work and insight.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper introduces a self-supervised image denoising method that leverages super-resolution from undersampled images. By doing this, unlike Noise2Noise (N2N), multiple noise observations are not required to train the self-supervised denoiser and comparable performance is achieved.
Although the average score of this paper is leaning toward weak accept, this paper received diverged reviews (ranging from 3 (reject) to 7 (strong accept)).
The stregnths of this paper include 1) novelty of using subsampler and SR for denoising, 2) improved performance compared to other self-supervised denoisers.
The weaknesses are 1) insufficient rational behind self-supervision and subsampling technique, 2) no ablation study, 3) not the best result (N2N is better)
Due to such diverging opinions, I carefully read the paper by myself and I found several issues (summarized below) that the authors should clarify to warrant acceptance.
1) The main idea of this method is generating arbitrary many undersampled images, which are up-scaled using the SR module, to make image pairs. However, such techniques (except up-scaling) are already introduced in Neighbor2Neighbor (Huang et al., CVPR 2021), which impairs the novelty of this method.
2) Having said that, the only difference between Neighbor2Neighbor and the proposed method is to define the self-supervised loss in the high-resolution domain using the SR module. What is the rationale behind this design decision (why is this better than using low-resolution domain as in Neighbor2Neighbor)? An ablation study to show the effect of SR module should have been included in the experimental result.
3) Neighnor2Neighbor should be properly cited and compared in the result. I suspect that the performance of the proposed method would be similar to that of Neighrbor2Neighbor.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3
Author Feedback
We are very grateful for the constructive comments of the meta-reviewer and reviewers.
Compared with Noise2Noise (N2N), the advantage and drawbacks of N2SR (R3) First of all, N2N requires at least 2 independent noisy observations of the same static scene for training. However, multiple observations of the same object are always impractical to obtain, thus constraining the applicable scenario of N2N. The proposed N2SR only needs a single noisy observation for training. In the training of N2SR, the input subsampled image and target image of higher resolution are from the same noisy image. Thus the denoising performance of N2SR is slightly lower than N2N, but it is more practical for more application scenarios where the imaging process is not repeatable.
Compared with Neighbor2Neighbor (NB2NB), the advantage of N2SR (meta) Most existing single-image denoising methods share the same purpose of constructing independent image pairs for training the denoising network. We clarify that N2SR can be considered an extension of previous blind-spot denoising networks (ex. N2Void, N2Self), generating more blind-spot pairs for training supervision. The major difference between NB2NB and blind-spot methods is that the paired pixels are not extracted from the same image coordinate but neighboring pixels, introducing a certain intensity gap between the two sub-sampled images. If NB2NB directly minimizes the reconstruction loss between the two sub-sampled images, the denoising results will be over smoothed. Thus, NB2NB proposes an additional regularization term to compensate for this gap in generating sharper denoised images. Therefore, compared with previous blind-spot methods, the denoising performance of NB2NB depends on the image quality and SNR. When the image SNR is relatively high, NB2NB outperforms other methods since it made more pixel pairs (about ¼ number of the original image) for training. With low SNR, denoising performance is degraded by more significant gaps btw pixel pair intensities. The manually tuning parameter also affects the denoising performance. Compared with NB2NB, N2SR intends to generate more pixel pairs for denoising network training without making an intensity gap between pixel pairs. We first use the subsampled image to train a super-resolution network to generate an image with the same dimension as the original image. Then the rest part of the original image (about ¾ number of the original image) and the SR image are used as denoising network training pairs, where we use pixel pairs extracted from the same image location and independent from the ones that used for SR network training. Therefore, N2SR gains an advantage from the increasing number of pixel pairs for training the denoising network, as well, avoiding the unstable intensity gap from NB2NB. Actually, the SR module is a simple up-sampling layer. In the future, we can explore if SOTA super-resolution networks can be applied in denoising tasks to improve performance.
Denoising performance compared with NB2NB (meta) We evaluated the NB2NB denoising performance on real noisy fluorescence images. For NB2NB, we carefully choose gamma=0.1 to achieve its best performance. Quantitative results (PSNR/SSIM) of NB2NB on the test set is 33.93/0.9120, while N2SR is 34.11/0.9150, N2N is 34.21/0.9175 and N2S is 33.83/0.9086. In NB2NB, gamma has a large impact on its performance. When gamma = 0.25 & 0.5, the results of NB2NB are 33.77/0.9096, 33.33/0.8987 respectively. We will add these results in the revised version.
Computational time comparison and evaluation on other image modalities. (R1,R2,R3) For a noisy grayscale image of 512*512, N2SR takes 0.3714s to get a denoising result, while N2S takes 0.0915s on NVIDIA 3060 GPU. Recently, we applied N2SR to noisy MRI and TEM data and obtained competitive results.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors addressed the most of issues well in the rebuttal. Specifically, the differences between existing methods (N2N, NB2NB) and their method are explained, and the supplementary result of NB2NB demonstrates the comparable performance of the proposed method. The revewers seem to be satisfied with the rebuttal as well.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
1
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper uses subsampling and super-resolution to denoise microscopy images. Two reviewers remain positives after the rebuttal, while the third one has critical comments on the incremental novelties and marginal improvements, compared to two related works. The meta-reviewer has one more question: is the zero-mean additive gaussian noise model the real noise model in fluorescent microscopy? The causes of noise can be from many aspects.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have addressed the main concerns of all the reviewers and metareviewer. In general, I think the paper has value even just in the fact that only single images are needed for training as opposed to multiple images of the same static scene. In addition, they have included quantitative results of comparisons to the other methods and the inclusion of this will make this paper suitable for acceptance.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5