List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Mingjie Pan, Yulu Gan, Fangxu Zhou, Jiaming Liu, Ying Zhang, Aimin Wang, Shanghang Zhang, Dawei Li
Abstract
Three-dimensional microscopy is often limited by anisotropic spatial resolution, resulting in lower axial resolution than lateral resolution. Current State-of-The-Art (SoTA) isotropic reconstruction methods utilizing deep neural networks can achieve impressive super-resolution performance in fixed imaging settings. However, their generality in practical use is limited by degraded performance caused by artifacts and blurring when facing unseen anisotropic factors. To address these issues, we propose DiffuseIR, an unsupervised method for isotropic reconstruction based on diffusion models. First, we pre-train a diffusion model to learn the structural distribution of biological tissue from lateral microscopic images, resulting in generating naturally high-resolution images. Then we use low-axial-resolution microscopy images to condition the generation process of the diffusion model and generate high-axial-resolution reconstruction results. Since the diffusion model learns the universal structural distribution of biological tissues, which is independent of the axial resolution, DiffuseIR can reconstruct authentic images with unseen low-axial resolutions into a high-axial resolution without requiring re-training. The proposed DiffuseIR achieves SoTA performance in experiments on EM data and can even compete with supervised methods.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43999-5_31
SharedIt: https://rdcu.be/dnwwK
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #2
- Please describe the contribution of the paper
The authors propose a new approach to the Isotropic Reconstruction problem in volumetric image restoration based on a clever use of Denoising Diffusion Probabilistic Models and achieve state of the art results.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper has many strengths; it is well motivated and clear. The technical contribution is simple but not obvious. The benefits include better accuracy and broader applicability without retraining, and these results are borne out clearly by the experiments which are interesting and relevant.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The primary weakness is the very large computational cost (can be 1000x slower than one-shot restoration models), but this is addressed in the text and experiments. The authors explore the tradeoff between computational cost and accuracy, clearly demonstrating that the cost is necessary to achieve the results in this framework.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
There are no obvious issues wrt reproducibility. The datasets are standard. The most complex part of the method is based on publicly available code.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Thanks for the nice paper!
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Only “accept” because the performance issue is especially worrying in anisotropic reconstruction given the often very large image volumes.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper proposes a diffusion-based generative model for the isotropic reconstruction of anisotropic EM data. The method includes a Refine-in-the-loop strategy to reduce the sampling step and is agnostic to the anisotropic factor, which is fixed in straightforward CycleGAN setups. The proposed method has been tested in various experiments and has achieved state-of-the-art quality.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The pre-training of the full resolution of the diffusion process at lateral images enables the framework to operate faster and more conveniently.
- The Refine-in-the-loop strategy is a bootstrap to accelerate the sampling and has been supported by an ablation study.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The Sparse Spatial Condition Sampling on the low-resolution branch is not generalizable. If the anisotropic factor is not an integer, a tweak is required to handle the number of inserting rows.
- The proposed method may work only in fixed orientations (xy) but not in others (yz or any other slicing parallel with z).
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The method is reproducible and can be reimplemented by a skillful graduate student.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- Major comments:
- The anisotropic factor and the scale of noise use the same notation \alpha. It is suggested that the authors adjust it to avoid confusion.
- While the Sparse Spatial Condition Sampling module is interesting and can be adapted to any anisotropic factor, inserting its content by zeros is not the only starting point. Various other techniques of interpolation, such as nearest, (bi-)linear, and (bi-)cubic, can be used while keeping the proposed formulas unchanged. The authors are requested to discuss the advantages of using zero-filling compared to others.
- The paper needs to discuss the consistency of the 3D structure while performing 2D slice-based isotropic reconstruction from one viewpoint. The authors should address this issue and show the three orthogonal view planes in the Results section.
- Minor comments:
- In algorithm 1, ‘I’ is an identity matrix. It is recommended to make it bold to avoid confusion with other scalars.
- Major comments:
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Based on the above comments, the manuscript has some merits regarding isotropic EM reconstruction. However, there are some concerns regarding the 3D consistency and interpolating-based initialization. While the concept is interesting, the exposition and presentation of the method are unclear and confusing. As a result, my evaluation of the paper is currently borderline toward weak rejection. However, I am happy to adjust my scores if the authors effectively address those issues during the rebuttal.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
The author addressed my concerns, especially in the initial interpolation step. Therefore I would like to raise my evaluation to weak accept.
Review #5
- Please describe the contribution of the paper
The authors propose diffusion models for the isotropic reconstruction for 3D microscopic images (e.g., match lower axial resolution to higher lateral resolution). The proposed model is robust because it can deal with various image settings without new training.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The main strength of the paper is the application of the diffusion model for isotropic reconstruction of 3D microscopic images. This is important because it doesn’t require the re-training of the network for various image settings (e.g., different anisotropic resolutions).
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The main weakness of the novelty of the method. The way of using the diffusion model for isotropic reconstruction with sparse spatial condition sampling is not novel as it greedily searches the match with different anisotropic resolutions. In addition, a refine-in-loop strategy can not guarantee theoretical convergence and require the empirical choice of stopping condition.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper is ok with reproducibility. They provide the details about the implementation.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The authors propose the diffusion model-based isotropic reconstruction of 3D EM images. The proposed method can improve the robustness of various anisotropic resolution settings. The method is well validated with qualitative and quantitative evaluation by comparing it with state-of-the-art methods such as CycleGAN-IR. The paper is well-organized and easy to read. However, I have a few major concerns about the novelty and feasibility of the method. The diffusion model itself is not new and the application of the diffusion model for IR such as sparse spatial condition sampling and refine-in-loop is a simple search and denoising algorithm without the theoretical proof of the convergence. In addition, the computational time should be reported to check the clinical feasibility as long recon time would be prohibited.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The major factor of my “week reject” is the lack of novelty on method in applying diffusion model to IR and concerns about the computational time and stopping criteria about refine-in-loop approach.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This is an intriguing application of the diffusion model to isotropic EM reconstruction. The central concept revolves around the process of inpainting the missing z-stacks using the diffusion model, which has been trained using x-y 2D images. While the work showcases both interest and utility, the reviewers have expressed the need for further clarification on various aspects. These include the running time of the proposed method, the feasibility of accommodating arbitrary orientations other than xy (ensuring consistency in 3D), and the convergence of the refine-in-loop strategy employed in this particular study. These should be properly addressed in the rebuttal.
Author Feedback
To all: We thank all for their valuable feedback. We are glad that our work has been recognized as novel and well-motivated (R2&R3), as our method eliminates the need for re-training models across different settings (R2&R3&R5), and well-supported by extensive experiments (R2&R3). Reviewers suggested us considering computational time, which is indeed worth paying attention to. For a 256x256x256 3D stack sample, the processing time of DiffuseIR is about 2 hours with 3090Ti, and 0.8 hours with A100. For offline scenarios, the computational time is totally acceptable to the collaborators because for them, the time spent on collecting electron microscopy images ranges from weeks to months.
—— To R2: Thank you for your high praise. Regarding the time cost of our method, you could kindly refer to the “to all” part. ——
To R3: Q1: Consistency of the 3D structure. A1: Like the unsupervised baseline CycleGAN-IR, we use 2D slices for isotropic reconstruction but also have 3D consistency. First, we achieved SOTA in both PSNR (32.37) and SSIM (0.832), indicating strong 3D consistency. We calculate PSNR using the entire 3D stack for a comprehensive assessment of the overall quality. For SSIM, we evaluate metrics slice by slice in both XZ and YZ viewpoints, averaging the scores to obtain the final 3D stack score. Second, our SSCS module plays a key role in achieving 3D consistency. It applies dense lattice structure constraints at each diffusion step, ensuring neighboring slices maintain consistent content throughout the 3D structure. By incorporating these constraints, our method preserves 3D structural coherence and enhances reconstruction quality.
Q2: Impact of interpolation method of SSCS. A2: We will refine our writing to eliminate readers’ confusion. We conducted comparative experiments as suggested by the reviewer. When the interpolation method of SSCS is zero, nearest neighbor, and bilinear, the PSNR scores are 32.37, 32.35, and 32.38, respectively. In the category of random perturbations, this represents that our method is not sensitive to the interpolation method. In principle, the interpolated pixels are noise-added to a standard Gaussian distribution at the initial time step of the diffusion model. Regardless of the interpolation method used, they are a unified (and far different from the image content) distribution, so their impact on the final result is very light.
—— To R5: Q1: Novelty of diffusion model with SSCS A1: We have pioneered the use of inpainting for IR and carefully designed the SSCS diffusion model to achieve inpainting. Existing methods rely on super-resolution techniques but are restricted to specific multiples, rendering them inadequate for adjusting the z-axis scanning interval in various 3D microscopy scenarios. This lack of versatility hinders their application in different contexts. In contrast, our method effectively overcomes the limitations of previous IR methods by offering generalization capabilities, eliminating the need to train separate models for each resolution. Additionally, our unsupervised approach achieves SOTA and can even compete with supervised methods.
Q2: Stability of Refine-in-Loop strategy A2: The Refine-in-Loop strategy lacks theoretical proof of convergence but demonstrates stability through numerous ablation experiments (Fig. 4). The observed regularity is consistent across all datasets. Increasing the number of refine loops leads to higher model accuracy within the same total calculation steps. Users can select refine-in-loop parameters based on their preferred balance between speed and accuracy, or utilize the default parameters we suggest in the paper. We draw a parallel between the Refine-in-Loop strategy and the shortcut connections of ResNet. Despite its simplicity, the Refine-in-Loop strategy proves to be highly effective. Shortcut connections, similarly simple in nature, have been widely adopted and achieved impressive results in practical applications.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
In the rebuttal, the authors effectively addressed the most crucial issues raised by the reviewers, including 3D consistency, the impact of the interpolation method, and the stability of the refine-in-the-loop strategy. R3, in response, raised the score from weak reject to weak accept. After careful consideration, I recommend accepting this paper due to the authors’ satisfactory resolution of concerns and the improvement in evaluation scores.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors present diffusion models as a solution for isotropic reconstruction in 3D microscopic images. The proposed method has undergone extensive experimentation and demonstrated exceptional quality, surpassing existing techniques. However, one limitation lies in its novelty. The utilization of a diffusion model for isotropic reconstruction with sparse spatial condition sampling is not a novel approach. Furthermore, the inclusion of a refine-in-loop strategy lacks theoretical convergence guarantees and necessitates significant computational resources.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
In the rebuttal, the authors have satisfactorily responded to some reviewer concerns, prompting the reviewers to upgrade their scores. Therefore, I recommend acceptance at this stage.