List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Samuele Papa, Efstratios Gavves, Jan-Jakob Sonke
Abstract
Respiratory Correlated Cone Beam Computed Tomography (4DCBCT) is a technique used to address respiratory motion artifacts that affect reconstruction quality, especially for the thorax and upper-abdomen. 4DCBCT sorts the acquired projection images in multiple respiratory correlated bins. This technique results in the emergence of aliasing artifacts caused by the low number of projection images per bin, which severely impacts the image quality and limits downstream use. Previous attempts to address this problem relied on traditional algorithms, while only recently deep learning techniques are being employed.
In this work, we propose Noise2Aliasing, which reduces both view-aliasing and statistical noise present in 4DCBCT scans. Using a fundamental property of the FDK reconstruction algorithm, and prior results from the literature, we prove mathematically the ability of the method to work and specify the underlying assumptions.
We apply the method to a public dataset and to an in-house dataset and show that it matches the performance of a supervised approach and outperforms it when measurement noise is present in the data.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43999-5_46
SharedIt: https://rdcu.be/dnwwZ
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes a deep learning model to reduce both aliasing artifacts and stochastic noise from 4D Cone-Beam Computed Tomography (4DCBCTs) in an unsupervised way. It is an extension a deep learning unsupervised denoising method called Noise2Inverse, originally used to reduce measurement noise. Repiratory artifacts are a real problem in the image guided radiotherapy due to the high precision in the target delimitation.
The model was tested on two datasets of lung CBCT: one with publicly available clinically realistic simulated data and the other an internal clinical dataset.
The paper is well structured, the methodology well presented and the results section well organized.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The main strengh of this work is to have a new unsupervised method to simultaneously reduce noise and aliasing artifacts in 4DCBCT images in IGRT.
The application of the model to public data is also a positive remark because other different methods can be fairly compared.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The main weakness is the number of patients (from the public database) used to train the model. It seems that only 4 patients for training and 1 for testing is a value that is really too low to be able to draw conclusions. Although the number of patients used from the private database increased to 25, it still seems a low number.
The authors present quantitative metrics for the evaluation of the model on the public dataset but for the private one, only a visual inspection is done.
In addition, the fact that ““the model also tends to remove small anatomical structures as high-frequency objects that cannot be distinguished from the noise.” is too important not to have any further consideration. If this fact persists, the model simply cannot be used in clinical practice.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
This paper has a good reproducibility.
The methods are well described and the application of one free available dataset makes is possible to compare results for other algorithms in the future.
One aspect that could improve the reproducibility is that although the methods are very well described, they are described in a very theoretical way which may difficult its replicability for others. If the code is released in the future, it may help to overcome this issue.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
-
The public dataset is a simulated clinical dataset. Although it is a realistic dataset, the authors should mention the fact that this data is simulated and explain how can this affect (or not) the results.
-
The number of training samples is very small,so the conclusion that “No additional data collection was required and the method can be applied without major changes to the current clinical practice” should be carefully modified.
-
Also, some additional comments should be provided about the fact that “the model also tends to remove small anatomical structures as high-frequency objects that cannot be distinguished from the noise.”. If this fact persists, the model simply cannot be used in clinical practice.
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This is a well written and organized paper, with a very hot topic about respiratory motion in IGRT, with a good methodology description but it lacks a more critical approach to weak points such as the low number of training data or the removal of anatomical structures that cannot be distinguished from the noise.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
7
- [Post rebuttal] Please justify your decision
In a general way, the authors reply to all my main concerns.
Review #2
- Please describe the contribution of the paper
The article presents a deep learning-based method called “Noise2Aliasing” for unsupervised perspective mixing and noise reduction in respiratory-correlated cone-beam computed tomography (4DCBCT). The authors demonstrate the mathematical ability of this method using the FDK reconstruction algorithm. The method was applied to both a publicly available dataset and an internal dataset, and the results show that the performance of this method is comparable to supervised methods and outperforms them when measurement noise is present in the data. Also demonstrated good performance in clinical applications.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The article provides a mathematical proof of the validity of the method, which adds to its theoretical foundation. When compared to existing methods, this method demonstrates comparable or superior advantages, making it a promising candidate to become a mainstream method for 4DCBCT imaging. Furthermore, this method does not require additional datasets and can be applied in clinical practice without the need for labeled data. It also avoids the need for supervised methods that require labeled data. Overall, this method shows good performance and potential for widespread use in the field.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
While performance metrics are important for comparing methods, a more comprehensive analysis of the experimental results could help demonstrate the superiority of the new method. It would also be beneficial to supplement the description of the method’s algorithm with appropriate visual aids, such as figures or diagrams. The use of visual aids in conjunction with text can make it easier for readers to understand the method and its underlying concepts.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper has provided sufficient details.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
You make excellent suggestions for improving the clarity and comprehensibility of the article. Clear explanations of formulas and their associated symbols can improve the readability of the article, especially for readers who are not familiar with the technical jargon used in the field. Introducing a wider range of evaluation metrics can provide a more comprehensive and nuanced assessment of the performance of the new method compared to existing methods. This can help identify strengths and weaknesses in the method and provide insights into how it can be further improved. Providing more examples of clinical applications can provide more robust evidence of the feasibility and effectiveness of the new method in real-world settings. This can help readers understand the potential impact of the new method and encourage further research and development in the field.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The work is of clinical interest. However, the results are not well evaluated.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #5
- Please describe the contribution of the paper
This work is an extension of an existing unsupervised method for noise reduction in CT called Noise2Inverse. The extension removes both view-aliasing and stochastic projection noise from 4D CBCT scans. The method is compared against a supervised denoising method and matches its performance on a popular benchmark.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The extension is straightforward and the theoretical foundations are well explained
- The method is evaluated on both an internal and an external dataset
- Proposition and proof are concise and clear
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- My main concern with this paper is whether the (relatively straightforward) extension of Noise2Inverse is enough to justify a MICCAI publication. The original Noise2Inverse method splits the sinograms to generate independent noisy reconstructions and train a denoising neural network. Although I appreciate the proposition and the proof provided by the authors, their extension is mostly an application of the Noise2Inverse to cone-beam CT data. They do not add anything to the method but merely define the data split in a different way.
- No hyperparameters tuning is performed to optimize the method/network architecture/training procedure etc. to the new dataset. Hyperparameters seem to be directly taken from the original Noise2Inverse paper.
- Only one benchmark comparison is provided next to the FDK reconstructions and this is a supervised approach. No comparison with other unsupervised methods is provided.
- Discussion of the results is limited and the unsupervised method performs similar to the supervised method. Why is it important to develop an unsupervised method for noise reduction? Of all image processing tasks, noise is easily added to a (cone-beam) CT dataset to train a denoising network in a supervised way.
- I am also worried about the removal of small details in the images. This is a frequently occurring problem of many denoising methods and could seriously impede the applicability of this method in practice.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
OK (no comment is made about whether code will be made publicly available).
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- Could the authors elaborate on the advantages and disadvantages of an unsupervised method for noise reduction?
- Justify hyperparameter choices
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This is an interesting paper and the approach and experiments are sound. My advise is a weak accept rather than a strong or definite accept because I doubt the novelty of the contribution. This paper relies quite heavily on a previously published method.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper presents a method to reduce noise and aliasing artifacts in Respiratory Correlated Cone Beam CT. The topic is important. But the novelty is limited and the results are not convincing. The paper includes some theoretical analysis that is hard to follow and apparently not very important in terms of its impact on the proposed method or the practical implications.
Derivations are not clearly explained. For example, why do you need the properties of Radon transform in Equation (6). Similarly, why do you need the properties of FDK to arrive at Equation (11). More importantly, given the considerations described at the end of Section 3, what are the significance of these proofs for your method and results? I see none. How did you theoretical proof guide the design of your method?
Figures 1-3 are hard to read and it is hard to tell which reconstruction is better. Table 1, which is the only place quantitative results are presented lacks any statistical significance tests.
Reviewer 1 raises a few important issues regarding evaluations, while reviewer 3 challenges the novelty of the method and limited comparisons, as I also mentioned. Two of the reviewers raise concern about the tendency of the method to remove important image details.
Minor comments:
-
There are instances of poor writing. Example on Page 1: Radiotherapy (RT) is one of the cornerstones of cancer patients.
-
Page 6: “ … and evaluate the metric over only the 4DCBCT volume boundaries.” Is that not a contradiction. Is not your aim to improve the reconstruction for regions that suffer from motion?
Although all three reviewers have recommended accept, I will send this to rebuttal. I will recommend rejection unless they address the comments above.
-
Author Feedback
We thank the meta-reviewer and reviewers. We take the recommended acceptance by reviewers as a sign of genuine interest in our work to the community and welcome the opportunity to improve clarity. Novelty vs Noise2Inverse (N2I) and supervised learning. The primary limitation of N2I is its inability to address view aliasing as it only uses the available projections: in 4DCBCT limited to a single phase. Additionally, it struggles to reduce measurement noise in this setting. To explain why, we explore the 2 ways to use N2I in 4DCBCT: 1. With respiratory-correlated reconstructions, very few projections are available, undesirable for N2I. 2. Using all projections introduces motion artifacts, as N2I requires averaging of the sub-reconstructions to obtain a clean one. A supervised approach is only as good as how realistic the simulated noise in the targets is, in practice often unrealistic. Relevance of theory and derivations. Thanks for noting that the theoretical analysis can become clearer. Our model and the theoretical substrate address the limitations of N2I and are relevant for the following reasons. 1) The derivations (eq 10&11) show that Noise2Aliasing (N2A) learns to predict the true expected reconstruction using only random subsets of projections. This is a crucial finding as it allows for learning reconstructions from much smaller data, addressing the problem of aliasing and 4DCBCT simultaneously. 2) Eq 10&11 rely on FDK as a reconstruction algorithm because it uses the Dual Radon Transform (DRT) as its fundamental step. The DRT is always normalized to scale the values as probabilities, with weights dependent on the subset of projections. From this and a few mathematical steps follows that, when combining subsets, the DRT equals the average of the individual subset transforms. This is key to reducing view-aliasing as we use small subsets of projections but still guarantee we reconstruct the true structures in expectation. 3) With eq 9, we show that the trained model produces a clean reconstruction by i) developing the loss function from eq 8 into the same form as eq 4 and ii) then proceeding in the same way that was done for eq 5. This informs many choices: -The subsets of projections must be disjoint -We must use a linear reconstruction algorithm -There must be enough information in the projections to reasonably determine that they correspond to the same structure. We will provide the clarifications for eq 6, 9, and 11 to motivate derivations. p-values and Tab. 1. Tab. 1 shows mean and SD across the phases of one patient for which a p-value would not be informative. For the internal dataset, we provide a quantitative analysis which we previously only reserved for the supplement. N2A trained on 25 patients and tested on 5 achieved mean PSNR of 35.24 and SSIM of 0.91, while the clinical method achieved mean PSNR of 29.97 and 0.74 SSIM with p-value of 0.048 for the PSNR and 0.0015 for the SSIM, so N2A was significantly better in both metrics. Figures and image details We provide high-quality videos of reconstructions from Figures 1-3 in the supplement. The 25 patients model is more aggressive in removing noise which naturally affects small vessels, as with similar methods. The video on the internal dataset clarifies this: the vessels that are not reconstructed are also not trivially identifiable in the noisy image. Notably, this does not happen when the model is trained on a single patient, indicating that it has the ability to reconstruct even those vessels, which we leave for future work. We agree our statement “No additional data collection…clinical practice” should be more modest. We change this to “Training deep learning models for medical applications often needs new data. This was not the case for N2A, and historical clinical data sufficed for training.” Thank you very much, we will incorporate the suggestions and dedicate more efforts to meticulously review the text, ensuring its utmost comprehensibility.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
I would have preferred to reject the paper due to (i) poor presentation and (ii) limited novelty. I decided to recommend accept because of the recommendation of the reviewers and because other papers in my pile are not so good.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
All three reviewers now have unanimously given accept for this work and the rebuttal satisfied concerns from previous reviews.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
After carefully considering the reviewers’ feedback and the authors’ rebuttal, a unanimous decision has been reached among the reviewers to accept the paper. The authors have diligently addressed all the concerns and provided clarifications in their rebuttal, which have satisfied the reviewers. As a result, the Meta Reviewer recommends accepting the paper for publication. The unanimous consensus among the reviewers, combined with the careful addressing of concerns, solidifies the decision to accept the paper and acknowledges its value in advancing the field.