List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Rui Hu, Yunmei Chen, Kyungsang Kim, Marcio Aloisio Bezerra Cavalcanti Rockenbach, Quanzheng Li, Huafeng Liu
Abstract
Deep learning based PET image reconstruction methods have achieved promising results recently. However, most of these methods follow a supervised learning paradigm, which rely heavily on the availability of high-quality training labels. In particular, the long scanning time required and high radiation exposure associated with PET scans make obtaining these labels impractical. In this paper, we propose a dual-domain unsupervised PET image reconstruction method based on learned descent algorithm, which reconstructs high-quality PET images from sinograms without the need for image labels. Specifically, we unroll the proximal gradient method with a learnable l2,1 norm for PET image reconstruction problem. The training is unsupervised, using measurement domain loss based on deep image prior as well as image domain loss based on rotation equivariance property. The experimental results demonstrate the superior performance of proposed method compared with maximum-likelihood expectation–maximization (MLEM), total-variation regularized EM (EM-TV) and deep image prior based method (DIP).
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43999-5_15
SharedIt: https://rdcu.be/dnwwt
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This work presents a dual-domain unsupervised PET image reconstruction method based on learned descent algorithm, which reconstructs high-quality PET images from sinograms without the need for image labels.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The formualtion is novel and interesting.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The man weakness of the work is the experiment. It is unclear what data and how experiment is performed.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
the reprodcbility is uncelar as model is not published.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Improve the experimental setup and clarify on datasets and comparative methods.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
main recommendation is based on methods novlety.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This paper summarized the drawbacks of the current PET reconstruction methods and proposed a novel unsupervised PET image reconstruction method based on learned descent algorithm, which can reconstruct high-quality PET images from sinograms without the use of paired PET images. Specifically, first, the paper proposed a model based deep learning network along with a learnable norm for more robust feature extraction. Second, the paper introduced unsupervised learning to PET reconstruction task, proposing a training unsupervised strategy which provides constraint on both image and sinogram domains. Third, the method showed superior performance over other comparative methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
a) This paper analyzed the pros and cons of the current PET reconstruction methods and proposed a novel unsupervised PET image reconstruction method which can overcome the drawbacks to some aspect.
b) The proposed method is an unrolled reconstruction method with acceptable mathematical foundation and interpretability.
c) Different from most of the previous methods, the paper introduced unsupervised learning to PET reconstruction. Moreover, the method set constraints from image domain and sinogram domain, which can be plug-in-and-play for reconstruction models.
d) Experiments are conducted on phantom dataset and clinical dataset, and the results demonstrated the superiority of the proposed method over other reconstruction methods.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
a) The paper argues strongly against end-to-end direct learning methods and deep generative models, citing several drawbacks. However, the paper lacks evidence supporting these arguments. Additionally, the proposed method has only been compared with basic reconstruction methods like MLEM, which makes its superiority less convincing based solely on current experiment results.
b) The introduction section could benefit from a more comprehensive review of previous works, including more recent advancements in end-to-end direct learning methods. To make the drawbacks listed in the introduction more convincing, the paper should provide supporting evidence.
c) In Figure 2, the quality of “Slice 199” reconstructed by MLEM-TV appears to be lower than that of the proposed method. However, the PSNR of the image reconstructed by MLEM-TV is higher. The paper should explain this discrepancy to avoid confusion.
d) It is unclear if the unrolled method performs better than other deep learning methods. The paper should prove if the proposed method is limited to unsupervised scenarios with designed loss terms or if it can outperform other fully-supervised methods when ground truth labels are available during training.
e) The paper lacks sufficient discussion regarding the contribution of network structure and loss terms in the ablation study. Table 2 indicates that the inclusion of L-measure in the framework increases PSNR, while the importance of other components is not clear. The paper should provide more details regarding the significance of L-measure and the contribution of other components. Moreover, the paper should clarify if the proposed method is inferior to comparative methods in the experiment section without L-measure.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Codes are not released
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
a) As mentioned above, the paper argued the drawbacks of end-to-end direct learning methods as well as the deep generative models, yet without any experimental proof. Thus, more end-to-end models, as well as deep learning regularized iterative methods should be included as comparative methods.
b) It is suggested that the introduction part includes more recent reconstruction methods, especially end-to-end reconstruction methods.
c) The comparative methods are insufficient. Why some unsupervised reconstruction methods [1,2] mentioned in introduction part are not chosen as comparative methods in experiments?
[1]Hashimoto, F., Ote, K., Onishi, Y.: PET image reconstruction incorporating deep image prior and a forward projection model. IEEE Transactions On Radiation And Plasma Medical Sciences. 6, 841-846 (2022) [2] Shen, C., Xia, W., Ye, H., Hou, M., Chen, H., Liu, Y., Zhou, J., Zhang, Y. Unsupervised Bayesian PET Reconstruction. IEEE Transactions On Radiation And Plasma Medical Sciences. (2022)d) I wonder whether the proposed deep unrolled method can achieve better performance than other deep learning methods in same scenario. This can be completed by introducing the supervision of ground truth label and comparing the methods with other supervised deep learning methods in the section of experiments.
e) The contribution of the network structure and loss terms should be discussed more in ablation study. According to Table 2, when L-measure is added to the framework, the PSNR increases from 15.41 to 19.61. Why L-measure is so important and the contribution of other components is not obvious? Does it mean that the proposed method can’t beat the comparative methods in experiment section without L-measure? This may make the proposed unrolled reconstruction framework lack the contribution.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
There are some major problems that should be addressed. First, the paper should provide more experimental evidence to support the arguments against current PET reconstruction methods, since the paper only compares the method with some traditional methods like MLEM. Second, the introduction section could benefit from a more comprehensive review of previous works, including more recent advancements in unsupervised learning methods and more recent end-to-end deep learning methods. Third, while the proposed deep learning network and learnable norm are promising, the paper needs to thoroughly discuss their contributions. The L-measure has a dominant contribution to the boosted performance, while the other components do not contribute so much. Finally, uploading the paper to Arxiv may be not proper for the process of double-blind peer review.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
4
- [Post rebuttal] Please justify your decision
I decide to maintain the initial rating for the paper. The author addressed some of my concerns, but I still doubt about the contribution of the proposed components. The performance increases much with LMeasure added, while the proposed unrolling the learned descent algorithm which is considered to be the main novelty, does not perform well without LMeasure. Additionally, if the author wishes to demonstrate that the designed loss terms is plug-and-play, it is necessary to test the loss term on other methods for further evaluation.
Review #3
- Please describe the contribution of the paper
The paper proposes an unsupervised method for PET image reconstruction in a low-dose setting with superior results. The proposed method incorporates a learnable norm for general and robust PET image reconstruction. The method was trained in a dual-domain manner, combining information from both the image domain and sinogram domain.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Instead of directly replacing the penalty term with a neural network, the paper proposed to parameterize the penalty term as the learnable norm, preserving mathematical rigor and making the model interpretable to a certain extent.
- The method showed superior performance compared with other reconstruction algorithms.
- Trained on simulated phantoms, the proposed method showed superior generalizability on a brain scan.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- the limited amount of testing on real patient data.
- It would be interesting to see intermediate network output at different iterations.
- Comparison of reconstruction time will be necessary.
- Trained on phantom data, how will the model perform on PET data of different tracers?
- Since the authors used 2D sinogram for training, is the model 2D? Any difficulties to generalize to 3D? If it is 2D, testing on realistic 3D data will be preferred.
- will the model be able to generalize to input data with different noise levels? If you trained the model with 20% of original counts, will the model still outperform MLEM/OSEM for input data with lower (10%) or higher (50%) counts?
- Authors only tested the model in a low-dose setting, will the model be able to generalize to different acquisition settings? such as fast PET?
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Code is available, so it should be easy to reproduce.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- Add more real patient data for testing the reconstruction algorithms. And also for images obtained with different imaging tracers.
- It would be interesting to see intermediate network output at different iterations.
- Add comparison of reconstruction time will be necessary.
- show results with input data of different input noise levels.
- Any idea why the model performance deteriorates after phase number 4?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This is an interesting paper unrolling the PET reconstruction in a neural network, preserving some mathematical interpretabilities. But there are still some weaknesses outlined in my comments above.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The authors propose an optimization-unrolled deep network for PET image reconstruction, which can be trained in an unsupervised fashion, leading to better performance than baselines. Although the reviewers and the AC consistently recognize the technical contribution of the propose method, its advantages over existing studies (especially those pure data-driven approaches) need more convincing justification.
Author Feedback
We appreciate the AC and reviewers’ time. We’re pleased that R1 and R3 highly endorse our “novel and interesting” formulation, highlighting the combination of deep unrolled method with unsupervised learning. R2 also highlights our novelty, with remaining concerns about supervised methods comparison. This rebuttal thoroughly addresses these questions and demonstrates with experiments. We’ll add these in the final version to enhance our contribution. Code will be released. (MR,R1,R2 Experiments): We’d like to declare that we weren’t set on strongly opposing the direct learning, the tone will be adjusted. Although our original focus is unsupervised learning and we’ve compared our method with the prevailing unsupervised PET reconstruction method (Hashimoto 2022), the proposed method can also be trained in a fully supervised manner (we call it SLDA). We evaluated the SLDA’s performance on the same datasets, comparing it to DeepPET (Haggstrom2019, prototypical end-to-end learning), and FBSEM (Mehranian2020, SOTA deep unrolled method). Quantitative results showed the mean(SD) PSNR and SSIM of SLDA (PSNR: 24.21±1.83, SSIM: 0.963±0.008) outperforming both DeepPET (PSNR: 23.40±2.87, SSIM: 0.962±0.011) and FBSEM (PSNR: 23.59±1.50, SSIM: 0.954±0.008). We also tested these methods’ generalization on patient data. Our proposed unsupervised method DULDA (PSNR: 25.86±0.68, SSIM: 0.868±0.009) outperformed DeepPET (PSNR:15.48±0.53, SSIM:0.669±0.009), FBSEM (PSNR:22.07±0.95, SSIM:0.849±0.010) and SLDA (PSNR:21.69±0.49, SSIM:0.813±0.008). This could stem from supervised methods, notably DeepPET, overfitting to an extent. These methods’ reconstructions of patient data preserve the sharp boundaries existing in the simulated training labels. However, unsupervised learning, focusing on the sinogram’s internal features, provides better generalization. We have tried to include Shen’s method in comparisons (R2). As the authors didn’t release their code, we are still working on developing the code in order to replicate the performance described in their paper. (R3 generalization): Our network unfolds the algorithm that solves a variational model, which integrates the prior knowledge of PET’s physical processes, to learn a regularizer for superior reconstruction. We believe that by training with large-scale heterogeneous data, the network’s ability to construct adaptive regularizers that capture common features, such as sparsity, and task specific features would greatly boost its generalizability. This method can be extended to 3D with 3D projection and convolution, but the training time and the memory requirements will be increased. (R2 L_measure): In unsupervised image reconstruction, L measure is key to loss design to boost the performance by converting the model output to sinogram domain and computing the loss as a data consistency constraint. (R1,R3 Data): The 128×128×40 phantom used in our study originated from Zubal phantom, added by two tumors. A Gaussian variable modelled the gray to white matter ratio, generating 40 unique 3D phantoms. A three compartmental model with Feng’s input function was used to simulate dynamic 18F-FDG scans, filling these phantoms with diverse time activity curves. The last frame of each scan were selected, yielding 1600 2D ground truth images. Sinograms were generated via the MIRT toolbox (Fessler). We’re collecting more patient data to train and test the model. (R3 Time): Unlike the Deep image prior method taking a long reconstruction time (training is equivalent to the reconstruction), our proposed method is considerably faster, averaging 0.0681s per single 2D image compared to MLEM’s 0.237s and EMTV’s 0.530s. (R2 Figure2): Apologies for the typo, the correct PSNR for MLEM-TV is 21.73dB, lower than our method’s 22.61dB (R3 Phase number): Increasing phase numbers can cause overfitting due to network capacity growth, leading to poorer performance. In this study, overfitting could occur when the phase number exceeds 4.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors did a thorough rebuttal to address the reviewers’ concerns, especially those regarding the comparisons between DULDA and fully supervised or pure data-driven approaches. I tend to side with most reviewers to accept this paper, mainly considering the practical meaningness of unsupervised recon and the interpretability of the unrolled deep network. A key limitation of the paper is that the evaluation was performed on synthetic data.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
In the rebuttal, the authors have addressed the raised comments carefully. Necessary experimental validations have been added. Nevertheless, I think more theoretical analysis or visualizations to demonstrate the effectiveness of the proposed method will be better.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper proposes an unsupervised unroll method for PET image reconstruction in a low-dose setting with superior results. Nonetheless, it remains uncertain whether the unrolled method surpasses alternative deep learning techniques. It is crucial for the paper to demonstrate whether the proposed method is solely applicable to unsupervised scenarios with specifically designed loss terms, or if it can outperform fully-supervised methods when provided with ground truth labels during training.