List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Noam Korngut, Elad Rotman, Onur Afacan, Sila Kurugol, Yael Zaffrani-Reznikov, Shira Nemirovsky-Rotman, Simon Warfield, Moti Freiman
Abstract
Intra-voxel incoherent motion (IVIM) analysis of fetal lungs Diffusion-Weighted MRI (DWI) data shows potential in providing quantitative imaging bio-markers that reflect, indirectly, diffusion and pseudo-diffusion for non-invasive fetal lung maturation assessment. However, long acquisition times, due to the large number of different ``b-value’’ images required for IVIM analysis, precluded clinical feasibility. We introduce SUPER-IVIM-DC a deep-neural-networks (DNN) approach which couples supervised loss with a data-consistency term to enable IVIM analysis of DWI data acquired with a limited number of b-values. We demonstrated the added-value of SUPER-IVIM-DC over both classical and recent DNN approaches for IVIM analysis through numerical simulations, healthy volunteer study, and IVIM analysis of fetal lung maturation from fetal DWI data. Our numerical simulations and healthy volunteer study show that SUPER-IVIM-DC estimates of the IVIM model parameters from limited DWI data had lower normalized root mean-squared error compared to previous DNN-based approaches. Further, SUPER-IVIM-DC estimates of the pseudo-diffusion fraction parameter from limited DWI data of fetal lungs correlate better with gestational age compared to both to classical and DNN-based approaches (0.555 vs. 0.463 and 0.310). SUPER-IVIM-DC has the potential to reduce the long acquisition times associated with IVIM analysis of DWI data and to provide clinically feasible bio-markers for non-invasive fetal lung maturity assessment.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16434-7_71
SharedIt: https://rdcu.be/cVRsD
Link to the code repository
https://github.com/TechnionComputationalMRILab/SUPER-IVIM-DC
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The authors propose a deep learning based method for the estimation of IVIM parameters. The proposed approach extends the state if the art by basically combining two previous approaches, where one works with a supervised loss measuring the difference between the ground truth IVIM parameters of a forward model and the parameters predicted by the DNN on the basis of the signal simulated using said model, and the other one works with an unsupervised loss measuring the difference between the measured/simulated signal and the signal generated by the DNN.
The authors perform experiments that are supposed to show that their extended approach yields more robust and accurate results as compared to a state of the art model.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The method is sound and the experiments are suitable for showing the improved performance of the approach. The method has the potential of reducing required acquisition times for the MR signal, which is clinically relevant.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The authors provide quantitative results to show the superior performance of their approach but they do not perform a statistical analysis of any kind to prove the significance of this performance gain. As it stands, the presented results only show that their approach might be better but this can only be confirmed with a thorough statistical analysis.
The correlation between f and the gestational age is pretty tenuous. I doubt that this relation is a useful indicator of clinical impact of the method. Maybe there are better measures relating the lung development to the signal/IVIM model parameters?
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors claim that their code will be made publicly available upon publication. The simulated data the authors are using is probably easy to reproduce. The authors do not mention that they are planning to publish the in vivo data used in their experiments, which probably makes the exact reproduction of their results impossible. AT least the experiments using healthy volunteers should probably be reproducible using own acquisitions though.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Apart from the points described above, it might be interesting to perform more extensive experiments and comparisons to other models, also classical non-DL methods. The authors only focus on one unsupervised approach.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I like the paper and the approach but without a proper statistical analysis it is hard to judge its real value. The paper would also benefit from including more baseline methods. This should also not be difficult to realize. The presented measure for clinical impact also seems not really useful.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
7
- [Post rebuttal] Please justify your decision
The authors provide results of statistical tests now and an improved version of their correlation analysis. With these corrections, I think the manuscript is indeed interesting for the community.
Review #3
- Please describe the contribution of the paper
Intravoxel incoherent motion imaging is an emerging MRI modality for the characterization of tissue microvascular perfusion and diffusion. Its clinical value is undoubtedly the ability to describe tissue viability and vascularity without the need for contrast agents, however, such measurements are often compromised by low SNR. The authors propose a DNN coupled with data-consistency term that may provide more reliable parameter estimates in low SNR settings. They test their hypothesis by undersampling a volunteer dataset. Furthermore, they evaluate the method in a challenging fetal MRI setting to characterize lung maturation. Compared to DNN based parameter estimation methods from the literature, the authors propose a supervised loss function coupled with a data-consistency term, which they hope to achieve more robust estimates in case of new data and in case of low SNR (fewer b-values). Their method yields lower standard error for the IVIM parameter estimates compared to the IVIM
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Novel concept for IVIM parameter estimation that would be applicable for the clinical routine since it needs fewer images to be acquired, saving some time
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
clinical scenario: the authors removed cases with motion, however, it is sure that the retained cases had some residual motion. It is also not clear what motion correction strategy they used. The low SNR of fetal MRI, particularly for motion-sensitive sequences like EPI, is often a result of spin dephasing. Therefore it is important to characterize residual motion and perhaps describe the reliability of the method as a function of motion or other artifacts
-
the authors tested their study on one participant only. It is not clear how the network would behave on new data
-
Fig 2. Is a bit misleading since the Y axis has been scaled and shifted in a way to exaggerate the magnitude of improvement: the NMSE of SUPER-IVIM-DC ranges from 0.26 to 0.3 while for IVIMnet is 0.25, however, in the plot, it looks like it is orders of magnitude better. The improvement for D is actually quite marginal.
-
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors provide sufficient details for reproducing their methodology, including a detailed description of the implementation of their code. While they pledged to make code available, they did not rely on open data, therefore a complete reproduction is not possible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Thank you very much for this high quality submission. The authors find my comments in the “strengths” and “weaknesses” section.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper shows clinical value and it would be relatively easy for other groups to implement the method.
- Number of papers in your stack
3
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #6
- Please describe the contribution of the paper
This paper introduced SUPER-IVIM-DC, a DNN approach for the estimation of the IVIM model parameters from DWI data acquired with limited number of b-values. Their numerical simulations and healthy volunteer study show that SUPER-IVIM-DC estimates of the IVIM model parameters from limited DWI data had lower normalized root meansquared error compared to previous DNN-based approaches.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper proposed a data-consistency term that enables the analysis of diffusion and pseudo-diffusion biomarkers based on DNN. Their work demonstrated the added-value of SUPER-IVIM-DC over both classical and recent DNN approaches for IVIM analysis through numerical simulations, healthy volunteer study, and IVIM analysis of fetal lung maturation from fetal DWI data. This paper has clinical significance in fetal lung maturity analysis.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
DNN-based methods have formalized the IVIM model parameters estimation problem as a prediction problem, such as [3][4][11]. The problem formulation is not novel. The clarity of the paper is not very clear. There are many mistakes in expression.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Their code and trained models will be made publicly available upon publication.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
This paper proposed a SUPER-IVIM-DC to alleviate the need to acquire DWI data with a large number of “b-values” by constraining the DNN training process through a supervised loss function coupled with a data consistency term. However, the SUPER-IVIM-DC is similar to the published work IVIM-NET [3] [11]. Compared with IVIM-NET, the correlation results of SUPER-IVIM-DC has no significantly improvements (0.239 vs 0.242).
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper focus on fetal lung maturity analysis based on DWI data. It has clinical significance in fetal lung maturity analysis.This paper proposed a data-consistency term that enables the analysis of diffusion and pseudo-diffusion biomarkers based on DNN. Their work demonstrated the added-value of SUPER-IVIM-DC over both classical and recent DNN approaches for IVIM analysis through numerical simulations, healthy volunteer study, and IVIM analysis of fetal lung maturation from fetal DWI data. However, the proposed model lack of novelty. It is an improvement of IVIM-Net. The improvement of comparison experiments is not significant.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Somewhat Confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
please comment regarding the opportunities for further statistical analysis, what metrics would be useful to show clinical impact beyond correlation? possible to go beyond testing on only one participant? novelty: are there valid improvements over baseline methods?
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2
Author Feedback
We thank the reviewers and meta-reviewers for insightful comments. We addressed each of them in detail below. In addition, we will review the final version thoroughly to make sure that there are no mistakes in expressions and language and that figures accurately reflect our results.
Novelty The practical novelty of our paper is the ability to predict IVIM model parameters from limited number of b-values(6) as compared to the original IVIM-NET and other works(>10). We achieved this by introducing a supervised loss coupled with a data-consistency term. While such combinations were proposed in various applications, we are the first, to the best of our knowledge, to propose this combination within the context of IVIM analysis. This results in improved capacity to apply IVIM analysis in clinical applications which involve motion such as fetal imaging.
Results Experiments 1&2 results demonstrate a clear improvement of SUPER IVIM DC over baseline methods(LS and IVIMNET). Further, while overall linear correlation presented in Fig. 4(experiment 3) is relatively low, a deeper analysis show that by dividing the GA axis of Fig. 4 into two stages of the fetal lung development-the Canalicular phase(weeks 16-25) and the Saccular phase(week 26-34) our approach demonstrate a better correlation with GA (0.733 with our SUPER IVIM DC vs. 0.679 and 0.49 with IVIMNET and LS) for the Canalicular phase of development. This result correlates with the intensive angiogenesis starts the formation of a dense capillary network in the Canalicular phase. Thus, has the potential to serve as a non-invasive prenatal diagnostic tool to study lung development and to quantify fetal growth. In turn, these data could have clinical relevance as benchmark values to distinguish normal fetuses from pathological fetuses with deformity. Further, due to the difference between the lung development phases, an exponential model is better suited for analysis rather than a linear model as we used. We will modify the analysis to better reflect these results in the final version.
Statistical analysis The focus of the paper is technical development. We used simulation and healthy volunteer study to demonstrate the improved accuracy and precision of our approach. The reduction in NRMSE achieved by SUPER IVIM DC was statistically significant (p«0.01) for D and f parameters in both experiements. The difference in D* was not statistically significant. The improved correlation between IVIM model parameters and GA (see above) indicates a potential clinical application. We agree with the reviewer that a future clinical study assessing group differences between abnormal fetal lung development subjects and a control group using standard statistical measures (t-test,ROC,F1,Cohen’s kappa score) is required to demonstrate actual clinical impact. However, this is beyond the scope of our MICCAI submission.
Extensive experiments and comparisons to other models In our preliminary studies we have indeed tried several classical non-DL methods (NLLS with different solvers: BOBYQA,LM,TRF) and Bayesian approaches to estimate IVIM parameters. NLLS with the TRF minimization algorithm ‘trf’ gave us the best fit and that why we took its results as the baseline NLLS approach. Due to limited space in the MICCAI submission, we have not shown a comparison to the rest of the methods as this was already presented in the original IVIMNET papers(Barbiery et al,MRM 2020 and Kaandorp et al,MRM, 2021).
Test data While experiment 2 presents results on a single subject, it is impotant to note that we used 6 different ROI selected from different slices and organs to demonstrate the potential added-value of our approach. We are planning to expand our analysis as we are scanning more subjects to further expand our assessment.
Reproduction of results As mentioned, we are planning to publish our code and numerical simulations upon acceptance. The clinical data is confidential due to IRB committee requirements.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
borderline
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
low
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The concept of predicting IVIM model parameters from limited number of b-values is novel and the experiments are suitable for showing the improved performance of the approach. The authors satisfactorily address the reviewer’s comments of statistical analysis, results, and test data
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
NR
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors’ rebuttal answered most of the concerns. Their response to novelty part is convincing about using a smaller number of b-values as well as for the loss function incorporation for IVIM analysis. They also added statistical results. They should add the “Extensive experiments and comparisons to other models” to the camera ready.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3