List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Maryam Toloubidokhti, Nilesh Kumar, Zhiyuan Li, Prashnna K. Gyawali, Brian Zenger, Wilson W. Good, Rob S. MacLeod, Linwei Wang
Abstract
Prior knowledge about the imaging physics provides a mechanistic forward operator that plays an important role in image reconstruction, although myriad sources of possible errors in the operator could negatively impact the reconstruction solutions. In this work, we propose to embed the traditional mechanistic forward operator inside a neural function, and focus on modeling and correcting its unknown errors in an interpretable manner. This is achieved by a conditional generative model that transforms a given mechanistic operator
with unknown errors, arising from a latent space of self-organizing clusters
of potential sources of error generation. Once learned, the generative model can be used in place of a fixed forward operator in any traditional optimization-based reconstruction process where, together with the inverse solution,
the error in prior mechanistic forward operator can be minimized and the potential source of error uncovered. We apply the presented method to
the reconstruction of heart electrical potential from body surface potential.
In controlled simulation experiments and in-vivo real data experiments, we demonstrate that the presented method allowed reduction of errors in the physics-based forward operator and thereby delivered inverse reconstruction of heart-surface potential with increased accuracy.
keywords: Inverse imaging, Forward modeling, Physics-based.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16452-1_44
SharedIt: https://rdcu.be/cVVpZ
Link to the code repository
https://github.com/miccai2022IMRE/miccai_2022_IMRE
Link to the dataset(s)
https://github.com/miccai2022IMRE/miccai_2022_IMRE
Reviews
Review #1
- Please describe the contribution of the paper
The paper extends the DAECGI method to improve the reconstruction of the forward operator when errors are present in the initial estimation. The method cyclically updates the estimated corrected forward operator by generating a latent vector z based on the initial forward operator and the current estimated forward operator. By using an SOM, the types of errors can be revealed by examining the latent vector z for a set of initial and final forward operators.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The method is capable of correcting errors in the forward operator when solving inverse problems. The authors claim that these errors are difficult to account for when solving inverse problems and instead update the forward operator in a cyclical process.
By clustering in the latent space of the vector controlling the update to the forward operator, the type of error in the original forward operator can be evaluated.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
While the paper does propose a method for interpreting the refinements and sources of error in the original modeling of the forward operator, it is unclear why this interpretability is clinically useful. It seems interesting from a methodological standpoint to understand the sources of error in the forward operator, however the paper does not make it clear if there is any clinical use for this information. If interpretation of the correction of the forward operator has a direct usefulness it should be made clear. The motivation for the interpretability of the method should be much more clearly motivated as this is emphasized in the title and paper itself.
The implementation details are not fully described and sometimes unclear. For example, I am uncertain if the SOM mapping plays a central role in the generative model or is performed after the fact to add interpretability to the latent variables z. Similarly a lot of training details and architectural information are missing.
The results are difficult to interpret and do not make the overall effectiveness of the method obvious. Figures like figure 2, which shows the forward operators and is not human interpretable, could be replaced with more easily interpretable figures such as figures of the simulated electrocardiographic sources displayed on the heart and their reconstructions. This would help the audience verify that the inverse solver is working as correctly.
The authors train on a variety of error sources. However, it is unclear if these errors correspond to those seen in actual practice. Motivation of these choices would be helpful to validate the experiments. Similarly, it would be interesting to see the quality of the reconstructions under different error conditions. For example, translations or rotations could be gradually increased and the error in the reconstruction could be plotted to show if the model is more capable of adjusting to different ranges of errors.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Given the level of detail in the paper, it would be difficult to reproduce the author’s work. The architectures and other training details aren’t mentioned. In addition, I am unclear on some portions of the method itself. The data used seems to be mostly synthetic so it seems like it would be possible to reproduce the data.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
In the introduction, the authors claim that there has been “substantial interest” in combining deep learning with prior knowledge of imaging physics. However they only cite two examples. More review of the prior work is necessary.
Similarly, the method presented is not contrasted with prior approaches in the literature. Such a comparison would help evaluate the method’s novelty.
More discussion of the sources of error and how they affect inverse problems in general would be helpful for assessing the applicability of the method.
The comparison to the DAECGI method given in the introduction is not very clear. It is important to make the comparison very understandable, as the present method seems to be a direct extension of the DAECGI method. It seems that the major difference between the two is that the presented method uses the both the original forward operator and the derived forward operator in inferring the latent vector z. The authors seem to claim that this is the innovation that allows for the sources of error to be identified.
The SOM method applied to the latent vector seems to me to be a post hoc method to add to the interpretability of the algorithm. From the description in the text, it seems like the SOM clustering is central to the generative model. I may not fully understand but it seems like the SOM is applied after inference. If this is the case, any clustering algorithm could be used. The SOM method is presented in the text alongside the generative model but it is not clear that the SOM plays a role in the generation of the forward operator. If it does, it needs to be clarified. If it does not it should be presented in a separate subsection.
I’m not sure I understand the motivation for using a U-Net as a generative model. It seems that any autoencoder architecture would be as applicable as a U-Net. Some motivation should be given for this choice. Furthermore, as the U-Net uses convolutions, it is not clear why the convolution operation is appropriate for the forward matrix H.
In equation (5) the authors add a hyperparameter to the loss function. With the beta parameter I don’t think the equation is quite equal to the ELBO and I’m not sure that the bound in the equation still holds. I think the addition of the hyperparameter is fine and makes sense to control the degree to which the KL term effects the loss, however I believe some care should be taken to ensure that it is described in a way that is mathematically correct.
As the model is fairly complicated, I think the training procedure could use explicit description. The main text details how the forward and inverse problems are solved but I am unclear on how the U-Net and variational inference network q(z Hi, Hf) are trained. Are the networks trained simultaneously with simulated ground truth forward operators and operators with errors? What are the settings used for the hyperparameters beta and lambda_reg? What are the architectures used for each component. These details would be helpful in the understanding of the overall method and increase its reproducibility. If there is not space in the main text they should be provided as supplemental materials. The authors claim the cyclical inverse estimation procedure “is expected to continuously reduce the error.” I’m not sure what this means. If the error is guaranteed to decrease they should explicitly state that. If not, they should also explain why the approach they have chosen is appropriate. This step seems important to describe in detail as it seems possible that a cyclical update such as this might not converge without some guarantees or some amount of appropriateness to the task. Convergence plots could be shown for these cyclical updates to verify the procedure is working as intended.
I am unable to understand what is being depicted in Figure 2. The images are small and difficult to read. In addition the visualization of the forward operator is not human interpretable. Instead the authors should choose to display some metrics that might more informative.
Similarly, Figure 5 is very difficult to interpret. I am not sure how to read these images.
In figure 4, the reconstruction error of the different methods is presented. It may be helpful to show the simulated and inferred u’s if they are interpretable. This would show that the model accurately reconstructs the underlying latent cartographic source data and that the inverse solutions are as expected.
The paper contains spelling errors and awkward formatting such as “variatioanl,” “gird,” and “Real Data : .” The manuscript should be more closely spell and formatting checked.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I think the method is very interesting. However I think the presentation could be improved to make a much stronger paper. The differences between the previous DAECGI model are not obvious so it is difficult to assess the novelty of the proposed method, as these methods are closely related. Some of the training details are hard to infer from the exposition in the paper. Furthermore, the presented results could be strengthened by adding more interpretable figures and descriptions. In addition I think some of the paper organization, especially with regard to the SOM procedure, could be improved. For these reasons, I would recommend that the paper be rejected so that the authors might strengthen it and resubmit an improved and stronger version.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
Use prior knowledge to improve the performance.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
A new model was proposed to reconstruct medical images.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The results are not adequate to fully support the claims in the manuscript. Solid results are required to justify the advantages of the proposed model.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
There is no reproducibility problem.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
In this manuscript, a new model was proposed to reconstruct images by relying on the forward operator and residual errors simultaneously.
It is tough to see that the proposed model is more interpretable. The interpretability was not demonstrated in the manuscript. I suggest removing this word from the title. Otherwise, solid results demonstrating interpretability should be supplemented.
The evaluation of the model is not complete. More quantitative results should be added to justify the advantages of the proposed model.
The numbers and fonts in the figures are too small. It would be better to have larger sizes.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
A new model was proposed, but there are no solid results to support the superior of the model.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
The paper described a physics-inspired neural method for the inverse problem of ECG. The method shows advantages over a pure physics-driven method (fully data-driven) or a pure neural network-based method (requires less training samples and has more interpretability).
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is well written and structured. Interpretability is highly desired in clinical settings.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The authors could have provided more results (on both synthetic and real data) using the remaining space.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Synthetic and public real data are used. Codes are also submitted for review.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
see point 3,4,5
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper addressed an important topic in electrocardiogram. The method bears some novelty and is preliminarily validated.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper received a mixed review of positives and negatives. All reviewers agreed that the idea of reducing unknown errors of the forward operator in image reconstruction is very appealing and the authors introduced a decent solution to that. While the presented methodology is technically solid, several concerns arise from different aspects: (i) the unclear picture of how the interpretability will be used in real clinical applications, (ii) missing implementation details, and (iii) a lack of sufficient experimental results to demonstrate the advantages of this proposed model and the difficulty to interpret current results. The authors are encouraged to respond to all reviewers’ questions and concerns, particularly those from R1 & R2.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6
Author Feedback
We appreciate the constructive comments and provide clarifications below. Interpretability (MR/R1/R2): The interpretability of IMRE refers to two aspects. One is interpretable error modeling. In response to R1, the sources of error we considered indeed represent the most prominent sources of errors in ECG imaging as concluded in literature [1]. We will add this reference to clarify our choices of errors. These errors arise from different stages in the ECG imaging pipeline such as data acquisition, image segmentation, and forward modeling: detecting error types will inform which part of the pipeline needs to be improved in a clinical workflow. This motivation was given on pp2 - paragraph 3 and will be expanded in our final version. The second aspect of interpretability refers to the fact that our generative model is not a black-box, but preserves imaging physics and focuses on explaining common sources of errors. Reproducibility/Methodological choices (MR/R1): Our code and data are public in an anonymized github link (https://github.com/miccai2022IMRE/miccai_2022_IMRE). We will include architecture and training details in the final version. a) SOM is done post-hoc and chosen to improve visualization and interpretation of cluster relationships. We will reorganize the method section as suggested. b) We use convolutions to extract spatial information in the heart and torso underlying the forward operator. We use Unet and its skip connection to supplement the decoder with information about the initial forward operator, so as to focus on error modeling. We train the Unet and variational network simultaneously using simulated ground truth. We will clarify that adding a hyperparameter indeed modifies the ELBO. c) The forward operator and inverse solutions are optimized iteratively. There is no theoretical guarantee of convergence, although empirically we have not observed divergence. We will revise the wording and add convergence plots. Comparison with literature (R1): Limited works used deep learning to improve forward modeling. We compared to: 1) traditional works using a fixed forward operator as baseline, and 2) deep learning based forward modeling (DAECGI). The novelty of IMRE over DAECGI includes: 1) DAECGI learns a black box to build the forward operator from geometry, whereas IMRE respects the physics-based forward operator; this allows IMRE to be applicable to other forward operators that are not a function of geometry; 2) DAECG is not able to reconstruct H_i without regularization; 3) IMRE included error type detection. Results: We focused on evaluating IMRE in two aspects: generating forward operators and improving inverse solutions. a) We use Fig 2 for a visual comparison of the forward operators generated by IMRE and DAECGI. We will add informative axis titles, and include quantitative metrics such as RMSE: IMRE: 0.01 vs DAECGI: 0.04. b) Fig. 5 includes examples of reconstructed cardiac electrical sources in clinical data. We will add the Euclidean distance of the predicted pacing sites to the ground truth: top – baseline: 26.3, IMRE: 12.7; bottom – baseline: 75.3, IMRE: 34.7. DAECGI failed in producing both results (Fig 5). We will add similar visualizations in synthetic cases per R1’s advice, in addition to the quantitative summary in Fig. 4. c) We appreciate R1’s suggestion for examining reconstruction vs error magnitude. We found that IMRE improved inverse solutions more for higher error magnitudes: e.g.,for z_translation (mm), IMRE offers a correlation improvement by 2.5% over the error range of (1,10) vs. 30% improvement for (30,40). With these additions/modifications, we hope it would be evident that IMRE outperforms DAECGI in both generating forward operators and inverse solutions.
[1] R. Throne and L. Olson, ”The effects of errors in assumed conductivities and geometry on numerical solutions to the inverse problem of electrocardiography,” IEEE Trans Biomed Eng, vol. 42, no. 12, pp. 1192–1200, 1995.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The overall opinion of all reviewers is in favor of accepting this paper, under the constraint that the authors will incorporate all changes in the camera-ready version as they committed in the rebuttal.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Authors have rebutted al main points by reviewers. Many of such points seem to be satisfactorily addressed; others are promised to be changed in the final manuscript, but there is no guarantee that this will be done. I believe this paper is borderline.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
10
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The rebuttal adddressed many of the point raised by the reviewers, although there are several modification which could be likely addressed only through a major revision (as recommended by R1). Nevertheless, the discussion post-rebuttal seems positive about the possibility of sufficiently improving the manuscript for presentation to the conference.
Overall, the paper is interesting while the proposed methodological approach is to my opinion relevant to the MICCAI community, and could stimulate a positive discussion to the conference.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6