List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Qi Chang, Zhennan Yan, Mu Zhou, Di Liu, Khalid Sawalha, Meng Ye, Qilong Zhangli, Mikael Kanski, Subhi Al’Aref, Leon Axel, Dimitris Metaxas
Abstract
Joint 2D cardiac segmentation and 3D volume reconstruction are fundamental to building statistical cardiac anatomy models and understanding functional mechanisms from motion patterns. However, due to the low through-plane resolution of cine MR and high inter-subject variance, accurately segmenting cardiac images and reconstructing the 3D volume are challenging. In this study, we propose an end-to-end latent-space-based framework, DeepRecon, that generates multiple clinically essential outcomes, including accurate image segmentation, synthetic high-resolution 3D image, and 3D reconstructed volume. Our method identifies the optimal latent representation of the cine image that contains accurate semantic information for cardiac structures. In particular, our model jointly generates synthetic images with accurate semantic information and segmentation of the cardiac structures using the optimal latent representation. We further explore downstream applications of 3D shape reconstruction and 4D motion pattern adaptation by the different latent-space manipulation strategies. The simultaneously generated high-resolution images present a high interpretable value to assess the cardiac shape and motion. Experimental results demonstrate the effectiveness of our approach on multiple fronts including 2D segmentation, 3D reconstruction, downstream 4D motion pattern adaption performance.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16440-8_54
SharedIt: https://rdcu.be/cVRwF
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The contribution of the paper is a method for synthesizing 2D cardiac MR images by interpolating an optimized latent embedding, which has been obtained by an encoding-generating network architecture. A segmentation network is furthermore trained, using the synthesized MRIs as input. The interpolation technique allows for reconstructing (super-resolving) 3D images and segmentation, as well as transferring information from subjects with longitudinal data, to subjects without such data. The method is validated on the tasks of: segmentation, 3D volume reconstruction, and motion pattern adaptation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The idea of reconstructing 3D images and segmentations from an optimized latent embedding, as well as transfering longitudinal information, is interesting and novel. Using the generator of a GAN for decoding the latent embeddings gives a rich model for synthesizing images. The validation is thorough and shows good results on the experiments performed.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The paper is good overall, however the following issues need clarification:
Dice on the segmentation produced by the generated images is a good proxy for the quality of the synthesized MRIs; however, I am surprised that no metrics on the image was computed in the experiments (e.g., SSIM, MSE, etc), as this could be a more direct measure of the synthesizing quality? Why was this not performed?
Some test for statistical significance (e.g., Wilcoxon) would provide improved interpretability of the results.
The loss function contains several balancing hyper-parameters, but the paper does not state how these were chosen (nor their values); please specify. Furthermore, is there also some weighting between the segmentation part of the loss, and the image part?
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Good, the author’s answers to the reproducibility checklist correspond to the content of their paper.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
“minimal radiation exposure”, MRI has no radiation exposure.
The input images (x) were motion corrected by a preprocessing step I assume (see Fig 1)? Please specify how this was done.
“resampling function that align”, it is not clear to me if this step performs some registration, or simply resamples the data? Please clarify.
The Settings paragraph of Section 3.1 is a bit cluttered because the five experiments are not in an enumerate{} environment. If space allows, please consider changing this for improved readability.
“3D DICE”, the Dice metric is agnostic to the dimensionality of the inputs.
It is surprising to me that linear interpolation seems to perform substantially worse than computing Dice on the original data, I would expect a very similar performance, or slightly better for linear interpolation (often used as a simple baseline in super-resolution work for example). Any ideas why this might be?
Fig 4: would be easier to interpret if each plot had a title (e.g., ‘normal-to-normal’, ‘diseased-to-normal.’)
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The methodology is novel and the validation shows encouraging results. The validation could have been more complete.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
This paper presents an end-to-end latent-space-based framework, DeepRecon, that generates multiple outcomes, including image segmentation, 3D reconstructed volume, and motion adaptation. Some cardiac datasets have been used for evaluation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
The motivation of this work looks interesting and useful, i.e., joint 2D cardiac segmentation and 3D volume reconstruction are fundamental to building statistical cardiac anatomy models and understanding functional mechanisms from motion patterns.
-
The datasets used for evaluation look very large.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
The novelty in the method and network design is not high.
-
Some technical components need to be clarified.
-
The evaluation and comparison are not sufficient.
More detailed comments are given in the following Sec. 8.
-
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Seems okay.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
There are some major concerns of this paper:
-
The general pipeline of this method is simple. It includes an autoencoder and U-Net network. There is not much novelty in the method and network design. Furthermore, the two tasks, i.e., image representation/reconstruction and segmentation, work sequentially from their workflow. It is unclear how they work jointly as they proposed?
-
In the proposed framework, it is unclear how it can realize high-/super-resolution 3D image generation (no technical description on it). There is also no evaluation on this task as they proposed.
-
In the 4D motion adaption, how to guarantee the smoothness and accuracy along the time space is unclear. From the supplemental video, we can see that the organ shape results look noisy and jittering.
-
What are the training steps on the comparison methods (e.g., DirectSeg, SemanticGAN, W+SegNet)? We can see that the proposed method has two training step settings. But it is unclear what is the setting for the comparison methods.
- In general, the experiment and evaluation sections are a bit weak:
- The evaluation only includes limited segmentation methods for comparison, more state-of-the-art deep learning-based 3D image segmentation methods should be included.
- The evaluation only includes some basic reconstruction methods (e.g., Linear Interp, CPD) for comparison, but there is no comparison with state-of-the-art deep learning-based 3D image reconstruction methods.
- The visualization results in Fig. 2 and Fig. 3 are difficult to see the difference between different methods.
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Overall, this paper proposes a latent-space-based generation method for multiple medical image tasks. However, there are some concerns on the limited novelty, some unclear technical components, and insufficient evaluations.
- Number of papers in your stack
6
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
4
- [Post rebuttal] Please justify your decision
Thanks for the authors’ hard-working feedback and explanation. After reading the rebuttal, I am on the fence of this paper.
One concern is not well responded about the limited evaluations in comparison. Another is responded that there is no ground truth for high-resolution images and 4D motions (4D smoothness and accuracy) in their work.
So in this case, the practical value and potential in the treatment planning of cardiovascular diseases are not very clear. If ACs and all other reviewers agree to accept this paper, I am fine with it.
Review #3
- Please describe the contribution of the paper
In this paper a 2D cardiac segmentation and a 3D volume reconstruction tool is shown, and after this they obtain the 4D motion pattern shown in volume flow graphics, to probe the correct segmentation and reconstruction.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Authors show the path that clinicians need to obtain a complete analysis preview of the heart. And get to obtain graphic volume flow. Although it is a known problem, they propose a different solution. The solution shows the workflow necessary to obtain a final product for clinicians. so it could have a immediate application.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The paper does not have great novelty in the technical area, it makes an ensemble of already existent architectures, to improve a known problem with a different solution. The solution could be too elaborated, although it has very good results. It is missing more information about parameters and the type of infrastructure needed. Some steps are not so clear, though I understand that 8 to 10 pages is not enough space to describe it completely.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Some parts are not so easy to reproduce specially concerning the methods used, because information is not enough, and the complexity of the architectures used, it may need more information about hyperparameters. The expertise reached by their tool it is based on coupling different architectures for the segmentation and reconstruction tasks previously known.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The complete system used are a good ensemble of the methods with very good results of the system is very good. Nevertheless, the novelty is not the strong part of the paper.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper presents an end-to-end latent-space-based framework, DeepRecon, that generates multiple outcomes, including image segmentation, 3D reconstructed volume, and motion adaptation. The reviewers agreed on that this work is well-motivated and interesting. However, they also pointed out the limited novelty of this work for MICCAI, missing results on the quality of synthesized images, and insufficient details and evaluation.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
9
Author Feedback
We appreciate all valuable comments from the reviewers (R1 to R3) and the meta review. The following is our response to the concerns on novelty clarification, synthesized images quality, insufficient details and evaluation. All updates will be made in the final submission.
Novelty clarification (R2, R3) To our best knowledge, this work is the first effort to explore latent code interpolation for 3D cardiac reconstruction and the first study using latent code manipulation for 4D cardiac motion adaptation. In addition, we proposed a normalized local cross-correlation loss in learning latent codes to ensure the synthetic images have accurate anatomical reconstruction in the heart area. To reiterate, we appreciate the support from R1: ”the idea of reconstructing 3D images and segmentations from an optimized latent embedding, as well as transfering longitudinal information, is interesting and novel”. After assessing several candidate architectures, we found our approach is simple yet powerful to outperform other sophisticated solutions like SematicGAN and W+SegNet based on a large cohort (6,846 training cases).
Synthetic image quality (R1) We carefully verified the PSNR and SIMM in the experiment, showing the improvement of the synthesized image quality. Details are: SematicGAN (PSNR: 19.66, SIMM:0.52); W+SegNet (PSNR: 19.57, SIMM:0.42); DeepRecon_no (PSNR: 18.21, SIMM:0.61); DeepRecon_1k (PSNR: 25.57, SIMM:0.654); DeepRecon_10k (PSNR: 27.68, SIMM: 0.72). We have already shown the FID score as 18.09 in our article as a major quality metric. The synthetic images are only intermediate outputs of our method.
Super-resolution 3D image generation (R2) We generate the super-resolution image along the z-axis by latent space interpolation since the original MRI has a lower through-plane resolution. There are no ground truth images (high-resolution short-axis images) to compare, as detailed in Sec 2.3 and Fig.3. Thus, learning-based 3D reconstruction is not applicable. Our analysis offers a new avenue for evaluation to use long axis view labels, enabling precise measurement of the super-resolution and 3D reconstruction results.
Joint segmentation and reconstruction (R2) We adopt a structural generative method to jointly generate 2D segmentation and 3D volume (by interpolating latent codes) simultaneously in the evaluation stage. Thus, we do not require another step to reconstruct the 3D volume from the sparse 2D segmentations.
Experimental details on 3D data (R2) We have evaluated the 3D-UNet segmentation on the original low-resolution Cine data, and additional results are LVC [DICE:0.938 (0.046), HD95: 3.78 (11.4)]; LVM [DICE: 0.861 (0.03), HD95: 2.31 (1.8)]; RVC [DICE: 0.90 (0.03), HD95: 5.79 (3.9)]. Since there is no high-resolution 3D data for training the 3D-UNet, it can not be directly applied to the generated high-resolution 3D images to obtain the 3D segmentation volume. The latent representations are learned from 2D images only. Thus we reported SemanticGAN and other latent-code-based methods for a fair comparison.
4D motion adaptation smoothness and accuracy (R2) Please note that there is no ground-truth data to measure the 4D smoothness and accuracy along the time dimension. We made an effort to qualitatively adapt the motion pattern from one subject to another.
Training steps of the comparison methods (R2) The DirectSeg uses the same architecture as the Seg in our method and real images as the input. SematicGAN [11] only has one training step but with lower dice accuracy. W+SegNet uses the latent code of StyleGAN as input to train a Seg network.
Statistical tests for Table 1, Correction of Table 2 (R1) Our DeepRecon_1k, DeepRecon_10k are statistically better than SematicGAN in all metrics (p<0.05) and comparable with DirectSeg (p>0.05) in most metrics. The corrected linear interpolation results are comparable to the original method: [avg: 0.78 (0.08); 2ch: 0.797 (0.05); 3ch: 0.773 (0.07); 4ch:0.768 (0.102)]
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have satisfactorily addressed the concern of R2. I am glad to accept the paper for publication at MICCAI 2022.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
1)The method of reconstructing images and segmentations is novel. The GAN for decoding the latent embedding is well-motivated in the paper. 2)The validation shows good results on the experiments performed and author explain and analyse the reason. 3)The paper test the model on a large dataset, so the solution shows the workflow necessary in the actual immediate application
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3