Authors

Xiaowu Sun, Li-Hsin Cheng, Sven Plein, Pankaj Garg, Rob J. van der Geest

Abstract

Four-dimensional flow magnetic resonance imaging (4D Flow MRI) enables visualization of intra-cardiac blood flow and quantification of cardiac function using time-resolved three directional velocity data. Segmentation of cardiac 4D Flow data is a big challenge due to the extremely poor contrast between the blood pool and myocardium. The magnitude and velocity images from a 4D Flow acquisition provide complementary information, but how to extract and fuse these features efficiently is unknown. Automated cardiac segmentation methods from 4D Flow MRI have not been fully investigated yet. In this paper, we take the velocity and magnitude image as the inputs of two branches separately, then propose a Transformer based cross- and self-fusion layer to explore the inter-relationship from two modalities and model the intra-relationship in the same modality. A large in-house dataset of 104 subjects (91 182 2D images) was used to train and evaluate our model using several metrics including the Dice, Average Surface Distance (ASD), end-diastolic volume (EDV), end-systolic volume (ESV), Left Ventricle Ejection Fraction (LVEF) and Kinetic Energy (KE). Our method achieved a mean Dice of 86.52%, and ASD of 2.51 mm. Evaluation on the clinical parameters demonstrated competitive results, yielding a Pearson corre-lation coefficient of 83.26%, 97.4%, 96.97% and 98.92% for LVEF, EDV, ESV and KE respectively.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16443-9_36

SharedIt: https://rdcu.be/cVRyQ

Link to the code repository

https://github.com/xsunn/4DFlowLVSeg

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The paper describes a novel approach for segmenting the left ventricle from MRI 4D flow images. Using deep learning based on a combination of U-Net architecture with Transformer components and feature fusion layers, the algorithm tackles the challenges of poor anatomical appearance in 4D flow images.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- generally an interesting problem, as it brings cardiac segmentation to a novel imaging method with potentially brings new diagnostic and therapeutic benefits
- novel approach to concurrently learn anatomical appearance from multiple images
- strong evaluation both in terms of technical (DICE & surface distance) and clinical measures (ejection fraction, volumes etc. - both absolute errs as well as statistical metrics)
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Overall a pretty good paper, I am not sure what to criticize ;-)
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

What can I say, the code and models are available on Github - as good as it gets! :-)
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

Please clarify whether the LV was annotated for all the 30 frames, or only for a subset?
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

See main strengths above: it brings a seemingly novel approach (transformer based multimodality learning) to a challenging problem (low quality imaging) to generate convincing results quantified with relevant metrics
Number of papers in your stack

4
What is the ranking of this paper in your review stack?

1
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

This paper proposes a Transformer based segmentation model for 4D flow MRI. 4D flow MRI is a recent blood flow velocity diagnostic on which automatic assessment has not been fully investigated. The authors design two self- and cross-attention-based methods to fuse the information from different modalities in 4D flow, and perform evaluations on a large in-house dataset.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. Application novelty. The automatic assessment on 4D flow MRI has been little investigated before. As the authors states, it may be the first work for this application.
2. The employed techniques are sound. The authors introduce different attention modules for intra- and inter-modality feature extraction and fusion. The results also support the performance improvement thanks to the attention mechanism.
3. The choice of evaluation metrics. Since the most significant of the paper is the application novelty, the authors employ both geometrical and clinical metrics.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Minor concern: since this is a relatively recent application, the work has little related work for comparison. Moreover, the compared methods are more considered baselines, hence, the evaluation against SOTA is not available.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The experiments are performed on a private dataset while the code is avaible on git.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

The authors have well proved the superiority of the proposed model. However, I would suggest adding the results on a single modality (2D MRI without velocities) with U-Net. These experiments can prove the effectiveness of 4D flow MRI against conventional CMR.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

7
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Two major factors affect my recommendation:
1. The novelty of this application.
2. The quality of evaluation.
Number of papers in your stack

4
What is the ranking of this paper in your review stack?

2
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #4

Please describe the contribution of the paper

The authors propose a novel method to take the velocity and magnitude image into a unified segmentation framework to achieve the automatic segmentation of the LV directly from 4D Flow. The experiment demonstrates that their method outperforms the best previously published results for this task.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

They propose the first study to segment the LV directly from 4D Flow MRI data. They propose a Transformer based cross- and self-fusion layer to explore the inter-relationship from two modalities and model the intra-relationship in the same modality.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The writing is not rigorous enough in some sections of this paper.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The description of the proposed method is clear and the paper is reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1. The figures in the paper is not clear enough, the font in all figures is inconsistent with the text, and there is some ambiguity. It is recommended to revise all figures carefully.
2. “dividing feature maps into patches leads to loss of spatial information” should be given detailed evidence to prove this point through ablation experiments.
3. The paper mentions “a learnable positional encoding sequence”, however the detailed descriptions are not given.
4. The typesetting is not neat enough. It is recommended to use “latex” for typesetting.
5. what the meaning of “in 91 182 annotated pairs”? is it a writing error because of the large space in one number? Please check it.
6. The sentance “KE was normalized to EDV as recommended by other researchers” is too casual and not rigorous, please explicate this point by citation who recommend or which paper inspired you to do like this.
7. “we did not employ any data augmentation methods to enlarge the dataset”, in future I suggest you to conduct some data augmentation like nnU-net to improve the performance of the proposed method.
8. “All of those Pvalues are larger than 0.05, which confirmed that there is no significant different…” is different from the “Pvalue was computed using Wilcoxon-signed-rank test. P<0.05 indicate a significant difference between two variables”. Please check which one is wrong.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The novelty from application is relatively good.
Number of papers in your stack

2
What is the ranking of this paper in your review stack?

2
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The authors propose a transformer based method to segment flow MRI images. All reviewers agree on the soundness of the proposal and only very minor criticisms have been found. I would suggest authors to try to address the minor concerns about clarity in theri final version.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

2

Author Feedback

Thank you for the reviewers’ positive comments. We explain the reviewers’ comments individually. Reviewer #1 Q1.The LV was annotated for all the 30 phases. Reviewer #3 Q1: It is a good idea to compare the segmentation results derived from conventional 2D MRI and 4D flow MRI, but it is not comparable. The segmentation results on the 2D MRI definitely are much better than 4D flow data due to its excellent image quality. However, the 4D flow data provides additional information in the hemodynamic parameters analysis and the blood flow visualization, which is over conventional cine MRI.

Reviewer #4 Q1 and Q4. We will use “Latex” to solve the format problem in the final revised version. Q2. “dividing feature maps into patches leads to loss of spatial information” this conclusion has been proved in many Transformer based papers, such as the Ref 6,7, 8. Q3. The standard learnable 1D position embeddings used in ViT is introduced in our work. Q5. It means 91,182. Q6. The reference which recommends the normalized KE is as follows. Garg, Pankaj, et al. “Left ventricular blood flow kinetic energy after myocardial infarction-insights from 4D flow cardiovascular magnetic resonance.” Journal of Cardiovascular Magnetic Resonance 20.1 (2018): 1-15. Q7. Data augmentation method is useful, but the data augmentation methods in nnU-Net cannot be applied for the 4D flow data directly. Because the velocity images do not only have the magnitude but also have the directions. If we rotate the images, the velocity direction will be changed. Therefore, a much more careful data augmentation method should be proposed specially for 4D flow data. Q8. Both these sentences are correct. In theory, if the P<0.05, it demonstrates that there is a significant difference between two variables. Our experiments showed that the P-values are larger than 0.05, it means there is no significant difference between our prediction and the ground truth.

back to top

Transformer based feature fusion for left ventricle segmentation in 4D flow MRI