List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Zechen Zhao, Heran Yang, Jian Sun
Abstract
Multi-modal Magnetic Resonance Imaging (MRI) plays a crucial role in brain tumor segmentation. However, missing modality is a common phenomenon in clinical practice, leading to performance degradation in tumor segmentation. Considering that there exist complementary information among modalities, feature interaction among modalities is important for tumor segmentation. In this work, we propose Modality- adaptive Feature Interaction (MFI) with multi-modal code to adaptively interact features among modalities in different modality missing situations. MFI is a simple yet effective unit, based on graph structure and attention mechanism, to learn and interact complementary features be- tween graph nodes (modalities). Meanwhile, the proposed multi-modal code, indicating whether each modality is missing or not, guides MFI to learn adaptive complementary information between nodes in differ- ent missing situations. Applying MFI with multi-modal code in different stages of a U-shaped architecture, we design a novel network U-Net-MFI to interact multi-modal features hierarchically and adaptively for brain tumor segmentation with missing modality(ies). Experiments show that our model outperforms the current state-of-the-art methods for brain tumor segmentation with missing modalities.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16443-9_18
SharedIt: https://rdcu.be/cVRyw
Link to the code repository
https://github.com/zzc1909/UNET-MFI
Link to the dataset(s)
N/A
Reviews
Review #2
- Please describe the contribution of the paper
The authors address the brain tumor segmentation with missing modalities by modeling feature interaction among modalities. To learn and interact complementary features between modalities, graph structure and attention mechanism are used.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1) The problem addressed in this paper is important. 2) The authors address the brain tumor segmentation with missing modalities by introducing Modalityadaptive Feature Interaction (MFI) with multi-modal code. 3) The method has novelty, although the novelty is not significant. 4) The validation results show the improved peformance.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- It seems that the comparsion in Table 1 is not under the same cross-validation slpit, which may result in unfair comparision. The results of some methods are directly extracted from their papers.
- Significance test is missing. Since the proposed method has small improvement in comparision with RFNet, significance test is recomended.
- Disccusion about the limitations of this method is missing.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors listed “yes” for both code and pre-trained models. In this case, it can be an easy task for both training and testing. If the reproduction was only based on the descriptions in the paper, it could be somewhat difficult.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- Significance test is needed. Since the proposed method has small improvement in comparision with RFNet, significance test is recomended.
- Disccusion about the limitations of this method is needed.
- Means–> Mean in Table 1
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The method has fair novelty with improved performance. The paper is generally well orgnized and clear. However, the validation is not solid. It seems that the comparsion in Table 1 is not under the same cross-validation slpit, which may result in unfair comparision. Significance test is missing. Since the proposed method has small improvement in comparision with RFNet, significance test is recomended.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #6
- Please describe the contribution of the paper
This paper proposed a modality adaptive feature interaction (MFI) with multi-modality code to adaptively interact features among modalities in different modality missing situations. The proposed MFI was incorporated with U-Net for segmenting brain tumors. Experimental results had compared with other state-of-the-art brain tumor segmentation methods and achieved superior segmentation performance.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1) To adaptively learn the complementary features among modalities (i.e., graph nodes), this paper introduces a multi-modal code to represent if different modalities are observed or not, to guide the learning process. Introducing MFI guided by multi-modal code into the different stages of a U-shaped architecture makes the multi-modal features hierarchically interact.
2) The experimental results demonstrated the positive effect of the proposed MFI.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) As the authors mentioned in Graph representation G=(V, E) in which E denotes the adjacency edge matrix representing the relations between nodes (modalities), the r_{ij} is finally computed using the formula (2) which is the output of Leaky Rectified Linear Unit by inputting the concatenation of voxels of v_i and v_j, why the authors defined the edge in such form? What’s the main purpose to define the edge in such a form? What are the advantages? All of these are not described in the paper and the relative experiments are not conducted.
2) The experimental results showed that the highest accuracy is obtained by using all of these four modalities images including F, T1, T1c, and T2, which suggest that the proposed MFI module seems to haven’t much better effect.
3) In Fig. 1, the GN hasn’t a description.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
acceptable
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1) The GN illustrated in Fig 1 should be detailed in this paper. 2) The motivations and the advantages of the computational method of edge shown in Formula (2) should be detailed, and some additional experiments should be conducted to demonstrate the advantages of this computational method.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The method proposed in this paper is more intuitive， the experiments are relatively sufficient.
- Number of papers in your stack
6
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The authors propose a modality adaptive feature interaction network (Net-MFI) based on graph structure for brain tumor segmentation with missing modalities. Compared to other approaches, Net-MFI focuses on learning the complementary information using an attention mechanism for adaptively missing modalities. Validation on the BraTS 2018 shows that Net-MFI enhances the tumor segmentation in different missing modalities situations outperforming other existing methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1- Simplicity: The proposed MFI is a simple yet effective approach for tumor segmentation with missing modalities.
2- The paper was validated on the BraTS 2018 dataset achieving the state-of-the-art results in incomplete mutli-modal brain tumor segmentation.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Limited experiments: The proposed method was only applied to the BraTS 2018 dataset which is not the latest BraTS dataset.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper meets the standard requirement in terms of reproducibility.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Recently, Wang et al. proposed “ACN: Adversarial Co-training Network for Brain Tumor Segmentation with Missing Modalities. MICCAI 2021”. Similarly, Azad et al. has published “SMU-Net: Style matching U-Net for brain tumor segmentation with missing modalities. MIDL 2022.” The authors should compare the proposed method with these previous works to demonstrate the main difference.
Net-MFI is applied with multi-modal code to the BraTS 2018 dataset which is not the latest BraTS dataset. The authors should apply to recent BraTS datasets such as BraTS 2020 and BraTS 2021.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Overall, the paper is very interesting, and the method shows great potential.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #7
- Please describe the contribution of the paper
This paper studies medical image segmentation with multi-modalities and when some modalities are missing. It presents a novel network component named MFI to solve the problem. The MFI does graph-based information fusion among available modalities and it is validated in UNet. The presented method is compared with four well established approaches on BraTS2018 dataset and it delivers superior performance than its counterparts.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
To the best of my knowledge, this paper presents a novel method. Graph-based information fusion is a well known concept, however, this is the first I see it is applied to deal with missing modalities. The method is clearly presented and the design of integrating MFI with U-Net is technically sound.
-
The improvement of the proposed U-Net-MFI over baselines is clear. It is a plus as it demonstrates its effectiveness on the well known BraTS2018 public dataset. On the BraTS2018, the U-Net-MFI’s improvement over the second-placed method seem’s systematical.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
I don’t see any major weaknesses of the paper. One thing it can be further improved is that it is unknown that whether the reported improvement are statistical significant or not.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Source code with be provided after this work is accepted. The BraTS2018 dataset is publicly available. Therefore, it should be possible for interested readers to reproduce the presented method and experiments.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
-
Provide statistical analysis of the reported improvement.
-
Report the computational efficiency of the presented work. For example, its inference time.
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper studies a practical problem in clinical routine. The presented method, to the best of my knowledge, is novel. It is effective when compared with other published methods on the BraTS2018 dataset. The presentation of this paper is overall very clear. I don’t see any major weaknesses.
- Number of papers in your stack
7
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The article proposes an interaction network based on a graph structure to adapt modality features for the segmentation of brain tumors with missing modalities. The proposed method is novel, as it focuses on learning complementary information using an attention mechanism for missing modalities in an adaptive way. The method is clearly presented and its design is technically sound. To improve the quality of the article, authors should address the following points in the final version:
- Why the latest BraTS datasets are not considered, but only the BraTS 2018 dataset was used?
- Is the comparison in Table 1 under the same cross-validation split? Otherwise, it may result in an unfair comparison.
- Since the proposed method presents a slight improvement compared to RFNet, a test of significance should be used to show a real improvement obtained.
- Discussion about the limitations of this method is needed.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2
Author Feedback
Q1：Why the 2018 BraTS dataset is used instead of the latest dataset？ The compared methods do not publish codes and release the dataset split for the latest BraTS dataset. For fair comparison, we use 2018 BraTS dataset, because most compared methods report their results based on the same data splitting proposed in Reuben Dorent’s paper “Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation”. Therefore, with 2018 dataset, we can fairly compare the methods using the same data splitting.
Q2：Is the comparison in Table 1 under the same cross-validation split？ Yes, the results of all the compared methods are obtained under the data split of Reuben Dorent’s work.
Q3：Test of significance for our method and RFNet. Since RFNet is not open source, we cannot conduct the test of significance. The results of our method and RFNet are obtained under the same cross-validation data split. Our method achieves 0.45%, 0.51%, 3.04% improvements for the whole tumor, tumor core and enhancing tumor regions, compared with RFNet.
Q4：The limitations of our method. The four contrast MR images are required to be pre-aligned (2018 BraTS dataset contains the well aligned multi-contrast MR images for each subject). In the future, we plan to conduct spatial alignment and feature interaction in feature space by attention mechanism at the same time.
Q5：The inference time of our method. For a subject data with 4 MR contrasts and 155 slides (240×240×155×4), the inference time of our method is about 2 minutes using Nvidia GeForce RTX 3090 GPU.