Authors

Zirui Wang, Yi Hong

Abstract

Magnetic Resonance Imaging (MRI) plays an important role in multi-modal brain tumor segmentation. However, missing modality is very common in clinical diagnosis, which will lead to severe segmentation performance degradation. In this paper, we propose a simple adaptive multi-modal fusion network for brain tumor segmentation, which has two stages of feature fusion, including a simple average fusion and an adaptive fusion based on an attention mechanism. Both fusion techniques are capable to handle the missing modality situation and contribute to the improvement of segmentation results, especially the adaptive one. We evaluate our method on the BraTS2020 dataset, achieving the state-of-the-art performance for the incomplete multi-modal brain tumor segmentation, compared to four recent methods. Our A2FSeg (Average and Adaptive Fusion Segmentation network) is simple yet effective and has the capability of handling any number of image modalities for incomplete multi-modal segmentation. Our source code is online and available at https://github.com/Zirui0623/A2FSeg.git.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43901-8_64

SharedIt: https://rdcu.be/dnwEc

Link to the code repository

https://github.com/Zirui0623/A2FSeg.git

Link to the dataset(s)

https://www.med.upenn.edu/cbica/brats2020/data.html

Reviews

Review #3

Please describe the contribution of the paper

The paper proposes a simple adaptive multi-model fusion network, A2FSeg, for incomplete multi-modal brain tumor segmentation. Two fusion modules are included, an average fusion and an adaptive fusion based on an attention mechanism, to handle the modality missing issue. State-of-the-art performance has been achieved on the BraTS2020 dataset under different incomplete multi-modal situations.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. Incomplete multi-modal data are commonly encountered in the clinic. The topic of the study is important.
2. The proposed method is simple and easy to implement.
3. The brain tumor segmentation performance is promising.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. It is surprising to see that the baseline performs better than the state-of-the-art method.
2. As stated by the authors, the proposed method is quite simple. Why it can achieve so good results? The authors should elaborate more on this.
3. I think T1c is an important modality for tumor characterization. However, from Fig. 3, its importance is lower than Flair and T2.
4. More comparison methods should be included (e.g., HeMIS, RobustSeg, and RFNet).
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Reproducibility is OK.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
1. Elaborate more on the results generated, linking to the method design.
2. Add more comparison methods.
3. Explain why Flair and T2 are more important than the other modalities from the clinical perspective.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. Overall, the topic is interesting and important. The method is simple. The results are promising.
2. The results are not enough convincing. More comparisons are required to make the conclusion.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #1

Please describe the contribution of the paper

A multimodal fusion that can handle missing modalities is proposed for the problem of brain tumor segmentation. An average fusion and a fusion with attention modules are implemented. To handle missing modalities, attention weights are normalized using the softmax function.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Simple fusion methodology proposed which seems to provide marginal, but consistent improvement over competing methods
- ablation study has been reported.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- check ref [13] (“These two techniques are demonstrated to be effective in handling missing modalities and multi- modal fusion [13].”) -link to source code has not been made available -in the ablation study results in Table 2, what are the meanings of Complete, Core, Enhancing and Average columns? This needs proper explanation.
- The conclusion section: authors question why a complex model is necessary when their rather simple implementation already provides better results. Rather than putting forward this question , authors should seek to explain why their simple model performs better than more complex, competing ones.
- authors should highlight the novelty of their approach.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

authors claim source code is provided but no link is provided in the text.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
check ref [13] (“These two techniques are demonstrated to be effective in handling missing modalities and multi- modal fusion [13].”) -link to source code has not been made available -in the ablation study results in Table 2, what are the meanings of Complete, Core, Enhancing and Average columns? This needs proper explanation.
- The conclusion section: authors question why a complex model is necessary when their rather simple implementation already provides better results. Rather than putting forward this question , authors should seek to explain why their simple model performs better than more complex, competing ones.
- authors should highlight the novelty of their approach.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The proposed fusion approach is very simple and straightforward. Authors should dive deeper into explaining why their simple model outperforms more advanced architectures.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

The study proposes a novel multimodal segmentation method robust to missing MRI modalities in the context of brain tumor segmentation. The method was tested on the well-known BraTS database. The main contribution is to adapt Modality-Aware Mutual Learning (MAML) feature fusion strategy in brain tumor segmentation under the missing modality scenario. The authors combine two fusion strategies one based on simple average and the other based on MAML using attention. The method performs better than other baseline methods. The ablation study shows incremental benefits of the newly proposed components of the methods.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The authors proposed a novel fusion strategy that improves segmentation performance in missing modalities scenarios. The approach showed better results than the existing ones.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

1) Introduction: <Since different modalities present different clarity of brain tumor components, …> The expression is awkward. Different MRI modalities provide complementary information for tumor segmentation. This is not the only place the expression is awkward. Written English is passable but certainly needs improvement throughout the text. Many acronyms are not fully spelled out (or explained). For example, HVED, RFNet, and MAML were not fully explained or spelled out. These are important baseline methods and thus should be explained. 2) Multimodal brain tumor segmentation with missing modalities is a well-studied topic. The manuscript is missing some recent studies such as <Region-of-interest Attentive Heteromodal Variational Encoder-Decoder for Segmentation with Missing Modalities; Proceedings of the Asian Conference on Computer Vision (ACCV), 2022>. It should be mentioned in the Introduction. 3) Methods: The authors adapted MAML to their problem domain. They seemed to replace concatenated multimodal features (from MAML) with averaged multimodal features (AF2Seg). Other minor things have been modified from the original MAML. The authors need to justify and fully explain the modifications they made from the original MAML. 4) Equation 5 seems to lack relative weights in each term. There seems to be three dice loss, which needs to be reflected in Figure 1. 5) Table 1 is arranged in a weird order. They should report one-modality results followed by two-modality results. After that three- and four-modalities results should be given. The table is missing p-values comparing approaches. They might need to perform bootstrapping to derive the distribution of performance metrics. 6) The method was not tested in an external validation set.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Largely reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

See the comments above.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

See the comments above.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposes a simple adaptive multi-modal fusion network (A2FSeg) for brain tumor segmentation. The proposed method combine two fusion strategies one based on simple average and the other based on MAML using attention. The results show the proposed method performs better than other baseline methods and the benefits of different components. Overall, the topic is interesting and the results are promising. Besides, some concerns raised by reviewers should be addressed in the final version.

Author Feedback

N/A

back to top

A2FSeg: Adaptive Multi-Modal Fusion Network for Medical Image Segmentation