Authors

Dong She, Yueyi Zhang, Zheyu Zhang, Hebei Li, Zihan Yan, Xiaoyan Sun

Abstract

Accurate segmentation of brain tumors in MRI images re- quires precise detection of the edges. However, this crucial information has been overlooked by existing methods. In this paper, we introduce the Edge-oriented Transformer (EoFormer) which specifically captures and enhances edge information for brain tumor segmentation. Our approach incorporates a CNN-Transformer encoder to comprehensively improve the feature representation capability. The CNN structure captures low- level local features in the image, while the Transformer structure estab- lishes long-range dependencies between features to generate high-level global features. Additionally, the decoder of our approach utilizes two edge sharpening modules, the Edge-oriented Sobel and Laplacian mod- ules, which enhance the edge information. We also introduce efficient attention and re-parameterization techniques that make EoFormer com- putationally efficient. Experimental results on the BraTS 2020 dataset and a private medulloblastoma dataset demonstrate the superiority of our approach compared with existing state-of-the-art methods. More- over, our method achieves this with limtied model parameters and lower FLOPs, making it a promising approach for future research. The code is available at https://github.com/sd0809/EoFormer.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43901-8_32

SharedIt: https://rdcu.be/dnwDx

Link to the code repository

https://github.com/sd0809/EoFormer

Link to the dataset(s)

https://www.med.upenn.edu/cbica/brats2020/data.html

Reviews

Review #2

Please describe the contribution of the paper

The contribution of the paper is proposing a novel Edge-Oriented Transformer for brain tumour segmentation which tackle the problem of precise detection of edges. The proposed approach comprises an Efficient Hybrid Encode (EHE), which extracts features from images balancing the strength of a CNN and Transformer architecture, and a Edge-Oriented Transformer decoder, which integrates Sobel and Laplacian edge detector filters, enhancing the capabilities of edge and texture information extraction. The authors also introduce efficient attention and re-parametrisation techniques to increase computational efficiency.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-Clear contribution: The papers proposed a novel approach for brain tumour segmentation aiming to a precise detection of the edges. The proposed architecture is based on a CNN-Transformer encoder, which extracts features from images balancing the strength of a CNN and Transformer architecture, and a Edge-Oriented Transformer decoder, which integrate Sobel and Laplacian edge detector filters, enhancing the capabilities of edge and texture information extraction.
- The authors consider the complexity of the proposed approach, and they introduce an efficient attention and re-parametrisation techniques to achieve better efficiency in terms of memory and computational complexity.
- Evaluation was thorough and rigorous. The authors evaluated their approach on two different datasets and compared their results with the state-of-the-art works. They also carried ablation study to demonstrate the advantages of each proposed block.
- Based on the evaluation the proposed methods looks to outperform also in terms of number of parameters and FLOPs.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Some parts of the methodology could provide further details and be more precise. For example, the first paragraph of section 2.1 could be clearer and use the same words used in Fig.1. Some blocks on the figure, such as the Identity mapping is not clear when and how it has been used.
- The authors provide some details design about the proposed approach but there is no link to the actual code. However in the reproducibility checklist they state the code will be released.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors provided a table with some design details of the proposed EoFormer network. The architecture and implementation details are clear. However there is no link or reference to any code. The authors stated that the code will be provided in the reproducibility checklist, along with all the information. Information about datasets have been included in the paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Overall, the paper is well written and clear. However, I would suggest the following: -The first paragraph in section 2.1 could be better rephrased based on the blocks represented in fig.1, or just please use the same capital letters used in the figure. For example, in the third stage and bottleneck should be “TransFormer block” instead of “Tranformer Block”.
- In Fig.1 there is an IdentityFormer Block as Token Mixer, but is not clear if this is part of the proposed methodology or if it has been used only in the ablation studies. I would clarify this.
- In the paragraph “Edge-Oriented Transformer Decoder please add the definition of C in the text (as C is a parameter).
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I would accept the paper as the work proposed a novel approach for the segmentation of brain tumour focusing on the extraction of edge and texture information. The results are outperforming the SOTA works and also efficiency in terms of parameter and FLOPs has been demonstrated. The only issues is that the authors stated that the code will be provided in the reproducibility checklist but there is no link (hidden with stars) on the paper.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

The paper presented an edge-oriented transformer network combining CNN-transformer encoder for efficient segmentation of brain tumors. The method is validated on the public BraTS dataset and private medulloblastoma dataset.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

-The paper is written well. -The results and the ablation study presented validates the contribution of the method. -The efficient combination of CNN and transformers, along with memory-efficient attention modules are interesting to the community.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Some additional text on EoS and EoL would be helpful for the readers, along with some discussion on the results.
- The method is evaluated on the training dataset of BraTS. How does this compare to the validation set from BraTS? This could be straightforward by uploading the results to the website and thus comparing the existing state-of-the-art approaches fairly.
- How the EoS and EoL blocks impact the performance needs to be clarified. Is there a way to qualitatively visualize/evaluate the enhancement of edge information that the authors claim to achieve with these two blocks?
- Why is the private dataset evaluated with 4-fold validation while the BraTS trained differently? Are the same samples in the split used as other works?
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper could be difficult to implement for some with multiple blocks and optimization parameters. Making the code available along with the trained model can help the community.Why is the private dataset evaluated with 4-fold validation while the BraTS trained differently? Are the same samples in the split used as other works?
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

My main concerns and questions are addressed in the weakness section.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper is interesting to the community. The merits of this paper outweigh the weakness. There are some concerns that need to be addressed.
Reviewer confidence

Somewhat confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #1

Please describe the contribution of the paper

This paper presents a technique for brain tumor segmentation that is encoder-based, using a combination of a CNN model and edge-oriented transformer blocks for MR image feature representation. For edge predictions, the authors designed two edge sharpening modules in the decoder part, namely the Edge-oriented Sobel and Laplacian modules. The method aims to address the problem of segmenting brain tumors in three regions: the whole tumor (WT), the tumor core (TC), and the enhancing tumor (ET). The method was evaluated using two datasets: the BraTS 2020, which is a public dataset, and a private medulloblastoma dataset. Although the authors claim that their method overcomes the state-of-the-art methods, they have not provided a proper assessment of the results.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This paper proposes a new Edge-oriented Transformer (EoFormer) for the segmentation of brain lesions, which has interesting novelties in both the encoder and the decoder. In the encoder, the method combines the strengths of CNN and transformer to extract both local and global information from the input data. The computational and memory complexity has been addressed by replacing the vanilla attention with a new extended 3D attention. The decoder used by the model was specially designed to allow the edge-oriented module to extract the edges and textures of the image features. The proposed method was comparatively assessed against six other methods and has shown better or similar results.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The main weakness of this study is that the average results were presented without showing the variances, and no statistical hypothesis test was used to determine the significance of the differences between the proposed and comparable methods using the Average Dice metric.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The details required to implement, train and to test the proposed method are provided in the paper. Despite the private dataset, the reproducibility of the method could be evaluated using the BraTS 2020 dataset, which is a public dataset.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- Page 1 - Please replace the typo “CNN-Transformer hybird networks” by “CNN-Transformer hybrid networks”
- Page 2 - Please replace the typo “Efficient Hybird Encoder” by “Efficient Hybrid Encoder”
- Page 2 - last paragraph - Please correct the typo “matrial” - should be “material”
- Page 3 - section 2.1 - Please correct the typo “Efficient Hybird Encoder” by “Efficient Hybrid Encoder”
- Page 3 - Fig. 1(a) - Why does the input has 4 channels (H x W x D x 4) ?
- Page 3 - Fig. 1(a) - It is not clear why the EoS and EoL parts are linked in series instead of parallel.
- Page 3 - section 2.1 - Please correct the typo “convolutation” to “convolution”
- Page 4 - section 2.1 - Please explicitly define the variables n and d_k and operator transpose T.
- Page 5 - Equation 6 should be split in two equations, one for W_rep and other for B_rep.
- Page 7 - The authors claim that “The visualization demonstrates that the EoFormer achieves the closest segmentation results to the ground truth.”. It would be highly recommended to include a quantitative measure compared to the Ground Truth as a caption for each segmented resulted.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper presents a new technique for the segmentation of brain lesions. Although the paper is very well-organized and the proposed method presents enough novelty to be considered for publication, the results are poorly evaluated. All the results lack variance and a statistical hypothesis test is needed to assess and confirm the significance of the differences between the methods claimed by the authors.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

All three reviews agree to accept this work. Hence, this work can be accepted. Please prepare the final version based on review comments.

Author Feedback

Dear reviewers, Thank you for your valuable comments on our paper. We appreciate your time and effort in reviewing our work. We will carefully consider your suggestions and make the necessary revisions to improve the quality of the paper. Furthermore, we will release our code in the near future to facilitate further research and collaboration in the field. Best regards, Anonymous Author(s)

back to top

EoFormer: Edge-oriented Transformer for Brain Tumor Segmentation