Authors

Eduard Lloret Carbonell, Yiqing Shen, Xin Yang, Jing Ke

Abstract

COVID-19 is a viral disease that causes severe acute respiratory inflammation. Although with less death rate, its increasing infectivity rate, together with its acute symptoms and high number of infections, is still attracting growing interests in the image analysis of COVID-19 pneumonia. Current accurate diagnosis by radiologists requires two modalities of X-Ray and Computed Tomography(CT) images from one patient. However, one modality might miss in clinical practice. In this study, we propose a novel multi-modality model to integrate X-Ray and CT data to further increase the versatility and robustness of the AI-assisted COVID-19 pneumonia diagnosis. We develop a Convolutional Neural Networks(CNN) and Transformers hybrid architecture, which extracts extensive features from the distinct data modalities. This classifier is designed to be able to predict COVID-19 images with X-Ray image, or CT image, or both, while at the same time preserving the robustness when missing modalities are found. Conjointly, a new method is proposed to fuse three-dimensional and two-dimensional images, which further increase the feature extraction and feature correlation of the input data. Thus, verified with a real-world public dataset of BIMCV-COVID19, the model outperform state-of-the-arts with the AUC score of 79.93%. Clinically, the model has important medical significance for COVID-19 examination when some image modalities are missing, offering relevant flexibility to medical teams. Besides, the structure may be extended to other chest abnormalities to be detected by X-ray or CT examinations.Code is available at https://github.com/edurbi/MICCAI2023 .

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43904-9_37

SharedIt: https://rdcu.be/dnwHg

Link to the code repository

https://github.com/edurbi/MICCAI2023

Link to the dataset(s)

https://bimcv.cipf.es/bimcv-projects/bimcv-covid19/

Reviews

Review #1

Please describe the contribution of the paper

The paper proposed a multi-modality model to integrate X-Ray and CT data to detect COVID-19 pneumonia.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

the proposed model is important for COVID-19 examination when some image modalities are missing.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

no comparison to other studies.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

the proposed model has the potential to be applied to other to other chest abnormalities using multiple modalities.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

Please add a comparison table to other previous works.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The work
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

The authors proposed two-stage feature fusion layer, random feature matrix dropout and feature correlation blocks to learn from multi modality image data. The proposed model can adapt to fusing 3D image and 2D image.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The model can adapt into learning both 3D data and 2D data with one model.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The dataset description is not clear. No numbers of positive and negative are given. It is unclear if the dataset is balanced or not. The authors did not compare their proposed model with baseline and state of art models.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Can be preproduced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
1. Please add a few baseline models and state of art models. I am really curious about how other models performed using the same data set.
2. Please add a more detailed description of the dataset. Number of images in each class, if augmentation was applied.
3. please add an ablation study about the effectiveness of each module.
4. Please address why use both early and late fusion.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Novelty of the paper.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

This paper proposes a novel multi-modality model for the AI-assisted COVID-19 pneumonia diagnosis. The model integrates X-Ray and CT data to enhance the versatility and robustness of the diagnosis. The model utilizes a hybrid architecture of Convolutional Neural Networks (CNN) and Transformers to extract extensive features from the distinct modalities. Additionally, the proposed method fuses three-dimensional and two-dimensional images to further increase feature extraction and feature correlation. The model’s effectiveness is evaluated on a real-world public dataset of BIMCV-COVID19, and it outperforms state-of-the-art methods with an AUC score of 79.93%. The proposed model has significant medical significance for COVID-19 examination when some image modalities are missing, offering flexibility to medical teams.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The paper proposes a hybrid CNN-Transformer model that can effectively extract features from distinct modalities using Transformers to find dependencies between different modalities.
2. The introduction of a Dual X-Ray Attention block is a unique approach to detecting dependencies between the two input X-ray images, which has demonstrated promising results.
3. The proposed feature fusion layer combines features from different modalities, which have varying dimensions, using convolutional layers to reduce the three-dimensional CT data to two-dimensional and fuse it with two-dimensional X-ray data twice, at early and late stages.
4. The model is evaluated using a real-world public dataset of BIMCV-COVID19 and outperforms existing state-of-the-art methods with an impressive AUC score of 79.93%, showcasing its effectiveness in COVID-19 diagnosis.
5. The ablation study demonstrates that the proposed model performs well in terms of AUC, recall, and precision metrics. The integration of CT and X-ray data enhances the accuracy of COVID-19 diagnosis, and the model generates saliency maps that can aid radiologists in identifying areas of interest and making diagnoses more efficiently.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The evaluation of the proposed model is limited to a single dataset and a small number of state-of-the-art methods, raising concerns about its generalizability and robustness in clinical practice.
2. The paper lacks a comprehensive explanation of the design choices made, including the use of specific Transformer layers and feature fusion approach, limiting the ability of other researchers to replicate or modify the proposed model and weakening the impact of the proposed method in the field of medical image analysis.
3. The authors do not provide sufficient justification for the use of ResNet and ViT in the hybrid architecture, nor do they experimentally demonstrate its effectiveness over other recent CNN and transformer architectures.
4. The experimental setup involves image size reduction for CT scans, which may impact the model’s performance on full-size images.
5. The paper does not include a detailed analysis of the misclassified cases, which would help identify the limitations of the proposed model.
6. The proposed model may be perceived as a “black box”, potentially hindering its interpretability and transparency in medical settings where such qualities are critical.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

he authors have presented comprehensive details and resources, indicating that the proposed method can be reproduced with ease.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

I have some concerns about the paper’s lack of clarity regarding design choices, feature fusion, and the justification for the use of ResNet and ViT in the hybrid architecture. The evaluation is limited to a single dataset, and a more detailed analysis of misclassified cases could help identify the model’s limitations in clinical practice.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

5
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I have carefully reviewed the paper and I believe that while the proposed hybrid CNN-Transformer model is promising, several weaknesses limit its impact and generalizability in the field of medical image analysis. Specifically, the paper lacks a comprehensive explanation of the design choices made, such as the use of specific Transformer layers and feature fusion approach, and does not provide sufficient justification for the use of ResNet and ViT in the hybrid architecture. Additionally, the evaluation of the proposed model is limited to a single dataset, and the paper does not include a detailed analysis of misclassified cases, which would help identify the limitations of the proposed model. However, the paper has provided sufficient information and resources to allow for the reproducibility of the proposed method, and the proposed model outperforms existing state-of-the-art methods on the BIMCV-COVID19 dataset. Based on these factors, I recommend that the authors revise their paper
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper presents a multi-modality model to integrate X-Ray and CT data to detect COVID-19 pneumonia. The proposed model is evaluated using a real-world public dataset of BIMCV-COVID19 and outperforms some state-of-the-art methods, indicating its effectiveness in COVID-19 diagnosis. In general, this paper is interesting, and it is useful for COVID-19 examination when some image modalities are missing. Besides, suggesting the authors revise the paper within the final version according to the reviewers’ comments.

Author Feedback

N/A

back to top

COVID-19 Pneumonia Classification with Transformer from Incomplete Modalities