List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Debayan Bhattacharya, Benjamin Tobias Becker, Finn Behrendt, Marcel Bengs, Dirk Beyersdorff, Dennis Eggert, Elina Petersen, Florian Jansen, Marvin Petersen, Bastian Cheng, Christian Betz, Alexander Schlaefer, Anna Sophie Hoffmann
Abstract
Using deep learning techniques, anomalies in the paranasal sinus system can be detected automatically in MRI images and can be further analyzed and classified based on their volume, shape and other parameters like local contrast. However, due to limited training data, traditional supervised learning methods often fail to generalize. Existing deep learning methods in paranasal anomaly classification have been used to diagnose at most one anomaly. In our work, we consider three anomalies. Specifically, we employ a 3D CNN to separate maxillary sinus volumes without anomaly from maxillary sinus volumes with anomaly. To learn robust representations from a small labelled dataset, we propose a novel learning paradigm that combines contrastive loss and cross-entropy loss. Particularly, we use a supervised contrastive loss that encourages embeddings of maxillary sinus volumes with and without anomaly to form two distinct clusters while the cross-entropy loss encourages the 3D CNN to maintain its discriminative ability. We report that optimising with both losses is advantageous over optimising with only one loss. We also find that our training strategy leads to label efficiency. With our method, a 3D CNN classifier achieves an AUROC of 0.85±0.03 while a 3D CNN classifier optimised with cross entropy loss achieves an AUROC of 0.66±0.1. Our source code is available at https://github.com/dawnofthedebayan/SupConCE_MICCAI_22.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16437-8_41
SharedIt: https://rdcu.be/cVRup
Link to the code repository
https://github.com/dawnofthedebayan/SupConCE_MICCAI_22
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The paper proposes to augment a cross-entropy based classification task with an adapted contrastive SimCLR loss which uses samples from the same class as positive pairs.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Nice addaption of supervised contrastive training for paranasal anomalies and combination with a CE-loss.
- Experiments and evaluation seem to be done well.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Contribution is not major, more application of an existing method to a medical dataset.
- The presentation and structure can be improved and the Figures 2&3 are terrible.
Minor:
- For [7] (citation/introduction of contrastive learning) please use the first original paper from van den Oord and not a survey paper.
- For t-SNE please always report the parameters, otherwise the results are hard to trust (basically t-SNE is not a good way to show separability, since no separability for one t-SNE setting does not mean there is no separability, it’s nice ‘anecdotally’ but does not show anything).
- Pg 5. “F())” there is a typo.
- I can just assume (knowing contrastive learning) the there also is a Proj_1 which is used for the contrastive task (Proj_1 is only briefly mentioned in the architecture but not what its used for).
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Overall, most hyperparamters are given and the methods themselves are not novel. The dataset is not provided so its hard so reproduce the exact results, but extending them to different datasets should be easy.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
I think its a decent paper with a minor novelty and application to a new dataset. However I think the presentation is still lacking and can be improved. In particular IMO the introduction is a bit all over the place and the general structure/ clarity can be improved. Furthermore the Figures 2&3 should be revised (especially Figure 3).
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
With a top presentation I think the application & minor novelty would make me see this as a “weak accept” - “accept”. However, given the current presentation I think it does not make the cut.
- Number of papers in your stack
6
- What is the ranking of this paper in your review stack?
6
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
The authors in this work propose to combine the supervised contrastive loss with a cross-entropy loss for classifying paranasal anomalies in the maxillary sinus.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper is mostly well-written and clear.
- The method achieves good performance on the authors’ private dataset.
- Most of the aspects are described in sufficient detail to enable the reproduction of results.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The novelty of the paper is not enough, which basically combines the supervised contrastive loss and binary cross-entropy loss.
- The experimental comparison lacks the current SOTA classification models. Thus, it is hard to justify the paper’s contribution.
- It would be better if the authors can provide some results on public medical classification benchmarks
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors will not release their code.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Although the paper is well-written, the major novelty is trival.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I appreciate the well-organised paper. I believe this paper can have good applicability potentials. But the novelty of this work is not good enough for MICCAI given that its a combination between existing works (See weakness). Also, the authors claimed that they will not release the code, which may cause reproducibility problems. Hence, I will suggest a reject score.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
5
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
4
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
The authors propose a self-supervised SimCLR method to maxillary sinus classification in MRI images. They also conduct a population study - experiments on large number of patients with wide distribution statistics - a clinically important but rather rare contribution in MICCAI.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Self-supervised learning methods such as the SimCLR used in the paper is a high interest to the readers in the field. The paper does a good job in introducing the method to the clinical problem with good experimental results.
The experiments are thorough and sufficiently backs the contribution.
The visual examples are well illustrated.
Limitations are clearly mentioned.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Experiments could have been better - it is possible that other methods could be comparable or even better than the proposed method. Showing scenarios where the proposed method really is more beneficial than the existing methods would be good.
Fig. 3. could be better - more explanations on what are shown - are the hyper-parameters fixed as much as possible? What about the hyper-parameters for t-SNE? What are the number of samples for the visualization and how are they chosen?
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper seems fairly reproducible - no code nor data is available but readers could try the suggested methods on their dataset of choice.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
It is probably early to conclude and generalize that contrastive loss + cross entropy loss is better for medical images and those with limited amount of labels. It’d be helpful to include one or more experiments with similar characteristics/problems for more generalizable conclusion.
More examples and visualizations to help understand the benefit of the proposed method would be good. For example, in which cases do the methods fail and succeed compared to the others? Can we get some insights into why?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper is clearly written and well structured. The claimed contribution is clear and experiments are done well. Nonetheless, the claimed contribution does not stand out strongly - neither based on the conducted experiments nor by the theoretical analysis.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper received mixed reviews and is recommended for rebuttal. This is indeed a borderline paper with no strong arguments for merits for acceptance.
“The paper is clearly written and well structured. The claimed contribution is clear and experiments are done well. Nonetheless, the claimed contribution does not stand out strongly - neither based on the conducted experiments nor by the theoretical analysis.”
“I appreciate the well-organised paper. I believe this paper can have good applicability potentials. But the novelty of this work is not good enough for MICCAI given that its a combination between existing works (See weakness). Also, the authors claimed that they will not release the code, which may cause reproducibility problems. Hence, I will suggest a reject score.”
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
13
Author Feedback
1)R1,R2,R3-Concerns on novelty:The main concern of the reviewers is that the combination of two losses is not novel for MICCAI community. However, we argue that the novelty should not be looked only from the combined loss but also from the careful design of our contrastive loss setting. Our work attempts to classify paranasal anomalies(mucosal thickening,polyps,cysts) occurring in the maxillary sinus(MS) from MS without anomaly on a large cohort of patients with wide distribution statistics. Previous studies[10,8,11] have classified at most one anomaly.We consider multiple anomalies with varying morphologies. Knowing that MS are found at particular locations of rigid registered MRIs, we extracted MS volumes.We then investigated the use of cross-entropy(CE) loss to leverage 3D information from MS volumes for anomaly classification. By using 3D data we were able to consider multiple anomalies that are differently located in the MS. However, in the given setting, 3D networks suffered from overfitting and failed to learn meaningful representations for our data. The anatomical and pathological variations existing in MS and the uneven distribution of the pathologies made the classification task challenging for CE trained models.We realised we needed to learn invariant and general representations to counter overfitting.This motivated us to employ contrastive learning. Our comparisons with SimCLR showed that pulling representations closer together for MS volume and its transformed view was not ideal. This was because we relied on transformations to learn good representations for classification. Therefore, we employed a supervised contrastive learning strategy to learn anomaly invariant representation for “anomaly” class i.e. MS with anomalies and anatomy invariant representation for “normal” class i.e. MS without anomalies.Together with the aforementioned pre-processing strategy, the supervised contrastive loss achieved satisfactory performance improvements already compared to our baselines(CE,SimCLR).By combining the supervised contrastive loss with the CE loss, we further improved the results and developed a data-efficient approach for paranasal anomaly classification.Having elaborated our approach, we want to emphasise again that the novelty of our work is not only the combination of two loss functions but also in the carefully designed pre-processing strategy that helps the combined loss learn invariant representations.
2)R2,R3-generalisation concern:We agree generalisation study is important, however, our work attempted at providing a solution to paranasal anomaly classification. We do not claim that this loss function is generalisable to other medical imaging tasks. However, we do think that such studies should be conducted to test the feasibility of our proposed loss function on other medical imaging tasks such as fetal abnormality classification.
3)R1-general structure/ clarity:We find this comment a little vague and therefore we are uncertain what R1 wants us to change.
4)R1,R2-Minor concerns: The minor concerns, t-sne parameters & figures will be redressed in camera-ready manuscript.
5)R1,R2,R3-Reproducibility:We will release our code on acceptance of the paper.
6)R3-claimed contribution does not stand out strongly: We performed a nested stratified K-fold experiments.We tested for statistically significant difference in our performance metrics using a permutation test with nP= 10000 samples and a significance level of alpha=0.05
7)R3-cases where methods fail and succeed: We found that most of the misclassifications were in the case of mucosal thickening. The dissimilarity of mucosal thickened MS from normal MS is small. Polyps and cysts were successfully classified in most cases. 8) R2-current SOTA models: Our work does not propose a new architecture.We use ResNet18 to show the effectiveness of our combined loss. Thus, comparison against other SOTA classification models seems unnecessary in our opinion.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper has the following merits even from a reviewer who suggested to reject.
“1. The paper is mostly well-written and clear.
- The method achieves good performance on the authors’ private dataset.
- Most of the aspects are described in sufficient detail to enable the reproduction of results.”
Although the novelty is not very strong, this is a very strong solid work on solving an important medical imaging problem with good technical clarity, reproducibility and impacts.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
9
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper does a good job in introducing the method to the clinical problem with good experimental results. In the rebuttal, authors point out that the novelty of their work is not only the combination of two loss functions but also the careful design of the contrastive loss setting. I think this addressed well the main comment of the reviewers
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
NR
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Paper is a borderline accept thanks to the authors clarifying the novelty and inspiration for their proposed combination of losses for use in existing networks. As such, the careful design suggests the paper is reasonable for MICCAI.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
8