List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Jaeung Lee, Keunho Byeon, Jin Tae Kwak
Abstract
Cancer grading is an essential task in pathology. The recent developments of artificial neural networks in computational pathology have shown that these methods hold great potential for improving the accuracy and quality of can-cer diagnosis. However, the issues with the robustness and reliability of such methods have not been fully resolved yet. Herein, we propose a centroid-aware feature recalibration network that can conduct cancer grading in an accurate and robust manner. The proposed network maps an input pathology image into an embedding space and adjusts it by using centroids embedding vectors of different cancer grades via attention mechanism. Equipped with the recalibrated embedding vector, the proposed network classifiers the input pathology image into a pertinent class label, i.e., cancer grade. We evaluate the proposed network using colorectal cancer datasets that were collected under different environments. The experimental results confirm that the proposed network is able to conduct cancer grading in pathology images with high accuracy regardless of the environmental changes in the datasets.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43895-0_20
SharedIt: https://rdcu.be/dnwx2
Link to the code repository
https://github.com/colin19950703/CaFeNet
Link to the dataset(s)
https://github.com/QuIIL/KBSMC_colon_cancer_grading_dataset
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes an attention-based neural network for cancer classification using pathology images. By attending to class centroids, the proposed model achieves better performance while still leads to small scale in terms of the number of parameters.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The main contribution/strength of the paper is the proposal of calibrated input embedding vectors by attending to the centroids of cancer grades. This leads to better learning capability and generalization capability.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Experiments on more datasets are expected;
- Provide rationale behind the attention mechanism: why attending to class centroids?
- Ablation study is also suggested.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Reproducibility is good.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Please see above for weaknesses.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Please see comments above.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This paper describes a feature recalibration technique to improve the feature representation of pathology images for cancer grading. The proposed method uses the feature centroids to enhance the original feature with recalibrated feature, which is obtained with an attention mechanism on the original feature and centroid features. The experimental results show that the proposed recalibration brings improvement across five evaluation metrics on two datasets.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Well written and easy to follow. The proposed feature recalibration is novel. It utilizes the centroid features to obtain the recalibrated features with an attention mechanism. This shows a specific way to explore the knowledge of centroids to enhance the features for each sample/image. This could be understood as explicitly using the dataset-level knowledge to enhance the sample level feature. It could also lead to better generatlization across datasets as the centroids will adapt with the dataset.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The proposed method lacks motivation, analysis and thorough evaluation. While it is demonstrate the proposed method can lead to improvement over five evaluation metrics on the test datasets, the motivation and reason to adopt such method is unclear. Why the attention on centroids can help? How does the recalibrated features perform if used independently (this can validate if the recalibrated features are complemented to the original feature or not)? How does the attention score maps look like? Is the concatenation the best way to fuse the original features and the recalibrated features? How does the centroids evolve during training?
In the evaluation, as I understand the proposed recalibration can be viewed as an additional component for any existing model. Further comparison with other type of network as backbone can help to show the contribution of the proposed method.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The version and platform of the model implementation should also be given for the reproducibility. The specific variant of the compared methods should also be made clear. E.g. which variant of Swin is used?
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The questions in the weakness can be addressed either from analysis or from observation. Answering those questions can help the reader to better understand the proposed method.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The proposed method of feature recalibration is novel. The weaknesses are more on better understanding the proposed method.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
I sit between weak accept and accept after reading the rebuttal. The authors addressed the concerns. The ablation study is presented but no numbers is presented as evidence. The motivation is clarified.
Review #3
- Please describe the contribution of the paper
This paper propose an attention-based approach to recalibrate centroid-aware feature within the network to solve a classification problem using pathology image.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper utilizes multiple datasets for evaluation and employs a diverse range of metrics, enhancing the robustness of the assessment. Additionally, the overall writing is well-organized, facilitating a clear understanding of the work presented.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The performance improvement of the proposed model is minor compared to the baselines. Also, only basic baselines like ResNet and DenseNet were used for comparison, while domain-specific models were not considered. Additionally, the paper lacks qualitative results to support its claims.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper offers implementation details and a description of the datasets used in the study.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
I suggest that the authors include additional domain-specific models as baselines, focusing on the analysis of colorectal pathology images. Furthermore, it appears that an ablation study of the proposed modules is missing. I suggest that the authors perform an ablation study by removing one module at a time and comparing the resulting performance to better understand the contribution of each module.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper presents a well-organized evaluation using multiple datasets and diverse metrics, but the proposed model’s performance improvement is minor, and it lacks comparisons to domain-specific models and ablation study/qualitative results.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
4
- [Post rebuttal] Please justify your decision
The rebuttal effectively addressed my concern regarding minor performance improvement, for which I appreciate the effort, but the two remaining issues still require attention. (1) The baselines used in the study, even considering MMAE-CEo (Medical Image Analysis, 2021), are outdated, and I recommend updating the baselines to ensure the relevance and validity of the results. (2) The ablation study results are insufficient in demonstrating the significance of the improvement achieved by each component. Without sufficient supporting evidence, the significance of these improvements remains uncertain based on the information provided.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper presents a feature recalibration method that improves the representation capability. Evaluation is conducted on two public datasets. Overall the reviewers find the paper well written and method presented some novelty. The main weaknesses are round evaluation, such as comparison with domain specific methods, evaluation using more datasets and more ablation studies to validate the design motivation.
Author Feedback
● The reviewers suggested that additional experiments with more datasets and domain-specific models and ablation studies are needed:
We agree with the reviewers that the experiments that we conducted are limited. It would have been much better if we could include other datasets. Unfortunately, we were not able to do so. We leave it for the future study and will discuss this as the limitation of our study in the final manuscript.
We would like to note that MMAE-CEo (Medical Image Analysis, 2021) is a domain-specific model that was specifically built for cancer classification in pathology images and achieved the SOTA performance on the two test sets used in this study. The original paper of MMAE-CEo shows that it outperforms 11 different models. However, the description of MMAE-CEo was insufficient. We will provide an extended description of MMAE-CEo in the final manuscript.
For ablation experiments, we were able to conduct a few experiments to examine the effect of different components. The exclusion of the CaFe module, i.e., EfficientNet, resulted in much worse performance than our model. Using the recalibrated embedding vectors only, there was a large performance drop. These two results indicate that the recalibrated embedding vectors are complement to the input embedding vectors. Also, using addition instead of concatenation to merge the input embedding vectors and recalibrated embedding vectors, the performance was lower than our model, and thus the concatenation is the best way to fuse them together. Overall, we believe that these results demonstrate the validity of our design and method. We will briefly discuss these findings in the final manuscript.
● The reviewers pointed out that the performance improvement of our model is minor compared to the baselines:
As for the performance improvement, we would like to emphasize the difference between the two test sets, which were obtained from different scanners and settings. The huge performance drop that was obtained by other models due to domain-shift, which is of great interests in digital pathology today. Therefore, we believe that the 2.4% improvement in accuracy against the SOTA model on this dataset should not be underestimated.
Moreover, we would like to emphasize that the core idea of our method is to recalibrate the embedding vectors, i.e., aiming at improving feature representation. For this reason, we included triplet loss and supervised contrastive loss that are extremely popular, related methods. Using the same backbone, our method outperforms the two methods (≥3% accuracy). This further emphasizes the effectiveness of our model.
● The reviewers pointed out that the motivation of our method is unclear:
We agree with the reviewers that the motivation and rationale behind our method was insufficient. The motivation of our method is to improve the feature representation that is efficient/fast but robust to changes in the data distribution. Assuming that the classes are well separated in the feature space, the centroid embedding vectors may provide the reference points to represent the data distribution of the training data. Hence, we propose to utilize the centroid embedding vectors to recalibrate the input embedding vectors of pathology images. The attention mechanism by design allow us to express the input embedding vectors as a weighted combination of centroid embedding vectors. In other words, we reformulate the input embedding vectors with respect to the class centroids (i.e., training data distribution). Since centroid embedding vectors are fixed during inference, the recalibrated embedding vectors do not vary much compared to the input embedding vectors even though the data distribution substantially changes, leading to improved stability and robustness of the feature representation. In the final manuscript, we will improve the description of the motivation of our method.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
After the rebuttal, some of the reviewers’ concerns have been addressed, but the main questions about ablation study and comparison with baselines still remain. It would require more thorough comparisons with more up-to-date methods to fully present the new contributions of the proposed method.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper presents a new attention method for WSI classification, called centroied-based attention. Although the proposed method should provide some insights for computational pathology community, however, as pointed by R3, this paper lacks comparison with WSI domain-specific methods, such as Trans-MIL, CLAM, etc, which are all well acknowledged by computational pathology community. Moreover, the proposed centriod-based recalibration method is very similar to the Clustering-constrained attention multiple instance learning in CLAM.
Consider the above information, this is a boardline paper and I slightly vote for reject.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The response and the commitment to include details of the methodology appear to be relevant - enough to create the confidence we need. I suggest accepting the document, with a reality check of the commitments made at the rebuttal stage.