List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Zhen Yu, Toan Nguyen, Yaniv Gal, Lie Ju, Shekhar S. Chandra, Lei Zhang, Paul Bonnington, Victoria Mar, Zhiyong Wang, Zongyuan Ge
Abstract
In practice, many medical datasets have an underlying taxonomy defined over the disease label space. However, existing classification algorithms for medical diagnoses often assume semantically independent labels. In this study, we aim to leverage class hierarchy with deep learning algorithms for more accurate and reliable skin lesion recognition. We propose a hyperbolic network to jointly learn image embeddings and class prototypes. The hyperbola provably provides a space for modeling hierarchical relations better than Euclidean geometry. Meanwhile, we restrict the distribution of hyperbolic prototypes with a distance matrix that is encoded from the class hierarchy. Accordingly, the learned prototypes preserve the semantic class relations in the embedding space and we can predict the label of an image by assigning its feature to the nearest hyperbolic class prototype. We use an in-house skin lesion dataset which consists of around 230k dermoscopic images on 65 skin diseases to verify our method. Extensive experiments provide evidence that our model can achieve higher accuracy with less severe classification errors compared to that models without considering class relations.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16437-8_57
SharedIt: https://rdcu.be/cVRuI
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
In this paper, a method for skin lesions embedding and classification based on hierarchical class relations encoding and hiperbolic embedding is presented. The two main contribuions are:
- They use hyperbolic geometry, instead of Euclidean, for the image embedding. . They incorporate to the loss a distance based on the hiearchical relations between classes.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper strengths of the paper are: 1) The introduction of hieralchical relations between classes and the introduction of this informataion in the loss function. 2) The use of hyperbolic geometry for image embedding, more suitable for hierarchical relations then Euclidean according to [5] and [11] 3) They include an ablation study in which they evaluate the improvements of each of the proposed contributions. 4) They include two evaluations metrics of great interest to analyse the contribution that they propose: mistake severity and hierarchical distance of he top k (HD-k).
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The weaknesses are reduced: 1) I consider questionable the data augmentation that they employ. In an application where color is essential for the diagnosis, it does not seem adequate to incorporate color transformations in the augmentation. 2) An explanation of the dynamic range and justification of the values of the different metrics in Table 1 would be desiderable. For example, they do not justify that 50% accuracy is reasonable for 65 classes or the maximum possible value of the mistake severity or the maximum possible value of the mistake severity or if 1.X is a good value for HD-k
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
They scarcely describe the database. They mention the number of images but they do not mention how many images of each class or if it is balanced. They cited a previous work of the authors, perhaps in this reference the database is better described. They do not provide implementation details. They state ithat it is included in an Appendix, but in the available appendix only a figure with the structure of the skin taxonomy is contained.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
In this paper, a method for skin lesions embedding and classification based on hierarchical class relations encodng and hiperbolic embedding is presented. The two main contribuions are:
- They use hyperbolic geometry, instead of Euclidean, for the image embedding. . They incorporate to class distance a distance based on the hierarchical relations between classes. Five comments that may improve the paper: 1) I consider questionable the data augmentation that they employ. In an application where color is essential for the diagnosis, it does not seem adequate to incorporate color transformations in the augmentation. 2) An explanation of the dynamic range and justification of the values of the different metrics in Table 1 would be desiderable. For example, they do not justify that 50% accuracy is reasonable for 65 classes or the maximum possible value of the mistake severity or if 1.X is a good value for HD-k 3) It would be interesting to highlight in the text the main contributions of the paper with respect to the literature. 4) Perhaps the theoretical explation 2.1 could be substituted by more details in the method. 5) Figure 4 is impossible to visualize.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The proposed research contributes to solve an important medical problem. It also include methodological contributions.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This work proposes to use class hierarchy with deep learning algorithms for more accurate and reliable skin lesion recognition. A hyperbolic network is proposed to learn image embeddings and class prototypes. Validation was carried out on an in-house skin lesion database which consisted of ~230k dermoscopic images on 65 skin diseases.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Elegant problem formulation.
- Clear theoretical background introduction.
- Thorough literature review on related work.
- Extensive validation experiments.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
The logic of the hyperbolic hierarchy structure of the skin class labels is debatable. What specific values does such formulation brings? In hyperbolic space, the distance becomes longer close to the boundary areas. However, the final classes, i.e., the leaf nodes, are all there. The reviewer does not fully understand the full motivation. Why are the distances between two nodes that share the same parent nodes are formulated with the same distance between two nodes that do not share the same parent nodes?
-
It is exciting to see a dataset with ~230k images from 65 disease subtypes. A natural question is whether the image numbers in each class are balanced. What are the maximum image numbers and minimum images numbers for a disease type? Any performance evaluation methods to address them?
-
The performance improvement was relatively incremental.
-
The training details are not provided in Appendix.
-
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Excellent.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
It would be nice if a concrete contribute list is provided in the paper.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
It is an inspiring work with solid clinical importance. It will promote the society to pay attention to advanced geometry research.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper proposes a skin lesion classification approach that leverages a hyperbolic embedding in a class hierarchy-aware setting. The authors’ goal is to increase classification performance on the task by projecting the deep features to a hyperbolic geometric embedding and performing classification on it. Their method also models and enforces the problem’s class hierarchy by incorporating a novel loss into the training process.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper has novel contributions because it incorporates recent developments in hyperbolic embedding learning. It also presents a novel loss function that uses the class hierarchy of the underlying problem in their architecture. The paper is technically sound, mathematically well-funded, and enjoyable to read.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Despite the interesting formulation of the method described in the paper, the results (mainly table 1 ) look relatively closer to the baselines. The performance improvements look marginal compared to the rest of included methods. In particular, it does not look like the hyperbolic approach contributes much to the performance as it does the hierarchy aware variants.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Upon release of the dataset employed in the study, the paper contains the necessary information to reproduce the paper’s results. However, I encourage the authors to share more details about the classification network and clarify whether multiple layers were used and if there is any type of batch normalization or non-linearity.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- The introduction succeeds at stating the problem and the related work. It is also well-motivated and clarifies the paper’s contributions. It would be helpful to see the standard deviation on all the values in table 1 to evaluate the proposed method’s performance further.
- The authors mention more implementation details in the Appendix, but they seem to be missing.
- Can the authors further clarify the default “0” parameter used in Eq. 3. It seems with 0, Eq. 2 simplifies a lot, and it is not clear how this affects, for instance, the expressivity of the network.
- I believe there is a typo on the abstract on the word “provably.” It seems that the authors meant “probably.”
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
My decision is primarly based on the marginal increments in performance of the proposed method with the baselines in table 1. I would be willing to change my rating if more evidence of the benefits of the proposed approach is shown.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
2
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2
Author Feedback
We would like to thank all the reviewers for their favorable comments and constructive suggestions. We give detailed responses as follow:
Q1. Concern about color transformation in skin lesion diagnosis? Reply: Clinical criteria used for skin lesion diagnosis often include lesion’s color, shape, size, and other patterns like textures. Hence, adding color transformation during model learning to some extent can prevent algorithm overfitting in this pattern. Study [1] conducts extensive experiment in data augmentation for skin lesion image analysis and show that color transformation is beneficial for AI-image based skin lesion recognition.
[1] Perez, Fábio, Cristina Vasconcelos, Sandra Avila, and Eduardo Valle. “Data augmentation for skin lesion analysis.” In OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis, pp. 303-311. Springer, Cham, 2018.
Q2. Concern about using hyperbolic space and hierarchy information? Reply: Indeed, the hyperbolic space has the distance characteristic that pairs-wise distance increase significantly from the origin point to the border. Thus, we can put the root class close to the origin point and put those leaf classes around the border. In this study, we mainly focus on classifying the leaf classes and leave hierarchical classification, i.e., classify a lesion from coarse-to-fine in our extension work.
For this work, our basic idea is to learn prototypes for each leaf class in the hyperbolic space and restrict the distance between class prototypes so that their relative arrangement in the embedding space follows distribution from the class hierarchy. We use HCD and LCD to encode truth class distribution in a class hierarchy which is defined by a class distance matrix. Classes belonging to a same super-class have same distance which is less than that of classes from different super-classes (as shown in Fig.2). The class distance increase when the semantic difference gets large, for example, classes from totally different super-classes like benign vs malignant have large distance than classes from different benign super-classes. Our paper is mainly inspired by [3], but we explored the learning of class-hierarchy regularized embedding in hyperbolic space for skin lesion classification, and we also studied different hierarchy encoding methods.
[2] Garnot, V.S.F., Landrieu, L.: Leveraging class hierarchies with metric-guided prototype learning, BMVC (2021)
Q3. Can the authors further clarify the default “0” parameter used in Eq. 3? Reply: The parameters “0” define the reference point for performing exponential projection, that is, projecting image features from Euclidean space to hyperbolic space. We definitely can use different start point by learning from random parameters to replace the “0”, but this complicate the model and moreover we did not observe a better performance in experiment. Similar practice also can be seen [3] and [4] that use “0” as fixed point for a transition between the Euclidean and hyperbolic ball representations of a vector.
[3] Khrulkov, V., Mirvakhabova, L., Ustinova, E., Oseledets, I. and Lempitsky, V., 2020. Hyperbolic image embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6418-6428).
[4] Liu, Shaoteng, Jingjing Chen, Liangming Pan, Chong-Wah Ngo, Tat-Seng Chua, and Yu-Gang Jiang. “Hyperbolic visual embedding learning for zero-shot recognition.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9273-9281. 2020.
Q4. Other minor concerns. Reply: We will carefully revise the typo errors and include the missed details in the camera-ready paper, i.e. training details, dataset distribution and more information about network architecture.