List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Haolin Yuan, John Aucott, Armin Hadzic, William Paul, Marcia Villegas de Flores, Philip Mathew, Philippe Burlina, Yinzhi Cao
Abstract
Lyme disease is a severe skin disease caused by tick bites, which affects hundreds of thousands of people. One task in diagnosing Lyme disease is lesion segmentation, i.e., separating benign skin from lesions, which can not only help clinicians to focus on lesions but also improve downstream tasks such as disease classification. However, it is challenging to segment Lyme disease lesions due to the lack of well-segmented, labeled Lyme datasets and the nature of Lyme, e.g., the typical bull’s eye lesion and its closeness to normal skin. In this paper, we design a simple yet novel data preprocessing and alteration method, called EdgeMixup, to help segment Lyme lesions on imbalanced training datasets. The key insight is to deploy a linear combination of lesion edge, either detected or computed, and the source image highlights the affected lesion area so that a learning model focuses more on the preserved lesion structure instead of skin tone, thus iteratively improving segmentation performance. Additionally, the improved edge from lesion segmentation can be further used for Lyme disease classification—e.g., in differentiating Lyme from other similar lesions including tinea corporis and herpes zoster—with improved model fairness on different subpopulations.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43901-8_36
SharedIt: https://rdcu.be/dnwDK
Link to the code repository
https://github.com/Haolin-Yuan/EdgeMixup
Link to the dataset(s)
https://github.com/Haolin-Yuan/EdgeMixup
Reviews
Review #2
- Please describe the contribution of the paper
The paper proposes a new data augmentation/pre-processing technique to improve lyme disease lesion segmentation. It is an iterative procedure that uses a linear combination of image and detected edges that gradually improves segmentation results.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The formulation of edge-mixup is novel i.e the idea of iteratively improving segmentation accuracy by using detected edges. Authors also consider different skin tones as computed using the Individual Topology Angles (ITA) which demostrates the utility of the algorithm in detection lyme disease lesions for diverse skin tones.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The detection or recognition of Lyme disease might be a more clinically useful application compared to the segmentation of Lyme disease lesions.
- in the section called Bias Mitigation, authors cite several papers[26-30] but it is unclear what the relationship is between the proposed approach and the references cited.
- the organization of the paper could be improved, for example the sections on related work, placement of Tables 1 & 2. There is no section for Results. Fig 4 is too small to be read properly so I suggest moving this to an Appendix and making them larger and more readable.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors have provided code and data for reproducibility
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- improve organization of the paper particularly placement of tables and figures, and typical sections like related work, results and conclusions etc must be properly organized.. -figures (for ex Fig 4) must be large and readable.
- explain the concept of “fairness” clearly and show how that relates to results given.
- Justify why Lyme disease segmentation is important for practical clinical applications as opposed to just recognition or classification of Lyme disease lesions.
- justify why ITA was used to classify skin tone as opposed to other scales like Fitzpatrick or Monk Skin Tone
- why do you report both Dice and Jaccard ? They are both related so just report one or the other.
- Show more results, including failure cases
- how are the edges of the lesion detected in the images ? Canny ? or something else. I know it is in the code but mention this explicitly. If there are multiple edges detected, does the algorithm use all edges? Even if some of these edges do not belong to the lesion ? (imagine if there is more cluttered background in the image).
- in Fig 1C - what is “legacy analysis?”
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The method proposed is novel and feasible.It has broader application even for other types of skin lesions.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #6
- Please describe the contribution of the paper
The paper proposed a method to iteratively enhance the edges of lyme disease skin lesions. The skin images were first transformed into HSV and RGB spaces to obtain the binary masks. Then the masks were combined with the original image to enhance the lesion edges for segmentation purposed. The segmented masks were used iteratively to improve the edge enhancement as well as the segmentation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- A large dataset containing 2,000 images with ground truth classification labels and 185 images with ground truth segmentation masks was built.
- An iterative segmentation approach was proposed.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Some of the details of the proposed methods are not very clear, e.g., when initializing the masks from RGB and HSV space, how are the binary masks obtained?
- It’s mentioned in the text that “all min and max values as the lower and upper bound for a candidate mask (Line 9)”, but it doesn’t match the pseudo code on Line 9.
- Not enough rationale were given in adopting some of the approaches mentioned in the paper, e.g., why “iteratively trains segmentation models from scratch”.
- The value of the classification model was not properly validated. Although the accuracies were reported in Table 2 using ResNet 34 model, but it’s unclear whether increasing the classification model accuracy would increase the final results.
- A mixture of representations were used in Table 2, e.g., some parentheses include margin of error, some include skin types as ds and ls.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The workflow is clearly described and reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- Since the performance of the classification model decides the quality of the edge candidate choices, it’s highly recommended that the authors could further compare ResNet 34 with other state-of-the-art classification models, e.g., ResNet 50 and ResNet 101.
- It’s confusing that the segmentation model is trained from scratch during each iteration. More discussions should be given here.
- A consistent format of Table 2 should be used.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The contribution of creating the dataset is very clear, but many details are not clear in the proposed algorithm.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #1
- Please describe the contribution of the paper
The authors present a new open dataset for Lyme disease skin detection and a novel simultaneous segmentation and classification process they call EdgeMixup. The EdgeMixup process involves generating mask candidates, blending input images and training a classification model for stage 1 with inputs mixed up with ground truths using the most probable mask. In stage 2, the segmentation model is trained with refinement again mixed up with original training samples. The goal is to focus more on the relevant regions distinguish Lyme disease.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is well written, clearly motivated, and important to the area. The overall idea is simple, yet effective according to the results, despite that the proposed process is still fairly sophisticated using a UNet, a ViT and the MFSNet within the pipeline. The performance is also solid across the board and convincing that the proposed method seems to work as intended. Their method is also available online for reproducibility.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
My only point to improve would be mentioning in a sentence, what happens after the Iterative Edge Improvement in the end of section 3 (i.e. fair right of Figure 2). Would be good to say that these are be adapted in the evaluation section that produce the results in tables? I may have gotten confused with the segmentation model and the segmentation task model.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors provide both their data and code
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Minor: Figure 2 text for arrows are too small to read, same for Figure 4. Extra space after Pytorch in section 5
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Everything is presented well, reproducible and results are convicing. Only minor comments about clarity raised.
- Reviewer confidence
Somewhat confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #5
- Please describe the contribution of the paper
A preprocessing method to highlight a skin lesion in an image is proposed. Experiments using two datasets investigate its effect on segmentation and classification, as well as on its relative performance on images with light and dark skin tone.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
A new dataset of 2000 skin disease images of which 185 are manually segmented is made available.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The major weakness is that the description of the EDGEMIXUP method is incomplete and very unclear (see comments in 9 below).
The paper has quite a few typos and grammar errors. The single motivating example in Section 2 is rather unconvincing as it is unusual to have fingers in an image like this, and gradcam heatmaps can in any case be misleading on such images in my experience.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The new dataset and source code are made available.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The method description (Algorithm 1 and accompanying text) is too unclear and incomplete:
- I believe the term “edge” is being used synonymously with “lesion boundary”. It would be clearer to say the latter where that is what is intended.
- In step 9 it is not specified how “edge candidates” are generated; the text on this (“EDGEMIXIP collects the minumum…”) is ambiguous and incomplete.
- In step 11 “optimal” appears to mean “most confident”, in which case it needs to be stated how confidence is defined/computed.
- I could not find a definition of the term “mixup” in this paper. If all that is meant is to take an average of two images, then it would be best to avoid the term mixup.
Given that EDGEMIXUP is a preprocessing method, why aren’t results presented for each of the ‘baseline’ methods with and without this preprocessing? It’s not clear which method has been used in the final column. When the results are presented in Table
Segmenting lesions is partly motivated in the paper (e.g. in the abstract) because it can “help clinicians focus on lesions”. Do the authors have any evidence that this is the case? I’m not sure a clinician needs any help to focus on lesions in the type of images used here.
Please provide further details on the provenance of the Skin dataset.
If I understand correctly, the SD-198 dataset used in the experiments does not include Lyme disease rashes. So I wonder if the paper title is therefore a bit misleading.
The Table 2 caption overlooks the SD-sub dataset.
In Section 4, say what datasplits were use in the first dataset.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
2
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The method is unclear and incomplete (see above). It was not clear in the results why the proposed pre-proceesing had not been used with each of the methods, so the conclusion that can be drawn is not clear.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper under review introduces a new open dataset for Lyme disease skin detection and a novel simultaneous segmentation and classification process termed “EdgeMixup.” This process blends input images and trains a classification model in stage one, with inputs mixed with ground truths using the most probable mask. In stage two, the segmentation model is trained with refinement again mixed with original training samples. The method aims to focus on the relevant regions that distinguish Lyme disease.
However, several points of critique were raised by the reviewers. The paper’s organization was found lacking, with issues regarding the placement of related work, Tables 1 & 2, and the absence of a dedicated Results section. The description of the EdgeMixup method was deemed incomplete and unclear.
There are also concerns about the paper’s clarity in its methods and rationale. For instance, the process of initializing masks from RGB and HSV space was not clearly explained. Similarly, the rationale behind some of the approaches adopted in the paper, such as the iterative training of segmentation models from scratch, was not adequately justified.
The reviewers also noted inconsistencies in the presentation of results in Table 2, with some entries including margin of error and others including skin types. Furthermore, they pointed out that the value of the classification model was not properly validated, and it was unclear whether increasing the classification model accuracy would improve the final results.
Grammatical errors and typos throughout the paper further detracted from its overall quality. The reviewers also mentioned that the single motivating example in Section 2 was unconvincing due to the unusual presence of fingers in the image and potential misrepresentation by gradcam heatmaps.
Despite these points of critique, the reviewers appreciated the creation of a large dataset containing 2,000 images with ground truth classification labels and 185 images with ground truth segmentation masks. This, combined with the proposed iterative segmentation approach, represents a significant contribution to the field.
Author Feedback
We would like to thank all the reviewers for the thoughtful comments. We answer some questions below. We will also improve our paper to address misunderstandings and writing issues in the camera ready. (1) Downstream tasks after iterative edge improvement EdgeMixup has two downstream tasks: (i) segmentation and (ii) classification. The former directly adopts improved edge for segmentation; the latter mixes up the improved edge with the original image for better classification results. (2) Clinical importance of Lyme lesion segmentation On one hand, Lyme lesion segmentation will facilitate Lyme lesion classification in prescreening of Lyme disease before clinician participation. On the other hand, such segmentation will remove unrelated skin areas, which may contain other potentially sensitive or private objects, and extract target lesion. (3) Motivating example with a finger We would like to emphasize that image quality often varies in practice and may not be ideal as in a lab environment. Practically, unrelated items—such as fingers and jewelry—often appear in images taken by either patients themselves or even clinicians. Specifically, fingers are common objects because they may not only help camera to better focus on the target skin, but also stabilize the target. (4) Mask initialization from RGB and HSV space There are two steps. First, EdgeMixup collects color range for each channel for the lesion area of all the samples and then filters out each image based on the range. Second, EdgeMixup applies canny operator to generate a binary mask based on the result from the first step and uses it as the initial mask. (5) We will improve the paper, according to reviews and the meta review, to better address writing-related misunderstandings, which include but are not limited to.
- Paper organization
- Placement of related work
- Table 1 & 2 (including result presentation such as margin of errors)
- Result section
- Grammar error