List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Yi Shi, Rui-Xiang Li, Wen-Qi Shao, Xin-Cen Duan, Han-Jia Ye, De-Chuan Zhan, Bai-Shen Pan, Bei-Li Wang, Wei Guo, Yuan Jiang
Abstract
In the field of plasma cell disorders diagnosis, the detection of abnormal monoclonal (M) proteins through Immunofixation Electrophoresis (IFE) is a widely accepted practice. However, the classification of IFE images into nine distinct categories is a complex task due to the significant class imbalance problem. To address this challenge, a two-sub-task classification approach is proposed, which divides the classification task into the determination of severe and mild cases, followed by their combination to produce the final result. This strategy is based on the expert understanding that the nine classes are different combinations of severe and mild cases. Additionally, the examination of the dense band co-location on the electrophoresis lane and other lanes is crucial in the expert evaluation of the image class. To incorporate this expert knowledge into the model training, inner-task and inter-task regularization is introduced. The effectiveness of the proposed method is demonstrated through experiments conducted on approximately 15,000 IFE images, resulting in interpretable visualization outcomes that are in alignment with expert expectations.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43987-2_15
SharedIt: https://rdcu.be/dnwJx
Link to the code repository
https://github.com/shiy19/IFE-classification
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes a two-sub-task classification approach for Immunofixation Electrophoresis (IFE) image classification. The classification task is first devided into severe and mild cases instead of 9 classification tasks to handle the class imbalance of the data. In addition, the expert knowledge is handled through inner-task and inter-task regularization. The method is tested on about 15000 IFE images and acheived a good performance.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The manuscript handles a less visited topic (IFE image classification) and address the class imbalance.
- Several experiments and ablation study is perfromed and the method seems to have good perfromance.
- The study collected a relatively big size clinical dataset and the source code is provided.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- So much description and many repeated sentences.
- No clear figure to the architecture of the proposed method.
- Few writing mistakes.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The link of the source code is given in the the Implementation details (Section 3).
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- Long sentences are not preferred as they increase the ambiguity.
- Too long captions of Figs. 1 and 2.
- Some abbreviations are used without being defined before, e.g., CT, CNN, SGD, …etc.
- There are some old and irrelevant references, please rectify them. -Typos:
- “is introduced” in the Abstract should be “are introduced”
- “represents” in the caption of Fig.1 should be “represent”.
- In Subsection 2.1, “Let ϕ(·), h(·)” should be “Let ϕ(·) and h(·)”
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Please see my comments on items 6 and 9.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
-
Propose an end-to-end multitask method for IFE image classification that effectively addresses the class imbalance issue
-
Implement two types of regularization mechanisms to more accurately model relationships within and between tasks
-
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
It is innovative to employed a multi-task method for immunofixation electrophoresis image classification
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
It is not clear that how the methods described in section 2 can solve class imbalance problem.
-
Some topics are significantly overstressed without any need. The authors need to structure the content better, to facilitate legibility and understanding.
-
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Although data and code are not provided, the method description is clear. I expect the method to be reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
-
The motivation behind the work is not fully clear and state of the art needs improvement.
-
The contributions of the work are not clear. What is novel about the proposed approach?
-
It is not explained how the approach deals with class imbalance. I can make a guess, but why not tell the reader, since this is one of the main points of the paper?
-
The authors need to give some experiments to show the influence of parameters.
-
In comparison with the published paper, what is the superiority of the proposed method in the manuscript? The authors should clarify.
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Taken as a whole, the paper provides good experimental results but we do not clearly understand why the technique is well appropriate to solve the problem. Moreover, the manuscript is affected by many flaws which limit the correct assessment of its value. Meanwhile, the presentation of the manuscript has to be improved.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
3
- [Post rebuttal] Please justify your decision
After reading the comments of all reviewers and the rebuttal from authors, I would like to share my opinions bellow.
- The paper is well-written and easy to follow.
- The proposed method may be effective, however, with simple content. Taking the above considerations, I think this manuscript may be not suitable for presentation at the prestigious conference MICCAI.
Review #4
- Please describe the contribution of the paper
The paper focuses on the classification of images of Immunofixation Electrophoresis in the context of class imbalance. The authors propose a domain-specific approach with multi-task classification allowing for effectively identifying the class of interest (i..e, IdG-k positive). The authors also propose inner- and inter-task regularizers to further improve the performances of the algorithm. The performances are evaluated in a k-fold manner and ablation results are provided illustrating the advantages of the proposed method.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper focuses on a clinically relevant task and addresses a technically appealing issue of class imbalance. While the multi-task setup is domain-specific, it can still be projected on other medical imaging fields so the paper might be of interest to a broader community. Overall it is quite well written and reads fluently.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Two main weaknesses apear to me. First, the state-of-the-art is worth a bit more discussion with regard to the multi-task approaches (e.g., https://doi.org/10.3389/fradi.2021.796078). Second, the results merit further discussion as the presented performances appear (Table 1 and Table 2) quite close.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors provide a fair amount of details about the training setup and the architecture being used. With the dataset being private, the dataset statistics are available in the supplementary material.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
I would like to thank the authors for their work. I have a few comments that I hope the authors will find useful. - I suggest the authors provide some more discussion about state-of-the-art of multi-task learning in medical imaging (e.g., https://doi.org/10.3389/fradi.2021.796078) for better comprehension of the contribution. - In the Dataset paragraph the authors state the “statistical results of the dataset” and refer to the supplementary material. I suggest rephrasing it as “dataset characteristics” and providing a few words in the paper (at least the class imbalance ratio) - In Performance Evaluation, Tables 1, and 2 show very comparable numbers. Could the authors provide more comments in that regard? Moreover, in Table 2 (and also in Table 1 of supplementary material), the M1 column contains some identical numbers. Could the authors verify if it is not a typo? - In Performance Evaluation, the authors mention the “conventional multi-class approach”. Could the authors clarify if it is the same as “Valilla multi-class” as in Fig. 1 (b)? If it is so, could the authors comment on the similarity of metrics? - Overall, I suggest revising the inclusion of the figures and tables. That is, the figures and tables (e.g., Tables 1, 2) are often appearing too early in relation to the content referencing them. This makes navigation through the paper difficult.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper is generally clear and the method is quite well presented (except the order of figures and tables). The multi-task nature of the method may be of interest of the community.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
The paper remains of interest. The authors feedback is relatively well organized and comprehensive. Given the difficulty of using M1-Acc metric I would suggest the authors to revise the presentation of metrics for the sake of clarity. Hence, my note do not change.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper has an interesting application, can be considered novel, and there are some technical adjustments related to class imbalance problem. There seems to be merit by two reviewers, and there are also some moderate to major weaknesses about the novelty and justification and details necessary to justify the intuition behind the method presented.
Author Feedback
We thank all reviewers for valuable feedback. We are pleased they find our multi-task method to be innovative (R3), the method description to be clear (R3), and the paper to be well-written and of interest to a broader community (R4).
We first reiterate our contributions: 1) we propose an end-to-end multi-task method for IFE image classification that effectively addresses the class imbalance issue, 2) we implement two regularizations to more accurately model relationships within and between tasks, and 3) Experiments and visualization demonstrate the effectiveness and explainability of our proposed method.
Next, we answer some general questions. @R3 Why our multi-task method can address class imbalance: When we decompose the original task into two sub-tasks, different minor classes of the original 9-class task that shares the same severe or mild cases are combined into larger minor classes for the two sub-tasks, e.g., both ‘IgA-k’ and ‘IgM-k’ contributes to the mild case ‘k’, which can be proven by the histogram in Fig.2(a). Therefore, this decomposition alleviates class imbalance. Inspired by @R4, if we define the class imbalance ratio as the minimum class number divided by the maximum class number, we can get the imbalance ratio of the original task as 63/10373=0.006, and that of two sub-tasks as 1955/10373=0.188 and 447/10549=0.042. Since a bigger ratio means less class imbalance, we can more intuitively feel that class imbalance has been alleviated. We will make it more clear in the final version.
@R3 What’s the novelty and superiority of our method: To the best of our knowledge, our paper is the first attempt to use a multi-task method in IFE image classification to address class imbalance. We agree with @R4 that this idea can still be projected on other medical imaging fields, so the paper might be of interest to a broader community. Besides, inspired by expert knowledge, we propose two novel regularizations to model relationships within and between tasks. Experiments demonstrate our method is superior to other methods, especially on metrics highly sensitive to class imbalance (F1-score, Fnr, M1-Acc, and M3-Acc). Furthermore, our method is the first end-to-end method in IFE image classification.
@R3, R4 More discussion on the performance: As shown in Table.1, our method is not significantly better than the vanilla multi-class method on the Acc metric, because Acc is the average accuracy on all samples, which is dominated by the accuracy of major class samples and NOT sensitive to that of minor class samples. Our method mainly deals with class imbalance and can perform better on minor classes. Since other metrics are more easily affected by minor class samples, our method is superior to other methods with a big gap on these metrics. Notably, in the medical field, positive samples are usually minor classes, but exactly what we should pay more attention to.
Following are some detailed questions. @R1 Figure to the architecture: Fig.1(c) illustrates our proposed multi-task framework. We will make it more clear in the final version.
@R3 Influence of parameters: Table.1- Table.4 in the supp shows the influence of different parameters.
@R4 Identical numbers in M1-Acc: It’s hard to clinically collect more samples of the top-1 minor class. Due to the small number of samples, there is no gap between some methods on M1-Acc.
@R4 Similarity of metrics: ‘conventional multi-class’ is the same as ‘vanilla multi-class’. Acc is less sensitive to minor class samples compared to other metrics. All the other metrics can better measure the performance of models on minor classes, and should be more concerned.
@R4 More discussion on the SOTA multi-task method: Thanks for the suggestion. We will cite the paper and give more discussion on multi-task methods in other medical fields.
@R1, R3, R4 Advises on writing, presentation, and references: Thanks for the suggestions. We will modify them one by one in the final version.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
answers in the rebuttal phase are satisfying, and encouraging. The paper is in good shape with further clarifications.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
After rebuttal, 2 reviewers lean to accept, and one tends to reject. Based on my reading of the paper and rebuttal, a decision of accept is recommended according to the overall quality of the paper (it is indeed well-written and of interest). However, I also encourage the authors to eliminate the concerns of the reviewer 2 which helps to enhance the impact of the paper on the MICCAI society.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Mixed review. Some level of novelty. One reviewer lowered the score after rebuttal. Some concerns seem to be moderate/minor. To me it is not exciting and I see it a borderline paper.