List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Hang Zhang, Rongguang Wang, Renjiu Hu, Jinwei Zhang, Jiahao Li
Abstract
Chronic active multiple sclerosis lesions, also referred to as rim+ lesions, are characterized by a hyperintense rim observed at the lesion’s edge on quantitative susceptibility maps.
Despite their geometrically simple structure, characterized by radially oriented gradients at the lesion edge with a greater gradient magnitude compared to non-rim+ (rim-) lesions, recent studies indicate that the identification performance for these lesions is subpar due to limited data and significant class imbalance.
In this paper, we propose a simple yet effective image processing operation, deep directed accumulator (DeDA), which provides a new perspective for injecting domain-specific inductive biases (priors) into neural networks for rim+ lesion identification.
Given a feature map and a set of sampling grids, DeDA creates and quantizes an accumulator space into finite intervals and accumulates corresponding feature values.
This DeDA operation can be regarded as a symmetric operation to the grid sampling within the forward-backward neural network framework, the process of which is order-agnostic, and can be efficiently implemented with the native CUDA programming.
Experimental results on a dataset with 177 rim+ and 3986 rim- lesions show that 10.1% of improvement in a partial (false positive rate < 0.1) area under the receiver operating characteristic curve (pROC AUC) and 10.2% of improvement in an area under the precision recall curve (PR AUC) can be achieved respectively comparing to other state-of-the-art methods.
The source code is available online at https://github.com/tinymilky/DeDA
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43895-0_72
SharedIt: https://rdcu.be/dnwzE
Link to the code repository
https://github.com/tinymilky/DeDA
Link to the dataset(s)
N/A
Reviews
Review #6
- Please describe the contribution of the paper
The authors introduce a deep directed accumulator (DeDA) and apply provide meaningful prioros to the training a network that detects rim+ lesions in MS. The proposed method is comapred to 3 alternatives introduced in the last few years. The proposed methods outperforms these methods. An ablation study is provied.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Thorough comparison to previous methods for rim+ lesion detection. Framework can be applied to further physically-motivated prioros.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Better references to existing literature.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Good. Using existing dataset (already published). Code to be made publicly available.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
N/A (not my domain)
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Interesting approach with wider applicability, and sensible performance gain on a interesting medical task.
- Reviewer confidence
Not confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #5
- Please describe the contribution of the paper
The paper describes two manually designed features for an improved MS lesion classification. The features can be partially learned and be integrated into a deep-learning architecture. The given approach is evaluated on a dataset and it is shown that it performs better than the current state of the art.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
I really liked the approach of adding prior knowledge to the network. With the current trend on machine-learning-based solutions, it is sometimes forgotten that a good engineered feature can be much more reliable and stronger. Also, the paper uses a non-standard approach which makes it interesting.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The main weakness of the paper is that it is rather application specific. I am not sure if the proposed technique can be applied to any other task. Also, training and evaluation were done on the same dataset, only using cross-validation for evaluation. This could lead to overfitting during the development (although that is not necessary). Finally, the training details are missing. (They might be in the appendix, but it exceed the maximum appendix length and does not conform to the required format)
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
I think that it is difficult to reproduce the results from the paper. A private dataset was used, and I found no (future) link to the code. The reproducibility statement has some points checked that I could not find in the paper (Same goes for some other papers I reviewed): Hyperparameter tuning, sensitivity to parameter change, baseline method implementation, tendency of results, and significance. Further, the following points are marked as not applicable but would have been relevant: Runtime, memory footprint, central tendency,
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- I missed details on the training procedure, the hyperparameter tuning etc.. This seriously affect the ability to judge the reliability of the findings.
- The authors claim that CNN cannot be used for RIM-classification. (Page 6). However, this ignores the fields of single-shot learning, transfer learning etc… . This statement should either be extended or removed.
- The ablation study is done on the same dataset as the main study. Consequently, this is an unfair benefit for the proposed method. Ablation studies should always be done on a left-out dataset or on a subset of the data only.
- Please report confidence intervals for all measures. The differences between the methods are really small, so it is important to know if they are by chance or systematically.
- In the evaluation, the authors use Pearson’s Correlation to relate predicted and human-based lesion counts. However, Pearson’s R requires a) normal distribution and b) outlier-free data. Both are not the case here (Judging from Figure 4a). Also, the data seems highly imbalanced with most images having only one or two lesions, leading generally to the question of whether correlations should be done.
- The authors state that there have been three methods developed for RIM+ identification (Page 6). This seems to be strange. That sounds oddly confident. Did the authors a complete literature research? What is with the work of Macin et al., (10.3390/app12104920)? I think this statement should be weakened. I do not think that a reference method is missing.
- Please also include a baseline approach using a standard ResNet (or any other standard architecture) with pretraining. This would be the most common approach right now.
- Figure 4: The scatter plot. I think this plot is highly misleading. As it displays the lesion count, the underlying data should not be continuous. Also, counting cannot be negative. I am aware that this was most likely caused by the plotting function, but nevertheless, the figure is far from an accurate representation.
- The paper would benefit significantly from having source code released. The described method is far from obvious and simple to implement.
- I am not convinced that the math in the paper is really needed. I think the paper would greatly benefit from a more straight-forward explanation of what was done rather than all the formulas.
- The second contribution mentioned is basically no contribution but the evaluation of the proposed method. I am fine with adding this, but the wording is strange. Is your contribution really that you performed a standard evaluation?
- Page 3 first paragraph. “two folds” should be twofold. Otherwise the meaning wouldn’t fit in there.
- Page 3, Section 2.1. “Fig… and Eq (3) have shown that this DA”. This is grammatically wrong. Either they show (change in grammatical time) or indicate or illustrate. Please correct.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The proposed approach of the paper is interesting, but the evaluation is limited to private datasets, the proposed solution is specific to a single application and the writing is unnecessarily complex. Also important information on implementation, optimization etc.. were missing. Given those pros and cons, I go for a weak acceptance. For this decision, I ignored the Appendix, which is not following the current MICCAI rules w.r.t. to page count and content.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This paper presents a novel image processing operation, known as DeDA, that improves the identification of chronic active multiple sclerosis lesions. A domain-specific inductive bias injection is proposed that enables identify rim+ lesions by creating and quantizing an accumulator space into finite intervals and accumulating feature values according to a set of sampling grids. The method demonstrated a significant improvement in the false positive rate and the area under the precision-recall curve, indicating its effectiveness in identifying rim+ lesions.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper is addressing an important problem in medical image analysis, i.e., how to explicitly encode the above prior information into networks.
- The experimental results seem promising, with quite significant improvement in both pROC AUC and PR AUC.
- The designed deep directed accumulator (DeDA) is a generic image processing operation.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The proposed method seems to be loosely linked to the original motivation or the problem to be resolve.
- The method sounds like a generic one, but the evaluation is only performed on a very specific task. More disscussion on the limitation or applicability of DeDA is desired.
- Comparison is done with three very specific methods.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
reproducibility looks okay.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- The deep directed accumulator (DeDA) seems to be distantly linked to explicitly encode the above prior information into networks. To me, it more like a operation to better obtain global representation. The authors should analysis the limitation of DeDA in encoding priors. Is it restricted to geometric priors only?
- I also wonder how it ‘explicitly’ encode the domain specific prior? Even in the case of chronic active multiple sclerosis lesion classification, I cannot see how it happens.
- DeDA seems to be a generic operation that would be beneficial to a range of applications with dependent on global representation. I would suggest the authors to experiment with more tasks and datasets to show it general applicability.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper is addressing a very specific problem with a seemingly generic method proposed. I would suggest the authors to consider evaluating on more wide set of tasks to show its effectiveness.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #7
- Please describe the contribution of the paper
This paper proposes the Deep Directed Accumulator (DeDA) approach for identifying chronic active multiple sclerosis lesions. By incorporating domain-specific inductive biases, DeDA significantly enhances lesion detection accuracy, as demonstrated in experiments conducted on a large dataset. This study offers a new perspective for improving medical imaging analysis by injecting domain-specific knowledge into neural networks.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The Deep Directed Accumulator (DeDA) is a novel approach to identifying rim+ lesions. The method to inject domain-specific inductive biases into neural networks is simple and effective.
The paper includes experiments on a large dataset with 177 rim+ and 3986 rim- lesions, demonstrating that DeDA outperforms other state-of-the-art methods. The evaluation includes both the area under the receiver operating characteristic curve (pROC AUC) and the area under the precision recall curve (PR AUC), providing a comprehensive assessment of performance.
The proposed method has potential clinical applications for identifying chronic active multiple sclerosis lesions, which can aid in diagnosis and treatment planning.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Improvemenet aganist QSMRimNet is marginal.
The DeDA approach is a complex operation that may be difficult to interpret or explain to clinicians or non-experts. Further work could explore ways to improve interpretability and transparency of the method.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The details of the method are described, which is technically sound. Codes are not available.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Typos, e.g., keywords: driected accumulator - > directed
Please refer to the weakness for more detail.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The method is novel and the technique part is sound.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper presents a novel image processing operation to improve the identification of chronic active multiple sclerosis lesions. The addressing topic is clinically important and the generalization of the technique is good enough. Since all four reviewers shared a great enthusiasm, I would recommend accept.
Author Feedback
N/A