List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Jun-Ho Kim, Mohammed A. Al-masni, Seul Lee, Haejoon Lee, Dong-Hyun Kim
Abstract
Cerebral Microbleeds (CMBs) are chronic deposits of small blood products in the brain tissues, which have explicit relation to cerebrovascular diseases, including cognitive decline, intracerebral hemorrhage, and cerebral infarction. However, manual detection of the CMBs is a time-consuming and error-prone process be-cause of the sparse and tiny properties of CMBs. Also, the detection of CMBs is commonly affected by the existence of many CMB mimics that cause a high false-positive rate (FPR), such as calcification, iron depositions, and pial vessels. This paper proposes an efficient single-stage deep learning framework for the au-tomatic detection of CMBs. The structure consists of a 3D U-Net employed as a backbone and Region Proposal Network (RPN). To significantly reduce the FPs, we developed a new scheme, containing Feature Fusion Module (FFM) that greatly detects small candidates utilizing contextual information and Hard Sample Prototype Learning (HSPL) that mines CMB mimics and generates additional loss term called concentration loss using Convolutional Prototype Learning (CPL). The proposed network utilizes Susceptibility-Weighted Imaging (SWI) and phase images as 3D input to efficiently capture 3D information. The proposed model was trained and tested using data containing 114 subjects with 365 CMBs. The performance of vanilla RPN shows a sensitivity of 93.33% and an average number of false positives per subject (FPavg) of 14.73. In contrast, the proposed Feature Fused RPN that utilizes the HSPL outperforms the vanilla RPN and achieves a sensitivity of 94.66% and FPavg of 0.86.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16431-6_43
SharedIt: https://rdcu.be/cVD6Z
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes a single-stage 3D deep learning method for automatic detection of cerebral microbleeds based on susceptibility-weighted imaging (SWI) and the phase images. Compared to the literature, the study removes the need of a second-stage learning for reducing false positives. Instead, it adds a feature fusion module (FFM), as well as a hard sample prototype learning (HSPL) approach. Collective strategies seem to outperform existing models.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1) Introduces a 3D single-stage deep convolutional neural network for automatic detection of cerebral microbleeds, which can be time-consuming and subject to error in clinical practice. 2) Includes comparison of the proposed method with 2 approaches without addition of the customized learning components 3) Overall results appear promising
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) The description of RPN is overly brief, and the mechanisms and utility of HSPL is not clear 2) All the image examples shown seem to be SWI scans. How the phase images are used are not illustrated.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
No strategy is mentioned
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
This paper presents a new deep neural network approach for detecting cerebral microbleeds based on a 1-stage implementation instead of 2 stages as seen in the literature. With the demonstrated performance, this method definitely warrants further validation. Additional attention to the following points should help enhance the paper.
Introduction 1) It is worth noting that neuroradiologists do not always have trouble to detect the microbleeds, and often do not need to quantify them if not clearly indicated. See end of 1st paragraph, page 2.
Method 1) In section 2.1, are the microbleeds incidental findings in the 114 subjects or under some specific diseases? What are the size ranges of the microbleeds? That would impact model performance. 2) In the same section, are there any co-registration procedures applied in data preprocessing, why and why not? What does ‘random cropped data’ mean – referring to any location of the brain? 3) In section 2.2, the term ‘number of lengths’ is confusing. E.g. how would a bounding box sized 20x20x20 give ‘the number of lengths for each dimension to zero’? A related question, in Fig. 1, do all the feature maps to be fused have the size of 32 with 16 channels? The middle part is unclear. 4) In section 2.3, please explain this sentence: ‘due to the sparse and tiny properties of CMBs, the HSPL crops the data based on the rule that the number of crops containing CMBs corresponded to the crops not containing the CMBs’; e.g. how would the two crops correspond to each other? Overall, as mentioned above, this and the RPN sections would benefit from additional information.
Experiments 1) The method proposes to use both SWI and phase images but it is unclear how that is implemented, alone or together. 2) (Minor) For figure captions, start with a brief title would help improve clarify.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper includes several innovations as seen in the strength, and the proposed method can be useful for research in different topics. A few weaknesses need to be addressed.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The authors propose a novel single-stange deep convolutional neural network that combines a 3D-Unet with the Region proposal network (of Faster R-CNN) as well as a feature fusion module and a convolutional prototype learning-based loss term to detect cerebral microbleeds in SWI MRI.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
• The authors combine several known techniques in a sensible manner and show that their method is outperforming simpler baselines. • The results may indicate a preferable performance compared to the cited state of the art. • The authors performed quantitative as well as extensive qualitative evaluations employing the visualization of feature maps and probability maps.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
• A direct comparison of the performance to methods of the state of the art is not possible because different data sets are used for every method including the proposed one, which was trained and evaluated on an in-house data set. The authors did not compare to a state-of-the-art method on their data set. • The authors state correctly that ratings of expert neurologists are error-prone and that there are substantial inter-rater differences, yet this potentially important issue is not addressed in the paper. The method was (supposedly) trained using labels provided by a single rater (of a group of experts) per subject. • No limitations or future work are discussed or mentioned. • more quantitative evaluations wrt. to the choice of a Unet backbone as compared to the original Faster R-CNN architecture are missing
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
• the method is insufficiently described as crucial details such as the used optimizer, the number of epochs or training iterations and batchsize (if any) are missing. • The reproducibility relies on the availability of the used code and the data set as well.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
• The most promising concurrent method of the references should be trained and evaluated on the author’s data set to be able to directly compare performance conclusively. • The limitations of this methods should be shown and discussed. • The wording should be improved, e.g. ◦ stating that with two-stage models “there is an annoyance” is hardly objective and valid criticism and objective arguments should be found to motivate a one-stage approach. ◦ The sentence “Obviously, the proposed net significantly detected the CMBs […]” should be rephrased as well. • All important details on the training procedure need to be added. • A cross validation-based evaluation scheme would be beneficial and avoid a possible testset bias
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
• A discussion or at least mentioning of the limitations, possible improvements and future work is missing entirely. • Furthermore, a prominent issue is that the results cannot be directly compared to the state-of-the-art methods hence the significance of them is somewhat unclear
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper proposes a single-stage deep learning framework for the automatic detection of cerebral microbleeds. The structure of the net consists of an initial encoding based on the 3D U-Net while the decoding part is merged with a region detection network (YOLO-based).
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Paper merges u-net with object detection architectures to detect and classify all in one step (avoiding the usual detection + false positive reduction step).
- Paper uses feature fusion model to incorporate contextual information-
- A hard sample prototype learning module is introduce to gauge the false positives location and use this information in the metric to reduce the number of false positives.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- It seems that the results are obtained with images of patients with CMBs. What would be the performance in healthy brains?
- Generalisation analysis (i.e. testing in a different dataset) is not shown.
- Analysis of the loss contribution is not performed.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Difficult to replicate. There are a large choice of hyerparameters which are not explained in detail.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
In this paper authors aim to detect cerebral microbleeds in MRI images. They fuse ideas from the U-Net and the YOLO-based architectures. A total of 114 subjects including 365 CMBs were used to train & test the approach.
The technological proposal seems feasible, although authors needed to add an “artificial” loss (ie, concentration loss) to enhance the results. Without this term the results were improved with respect the U-Net, but not significantly. The performance of the simple U-Net using the concentration loss is not shown.
I have some issues regarding the dataset. Firstly, since authors used only diseased brains, I’m not sure if the approach will be able to detect non-diseased brains (ie, if the brain is healthy, would the net do not detect any lesion?). An analysis with normal brains should be added in the experimental section.
My second main concern is about generalisation. Using images from the same dataset is a very favourable scenario for deep learning algorithms. Authors should test what is the performance of the algorithm in images from a different dataset.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The aim of the paper is interesting. However, the experimental evaluation is limited and specific to a single dataset.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Somewhat Confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper presents a single-stage deep learning approach for automatic detection of cerebral microbleeds. Experimental results demonstrate better overall performance. The reviewers have affirmed the merits of paper while issues such as implementation details, discussion of drawback and generalization analysis need to be resolved in the final version.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2
Author Feedback
Reviewer 1 5.1) The description of RPN is overly brief => The RPN of proposed model is almost same with vanilla Faster R-CNN. The important part of RPN is the number of channels of output and i explained that in detail (2.2, page 3).
-Method 8.1) Are the CMBs incidental findings in the subjects or under some diseases? What are the size ranges of the CMBs? => The subjects are composed of normal elder, dementia, Alzheimer’s disease, vascular dementia and mild cognitive impairment. The size ranges of CMBs is almost under 5mm and the limit of size is 10mm.
8.2) Are there any registration procedures? What does ‘random cropped data’ mean? => There are no registration procedures. SWI images are generated with magnitude images and phase images. So, there spatial locations are always matched. Random cropped data referring to crop any location of brain.
8.3) The term ‘number of lengths’ is confusing. how would a bounding box sized 20x20x20 give ‘the number of lengths for each dimension to zero’? => The term ‘number of lengths’ means length of bounding box for each dimension. The size of bounding box is fixed 20 x 20 x 20, which draw rectangular at the UI. If bounding box is not fixed and have to be predicted, we need more channels (Nld) for bounding box size prediction.
8.4) Explain ‘the HSPL crops the data based on the rule that the number of crops containing CMBs corresponded to the crops not containing the CMBs’. => The number of cropped data containing CMBs equal to the number of cropped data not containing CMBs.
Experiments 8.1) The method proposes to use both SWI and phase images but it is unclear how that is implemented. => As you can see Fig. 1, the number of input data channel is 2. The input data is composed of SWI image and Phase image.
Reviewer 2 5.1) The authors did not compare to a state-of-the-art method on their data set => The methods of the state-of-the-art do not provide code or data set. We will consider about coding methods based on paper and comparing the models with same data set in future journal paper.
5.2) The method was (supposedly) trained using labels provided by a single rater (of a group of experts) per subject. The inter-rater differences is not addressed. => In this work, two experts have labeled, where the gold standard was obtained based on the consensus of all the raters. We did not inspect the rater differences.
5.3) No limitations or future work are discussed or mentioned. => we have added the limitation and future work in revised paper.
5.4) The choice of a Unet backbone as compared to the original Faster R-CNN architecture are missing. => The proposed model is 3 dimension version of Faster R-CNN and just used 3D U-Net as backbone. There is no original 3D Faster R-CNN.
8.5) A cross validation-based evaluation scheme would be beneficial and avoid a possible testset bias => We agree with the reviewer, and we will add this cross evaluation in the future journal paper along with additional test data from different scanner to address the generalization issue.
Reviewer 3 5.1) What would be the performance in healthy brains? => We think that the non-diseased brain also show a false positive rate similar to that of the diseased brains. The difference among diseased brains and a non-diseased brains is just the presence or absence of CMB, which does not affect the false positive rate. However, we will collect healhy subject data and examine the false positive rate in future work.
5.2) Generalisation analysis (i.e. testing in a different dataset) is not shown. => It’s good idea. we will involve this analysis in the future journal paper.
5.3) Analysis of the loss contribution is not performed. => Actually, we perfomed this analysis in our paper. You can observe how much improvement in the performance was achieved after adding the concentration loss (equation 4) compared to the Feature Fused RPN that did not use this loss. Please refer to Table 1.