List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Žiga Bizjak, June Ho Choi, Wonhyoung Park, Žiga Špiclin
Abstract
Early detection of intracranial aneurysms (IAs) allows early treatment and therefore a better outcome for the patient.
Deep learning-based models trained and executed on angiographic scans can highlight possible IA locations, which could increase visual detection sensitivity and substantially reduce the assessment time. Thus far methods were mostly trained and tested on single modality, while their reported performances within and across modalities seems insufficient for clinical application. This paper presents a modality-independent method for detection of IAs on MRAs and CTAs. First, the vascular surface meshes were automatically extracted from the CTA and MRA angiograms, using nnUnet approach. For IA detection purpose, the extracted surfaces were randomly parcellated into local patches and then a translation, rotation and scale invariant classifier based on deep neural network (DNN) was trained. Test stage proceeded by mimicking the surface extraction and parcellation, and the results across parcels were aggregated into IA detection heatmap of the entire vascular surface. Using 200 MRAs and 300 CTAs we trained and tested three models, two in cross modality setting (training on MRAs/CTAs and testing on CTAs/MRAs, respectively), while the third was a mixed-modality model, trained and tested on both modalities. The best model resulted in a 96% sensitivity at 0.81 false positive detections per image. Experimental results show that proposed approach not only significantly improved detection sensitivity and specificity compared to state-of-the-art methods, but is also modality agnostic, may aggregate information across modalities and thus seems better suited for clinical application.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16437-8_73
SharedIt: https://rdcu.be/cVRuY
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The authors present a novel modality agnostic aneurysm detection method
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- well-written paper
- interesting approach to use surface models and pointNet
- good evaluation
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- no true baseline method, comparison to other methods is based on literature values that were achieved using different datasets.
- Dataset is biased towards larger aneurysms, which are easier to detect.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Challenging to reproduce without access to the data
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- This is a really well-written and interesting paper.
- The data should be described in more detail, especially the spatial resolution is needed. It may be the case that the method just works well because the resolution off the CTA and MRA is similar.
- The CTA in Fig 2 looks odd.
- The ground truth used to train the vessel segmentation needs to be explained in more detail.
- Another alternative for a modality agnostic detection method would be to use image-to-image translation techniques.
- No true baseline comparison
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The missing comparison to other methods is the major weakness.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The manuscript proposes a deep learning-based, modality-independent method for intracranial aneurysm detection based on a previous extraction of the vascular surface mesh. The method was demonstrated in cross-modality and mixed-modality experiments based on 500 datasets and achieved 96% sensitivity and 0.81 false positive detections per image.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The main strengths of the manuscript are:
- modality-independent method for aneurysm detection, which achieves very good results on both, CTA and MRA datasets in a study with 500 datasets, and outperforms the state-of-the-art
- well-written, concise manuscript which is easy to read
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The main weaknesses of the manuscript are:
- missing related work on automatic intracranial aneurysm detection in both modalities, CTA and MRA by Hentschke et al.: https://ieeexplore.ieee.org/document/6235669
- It remains unclear from the data description how well the used data sample reflects data from clinical routine. Has the data been acquired using the same MR and CT scanners by the same trained personell? Were there differences in the MR scanning protocol and sequence parameters? Additional information is necessary to assess whether the approach can be transferred to data from clinical routine. I also miss a brief discussion of data quality/imaging artifacts and their potential impact.
- The key part making the method modality-agnostic is the vessel extraction. As such, it must be described in more detail. It remains unclear how good the segmentation is, whether it fails in which cases and if manual refinement is necessary. The post-processing of the initial surface, e.g., smoothing, usually requires the setting of parameters. Which parameter values were chosen? In summary, it remains unclear how good the vessel extraction performs and how much the aneurysm detection depends on the quality and the parameter adjustment.
- Manual aneurysm identification was conducted by one rater. Since this is a difficult task and small aneurysms may remain unnoticed, I suggest for future work to include a second rater and also, determine inter-rater variability.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The reproducability of the vessel extraction part is rather poor.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- Background: “…methods are summarized in Table 1.” -> Table 2
- Fig 2. could be enlarged to improve readability, in particular of the heatmap.
- More details on the imaging parameters must be provided.
- More details on the vessel extraction and its accuracy must be provided.
- Impact of the vessel extraction on the detection performance must be investigated.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper presents an interesting approach and achieves very good results but the impact of the vessel extraction on the detection accuracy and the generalizability of the approach with respect to imaging parameters and therefore, clinical routine remain unclear.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The primary contribution of this manuscript is a deep learning system to detect intracranial aneurysms, tested in both MRA and CTCA imaging. This approach shows good results compared to the state of the art and does seems robust to changes in training data, at least between MRA and CTA.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Generalization of machine learning models can be difficult, and moderns trained on a specific dataset may not actually work well in practice. This work has addressed these issues atleast in part by showing that their deep learning method for detecting intracranial aneurysms performs on part or better than manual segmentation, and changes in imaging modality do not particularly affect the performance of their method.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The PointNet architecture used is somewhat outdated, and there have been significant advances for these types of architectures. Might be worth mentioning this in the discussion.
I also wonder if testing on CTA data that has at least 1 aneurysm per patient potentially underestimates the false positive rate, although the sampling performed somewhat addresses this concern.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The methodology used is reasonably clear, primarily relying on two previously published architectures (nnU-Net and PointNet). Preprocessing and parameters required are mentioned in the manuscript.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
I would suggest including the processing time for each case, as it would make evaluation of the time savings involved clearer.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The method presented shows good performance, along with robust validation strategy that shows the model can generalize to some degree to different imaging modalities. The problem of intracranial aneurysm detection is fairly important, and it is great that the authors are aware of the common pitfalls regarding generalizability of the machine learning models and attempting to rectify this.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Application paper uses effectively the known methods (nnU-net and PointNet) to solve an important clinical task of anuerysm detection. All the reviewers found the paper to be well written and the expertiments were extensive and robust, involving two main modalities. the main remark is the lack of direct comparison to alternative approaches as the reported numbers of other methods came from different datets. In the final version the authors should provide the details on the representativeness of the data used and vessel extraction performance.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2
Author Feedback
N/A