List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Youssef Assis, Liang Liao, Fabien Pierre, René Anxionnat, Erwan Kerrien
Abstract
The diagnosis of unruptured intracranial aneurysms from Magnetic Resonance Angiography (MRA) images is a challenging clinical problem that is extremely difficult to automate. We propose to go beyond the mere detection of each aneurysm and also estimate its size and the orientation of its main axis for an immediate visualization in appropriate reformatted cut planes. To address this issue, and inspired by the idea behind YOLO architecture, a novel one-stage deep learning approach is described to simultaneously estimate the localization, size and orientation of each aneurysm in 3D time-of-flight MRA images. It combines fast and approximate annotation, data sampling and generation to tackle the class imbalance problem, and a cosine similarity loss to optimize the orientation. We evaluate our approach on two large datasets containing 416 patients with 317 aneurysms using a 5-fold cross-validation scheme. Our method achieves a median localization error of 0.48mm and a median 3D orientation error of 12.27 degrees, demonstrating an accurate localization of aneurysms and an orientation estimation that comply with clinical practice. Further evaluation is performed in a more classical detection setting to compare with state-of-the-art nnDetecton and nnUnet methods. Competitive performance is reported with an average precision of 76.60%, a sensitivity score of 82.93%, and 0.44 false positives per case.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43895-0_51
SharedIt: https://rdcu.be/dnwzj
Link to the code repository
https://gitlab.inria.fr/yassis/DeepAnePose
Link to the dataset(s)
https://openneuro.org/datasets/ds003949/versions/1.0.1
Reviews
Review #1
- Please describe the contribution of the paper
This paper propose a novel approach to detect as well as to estimate the pose of intracranial aneurysms simultaneously using ToF-MRA images. Furthermore, the ground truth data are generated use a simplified annotation methods, which reduce the effort of the data collection largely. The quantitative evaluation results are generated with clinical datasets, where one dataset is acquired dedicatedly for this study and one dataset is publicly available.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The manuscript is well structured and well-written. A extensive description of the related work is provided, which makes it easier to understand the novelty of the proposed work. Although YOLO architecture is widely used for pose estimation in computer vision (Human, cars. etc.), it s rarely adapted for the detection and pose estimation of aneurysms. Furthermore, the annotation process is more straightforward then the state-of-the-art voxel level annotation. The validation on the two datasets are also comprehensive. The results of pose estimation correlates well with the measurements in real clinical workflow. For object detection, the proposed method outperforms the state-of-the-art methods.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
No major weakness. If the author could provide the implementation as open source code, it will increase the impact of this work. The authors should chose a more suitable typeset for the math expression, especially for the eq. 1-4, to make the equations more readable.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Except the evaluation with the private dataset, other aspect of the paper can be reproduced. The network architecture is well described, implementation detail are given, both evaluation metrics and validation schema are detailed.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
For me, this is a good work, which as a great potential to be adapted clinically. If possible, I suggest the author to run a clinical validation of this method in there future work.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
well motivated, has a clear structure, addresses a clinically relevant problem. Novelties are clear and the method shows very promising quantitative results with great potential for clinical adaptation.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The paper proposes a novel one-stage deep learning approach, inspired by the YOLO architecture, for the simultaneous localization, size estimation, and orientation estimation of intracranial aneurysms in 3D time-of-flight Magnetic Resonance Angiography (TOF-MRA) images. The proposed method addresses the class imbalance problem using fast and approximate annotation, data sampling, and generation. It also uses a cosine similarity loss to optimize the orientation estimation. The approach achieves a median localization error of 0.48mm and a median 3D orientation error of 12.27 degrees, demonstrating accurate localization and orientation estimation of aneurysms that comply with clinical practice. Additionally, the method is compared with state-of-the-art methods in a more classical detection setting, showing competitive performance with an average precision of 76.60%, a sensitivity score of 82.93%, and 0.44 false positives per case.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Main strengths of this work are:
- Presents a novel application of state-of-the-art YOLO object detection to both localize and estimate orientation of aneurysm (using multiple heads on top of YOLO model) - Application is on 3D MI data
- Contains a unique dataset that (if released publicly) would be beneficial to the community, although no such indication is made in the paper
- It is clear and presented well with description of datasets, proposed methods and experimental validation
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
While the paper presents a novel application of YOLO object detection model and utilizes novel aneurysm detection and pose datasets, it does have the following weaknesses:
- While not a major weakness, the paper lacks originality/novelty of proposed methods where the main contribution comes from application of YOLO on 3D detection and pose estimation. Therefore, I have classified this to be an application paper where the main novelty/originality is in application of YOLO on aneurysm detection and pose estimation and the datasets used.
- Experimental validation section lacks significantly where only a comparison is made with nnUNet/nnDetection models. As this is an application paper, I expect it to include detailed analysis and comparison with existing state-of-the-art detection models (e.g. see a list here: https://paperswithcode.com/task/object-detection). Some of the existing models are SSD, EfficientDet, Faster R-CNN etc..
- Pose estimation results need to be presented with the help of a table, where a comparison should be made with existing techniques that can be used for pose estimation. At the moment such comparison is missing, at the least a baseline comparison is required, however including more methods may be beneficial in putting this work into broader community perspective.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
This paper includes implementation detail section with sufficient information which allows it to be reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
See above
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
See weaknesses. The main being lacking experimental validation/comparison with existing sota in detection as well as pose estimation.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
There are still a few weaknesses in the experimental validation and comparison with SOTA detection and pose methods. However, given additional context and improvements promised in rebuttal, I am happy to revise my decision as leaning towards a weak accept.
Review #3
- Please describe the contribution of the paper
The paper describes a network approach for aneurysm detection and joint estimation of positing and pose (e.g. orientation). A simple CNN architecture is chosen, with the complete layout inspired by YOLO. Two datasets are used to evaluate the performance of the given approach, and it is evaluated w.r.t. both pose estimation and detection performance. For the latter, it is shown that the performance is better than two current state-of-the-art approaches (nnDetection and nnUnet).
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper represents an interesting extension of current research. The underlying approach has a clear goal and a straightforward working thesis. The joint estimation of pose and detection makes sense and the performance justifies the extension. The evaluation is well done and trustworthy. The work is also well written and easy to follow.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The paper itself is simple and the results presented are not surprising. Consequently, the paper has a very solid feel to it, but does not appear to be rocket science.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The methods described in the paper seem sufficient to reproduce the results. The reproducibility points given are mostly valid, more so than in any other paper I have reviewed. It was particularly nice that the authors (only in 5 papers reviewed) included a placeholder for a link to the cod. The following points were checked in the reproducibility response, but I couldn’t find them in the paper: I) Hyperparameter tuning II) Runtime, III) Memory footprint, IV)
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The paper is generally interesting and well written. I can only encourage the authors to maintain this good writing style. However, there are a few points that could be improved:
- The figures in the paper are very small. It is almost impossible to read the text in the figure without zooming in considerably. This applies to all the figures, including the screenshot showing Slicer.
- The authors made two different evaluations. However, it is a bit surprising that they were done on independent datasets. Please also report the pose estimation quality on dataset 2 and the detection performance on dataset 1.
- Please check the paper for unwarranted claims. For example, at the bottom of page 6: The authors claim that they have “excellent performance” on both datasets. This is a high claim and the numbers do not justify this claim, at least in my view. (This is not to say that there is poor performance…)
- Why did the authors choose to compare their method with the first and third ranked competitors and not the second?
- Please report the results for the pose results as a table, maybe in the appendix?
- Does Table 1 show the standard deviation or the confidence interval? Please clarify.
- Please justify why sensitivity and FPs/case are a good trade-off for your method? What is the desired objective here? This seems to be a rather arbitrary claim.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper is solid and easy to read. The topic is of interest to the main MICCAI audience and the evaluation is reliable. However, neither the findings nor the technique are outstanding and there are minor flaws in the paper. The paper also slightly exaggerates the results. This leads me to a weak accept with a tendency to accept.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
For me, the paper is interesting and adds to the current knowledge. The analysis is reasonable and with two rather large datasets, there are enough data to validate the approach. I follow the authors with their argument that the application and the usage of non-point-based data is a novelty. The current state of the art that they used seems to be sufficient and fair to me. Therefore, I am changing my opinion to accept the paper.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper is about the detection of intracranial aneurysms and the estimation of their pose in TOF-MRA images. The method is developed based on the deep learning method and inspired by YOLO. Comparing with nnDetection and nnUNet, the evaluation results are good in terms of median localization error and median 3D orientation error. There are some concerns raised by the reviewers. Clarifications are needed on the method novelty, the method comparison with other related state-of-the-art aneurysm detection methods, and the pose estimation and detection performance comparison on both datasets.
Author Feedback
We sincerely appreciate the diligent evaluation and valuable feedback from the reviewers. Intracranial aneurysm pose estimation is challenging because it combines many difficulties: scarcity, small size and unknown number of objects to detect, input data dimension, limited datasets, and burden of data annotation. Our method addresses them all through dedicated data sampling (small patches, data augmentation, and synthesis, balanced sampling), a 3D anchor-free architecture and fast data annotation.
Method novelty (R2,R3): We believe our method is clinically original. It is the first, to the best of our knowledge, to propose automated aneurysm pose estimation. We conquer with R1 that this has a great clinical potential. Besides, existing medical works mainly deal with single instances of large organs (e.g, knee, shoulder) [29] with specific shapes and standard positions and orientations. Aneurysms are very small with unspecific shapes. There can be many to none aneurysms at varying locations. Previous works on non-medical images have mainly focused on 2D and/or point cloud datasets, primarily for camera pose estimation (R2). Adapting existing approaches to handle the challenges of aneurysms in 3D images is non trivial. For instance, we attempted to use the PoseNet architecture to directly regress aneurysm orientation angles, instead of the adopted markup approach. However, we faced convergence issues, highlighting the need for an approach tailored to 3D medical data. Therefore, we think our proposed combined strategy to attack the problem is of interest to MICCAI community.
SOTA comparison : As far as we know, SOTA only addresses the aneurysm detection subproblem, with methods mostly based on the UNet backbone. The only applied object detection approach was nnDetection. We excluded the 2nd-ranked method in the ADAM challenge from our comparison (R3) due to the unavailability of public code or a technical description. It is based on the same UNet architecture as nnUNet (ranked 3rd). The authors’ codes for nnUNet and nnDetection (considered in our paper) are available, which ensures reproducibility and fairness for our evaluation. Comparing with other object detection approaches as suggested by R2 would indeed be valuable, but would also require some adaptation which could taint the evaluation, and to be documented in the paper. Moreover, nnDetection is based on an improved RetinaNet architecture, which has demonstrated superior performance compared to SSD and Faster RCNN (doi.org/10.48550/arXiv.1708.02002).
Our method demonstrated good balance between sensitivity and FP/case (R3). FROC curves analysis indicates a sensitivity of 82.93% with a FP/case of 0.44. In comparison, nnUNet achieves a maximum sensitivity of 81.90% with a higher FP/case of 1.04, while nnDetection achieves the same sensitivity of 82.93% but with a higher FP/case of 0.51.
On dataset [7], the training and inference times for our method are 24 hours/fold, versus 28 hours for nnDetection and 4 days for nnUNet (R3).
- Results on datasets : Hyperparameters were tuned using a part of the in-house dataset, which induces a risk of bias. Therefore, we kept the datasets separate and the final evaluation was performed on the public dataset [7] (larger, reflecting real-world scenarios) to ease comparison with upcoming works. Object detection results of our method were added to the paper for the in-house dataset (R3). These results (AP=82.48%, Sensitivity=83.01%, 0.34 FP/case) exhibit performance comparable to dataset [7].
Our code, annotation tool, and labels used for dataset [7] will be made publicly available upon acceptance (R1, R2, R3, see Section 2.5). Most other requests have now been satisfied in the revised version of the paper (typeset for equations 1-4, “±” is for standard deviation, larger figure for the architecture). Some claims were indeed unwarranted and have been mitigated (R3). The pose estimation results are now presented in a table (R2, R3).
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
I have read the comments and rebuttal. This paper is about the detection of intracranial aneurysms and the estimation of their pose in TOF-MRA images. The method is developed based on the deep learning method and inspired by YOLO. Compared with nnDetection and nnUNet, the evaluation results are good in terms of median localization error and median 3D orientation error. Most of the concerns have been addressed in the rebuttal.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper focuses on using 3D magnetic resonance angiograms for intracranial aneurysm detection and pose estimation. They also provide simplified annotation methods. The authors adapt YOLO for this task and evaluate the results on 2 clinical datasets. This is a good application paper and the rebuttal sufficiently addresses reviewer feedback. That said, I do concur that the experimental validation and SOTA comparisons can be enhanced, and authors should discuss this matter carefully in their camera ready. Recommendation is to accept.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper describes a YOLO-based method for the detection of an oriented bounding box around intracranial aneurysms in time-of-flight MR angiography. The method is evaluated in two large datasets. Reviewers raised several issues that were properly addressed in the rebuttal. I think the combination of a new but clinically relevant task and evaluation in relatively large datasets with good results makes this an interesting paper for MICCAI and suggest acceptance.