List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Soorena Salari, Amirhossein Rasoulian, Hassan Rivaz, Yiming Xiao
Abstract
In brain tumor resection, accurate removal of cancerous tissues while preserving eloquent regions is crucial to the safety and outcomes of the treatment. However, intra-operative tissue deformation (called brain shift) can move the surgical target and render the pre-surgical plan invalid. Intra-operative ultrasound (iUS) has been adopted to provide real-time images to track brain shift, and inter-modal (i.e., MRI-iUS) registration is often required to update the pre-surgical plan. Quality control for the registration results during surgery is important to avoid adverse outcomes, but manual verification faces great challenges due to difficult 3D visualization and the low contrast of iUS. Automatic algorithms are urgently needed to address this issue, but the problem was rarely attempted. Therefore, we propose a novel deep learning technique based on 3D focal modulation in conjunction with uncertainty estimation to accurately assess MRI-iUS registration errors for brain tumor surgery. Developed and validated with the public RESECT clinical database, the resulting algorithm can achieve an estimation error of 0.59±0.57 mm.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43996-4_66
SharedIt: https://rdcu.be/dnu1G
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The authors have touched on an important problem which is inter-modal registration error estimation in ultrasound-guided neurosurgery.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Well-described method
- Promising results.
- Method validated on public data.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
At the conference level, the article has the correct structure. In the case of method development, a more detailed description will be indicated.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Due to the complexity of the architecture, the information described may not be sufficient to easily reproduce the results.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The authors have touched on an important problem which is inter-modal registration error estimation in ultrasound-guided neurosurgery.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The authors proposed a new approach to a significant clinical problem by adapting the known architecture and extending it with elements of the Monte Carlo method.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper describes a new deep learning approach for prediction registration error between intraoperative ultrasound and MRI data, addressing the challenges associated with brain shift during brain tumor resection. The paper describes the generation of training data (due to impossibility of gaining a ground truth) in detail as well as the modifications made to a network that was previously used to predict registration error on 2D images. The results suggest that this method out-performs baseline and can accurately predict the registration error between MRI and iUS.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-The paper is very well written, easy to follow and has great visuals -The manuscript goes into extensive detail on the background and existing literature surrounding this problem -The incorporation of uncertainty estimation for registration error prediction seems to be a very good idea (particularly for clinical translation and giving surgeon’s some context on what the network is doing)
- The authors use a public dataset which is great
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Please see justification below (in suggestions for the author) for a more detailed explanation of each of these comments:
-A few mistakes / typos in the paper -Incremental approach for error classification
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
This paper is reproducible. They use a public clinical dataset and accurately describe the methodology and implementation used.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
This paper is very nicely written and easy to follow, these comments are directly related to the weaknesses pointed out above and potential areas for improvement.
-A few mistakes / typos in the paper: On page 6, several times it says FocalErroNet instead of FocalErrorNet. On page 3, it states: “To ensure that simulated registration errors are of different varieties and sizes, we randomly selected the number of control points and the associated displacements (in each 3D axis) with a maximum of 20 and 30 mm, respectively.” – should this say, with a minimum and maximum?
-Incremental classification approach (variation of network input size): I am a bit concerned that the contribution of this paper is incremental. The manuscript mentions that the novel framework is adapted from an existing architecture that was proposed for 2D MRI and iUS registration error prediction: [13] J. Yang, C. Li, X. Dai, and J. Gao, “Focal modulation networks,” 2022. Perhaps elaborating on the significance of the modifications to this network would make it more clear why this methodology is interesting / novel
- The text in figure 2 is difficult to read. Maybe reformat / increase font size
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Overall, I think this paper presents a detailed and interesting description of using deep learning to predict registration error for brain shift in neurosurgery. The idea of incorporating uncertainty estimation is very interesting. My concerns regarding the incremental nature of the error prediction are outweighed but other interesting contributions in this paper, such as the uncertainty estimation and compelling results.
- Reviewer confidence
Somewhat confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The authors presented a framework named FocalErrorNet for MR-iUS registration error estimation for brain tumor resection surgery. The study reported an average estimation error of 0.59 mm with RESECT database.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- With the increased popularity and adoption of ultrasound, understanding registration quality with MR-iUS co-registration is an important topic. An accurate, fast, and automated method for registration quality assessment can significantly improve the utilization and confidence in using iUS for navigation in brain tumor resection. This work is a positive step in that direction.
- The validation and assessment of the framework are thorough and reasonable; the use of RESECT provides a common baseline for comparisons to gauge performance.
- The use of focal modulation networks is an interesting attempt to improve upon a vanilla neural network framework.
- It appears the authors would make this code as well as data available; that initiative should be commended.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- A major concern is regarding the performance decline when testing robustness of the model: when challenged, the mean prediction error for FocalErrorNet increased from 0.59 mm to 1.28 mm, standard deviation increased from 0.57 mm to 0.99 mm, and correlation between estimated and true error decreased from 0.82 to 0.41; similar performance decline was observed with the baseline 3D CNN, in particular the correlation drops from 0.61 to 0.20. While still showing promises, it is unclear if this performance degradation is (a) acceptable for clinical use; (b) would suffer further when the challenge is more than the random linear shifts introduced in this work.
- Another concern is the size and homogeneity of the data examined: the RESECT dataset is limited to 23 cases, and the additional data are also limited to 22 cases with T2 FLAIR MRI. The overall size of the dataset in the study makes it challenging to assess the framework and results of the study especially considering the heterogeneity of the clinical situations. Furthermore, it is unclear if the performance differs among different MR modalities and scanners, e.g. if T2 outperforms T1, which could have implications and constraints on the clinical usability of the proposed approach.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper should be fairly reproducible given the authors’ indication of providing code and data in an open-source manner.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- On Page 6, it appears to be a typo of “FocalErroNet” in Sec 3.1.
- It is unclear the statistical test performed to obtain the p values.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The recommendation is based on the clinical usefulness of the direction that this work contributes to, especially in consideration of the potential of ultrasound in the future in image guided navigation.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
All reviewers recommend acceptance based on an interesting approach to a well known problem, showing promising performance. The method is further perceived as well presented and reviewers commend the use of public data.
However, reviewers also identified some weaknesses (especially R2); the chied concern is about the robustness and overall validity of the results, as performance declines dramatcially in challenging cases and evaluation is limited to public but small and homogeneous data.
These concerns should be addressed to the degree possible in an updated version.
Author Feedback
We thank the reviewers for their positive and constructive feedback. To answer the posed questions:
Novelty of FocalErrorNet: The Focal modulation network paper (Yang et al., 2022) introduced a general framework to model spatial context in an image, but only simple hierarchical architectures were designed and tested for 2D images, similar to the initial introduction of ViT. We proposed the first ResNet-like network architecture based on Focal modulation to better encode local features and spatial context and ensure better gradient flow. In addition, we have extended the application from 2D to 3D and further incorporated uncertainty estimation to offer assurance for error regression.
Performance changes in validation and model robustness: Due to the limited FOV of iUS and to ensure sufficient anatomical features for error estimation, we extracted image patches around pre-selected landmarks with well-defined annotation protocols to train and test our technique, achieving great results. In the follow-up robustness study, we obtained sample iUS patches from test subjects away from the feature-rich areas (with random linear shifts). These patches can contain large areas of zeros (image content out of the imaging FOV of iUS). Here, the main reason for the observed performance decline is due to the reduction in sufficient image features in iUS. However, despite these challenges, we saw an acceptable outcome from the model (MAE=1.28mm or ~1 voxel in clinical MRIs). This helped confirm the robustness of our model.
Limited subjects and data variability: Both public and reported private datasets for iUS-guided brain tumor resection are still rare. The RESECT dataset contains brain tumors of different shapes and sizes at various locations. These provide a good variability in the data for training and testing the proposed method. Based on the 22 cases, we have implemented 3 strategies to increase the number of samples and their diversity for training and validation. First, we used image patches extracted around pre-selected anatomical landmarks (~15 patches per subject). Then, each co-registered iUS scan was deformed randomly with different parameters 10 times. Finally, we used data augmentations during training to further improve training quality. Note that the test subjects were different from the training and validation sets, and their image patches were also deformed randomly. These measures ensured the quality of system validation and confirmed the robustness of our proposed method. In addition, the use of public data can enhance the reproducibility of our method.
Choice of MRI contrast: T2w FLAIR MRI better depicts the tumor than T1w MRI, especially for the Grade I and II tumors in the RESECT database, and is routinely used in planning brain tumor surgery. Thus, it is widely used as the primary contrast in the relevant literature, including ours.
We thank the reviewers for pointing out the typos in the submission; they will be corrected in the final version of the manuscript.