List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Boran Hao, Guoyao Shen, Ruidi Chen, Chad W. Farris, Stephan W. Anderson, Xin Zhang, Ioannis Ch. Paschalidis
Abstract
Magnetic Resonance Imaging (MRI) acceleration techniques using k-space sub-sampling (KS) can greatly improve the efficiency of MRI-based stroke diagnosis. Although Deep Neural Networks (DNN) have shown great potential on stroke lesion recognition tasks when the MR images are reconstructed from the full k-space, they are vulnerable to the lower quality MR images generated by KS. In this paper, we propose a Distributionally Robust Learning (DRL) approach to improve the performance of stroke recognition DNN models when the MR images are reconstructed from the sub-sampled k-space. For Convolutional Neural Network (CNN) and Vision Transformer (ViT)-based models, our methods improve the stroke classification AUROC and AUPRC by up to 11.91% and 9.32% on the KS-perturbed brain MR images, respectively, compared against Empirical Risk Minimization (ERM) and other baseline defensive methods. We further show that DRL models can successfully recognize the stroke cases from highly perturbed MR images where clinicians may fail, which provides a solution for improved diagnosis in an accelerated MRI setting.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43904-9_74
SharedIt: https://rdcu.be/dnwIm
Link to the code repository
https://github.com/noc-lab/drl_mri
Link to the dataset(s)
The IXI dataset we used for MAE pre-training: https://brain-development.org/ixi-dataset/
The main stroke data that support the findings of this study are available from our institute but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of our institute.
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes Distributionally Robust Learning (DRL) approach to improve the performance of stroke recognition DNN models when the MR images are reconstructed from the sub-sampled k-space. The approach improves stroke classification AUROC and AUPRC by about 10% on KS-perturbed brain MR images given CNN and ViT classification models. The proposed DRL method is based on minimizing the worst-case expected loss, and results in lower degradation of performance in the case of white gaussian noise and cartesian under-sampling than other metrics.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper tackles the problem of robust classification in presense of sparse k-space MRI, where information is missing. The authors method shows stroke classification performance from MRI that is more robust than others given k-space subsampling degradation.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The method makes networks more robust to perturbation, but does not improve their performance.
It’s interesting how the DRL method allows classification accuracy to degrade less rapidly than other methods. However, all improvements shown here are due to simulated noise. It’s hard to understand the immediate practical usefulness of k-space subsampling degradation, presumably in a practical setting for best classification the full k-space would be acquired.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Equation 5: The method is implemented as a in equation (5) as minimizing the difference between ‘hidden states’ phi^t(xi) of sequences associated with MRI slices xi. The idea of hidden states is introduced rather rapidly, what specifically are these hidden states?
W weight matrix W is estimated minimizing the difference between hidden states in sequences of original and artificially perturbed MRI images.
It’s interesting how the DRL method allows classification accuracy to degrade less rapidly than other methods. However, all improvements shown here are due to simulated noise. It’s hard to understand the immediate practical usefulness of k-space subsampling degradation, presumably in a practical setting for best classification the full k-space would be acquired.
Fig. 3: AUROC curves show the DRL method improves upon BAT, ERC, PGD baselines in the case of degradation. With no degradation, the curves seem to indicate the DRL method actually performs a little worse than baseline methods - is this the case?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The authors achieve robustness in their scenario, however the task is simulated noise and seems artificial.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #5
- Please describe the contribution of the paper
In this work, the authors proposed a Distributionally Robust Learning (DRL) approach to enhance the performance of deep neural network (DNN) models for stroke recognition, particularly when the MR images are reconstructed from sub-sampled k-space. The results indicate the efficacy of the proposed method.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
(1) The proposed DRL method is easily adopted by the deep learning model.
(2) The results demonstrated the efficacy of the proposed method.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
(1) It appears that the author performed the data split at the slice level, but it is suggested that the split should be done at the patient level for more accurate evaluation.
(2) It would be beneficial for the author to compare the proposed method with state-of-the-art methods to demonstrate its effectiveness.
(3) More details should be added to Figure 3, such as labeling the x-axis with f. Without the accompanying paragraph, the meaning of the graph may not be clear.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The code should be released.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Please address the problem in the main weaknesses section.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Based on the main strengths and main weaknesses of the paper.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #4
- Please describe the contribution of the paper
This paper presents Distributionally Robust Learning (DRL) approach to improve performance of the stroke recognition DNN models when the MRI images are reconstructed from the full k-space.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
• The approach used in this work shows that its better than the Empirical Risk Minimization (ERM) and other state-of-art-techniques. • The paper claims that DRL models can successfully recognize stroke cases where every clinician may fail.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
• The repetitive studies have been performed but the inference is missing
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The author/authors have filled out the reproducibility section thoroughly and have taken steps to ensure their research is transparent and replicable. The dataset, dataset split, and hyper-parameters used to train the model have been clearly discussed in the paper. It would be good to also have the 226 patients’ demographic data in the experiments section.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
• Since the dataset is imbalanced and stroke cases constitute for just 1/4th of the entire dataset, did the author/authors use any type of sampling? Is there a possibility to use stratified sampling and compare the AUCs? • Details about approximate overall slices per/patient would be good to have in the paper. What is the isotropic spacing between the slices? What is the input image dimensions? • It would be good to see DWI and FLAIR sequences also in Fig 2. • What is the complexity of the proposed approach? Can the authors justify why a small learning rate, and no weight decay was used?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Compared to robust baseline methods, this approach yields better results.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The study focuses on using Distributed Robust Learning to detect stroke in patients with abbreviated MRI sequences (needed for clinical care, as the long MRI acquisitions are not recommended in such scenarios).
The strengths of this study include:
- nicely written and clear study.
- great experimental design and results.
- consider multiple approaches (Vit and CNN) at the same time during evaluation.
Weaknesses of the study include:
- Unclear performance of the approach on normal cases, as it does not appear that normal cases were included. While many slices don’t have stroke, an increase cohorts should also include subjects without stroke.
- reduced cohort size of only 226 patients.
- Since the data was split based on slices, is unclear whether slices for some patients were devived between training/validation/testing. Please clarify whether the splits take in account that all slices of one patient should only be in one of the sets and not spread across sets.
- in a brain MRi, a flip augmentation make sense due to the symmetry of the brain, but does a rotation makes sense ?
- how do the slice based prediction look like when comparing all slices in the study
Despite some of the shortcomings, this is a strong study that should be considered for this year’s meeting.
Author Feedback
We appreciate the careful reviews and valuable feedback from all the reviewers. Due to the word limit, we summarize the main comments as follows, and we hope our response can address the reviewers’ concerns.
- Method. (1) The idea of hidden states is introduced rather rapidly, what specifically are these hidden states? (2) With no degradation, DRL performs a little worse than baseline methods. (3) It’s hard to understand the immediate practical usefulness of k-space subsampling degradation, presumably in a practical setting for best classification the full k-space would be acquired.
Response: (1) In our ViT model, hidden states are the representation vectors for the sequence of MR image patches, output by and fed into different layers in the transformer encoder. We will introduce this more clearly in the paper. (2) When a larger regularization coefficient is used, DRL can be more robust to the perturbations in MR images, though its performance on clean images can drop as a trade-off. Therefore, we tuned the regularization coefficient using a validation set to achieve a balance, improving the DRL model performance on the perturbed images as much as possible, while controlling the AUROC sacrifice on clean images at around 1%. In practice, people can flexibly decide the coefficients to fit their setting. (3) As we claimed in the Introduction section, KS is a simple way for MRI acceleration, which is meaningful for acute stroke diagnosis. DRL significantly improves the model performance on the resulting degraded MR images, which has practical usefulness.
- Data. (1) An increased cohort size should be used, and while many slices don’t have stroke, subjects without stroke should be added. (2) It is suggested that the split should be done at the patient level for more accurate evaluation. (3) It would be good to include the patients’ demographic data.
Response: Thank you for these suggestions. Unfortunately due to limited available patient cases and the high-cost of labeling images, we only obtained access to 226 stroke patients and used their normal MR slices as the normal samples. A patient-level split can reduce the correlation between the training and test slices but will lead to less stable results due to our limited cohort. We have taken measures in the MR imaging (e.g., using a 2-d acquisition) and preprocessing (e.g., slice-level normalization) to avoid the dependency among the slices from the same subject to a great extent. As our main goal is to justify the enhanced robustness of DRL on slice-level classification models over other methods, we believe the current data of 6,533 slices (1,650 stroke) and a slice-level split can reasonably support our findings. In future work, we will increase the cohort size and consider a patient-level split as well. Due to the double-blind rules, we provided less information about our data source and study cohorts in the submission, but we will add them to the camera-ready version in accordance with the IRB protocol, and the source code will be made publicly available as well.
- Experimental settings. (1) Can the authors justify why a small learning rate, and no weight decay was used? (2) It would be beneficial for the author to compare DRL with state-of-the-art methods to demonstrate its effectiveness.
Response: (1) As claimed in Sec 2.2, we selected the best hyperparameters based on the validation set performance. Since we used DRL to train multiple layers in ViT in a randomized manner, a relatively small learning rate can avoid one layer from changing significantly in each epoch, which gives us more stable convergence. (2) Thank you for this suggestion. As stated in Sec 2.2, the PGD baseline we chose is indeed one of the state-of-the-art adversarial training methods which resists a wide range of perturbation types. Still, in our future work we will try to combine DRL with other types of methods that improve the accelerated MRI (e.g., enhanced reconstruction approaches).