List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Chinmay Prabhakar, Hongwei Bran Li, Johannes C. Paetzold, Timo Loehr, Chen Niu, Mark Mühlau, Daniel Rueckert, Benedikt Wiestler, Bjoern Menze
Abstract
Multiple Sclerosis (MS) is a severe neurological disease characterized by inflammatory lesions in the central nervous system. Hence, predicting inflammatory disease activity is crucial for disease assessment and treatment. However, MS lesions can occur throughout the brain and vary in shape, size and total count among patients. The high variance in lesion load and locations makes it challenging for machine learning methods to learn a globally effective representation of whole-brain MRI scans to assess and predict disease. Technically it is non-trivial to incorporate essential biomarkers such as lesion load or spatial proximity. Our work represents the first attempt to utilize graph neural networks (GNN) to aggregate these biomarkers for a novel global representation. We propose a two-stage MS inflammatory disease activity prediction approach. First, a 3D segmentation network detects lesions, and a self-supervised algorithm extracts their image features. Second, the detected lesions are used to build a patient graph. The lesions act as nodes in the graph and are initialized with image features extracted in the first stage. Finally, the lesions are connected based on their spatial proximity and the inflammatory disease activity prediction is formulated as a graph classification task. Furthermore, we propose a self-pruning strategy to auto-select the most critical lesions for prediction. Our proposed method outperforms the existing baseline by a large margin (AUCs of 0.67 vs. 0.61 and 0.66 vs 0.60 for one-year and two-year inflammatory disease activity, respectively). Finally, our proposed method enjoys inherent explainability by assigning an importance score to each lesion for the overall prediction. Code is available at https://github.com/chinmay5/ms_ida.git
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43993-3_22
SharedIt: https://rdcu.be/dnwNn
Link to the code repository
https://github.com/chinmay5/ms_progression
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper describes an approach for predicting whether new lesions will appear given the T1-weighted and FLAIR images acquired from multiple sclerosis (MS) patients. The method builds a graph to characterize lesion automatically extracted features and spatial properties, pruned and the then classified with a final MLP based neural network.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- It is interesting that the appearance of a new lesion could be predicted by the current pattern of lesions.
- This combination of methods to generate lesion graphs is novel for this application.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- I found this paper hard to follow. Part of the reason is the poor use of terminology, specifically related to MS. For example, “MS disease progression” is defined as the appearance of a new lesion. Disease progression usually relates to clinical progression. “Progression information” is used several times and is similarly ambiguous. The term “active lesions” is also used in the paper, even though this usually refers to lesions that are enhancing on post-contrast MRI or lesions with a paramagnetic rim.
- It is not clear whether the ability to classify whether a new lesion will appear is that useful clinically. What if no new lesions appear but the current lesions grow substantially? Why is that not considered disease progression? A more meaningful approach to predicting MS progression can be seen in Storelli et al, “A Deep Learning Approach to Predicting Disease Progression in Multiple Sclerosis Using Magnetic Resonance Imaging”, Investigative Radiology 57(7):p 423-432, July 2022.
- What was the ground truth in the data set that was used? Was this from the radiology report stating that a new lesion appeared? This needs to be described explicitly. It is also possible in MS for new lesions to appear and then resolve before the next imaging interval, so the ground truth is not perfect.
- The method achieves an AUC of 0.67, which is somewhat unremarkable when combined with the fact that a high perecentage of the subjects do have a new lesion in the follow-up scan in the data set used. Additional classification metrics would have been helpful
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Methods are reasonably well-described. There is no statement of whether the code or data will be available.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- It is known that T2 lesions alone do not tell the whole story of MS. Central vein sign, enhancing lesions, and paramagnetic rim lesions are among the features of lesions that have been shown to better predict disease progression and are important to characterize.
- The data set here is focused on early MS. Lesions become confluent at later stages. It is not clear if the proposed techniques would be useful when this occurs as the confluence would affect the graph structure.
- Was the algorithm trained separately for the 1 year interval and 2 year interval data?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The method has some interesting qualities but the proposed application to MS seems misplaced.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper makes an important contribution to the field of disease progression in MS. In particular the paper uses a two stage algorithm to detect MS lesions, then extracts a patient-specific graph of the lesions connected by spatial proximity. The main contribution is prediction of progression as a graph classification task.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The main strength of this paper is formulating the prediction problem as a graph classification task and incorporation of spatial proximity into building a patient-specific graph. This idea beautifully addresses lesion locality, severity, and activity using both local lesion data and global locality. The authors correctly acknowledge the data imbalance in their training data, however such imbalances can not be avoided in the clinical context and is correctly addressed. The ablation study is also a strong point of this paper and demonstrates the authors performed careful analysis of their algorithm. Overall the paper was well presented, clear and supported the claims about the algorithm.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The description of the patient data was a bit unclear. Was the data collected at the author’s institution (“We collect the FLAIR, and T1w MR scans…”) or was it obtained from [18]. Age and sex information was missing. Data on time from MS diagnosis to scan would have also been relevant. An ethics statement was not present and is important.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
According to the reproducibility Response, the data for this paper are not publicly available and would therefore present reproducing the results.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Aside from the aforementioned details on patient details, I found the paper extremely well written and clear. All points were well supported by the results and support the conclusions.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper presented a novel approach to a difficult clinical task, namely MS disease. The claims of the paper were well supported by the methods and analysis. The results were presented clearly.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Since the review options are polarized, I would like to invite rebuttal and the authors should address those main concerns raised by reviewer 1, especially those about MS disease.
Author Feedback
We appreciate the valuable comments from both reviewers, who acknowledge the technical novelty of our approach (R1, R3) and emphasize how the graph formulation “beautifully addresses lesion locality, severity, and activity using both local lesion data and global locality” (R3). We understand and answer the concerns of R1.
First, we address concerns about method design and experiment since we aim to bring a novel, highly applicable methodology to clinical problems.
(1) Additional features (R1) R1 is correct in stating that many additional imaging (and meta-clinical) features (such as paramagnetic rim lesions) contribute to “the whole story of MS” and, as such, are relevant factors in predicting inflammatory disease activity. While we did not have access to, e.g., susceptibility-weighted imaging in our dataset, this comment highlights another key strength of our approach. A graph-based representation allows us to efficiently incorporate additional lesion- or patient-specific features with minor algorithmic modifications and, thus, enables us to learn from these features as well. We will extend our approach to include additional features in future work.
(2) Classification metrics (R1). We have computed additional metrics (precision, recall, and F1 score). We present the F1 score for brevity, demonstrating our superiority over the two baselines. The final version will include all metrics.
Methods 1-y 2-y 3DRes 0.72 0.80 CNN 0.74 0.81 Ours 0.79 0.88 Second, regarding clinical/medical perspectives, we extensively consulted our co-authors with >20 years of experience and a MAGNIMS (an internationally renowned group of MS imaging experts) member. We enhance the clarity and address the raised concerns below.
(3) Dataset (R1, R3) Data was collected at our university hospital from a prospective cohort of newly-diagnosed relapsing-remitting MS patients. The local IRB approved the study. To generate a reliable ground-truth assessment of inflammatory disease activity, three neuro-radiologists independently read longitudinal subtraction imaging, where FLAIR images from two time points were co-registered and subtracted. In this vein, not only new lesions but also significantly enlarged lesions are identified as inflammatory disease activity. We will add this to the final manuscript.
(4) Terminology (R1) We agree with R1 regarding the need for more precise language. Based on the discussion with the MS experts, we make the following revisions: (1) “MS disease progression” will be modified to “inflammatory disease activity,” and (2) “progression information” will be replaced with “information about new or significantly enlarged lesions” to provide a more accurate description. These revisions align with more commonly accepted MS terminology and better convey the clinical relevance of our work.
(5) Clinical utility (R1) The high prognostic value of inflammatory disease activity (signified by new FLAIR-hyperintense lesions) is clinically very well established ([1], [2]) and codified in the NEDA (No Evidence of Disease Activity) criteria, which themselves are strongly associated with clinical disability worsening [3]. Thus, emphasizing the clinical relevance of inflammatory disease activity, phase II/III trials consider the reduction of new FLAIR-hyperintense lesions a primary outcome measure. Hence, we respectfully disagree with the notion that predicting inflammatory disease activity lacks clinical utility and want to point out instead its clinical relevance, which is also stated in a recent MAGNIMS-CMSC-NAIMS consensus statement [4]. We will emphasize this in the final version.
We again thank the reviewers and area chairs for their positive and constructive comments. We are certain the new clarifications, results and references address the raised concerns and improve the presentation of our novel GNN-based approach to a clinically relevant problem.
References: [1] PMID: 31342055 [2] PMID: 23743084 [3] PMID: 36224046 [4] PMID: 34139157
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors addressed all the concerns from reviewers and considering the potential implication and impact for Multiple Sclerosis Lesions, I would recommend accept.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Based on reviews and authors’ feedback, this work needs further improvement. The paper does not meet MICCAI’s standard presently. Thus, I recommend rejecting this work.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The clinical relevance of this work and the response to the reviewers push me to give a chance to this technology to be shown and, hopefully adopted. Pls. include the remarks concerning the critics of the reviewer 1, as you promise you will do, during the rebuttal phase. Wo hope making the right move to allow you to have a successful communication.