List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Lintao Zhang, Jinjian Wu, Lihong Wang, Li Wang, David C. Steffens, Shijun Qiu, Guy G. Potter, Mingxia Liu
Abstract
Brain structural MRI has been widely used for assessing future progression of cognitive impairment (CI) based on learning-based methods. Previous studies generally suffer from the limited number of labeled training data, while there exists a huge amount of MRIs in large-scale public databases. Even without task-specific label information, brain anatomical structures provided by these MRIs can be used to boost learning performance intuitively. Unfortunately, existing research seldom takes advantage of such brain anatomy prior. To this end, this paper proposes a brain anatomy-guided representation (BAR) learning framework for assessing the clinical progression of cognitive impairment with T1-weighted MRIs. The BAR consists of (a) a pretext model and (b) a downstream model, with a shared brain anatomy-guided encoder for MRI feature extraction. The pretext model also contains a decoder for brain tissue segmentation, while the downstream model relies on a predictor for classification. We first train the pretext model through a brain tissue segmentation task on large-scale auxiliary MRIs, yielding a generalizable encoder. To provide accurate brain anatomy, we perform tissue segmentation for 9,544 MRIs from ADNI to generate ground truth using an established toolbox with manual verification. The downstream model with learned encoder is further fine-tuned on target MRIs for prediction tasks. We validate the proposed BAR on two CI-related studies of late-life depression analysis with 309 subjects and diabetes mellitus analysis with 82 subjects. Experimental results suggest that the BAR outperforms several state-of-the-art (SOTA) methods in MRI-based depression recognition and cognitive impairment identification, and the pretext model can be potentially used for tissue segmentation in other MRI-based studies.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43993-3_11
SharedIt: https://rdcu.be/dnwNc
Link to the code repository
https://github.com/goodaycoder/BAR
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The paper falls within the context of cognitive impairment prediction using structural MRI. To achieve this goal, they propose to pretrain an encoder network on a pretext task (segmentation or reconstruction) and use it together with a classification head to predict diagnostic categories.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- It is interesting to see that one could use a large sample of (unrelated) auxiliary data to boost the performance of a given task. While pre-training a network on another task is not novel, it is typically is done on the same dataset.
- Be able to detect diagnostic differences using high-dimensional images for the task of mild cognitive impairment, a complex class.
- Strong evaluation for different tasks and comparison against a myriad of methods and even and ablation of their own.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The need of ground-truth tissue classes (not automatic nor semi-automatic as per my understanding) for the entire auxiliary data hampers the widespread use of the method and limits the amount of auxiliary data to be used.
- For small sample size studies, a cross-validation strategy may be better suited for evaluation purposes.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The code is not made publicly available, yet. Nonetheless, they provide some implementation details on the architecture, hyperparameters, loss function, hardware and software. The data for the pretext task is publicly available, while the data for the prediction task seems private (thus not reproducible).
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The manuscript is written in a clear language, and the stated goals and reported experiments match closely to show the advantage of their method. The main concern is about evaluations, where no cross-validation is performed, better suited for sample sizes used in this work. Moreover, multiple similar methods are reported which makes the tables harder to read. Maybe keeping the best ResNest and Med3D variants would be good.
Some other minor comments:
- LLD acronym is not defined.
- The paragraph in Section 2: “ Accordingly, we […] MRI feature learning” seems repetitive with the introduction.
- Are all 6 pre-processing steps needed for all methods? E.g., clearly step 6 seems only for SVM and XGB methods. What about step 5?
- Linked to the previous, it seems that SVM and XGB use ROI-based features according tho the AAL3 atlas, but it is not clear.
- It is also not clear whether the segmentation head is also used to fine-tune the encoder on the target datasets (LLD, DM) or, instead, the encoder weights are frozen and only used a feature extrator for the prediction task.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Even though the formulation is not novel, I think that the task is difficult and they propose an easy yet effective way to handle it. The experiments and results section are comprehensive and show the benefits of their method.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
The authors carefully addressed my main concern (evaluation strategy).
They also provide a discussion about the use of manual labels as ground-truth for pretext task. They will make available the ground truth maps used in this manuscript.
Review #2
- Please describe the contribution of the paper
Proposed a pre-training task of brain tissue segmentation to learn anatomy information from MRIs.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
It is important to endow models with relevant medical knowledge such as an understanding of brain anatomy. This prevents the model from overfitting on spurious patterns in the downstream tasks.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The method requires labeled MRI data with segmentation map of different tissues (i.e., background, WM, GM, and CSF). This makes the proposed approach less scalable. The authors could include further analysis on how the quality of the segmentation maps affect the downstream performance.
There is no comparison with unsupervised pre-training techniques. MRI reconstruction is more of a classical approach. Many recent works have proposed newer methods. Please see papers below: Zhou, Z., Sodha, V., Pang, J., Gotway, M. B., & Liang, J. (2021). Models genesis. Medical image analysis, 67, 101840. Zhou, H. Y., Lu, C., Yang, S., Han, X., & Yu, Y. (2021). Preservational learning improves self-supervised medical image models by reconstructing diverse contexts. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 3499-3509).
Although the authors have compared with SOTA models like ResNet and EfficientNet, these models have not been pre-trained and are only trained from scratch on the downstream data. In contrast, their proposed BAR has been pre-trained on the ADNI data. This makes the comparison unfair and difficult to evaluate how good the pretext task of tissue segmentation really is. Med3D is the only pre-trained model out of all methods evaluated. For a fair comparison, the authors should first conduct supervised pre-training (i.e., tissue segmentation) or unsupervised pre-training (e.g., reconstruction) on the ADNI data for the various SOTA models.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Method should not be difficult to reproduce. For the data, authors are using 9,544 MRIs from ADNI. However, it is unclear if they plan to release the tissue segmentation maps.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Please refer to main weaknesses.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The methodological contribution is limited. Comparison with SOTA unsupervised pre-training task should be included. However, I appreciate that the authors have conducted a proper analysis of their method (e.g., ablation study, segmentation result analysis).
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
I do not agree with the author’s claim that “The pre-training with a pretext task is the key innovation of this work”. It is important that they include comparison with other self-supervised learning/pretext task. However, they have not included these results in the rebuttal and leave it as future work.
Another concern is that they say “Tissue segmentation quality has very little influence (no statistical significance) on the performance of our downstream model”. This makes it questionable as to what kind of knowledge their proposed pretext task is actually learning. If the model actually learns brain anatomy information, as claimed in the paper, the quality of the tissue segmentation maps should be crucial.
Review #3
- Please describe the contribution of the paper
The paper proposed a pre-training model for disease progression prediction and tissue segmentation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper proposed a pre-training model for disease progression prediction and tissue segmentation.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Whether the LLD and DM dataset are private or public dataset?
- Similar to current pre-trained model, the proposed method first pre-trains the encoder and decoder with unsupervised tasks, then trains the classifier and fine-tunes the encoding using supervised tasks. The authors should highlight the different between the proposed method and current studies.
- The experiments designed were only for the disease diagnosis, so it was not possible to demonstrate the performance of the proposed method in the disease progression prediction task.
- “This suggests that using more data for downstream model fine-tuning helps promote learning performance.” This seems to be a consensus, so it’s not clear what the point of this experiment is
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
None
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
See Q6
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The novelty of the proposed method is limited, the experiments designed do not demonstrate the contribution of the proposed method.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This work proposes a pretext-based learning framework for cognitive impairment prediction using T1w data. The authors all see the benefit of using existing large data to boost the performance of a downstream task. Major concerns from the reviewers include the need for tissue segmentation of a large number of datasets and no experimental results about how the segmentation can affect the downstream analysis. Reviewers also raised questions about the compared methods do not include a pretaining process.
Author Feedback
We appreciate ACs and Reviewers for the constructive comments. We address major comments here.
AC&R1&R2: Need for tissue segmentation
- As mentioned in “Ablation Study”, besides tissue segmentation, we can use MRI reconstruction as guidance for the pretext model in a fully unsupervised manner (called BAR-R). Results in Fig. 3(a) suggest that both BAR-R and BAR outperform BAR-B without pretext model, implying that brain anatomy prior derived from tissue segmentation and MRI reconstruction help improve prediction performance.
- For tissue segmentation, many established tools (e.g., FSL, FreeSurfer and SPM) can be used to provide good ground-truth segmentation maps. To promote reproducible research, segmentation maps used in this work will be shared with the public.
- Tissue segmentation quality has very little influence (no statistical significance) on the performance of our downstream model. The reason is that we only use the trained encoder for downstream prediction.
- Aside from our proposed two pretext tasks, one can use other tasks to model brain anatomy such as brain parcellation and brain MRI to CT translation.
- Such discussions will be included in the final version.
AC&R1&R2&R3: Concern about competing methods without pretraining process
- The pre-training with a pretext task is the key innovation of this work. Our method aims to learn a generalizable feature encoder via pretext task, so the pretext and downstream tasks can have totally different label distributions. This is different from previous studies that first pretrain a model on source data and then directly apply or finetune it to downstream task on target data, where source and target data share the same label set [1,2]. Particularly, our BAR-R does not require any label information, which significantly enhances its adaptability in practice. The related results can be found in “Ablation Study” section.
- Aside from the original two competing methods with pretraining (i.e., Med3D and BAR-R), we also compared our BAR with ResNet18-P that shares the same pretraining and finetuning strategy as BAR. The results of ResNet18-P are: 1) AUC=63.1±3.2&ACC=61.7±3.8 for CND vs. CN classification, 2) AUC=63.4±4.2&ACC=56.3±4.3 for CI vs. CND classification, and 3) AUC=56.0±2.2&ACC=58.0±4.1 for MCI vs. HC classification, which are superior to RestNet18 without pre-training but are still worse than our BAR. -We will include the results of ResNet18 with pretraining in the final version, and compare more unsupervised pre-training methods[1,2] in future. [1] DOI: 10.1016/j.media.2020.101840 [2] DOI: 10.48550/arXiv.2109.04379
R1: Evaluation strategy
- Two strategies are commonly used in machine learning and neuroimage analysis: random data partition and 5/10-fold cross-validation (CV). In this work, we use the 1st one and repeat the random partition process 5 times.
- With 5-fold CV, our BAR produced comparable results in 3 tasks: 1) CND vs. CN classification: AUC=71.8±2.5&ACC=63.7±5.5, 2) CI vs. CND classification: AUC=64.1±2.6&ACC=58.0±5.8, and 3) MCI vs. HC classification: AUC=65.9±6.1&ACC=62.5±4.7.
R1&R2&R3: Reproducibility
- Our source code and pretrained models will be shared to the public via GitHub, with the hyperlink given in the final version.
- ADNI is publicly available, but we have no right to share LLD and DM datasets currently. Readers can request to use LLD and DM by submitting Data Use Agreement to dataset providers.
R3: Disease progression prediction
- We actually performed two types of tasks in this work: 1) MCI vs. HC classification on DM, and 2) progression prediction of late-life depression (LLD) over 5 years (i.e., CND vs. CN classification and CI vs. CND classification). Note that category labels in the LLD study were determined based on subjects’ 5-year follow-up diagnostic information, while MRIs were acquired at baseline time.
- We will clarify this in the final version.
R1: Full name of LLD
- LLD: late-life depression.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have responded properly to the major concerns in their rebuttal so the primary AC suggests acceptance of the paper at MICCAI. R#2 provides further comments, which would be ideally considered for future improvement if extended work will be performed.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors addressed most of the concerns from the reviewers and I would recommend accept.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper proposed a pretext-based learning framework for cognitive impairment prediction using structural MRI data, which is a hot topic in MICCAI field. Although the authors have provided detailed rebuttal to address the reviewers and AC’s concerns. I agree with a majority of reviewers that the technical novelty is not that great. Some key comparisons are also missed in the current version of study. Therefore I recommend reject of this paper.