List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Ignacio Sarasua, Sebastian Pölsterl, Christian Wachinger
Abstract
Modeling temporal changes in subcortical structures is crucial for a better understanding of the progression of Alzheimer’s disease (AD). Given their flexibility to adapt to heterogeneous sequence lengths, mesh-based transformer architectures have been proposed in the past for predicting hippocampus deformations across time. However, one of the main limitations of transformers is the large amount of trainable parameters, which makes the application on small datasets very challenging. In addition, current methods do not include relevant non-image information that can help to identify AD-related patterns in the progression.
To this end, we introduce CASHformer, a transformer-based framework to model longitudinal shape trajectories in AD.
CASHformer incorporates the idea of pre-trained transformers as universal compute engines that generalize across a wide range of tasks by freezing most layers during fine-tuning.
This reduces the number of parameters by over 90% with respect to the original model and therefore enables the application of large models on small datasets without overfitting.In addition, CASHformer models cognitive decline to reveal AD atrophy patterns in the temporal sequence.Our results show that CASHformer reduces the reconstruction error by 73% compared to previously proposed methods.
Moreover, the accuracy of detecting patients progressing to AD increases by 3% with imputing missing longitudinal shape data.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16431-6_5
SharedIt: https://rdcu.be/cVD4M
Link to the code repository
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposed a Cognition Aware Shape Transformer for longitudinal shape analysis. The CASHformer uses a frozen pre-trained Transformer, where only LN layers are fine-tuned in small Alzheimer’s dataset, to predict the mesh deformation along time. Congnitive embeddings and congnitive decline asare loss are also introduced as regularization.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Using large pre-trained Transformers as the universal computing engine and finetune it in the small downstream tasks is an interesting research topic, which might helpful to deal with the limited data problem in the medical field.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The ablation study is not very sufficient. The first two rows of the bottom Table 1, proof that finetune LN layers with pretrain weights is better than train from scratch, but we can’t know if only finetuning LN layers is better than finetuning all layers.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors agree to release the training code.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Provide more experiments to address my questions in the wakness question.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Overall, the paper has good quality, however, the comparison and ablation is not sufficient.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Somewhat Confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
This paper proposed a method named CASHformer, a transformer-based framework for the longitudinal modeling of neurodegenerative diseases. CASHformer consists of the mesh network, frozen pre-trained transformer, cognitive embedding, and cognitive decline aware loss. The results show CASHformer reduces the reconstruction error by 73% and increases AD disease detection by 3% to the baselines.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This paper effectively incorporates the transformer model into its framework to solve the low-resource data issues with detailed experimental discussion.
- CASHformer designed cognitive decline aware loss for the longitudinal analysis.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- I notice the frozen pre-trained transformer is also similar to the ideas of Adaptor [1], Prompt [2] in the natural language process.
[1] Houlsby, Neil, et al. “Parameter-efficient transfer learning for NLP.” International Conference on Machine Learning. PMLR, 2019. [2] Schick, Timo, and Hinrich Schütze. “Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference.” Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The results are reproducible with some effort.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- Please provide more details on the dataset. How to choose the subset from ADNI, which would help researchers follow and reproduce your works.
- The author could provide more exploration in their further works on the frozen pre-trained transformer, e.g., fine-tuning partial layers instead of layer normalization (LN), prompts.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Overall the paper is well written, and the experiments are good and solid.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Somewhat Confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
Authors propose CASHformer, a transformer based approach to model hippocampus deformations across time. MRI follow-ups of hippocampus are segmented using an already existing software (FIRST) and embedded using a mesh-based neural network encoder (SpiralResNet). A pretrained transformer is trained to predict hippocampus deformations embeddings (only fine-tuning the Layer Normalization blocks). Authors propose to incorporate clinical knowledge to the model by including a cognitive score embedding (similarly to positional encoding) to modulate the hippocampus latent representations, as well as a Cognitive Decline Aware Loss based on cosine similarity to enforce larger deformations for patients with a higher cognitive decline. Authors evaluate their model on three proposed longitudinal shape modeling tasks: interpolation, extrapolation and trajectory prediction. An ablation study on the proposed contributions and a discussion on the size of the models are also presented.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Overall clarity: the context, goal, contributions are clearly described and the article seems well written to me. Clinically aware: the effort to anchor clinical relevance and medical expertise throughout the introduction, methods and evaluation is highly appreciated. Generalizable methodology and contributions: although it may seem contradictory with the previous comment, the approach presented seem to be easily transposed to the analysis of other longitudinal data (for different pathologies or clinical need) using any encoding/decoding network (relevant to the task) and the same training strategy of the pre-trained transformer backbone, aswell as some task specific loss / embeddings.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Overlap with ref. [24]: TransforMesh [24] is the main state-of-the-art comparison method, and when reading the original paper, some considerable overlaps with the proposed submission are noticeable: in the structure (section 2.4, evaluation…), notations, figures (fig. 2 in [24] vs fig S2 here), and quite some amount of text (transformer architecture description, “missing shapes” paragraph, etc.). Still the proposed method is showed to outperform [24] and some clear methodological novelties are provided with the Cognitive Embeddings, Cognitive Decline Aware Loss and Frozen pre-trained Transformer training. Authors could provide further discussion on the use of such modelization: predict diseases among neurodegenerative pathologies in clinical practice ? if yes, in what time horizons ? or maybe to better understand those complex diseases for researchers, neurologists ?
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
All parts of the model architecture blocks are well described as well as the training procedure (mostly in the supplementary materials). The dataset is exhaustively detailed as well as the splits and the proposed evaluation process. As stated in the reproducibility checklist, not all hyper-parameters tuning / setting is reported, e.g. the loss weight lambda was empirically set.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Major comments:
- an alternative evualuation idea could focus on trying to predict the ADAS score with different time horizons ? which seems to be highly valuable for clinical pratice.
Minor comments:
- p.5 line 3: “between the the line”: remove 1 the
- p.5 line 16: “The increase of the classification accuracy by 3%” : percentage point increase
- does the mean absolute error (metric used) between meshes has a unit ? (mm?)
- fig S3. authors could add other usual binary classification metrics (e.g. precision, recall, f1, roc)
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The overall clarity, medical significance and clinical anchor are appreciable. The methodological contributions and outperforming results are clear, as well as a good generalization potential to approaches for the processing of longitudinal data. Nevertheless, the considerable overlap with ref [24] is quite remarkable and remains problematic to me.
- Number of papers in your stack
6
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
7
- [Post rebuttal] Please justify your decision
As my main justification for my former rating was this overlap issue, and I understand meta-reviewer’s decision not to consider it, my paper’s rating change is quite important. My overall opinion of the work done was/is very positive, acknowledging the quality and significance of the study. Authors’ rebuttal addresses precisely reviewers concerns to me, providing additional experiments to demonstrate the relevance of the LN parameters training only, as well as further discussion on the clinical use.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The reviewers are generally positive about the work, but have some remaining questions. Strong points are the novelty of the proposed methodology, clarity of the paper, clinical awareness, and the generalizability of the method.
One reviewer raises the that there some overlap in paper structure with ref [24]. Given that the current work has clear novel contributions compared to that work, I do not consider this to be a problem. This does not require further explanation in the rebuttal.
For the rebuttal:
- Please address the comments by R1 on the ablation study and comparison
- Please address the horizon for clinical use (R3)
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
4
Author Feedback
We thank the reviewers for their constructive comments. We appreciate that the clarity and novelty of our paper was acknowledged by all reviewers and the meta-reviewer (MR), and that R2 and R3 have ranked our paper 1st (out of 5 and 6 papers, respectively). It seems that R3’s main concern was with the overlap to ref 24. However, we would like to emphasize that while the headings may be similar, the contributions differ significantly, so that the novelty of our work is not affected. This is supported by the MR, who wrote: “the current work has clear novel contributions compared to that work, I do not consider this to be a problem. This does not require further explanation in the rebuttal”. Below, we would like to address specific comments from R1 and R3 highlighted by the MR.
R1: “but we can’t know if only finetuning LN layers is better than finetuning all layers”
Following R1’s feedback, we initialized the network with the ImageNet weights and fine-tuned all the layers. We obtained a mean interpolation error of 7.35 ± 0.80. In addition, we also tried two more experiments: freezing all the layers and fine-tuning only 1) the attention layers 2) the feed forward layers. We obtained mean interpolation errors of 4.34 ± 0.72 and 7.37 ± 0.68, respectively. These results substantiate that freezing all the layers and only fine-tuning the normalization layers, as done in the paper, leads to the best result: 2.61 ± 0.15 (taken from Table 1), which is consistent with findings in [1].
Please address the horizon for clinical use (R3)
We agree with R3 that predicting “the ADAS score with different time horizons” could be highly valuable in clinical practice. As R3 mentioned, our work presents “Generalizable methodology and contributions” and “the approach presented seem to be easily transposed to the analysis of other longitudinal data” such as predicting future ADAS scores. We thank R3 for the idea and we would be very interested in exploring this evaluation scheme in future versions of our work (e.g. journal version). Regarding the clinical application of our framework, we believe that its main application is currently in research to “better understand those complex diseases for researchers” (as described by R3). However, in recent years we have observed that more computational methods have been adopted in clinical practice. Hence, we believe that this will also open new avenues for deploying our framework in the clinic, which could help with the diagnosis and treatment process in the future.
[1] Lu, Kevin, et al. “Pretrained transformers as universal computation engines.” arXiv preprint arXiv:2103.05247 (2021)
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Strong points are the novelty of the proposed methodology, clarity of the paper, clinical awareness, and the generalizability of the method.
Reviewers questions are answered well and reviewer have increased their scores or kept hem the same, making the average positive.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The rebuttal addresses satisfactorily reviewers concerns, providing additional experiments to demonstrate the relevance of the finetuning of LN layers, and further discussion on the clinical use. This is also acknowledged by one of the reviewers that changed his/her rating from reject to accept
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
NR
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
All reviewers agree there are merits to this paper. The rebuttal addressed most of the concerns. The paper offers new methods and insights for the medical imaging community. The authors are encouraged to address the comments in the final version of the paper.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
8