List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Fan Zhang, Tengfei Xue, Weidong Cai, Yogesh Rathi, Carl-Fredrik Westin, Lauren J. O’Donnell
Abstract
Diffusion MRI tractography is an advanced imaging technique for quantitative mapping of the brain’s structural connectivity. Whole brain tractography (WBT) data contains over hundreds of thousands of individual fiber streamlines (estimated brain connections), and this data is usually parcellated to create compact representations for data analysis applications such as disease classification. In this paper, we propose a novel parcellation-free WBT analysis framework, TractoFormer, that leverages tractography information at the level of individual fiber streamlines and provides a natural mechanism for interpretation of results using the attention mechanism of transformers. TractoFormer includes two main contributions. First, we propose a novel and simple 2D image representation of WBT, TractoEmbedding, to encode 3D fiber spatial relationships and any feature of interest that can be computed from individual fibers (such as FA or MD). Second, we design a network based on vision transformers (ViTs) that includes: 1) data augmentation to overcome model overfitting on small datasets, 2) identification of discriminative fibers for interpretation of results, and 3) ensemble learning to leverage fiber information from different brain regions. In a synthetic data experiment, TractoFormer successfully identifies discriminative fibers with simulated group differences. In a disease classification experiment comparing several methods, TractoFormer achieves the highest accuracy in classifying schizophrenia vs control. Discriminative fibers are identified in left hemispheric frontal and parietal superficial white matter regions, which have previously been shown to be affected in schizophrenia patients.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16431-6_19
SharedIt: https://rdcu.be/cVD41
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The key contribution is to construct a representation of whole-brain tractograms with consistent structure enabling applications of machine learning algorithms that take tractograms as input.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
The formulation of the representation is novel, interesting and sensible.
-
It offers genuine advantages over more ad-hoc solutions one might imagine.
-
Simulation results are promising and show the approach works as expected in one example scenario.
-
Real-data results are reasonably compelling showing good discrimination between psychiatric groups and highlighting reasonable features of the tractogram as salient features.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
Writing is a bit hard to penetrate. I had to read it a few times. Intro could be more focussed and define more clearly what the actual task is - it takes until we get to the experiments before that becomes clear.
-
Could do with some ad-hoc baselines in the experiments e.g. a simple tract-density image.
-
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Seems fine.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Not much more to add. I like the idea and I can see it being useful in some applications. As I note under weaknesses, the most obvious way to feed a whole-brain tractogram into CNNs etc would be to create a tract-density image (streamline count at each voxel); an experiment with this as a baseline would highlight the benefits of the proposed approach better.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Novel and useful method even if not the most clearly written paper.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
The paper approaches the problem of parcellation-free WBT analysis by the proposed TractoFormer. The proposed method first map the tractography information at the level of individual fiber streamlines to 2D representation then leverages ViT to performance classification.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The topic is interesting and clinically significant.
- The paper is well organized and easy to follow.
- The idea of embedding tractography information into 2D representation seems effective and interesting.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Choosing the first two eigenvectors of the affinity matrix as the coordinates of the 2D embedding grid might lead to information loss.
- The interpretability of CNN can also be achieved by Class Activation Maps (CAM). Authors does not provide experiments to show superiority of the proposed method against it.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Authors mentioned that the code will be made available upon request. The availability of the pre-trained model is unclear.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- Authors may explore the embedding in higher dimensions. The proposed workflow should be easily extended to that. It is interesting to see how emending affect the performance and will be a good ablation study.
- Comparison between CAM and ViT’s attention seems necessary to demonstrate the interpretability of TractoFormer.
- Since the proposed method is able to obtain 2d grid representation, it is interesting to see using other kinds of conventional data augmentation techniques in computer vison.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The rating reflects innovation of the approach for of parcellation-free WBT analysis. The merits outweigh the weaknesses.
- Number of papers in your stack
1
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #4
- Please describe the contribution of the paper
This paper proposed a TractoEmbedding method to encode 3D fiber spatial relationships by 2D image. This representation method can represent fibers of different regions by different channels and by performing random downsampling, multiple TractoEmbedding images can be generated. Based on the generated 2D TractoEmbedding image, a ViT-based TractoFormer was proposed to conduct HC/SCZ classification.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
The proposed TractoEmbedding method can encode the 3D fiber spatial relationships by 2D image, which allows image-based models such as CNNs and ViT to leverage fiber spatial similarity information.
-
The figures of this paper are clear, which is very helpful to understand the main idea of the proposed method.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
The introduction of TractoEmbedding method is not clear. First, spectral embedding as an important step, even citation is given, should give some basic introduction. Second, in the second step, only first two dimensions for each point was used to generate the 2D embedding grid. There is no experiments to discuss if the two dimensions can effectively represent the spatial information of the 3D fibers. Third, the paper mentioned “multiple fibers that are spatially proximate are mapped to the same voxel”, based on this kind of representation, the differences of multiple fibers which are mapped to the same pixel are ignored. It is better to conduct experiments to measure similarity and differences of the fibers at the same pixel.
-
Exp 1: Synthetic data. To evaluate if the proposed method can identify the fibers with group differences in the WBT data for interpretation, this paper generated two synthetic groups of G1 and G2, compared G1, the mean FA of each CST fiber in G2 was decreased by 20%. This kind of group difference is too simple compared the complex change caused by SCZ. The results that the proposed model can get 100% acc for G1/G2 classification but a much lower acc for HC/SCZ (table1) can also illustrate this problem. Therefore, the interpretability of the results may be not very accurate.
-
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The code and data will be released according to the reproducibility checklist.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Q5
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I have carefully read all part of the paper. Based on the strength and weakness elaborated in Q5, I gave the score.
- Number of papers in your stack
8
- What is the ranking of this paper in your review stack?
6
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
I agree with the reviewers that this is a novel and interesting work. I would like to suggest the authors comprehensively address the concerns of all the reviewers.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6
Author Feedback
We thank the AC and reviewers. After reading the comments, we think that ablation experiment results are needed to show comparison with additional baseline methods and also to show the effectiveness of using the first two embedding coordinates for creating TractoEmbedding images. We have performed the requested experiments and will add new results, as follows.
For baseline comparisons: We will add a baseline method comparison using CNNs and tract-density images (TDI) (suggested by R1). Briefly, a 3D TDI, where each voxel represents streamline count, is generated per subject and fed into a 3D Resnet for group classification (i.e. healthy vs schizophrenia). This method generates a mean accuracy of 0.764 and a mean F1 of 0.589, similar to another baseline method (FC-1DCNN) that also uses streamline count information. We will also add CNN interpretability results using Class Activation Maps (CAMs) in Resnet (suggested by R3). In Exp 1 (synthetic data), CAM also identifies the fibers with synthetic changes. In Exp 2 (disease classification), CAM identifies the fibers related to the brainstem and cerebellum, different from our ViT-based method, suggesting that the two methods focus on different brain regions and possibly explaining the accuracy difference.
For embedding coordinate evaluation: We will add a new result from using 3D TractoEmbedding FA images generated using the first 3 eigenvectors (following R3’s suggestion to explore embeddings in higher dimensions). For simplicity, we use 3D Resnet to perform group classification. Interestingly, we find that using 3D representations generates worse classification performance than using the 2D representation and a 2D Resnet. This is likely due to the sparsity of the 3D image data, where many voxels on the 3D grid do not have any mapped fibers. In addition, we will also add a new result from quantifying the similarity of fibers mapped to the same voxels (suggested by R4). Briefly, for each voxel with multiple fibers, we compute the mean pairwise fiber distance (MPFD) across the fibers. The average of MPFDs across all voxels with multiple fibers is 5.7 mm, which is a low value representing highly similar fibers through the same voxel. (For a ballpark comparison, the average MPFD within fiber clusters in an atlas is 21.8mm (Zhang et al. NIMG 2018).) We also perform a visual check, which shows fibers passing through the same voxel in general have similar shape. We will add this visualization in Fig 2(1). This new result shows that a 2D TractoEmbedding image can effectively represent the spatial information of the 3D fibers in terms of capturing relative fiber similarity (asked by R4).
We agree with R4 that the synthetic data can not reflect complex brain changes caused by schizophrenia. However, we would like to note that the synthetic data experiment is used as a proof of concept evaluation, as adopted in (Smith et al. HBM 2009; Zhang et al. NIMG 2018), rather than simulating real disease changes (to our best knowledge, it is not possible to generate synthetic data as complex as a disease). We acknowledge that classification between healthy and schizophrenia is a challenging task and an ongoing research topic. In our study, we achieve an accuracy of 0.849, which is comparable to several recent imaging-based schizophrenia-healthy classification studies, e.g. 81.02% reported in Hu et al Schiz. Res. 2021 and 83 to 87% reported in Chilla et al. Sci. Rep. 2022. While there is no ground truth to validate our interpretability results, multiple studies have reported similar affected white matter regions in schizophrenia, supporting our results.
Finally, we will address all other comments, including clarifying the classification task in the introduction (R1), describing spectral embedding basics in the methods (R4), and discussing combining our method with CV data augmentation methods in the conclusions (R3). To gain space for updates, we will shorten the method details in the introduction.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
I agree with the reviewer that this is novel and useful work. Please add the comparison and improve the inroduction in the final version as Reviewer 1 suggested.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
I agree with the consensus from other Meta reviewers about the novelty and contributions of this work.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The key strength of this paper is to propose a representation of whole-brain tractograms with consistent structure based on novel deep learning models. The authors have adequately and reasonably addressed the major concerns of all reviewers. Therefore, I would suggest acceptance of this paper and revise the final version of this paper to integrate all useful comments.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2