List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Kai Ye, Haoteng Tang, Siyuan Dai, Lei Guo, Johnny Yuehan Liu, Yalin Wang, Alex Leow, Paul M. Thompson, Heng Huang, Liang Zhan
Abstract
The modeling of the interaction between brain structure and function using deep learning techniques has yielded remarkable success in identifying potential biomarkers for different clinical phenotypes and brain diseases. However, most existing studies focus on one-way mapping, either projecting brain function to brain structure or inversely. This type of unidirectional mapping approach is limited by the fact that it treats the mapping as a one-way task and neglects the intrinsic unity between these two modalities. Moreover, when dealing with the same biological brain, mapping from structure to function and from function to structure yields dissimilar outcomes, highlighting the likelihood of bias in one-way mapping. To address this issue, we propose a novel bidirectional mapping model, named Bidirectional Mapping with Contrastive Learning (BMCL), to reduce the bias between these two unidirectional mappings via ROI-level contrastive learning. We evaluate our framework on clinical phenotype and neurodegenerative disease predictions using two publicly available datasets (HCP and OASIS).
The experimental results demonstrate the superiority of our model compared to several state-of-the-art methods.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43898-1_14
SharedIt: https://rdcu.be/dnwAL
Link to the code repository
https://github.com/FlynnYe/BMCL
Link to the dataset(s)
N/A
Reviews
Review #3
- Please describe the contribution of the paper
This paper proposed a novel bidirectional framework to yield multimodal brain MRI representations based on the brain structural and the functional counterpart’s interactions. The proposed method take use of the contrastive learning to extract the intrinsic unity of such modalities, and good results have been achieved based on two different datasets, which shows the foreground in clinical applications.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The proposed method is novel and ingenious that does not necessitate a GNN-based framework, which could allow the proposed method to directly utilize the adjacency matrix of structural network. Besides, the ROI-level contrastive learning used in this paper could help to relate the interactions between two key brain modalities. Furthermore, exciting resutls both on gender and AD clissification are achieved based on the proposed method.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
This paper has sufficient experiments and detailed explaination for formulation. The advices are around the figures in this paper. Fig. 1 could be refined further with more details of the proposed method, which could help researchers to understand the method simply.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Good.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- What is the LCE that didn’t appear before Eq. (2);
- The caption of figures should be consitent with the figures.
- It is suggested that the headings of axises should be added.
- For Fig.3, the transparency could be further adjusted to make the brain structure more clear.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The novelty of the proposed method and the sufficient experiments that provide powerful proof.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This work propose a novel bidirectional mapping model by means of contrastive learning, to reduce the bias between unidirectional mappings from either ways. This proposed algorithm is applied to the mapping between ROI-level fMRI signals and DMRI structural connective matrix, and is evaluated on clinical phenotype and neurodegenerative disease predictions using two publicly available datasets (HCP and OASIS). The experimental results are compared to several state-of-the-art methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This work proposed an interesting concept of estimating mappings between two sets of features bilaterally. This concept is supposed to reduce the bias introduced by unilaterally mappings. The bidirectional mapping is fulfilled by a contrastive learning scheme, where the latent features from the bidirectional encoder-decoder pipelines are contrasted. This idea is simple but effective, and well presented by both texts and diagrams.
This work makes a good use of interpretability of contrastive learning, where integrated gradients are utilized to generate brain saliency maps for the interpretation of the outcomes, showing promise in a variaty of applications, some of which have been shown in this work.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
In the brain ROI intrepretation section, many clinical/demographic tasks are conducted, leaving very limited room for an in-depth analysis of the results.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
This work is clear in algorithms and datasets description, experiemental result reporting and code release.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Q1: why ROI signals are used on the brain function side, but not functional connective matrix? The latter seems better at coupling with structural connective matrix. I guess using ROI based feature will be more easier to yield a brain salience map, while connective matrices mapping will highlight connections, and interpretation of brain ROIs will be more straightforward than that for connections. Will undesired uncertainty be introduced to the mapping in this work between two features from two formats?
Q2: In equation 1, why latent features from different ROIs are requested to have minimum similarity? Those from a subnetwork but distant in space could correlate with each other.
Q3: HCP dataset and OASIS dataset are used in separate tasks. Is it possible to use both in one task to investigate the reproducibility of the algorithm?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This work is in the top rank of my stack, due to its novelty of the algorithm, the abundance of comparison result, as well as the interpretability of this algorithm on brain regions。
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #1
- Please describe the contribution of the paper
The proposed bidirectional method can model the interactions between brain structure and brain function. Based on the contrastive learning, the proposed method can make the bidirectional mappings as similar as possible so that the intrinsic unity of both modalities can be captured. The effectiveness of the proposed framework was demonstrated on two different datasets. The experimental results verified the superiority of the proposed method in disease classification and clinical phenotypes prediction. Moreover, by using integrated gradients to generate the brain saliency maps, the top key brain regions are identified, which are treated as the potential biomarkers for the disease and clinical phenotypes.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
By using two auto-encoders, the bidirectional mappings between bold signal of fMRI and diffusion MRI-derived brain structural network were explored in this work. Unlike other existing one-way mapping methods, the proposed method leverage the contrastive learning strategy to minimize the distinction between the two latent vectors obtained from two types of MRI data, so that the bias between these two unidirectional mappings can be reduced. Experimental results on two public datasets proved the effectiveness of the proposed method in multiple tasks, and the identified key brain regions are consistent with the results of previous studies, which also demonstrated the usefulness of the proposed method. In my opinions, the contrastive learning is an emerging strategy in deep learning, which is increasingly used in multiple research fields. In this work, the authors used the contrastive learning to explore the intrinsic unity of fMRI and diffusion MRI, further improving the following classification and prediction tasks. The idea is straightforward and the results proved the effectiveness of this model.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The method of this work is straightforward and effective. However, the contrastive learning is used to ensure the latent vectors within bidirectional mappings to be similar. During the learning process, it seems that the specific information of each modality would be ignored, which is also useful for the clinical phenotype and neurodegenerative disease predictions. And the description of Figure 2 is not correct.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The experimental results are based on two public-released datasets. The main settings about the experiments and model parameters are given. However, the specific selected participants of the data, data preprocessing steps, and the cross-validation results are not clear enough. Based on the consideration about these contents, the idea was reasonable, while it is doubtful to be reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
First, the input feature of the structural network reflects the interaction between two brain regions, whereas the feature of the bold signal reflects the temporal information in a single brain region. Is it reasonable to learn the mapping between these two kinds of features? Second, the proposed method aims to enforce the multimodal features of same brain region to be similar. However, not all the brain regions are important in both of these modalities. So is it possible to introduce any tricks to make a pre-selection before the contrastive learning. Third, the competing methods used the functional brain networks as the input, which is not consistent with the proposed method. Does this setting affect the fairness of the comparison?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The idea of this work is interesting and the results seems like effective, while it is doubtful to be reproducible. It will be better to release the code with the experimental data for testing.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
All the reviewers agree that this work proposed an novel and effective approach to estimate mappings between two sets of features bilaterally. This contrastive learning scheme is well designed to reduce the bias introduced by unilaterally mappings.
Author Feedback
Dear Reviewers,
We appreciate your insightful reviews. Your constructive feedback has played a significant role in enhancing the quality of our research work. The following is our response to your comments:
Reviewer #1: 1.The contrastive learning method indeed emphasizes the alignment of features from two modalities but may overlook specific characteristics of each modality. We plan to extend our method by considering a balance between alignment and modality-specific information preservation in our future work. Thanks for the suggestion.
2.We apologize for the confusion in Fig 2 and will revise the description in the camera-ready version. Also, we will release the source code after the official acceptance.
3.Regarding the concern about the mappings between modalities, we believe our approach is reasonable as we want to capture the brain dynamics through the BOLD time sequence. Using functional connective matrix may potentially disrupt such dynamic information due to the calculation of the correlations. Also, our experiments indicate that our encoder can model the temporal relations between different brain regions directly from BOLD signals (therefore no need to construct functional networks). Nevertheless, we agree that this should be more explicitly justified, and we will include related discussions in the camera-ready version.
4.We agree that pre-selected important brain regions could potentially improve model’s performance. We will explore this in future work. Thanks for the suggestion.
5.Concerning the fairness of the comparison, to the best of our knowledge, we didn’t find well-recognized methods using both structural and functional information as input. Our work shows that using both as input could significantly improve the prediction performance.
Reviewer #2: 1.Our experiments include different tasks from different datasets, which is to test the robustness of our new framework. We believe this is necessary for a new proposed algorithm. 2.We did not use functional connective matrix because ROI time sequence can better capture the brain activity dynamics. Our experiments indicate that our new encoder is capable of modeling the temporal relations between different brain regions directly from BOLD signals (therefore no need to construct functional networks). Nevertheless, we agree that this point should be more explicitly discussed and justified in the paper. We will include related discussion in the camera-ready version. 3.Our assumption is that different ROIs have minimum similarity in the latest space features and our algorithm assumes each ROI is distinct from other ROIs. Our current experiments support this assumption. However, we agree that subnetwork may be another important factor to affect the relationship among latent space features and we will consider this in our future work. Thanks for the suggestion. 4.In our experiments, we deliberately chose different atlas to reconstruct the bran networks for different datasets. This is to test our algorithm’s performance on different atlas (in other words, different network resolution) and on different datasets. Moreover, HCP and OASIS have different behavior/clinical measures, so it’s less likely to test the reproducibility of the algorithm using these two datasets.
Reviewer #3: 1.We are grateful for your positive feedback. Your advice on figure improvements is appreciated and we will carefully modify Figs 1 and 3.
2.Regarding the term “LCE” not being defined before Eq. (2), we apologize for the oversight and will provide a clear definition in the camera-ready version. L_CE means cross-entropy loss which is a common loss function for classification tasks.
We hope that our responses address the reviewers’ concerns and suggestions. We will incorporate these changes into the camera-ready version, which will significantly improve the quality and clarity of our work. We extend our gratitude once more to reviewers for the constructive feedback.
Best regards, authors