List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Kai Ma, Xuyun Wen, Qi Zhu, Daoqiang Zhang
Abstract
In brain functional networks, nodes represent brain regions while edges symbolize the functional connections that enable the transfer of information between brain regions. However, measuring the transportation cost of information transfer between brain regions is a challenge for most existing methods in brain network analysis. To address this problem, we propose a graph sliced Wasserstein distance to measure the cost of transporting information between brain regions in a brain functional network. Building upon the graph sliced Wasserstein distance, we propose a new graph kernel called sliced Wasserstein graph kernel to measure the similarity of brain functional networks. Compared to existing graph methods, including graph kernels and graph neural networks, our proposed sliced Wasserstein graph kernel is positive definite and a faster method for comparing brain functional networks. To evaluate the effectiveness of our proposed method, we conducted classification experiments on functional magnetic resonance imaging data of brain diseases. Our experimental results demonstrate that our method can significantly improve classification accuracy and computational speed compared to state-of-the-art graph methods for classifying brain diseases.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43904-9_17
SharedIt: https://rdcu.be/dnwGU
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The contribution of this paper is to present the sliced Wasserstein distance as the metric for comparing brain functional networks, and further uses that for disease classification and important region identification.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The novelty is to propose a sliced Wasserstein graph kernel to measure the transportation cost of brain functional networks and brain regions. The performance on disease-related brain classification tasks is over the state-of-the-art graph kernels and graph neural networks.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Table 2 only includes some of methods shown in Table 1. No description of computing settings.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
I think it is reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Please enrich the table 2 with the same method list as Table1 and state clearly the computation software and harware settings.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The novelty of the idea and the completeness of presentation and experimental verification.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
7
- [Post rebuttal] Please justify your decision
The authors answered the mentioned questions and the information needs to be added in the official verison.
Review #2
- Please describe the contribution of the paper
The authors propose a kernel to estimate distances between brain connectivity graphs based on the sliced Wasserstein distance.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors uses the eigendecomposition of the Laplacian matrices of the graphs to create embeddings in which it is possible to apply the sliced Wasserstein distance. I think this idea is interesting and valuable. The experiments are thorough and convincing.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
From the title, the authors wanted to remark that non-possitivenes is an issue. Still, the experiments do not assess if that is an issue for the applications at hand. This can easily be tackled by changing the title of the paper.
-
One of the issues of the paper is that the mathematical formulations are not always complete, with variables that are not well explained. This makes it difficult to follow some of the details of the method.
-
The comparison of Table 2 does not look fair. E.g., deep learning-based methods are expected to be extremely efficient. Instead, sliced Wasserstein distances are known for being slow. The only way to be faster than deep learning is that training time was considered and/or only a few slices were computed (which can compromise the properties of the proposed method).
-
In the experiments, the authors used connectivity graphs from rs-fMRI data. One of the most critical steps in creating such graphs is the selection of a threshold. Thus, the results from the last experiment can be biased. A sensitivity analysis on such threshold is necessary before drawing any conclusion on the brain function.
-
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
As I said, the equations are not well explained sometimes, which makes it difficult to follow the paper. That means that some details will be difficult to replicate. No code is provided.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
A better explanation of the method is required. Also a better differentiation with previous approaches is needed (e.g., with WWL method).
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although it has flaws, the idea is interesting and valuable.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This work proposes a new approach for investigating the transfer of information between brain regions with sliced Wasserstein distance, the proposed graph kernel is positive definite and improves classification accuracy and computational efficiency comparing with exiting methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This work proposes a novel formulation, which investigate the transfer of information between brain regions with sliced Wasserstein distance, the sliced Wasserstein graph kernel is positive definite and improves accuracy and efficiency by comparing with the SOTA graph methods for classifying brain diseases.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The presentation can be further improved. Many mathematical and algorithmic details need to be explained in more details. For example, the eigen embedding of graphs is mentioned but no explicit formula is given; the sliced Wasserstein distance defined in Eqn.(3), the parameter \theta is not explained; the construction of measures r and c is not clear. The whole algorithm 1 is too sketchy, low level details should be given more explicitly.
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The algorithm 1 is sketchy, too high level, many algorithmic details should be explained more thoroughly. It will be difficult for a graduate student to reproduce the result.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The work provides a novel method for investigating the transfer of information between brain regions with sliced Wasserstein distance. The work is based on many existing methods, which are introduced briefly. It will be more helpful to give mathematical formulae and more explanation, such as eigen embedding of a graph, the construction of the empirical distribution, the meaning the parameter in the sliced Wasserstein distance formula and so on.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The novel formulation is very promising. The main issue is the presentation, many mathematical and algorithmic details are introduced too briefly, more explanation and explicit formulae will be much helpful.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #4
- Please describe the contribution of the paper
The authors define an approach sliced Wasserstein distance (WD) which is different from WD to measure information transfer and a method sliced Wasserstein graph kernel for faster computation. They finally measured their performance with a SVM setting for brain disease classification.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Firstly and most importantly, their performance are very impressive. The computation time is dramatically increased with their matrix positive defined, and the classification accuracies between ADHD/NC and EMCI/NC are also great. Since they are comparing most trending methods, their ways are definitely SOTA. Secondly, the organization of their papers are good, espacially in their formulas and algorithms derivation, which make their work easy to understand and checked as valid. Thirdly, their definition of sliced WD is novel and intuitive, especially the contribution making their matrix positive define. PD property dramatically speed up their computation and maintained accuracy. It is quite impressive that the accuracy even increased to SOTA with dramatic decrease of computation time.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Firstly, the illustration of their kernel is not clear, especially compared to those papers they are competing to. This may cause difficulties to understand for researchers who are not familiar with this area. However, the formula organizations are clear for readers to refer to, but a clear and intuitive illustration is necessary. Secondly, despite their clear organization of algorithms, their provement of theorems are relatively short and requires more detailed explanations. This will weaken their argument and make it less persuative.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors agree most requirements in reproducibility checklist, with the help of their future training codes and evaluations codes, others may easily reproduce their results and even transfer to other dataset, such as their own applications. Further, with the clear explanation of their methods, others are also able to reproduce their results with light coding even without their codes.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
This paper is well-organized and the sliced WD and corresponding kernels are good methods with SOTA performance. However, it is recommended to generate more clear illustration of the sliced WD methods and more classification test other than SVM.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I made a strong accept recommendation because their work reaches SOTA performance and the sliced WD is a novel but useful approach.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This work proposed a new approach for investigating the transfer of information between brain regions with sliced Wasserstein distance. The proposed graph kernel was positive definite and improved classification accuracy and computational efficiency comparing with exiting methods. All reviewers acknowledged its novelty. Currnetly, the main problem is with the its presentation quality. There is vagueness on both method and experiment details. For example, which datasets were used in the experiments? How large were the datasets? How did the authors get the comparison method results, via meta analysis or implementation? Theorem and method description was not clear. Please use the rebuttal opportunity to improve the presentation quality and make it a competitive submission.
Author Feedback
We thank the reviewers for the encouraging comments like “novelty of the idea” for R1; “idea is interesting and valuable” for R2; “novel formulation is very promising” for R3; “very impressive, a novel but useful approach” for R4. For the construction feedbacks, we will discuss below.
(1) Q: “mathematical and algorithmic details are not clear and need to be explained” for R3, R2, R4, META-R1 A: In our work, we focus on a positive definite kernel called sliced Wasserstein graph (SWG) kernel. The mainly core mathematical formulations and algorithms for this kernel have been given in our paper. The details of background knowledge (e.g., sliced Wasserstein) related to SWG kernel have been known in [10, 2] (Kolouri, et al., CVPR, 2016; Carriere et al., ICML, 2017). For clearly understanding SWG kernel, we will explicitly explain its mathematical and algorithmic details (e.g., sliced Wasserstein) in the official version.
(2) Q: “eigen embedding of a graph, parameter \theta and variable r and c are not clear” for R3. A: The eigen embedding of a graph is based on Laplacian and matrix eigen-decomposition. We calculate Laplacian matrix from graph connection matrix and then calculate the eigen-decomposition of Laplacian matrix. We use this eigen-decomposition to represent the eigen embedding of a graph. Theta is a one dimensional absolutely continuous positive probability density function. r and c are probability measures. In the official version, we will explicitly explain these parameters and variables.
(3) Q: “selection of a threshold for connectivity graphs from rs-fMRI data” for R2 A: The anti-correlations are still biologically unclear (Garrison et al., Neuroimage, 2015). Therefore, in our work, only the positive connections are kept and all negative values are set as zeros. To verify the performance of our proposed SWG kernel, we have performed classification experiments in brain networks with different thresholds (e.g., T={0.3, 0.35}). The classification results demonstrate the superiority of our proposed SWG kernel. Different thresholds have few effects on the performance of our proposed method. Hence, we only present the results from brain networks with positive connections.
(4) Q: “comparison of Table 2 does not look fair” for R2 A: For fair comparisons in the classification experiments on ADHD, ASD and EMCI dataset, all listed methods in table 2 follow parameter setting protocol with same train, validation, and test sets. In our work, we concentrate on the performance of the entire computational process in classification experiments. Hence, we calculate the time of training, validation and testing for all methods.
(5) Q: “enrich the table 2 with the same method list as Table1 and state clearly the computation software and hardware settings” for R1 A: The methods listed in table 2 are selected from table 1. These selected methods achieve high classification results that outperform the other graph kernels (e.g., WL-ST, WL-SP, RW, WWL and GH kernel) in table 1. Among these kernels, WL-SP exhibits computational times of 71.16s, 5.37s, and 10.87s, respectively, which are faster than the SWG kernel. However, its classification results are inferior to those of the SWG kernel. Hence, we include only a subset of methods in table 2. Software: Spyder. Hardware: 32GB RAM, Intel i7 6-core processor.
(6) Q: “which datasets were used in the experiments? How large were the datasets?” How did the authors get the comparison method results?” for META-R1, A: In our work, we use ADHD, ASD and EMCI dataset to perform classification experiments. ADHD, ASD and EMCI dataset is respectively from ADHD-200, ABIDE and ADNI. ADHD dataset consists of 121 ADHD patients and 101 normal controls (NCs). ASD dataset includes 36 ASD patients and 38 NCs. EMCI dataset includes 56 EMCI patients and 50 NCs. The comparison method results are achieved by implementing classification experiments for competing methods and SWG kernel on ADHD, ASD and EMCI dataset.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors made complete and responsive response to the review questions. The manuscript quality has been improved. It is recommended for publication in MICCAI 2023.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper defines a positive definite Wasserstein graph kernel and applied it to brain disease diagnosis from brain connectivity graphs.
The original reviews had positive scores but comments that I do not find to align with these scores:
- Review 1 is almost empty but criticises the lack of detail on experimental setup as well as how computational time is only reported on a subset of methods. This very short review gives a “7”.
- Review 2 is concerned about the clarity of the method description, concerned about potential biases in the data, and brings up that the comparison of computational time is not fair because training time was included for the deep learning methods. This reviewer gives a “5”.
- Review 3 calls the algorithm “sketchy” and is highly concerned about the clarity of the method description, where crucial definitions are missing. This review gives a “4”
- Review 4 is impressed with performance, but concerned with clarity of the method and proofs. This reviewer gives a “7”.
All reviews were fairly short, and only R1 updated their review. There was no discussion.
Please note that the experiments are carried out on datasets of size 222, 74 and 106, respectively. A paper with concerns regarding soundness should not be accepted on the basis of their great experimental performance alone when the datasets are this small – this should be ringing all alarm bells.
So, while I generally do not like to go against the recommendation of reviewers, I do not think this paper should be accepted as is.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper designs a sliced Wasserstein distance (WD) to calculate information transfer and a method sliced Wasserstein graph kernel for faster computation. The proposed method improves classification accuracy and computational efficiency. This is a novel method. The experiments show the effectiveness for brain disease classification. Overall, this paper is good to published in MICCAI.