Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews Back to top
List of Papers By topics Author List
Paper Info Reviews Meta-review Author Feedback Post-Rebuttal Meta-reviews

Authors

Kai Ma, Xuyun Wen, Qi Zhu, Daoqiang Zhang

Abstract

Finding a faithful connection pattern of brain network is a challenging task for most of the existing methods in brain network analysis. To process this problem, we propose a novel method called ordinal pattern tree (OPT) for representing the connection pattern of network by using the ordinal pattern relationships of edge weights in brain network. On OPT, nodes are connected by ordinal edges which make nodes have hierarchical structures. The changes of edge weights in brain network will affect ordinal edges and result in the differences of OPTs. We further leverage optimal transport distances to measure the transport costs between the nodes on the paired of OPTs. Based on these optimal transport distances, we develop a new graph kernel called optimal transport based ordinal pattern tree kernel to measure the similarity between the paired brain networks. To evaluate the effectiveness of the proposed method, we perform classification and regression experiments in functional magnetic resonance imaging data of brain diseases. The experimental results demonstrate that our proposed method can achieve significant improvement compared with the state-of-the-art graph kernel methods on classification and regression tasks.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16437-8_18

SharedIt: https://rdcu.be/cVRs4

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

  • Please describe the contribution of the paper

    The authors propose an approach to tackle the problem of brain network connection pattern estimation. they use a new graph kernel called optimal transport based ordinal pattern tree kernel. Experimental evaluation shows significant improvement.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The paper details a method that on the base on my (limited) knowledge of the state of the art is novel.
    • The approach is explained in detail and the paper could be reproduced by other research groups engaging in similar topics.
    • Table 1 shows a comparison with other state of the art approaches. The proposed method yields better results.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    The notation could probably be simplified to make the topic more approachable.

  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    The paper is well explained and algorithms are detailed in a way that favours reproducibility

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

    Due to my limited expertise in this field I cannot comment further.

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    6

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    the paper yields very good results, it seems technically sound and it improves the state of the art.

  • Number of papers in your stack

    4

  • What is the ranking of this paper in your review stack?

    1

  • Reviewer confidence

    Not Confident

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Not Answered

  • [Post rebuttal] Please justify your decision

    Not Answered



Review #3

  • Please describe the contribution of the paper

    The paper presented a novel method called Optimal Transport (OT for short) based Ordinal pattern tree (OPT for short), for achieving better performance on classification and regression tasks in brain disease classification. This method is achieved by extracting OPT from brain network generated by fMRI, and measuring optimal transport distance in different OPT levels. Experiments from 4 datasets are reported and compared with other graph kernel classification methods.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. The adoption of OPT for brain network analysis is novel.
    2. The OT kernel is also novel for the OPT comparison.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    1. It is unclear why a tree structure is a good way for brain network analysis. It may reflect the hierarchical structure but it also loses important brain connections and its connection strength information. In other words, the proposed algorithm just analyzes partial information. Brain network is different from general graph analysis since brain network node structures are limited and fixed. The resulting tree structures are not necessarily consistent from person to person.

    2. It may need some further study to analyze whether the OPT or the OT makes significant contributions to the improved performance. For example, the OT can also be applied to other structures like Tree++. Such experiments will help justify their discoveries.

    3. The description of OT algorithm is not clear. It hurts its reproducibility.

  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    The reproducibility is okay. However, a more detailed description of OT will help further improve it.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
    1. The authors may add a more detailed description of how OT is implemented to compute the OT distance between sub-trees.
    2. The authors may add more controlled experiments on a different combination of methods.
    3. The authors may further justify the relevance between the OPT and brain network analysis.
  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    6

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper organization is clean.

    The work has certain novelties.

  • Number of papers in your stack

    5

  • What is the ranking of this paper in your review stack?

    3

  • Reviewer confidence

    Very confident

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    6

  • [Post rebuttal] Please justify your decision

    If it is accepted, the reviewer hopes the authors will be able to elaborate more on the motivation and computational process for the OPT part in their camera-ready version.



Review #2

  • Please describe the contribution of the paper

    In this submission, the authors introduce a new data structure to transform a graph to a tree structure. An advantage of doing so is that the proposed tree structure contains hierarchical relationships. An optimal transport distance between two trees is computed and used to build a kernel. Based on this kernel, a series of learning tasks can be applied.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    This paper introduces a novel idea of utilizing the brain network to analyze brain disease patterns. The proposed ordinal pattern tree is a good idea of establishing hierarchical relationship of nodes in a brain network. The whole paper is written in a smooth way and it is easy to follow.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    (1) Motivations of using OT distance are expected to be discussed. Tree edit distance is commonly used to measure the similarity of two tree-structured data and it is easier to compute. Some comparisons are preferred. (2) Ablation studies are desired, such as directly applying OT to the brain network and build the kernel, different OPT starting from different regions, etc (3) More statistical metrics are expected. Considering the possibility of imbalanced data, some other metrics are expected in classification experiments

  • Please rate the clarity and organization of this paper

    Excellent

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    No code is provided.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

    Besides the question in Section 5, here are some other questions: (1) How to guarantee that two graphs have the same number of levels in the form of your tree? (2) If the same region has multiple nodes, how to choose which one is the root in the tree structure? (3) Evaluation of classification results is expected to have precision, recall and F1 score.

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    6

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    My current judgement is based on the methodology in this submission. I look forward to seeing the rebuttal from the authors.

  • Number of papers in your stack

    5

  • What is the ranking of this paper in your review stack?

    1

  • Reviewer confidence

    Very confident

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    5

  • [Post rebuttal] Please justify your decision

    I change my final score of this submission to “weak accept”. Part of my concerns are not addressed in the rebuttal. Missing code for reproduction is also a reason for me to decrease the score.



Review #4

  • Please describe the contribution of the paper
    1. This paper proposed a graph kernel algorithm called optimal transport based ordinal pattern tree (OT-OPT) kernel to represent the graph structure.
    2. This paper combine optimal transport distances with OPT to compute the distance between graphs, which can be used to distinguish the discriminative tree structures from patients with brain disease.
  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The OT-OPT kernel method proposed in this paper considered the hierarchical structure information of nodes in brain network. It divides the graph structure into many tree structures. Each tree structure represents the ordinal information of one node.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    1. The serial numbers of the vertexes are regarded as classification features. While brain regions are not continuous variables, the distance between two brain regions is irrelevant with the serial number distance. Therefore, it is lack of interpretability to treat serial numbers as features.
    2. The comparison method are classical graph kernel methods, it’s better to compare with the state-of-art methods.
    3. Only part of the subjects in three datasets are selected for experiments. The selection criteria of the subjects are unclear.
  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    The description of the model is clear. However, the code of proposed method is not available.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
    1. The classification of brain networks is essentially a graph classification problem.The proposed method combine all nodes representation features as the graph features. It may lead to a plethora of redundant features. Therefore, a whole graph representation method is more suitable.
    2. It is suggested to explain the selection criteria of the subjects, and compare the proposed method with the state-of-art methods.
  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    4

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The interpretability of the model needs to be improved, and the experiments are not sufficient to prove the effectiveness of the proposed method.

  • Number of papers in your stack

    7

  • What is the ranking of this paper in your review stack?

    4

  • Reviewer confidence

    Very confident

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Not Answered

  • [Post rebuttal] Please justify your decision

    Not Answered



Primary Meta-Review

  • Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    This paper introduces ordinal pattern tree kernels for the analysis of functional brain connectivity. While the reviewers appreciate the use of the ordinal pattern trees, there are also a number of concerns, and the authors should take these into account in their rebuttal:

    • Reviewer 4 has a number of concerns regarding the experimental validation, especially the lack of comparison with state-of-the-art methods such as graph neural networks
    • Reviewer 3 expresses concern regarding the suitability of tree representations in brain connectivity
    • Reviewer 2 questions the suitability of optimal transport distances over other common tree distances.
    • Moreover, all reviewers express a number of general concerns regarding the soundness of the experimental validation.
  • What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    9



Author Feedback

We thank the reviewers for the encouraging comments like “a novel idea, easy to follow, better results” for R1, R2, R3, “consider hierarchical information of nodes” for R4. For the construction feedbacks, we will discuss below.

(1) Q: “lack of interpretability to treat serial numbers as features” for R4 A: Serial nodes reflect the connection pattern of brain regions and are used to detect neural interaction mechanism of brain regions (Wang et al., Brain, 2010; Zou et al., Biomed Signal Proces, 2020). Serial numbers are a measurement of the connection patterns between serial nodes and have been used as the biomarker of brain disease diagnosis (Curado et al., Entropy, 2020). In most of the existing methods related to serial numbers, edge weight information in brain network is usually ignored. To address this problem, we propose an OPT for brain topological analysis by using edge weight information. The nodes in OPT are serial nodes. We calculate the similarity measurement of OPT by using OT on the features of serial nodes and use it for brain disease classification.

(2) Q: “comparison methods are classical graph kernel methods, not compare with the state-of-art methods such as graph neural network （GNN）” for R4, META-R1 A: The main purpose of this study is to develop a new network topology (i.e. OPT), and integrate it with OT to quantify the similarity of pair-wise brain networks (i.e., graph kernel). Therefore, all selected algorithms for comparison in Table 1 are state-of-art methods related to graph kernels, such as Tree++ (Ye et al., TKDE, 2021), WWL (Togninalli et al., NeurIPS, 2019), etc. Another reason for not choosing GNN as the comparison algorithm is due to the limited sample size in this work. As deep learning method, the small number of subjects cannot guarantee the training requirements, thus leading to a low classification accuracy. Our validation experiments confirmed the above speculation. By adopting a classical GNN method (DIFFPOOL, Ying et al., NeurIPS, 2018), we reconducted three binary classification tasks (i.e., ADHD & NC, ASD & NC, MCI & NC) and obtained the accuracy of 67.62%, 70.36% and 71.21%, respectively, which are lower than our methods (76.13%, 78.38% and 89.36%).

(3) Q: “all nodes features as graph features may lead to a plethora of redundant features” for R4 A: Our method adopts the ordinal patterns of edge weights to construct OPT for brain network where the nodes connected by ordinal pattern edges are used. OPT makes a dense brain network become a sparse tree structure which is regarded as a whole graph representation. On tree structure (i.e., OPT), we use OT distances measure the similarity of node features and calculate graph kernel for brain disease classification.

(4) Q: “selection criteria for subjects” for R4 A: The selection criteria of subjects are screened by physicians according to clinical scales, such as MMSE for MCI, WASI for ASD, and DSM-IV for ADHD. Due to the page limitation, detailed descriptions are not given in manuscript. For details, please refer to (Chris et al., NeuroImage, 2011; Xie et al., Biol Psychiat, 2021; Rocco et al., BMC psychiatry, 2021).

(5) Q: “suitability of tree representations, it loses brain connection strength information” for R3, META-R1 A: In our method, OPT is constructed by using edge weights reflecting the connection strength between brain regions. It can show the hierarchical structure of brain network.

(6) Q: “suitability of OT over other tree distance, ablation studies about OT, OPT” for R2, R3, META-R1 A: OT distance can capture geometric information of the underlying data space and is suitable for graph data processing (Togninalli et al., NeurIPS, 2019). We also measure graph similarity using other tree distances and the results from them are not good, which are usually less than 70%. The results based on the methods without OT and OPT are less than 68%. Hence, OT, OPT are helpful for brain network classification.



Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    Most of the reviewer scores say “accept”, but the flaws listed in their reviews are actually rather serious:

    • Reviewer 4 has a number of concerns regarding the experimental validation, especially the lack of comparison with state-of-the-art methods such as graph neural networks
    • Reviewer 3 expresses concern regarding the suitability of tree representations in brain connectivity
    • Reviewer 2 questions the suitability of optimal transport distances over other common tree distances.
    • Moreover, all reviewers express a number of general concerns regarding the soundness of the experimental validation.

    I will accept the majority accept vote of the reviewers and recommend acceptance, but I also strongly encourage the authors to take the reviewer feedback into account in their final version.

  • After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

    Accept

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    10



Meta-review #2

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    By considering all reviewers’ comments and the positive response, I accept this paper,

  • After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

    Accept

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    5



Meta-review #3

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    Adapting ordinal pattern tree kernels as a topology metric for brain networks was received as a novel idea by reviewers. Although its motivation for using a tree structure was not clear in the paper (hope it is further clarified in the final version), the rebuttal addressed major critiques from reviewers, including experimental details and comparisons with SOTA methods.

  • After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

    Accept

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    4



back to top