List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Wei Dai, Stephanie Noble, Dustin Scheinost
Abstract
Functional connectomics has become a popular topic over the last two decades. Researchers often conduct inference at the level of groups of edges, or components", with various versions of the Network-Based Statistic (NBS) to tackle the problem of multiple comparisons and to improve statistical power. Existing NBS methods pool information at one of two scales: within the local neighborhood as estimated from the data or within predefined large-scale brain networks. As such, these methods do not yet account for both local and network-level interactions that may have clinical significance.
In this paper, we introduce theSemi-constrained Network-Based Statistic” or scNBS, a novel method that uses a data-driven selection procedure to pool individual edges bounded by predefined large-scale networks. We also provide a comprehensive statistical pipeline for inference at a large-scale network-level.
Through benchmarking studies using both synthetic and empirical data, we demonstrate the increased power and validity of scNBS as compared to traditional approaches. We also demonstrate that scNBS results are consistent for repeated measurements, meaning it is robust. Finally, we highlight the importance of methods designed to achieve a balance between focal and broad-scale levels of inference, thus enabling researchers to more accurately capture the spatial extent of effects that emerge across the functional connectome.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16431-6_38
SharedIt: https://rdcu.be/cVD6U
Link to the code repository
https://github.com/daiw3/scNBS.git.
Link to the dataset(s)
https://www.humanconnectome.org
Reviews
Review #1
- Please describe the contribution of the paper
The manuscript presents an extension to the Network Based Statistic (NBS) and constrained NBS (cNBS), by using a data driven policy to refine the large-scale constraints used in cNBS. The method, called semi-constrained NBS (scNBS), is based on four steps: network partition, marginal ranking, cut-off selection, and network-level inference. On semi-synthetic data, the Authors compare scNBS with multiple alternative solutions, like cNBS and NBS, claiming increased specificity, power and consistency.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The strong aspects of this work are: the detailed description of the scNBS, corroborated by excellent figures; the very interesting experimental design and data, which mixes Human Connectome Project resting-state fMRI data with somewhat-realistic synthetic phenotypic data.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
It is not entirely clear to me if there is some potential issue of circularity (a.k.a. double-dipping, bias) in the proposed methodology. For example, in Section 2.c, where the cut-off selection is described, the Authors explain that “the final threshold is the one that gives the largest effect among all possible cut-offs”. Typically, steps like this one, requires a careful nested cross-validation implementation to avoid that selection of parameters is operated in a biased way. Unfortunately, the manuscript is not clear on this (and other) implementation details but they claim to disclose the code of their experiments.
A second source of concern is the experiment’s section, which is entirely based on “synthetic” data (I’d say semi-synthetic, since it heavily relies on actual fMRI data). The noise models are quite elementary and may not properly represent physiological situations.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
As said above, the method is described in detail but it may be prone to circularity issues in the implementation. In the reproducibility statement, the Authors claim to disclose the code of all experiments, which should enable meta-reviewers and - if accepted - future readers to verify important technical aspects.
I invite the Authors to clearly mention in the manuscript the complete availability of the source code.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
My main constructive comment is about the experiment’s section. The Authors should at least provide more ground to the choice of presenting only “synthetic” experiments and should at least provide perspectives for future experiments.
My second main constructive comment is about discussing the limitations of the proposed method. The Authors should spend more effort on this side and characterize within which limits the method is expected to perform and underperform.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The manuscript present an interesting extension of NBS and cNBS that overcome some limitations of those methods. The topic is surely of interest to this community. The proposed method is pretty interesting and the experiments provide support to the claims of increased specificity, power and consistency. The experiments are only on (semi) synthetic data and some steps of the description raise some doubts on circularity issues.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
This paper proposed a connectome-based inference that integrates both local and global information, called semi-constrained network-based statistic (scNBS). In experiments, the proposed method was applied to synthetic and true brain imaging data. The results showed that the proposed method increased statistical power and validity in synthetic data, and showed consistency for repeated measurements in resting state brain imaging data.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The proposed method could account for both local and network-level interactions that might have clinical significance.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The algorithm performed after subnetworks were predefined. If the subnetworks were not known, how could the proposed method work?
- The cut-off values, t_{g}^{+} and t_{g}^{-} were obtained by the correlation with y, and the final inference was also estimated by the relationship between V_{g}^{+}(t_{g}^{+}) and y, and between V_{g}^{-}(t_{g}^{-}) and y. The results obtained during the process of the proposed method were reused for the final results of the proposed method.
- It would be better if the results of the consistency analysis of the other methods was also shown in Fig. 6.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- The order of the paper was a little unfamiliar. It would be better for Sec. 2 scNBS to be included in Sec. 3 method, and for the subsection ‘Synthetic Data for Benchmarking’ to be included in Sec. 4 Results.
- The words in the figures and the figures were too small to read.
- In Fig. 3, R is not defined.
- In the second line on page 4, the parentheses are missed.
- There is no explanation of how S_{g}^{+} and S_{g}^{-} were used.
- t_{g}^{+} was selected when the correlation between V_{g}^{+}(c) and y was maximized, however, t_{g}^{-} was selected when the correlation between V_{g}^{-}(c) and y was minimized. Why was the negative effect in t_{g}^{-} selected when the correlation between V_{g}^{-}(c) and y was minimized?
- The cut-off values, t_{g}^{+} and t_{g}^{-} were obtained by the correlation with y, and the final inference was also estimated by the relationship between V_{g}^{+}(t_{g}^{+}) and y, and between V_{g}^{-}(t_{g}^{-}) and y.
- It would be better if the results of the consistency analysis of the other methods was also shown in Fig. 6.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
It would be nice to make the picture a little bigger and add performance comparison with the other inference methods to the result part.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The proposed method could do statistical inference by accounting for both local and network-level interactions that might have clinical significance.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Somewhat Confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
In this paper the authors propose a novel method, called Semi-constrained Network-Based Statistic or scNBS, that uses a data-driven selection procedure to pool individual edges bounded by predefined large-scale networks to compare functional connectivity matrices and tries to overcome the main issues raised by conventional edge-wise approaches.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
• Novel approach grounding on solid statistical basis • Important impact in the field, with possible applications even in clinical scenarios (e.g., comparing patient data)
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
• I do not see major weaknesses in the paper
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Good. I would recommend the authors to release their code upon acceptance of their manuscript.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
In this paper the authors present a novel method (scNBS) to statistically compare the functional connectivity matrices from different conditions/populations trying to increase the statistical power which is greatly reduced when conventional edge-based methods are used. The paper is interesting and well-written, I enjoyed reading it, the figures are appropriate to convey the message and help the readers to follow the pipeline. I only have some minor points and suggestions:
- I would try to give more details and revise the section related to the generation of the synthetic data as at the moment it lacks a bit of clearness;
- Would it be possible to adapt this method to structural connectivity matrices? This would make the method even more applicable and general and I would mention this in the discussion;
- Please carefully proofread the manuscript as there are several typos across the different sections;
- I would suggest the authors to make their code freely available upon acceptance of the manuscript, as this would allow other interested researchers to explore and use this novel method.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Interesting and clear paper, the methodology is sound and robust.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper extended existing NBS and cNBS models and proposed a data-driven framework called semi-constrained NBS (scNBS) to refine the large-scale constraints used in cNBS based on both local and global information. They tested the effectiveness of the proposed model based on both synthetic and real resting state fMRI data, and demonstrated superior statistical power and validity in synthetic data, as well as consistency for repeated measurements in resting state data. The key strength of this paper is to integrate both local and network-level interactions for NBS based on solid statistics, which has potential clinical applications on modeling functional brain networks in healthy and patient subjects. Although it is an interesting paper aiming to improve the widely used NBS approach, the meta-reviewer as well as the reviewers have some concerns and confusions, and invite the authors to provide the rebuttal to clarify these major concerns: 1. Potential issues of circularity in the proposed method (see details from reviewers 1 and 2). 2. The experiment section is based on synthetic data and the typical noise models may not properly represent actual physiological situations. 3. How to define the reasonable subnetworks? 4. Paper organization concern (Reviewer 2). 5. Performance comparison with the other inference methods. Please also refer to the detailed comments from each reviewer.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5
Author Feedback
We thank all reviewers for their comments and briefly summarize their reviews. R1 & R3 were positive, stating that the method is interesting and novel with detailed descriptions and excellent illustrations through figures. Both agree that results of both synthetic and real data demonstrate the power, specificity, and consistency of the method and potential applications in clinical scenarios. They recommended accepting the paper with confidence. R2 was less positive, suggesting a weak rejection with “somewhat confidence”. We address the major concerns noted by the meta-reviewer below. We will incorporate these changes into the main text and release the code as suggested by all reviewers upon acceptance of our work.
Potential issues of circularity We apologize for not making this clearer and clarify that scNBS includes a cross-validation step to divide the data into two and avoid circularity. We use the 1st part to perform the threshold selection, reserving the 2nd for inference. We originally outlined this in the caption of Fig 1, and can emphasize it in the main text as well. All presented results were performed with this cross-validation step. Thus, circularity in inference is not driving the increased performance of scNBS.
Use of synthetic data; noise may not properly represent actual physiology We would like to clarify that only the results shown Fig. 4 used synthetic data. The results in Fig 6 used real fMRI for the consistency analysis. Also, to create the synthetic data, we started real data to incorporate actual physiological noise. We will add the need to perform additional experiments in real fMRI data for future work as noted by R1 and clarify the procedures as asked by R3.
How to define reasonable subnetworks? This is an open question in fMRI studies. Most standard atlases in the field have a corresponding set of 10-20 networks [1]; while exact boundaries of these networks vary, the spatial extent of networks tends to be fairly reproducible. For most connectomes generated from standard atlases, reasonable subnetworks should be available. We agree that different subnetworks could be used to explore their performance of scNBS, but do not believe that others would change the results of scNBS.
Paper organization R2 noted that the order of our paper was unfamiliar while R3 stated that the paper was well-written. We can reorganize sections as suggested by R2 and not change the length of the paper.
Performance comparison with the other inference methods We will add the consistency analyses for the other inference methods. In all cases, scNBA performed significantly better compared to the other inference methods. These results are presented here as method(r; ρ; Dice): Edge(.47; .47; .46); Edge-fdr(.54; .54; .62); NBS(.55; .55; .60); cNBS(.44; .53; .83); cNBS-fdr(.42; .36; .89); scNBS+(.86; .88;.92); scNBS-(.90; .88; .92); scNBS-fdr+(.96; .96; .94); scNBS-fdr-(.95; .98; .91).
Other comments: R2 & R3 noted some typos and errors. We will further proofread and fix any errors. R2 noted that the font was too small in some figures. We can increase the font size while keeping the figures the same size, as not to increase the length of the paper. R2 had a question about how t_{g}^{-} was selected. Both t_{g}^{+} and t_{g}^{-} were selected by maximizing the correlation between V_{g}^{+}(c) or V_{g}^{-}(c) and y. The “minimized” was a typo. We thank R2 for noting this. R1 & R3 suggested additional discussion items—limitations and structural connectome. We should be able to incorporate these within the page limit. RE limitations: the main limitation of scNBS is reduced spatial localization as only a network, rather than individual edges within, would be significant. RE structural connectome: yes, scNBS would work on any connectome, including structural connectome. [1] Uddin LQ, et al. Towards a Universal Taxonomy of Macro-scale Functional Human Brain Networks. Brain Topogr. 2019 Nov;32(6):926-942
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The key strength of this paper is to integrate both local and network-level interactions for NBS based on solid statistics, which has potential clinical applications on modeling functional brain networks in healthy and patient subjects. The authors have adequately and reasonably addressed the major concerns of all reviewers. Although there are still some concerns, I think it is good enough for acceptance of this paper. I would strongly suggest the authors revise the final version of this paper to integrate all useful comments.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Though I doubt the practical value of the proposed method in clinical studies, this paper is well written and may be potentially useful in the comunity.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
18
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Meta reviewer summarizes the paper. Basically, and extension for fMRI network analysis that uses both local and global information compared to previous approaches that diod not combine these two cues. Experiments on synthetic data generated from teh human connectome data demonstrated some effectiveness. Major reviewers issuess are withe experimental methodology. These include 1) oversimplification of the actual physiological noise in the simulated data, and lack of experiements with real data. Author responded in their rebuttal, highlighted Fig 6. which demosntrate repeatability on real data. However, without comparison to repeatability of previously suggested method. This leave the main question of the added-value of the proposed approach sensitivty over currently available repeatability which is highly relevant to the MICCAI community yet to be determined.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
NR