List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Xueli Chen, Xinqi Fan, Bernard Chiu
Abstract
We developed an interpretable deep biomarker known as Siamese change biomarker generation network (SCBG-Net) to evaluate the effects of therapies on carotid atherosclerosis based on the vessel wall and plaque volume and texture features extracted from three-dimensional ultrasound (3DUS) images. To the best of our knowledge, SCBG-Net is the first deep network developed for serial monitoring of carotid plaque changes. SCBG-Net automatically integrates volume and textural features extracted from 3DUS to generate a change biomarker called AutoVT (standing for Automatic integration of Volume and Textural features) that is sensitive to dietary treatments. The proposed AutoVT improves the cost-effectiveness of clinical trials required to establish the benefit of novel treatments, thereby decreasing the period that new anti-atherosclerotic treatments are withheld from patients needing them. To facilitate the interpretation of AutoVT, we developed an algorithm to generate change biomarker activation maps (CBAM) localizing regions having an important effect on AutoVT. The ability to visualize locations with prominent plaque progression/regression afforded by CBAM improves the interpretability of the proposed deep biomarker. Improvement in interpretability would allow the deep biomarker to gain sufficient trust from clinicians for them to incorporate the model into clinical workflow.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43987-2_29
SharedIt: https://rdcu.be/dnwJL
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #2
- Please describe the contribution of the paper
This paper proposed a Siameses network to monitor the change of vessel wall and plaque volume in 3D ultrasound images and generate a biomarker to quantify this change called AutoVT. To achieve interpretability, the framework also includes a activation maps to visualize the location of plaque change,
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- this model is able to extract both text feature change and absolute vessel wall volume change, which should be better than previous metrics that only focus on a certain aspect.
- From presented visualization, the change biomarker activation maps seems to be able to focus on plague.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- This model still requires intensive manual segmentation for all slice, which could constrain the efficiency in practical use.
- The common displacement between different volume scan is not discussed in this work.
- The main performance evaluation is based on the discriminative power (p-value) between placebo and pomegranate groups. However, as a dietary supplement, the effectiveness pomegranate is not validated by clinicians, at least not mentioned in the paper, to ensure distinct volume differences between two groups. Therefore, ii is also possible that the improvements in p-value could be overfitting.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Reproducibility is good, as the author has committed to publish code and dataset.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- It is claimed to be ‘serial’ change monitor but there are no experiments conducted in the paper including evaluation more than two volumes. A discussion about application of longer-term should be included.
- the visualization of activation map is too small. Moreover, without the original scans, it is hard to appreciate the result of CBAM.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper present a simple yet effective framework to detect plague change. However, there is not much novelty in the framework design and it still requires significant manual work for application in more complicated clinical cases.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper propose a interpretable deep biomarker, AutoVT, for carotid plaque monitoring, and develop a SCBG-Net to generate this biomarker.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper introduces a novel biomarker called AutoVT, along with an algorithm that generates biomarker activation maps (CBAM) to identify regions with significant impact on AutoVT. The results of the study demonstrate that AutoVT is more responsive to treatment effects than traditional measurements such as vessel wall and plaque volume. Overall, the study presents promising findings and highlights the potential of AutoVT as a more sensitive biomarker for assessing treatment efficacy.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
(1) The experiment only uses a single dataset, which limits the ability to assess the algorithm’s generalization and reproducibility. (2) While the paper presents significant results with p-values, it would benefit from a more comprehensive analysis of different biomarkers’ classification performance for placebo and Pomegranate patients. Such analysis could further validate AutoVT’s effectiveness.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The reproducibility of the paper is very good.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
(1) The experiment only uses a single dataset, which limits the ability to assess the algorithm’s generalization and reproducibility across multiple centers. It is suggested that the authors use multi-center data to address this limitation. (2) While the paper presents significant results with p-values, it would benefit from a more comprehensive analysis of different biomarkers’ classification performance for placebo and Pomegranate patients. Such analysis could further validate AutoVT’s effectiveness. (3) The hyperparameter “m” in Eq. (2) is not adequately explained in the paper, and it is unclear what factors may influence its value. (4) It would be helpful to clarify why the authors chose to use three-fold cross-validation instead of a higher number of cross-validations such as five or ten. (5) The paper would benefit from a more in-depth discussion of why the CBAM is superior to other activation maps. (6) Some parts of the paper’s writing are not sufficiently clear. For example, the first appearance of AutoVT in the abstract lacks explanation.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper introduces an interpretable deep biomarker, AutoVT, which can evaluate the effectiveness of dietary treatments for patients with carotid atherosclerosis. The paper also presents a deep learning algorithm, SCBG-Net, which innovatively utilizes Treatment Label Contrastive Loss and Plaque-focus Constraint to design AutoVT’s generation method. The idea is unique and has important clinical value. However, some issues still need to be addressed, such as the insufficient experimental design and unclear descriptions of certain aspects, such as hyperparameter selection.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #1
- Please describe the contribution of the paper
This paper developed an interpretable deep biomarker known as Siamese change biomarker generation network (SCBG-Net) to evaluate the effects of therapies on carotid atherosclerosis based on the vessel wall and plaque volume and texture features extracted from three-dimensional ultrasound (3DUS) images.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
SCBG-Net is the first deep network developed for serial monitoring of carotid plaque changes. SCBG-Net integrates volume and textural features extracted from 3DUS to generate a change biomarker called AutoVT that is sensitive to dietary treatments.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The paper does not sufficiently describe the experimental results of the designed network.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Is the dataset publicly available?
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
1、The paper does not sufficiently describe the experimental results of the designed network. 2、Can the designed network be applied to continuous two-dimensional ultrasound images? 3、The description of the dataset is not sufficient. For example, the description of the parameters of the acquisition machine should be added.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
SCBG-Net is the first deep network developed for serial monitoring of carotid plaque changes. SCBG-Net integrates volume and textural features extracted from 3DUS to generate a change biomarker called AutoVT that is sensitive to dietary treatments.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper provides to evaluate the effects of therapies on carotid atherosclerosis based on the vessel wall and plaque volume and texture features extracted from three-dimensional ultrasound images. The authors have conducted relatively sufficient experiments to support their arguments. The paper is well-organized and easy to follow, with clear arguments and well-supported conclusions.
Author Feedback
We thank all reviewers for their valuable comments and are encouraged by the positive comments on the novelty, clinical value of the proposed method and the clarity of our presentation. We address major comments below:
Q1. Dataset acquisition and effectiveness of pomegranate [R1, R2]: We will add a description of the acquisition machine’s parameters. Pomegranate is anti-oxidative and it has been shown to attenuate atherosclerosis in animal models. Previous studies have established that plaque texture features [6] and the weighted average of vessel-wall-plus-plaque thickness [1] are able to show the effect of pomegranate in the same cohort investigated in this study.
Q2. Displacement between different volumes [R2]: Potential misalignment of the baseline and follow-up 3DUS volumes is a major consideration leading to the introduction of the neighbourhood slice smoothing module, as discussed in Sec. 2.2.
Q3. Selection of Margin m [R3]: Intuitively, we would expect that increase of m would also increase the discrimination between the placebo and pomegranate groups (i.e., lower the p-value). However, in our experiment, we found that setting an m larger than the optimal value of 0.8 increases the standard deviation of AutoVT in the placebo group, thereby making AutoVT less discriminative. Therefore, we selected m by parameter tuning based on the validation dataset.
Q4: Why 3-fold? [R3]: Typically, 2/3~4/5 of the dataset is used for training to ensure that the train and test sets are large enough to statistically representative of the dataset. As the sample size of our dataset is not large (120), we opted for 3-fold cross-validation with a higher number of subjects in the test set.
Q5. More discussion about activation maps [R1, R3]: Most CNN visualization tools are only applicable to CNN classification models with class scores. Such a score does not exist in the current problem as it is not a classification problem. In this work, we developed a novel activation map CBAM, which is tailored for our proposed SCBG-Net. Unlike the existing CG-RAM, which may suffer from gradient saturation issues and produce artefacts and noise, the proposed CBAM is gradient-free and offers better visualization without much noise, as shown in Fig. 4.
Q6. Application on other cases [R1, R2]: (1) 2DUS images: In our current application on 3DUS, SCBG-Net analyzes 2D slices and integrates results in contiguous 2D axial slices to generate a biomarker for the 3DUS volume. Analyzing 2DUS longitudinal images using SCBG-Net would be even more straightforward as there is no need to integrate contents in different slices. (2) Longer-term cases: SCBG-Net can detect the effect of the treatment in the change of carotid atherosclerosis between two time points in about one year. The reason that changes are quantified based on two time points is that the rate of change of carotid plaque has been established as linear between the age of 50 to 75 in two studies involving 7345 patients in a stroke prevention clinic and 6727 healthy volunteers in the Troms study (Spence et al. Atherosclerosis, 2016;255:122-3). The clinical relevance of two scans approximately a year apart has been shown in a study investigating change in total plaque area (TPA) of 1685 patients (Spence et al. Stroke, 2002;33(12):2916-22) and change in total plaque volume (TPV) of 349 patients (Wannarong et al. Stroke. 2013;44(7):1859-65). Both studies show that patients with plaque progression in the first year of follow-up have a higher risk of cardiovascular events over 5 years.
Q7. Requirement of manual segmentation as a limitation [R2, R3]: We acknowledge this as a limitation. Automated segmentation methods have been developed recently to segment the carotid vessel wall and lumen. These methods will be incorporated into the proposed measurement workflow in our future work.