List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Yiting Lu, Jun Fu, Xin Li, Wei Zhou, Sen Liu, Xinxin Zhang, Wei Wu, Congfu Jia, Ying Liu, Zhibo Chen
Abstract
Coronary CT Angiography (CCTA) is susceptible to various distortions (e.g., artifacts and noise), which severely compromise the exact diagnosis of cardiovascular diseases. The appropriate CCTA Vessel-level Image Quality Assessment (CCTA VIQA) algorithm can be used to reduce the risk of error diagnosis. The primary challenges of CCTA VIQA are that the local part of coronary that determines final quality is hard to locate. To tackle the challenge, we formulate CCTA VIQA as a multiple-instance learning (MIL) problem, and exploit Transformer-based MIL module (termed as T-MIL) to aggregate the multiple instances along the coronary centerline into the final quality. However, not all instances are informative for final quality. There are some quality-irrelevant/negative instances intervening the exact quality assessment(e.g., instances covering only background or the coronary in instances is not identifiable). Therefore, we propose a Progressive Reinforcement learning based Instance Discarding module (termed as PRID) to progressively remove quality-irrelevant/negative instances for CCTA VIQA. Based on the above two modules, we propose a Reinforced Transformer Network (RTN) for automatic CCTA VIQA based on end-to-end optimization. The experimental results demonstrate that our proposed method achieves the state-of-the-art performance on the real-world
CCTA dataset, exceeding previous MIL methods by a large margin.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16431-6_61
SharedIt: https://rdcu.be/cVD7h
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #2
- Please describe the contribution of the paper
The paper presents a method to assess image quality of 3D coronary computed-tomography angiography, i.e., with contrast enhancement, scans. Image quality assessment is approached as a binary classification problem per coronary branch. The problem is addressed as multiple instance learning problem using artificial neural networks. As a minor contribution a reinforcement learning-based instance discarding module is introduced that selects the most relevant instances within the multiple instance learning framework for final classification.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- On the used data collection, the experimental results clearly show the benefit of adopting the suggested approach and additional network components in comparison to using less elaborated schemes.
- The ambition to formulate the method in a formal mathematical manner has to be appreciated.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- One of the major weaknesses of this paper is the relatively small amount of data (CCTA scans from only 40 patients) that is used for evaluation. It isn’t publicly available, either.
- From my point of view, the number of small issues pile up to a significant overall weakness of the submission.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Challenging: no obvious access to the data and the implementation
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- The first step in the processing pipeline, i.e., centerline tracing, may suffer from image artifacts as well. I don’t see how this is addressed by the suggested method.
- English spelling and in particular grammar should be carefully double-checked once again. (“We follows …”, …)
- It’s not entirely clear to me, which problem is solved at all. Are all centerlines processed at once and then a final quality score for the whole image is computed?
- The definitions and equations on page 4 are hard to follow: how do I have to imagine an instance embedding Z_L[1:n]? Is this a family of values? . As an improvement one could think of using the following convention for mathematical notation: scalars as regular small letters, vectors as small letters in bold, matrices as capital letters in bold, sets as capital letters with \mathcal, functions also with domain and value range …
-
PRID: from the explanations I can’t relate the reinforcement learning strategy to a Markov decision process as stated I the paper.
- Minor things . The number of discarded instances seems to be a free parameter of the system. How shall it be chosen in practice? Isn’t it data dependent? . The term “follow-up” on page 1 might not be used in the right way as “follow-up” most often refers to examinations carried out days or months later. . “the coronary artery”? I’d rather speak of “the coronary arteries”. . I’d recommend introducing the abbreviations again in the main text not only in the abstract. . “SOTA performance” for “state-of-the-art performance”: I never heard this abbreviation. . Sentences shouldn’t be started with abbreviations. . Abbreviations are use inconsistently: e.g., “Fig.” vs. “Figure”. . Word repetition “MIL aggregators in MIL methods” . The abstract speaks of “above two modules” while there’s been only one introduced before. I don’t understand this.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- Mathematical formulation not clear enough
- Experiments only on inhouse data and with inhouse implementations of alternative approaches
- Conceptual flaws . Fully automatic centerline tracing as preprocessing step, whose result may as well be impacted by poor image quality . Are really all conceivable kinds of artifacts handled by the method and properly represented in the data collection?
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
3
- [Post rebuttal] Please justify your decision
Rebuttal not convincing; too many things authors say they want to change in the final version; we cannot check this.
Review #3
- Please describe the contribution of the paper
This paper formulates CCTA vessel level image quality assessment (VIQA) as a multiple-instance learning (MIL) problem, and exploit Transformer-based MIL backbone (termed as T-MIL) to aggregate the multiple instances along the coronary centerline into the final quality.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The author formulate the CCTA vessel-level quality assessment as the typical multi instance learning problem, and introduce transformer to aggregate multiple instances and map them to final quality.
- The authors proposed a progressive reinforced learning based instance discarding strategy to mine the most informative instances for transformer network.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The proposed method was evaluated on one hospital which did not include CCTA images with different resolution. Only one doctor did the visual assessment of the quality for the CCTA images. So, the presented work might need more evaluation on its accuracy, effectiveness and robustness
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper describes the method clearly, and listed their training details such as: input size, batch-size, training epochs, loss function and evaluation metrics, so the readers should be able to reproduce the paper.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The proposed method classify the quality of the vessel by first removing the instances that not relevant for determining the quality of the vessels, then using the remaining instances to decide the quality of the vessel.
- The authors did some augmentation on the training dataset, could you please provide the volume of the dataset after augmentation?
- Different coronary arteries have different size and different length, the proposed method used the same number of cubes (n=19). Could you please add the details on how do you deal with the different lengths?
- Table l compares the results of the proposed method with state-of-art methods which also includes the results with/without PRID. Table 2 compares the results with different discarding numbers. It seems T-MIL without PRID has higher accuracy than the PRID with less amount of discarding number of instances. Could you please help me understand this?
- The authors provided supplementary materials for the propose method but lack of reference or description of the contents in the supplementary materials. Please consider add one or more reference sentences to the Figures, especially the Figure1. And Based on Figure1, it will be difficult to claim that the remaining examples are mainly concentrated in the front of coronary artery since index 2,4,7 has low frequency and, only 1,3,5,12,17 has a frequency more than 0.5. So, I would like to suggest the user to rephrase the claim/conclusion on Figure1. Instead of saying that the main concentration was on the front of the coronary, the conclusion could be that the assumption that only limited instances play important role in the VIQA is correct.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper is well-written and well-organized. The proposed method formulate the image quality problem as a multi-instance learning problem, and combined with reinforcement learning to remove irrelevant features which makes the network focus more on important features. And the authors did ablation study and comparison experiments to show the effectiveness of the proposed method.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
The authors explained every questions clearly and provided details and I am happy with the answers. I will keep my opinion as accept this paper.
Review #4
- Please describe the contribution of the paper
In this paper, the authors present a novel Reinforced Transformer Network (RTN) model for the Vessel-level Image Quality Assessment task on Coronary CT Angiography images. This model contains two parts: The transformer-based multi-instance learning (T-MIL) backbone and the Progressive Reinforcement learning based Instance Discarding module (PRID). T-MIL takes the responsibility for feature extraction (from image cube instances), providing states for the latter PRID and final classification. PRID is to prevent the intervention of irrelevant instances. This model outperforms other existing approaches by a large margin on a private dataset.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Overall, this paper is well organized and all technical modules are well explained. Below are the specific strengths: 1, the idea to leverage the RL based instance discarding strategy to exclude irrelevant instances is novel and proved to be quite efficient in the end to improve the performance. It adopts Markov Decision Process and uses the output of T-MIL as the state. This design enables an end-to-end training pipeline and keeps the architecture light.
2, the transformer based architecture fits well with the dynamic input sequences introduced by the instance discarding process. Experiments show that without PRID, this T-MIL is already a very strong baseline outperforming other existing approaches.
3, the ablation study on the pooling layer inside PRID (Table 3) justifies the choice of PMA versus other common pooling strategies such as max and average pooling.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Despite the strengths, I have the following concerns and questions: 1, the dataset is relatively small making the significance of the finding suffer. Also given the fact of a private dataset is used here, more details of the labeling process should be disclosed.
2, since the centerline tracking algorithm comes first place before the proposed algorithm, I am wondering how the performance will be if the failure of this algorithm happens.
3, how is the crop size is determined? Would it affect the final performance?
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
positive
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
please address the concerns and questions above.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The whole idea of transformer based multi-instance-learning + RL based instance discarding model is novel.
- Number of papers in your stack
6
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper assesses image quality of 3D coronary computed-tomography angiography by a binary classification problem using a neural network to combine multiple instance learning and reinforcement learning.
The reviewers mostly think this paper has contributions to formulate the medical problem in a tangible mathematical problem and provides a reasonable solution, including R#1 who rated the paper negatively. In the rebuttal, the authors are expected to address the numerous concerns raised by the reviewers. Among others, two reviewers are concerned with the experimental data because of its size. In addition, a number of questions are raised for the experiment settings and result explanations. Please address these in the rebuttal.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6
Author Feedback
To All Reviewers: We thank all reviewers and meta reviewers for the constructive suggestions and positive comments on the problem importance, idea novelty, paper writing, comprehensive experiments, and method effectiveness. Q1: The size of our CCTA dataset A1:In deed, our proposed CCTA dataset is relatively large and fine-grained (vessel-level labeling) in this field, which contains 210 labeled vessels from 80 CTs. e.g., TR-Net(MICCAI2021) only has 76 CTs. The reasons are 1) privacy issues. 2) labeling difficulties for vessel-level IQA. To ensure the reliability of the results on our dataset, we augment data by 114 times and adopt 5 cross-validation used in AttentionMIL(PMLR2018) to reduce the influence of data size. Thanks for your suggestions, we will extend and share the dataset in our future work. Q2: The influence of Centerline algorithm on our RTN A2: Our used centerline algorithm has high accuracy of 94% for centerline extraction, which is robust for most artifacts in CCTA image. In our paper, we also apply the semi-automatic centerline algorithm to further detect the centerline, which can assist automatic centerline algorithm. Therefore, the influence of centerline algorithm on RTN is minor, which will be clarified in the revision. 1 To Reviewer 2 About dataset and centerline, please see Q1 and Q2 of To All Reviewers. Q1: Which problem is solved A1: Our RTN aims to solve problem of more fine-grained vessel-level IQA, which can also be fused into image-level IQA. Q2: Definition and equations on page 4 A2: The instance embeddings ZL[1 : n] are features of n cubes/instances in one vessel extracted by T-MIL. We will refine the description in revision. Q3: Reinforced learning & Markov decision process A3: Markov decision process is established as follows: 1) At the t discarding step, the RL agent take the action At based on the state St. 2) Then the state St+1 at the (t + 1) step will depend on {St, At}. The definitions of state, action, reward are clarified in our paper. Q4: Minor things A4: The optimal number of discarded instances is based on experiments, which is fixed in overall experiments. We will add the related description in revision. 2 To Reviewer 3 Q1:Different resolutions A1:The resolution in our paper is commonly-used in hospital. Thanks for your advice and we will extend the different resolutions in the future work. Q2:Doctor annotation A2:Thanks for your suggestions. The annotation for our dataset is reliable. 1), the doctor is specialized from the imaging department. 2), the dataset will be double-checked to remove unsure labeling vessel by doctor. But your advice is significant, we will extent dataset with more specilized doctors for labelling. Q3:Data volume after augmentation A3:The volume of augmented dataset is 6 × 19 × 210 = 23940 (moving direction:6, cube numbers:19) Q4:Different lengths of vessels A4:We set optimal crop size (20) and number (19) of cubes to ensure that the sampled cubes can cover different lengths of vessels. Q5:baseline T-MIL & RTN with discarding number 4 (RTN4) A5:The reasons for that the performance of the baseline T-MIL is slightly higher than RTN4 are as follows: 1) The randomness caused by system. 2) The discarding number 4 is too smaller, which has tiny effects. Despite that, the RTN4 has higher AUC than T-MIL. A6: Thanks for your suggestions of supplementary, we will refine them in the revision. 3 To Reviewer 4 Q1:Dataset and labeling process A1:Please see Q1 of To All Reviewers. We collected CT scans of patients with age ranging from 33 to 91 years old, which are visually examined by professional doctors. Q2:Centerline algorithm A2: Please see Q2 of To All Reviewers. Q3:Crop size A3:The accuracy of different crop size h : h=15: 0.8042, h=20: 0.8546, h=30: 0.8510. The reasons are: 1) When h is smaller, the cubes cannot cover the whole vessel. 2) If h is higher, the redundancy of quality-unrelated content will increase. We will add the ablation studies in our revised paper.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
As commented in my pre-rebuttal summary, this paper has some merits. However, I agree with Reviewer #2’s post-rebuttal comment in that quite a few questions/issuses raised by the reviewers are not well addressed and this paper needs a significant revision before it is ready for publication.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
7
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper proposed a novel Reinforced Transformer Network for the vessel-level image quality assessment (which can be integrated to an image-level metric) for coronary CT angiography. It has two major novelties: the transformer-based multi-instance learning module and the progressive reinforcement learning based instance discarding module. The proposed method is novel. The reviews are mixed. The main concern is the small dataset and single-expert annotation. However, due to the difficulty of the task, it is hard to collect a large dataset and the current dataset with 80 CT scans is relatively large in this field. The authors clarified some issues in the rebuttal and promised to address some in the final version. Overall, I think its merits out-weigh limitations and I recommend accepting this work.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The major merits of this paper include the novelty of the proposed reinforced transformer network. The dataset size would be considered large given the the difficulty of collecting thus dataset and labeling. I would recommend acceptance of the paper.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5