List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Jiansheng Fang, Jingwen Wang, Anwei Li, Yuguang Yan, Yonghe Hou, Chao Song, Hongbo Liu, Jiang Liu
Abstract
In the management of lung nodules, we are desirable to predict nodule evolution in terms of its diameter variation on Computed Tomography (CT) scans and then provide a follow-up recommendation according to the predicted result of the growing trend of the nodule. In order to improve the performance of growth trend prediction for lung nodules, it is vital to compare the changes of the same nodule in consecutive CT scans. Motivated by this, we screened out 4,666 subjects with more than two consecutive CT scans from the National Lung Screening Trial (NLST) dataset to organize a temporal dataset called NLSTt. In specific, we first detect and pair regions of interest (ROIs) covering the same nodule based on registered CT scans. After that, we predict the texture category and diameter size of the nodules through models. Last, we annotate the evolution class of each nodule according to its changes in diameter. Based on the built NLSTt dataset, we propose a siamese encoder to simultaneously exploit the discriminative features of 3D ROIs detected from consecutive CT scans. Then we novelly design a spatial-temporal mixer (STM) to leverage the interval changes of the same nodule in sequential 3D ROIs and capture spatial dependencies of nodule regions and the current 3D ROI. According to the clinical diagnosis routine, we employ hierarchical loss to pay more attention to growing nodules. The extensive experiments on our organized dataset demonstrate the advantage of our proposed method. We also conduct experiments on an in-house dataset to evaluate the clinical utility of our method by comparing it against skilled clinicians.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16431-6_46
SharedIt: https://rdcu.be/cVD62
Link to the code repository
https://github.com/liaw05/STMixer
Link to the dataset(s)
https://github.com/liaw05/STMixer
Reviews
Review #1
- Please describe the contribution of the paper
Main contributions are: (1) Authors derived the prediction of lung nodule evolution by organizing a new temporal CT dataset called NLSTt by combing automatic annotation and manual review. (2) Authors proposed a spatial-temporal mixer (STM) to leverage both temporal and spatial information involved in the global and lesion features generated from 3D ROI pairs. (3) Authors conducted extensive experiments on the NLSTt dataset to evaluate the performance of their proposed method and confirmed the effectiveness of their model on an in-house dataset from the perspective of clinical practice.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Authors proposed a Siamese Encoder-based Spatial-Temporal Mixer for Growth Trend Prediction of Lung Nodules on CT scans.
Their method can help predict the growth of the nodule, to facilitate the diagnosis and treatment of theses nodules, which is a very targeted task in now a days medical field application.
The framework is simple, though authors succeeded to achieve reasonable classification results.
This work can serve as a starting point of formulating a much complex problem as: whether a treatment is efficient or no in the treatment of lung cancer or whether a nodule will grow into a malignant nodule.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
It is better to add some examples of images where the algorithm fails / succeed to classify correctly the output, a lake of images make the quality of the discussion section poor. The mathematical formulation of the loss functions and model in general is poor. It would improve the quality of the paper to improve the mathematical formulation.
The discussion section is poor, and authors did not specify why their algorithm classify correctly or misclassify samples.
The proposed method is not innovative, and components of the framework have been used before for similar tasks. This paper would have been more suitable for miccai if a more innovative approach was used.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
reproducible
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
8- It is better to add some examples of images where the algorithm fails / succeed to classify correctly the output, a lake of images makes the quality of the discussion section poor. Illustrations improve the quality of the paper.
The discussion section is poor, and it would add to the quality of the paper to study more the strength/weaknesses of such method.
The framework is simple and seemed to give good results though the components are not innovative and have been used before which make this paper more suitable for other conferences rather than miccai.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The idea is of the paper interesting and very useful in for the medical field. The prediction of the growth of lung nodules can assist in multiple tasks like future treatment orientation or overall survival estimation of patients. It would be good to show some examples where the algorithm failed and explain why The components of the framework are not innovative and have been used before.
- Number of papers in your stack
2
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
To advance the prediction of lung nodule evolution, the authors construct a new temporal CT dataset NLSTt, and propose a spatial-temporal mixer (STM) to extract the temporal and spatial information involved in the global and lesion features of CT images information.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors of this paper construct a CT dataset NLSTt for predicting the evolution of lung nodules based on the NLST dataset, which combines automatic inference and manual review to obtain reliable labels for nodule texture types and evolution classes. In our proposed method for pulmonary nodule growth trend prediction, a s spatial-temporal mixer (STM) module is proposed to exploit both spatial and temporal information. A two-layer H-loss is also proposed to pay more attention to the nodule growth (dilatation class) related to possible cancerogenesis in clinical practice.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The characteristics of pulmonary nodules considered in this paper are only diameter and texture, and do not utilize other sign information and clinical information of each patient. Some important formulas are not listed in the Methods section. There are few methods compared in the experimental part, and the results of the comparison experiments are not visualized. A description of the limitations of the method and future work is missing.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
This paper does not specify the software framework and version used, the description of the model is not clear enough, the experimental code and other experimental details are not provided, and the reproducibility of the paper is poor.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
It is suggested that other signs of pulmonary nodules and clinical information of the case be used in this paper, which is closer to the doctor’s clinical diagnosis process. In the experimental part, more comparative experiments and ablation experiments should be done to show the superiority and effectiveness of the method more comprehensively and accurately.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The CT dataset NLSTt constructed in this paper is a good work to predict the evolution of lung nodules, and it is a good idea to construct a spatial-temporal mixer (STM) module to effectively utilize the spatial and temporal information of the ROI. However, the detailed description of the method is not clear enough, and the experimental part is not perfect.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
This paper focuses on the prediction of lung nodule growth trend in CT scans. Authors organized a new temporal CT dataset from NLST dataset and proposed a growth trend prediction framework including a Siamese encoder, a spatial-temporal mixer and a hierarchical loss. Authors validated the framework on the organized dataset as well as on an in-house dataset to demonstrate its feasibility.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Clinical feasibility: The test with in-house datasets demonstrates the feasibility of the proposed framework.Extendibility: this work can be extended in lesion trend prediction in other diseases with time series images, such as cervical cancer lesion trend in colposcopy.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Limited comparison to state-of-the-art: 1) In the subsection of NLSTt dataset acquisition, it would be better if authors could explain clearly how they pair nodules whose locations change a lot between two Ti and T0, and what is the accuracy of pairing correctly nodules. 2)Authors compared between the CNN and ViT encoder, as well as between different mixers, whereas the proposed method is not compared with existing methods, such as [17,19] and [R1,R2]. It would be more interesting and convinced to compare between such methods. [R1]. Rafael-Palou, X., Aubanell, A., Bonavita, I., Ceresa, M., Piella, G., Ribas, V.,Ballester, M. A. G.: Re-identification and growth detection of pulmonary nodules without image registration using 3d siamese neural networks. Medical Image Analysis, 67, 101823 (2021) [R2] Tao, Guangyu et al. “Prediction of future imagery of lung nodule as growth modeling with follow-up computed tomography scans using deep learning: a retrospective cohort study.” Translational lung cancer research vol. 11,2 (2022): 250-262. doi:10.21037/tlcr-22-59
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Good reproducibility: Authors provided clear description about the pipeline of their methods, clear description about the datasets they used for experiments, as well as the parameters and environments used for training models.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Lack of clarity: Better explanation of the figures: it would be helpful if authors could make the figure captions, especially Fig.3, more clearly. Abbreviations in the figure should be explained in the caption, for example, MLP, FG1,etc. For future work, I would recommend to apply the proposed framework on more diseases and more image modalities, such as lesion variation in cine MRI, cervical cancer lesion variation in time serie colposcopy, etc.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Authors proposed a novel framework for nodule trend prediction but comparisons with the state-art-the-art is limited. I would suggest accepting it.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
Authors provided reasonable answer to my main concerned issue (comparisons with the state-of-art methods).
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper received mixed reviews and will recommend for rebuttal. This is an interesting and informative work addressing an important problem as well, Partially, as stated by reviewers such that “The CT dataset NLSTt constructed in this paper is a good work to predict the evolution of lung nodules, and it is a good idea to construct a spatial-temporal mixer (STM) module to effectively utilize the spatial and temporal information of the ROI. However, the detailed description of the method is not clear enough, and the experimental part is not perfect.” In the rebuttal, please address: 1) “Lack of clarity: Better explanation of the figures:” 2) statement on the novelty: “The components of the framework are not innovative and have been used before.”
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
10
Author Feedback
We appreciate all the reviewers for the insightful comments.
General response Q1: Comparison to state-of-the-art methods A1: (i) [17], [19], and [R2] rely on hand-crafted features, e.g., bronchiole change and vascular change, which are annotated by the authors while are absent in our organized dataset. Therefore, we cannot conduct these methods in our experiments. (ii) Similar to our method, [R1] ([14] in our paper) leverages the textures of 3D ROI pairs and employs CNN as the encoder. Hence, we make a comparison between [R1] and our method based on the CNN encoder in the following table. Our method consistently outperforms [R1] on two datasets in terms of two metrics, which verifies the efficacy of our method. (iii) We will discuss the recommended papers [R1] and [R2] in the revision.
Method__TestSet(AUC@H1)TestSet(AUC@H2)__InhouseSet(AUC@H1)__InhouseSet(AUC@H2) R1__81.2____75.2____69.4____66.7 STM(Ours)__83.0____76.3____73.5_____71.6
Q2: Mathematical formulation A2: In the revision, we will revise the formulas of the loss function and the model, and add the formula of H-loss.
R#1 Q1: Statement on the novelty A1: In order to extract spatial and temporal information, we propose a novel module called spatial-temporal mixer (STM). In specific, STM fuses spatial information based on local and global embeddings from one CT image and temporal information based on the embeddings of the same nodule from two consecutive CT images, which has not been considered in existing works of nodule trend prediction. In addition, our method is able to automatically augment a learnable embedding for FL0 when the T0 scan is unavailable, which makes our method more applicable in practice. Reviewer #3 also agreed that “authors proposed a novel framework for nodule trend prediction”.
Q2: Examples and weaknesses of the method A2: (i) Please refer to Appendix A1 for five examples of the classification results predicted by our model and clinicians A and B in Appendix A1. (ii) Due to the limited number of part-solid nodules in the dataset, it is difficult for our method to achieve promising performance on samples with part-solid nodules. In addition, the clinical information is not accessible in the current dataset, which restricts the performance of our method. In the future, we will investigate how to leverage more clinical information for growth prediction.
R#2 Q1: Usage of more clinical information A1: Currently, according to the management guideline for lung nodules [10], we consider diameter and texture for dataset organization and model training. In the future, we will investigate how to introduce more clinical information for growth prediction.
Q2: The reproducibility of the paper A2: We will add more implementation details in the revision and release our code and NLSTt dataset after acceptance to facilitate the research of lung nodule management.
Q3: Illustrations of classification results A3: Please refer to Appendix A1 for five examples of the classification results predicted by our model and clinicians A and B in Appendix A1.
R#3 Q1: Clear explanation for the figures A1: We will add the explanation of abbreviations in the figure captions to clarify our method better.
Q2: More Details of nodule pairing in NLSTt dataset acquisition A2: In the pipeline of organizing the NLSTt dataset, we first make registrations for the consecutive CT scans of each patient, then detect nodules. For the detected two nodules at T0 and T1, the pair criterion is that the Euclidean distance between the center points of the two nodules is less than 1.5mm. If the nodule location changes significantly between T0 and T1 (>1.5mm), we assert that they are not the same nodule. Finally, we ask experienced clinicians to review to ensure the accuracy of paring nodules correctly.
We will add new results in the appendix and revise the paper according to the above responses.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper constructed a very relevant dataset an dproposed a module of spatial-temporal mixer (STM) to incorporate the spatial-temporal appearance and changes of a lung nodule 3D ROI instance to classify its growth pattern. Although the experiments are not perfect and the study itself looks initial to some degree, overall it may pass the acceptance threashold for MICCAI acceptance. A meaningful problem, reasonably novel technical solutions and experiemntal evaluation should be considered as adequate.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The work is well-motivated and the proposed solution makes sense. Major concern from the reviewer is regarding the method details, and the rebuttal addressed the questions.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Rebuttal is partially successful as the major concern of innovation stays limited. However, experiments are convincing for yet another method for lung cancer studies with AI.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
NR