List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Qing Lu, Xiaowei Xu, Shunjie Dong, Cong Hao, Lei Yang, Cheng Zhuo, Yiyu Shi
Abstract
Accurately segmenting temporal frames of cine magnetic resonance imaging (MRI) is a crucial step in various real-time MRI guided cardiac interventions. To achieve fast and accurate visual assistance, there are strict requirements on the maximum latency and minimum throughput of the segmentation framework. State-of-the-art neural networks on this task are mostly hand-crafted to satisfy these constraints while achieving high accuracy. On the other hand, existing literature has demonstrated the power of neural architecture search (NAS) in automatically identifying the best neural architectures for various medical applications, within which differentiable NAS is a prevailing and efficient approach. However, they are mostly guided by accuracy, sometimes with computation complexity, but the importance of real-time constraints are overlooked. A major challenge is that such constraints are non-differentiable and thus are not compatible with the widely used differentiable NAS frameworks. In this paper, we present a strategy that can
directly handle real-time constraints in differentiable NAS frameworks,
named RT-DNAS. Experiments on extended 2017 MICCAI ACDC
dataset show that compared with state-of-the-art manually and auto-
matically designed architectures, RT-DNAS is able to identify neural
architectures that can achieve better accuracy while satisfying the real-
time constraints
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16443-9_58
SharedIt: https://rdcu.be/cVRzc
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The paper proposed a differentiable neural architecture search (NAS) method for 3D Cardiac Cine MRI Segmentation. The experimental results found a suitable network architecture with good segmentation performance as well as satisfying the latency and throughput constraints. Experimental results on the extended 2017 MICCAI ACDC dataset showed that the proposed method RT-DNAS obtained better overall results.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
• The proposed method RT-DNAS incorporated MS-NAS with a genetic algorithm to consider the latency and throughput constraints • RT-DNAS could be a good application for real-time cardiac MRI images which needs on-the-fly segmentation to avoid noticeable visual lag • RT-DNAS was evaluated on the extended 2017 MICCAI ACDC dataset and obtained overall better results when compared to both manually and automatically designed network architectures
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
• MS-NAS [1] is an existing method. The NAS part of the proposed method is also similar to one of the comparison methods HW-Aware NAS [2] and [3] with cell-level and network-level search, the only technic contribution would be the combination with a genetic algorithm which is used for network path selection optimization • Details of data processing are missing, which may add difficulties to the reproducibility of the paper • The experiments setting is not convincing. the comparison methods may not be the state-of-the-art, there is one paper from Medical Image Analysis that worked on the 3D cardiac cine MRI segmentation with manually designed network architecture and obtained better results [4]. Also, there are other latest NAS methods for medical image segmentation, such as [5]. • The results gap between ICA-UNet and RT-DNAS is quite trivial based one the second row and last second row in table 2. It seems the difference between these two methods will decrease when the hyperparameter n of ICA-UNet and the N of RT-DNAS increases
[1] Yan, X., Jiang, W., Shi, Y., Zhuo, C.: Ms-nas: Multi-scale neural architecture search for medical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 388–397. Springer (2020)
[2] Zeng, D., Jiang, W., Wang, T., Xu, X., Yuan, H., Huang, M., Zhuang, J., Hu, J., Shi, Y.: Towards cardiac intervention assistance: hardware-aware neural architecture exploration for real-time 3d cardiac cine mri segmentation. In: Proceedings of the 39th International Conference on Computer-Aided Design. pp. 1–8 (2020) [3] Bosma, M., Dushatskiy, A., Grewal, M., Alderliesten, T. and Bosman, P.A., 2022. Mixed-Block Neural Architecture Search for Medical Image Segmentation. arXiv preprint arXiv:2202.11401. [4] Dong, S., Pan, Z., Fu, Y., Yang, Q., Gao, Y., Yu, T., Shi, Y. and Zhuo, C., 2022. DeU-Net 2.0: Enhanced deformable U-Net for 3D cardiac cine MRI segmentation. Medical Image Analysis, 78, p.102389. [5] He, Y., Yang, D., Roth, H., Zhao, C. and Xu, D., 2021. Dints: Differentiable neural network topology search for 3d medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5841-5850).
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Satisfactory, if the code and extended data will be made public.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
• More thorough comparison experiments should be done. For example, other latest CNN and NAS based methods should be compared as mentioned in the main weaknesses of this paper • Although this paper is not difficult to read, the writing quality could be further improved by careful proofreading. There is one typo in the formulation (2).
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
• This paper is an incremental work from the prior art, and there are some similar works as mentioned in the weaknesses of this paper • The comparison experiments are not comprehensive. Lacking comparison to the state-of-the-art methods
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
The paper presents a method for neural architecture search which considers latency (in this case, 50ms) and throughput (22 fps) constraints. Latency is incorporated in the loss function and the architectures which do not meet the throughput constraints are ignored and the search is conducted again. Genetic algorithm is used to create new network architectures.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is well-written and provides a good description of the proposed method. The inclusion of latency and/or throughput in NAS has relevance for many other medical imaging applications. The method finds an architecture which produces better performance compared to baselines segmentation and NAS methods.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Some information/comments about the computation/time required for neural architecture search would have been nice.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper provides some details but there are still things which are not described sufficiently, probably not possible in the allowed page limit, so reproducing the paper is probably going to be challenging. If possible, it would be nice to publish the code.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Slightly larger text in Fig. 1 would be more easily readable.
Minor language issues:
- “In the past a few years,” -> “In the past few years,”
- “each paths is formed by” -> “each path is formed by”
- “the performance each path can be” -> “the performance of each path can be”
- on page 4: “L_{up}” -> “L_{ub}”
- “After the all the parameters are fixed” -> “After all the parameters are fixed”
- “metohd” -> “method”
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Latency is relavant for some medical image segmentation problems and the proposed method incorporates it in the NAS and achieves better results than the baseline manually designed methods and other NAS derived methods.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Not Confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
The authors propose an extension of the Neural architecture search (NAS) to incorporate latency/throughput trade-off for cardiac cine-MRI application. They extend MS-NAS and use a genetic search algorithm to find the optimal paths with proposed trade-offs. Ablation experiments and comparisons with state-of-the-art architectures and other NAS methods demonstrate the benefits of their approach.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- meaningful extenstion to neural architecture search to optimize for latency.
- good set of experiments to make a case for their hypothesis.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- few bits of information is missing.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Ok. Authors mention in the reproducibility checklist that the code will be available.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- In terms of the computation time, how long did it take to run these hyper-parameter searches in the RT-DNAS method?
- Once the architecture has been settled on, the authors find which optimal path had the highest latency to accuracy trade-off? Can they describe/quantify if it was more Conv or skip connections, downsampling/upsampling? It would be exciting for the readers to see what optimized this final set of layers and overall network architecture was.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Their logical extension of the MS-NAS to incorporate latency and optimizing for both backed up by good experiments.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Somewhat Confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #4
- Please describe the contribution of the paper
In this paper, a latency-aware neural architecture search method is proposed for cardiac MR segmentation. The real-time constraints has been applied and made differentiable. The results are evaluated in one public dataset.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Introducing the latency constraints to the NAS framework. Previously the focus is on reducing the FLOPS by compress the model size.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
(1) Some ablation studies missing. The framework depends heavily on MS-NAS[24] but in table 2 there is no comparison with [24]. [24] could be used as a baseline to compare with. (2) compared with [25], the improvement in accuracy is relatively small, the MYO is even lower than it.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The dataset is public available and the authors mentioned they will release the code in the questionnaire.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
(1) [24] would be a strong baseline to compare with and showing the latency improvement while keep the accuracy. (2) It will be safer to say “improve the latency issue while keep comparable accuracy”, as the improvement in accuracy is not marginal.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The concerns are about the missing baseline performance and the accuracy improvement is not significant.
- Number of papers in your stack
8
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
Most of concerns addressed, thus raise the rating
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper proposed a differentiable neural architecture search (NAS) method for 3D Cardiac Cine MRI Segmentation. It extends MS-NAS and uses a genetic search algorithm to find the optimal paths. As the reviewers mentioned, this is an interesting topic and latency is relevant for some medical image segmentation problems. In the rebuttal, please address the concerns on novelty of this particular work and experimental design. Also please address the concern on the trivial difference between the results in comparison.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
7
Author Feedback
We appreciate all the valuable comments and strive to resolve the major concerns and misunderstandings raised herein.
Q1: The technical contribution is minimal compared with MS-NAS (R1)
R: 1) MS-NAS focuses on accuracy only. The main objective of our work, however, is to propose a latency-aware NAS for real-time medical image segmentation with a strict latency upper-bound (< 50 ms). This is achieved by a differentiable latency estimation loss function and a penalty term for exceeding the upper-bound, neither of which are included in MS-NAS. MS-NAS is selected as our backbone due to its architectural flexibility and state-of-the-art performance on medical image segmentation. Our work can also build upon other NAS frameworks.
2) On top of the differentiable latency loss, we propose genetic algorithm for latency-aware network derivation, which provides further opportunities for lower latency and higher accuracy.
Q2: Insignificant improvement and missing comparisons to other work. (R1, R4)
R: 1) Comparing with ICA-UNet (R1). Note that the goal of this work is to find a network that can attain the highest possible accuracy while satisfying latency and throughput constraints. So the best result identified by RT-DNAS is when Nl=8, and that by ICA-UNet is when n=4. Further increasing these hyperparameters will result in violation of latency/throughput constraints and are therefore not feasible. Comparing these two best results, RT-DNAS achieves over 1.1% higher average Dice. Note that we include the results of other n and Nl values to show that it is possible to tradeoff accuracy for speed, and it is not meaningful to compare such inferior solutions.
2) Comparing with HW-aware NAS [25] (R4). HW-aware NAS has a similar Dice but violates latency and throughput constraints.
3) Comparing with other SOTA networks and MS-NAS (R1, R4). We did not include these in the paper originally because none of the suggested works take latency into consideration and therefore would not be applicable to real-time segmentation. Following reviewer’s comments, we added the comparison with MS-NAS, DeU-Net and Dints below and will include them in the paper. We did not have time to finish training Dints to get accuracy, but with the architecture we are able to profile the latency/throughput.
Method RV MYO LV Average LT (ms) TP (FPS)
MS-NAS .928±.011 .911±.024 .961±.025 .933±.023 55 18.8 DeU-NAS .961±.015 .927±.025 .971±.018 .953±.019 121 10.2 Dints - - - - 79 17.5
RT-DNAS .933±.018 .902±.029 .959±.011 .931±.019 46 23.4 Although these methods can achieve higher Dice in some cases, they all exceed the latency (<=50 ms) or throughput (>= 22 FPS) constraints.
4) From the above comparison, we argue that by incorporating latency loss into the search process, RT-NAS (Nl=8) achieves higher accuracy than all the state-of-the-art that can satisfy real-time constraints.
Q3: Computation time (R2, R3)
R: On a P100 GPU, it takes 1~10 hours to complete a full search depending on Nl. We appreciate the suggestion and will include it in the final paper.
Q4: Data processing information (R1)
R: We follow the same procedures of ICA-UNet to process data and the details can be found in our code once published.
Q5: Insights on the resultant architecture (R2)
R: Thanks for bringing this interesting topic up. We have provided the optimal architecture found in the supplemental material. There is a trend that non-scaling operations are likely to occur at an early stage of each path. We will include such discussion in the paper.
Q6: Advice on improving clarity of writing. R: Much appreciated and will revise the paper accordingly.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have address the comments of the reviewers to a large extent. NAS work is also an interesting addition to MICCAI. Therefore I’d like to recommend acceptance.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors well clarified the issues raised by the reviewers in the rebuttal, such as technical contribution, insufficient validation/comparison with other SOTA methods, etc. One reviewer changed the rating from 4 to 5, so three reviewers are supporting this paper for acceptance.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
10
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The response addressed my concerns.Experimental results have added ,including ablation studies and comparison method.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
9