List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Bolun Zeng, Li Chen, Yuanyi Zheng, Ron Kikinis, Xiaojun Chen
Abstract
Ultrasound imaging is a promising tool for clinical hand examination due to its radiation-free and cost-effective nature. To mitigate the impact of ultrasonic imaging defects on accurate clinical diagnosis, automatic fine-grained hand bone segmentation is highly desired. However, existing ultrasound image segmentation methods face difficulties in performing this task due to the presence of numerous categories and insignificant inter-class differences. To address these challenges, we propose a novel Adaptive Multi-dimensional Convolutional Network (AMCNet) for fine-grained hand bone segmentation. It is capable of dynamically adjusting the weights of 2D and 3D convolutional features at different levels via an adaptive multi-dimensional feature fusion mechanism. We also design an anatomy-constraint loss to encourage the model to learn anatomical relationships and effectively mine hard samples. Experiments demonstrate that our method outperforms other comparison methods and effectively addresses the task of fine-grained hand bone segmentation in ultrasound volume. We have developed a user-friendly and extensible module on the 3D Slicer platform based on the proposed method and will release it globally to promote greater value in clinical applications. The source code is available at https://github.com/BL-Zeng/AMCNet.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43901-8_38
SharedIt: https://rdcu.be/dnwDM
Link to the code repository
https://github.com/BL-Zeng/AMCNet
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes a UNet inspired CNN named as Adaptive Multi-dimensional Convolution Network (AMCNet) for fine-grain hand bone segmentation in 3D ultrasound volumes. AMCNet is built using convolutional modules (ACMs) which integrates the features extracted using 2D and 3D convolutional filters with learned weights. In addition a loss term which is a function of the difference between ground truth and dilated predicted masks is proposed to complement the standard dice loss. The proposed method is compared with the popular 2D/3D medical image segmentation methods and a comparatively better performance is reported.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Fine-grain segmentation is a challenging task and the proposed method aptly handles it.
- The paper is easy to follow.
- Ablation studies are conducted to demonstrate the effect of different components of the rosed network.
- A software plugin to integrate the proposed model with 3D slicer is developed.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The idea of simultaneously using 2D and 3D features already exits in literature. This paper should have clearly highlighted the technical differences between the proposed and existing similar approaches.
- The so called Antomy-constraint loss is not properly motivated. The design choices are not explained either, e.g. ReLU activation in eq. (5) will possibly ignore the cases of undersegmentation. This is evident from the F1-score values which are relatively smaller than Recall values in Table 1.
- The margin of improvement in Table 1 is small. The experiments should have been conducted for multiple runs with different seeds to report mean and standard deviation.
- Ablation should have included the results for AMCNet 2D and AMCNet 3D without Loss_AC to show the impact of simultaneous acquisition of 2D and 3D features.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Dataset used for experiments is not shared.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Authors are encouraged to clearly highlight differences between the proposed and existing similar approaches. The Antomy-constraint loss should be explained in a better way with justification for design choices.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper provides a solution to a difficult problem however there are certain weakness, as mentioned above.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
The authors have partially addressed the concerns, e.g. ablation studies. However the other important issues including significance of advancements over the existing 2D+3D approaches, rationale behind design choices etc. are still not addressed appropriately. I, therefore, keep my score unchanged.
Review #2
- Please describe the contribution of the paper
The paper presents an approach based on deep learning for ultrasound hand bone segmentation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The network architecture and loss function are properly designed based on the requirements and limitations of the problem. The results are interesting and the comparison is acceptable. The paper is very well-written.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The size of the dataset is small, and more discussions on the generalization performance is needed. The authors have only evaluated the proposed method on a limited size dataset Details regarding dataset in terms of imaging setting, subjects, data collection, etc are missing. It is not clear which form of ultrasound data are being processed (i.e., RF or Bmode).
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
As far as the codes/ dataset become publicly available, the results are reproducible. Otherwise, I do not think that others can work on top of their results.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
As mentioned above more discussions on the generalization performance is necessary. Please provide all details regarding dataset in terms of imaging setting, subjects, data collection, etc. It is not clear which form of ultrasound data are being processed (i.e., RF or Bmode).
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The problem of interest is clinically significant, and the proposed method is well designed. However, the submission has to be either a strong technical/methodological contributions of sufficient novelty that may be evaluated on smaller sized datasets, or well designed pipelines/networks that tackle challenging practical problems what are evaluated thoroughly on large datasets.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
I would like to thank authors for answering my questions/ concerns. What I meant by limitation in the size of dataset is the number of subjects not the number of slices/ training samples. As for my second comment. I did not find the authors’ response complete/ convincing.
Anyways, although my comments are still valid, the paper might be considered as a good staring point!
Review #3
- Please describe the contribution of the paper
This paper proposes the use of a 2D+3D convolutional network for the semantic segmentation of children’s hand bones. For the fusion of 2D and 3D information they create the adaptive 2D/3D convolutional Module (ACM) that fuses the information, defuses it and fuses it again. On the other hand, they propose an anatomy constraint loss, which penalizes more the mispredicted pixels outside the anatomical error map (Fe). This Fe is obtained by dilating the label with a kernel of 3, extending the ground truth zone. On the other hand, a cross entropy loss is also calculated, but this time taking into account the intersection of the label and the prediction with the Fe.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
THeir main strengths are: 1.They create a slicer plugin user friendly. 2.They compare with several open source 2D and 3D architectures, including MoNet(2d+3d network). All of those were trained with their public available hyperparameters, what allow the authors to provide a fair comparison.
- During the ablation study they prove the efficiency of their anatony loss on their network but also on the unet architecture. 4.They compute different segmentetaion metrics not olnly dice.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The main weaknesses of this paper are:
- They do not explain how they choose alpha to penalize what they call the anatomy constrain loss.
- They say this method can help in the diagnosis of children’s bone dissorders or detection of problem during growing. But all the metrics are segmentation metrics. Would it be interesting to have some medical metrics.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
They made a plugin in slicer, it is definitely reproducible. Although I don’t know if the architecture code and a tutorial on how to train it will also be made available.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Really nice application. The slicer app make translation to medical application straight forward.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
It seems interesting and directly transfer to the medical field
- Reviewer confidence
Somewhat confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This work is on multi label segmentation of individual hand bones from 3D ultrasound images. It proposes a UNet derived network which uses 2D slices as well as the full 3D volume as separate inputs and mixes features from both these steps via 1x1x1 convolutions, which then are fused for a final segmentation output (Adaptive 2D/3D Convolutional Module - ACM). Moreover it extends the loss by an anatomical constraint term based on a morphological dilation of the target labels, leading to higher penalization of wrong label predictions when the predictions are outside a 1 pixel border of the ground truth labels. Evaluations are performed on an in house dataset of 103 pediatric ultrasound images with manual labeling from a single expert ultrasonographer.
As confirmed by reviewers, the paper is well structured and in most parts relatively easy to follow. A strength seems to be that the proposed pipeline is the first to demonstrate fine grained hand bone segmentation and identification from 3D ultrasound, including 39 categories of bones in the presence of epiphyseal gaps that occur during growth in children and adolescents. Authors provide a 3D slicer plugin that performs segmentation of ultrasound images via a web based service for reproducibility of the software, however, no code is published.
My assessment and comments from reviewers indicate a number of weaknesses of this work, which need to be addressed in a rebuttal:
- The use of 2D and 3D filters from the same dataset has been used in other works. It is not clear how the presented work distinguishes itself from other works in this area.
- The motivation why to use 2D and 3D filters at the same time and mix their responses is not properly motivated for this application. It is not clear why this would bring extra performance, compared to a 3D only approach, because the 2D filters are a subset (i.e. can be expressed if found necessary during training) of the 3D filters. GPU memory could be a reason, however, in the proposed scenario, GPU memory will still be limited since two streams of network weights are coming from the 2D and 3D encoders. Experimentally it is shown that performance is improved using the proposed architecture, however, this could also be an artifact of using standard, published parameters for baseline models, and tuning the hyperparameters of the proposed method much more thoroughly. A strong argument to counter this is going to be needed in the rebuttal.
- The anatomy loss is very unclear in its description. As far as I understand it, it is another penalization of wrongly predicted labels in addition to the Dice loss, but focusing only on regions one pixel apart from the ground truth labels (via the morphological dilation of ground truth labels). This seems to be a very weak anatomical constraint (AC). This weakness is also reflected experimentally, since the added AC loss only brings minor improvements (that may be due to chance) and in some cases even worse performance compared with respective baselines in Tables 1 and 2. Explanation and motivation for the relevance of this loss term have to be given in the rebuttal.
- Results of ablation experiments and baseline experiments are very close and vary a lot between different variants in terms of best performance. Repeating experiments with different random seeds and analyzing statistical significance via non-parametric (ranking based) methods seems to be important to rule out that differences are just due to chance.
- Details on how the data was collected, how the ultrasound device was set up and how the annotations were performed are missing. Ultrasound introduces an operator bias, therefore, it would be necessary to know more exactly if all datasets for example were taken by the same operator and if there was a standardized imaging protocol. Otherwise, it is not possible to judge if one can apply the same method on one’s own data (e.g. via the Slicer plugin).
- Reproducibility of the specific experiments is strongly limited since solely an in house dataset is used in the experiments, no public data (maybe of different anatomy) is shown. Also it is not planned to make available the dataset plus segmentations publicly. Please comment on this aspect and revise if this has changed.
- In the end, the proposed method has small methodological contribution and shows an evaluation on a limited size in house dataset. Authors have to justify in the rebuttal why this adheres to the statement that MICCAI aims to publish either (1) Methods showing high degree of innovation evaluated on small size datasets or (2) Carefully designed pipelines of existing modules for clinically relevant applications that are evaluated thoroughly on large and/or diverse datasets.
Author Feedback
We sincerely thank the reviewers (R1, R2, R3, Meta-R) for professional and constructive comments. We will highlight the motivation, add the mentioned experiments and the data description. 1.The novelty of the Adaptive 2D/3D Convolutional Module (ACM) (Meta-R, R1). A1: Fine-grained segmentation demands a model to maintain the inter-slice anatomical relationships while extracting intra-slice detailed features. 2D filter excels at capturing dense information but lacks of inter-slice information, while 3D filter is the opposite [6]. This motivates us to develop a proper adaptive fusion method. Previous studies used fixed 2D/3D filter layouts to address data anisotropy problem. Differently, our method aims to enhance fine-grained representation. We proposed the learnable ACM, which simultaneously extracts 2D/3D features and dynamically assigns feature weights using an attention mechanism at multi levels. This enables the network to achieve an effective balance between fine-grained and anatomical features.
- The motivation of the anatomy-constraint loss (AC loss) (Meta-R, R1, R3). A2: Class confusion and missing are common errors in fine-grained segmentation, e.g., epiphysis missing, symmetrical confusion. Thus, we proposed AC loss which penalties for all pixels that contribute to class errors in the predicted map. This loss focuses the local errors rather than overall region. Thus, the penalty is confined within the label region by ReLU. Furthermore, considering the issue of unclear boundaries caused by ultrasound noise, we employ a differentiable method to establish an error tolerance, mitigating the impact of subjective labeling factors. As the γ mentioned by R3, we have compared the effect of γ=1 and γ=0.5 for AMCNet. The results of γ=0.5 slightly outperforms the case of γ=1. That’s why we chose γ=0.5.
- Dataset problem (Meta-R, R2). A3: The dataset was collected using device IBUS BE3 with 12MHz linear transducer. The device performs automatic volume scanning without intervention, thereby minimizing operator bias. The dataset consists of 103 3D B-mode ultrasound images. The mean voxel resolution is 0.088×0.130×0.279mm3. The mean image size is 512×1023×609. The data annotation was described in Section 3.1. As for the dataset size, note that our dataset consists of 3D ultrasound data, with each containing more than 500 valid slices, totaling over 50000 slices. In comparison, [14] used 1027 slices and [10] used 562 slices. Moreover, we employed a sliding sampling strategy allowing to obtain over 5000 3D patches. The quantity is sufficient for training a good model and conducting statistically meaningful analysis.
- Performance improvement in Table 2 (Meta-R, R1). A4: We have added ablation experiments of AMCNet 3D and AMCNet 2D without AC loss. In DSC/Jaccard/Recall/F1-score/HD95, the former results are 0.887/0.797/0.787/0.648/2.70, the latter are 0.889/0.800/0.760/0.744/1.10. (1) ACM: In the two comparisons (with or without AC loss), the proposed 2D/3D fusion method is superior to the 2D or 3D method alone, with most metrics improving over 1%. The degree achieved is comparable to the reported of [6] and [14]. (2) AC loss: Based on the added experiments, we have conducted four ablation studies for AC loss (AMCNet 2D, AMCNet 3D, AMCNet 2D+3D, UNet). All the results demonstrated improvements in most metrics, providing statistically evidence for the effect of our method, rather than chance factors.
- Generalization and reproducibility (Meta-R, R1, R2, R3). A5: Since there is currently no similar task in 3D ultrasound, we have tried for multi-class tooth segmentation (dataset in Doi: 10.1038/s41467-022-29637-2). Note that AMCNet improves 2.3%/2.5%/3.6%/1.2% in DSC/Jaccard/Recall/F1-score compared to UNet, showing the generalizability. Although we are temporarily unable to release the dataset due to collaboration policies, our plugin in 3D Slicer will release and the code will be available at https://github.com/BL-Zeng/AMCNet.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This work on fine grained hand bone segmentation from ultrasound seems to be the first one that tackles this task. Methodologically, contributions are an adaptive 2D 3D filter combining network design, and an additional loss function component that penalizes mistaken segmentations far from the ground truth labels of anatomical parts more strongly. It shows a reasonable evaluation and authors promise to make the code publicly available, in addition to implementing the software as a Slicer plugin for the community.
Concerns by reviewers and meta reviewer were about the originality of combining 2D and 3D filters, the confusing description of the anatomy constraint loss and the seemingly minor performance gains compared with state of the art, which were not assessed with respect of their statistical significance. Authors addressed the concerns about the adaptive combination of 2D and 3D filters reasonably well, while the anatomy constraint loss in my opinion is still confusing. Additional ablation experiments and an outlook on a different dataset showed that the method is promising. Statistical signifcance tests were performed, but no quantitative results were given in the rebuttal.
Authors unfortunately did not state how they intend to change the manuscript to address reviewer concerns in case of publication. However, I trust the authors will work thoroughly on (1) including the main issues (except for novel experiments on different data, these are not necessary to be added) in a revised version of the paper, (2) improving the description of the anatomy constraint loss, and (3) doing another thorough proof read to get rid of a number of weak grammar sections.
In summary, all three reviewers agreed in assessing the paper with an acceptance vote. In the light of the above mentioned changes, and given that - even without the publication of the annotated dataset - the code will be made public and there is a plugin for Slicer, I tend to vote for accepting this paper.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
In the rebuttal the authors were able to clear up the issues raised by the reviewers, especially regarding motivation on using 2D and 3D filters, some explanation on the anatomy-constraint loss, data collection, generalization and reproducibility. This allowed the reviewers ratings to reach a consensus of at least weak accept, hence I recommend acceptance of this paper.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
All of the raised issues have been responded to by the authors. The reviews are reasonable and only one review has recommended reject. Based on the issues raised and the responses, I would recommend to accept this paper.