List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Muhammad Osama Khan, Yi Fang
Abstract
3D signals in medical imaging, such as CT scans, are usually parameterized as a discrete grid of voxels. For instance, existing state-of-the-art organ segmentation methods learn discrete segmentation maps. Unfortunately, the memory requirements of such methods grow cubically with increasing spatial resolution, which makes them unsuitable for processing high resolution scans. To overcome this, we design an Implicit Organ Segmentation Network (IOSNet) that utilizes continuous Implicit Neural Representations and has several useful properties. Firstly, the IOSNet decoder memory is roughly constant and independent of the spatial resolution since it parameterizes the segmentation map as a continuous function. Secondly, IOSNet converges much faster than discrete voxel based methods due to its ability to accurately segment organs irrespective of organ sizes, thereby alleviating size imbalance issues without requiring any auxiliary tricks. Thirdly, IOSNet naturally supports super-resolution (i.e. sampling at arbitrary resolutions during inference) due to its continuous learnt representations. Moreover, despite using a simple lightweight decoder, IOSNet consistently outperforms the discrete specialized segmentation architecture UNet. Hence, our approach demonstrates that Implicit Neural Representations are well-suited for medical imaging applications, especially for processing high-resolution 3D medical scans. As open science, we make our code publicly available to aid future research in this area: https://github.com/osamakhaan/iosnet.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16443-9_42
SharedIt: https://rdcu.be/cVRyW
Link to the code repository
https://github.com/osamakhaan/iosnet
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper adopts a continuous implicit neural representations for medical image segmentation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The training speed is accelerated. The proposed algorithm is suitable to solve problems which are computationally heavy, like super resolution image processing.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
The contribution of the paper is not clear since the literature review lacks the comparison with other similar algorithms like the mentioned reference [2, 14]. Are there any other algorithms?
-
What is the difference between the classic discrete segmentation algorithms with a sampling mechanism and the proposed algorithm?
-
More related algorithms should be compared to illustrate the effectiveness of the proposed algorithm.
4, What is the shortcoming of the proposed algorithm?
-
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The effectiveness of the influence of sampling hyper-parameters is not analyzed, it could affect the reproducibility of the algorithm.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Please check part 5 for more detailed information.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The contribution of the paper is not clear. The important reference discussion is missing.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #2
- Please describe the contribution of the paper
This paper propose a new segmentation method via using implicit neural representation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is well organized and it is easy to follow. Combine different resolution features to make segmentation prediction.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
For segmentation task, the method is not novel. UNet has similar characteristic.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Yeah, it looks reproducible
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- The motivation looks like not reasonable. Compared with typical UNet, the method also still uses different resolution features without decoding layer-by-layer. It also needs big memory resource. Moreover, use some FC layers to incur more parameters.
- The baselines are not medical segmentation related methods.
- IF-Net has big advantages in reconstructing and smoothing. The work follows it, but doesn’t show the advantages in segmentation task.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The novelty is limited. The motivation isn’t sufficient and reasonable.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
This paper proposed a novel implicit organ segmentation network (IOSNet) that utilizes continuous implicit neural representations (INRs) to achieve memory-efficient and high-resolution medical image segmentation. Benefiting from the introduction of INRs, the presented IOSNet showed significantly smaller memory footprints and much faster convergence speed than the conventional fully convolutional networks. Experimental results on a clinical head and neck CT dataset demonstrated the superior performance and effectiveness of the proposed method in dealing with the 3D organ segmentation task, especially for small targets.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This study is well-motivated by a practical but longstanding problem, i.e., the trade-off between the image resolution and the memory footprint of the deep networks for image segmentation.
- The application of the INRs in the medical image segmentation task is novel and interesting and has been demonstrated well-suited for solving the above problem.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- An analysis of the inference efficiency of the proposed method is required but missed in the paper. According to the introduction of the proposed method, its inference efficiency is highly correlated with the resolution of the output image. The trained model may need to perform a large number of forwarding passes through the decoder to get the segmentation labels for each output pixel, which could be a time-consuming procedure. However, the author did not provide any analysis on this issue.
- The size of the testing set is too small (10 cases) to make the experimental result convincing. The author used a large training set (261 cases) to train the proposed network but evaluated it using 10 cases, which makes the experimental results less convincing. The author is suggested to rebalance the size of the training set and the testing set.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The source code is available according to the author’s answer to the reproducibility checklist.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- Section 2.1, “we learn a continuous segmentation (occupancy) function si for each organ i \in {0, 1, . . . ,R} where R is the number of OARs.”: Do we need to learn a continuous segmentation function for the background? If not, the value of i should start from one, not zero.
- Section 2.1, Equation 3: Is the Dice loss calculated inside the sigma symbol? If so, the loss value could be very large. Do we need to normalize it to a fixed range such as [0, 1]?
- Section 2.1, “s_i are the predicted organ occupancies and ground truth segmentations”: “ground truth” -> “ground-truth”
- Section 2.2, “The data sampling parameters are K=4000, L=4, and M=5000.”: What’s the unit of L? It is better to use another variable to indicate the padding size since L has been already used for the loss function in Equation 3.
- Table 1 (left): Why do the upper and lower parts of the table list different sampling strategies? Why not evaluate all the four sampling strategies in both the upper and lower parts of the table? And the inconsistent number of sampling points of different sampling strategies may also make the comparison unfair.
- Section 3.4, “Although boundary sampling segments well around organ boundaries (see Table 1 (left)) …”: This conclusion is hard to see from the number reported in Table 1.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Overall, this paper is well-written and easy to follow. The study is well-motivated by a practical problem and the proposed solution is also novel and effective. However, I still have some concerns regarding the inference efficiency of the proposed method and the small size of the testing set (see main weakness list), which should be well addressed before I can suggest the acceptance of this paper.
- Number of papers in your stack
6
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
7
- [Post rebuttal] Please justify your decision
Both of my two major concerns (one about the efficiency, one about the small testing set size) have been well-addressed according to the author’s response. For the issue of efficiency, it can be improved by applying some optimized sampling strategies such as Multiresolution IsoSurface Extraction [15]. For the issue of small testing set size, the author also promised to provide additional analysis with different train/test splits in the final version. After the revision promised by the author, I think the merits of the proposed method outweigh its weakness. I have no more comments but a suggestion of acceptance on this paper (I would like to revise my score from 5: weak accept to 7: strong accept).
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposes a segmentation network (IOSNet) that makes use continuous implicit neural representations (INRs) to achieve memory-efficient and high-resolution medical image segmentation.
Strengths:
- smaller memory footprints and much faster convergence speed than conventional fully convolutional networks is shown on the HaN dataset
- The application of the implicit neural representations in the medical image segmentation task is novel and interesting
Weaknesses:
- the contribution should be highlighted
- the cost at inference should be discussed
Rebuttal: Based on the reviewers comments, it is kindly advised that the authors:
- Present an analysis of the inference efficiency of the proposed method (cf Rev3)
- Discuss the limitations of the proposed approach (rev1)
- Answer the following questions:
1) From Rev1: The contribution of the paper is not clear since the literature review lacks the comparison with other similar algorithms like the mentioned reference [2, 14]. Are there any other algorithms?
2)From Rev1: What is the difference between the classic discrete segmentation algorithms with a sampling mechanism and the proposed algorithm?
3)Questions by Rev3 in his/her section « detailed and constructive comments for the authors » - Reply to Rev2 comment: « The motivation looks like not reasonable. Compared with typical UNet, the method also still uses different resolution features without decoding layer-by-layer. It also needs big memory resource. Moreover, use some FC layers to incur more parameters. »
In addition, the authors are invited to add any other comment they judge relevant.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2
Author Feedback
We thank reviewers for the constructive feedback and appreciate that you found our work “well-motivated by a practical but long-standing problem” (R3). “The application of implicit neural representations in the medical image segmentation task is novel and interesting” (R3,MR) which allows us to have “smaller memory footprints and much faster convergence speed than conventional fully convolutional networks” (MR).
Inference Efficiency/Limitations (R1,R3): During inference, the computation of the encoder feature maps is done only once for the entire scan. Also, for the decoder itself, which is relatively lightweight, the inference process is parallelized since we compute segmentations for multiple voxels in the same forward pass. For reference, we observe that even with an unoptimized naive implementation, inference requires at most 8 forward passes on the largest CT scan. For a test scan of size [80,200,152], this naive inference takes 3.28s on IOSNet in contrast to 0.31s on UNet. Although IOSNet has higher inference cost, it allows us to train at much higher resolutions than that possible with fully convolutional approaches (UNet) due to memory constraints. Moreover, during inference, we can generate segmentation maps of any desired (potentially infinite) resolution due to the continuous representations (Sec. 3.3). Lastly, for faster inference, we can leverage optimized algorithms such as Multiresolution IsoSurface Extraction [15], which does not need to densely evaluate at all points of a high-resolution grid since it incrementally builds an octree (i.e. starting from a lower resolution, it densely evaluates only blocks that are at organ boundaries). To summarize, we believe that the IOSNet benefits (continuous representations, memory efficiency, faster convergence, super-resolution) far outweigh the minor reduction in inference efficiency and we will provide further analysis by integrating faster inference algorithms in the final version.
Size of Testing Set (R3): Consistent with prior works (AnatomyNet [28], FocusNet [8], SOARS [9]), we used the same test set (10 cases) from the 2015 MICCAI Head and Neck challenge. Furthermore, as requested, we will provide additional analysis with different train/test splits in the final version.
Comparison with [2,14] and Discrete Segmentation Algorithms (R1): We present a comparison with BAENet [2] in Sec. 3.4 (Table 1), where IOSNet significantly outperforms [2]. On the other hand, in [14], the underlying SRN is pre-trained for novel view synthesis using posed RGB images and camera intrinsics, which is not directly translatable to the medical domain. In contrast to discrete segmentation algorithms with a sampling mechanism, our approach learns continuous representations (i.e. we can evaluate the learnt function at any continuous point instead of at discrete voxels), which has several benefits (memory efficiency, faster convergence, super-resolution) (Secs. 3.1-3.3).
R2: Whereas IOSNet also uses different resolution features similar to UNet, this is not the main contribution of our work. Instead, IOSNet learns continuous segmentation functions (Sec. 2.1) in contrast to UNet which outputs discrete segmentation maps. Moreover, the IOSNet decoder is lightweight and IOSNet is significantly more memory efficient than UNet (Sec. 3.1, Fig. 2). We use UNet as a baseline since it is competitive with SOTA HaN methods without using any auxiliary tricks (Sec. 3 1st paragraph). Inspired by the fine-grained representations in IF-Nets, we propose the first 3D medical segmentation method based on INRs that has the several benefits highlighted above.
Detailed Comments (R3): Empirically, we did not observe a huge difference w/ or w/o a segmentation function for the background. Yes, the Dice loss is in fact normalized by the number of sampled points. L has units of pixels. Thank you for the Table 1 suggestions. We will restructure the sampling ablation results so that the conclusions are easier to see.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
In this paper the authors leverage the recent advance in neural implicit representation to advance on the practical problem of memory efficiency in segmentation networks.
In the rebuttal, the authors were able to respond to the reviewer’s questions on the inference efficiency, the limitations, the comparison to SotA, the size of the test set and several implementation questions.
Thanks to the reviewer’s suggestions, I believe that the paper has now been improved and should be accepted at MICCAI.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
4
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
I very much appreciate the general approach of this work, i.e., using implicit neural functions as part of semantic segmentation. I think this general approach has great potential and these ideas are indeed something that should be explored in greater deal by the MICCAI community.
However, there are several issues that I feel were not sufficiently addressed in the original submission or the rebuttal:
Validation: authors are correct to note that comparisons against Unet are informative, since they are using the same encoder. Even so, authors should still report the MICCAI challenge results from the leaderboard, so readers can understand where the method performance sits compared to the SOTA. As it happens, the SOTA still does outperform the authors’ approach. This is ok, given all the bells and whistles that these other approaches uses. But authors still need to own these results and contextualize them for the reader, and make the case for their method and results in full light.
Limitations: Authors do need to explain when and where their approach can be best used. Since they concatenate the point coordinates to the features, they are implicitly assuming shapes are roughly rigidly aligned. Their H&N experiments use datasets that are indeed roughly aligned. What about other situations where rigid pose has great variability?
As a result, I recommend accept, but at the borderline level.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
9
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors’ addressed reviewers’ concerns regarding the efficiency and also explained the difference of their approach with existing methods, which better delineates the methodological novelty and contribution. The idea of using a lightweight network with continuous segmentation functions to generate segmentations is novel. The paper is also fairly well written with reasonable comparisons, although some lingering concerns regarding method limitations as requested by reviewers could have been addressed.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5