List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Zuopeng Tan, Hengyu Zhang, Feng Tian, Lihe Zhang, Weibing Sun, Huchuan Lu
Abstract
Segmentation-assisted registration models can leverage few available labels in exchange for large performance gains by their complementarity. Recent related works independently build the prediction branches of deformation field and segmentation label without any information interaction except for the joint supervision. They ignore underlying relationship between the two tasks, thereby failing to fully exploit their complementary nature. To this end, we propose a ProGressively Coupling Network (PGCNet) that relies on segmentation to regularize the correct projecting of registration. Our overall framework is a multi-task learning paradigm in which features are extracted by one shared encoder and then separate prediction branches are built for segmentation and registration. In the prediction phase, we utilize the bidirectional deformation fields as bridges to warp the features of moving and fixed images to each other’s segmentation branches, thereby progressively and interactively supplementing additional context information at multiple levels for their segmentation. By establishing the entangled correspondence, segmentation supervision can indirectly regularize registration stream to accurately project semantic layout for segmentation branches. In addition, we design the position correlation calculation for registration to easier capture the spatial correlation of the images from the shared features. Experimental results on public 3D brain MRI datasets show that our work performs favorably against the state-of-the-art methods.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43999-5_59
SharedIt: https://rdcu.be/dnwxc
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The authors propose a coupled segmentation and registration method that demonstrates state-of-the-art performance on two brain datasets (OASIS, IXI), which seems to be mainly due to the proposed deep network architecture.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is easy to understand, the main ideas are clearly presented, and the experiments show the state of the art. The ablation experiments help to understand where the performance gains come from.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
I have two concerns with the paper:
-
The network architecture of reference 8 seems to be very closely related to the proposed work, but is not included in the comparison. The authors mention that the source code is not available, which is false, as a short search on GitHub shows (https://github.com/TheoEst/coupling_registration_segmentation).
-
The experiments are performed with two MRI datasets for the brain, which makes it difficult to assess whether the framework is suitable for general resgisration tasks (although there are no components designed only for brain MRI datasets).
-
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors use publicly available datasets for their experiments and promise to release training and test code. Reproducibility should therefore be given.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The inclusion of closely related work in a comparison is very important to assess the contribution of the proposed method. Not having source code should not be an exclusion criterion for this. In particular, for reference 8, an implementation of the network architecture based on the paper would be straightforward (as noted above, the source code is actually even available).
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I think the comparison to a closely related work is missing, so I cannot conclusively evaluate the paper’s contribution. If the comparison was included, I would gladly change my recommendation to accept.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
As suggested the authors included comparison results of closely related work, so as stated i will change my rating to weak accept. However, the results of the authors are significantly better with a really similar network architecture which makes the evaluation a little unbelievable and me uncertain about acceptance. If accepted I would expect a short statement/discussion of the authors in their paper why/which component makes their method so much better than reference 8.
Review #2
- Please describe the contribution of the paper
The authors propose a learning framework for joint registration and segmentation. The tasks use a shared encoder and the authors introduce a coupling decoder to share semantic information between both tasks. As warped feature maps are used for cost volume calculation in the intermediate steps, they also propose a position correlation calculation. The framework is tested on MRI brain scans and compared to multiple state-of-the-art registration methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Using the warped encoder features of moving and fixed image in the respective other decoder branch is an interesting idea to strongly couple registration and segmentation task. It is a novel approach to combine both tasks with shared information instead of just linking the tasks with a shared loss function. The evaluation is extensive and the results are very good.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The presented registration approach refers to the registration of query images to five pre-selected atlas images. To assess the registration applicability, it would be interesting to test the method for the registration of random pairs. The paper does not include discussion on remaining limitations/improvement ideas.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The data sets are publicly available. The results should be reproducable if the code is made public and if some paramaters, such as the weights of the different loss components, are added.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
-
Please include a statistical test for comparison of metrics to those of other methods, especially for similar performaing methods like PC-Reg. Also state the statistical relevance of improvements presented in Table 2/3 for you ablation study.
-
The few-shot aspect of the method is not discussed sufficiently. What impact does the number of utilized labels have on the final model performance?
-
I assume that the evaluation was done on all 35 (44) label categories, but please state on which labels you evaluate the method to emphasize the difference between used number of labels during training and inference.
-
Please provide information on training and inference time.
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The method is novel and was evaluated thoroughly, showing good results compared to many other registration methods. The few unclarities could be improved by some reformulations.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper utilize a ‘shared encoder-dual decoder’ architecture to fully exploit complementary of the registration and segmentation. The registration provides the segmentation with training data, and the segmentation regularizes the correct projecting of registration. The experimental results on two public datasets show the promising performance of the proposed method.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This paper is well-written, and the motivation is clear.
- The results look good and it seems that the method outperforms other methods based on joint training of registration and segmentation.
- The ablation study is important and clearly demonstrate the role of each part of the proposed method.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Given the correlation of the two features, the details of computing the forward and inverse deformation field are missing.
- Why only calculate the forward similarity loss rather than the bidirection similarity loss?
- Do the author only use the labeled atlas as the moving image? Why not try to add the weakly supervised loss to train the registration part? That is, two images are randomly selected from 5 atlases as moving and fixed image to train the registration decoder. This may further improve the registration accuracy.
- The author should compare their method with at least one method based on ‘one-two models’.
- The statistical analysis is necessary to demonstrate the effectiveness of the method.
- The description of Figure 4 is missing.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Lacking sufficient implementation/code details. The authors could make their source code publicly available if this paper is accepted.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Please address the issues mentioned in “Weaknesses of the paper”.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This is a well-written paper with a clear motivation. The idea seems interesting and the results is promising. However, there are some major issues for the authors to address during the rebuttal phase.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
My concerns have been well addressed, and I recommend accepting this paper.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Overall, the paper is borderline, with the more thorough reviews (1 and 3) expressing quite a bit of concern.
It will be important for the authors to discuss, in the rebuttal, the similarity with the method pointed out with R1, and the lack of comparison, as this seems like an important omission. R2 find the work novel, but it seems like an important (related) piece of literature may have been missed. R3 brings several concerns to be addressed, including this comparison issue.
Author Feedback
We thank the reviewers for their helpful comments. Below we clarify questions from every individual reviewer. R#1Q1: Comparison with ref 8. Reply: We indeed didn’t find its code before. Its results are: On OASIS, DSC=77.83, HD95=2.16, |J<0| =0.020; On IXI, DSC=67.76, HD95=3.70, |J<0|=0.019. Our work significantly outperforms this method. We will include these comparisons in the final paper. R#1Q2: Only two brain MRI datasets. Reply: Our research originally focuses on brain MRI and uses two main brain datasets. It is necessary and important to extend the experiments on other datasets for generalization validation. Due to the limited time, we will add these results in the supplementary materials. As you noted, there is no specialized design for the brain, we believe the proposed method can work well.
R#2Q1: The statistical test. Reply: Thanks for your suggestions. We do the two-sided t-test as follows: The degrees of freedom are set to 6 (four runs with different seeds) and alpha to 0.05. Due to the limited time, we only finish the p-value (PGCNet vs. PC-Reg) on OASIS. The results indicate a significant advantage for our method with a p-value of 1.2e-6 for DSC metric and 5.9e-4 for HD95 metric. More results against other methods and the related ablation study will be provided in the final paper. R#2Q2: The impact of the number of labels. Reply: When labels are few, the performance is sensitive to them. However, once a certain number of labels are reached, its fluctuation becomes small. Specifically, when the number is 1, 3, 5, 10, 20, and 50, the Dice coefficient is 83.13, 85.92, 87.72, 89.50, 90.06, and 91.02, respectively. R#2Q3: Label categories during training and inference. Reply: For OASIS dataset, the categories are always 35. However, for IXI dataset, the same regions in the left and right brain are combined during training, which has 26 categories. During inference, all 44 categories are utilized. R#2Q4: Training and inference time. Reply: We require approximately two days of training on a single RTX 3090. On OASIS dataset, our inference time is 1.08 seconds, which is faster than PC-Reg (1.51 seconds).
R#3Q1: Details of computing the forward and inverse deformation field. Reply: At each level, the forward and inverse field estimators, which share parameters, use three residual blocks to estimate the forward and inverse increments of deformation fields based on two different correlation maps. The resulting increment of deformation field is added to the previous deformation field to obtain a new field for the next level, as illustrated in Fig. 3. R#3Q2: Why not calculate bidirectional similarity loss? Reply: In previous experiments, it was observed that bidirectional similarity loss led to an insufficient forward deformation and decreased the accuracy. Our model is structured symmetrically, with shared parameters for the bidirectional field estimators at each level. The forward similarity loss guides the forward field estimator to predict the deformation field from the correlation map of warped moving features and fixed features. The inverse field estimator automatically learns to predict the inverse deformation field from the correlation map of warped fixed features and moving features. The two processes are equivalent. Therefore, our model can be effectively implemented without the bidirectional similarity loss. R#3Q3: Do the author only use labeled atlas…? Reply: Yes, our study only utilizes labeled atlas as fixed images. Your suggestions inspire us to further explore the training strategy. R#3Q4-Q5: About the comparison with one-two model and the statistical analysis. Please refer to the reply for R#1Q1 and R#2Q1, respectively. R#3Q6: The description of Figure 4. Reply: We will add the related description in the final paper.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
After the rebuttals, the reviewers re-evaluated their scores and agree on acceptance. However, they also note that there’s a couple of things that the authors need to address by the camera ready, as promissed. In particular please note the R1 comment: “If accepted I would expect a short statement/discussion of the authors in their paper why/which component makes their method so much better than reference 8.”.
Overall there is agreement that the paper is sufficiently interesting and should be discussed at MICCAI. Congratulations to the authors.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
All reviewers found some merit in the proposed coupling of segmentation and registration and despite being similar to prior work (Estienne, Vakalopoulou et al) the specific coupling mechanism (which is not trivial) leads to improvements that are rather substantial. As reviewer #1 puts it almost to good to be true ;-) Since the source code will be released and I trust the authors will include some further statements of why the module performs so much better than prior work, I recommend acceptance.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The sufficient level of novelty of the proposed coupling segmentation and registration was recognized by reviewers. The test cases of the various numbers of labels available were investigated well. The rebuttal addressed other major concerns were addressed (if not all).