List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Michal Byra, Charissa Poon, Tomomi Shimogori, Henrik Skibbe
Abstract
We propose a novel image registration method based on implicit neural representations that addresses the challenging problem of registering a pair of brain images with similar anatomical structures, but where one image contains additional features or artifacts that are not present in the other image. To demonstrate its effectiveness, we use 2D microscopy in situ hybridization gene expression images of the marmoset brain. Accurately quantifying gene expression requires image registration to a brain template, which is difficult due to the diversity of patterns causing variations in visible anatomical brain structures. Our approach uses implicit networks in combination with an image exclusion loss to jointly perform the registration and decompose the image into a support and residual image. The support image aligns well with the template, while the residual image captures individual image characteristics that diverge from the template. In experiments, our method provided excellent results and outperformed other registration techniques.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43999-5_61
SharedIt: https://rdcu.be/dnwxe
Link to the code repository
https://github.com/BrainImageAnalysis/ImpRegDec
Link to the dataset(s)
https://gene-atlas.brainminds.jp/
Reviews
Review #1
- Please describe the contribution of the paper
1.This paper proposed a novel INR-based framework well-suited to address the challenging problem of gene expression brain image registration. 2.The proposed registration-guided image decomposition mechanism not only improved the registration performance, but also could be used to effectively separate the patterns that diverge from the target fixed image.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper proposes an idea similar to task decoupling, where the moving image M can be decomposed with separate implicit networks, S and R, into two images: the support image MS and the residual image MR. And the support image should correspond to the part of the moving image that contributes to the registration performance.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- From the qualitative comparison results shown in Fig.4, there is no comparison with the latest research works.
- The main task of this paper is close to medical image registration cross different modality. However, this paper lacks the comparison with the multi-modal medical image registration methods.
- In the section on experimental results analysis, the visualization results drawn are not very consistent. For example, in Figure 2, the gray boxes around each image are different.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
1.From the qualitative and quantitative experimental results, I believe that this paper can be reproduced. 2.From the author’s detailed description of the method, it is believed that the paper can be reproduced. 3.In the implementation section (2.4) of the paper, the selection of the environment and method parameters for the experiment is provided, which provides important support for the reproducibility of the paper. Therefore, we consider this paper reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- In the Methods section, for the first loss function (1), explain why both LNCC and NCC are used.
- The experimental results need to be compared with the latest research results and multi-modal medical image registration methods.
- In this paper, the quality of the charts is not good enough, specifically, there are problems with the aesthetics and consistency of the charts. Among them, the quality of the image is inconsistent, and the drawing of the table can be improved. Therefore, there is a need to rethink how to design and draw diagrams to improve the quality and readability of the paper.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The experimental results are not complete.
- The quality of the charts is not high enough.
- Due to the lack of a detailed and structured description of the network architecture for the proposed method by the author, it is difficult for readers to have a clear understanding of the specific algorithmic process.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This paper deals with deformable image registration between images with non corresponding intensity information. To this end, the authors suggest to decompose the moving image into two images, one sharing similar intensity information as the fixed image and which is considered to drive the registration process and one gathering the intensity discrepancies. Implicit neural representation is considered to encode the deformation field and the two images resulting from the decomposition step. The decomposition and registration tasks are formulated as a single optimisation problem and are estimated jointly. The proposed framework is evaluated in the context of 2D microscopy images of in situ hybridization gene expression images of the marmoset brain.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is clear and well written. It addresses a topic of great interest for the miccai community. The proposed solution to handle intensity discrepancies in the registration process is elegant and the use of implicit neural representations in that context is an attractive point. The proposed framework is technically sound and has been rigorously evaluated.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The paper has no major weakness
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper meets the standard criteria of reproducibility
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Concerning the choice of the jacobian-based deformation regularization term, I would have considered |log |J|| rather than |1-|J|| in order to give similar importance to the contracting (0<|J|<1) and the dilating (|J|>1) behavior of the deformation field. Maybe the authors can comment on that choice ?
It would have been interesting to present also visual representation of the deformation field (either jacobian map, module image, or color-coded image of the deformation field) to assess the regularity of the estimated deformations.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The proposed approach is innovative, interesting and versatile. It will potentially give ideas to the MICCAI community to adapt this approach to other kind of issues and/or other applications.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
7
- [Post rebuttal] Please justify your decision
I still think that the paper is of good quality. The authors have made an effort to take into account the comments of the various reviewers. I think it is worth presenting this paper to the Miccai community.
Review #3
- Please describe the contribution of the paper
This paper proposed an implicit neural representations-based method for marmoset brain 2D image registration. Two implicit decomposition networks are proposed to decompose the moving image into a support image and a residual image to solve the domain shift problem and to improve the registration performance.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper is well written and easy to follow. To make the two proposed decomposition networks to separately learn the difference features of the moving image, a decorrelated loss function is introduced.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Although the proposed decomposition networks are provided, from Fig.1 we can note that the intensity values of the support $M_s$ and Fixed $F$ are quite different. It’s hard to support some main claims of this paper. In addition, from the table 1 and 2, we can note that the differences between INRs dec and INRS dec+excl are quite small. It is hard to demonstrate the positive function of the proposed excl loss function.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors mentioned that the codes and data will be available, and the reproducibility of the paper may be acceptable.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
1) In the section of the Introduction, the authors mentioned that the residual image presents features of the ISH image, such as artifacts or texture patterns (eg. gene expression), how to get such a conclusion? 2) From the figures, the moving and fixed images can be considered as two different kinds of images from two different modalities, so the mutual information (MI) metric can be used in ANTs SyN to conduct the registration process. Authors should compare the experimental results of ANTs SyN using MI. 3) What’s the mean of the formula of transformation field $\phi(x)=\overline{x}+\delta{\overline{x}}$, what’s the relation between $x$ and $\overline{x}$, authors should clarify it. 4) Minors: from to 50 gene expression, 360x360 not $\times$. 5) Although the Dice of the proposed method is higher than that of some other methods, the percentage of Jacobian determinations is much higher than the other methods which is very important in the field of medical image registration. Such a limitation should be mentioned in the manuscript.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The decomposed moved support images are still quite different from the fixed images, it is hard to support the claim of “This way the training of the deformation network is guided to provide a more detailed transformation field for the contents of the moving image that accurately…”. The main limitation of lacking diffeomorphic didn’t describe in this paper.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
Although there are some minor weaknesses, the response largely resolve my concerns.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper presents an implicit neural representation based registration method for gene expression images of the marmoset brain. The image registration is linked to image decomposition task, where ISH images are decomposed into support and residual images. The proposed method is interesting and technically sound; however, certain points need to be clarified:
- Provide details of the network architecture.
- Revise figures for clarity and consistency.
- Comment on the choice of the Jacobian-based deformation regularization term.
- Statistical analysis should be added to check whether the results are statistically significant.
- Explain why both NCC and LNCC are used?
Author Feedback
We thank the Reviewers (R1-R3) and the Meta-Reviewer (MR) for the constructive and insightful comments (C). S – review section. Addressing reviewer feedback, we’ve enriched our appendix with more details about the network architecture, rectified figure issues, clarified our choice of the Jacobian regularization term, described the results of statistical tests, and discussed the use of NCC and LNCC.
MR/C1, R1/S8/C3: We’ve incorporated detailed text and illustrations regarding the SIREN model and the Fourier mapping into the appendix. Implementations will be available in our Github repository.
MR/C2, R1/S3/C3: We thank you for noticing the “box” issues in Fig. 2. We’ve revised the figures.
MR/C3, R2/S6/C1: We thank you for the suggestion. Our approach was inspired by the two previous studies on INRs-based registration methods [12,14]. We used the Jacobian regularization term from [14], which encourages local deformations and positive determinants, and it worked well in our study. However, R2 is right, $|log |J||$ may be a better choice; we plan to investigate this in the future.
MR/C4, R1/S8/C1: We used the t-test (alpha=0.05) to assess the results. Dice scores for our approach (dec/dec+excl) were significantly higher for 4 out of 5 regions compared to regular INRs. SSIM values were significantly higher for our approach in all cases. For the determinant-related score, there were no significant differences between the INRs dec and the regular INRs, but the metric values were significantly higher for the INRs dec+excl. These findings confirm the effectiveness of our method and have been added to the tables and the Results section (+2 sentences).
MR/C5, R1/S6/C1: Previous INR registration methods used either the NCC [12] or a combination of NCC and LNCC [14]. In [14], for 3D image, both small and large coordinate patches were sampled for loss calculations. In our 2D case, we used all pixel coordinates as input (as in the SIREN paper [11]). We found that combining both losses led to the best results. The NCC seems to stabilize training, while the LNCC yields good local registration results. While we do not claim novelty in this combination, given the previous works [12,14], we’ve added additional text to better clarify our approach (+2 sentences).
R3/S3/C1, R3/S8/C1: In Fig. 1, although the objective is that the support image resembles the fixed image, the intensity values of these two images can differ because the decomposition is constrained by the reconstruction loss, and the cross-correlation loss, which is insensitive to the scaling of the pixel intensities. In our work, we did focus on the decomposition method and did not investigate the diffeomorphism, but this can be included in our framework.
R3/S3/C2: We agree that INR dec and INR dec+excl achieved similar scores according to Tables. However, our qualitative results, Fig. 2 and 3, show that the inclusion of the excl loss resulted in more visually plausible image decomposition, which we plan to exploit in future extensions of our method (e.g. detection of artifacts).
R3/S6/C1: In our study, ISH images include brain tissue, gene expressions, and imaging artifacts. During registration, the optimization enhances crucial brain tissue parts, and moves non-essential components for registration to the residual image. This conclusion is based on our qualitative results (Fig. 2 and 3).
R1/S6/C2, R3/S6/C2: We apologize for the lack of clarity. ANTs SyN used in our work utilized the MI metric (default SyN metric in ANTsPY), which we now highlight in the paper. We compared the INRs to methods that can serve as off-the-shelf registration techniques. Due to the lack of space, a more detailed comparison of the different methods will be more suitable as future work.
R3/S6/C3: We apologize for this typo, $x$ should be changed to $\overline{x}$. We’ve proofread the manuscript.
R3/S6/C5: We’ve highlighted that the determinant related score was higher for the INRs (+1 sentence).
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors addressed all the concerns in the rebuttal; therefore, I recommend accepting this paper.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
After rebuttal, the reviewers agree on acceptance. Importantly, the review that had recommended rejection has now improved.
The field of implicit neural representations is quite rich, and the reviewers had several concerns throughout the review process. The rebuttal has helped but I encourage the authors to address these points in the camera ready to help the paper carry out a more productive conversation at MICCAI.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper has novelty which weighs over weakness and presents a sufficient contribution to the MICCAI community. The rebuttal clarified the details of the proposed architecture for image decomposition and registration, experimental setting, and evaluation metric.