Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews Back to top
List of Papers By topics Author List
Paper Info Reviews Meta-review Author Feedback Post-Rebuttal Meta-reviews

Authors

Amaury Leroy, Alexandre Cafaro, Grégoire Gessain, Anne Champagnac, Vincent Grégoire, Eric Deutsch, Vincent Lepetit, Nikos Paragios

Abstract

Multimodal 2D-3D co-registration is a challenging problem with numerous clinical applications, including multimodal diagnosis, radiation therapy, or interventional radiology. In this paper, we present StructuRegNet, a deep-learning framework that addresses this problem with three novel contributions. First, we combine a 2D-3D deformable registration network with an adversarial modality translation module, allowing each block to benefit from the signal of the other. Second, we solve the initialization challenge for 2D-3D registration by leveraging tissue structure through cascaded rigid areas guidance and distance field regularization. Third, StructuRegNet handles out-of-plane deformation without requiring any 3D reconstruction thanks to a recursive plane selection. We evaluate the quantitative performance of StructuRegNet for head and neck cancer between 3D CT scans and 2D histopathological slides, enabling pixel-wise mapping of low-quality radiologic imaging to gold-standard tumor extent and bringing biological insights toward homogenized clinical guidelines. Additionally, our method can be used in radiation therapy by mapping 3D planning CT into the 2D MR frame of the treatment day for accurate positioning and dose delivery. Our framework demonstrates superior results to traditional methods for both applications. It is versatile to different locations or magnitudes of deformation and can serve as a backbone for any relevant clinical context.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43999-5_73

SharedIt: https://rdcu.be/dnwxq

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

  • Please describe the contribution of the paper

    This paper proposes a novel method for registration of 2D to 3D volumes using previously defined structures. The method has been validated on the very challenging problem of histopathology to CT.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    Very clear clinical motivation for this work.
    Paper is well written Results in Figure 3 and Table 1 are impressive

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    As the authors point out the method relies on tissue with rigid structures, which may exclude some clinical use cases (eg. prostate). Method appears to currently require manual contours of rigid structures

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    No mention of ethics or patient informed consent.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

    Please clarify the (typical) original size of the whole slide imaging. Can you provide more details of the resampling method used?

    • Section 4.1: the three samples are not shown in Figure 1. The generated sCTs are shown but not the histology.
    • Fig3. (f) does tumour contour = GTV? Please be consistent.
    • Table 1: mention that these results are for the cartilage, not the GTV.
    • Section 5: “ease of manipulation” for the prostate seems out of place and is confusing
  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    6

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Very nice work, addressing an important and very challenging clinical problem

  • Reviewer confidence

    Confident but not absolutely certain

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #3

  • Please describe the contribution of the paper

    The paper introduces StructuRegNet, a deep-learning framework for multimodal 2D-3D co-registration. It combines a 2D-3D deformable registration network with an adversarial modality translation module and leverages tissue structure to handle out-of-plane deformation. The framework does not require preprocessing or 3D reconstruction and demonstrates superior results for head and neck cancer.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. The StructuRegNet proposed in the paper has significant clinical applications for multimodal 2D-3D registration, which is a challenging problem with various medical implications.
    2. The paper showcases improved results compared to existing methods for head and neck cancer and mapping 3D planning CT to 2D MR frame, demonstrating its potential in radiation therapy for precise positioning and dose delivery.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    1. The use of image-to-image translation networks for multi-modal image registration has been explored in prior work.
    2. The evaluation of the cascaded rigid initialization step could have been more thorough.
    3. It appears that manual cartilage segmentation masks are necessary for each run, which could limit scalability.
    4. The training process for the pairwise registration network could be better explained in the main paper.
    5. While Table 1 shows some improvement over MSV-RegSynNet, it is not a significant improvement.
  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    The paper does not have high reproducibility due to a lack of publicly available code and a lack of detailed information on the image registration network.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
    1. The efficacy of the proposed registration framework should be assessed by comparing its performance with and without the use of image-to-image translation.
    2. It is essential to evaluate the accuracy of slice-to-slice correspondences in the registration framework to ensure reliable alignment.
    3. The description of the registration network should be moved to the main text for better accessibility and understanding.
    4. A clear and detailed description of the method used to reconstruct the histology volume and ensure consistency of 2D histology slices in 3D should be provided in the paper.
  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    3

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The technical novelty of this paper is lacking. I suggest the authors improve their paper by (1) showing the benefits of using image-to-image translation (2) automating cartilage segmentation.

  • Reviewer confidence

    Confident but not absolutely certain

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #2

  • Please describe the contribution of the paper

    This paper proposes a 2D-3D multimodal registration method that leverages structural information through recursive cascaded plane selection and a distance field-based regularization is proposed. Two types of data are evaluated, and the ablation studies show the efficacy of the proposed modules.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. The clinical problem is challenging and of great importance.
    2. The Recursive Cascaded Plane Selection module is novel and effective.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    1. The results in Table 1 do not seem consistent. On the first dataset the performance gain is huge, but on the second dataset the performance is increased by a tiny bit. There is no statistical significance test, so I am not convinced by either the effectiveness of the distance field regularization (Dice from 85.1 to 86.9) or the proposed method on the CT/MR dataset (Dice from 84.6 to 84.8).

    2. CycleGAN part is not detailed enough. Even from the supplementary materials, I cannot know how to select the best epoch during training.

    3. The paper is not easy to follow. The description for the method part (e.g., recursive plane selection) is confusing. Some important information should have been provided in the main manuscript but put in the supplementary materials instead.

    4. Apart from the Recursive Cascaded Plane Selection, I am not sure if there are other differences between the proposed method and the MSV-RegSynNet (Ref. [14]).

  • Please rate the clarity and organization of this paper

    Poor

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    The authors will not make code and data publicly available.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

    1. I would suggest performing statistical tests given the small test set and small improvement on some experiments.

    2. Please discuss the differences between the proposed method and the MSV-RegSynNet (Ref. [14]).

    3. Please make the main manuscript more complete instead of putting some important information in the supplementary materials.

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    5

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The proposed method tries to solve a challenging problem and works well in the H&N CT/WSI setting. However, the paper writing needs to be improved and there is no statistical test which makes it hard to compare on the CT/MR dataset.

  • Reviewer confidence

    Somewhat confident

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Primary Meta-Review

  • Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The paper presents a neat combination of strong clinical motivation, a new solution for a relevant problem (histopathology to CT registration) with an overall good description. While the comments were a little bit short on technical details, all reviewers praise the clinical relevance and overall good results. The method is deemed somewhat incremental wrt the prior work “MSV-RegSyn-Net” from last MICCAI and the differences on the private CT/MR dataset of the two approaches are not significant. But I found that the recursive plane selection provides a useful technical solution for 2D / 3D registration - in combination with structure guidance (cartilage segmentation). The reviewers provided detailed comments that could further improve the clarity of the final paper. In particular the use of MIND within CycleGAN could be made clearer.
    One disadvantage would be the quite limited reproducibility in case the dataset remains private and the source code is not published (interestingly the authors are thankful for other code/data that was made available to them but report “no” to all questions regarding the release of implementation detail in the respective questionnaire).



Author Feedback

N/A



back to top