List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Songhui Diao, Wenxue Zhou, Chenchen Qin, Jun Liao, Junzhou Huang, Wenming Yang, Jianhua Yao
Abstract
Multispectral imaging has a broad, promising and advantageous application prospect in the diagnosis of skin diseases. However, there are inherent deviations such as rigid or non-rigid deformation among multispectral images (MSI), which makes accurate and robust registration algorithms desirable to extract reliable multispectral features. Existing registration algorithms are susceptible to significant and nonlinear amplitude differences and geometric distortions among MSI, resulting in an unsatisfactory estimation of the registration field (RF). In this study, we propose an end-to-end multispectral image registration (MSIR) network with unsupervised learning for human skin disease diagnosis. First, we propose a basic adjacent-band pair registration (ABPR) model to obtain the corresponding RFs through simultaneously modeling a series of image pairs from adjacent bands. Second, we introduce a multispectral attention module (MAM) for extraction and adaptive weight allocation of the high-level pathological features of multiple MSI pairs. Third, we design a registration field refinement module (RFRM) to rectify and reconstruct a general RF solution. Fourth, we propose an unsupervised center-toward registration loss function, combining a similarity loss for features in the frequency domain and a smoothness loss for RF. In addition, we built a MSI dataset of multi-type skin diseases and conducted extensive experiments. The results show that our method not only outperforms state-of-the-art methods on MSI registration task, but also contributes to the subsequent task of benign and malignant disease classification.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43999-5_68
SharedIt: https://rdcu.be/dnwxl
Link to the code repository
https://github.com/SH-Diao123/MSIR
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The authors introduce a novel method for registration of multi-spectral imaging, a useful for diagnosis of skin diseases with the goal to overcome some of the current limitations of the task, such as large geometric and intensity distortions between different frequency bands.
The presented approach is based on a group-wise deformation field estimated as the refinement of the pairwise registration of adjacent frequency bands. To compute such registration, an attention module is embedded to the model to weight the contribution of each image in the final deformation field and is optimised end-to-end using a similarity function on the frequency domain.
The authors provide a nice ablation study of several design choices, show the improvements of their approach as compared to several competitors and finally show the benefit of using the method in a downstream classification task.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Large dataset of MSI imaging, including 22 frequency bands
- End-to-end optimisation of pairwise and groupwise image registration.
- Novel formulation to estimate a groupwise registration based on the refinement (concat + 3D conv) of the pairwise deformation fields.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The model is tested in a single institution dataset.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- The data used is not publicly available, making it hard both to reproduce and compare in future studies.
- The architecture hyperparameters and training details (loss, train/test splits, etc..) are provided.
- The code is yet not made available (maybe for anonymisation of the submission).
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
I have some minor comments, specially in the Methods section:
- F is used for the registration function and for the encoded features.
- In Sec 2.3 I’ is defined as the “image to be registered” while it seems that it is “the image registered”.
- From the text, one infer that I_{i+1} is the reference image and I_i is the moving, but it is not clearly defined in the text.
- Gamma_i (used in the smoothness constraint of the loss) is not defined.
- It is not clear why the similarity cost function in the frequency domain needs to match both the reference and the transformed images? Could you elaborate a bit or include it in the ablation study?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The manuscript is well written with some minor corrections in the methods sections. The presented approach is technically sound and useful for a very interesting application. Moreover, the reported experiments match with their model choices and claims. They are able to show the benefits of their design choices (ablation study) as well as the overall method compared to other methods and for downstream (clinical) applications.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
The authors carefully addressed all my concerns
Review #3
- Please describe the contribution of the paper
This paper proposed an unsupervised non-rigid registration framework for multispectral image registration to help with skin diseases diagnosis. The underlying method consists of four components: adjacent-band pair registration, multispectral attention module, registration field refinement module, and a center-toward registration loss. The authors claim a SOTA MSI registration performance and the proposed framework can also benefit disease classification.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
This paper is very well written and structured. Due to the lack of groundtruth annotation and large non-rigid distortion in MSI, previous image registration method may have limited performance. The authors made a valid statement for each building components, and also carried out ablation studies to validate the loss function.
-
The authors designed ABPR to compensate for large deformations, RFRM for local deformations, and center-toward registration loss as a task-specific reward function. The developed loss function minimizes difference in feature space, aiming to rule out the influence of different lighting conditions. The authors combine both similarity loss and grid regularization (smooth) loss to train the registration network, which is very well reasoned.
-
The authors also carried out a significant amount of experiments on a fair-sized MSI dataset. They further extend this work from registration to benign/malignant classification, validating that their method (MSIR) is beneficial in disease diagnosis. The results in Table. 5 show a significant boost in classification accuracy.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- It seems that the authors are missing the network ablation study for ABPR and RFRM. If I understand correctly, RFRM is a downstream network following ABPR, aiming to further refine the registration field. It would be great to see how much improvement the RFRM brings comparing to solely ABPR.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors mark all “Yes” in the reproducibility checklist. Based on the model explanation and loss functions in the paper, reproducing the method should be feasible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Minor comments:
- It might be helpful to briefly explain what is “adjacent band” for reader who are not so familiar with MSI.
- Is it possible to show the registration error for the initializations?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper itself is well structured and written. Each design component is well reasoned and explained. The authors also carried sufficient amount of experiments and ablation studies to show the effectiveness of their work.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
The authors have justified that Voxelmorph is a kind of ablation study. Based on the quality of this paper, I would recommend a weak accept.
Review #5
- Please describe the contribution of the paper
This paper proposes an unsupervised end-to-end multispectral image registration (MSIR) network. It has been shown that the proposed method outperforms existing registration methods and contributes to the task of benign/malignant disease classification.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper is overall well-written and presented clearly. The experiments are very thorough, which not only include extensive comparison and ablation studies but also provide an evaluation for the subsequent task of disease classification.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The background and literature review were not sufficiently discussed, which makes it hard to evaluate the technical contributions and novelty of this paper against existing registration methods.
- The three comparison methods are not state-of-the-art medical image registration methods.
- The data split is potentially problematic as there is no validation set used during training. Please refer to the detailed comments below for each weakness.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Despite the lack of open-source code, the reproducibility of this paper is acceptable.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- In the Introduction, group image registration was not well introduced. It should be explained (i) what is the definition of group image registration and how it differs from common image registration, (ii) what are the challenges of group image registration, and (iii) why multispectral Images require group image registration.
- For the literature review, the mentioned methods seem to be general image-pair registration methods. The existing group registration methods were not discussed. If the proposed method is the first one in this track, this should be clearly stated.
- Since there is no sufficient discussion of previous group image registration methods for multispectral Images, it is hard to position the proposed modules and models among the literature. For example, it should be clarified (i) which parts of the MSIR are firstly proposed and (ii) which parts follow existing designs.
- The three comparison methods (Voxelmorph, DSIM, and Mu-Net) are not state-of-the-art medical registration methods. The Voxelmorph was proposed in 2018 (in the CVPR version) and has been surpassed by existing methods [1-4] by a large margin. The DSIM performs only tomography transformation instead of deformable registration. The MU-Net was designed for remote sensing image registration, and its key idea (multiple CNNs to generate coarse-to-fine registration) actually has been adopted early in 2020 for medical image registration [1-3].
- There is no validation set in the experiment setting. How did the authors optimize the hyperparameters and monitor the training process? If the testing set was used during the training stage, there will be the risk of overfitting to the testing set.
- Eq.3 should be further explained. Why does it simultaneously encourage the similarity between (i) the warped image and the adjacent band image, and (ii) the warped image and the corrected image? If so, does this mean the adjacent band image and its corrected version should be the same?
Reference [1] Mok, T.C. and Chung, A.C., 2020. Large deformation diffeomorphic image registration with Laplacian pyramid networks. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2020. [2] Shu, Y., Wang, H., Xiao, B., Bi, X. and Li, W., 2021, September. Medical image registration based on uncoupled learning and accumulative enhancement. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2021. [3] Meng, M., Bi, L., Feng, D. and Kim, J., 2022, September. Non-iterative Coarse-to-Fine Registration Based on Single-Pass Deep Cumulative Learning. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2022. [4] Chen, J., Frey, E.C., He, Y., Segars, W.P., Li, Y. and Du, Y., 2022. Transmorph: Transformer for unsupervised medical image registration. Medical image analysis, 82, p.102615.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although the experiments in this paper are very thorough, the technical contributions and novelty are still vague due to insufficient discussion of background and literature review. The comparison with state-of-the-art methods also should be added. I will consider increasing the final score after the rebuttal if my concerns can be well addressed.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
The rebuttal well addressed my concerns, so I am willing to increase the score and recommend its acceptance.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The proposed method for multi-spectral image registration for skin disease diagnosis through refining adjacent-band pairwise registration has novelty. It clearly describes each component of the proposed network, and validation was done thoroughly. There are still some concerns regarding the literature review on groupwise registration, comparison with STOA, and ablation study for ABPR and RFRM. We are not asking for new experiments during the rebuttal period, but the rebuttal should include discussions on those critiques.
Author Feedback
We appreciate that the reviewers and AC consistently think the presented method is novel and the paper has been clearly written. We will address the issues briefly and clarify the details in the final version.
R#1: The model is tested on a single institution dataset There’s currently no publicly available dataset for multispectral images (MSI) of skin diseases applicable to our task. Therefore, we not only utilized a large self collected clinical dataset, but also a large synthetic dataset.
R#1: The data is not publicly available, and the code is yet not made available We’ll disclose our data upon IRB approval. Codes will be available after the double-blinded review.
R#1: Minor comments in Method We’ll carefully revise potentially unclear words, formulas and symbols to improve the paper readability.
R#1: Why Eq.3 is calculated in the frequency domain? Having ablation study? R#5: Eq.3 should be further explained The pixel similarity in spatial domain among MSI is prone to large difference due to spectral intensity, while the feature similarity in the frequency domain is more stable. Eq.3 calculates the similarity in the frequency domain between the warped image I_i’ and the reference image I_i+1, as well as that between I_i’ and another warped image I_i+1’ with the adjacent band, which not only ensures that the transformed images do not deviate from the original spatial distribution, but also realizes center-toward registration of a group of images synchronously and uniformly. We conducted an ablation study on loss function for features in frequency domain (Our Loss) and spatial domain (NCC Loss) (Table 3).
R#3: Ablation study for ABPR and RFRM Once ABPR is removed, the network will degenerate into a Voxelmorph-like registration between two images, and cannot achieve the group image registration of MSI. In other words, Voxelmorph is equivalent to the ablation study of removing ABPR. Similarly, RFRM is a necessary module for the fusion of multiple registration fields and its ablation is reduced to registration of image pair.
R#3: Explain “adjacent band” Each MSI consists of 22 spectral images with wavelengths ranging from 405 to 1650nm. Images with “adjacent band” refer to those imaged at two adjacent wavelengths, e.g. 420 and 450nm.
R#3: To show the initialization registration error Its mean value is 8.77 pixels.
R#5: Insufficient background and literature review for group image registration (GIR) Conventional registration method is to register two images, while GIR is the joint registration of a group of related images. Current GIR research focuses on time-series MRIs. MSI contains multiple images with significant and nonlinear amplitude differences and geometric distortions, not only making the pair-wise image registration not applicable, but also bringing great challenges to GIR due to the inability to take advantage of image intensity or structural similarity. Moreover, there are few studies on GIR for MSI, and we are indeed the first MSI registration study for skin diseases. We will add the review and discussion of GIR and highlight the innovation of our method, namely we propose a MSIR network that integrates cross-band attention mechanism and multi-band deformation field refinement module.
R#5: Comparison methods are not SOTA medical image registration methods We sincerely appreciate the 4 methods you provided. However, they are all registration methods for image pairs, which do not apply to GIR in our work. We will review these methods in Introduction as comparison to GIR. Voxelmorph is a classic registration method widely cited by many papers. We chose it as the baseline of our method and adjusted it to fit the MSI registration task. The 2nd (2021) and 3rd (2022) comparison methods are SOTA methods specifically designed for MSI registration so that the comparison would be fair.
R#5: Is there a validation set Yes, we conducted a 4-fold cross-validation in training set to select models and hyperparameters.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Both the paper and rebuttal are read clearly. The major critique about the relation with groupwise registration methods was addressed well. All reviewers checked the rebuttal and updated their final ratings toward consensus. Other review comments, such as ablation studies of network components, were clarified in the rebuttal.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
While reviewers initially criticised the somewhat unclear relations to prior work and lack of comparison to SOTA registration methods, the authors’ rebuttal alleviated most concerns. I believe the paper makes a decent contribution for a specialised registration problem and while this might not directly have a huge methodological impact on different registration tasks contributes to the solution of a clinical problem. I therefore recommend acceptance.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper proposes a registration method for multispectral images of skin disease. The method is interesting and novel. The paper is clearly written, and the experimental setup and ablation study are sound. I like particularly the downstream task experiment, where the authors show how registration can improve skin disease diagnosis. The authors have addressed the main concerns of the reviewers in the rebuttal. Therefore I recommend acceptance.