Authors

Boqi Chen, Marc Niethammer

Abstract

Multiple imaging modalities are often used for disease diagnosis, prediction, or population-based analyses. However, not all modalities might be available due to cost, different study designs, or changes in imaging technology. If the differences between the types of imaging are small, data harmonization approaches can be used; for larger changes, direct image synthesis approaches have been explored. In this paper, we develop an approach, MRIS, based on multi-modal metric learning to synthesize images of diverse modalities. We use metric learning via multi-modal image retrieval, resulting in embeddings that can relate images of different modalities. Given a large image database, the learned image embeddings allow us to use k-nearest neighbor regression for image synthesis. Our driving medical problem is knee osteoarthritis, but our developed method is general after proper image alignment. We test our approach by synthesizing cartilage thickness maps obtained from 3D magnetic resonance images using 2D radiographs. Our experiments show that the proposed method outperforms direct image synthesis and that the synthesized thickness maps retain information relevant to downstream tasks such as progression prediction and Kellgren-Lawrence grading. Our results suggest that retrieval approaches can be used to obtain high-quality and meaningful image synthesis results given large image databases.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43999-5_26

SharedIt: https://rdcu.be/dnwwF

Link to the code repository

https://github.com/uncbiag/MRIS

Link to the dataset(s)

https://nda.nih.gov/oai/

Reviews

Review #4

Please describe the contribution of the paper

The authors propose a retrieval-based image synthesis approach, where the source modality and target modality are aligned into the same space using contrastive learning. In the inference time, for a test image in the source domain, similar images from the target domain can be extracted using kNN and the final synthesis image can be generated using a weighted sum of those similar images.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The proposed method combines several ideas, such as protecting different modalities into the same latent space using contrastive learning, extracting similar image neighbors from an atlas using kNN, and synthesizing images using the retrieval-based method. Overall, the proposed method is interesting and novel. The paper is well-written and easy to follow. The proposed method is well demonstrated by the experiments and results.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Some related works are missing in the introduction/method sections, such as other retrieval-based image synthesis methods
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The code is not publicly available
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

Adding some related works on retrieval-based image synthesis and multi-modal medical image synthesis will be helpful to general readers.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Overall, this paper is interesting and proposes a new idea for multimodal medical image synthesis.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #1

Please describe the contribution of the paper

In this paper, authors propose an image synthesis method using metric learning via multi-modal image retrieval and k-NN regression. The proposed approach is evaluated using plain radiograph and MR images from the OsteoArthritis Initiative (OAI) database.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Collecting very large medical image datasets is a challenging task. Particularly, when it comes to images from different multi-modalities for the same patient. This study tackles this problem and provides an original and simple approach. Obtained results deserve publication.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

According to the reviewer, the proposed approach enables retrieving cartilages and not synthesizing them. To evaluate if the proposed approach could do so, only plain radiographs should be provided as inputs to the model. Thus, the title of the paper is not appropriate. Please motivate.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

A public database, the OsteoArthritis Initiative is used for the validation of the proposed approach.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

In this paper, authors propose an image synthesis method using metric learning via multi-modal image retrieval and k-NN regression. The proposed approach is evaluated using plain radiograph and MR images from the OsteoArthritis Initiative (OAI) database.

The paper is well written.

How do authors ensure the output of f and g to have the same dimension? As f refers to X-rays and g to MRIs.

For the purpose of classification of the patients using the generated cartilages, it is questionable if the synthesized cartilages could be use along with the K&L scores used for X-rays. As known, when it comes to MR images, different scores (e.g. MOAKS, etc.) are used for such task. Please motivate.

MRIS was not defined before use in the text.

Strengths Collecting very large medical image datasets is a challenging task. Particularly, when it comes to images from different multi-modalities for the same patient. This study tackles this problem and provides an original and simple approach. Obtained results deserve publication.

Weaknesses According to the reviewer, the proposed approach enables retrieving cartilages and not synthesizing them. To evaluate if the proposed approach could do so, only plain radiographs should be provided as inputs to the model. Please motivate. Thus, the title of the paper is not appropriate.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Originality of the proposed approach which may give new directions of research.
Reviewer confidence

Very confident
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

The author is seeking a function that can take a knee x-ray as input and generate a cartilage thickness map as output. They have identified a compelling narrative to showcase the function’s capabilities, namely Multi-modal Retrieval. In addition, the author has implemented the triplet loss to optimize the function’s performance.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- A novel paper for me.
- Good method and good writing.
- I like the organization of experimental results very much.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. I’m not fully convinced why this is very useful. But you might say all synthesis-task paper will receive this question in MICCAI review. It’s okay for me.
2. This learning scheme looks cool, but I not sure if triplet loss make a difference comparing to CE since we only have five classes while in task like face-recognition (while triplet loss is usually applied) we could have more than a million classes.
3. As the severity increases, we can notice that the errors become larger and more unstable. This is understandable since the network might have learned to output an “average cartilage” and only make slight adjustments when the input x-ray has a narrower (left or right) tibial plateau.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Not hard to reproduce the baseline if author release the paired seg mask and x-ray.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
1. This learning scheme looks cool, but I not sure if triplet loss make a difference comparing to CE since we only have five classes while in task like face-recognition (while triplet loss is usually applied) we could have more than a million classes. -> Maybe add some ablation study?
2. As the severity increases, we can notice that the errors become larger and more unstable. This is understandable since the network might have learned to output an “average cartilage” and only make slight adjustments when the input x-ray has a narrower (left or right) tibial plateau. -> Maybe add mean and std of: all synthesised cartilage - avg synthesised cartilage. compare to all MR cartilage - avg MR cartilage.
Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

6
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- A novel paper for me.
- Good method and good writing.
- I like the organization of experimental results very much.
Reviewer confidence

Confident but not absolutely certain
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposed an image synthesis method using metric learning for image retrieval. Three reviewers recommended to accept this paper.

Author Feedback

N/A

back to top

MRIS: A Multi-modal Retrieval Approach for Image Synthesis on Diverse Modalities