List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Kazuma Kobayashi, Lin Gu, Ryuichiro Hataya, Mototaka Miyake, Yasuyuki Takamizawa, Sono Ito, Hirokazu Watanabe, Yukihiro Yoshida, Hiroki Yoshimura, Tatsuya Harada, Ryuji Hamamoto
Abstract
Medical education is essential for providing the best patient care in medicine, but creating educational materials using real-world data poses many challenges. For example, the diagnosis and treatment of a disease can be affected by small but significant differences in medical images; however, collecting images to highlight such differences is often costly. Therefore, medical image editing, which allows users to create their intended disease characteristics, can be useful for education. However, existing image-editing methods typically require manually annotated labels, which are labor-intensive and often challenging to represent fine-grained anatomical elements precisely. Herein, we present a novel algorithm for editing anatomical elements using segmentation labels acquired through self-supervised learning. Our self-supervised segmentation achieves pixel-wise clustering under the constraint of invariance to photometric and geometric transformations, which are assumed not to change the clinical interpretation of anatomical elements. The user then edits the segmentation map to produce a medical image with the intended detailed findings. Evaluation by five expert physicians demonstrated that the edited images appeared natural as medical images and that the disease characteristics were accurately reproduced.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43895-0_38
SharedIt: https://rdcu.be/dnwyR
Link to the code repository
https://github.com/Kaz-K/medical-image-editing
Link to the dataset(s)
N/A
Reviews
Review #4
- Please describe the contribution of the paper
The authors propose a novel method to edit segmentation masks and generate realistic medical images from these. The segmentation masks can be edited in order to express determined anatomical characteristics.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The novelty brought by this work is the image-editing algorithm proposed by the authors. This algorithm is capable of synthesizing medical images via self-supervised segmentation. The segmentation masks can be edited to reflect some pathology or determined anatomical structures, this way creating an automatic model to generate variate medical data. The authors used an algorithm based on vector quantization, creating another novelty brought by their work.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The main weakness is the analysis of the final images. The authors provide values for the PSNR which are not within the range of “good quality image”. Also the chosen training strategy is not clear: in section 2.1 it’s mentioned that the encoder and decoder are training on a first stage, and on section 2.2 the decoder and the discriminator are trained on a second stage. It’s not clear how many times the decoder was trained. Regarding the method chosen to generate the images, it relies on vector quantization. The authors could mention the original work where a VQ-GAN was proposed to generate data (https://arxiv.org/abs/2012.09841), as this is relevant for the topic they work on.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors provide enough information regarding the reproducibility of their results, as their synthesized images and code will be publicly available.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Overall the paper reads well and is clear. Regarding the state of the art I believe the authors should add information about using deep generative models to synthesize medical images (https://arxiv.org/pdf/2207.08208.pdf); (https://ieeexplore.ieee.org/document/10049068); (https://ieeexplore.ieee.org/document/9324763) These mentioned references, are also good examples of methods that use accurate labels to synthesize medical images. This way, the statement that the authors make regarding the usage of inaccurate detailed anatomical masks, in section 1, should be rewritten. In section 3, implementation and datasets, in the description of the pelvic dataset, the authors refer to the data as images series. This is not clear about the meaning of what an image series is (2D+t, 2D at different views, 3D spatial,…?). Should be rephrased or explained in more detail.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The work done by the authors brings novel contributions to the field of medical image synthesis, together with anatomical labels. If the weaknesses are addressed and corrected, this paper brings a solid contribution to the field.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The authors introduce a novel human in the loop image editing tool to allow for generation of educational images with fine-grained control over anatomical elements. The method learns an encoder to segment an image and a decoder to reconstruct from the segmentation. Additionally a discriminator is learned for image quality of the reconstructions. After training, the pixel class in the segmentation map can be changed to perform edits before reconstructing using the decoder to acquire specific anatomical traits (e.g. tumor).
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper is well written with a strong evaluation
- The authors address an important topic of generation of medical images for education with fine-grained control over disease characteristics
- They improve existing self-supervised segmentation techniques by introducing reconstruction loss, thus benefiting from self-supervision and improving segmentation quality.
- The evaluation of the generated images based on five expert physicians indicates no significant differences between synthesized and real samples
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- No major weaknesses
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The code is not yet made available due to anonymization. However, other details for reproducibility are provided in accordance with the author statement.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Nice paper with an interesting method solving a relevant task.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
7
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Interesting, well-written paper solving a relevant task with only minor to no weaknesses.
- Reviewer confidence
Somewhat confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This paper proposes a medical image editing method, which consists of two steps. In the first step, the segmentation map is obtained by using self-supervised learning with the help of k-means clustering. In the second step, the segmentation map is reconstructed to the original input. The proposed method is evaluated on two in-house datasets.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This paper is well written and easy to follow.
- The evaluation procedure involves human experts to evaluate the results, which is reasonable and closer to the real-world applications.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The authors are suggested to add comparisons with other methods, such as state-of-the-art methods, to demonstrate the novelty and effectiveness of the proposed method. For example, self-supervised learning with clustering is widely adopted for medical image segmentation [1,2]. Several methods can be used for medical image editing [3,4], and the authors are suggested to add experimental comparisons with these methods.
[1] Li. Superpixel-guided label softening for medical image segmentation. MICCAI 2020. [2] Ouyang. Self-supervision with superpixels: Training few-shot medical image segmentation without annotation. ECCV 2020. [3] Blanco. Medical image editing in the latent space of Generative Adversarial Networks. Intelligence-Based Medicine, 2021. [4] Zhou. Interactive Deep Editing Framework for Medical Image Segmentation. MICCAI 2019.
-
There are many hyper-parameters in the proposed method, such as $w_{cluster}$, $w_{dist}$, $w_{cross}$, $w_{recon}$, $w_{mse}$, $w_{ffl}$, $w_{lpips}$, $w_{int}$. How to determine and balance these hyper-parameters? Are these hyper-parameters sensitive to the performance of the proposed method? The authors are suggested to add the experimental results to demonstrate the sensitivity of these hyper-parameters.
-
The two in-house datasets are not publicly available. The authors are suggested to add the experimental results on the public datasets. Additionally, there’s no validation set in the experiments. Did the authors use the test set to determine the hyper-parameters? If so, the results may be over-optimistic.
-
The authors are suggested to add the experiments to verify the effectiveness of the proposed method on the downstream tasks.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The proposed method is evaluated on in-house datasets, and the author did not promise to release the data. Many hyper-parameters need to be determined, the paper should clarify it.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Refer to Q6.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- Experiments need to be strengthen. Lack of ablation study of hyper-parameters, comparison with the state-of-the-arts.
- Comparison with related methods to clarify the novelty.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The work proposes a novel image-editing approach to synthesise images for medical education based on self-supervised segmentation, which also shows interesting results on generating pathologies. The evaluation by five human expert also indicates promising performance of the method. The idea and application are both novel and will be of high interest to the community. It is therefore recommended an acceptance. Further clarifications on the paper as highlighted by the reviewers are suggested to be addressed before the final submission.
Author Feedback
We are grateful for the constructive feedback from the meta-reviewer and reviewers on our paper’s presentation and conceptual contribution. In the final version, we aim to refine our paper to address the issues raised by both Reviewers #2 and #4, giving particular attention to the constructive feedback from Reviewer #4 (see the updated Introduction and corresponding section of the Abstract).
Reviewer #2 Pt. 1: We agree that a comparison with previous methods would provide valuable insights, especially in highlighting the performance differences among the superpixel-based algorithms. Since we are unable to include additional experiments at this time, we acknowledge this as a limitation that should be addressed (see the Conclusion section).
Reviewer #2 Pt. 2: In response to the issue of hyperparameter optimization, we have included the relevant experimental results in the Supplementary Information. This also includes some ablation studies. As shown in Fig. S1, the self-supervised segmentation was sensitive to the hyperparameters.
Reviewer #2 Pt. 3: Concerning the lack of public datasets, future work may incorporate additional experiments using available public datasets. Responding to the absence of validation datasets, we conducted the hyperparameter optimization based on the results from the training dataset due to the absence of objective metrics, as stated in the manuscript. It’s important to note that we refrained from using the testing dataset for hyperparameter search. This ensures that the results are not overfitted to the testing datasets.
Reviewer #2 Pt. 4: In relation to downstream tasks, our future plans involve evaluating how synthetic medical images can enhance practical medical education. Although not discussed in this manuscript for the sake of reader simplicity, our work can potentially be used for synthetic data augmentation in the future.
Reviewer #4 Pt. 1: Concerning the quality of image generation, we agree that by crafting a more efficient algorithm, we could enhance the PSNR. Indeed, despite employing a variety of techniques like focal frequency loss, perceptual loss, intermediate loss, and the U-Net discriminator, the image generation quality reached a certain limit. This may partly be due to the segmentation map, which serves as the input for the decoder, not containing sufficient information from the original image. This results in reconstructions that are visually natural, but not fully identical to the original image. In our future work, we intend to improve the algorithm by incorporating diffusion models to achieve superior image quality.
Reviewer #4 Pt. 2: The decoder was trained twice: during the first and second stages. In the initial stage, the decoder acts as an aid to achieve robust self-supervised segmentation, and it is intensively trained again in the second stage to generate realistic images. This is due to the complexity of jointly training both self-supervised segmentation and realistic image generation using adversarial training, which is challenging to stabilize. We have included further explanation to elucidate this aspect in Section 2.2 of the updated manuscript.
Reviewer #4 Pt. 3: In terms of its relation to VQ-GAN, while it uses VQ in the latent space, we apply VQ in the image space to attain self-supervised segmentation. Even though portions of our implementation depend on VQ, we believed that explaining the algorithm in the context of K-means clustering would simplify understanding, which is why we cited only relevant papers without directly mentioning VQ.