List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Agnieszka Tomczak, Aarushi Gupta, Slobodan Ilic, Nassir Navab, Shadi Albarqouni
Abstract
Generative adversarial networks (GANs) offer an effective solution to the image-to-image translation problem, thereby allowing for new possibilities in medical imaging. They can translate images from one imaging modality to another at a low cost. For unpaired datasets, they rely mostly on cycle loss. Despite its effectiveness in learning the underlying data distribution, it can lead to a discrepancy between input and output data. The purpose of this work is to investigate the hypothesis that we can predict image quality based on its latent representation in the GANs bottleneck. We achieve this by corrupting the latent representation with noise and generating multiple outputs. The degree of differences between them is interpreted as the strength of the representation: the more robust the latent representation, the fewer changes in the output image the corruption causes. Our results demonstrate that our proposed method has the ability to i) predict uncertain parts of synthesized images, and ii) identify samples that may not be reliable for downstream tasks, e.g., liver segmentation task.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16446-0_48
SharedIt: https://rdcu.be/cVRTH
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper presents a noise injection technique that allows generating multiple outputs, thus quantifying the differences between these outputs and providing a confidence score that can be used to determine the uncertain parts of the generated image, the quality of the generated sample, and to some extent their impact on a downstream task.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This paper is well written and easy to read.
- Sufficient experiments.
- The results are described and analyzed in details
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
– There is no need to do experiments, We can also be sure that the scores of these two indicators(variance and MI) will be very poor in Sec.3.2. The generator has a high generation freedom with a black patch. So the author claims ‘The results suggest the possibility of finding a potential confidence threshold to eliminate uncertain synthesized images.’ is also worth researching.highly reproducable
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
highly reproducible
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- The caption of Figure 3 should be ‘corresponding tumor segmentation map (and) the uncertainty heat map, respectively’
- refer to the weakness
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Nowadays, many works on medical image processing using GANs for data augmentation or modality translation. But sometimes it doesn’t bring more useful information. This paper can bring benefits to the community and give these works a reference.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
- Proposing noise injection technique for testing latent representation;
- Proposing two metrics to calculate uncertainty for synthesized images;
- Validating the hypothesis that robust latent representation leads to better quality of generated image.
- Small noise injections during the training phase lead to more robust representation and slightly higher image quality.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Noise injection is easy to use in deep learning models, such as GANs;
- Conducting pre-experiment to observe that noise injection can identify uncertainty parts;
- Carried on adequate experiments including image translation and downstream tasks.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The validation experiment of the proposed confidence score correlates with the quality of downstream task, e.g., segmentation, on the synthesized image is not enough and straight forward. Segmentation tasks may not be enough, more tasks such as classification and detection can be implemented.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Authors will not make the code publicly available. But the method is easy to be implemented.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The experiment and results in sections 3.2 and 3.3 are very clear and straight forward, and the conclusion is obvious. But the experiment in section 3.4 which validates that the proposed confidence score correlates with the quality of downstream task, is not straight forward and convincing. In Table 4, the absolute value of a correlation between the confidence score and DICE coefficient for the proposed method is up to 0.54, which indicates moderate correlation. These results can not support the conclusion “This suggests that our method can be used most efficiently in cases where the images are generated well enough for the downstream task network to also perform well.”
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The idea is new and interesting. But more convincing results are necessary.
- Number of papers in your stack
3
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
the authors investigated the hypothesis that we the image quality can be predicted based on its latent representation in the GANs bottleneck.The authors presented a method using latent representation corruption with noise as such that multiple outputs can be generated to obtain uncertainty. The results demonstrated in this paper show that the method has the ability to predict uncertain parts of synthesized images, and identify samples that may not be reliable for downstream tasks.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Easy to read. The idea presented in this paper is very clear and easy to follow.
- The experiments are solid
- The idea presented in this paper is somehow novel and can provide insights in medical imaging. Becasue the uncertainty estimation method is unsupervised, which make it value in medical imaging domain considering that most datasets cannot provide proper label.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The experiements shown in Figure 3 are somehow vague. I feel hard to understand the relationship between segmentation mask and the generated confidence score maps. Better to overlap both images to show correlation.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Implementing the method should not be a problem because the idea is straightforward.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- Need to explain more details about how to relate segmentation mask with uncertainty maps.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The novelty of the idea - somehow novel.
- The completeness of the experiment settings - the visualisation of the relationship between uncertainty maps and the segmentation is hard to follow. The relationship betweent hem is not very strong. Maybe authors need to find a new way to demonstrate how to use this uncertainty estimation technique.
- The clarity and organisation of the paper - the organisation is very good.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper presents a method of injecting noise to the latent representation to test the hypothesis that robust latent representation leads to better quality of generated image. The reviewers agree that this is a novel and interesting work. One weakness is that the results in Fig. 3, the correlation between uncertainty and segmentation results is hard to interpret (R2 and R3). The other weakness is that the experiments in Section 3.2 may be better designed as pointed out by R1.
The paper missed some highly relevant references, such as https://arxiv.org/abs/1905.02175
In general, it may be interesting to further extend the work using adversarial attack techniques, such as https://arxiv.org/abs/2103.13557 https://www.medrxiv.org/content/10.1101/2021.12.23.21268289v1
Overall, this is an interesting piece of work in a well organized form, which should be presented at MICCAI.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
1
Author Feedback
We sincerely thank all the reviewers and AC for the insightful comments and valuable suggestions on how to improve and further develop our work. We fully agree that the correlation in Fig. 3 could be presented better. For the camera-ready version, we fixed it by overlapping the uncertainty maps with the tumour segmentation masks as suggested by R3, which indeed improved the clarity of the figure and highlighted the correlation between the detected uncertainty and tumour segmentation mask. We also agree with R1 that the experiment results in section 3.2 are to be expected. While we have provided the histogram of these different scores, and one could draw the threshold to filter out the poorly generated samples, we acknowledge that as it is an artificial example and would require further validation in real-life scenarios. Thank you for this remark; we will further elaborate on this issue in the discussion of this experiment. Additionally, we added the suggested references and commented on them in the future work paragraph. Thank you very much once again for the thoughtful comments. After addressing them in the text and fixing the issues, we believe that the quality of the paper improved.