List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Valentin Koch, Olle Holmberg, Hannah Spitzer, Johannes Schiefelbein, Ben Asani, Michael Hafner, Fabian J Theis
Abstract
Optical coherence tomography (OCT) imaging from different camera devices causes challenging domain shifts and can cause a severe drop in accuracy for machine learning models. In this work, we introduce a minimal noise adaptation method based on a singular value decomposition (SVDNA) to overcome the domain gap between target domains from three different device manufacturers in retinal OCT imaging. Our method utilizes the difference in noise structure to successfully bridge the domain gap between different OCT devices and transfer the style from unlabeled target domain images to source images for which manual annotations are available. We demonstrate how this method, despite its simplicity, compares or even outperforms state-of-the-art unsupervised domain adaptation methods for semantic segmentation on a public OCT dataset. SVDNA can be integrated with just a few lines of code into the augmentation pipeline of any network which is in contrast
to many state-of-the-art domain adaptation methods which often need to change the underlying model architecture or train a separate style transfer model. The full code implementation for SVDNA will be made available at https://github.com/ValentinKoch/SVDNA.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16434-7_67
SharedIt: https://rdcu.be/cVRsz
Link to the code repository
https://github.com/ValentinKoch/SVDNA
Link to the dataset(s)
https://retouch.grand-challenge.org/
https://www.kaggle.com/paultimothymooney/farsiu-2014
Reviews
Review #2
- Please describe the contribution of the paper
This paper proposes to transfer device-specific noise to OCT images taken with different device(s), thus mitigating the domain gap that often affects performance with models trained on one device source, but used on another device source. The transfer is done by singular value decomposition-based noise adaptation (SVDNA), which adds the reconstructed noise signal from some target domain, to a source image. With the SVDNA method, noise from various (known) domains can be included as part of the data augmentation in training a UNet++ segmentation model. It is claimed that this improves OCT image layer segmentation performance over state of the art unsupervised domain methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- SVDNA is relatively simple to apply, and does not appear to require in-depth training with deep models
- Evaluation on public datasets
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- SVDNA evaluation in this study applies only to known domains
- Key contribution appears to actually be the final histogram matching, and not the SVD-based noise transfer, which might not have been emphasized
- Comparison against prior methods not very clearly presented
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The SVDNA transfer method (Algorithm 1) is fairly straightforward.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- In Section 2.1, it is stated that the pixel values between the restyled image and target image is (often) still different, which is fixed by further histogram matching. The histogram matching is empirically shown to further improve performance somewhat (Table 1, supplementary material).
However, it is not clear whether the (final) restyled image should necessarily actually share the same pixel value distribution as the target image, since the true source image (assumed with noise from source device distribution removed) and the true target image (assumed with noise from target device distribution removed) might well have different distributions. Indeed, the noise addition formulation would appear to have the restyled images be (true source image)+(source noise)+(added estimated target noise), which is then adjusted to fit (true target image)+(target noise). Theoretically, it might be more proper to first subtract estimated source noice, if the source is known. This might be considered.
-
Related to the above, the clipping of values outside the interval [0,255] after noise transfer would appear a potential source of signal loss at its extremes. It might be clarified if such out-of-bound values apply to a significant proportion of the signals, in general.
-
In Section 2.2, it is stated that for SVDNA style transfer, when a target domain is required, one image from that domain is randomly chosen as the source. While Figure 2 suggests that images from the same domain do have relatively similar noise characteristics, it might be clarified whether the adapted data does in fact mimic the intended target domain well. SVDNA to Bioptigen in Figure 2(b) appears clearly separate from Bioptigen, for example, and SVDNA to Cirrus/Topcon do not appear to be shown.
-
Moreover, Table 1 in supplementary material suggests that the histogram matching alone contributes the majority of the performance improvement, with noise transfer alone contributing next to nothing. As such, the natural comparison would seem to be against histogram matching/normalization methods such as CLAHE etc.
-
In Section 3.2, it is not clear why CycleGAN required a different training methodology.
-
While Table 1 in the main text claims comparison of SVDNA to supervised trained models from 8 teams, only three results are presented. If only the best model from the 8 teams was used, this might be clearly stated.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The main contribution of the proposed method appears to be from histogram matching, and not noise transfer as emphasized in the title and presentation. While this is not undesirable in itself, it should be clearly stated. Moreover, the comparisons against other methods might be more fully presented.
- Number of papers in your stack
6
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
4
- [Post rebuttal] Please justify your decision
The authors have addressed the additional contribution of unsupervised domain adaptation, over histogram matching alone. However the lack of comprehensive benchmarking against prior unsupervised domain adaptation methods makes it somewhat difficult to recognize the particular significance of the proposed method.
Review #3
- Please describe the contribution of the paper
The paper proposes a new domain adaptation technique, named SVDNA, for improving the performance of segmentation models when trained in a source domain and tested on a different/target domain. The paper is validated when domains are images generated by different camera devices for the task of Optical Coherence Tomography (OCT).
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The proposed idea seems to be novel. The idea is very simple and it seems to properly adapt the domain when moving from a less noisy one to a more noisy one. The performance in the opposite scenario is uncertain.
- The paper is well written and clear.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- By design, and as mentioned by the authors, the proposed technique has limitations. Specially, when the source domain is more noisy than the target domain, the proposed technique will potentially not perform well. The performance is uncertain in this scenario. This important limitation reduces the applicability of the technique to more scenarios.
- The Algorithm 1 is not clear enough. Variables within the algorithm are never defined so that it cannot be properly understood. Authors should properly define all the variables employed and add comments on the algorithm when needed. The description of the algorithm in the body of the paper helps understanding the technique applied.
- There exists mismatches between Algorithm 1 and text. Clipping and histogram matching are not within the Algorithm but they described in the text.
- There are some variables defined in the text (e.g. probability ‘p’) which are never applied anywhere. If a variable is defined, it should be used somewhere (e.g. in an equation)
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Code will be available upon acceptance Public datasets/benchmarks are used
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- Improve the Algorithm 1. Including the cohesiveness between the text description and the algorithm.
- Make sure that all variables used are clearly defined.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The paper is clear, well written, and the validation experiments are enough
- The proposed technique has minor novelty and contributions as the applicability seems to be very reduced to when the source domain is less noisy than the target domain. In real settings, target domain features might be unknown or likely more noisy, where this algorithm will not perform well
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #5
- Please describe the contribution of the paper
In order to to bridge the domain gap between different retinal OCT imaging devices, the authors proposed a minimal noise adaptation method based on a singular value decomposition (SVDNA), without modifying the basic model architecture or training an extra style transfer model.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
It is interesting that style transfer between the source and target domain can be achieved by combining content components from the source image and noise components from a target image after singular value decomposition. The proposed method is simple and does not require the modification of the basic model architecture or extra training of a style transfer model. The experimental results and discussion presented in the paper and supplementary material are sufficient.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Some minor mistakes exist in the paper (see detailed and constructive comments).
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The proposed method are described in details and the full code implementation will be made available, which would make it easy to reproduce the methods.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Minor issues:
- “For more details on the feasibility of the used values of k see supplementary figure 1” in Page 4 should be modified to “For more details on the feasibility of the used values of k see supplementary figure 2”.
- “for examples of content distortions achieved by an optimized CycleGAN see supplementary figure 2” in Discussions and Limitations should be modified to “for examples of content distortions achieved by an optimized CycleGAN see supplementary figure 4”.
- The last line in Table 1 of the supplementary material seems inconsistent with the results of Topcon (left) in Fig. 3. Please check.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I recommend accepting this paper. The authors sufficiently demonstrate that style transfer between the source and target domain can be achieved by combining content components from the source image and noise components from a target image after singular value decomposition. Their simple method does not require the modification of the basic model architecture or extra training of a style transfer model. In addition, the proposed method might be applied in domain adaptation of other medical imaging modalities such as CT with different doses.
- Number of papers in your stack
2
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
The authors have made an appropriate explanations and modifications according to the reviewers. I suggest accepting the paper.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
TBA
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5
Author Feedback
Dear Area Chair, dear Reviewers
We thank you for your constructive feedback on our manuscript and for giving us the opportunity to clarify the points raised.
Our approach is acknowledged as a “simple method” that “does not require the modification of the basic model architecture” and could “… be applied in domain adaptation of other medical imaging modalities” (R5). The novelty of our approach is acknowledged by R3 and R5 finds it “… interesting that style transfer between the source and target domain can be achieved”. All reviewers appreciate the clarity and organization of this paper and R3 concludes it to be “well written and clear”.
Yet the three reviewers also have concerns that we are addressing below:
R2 is critical of the role of the noise transfer step and finds “… that the histogram matching alone contributes the majority of the performance improvement …” We agree with the reviewer that histogram matching is an important part of the algorithm and attributes heavily to the performance gain. However we argue that
a) performance gain is class dependent. While histogram matching contributes most for the IRF class when comparing to the baseline (0.26 -> 0.70 dice score), the noise transfer step contributes most for the PED class (0.56 -> 0.69). As expected, the combination of both steps always yields the best result. b) Only with our SVDNA method are we able to match noise characteristics (as can be seen in the figure 2b), which is one of the main points of SVDNA. While the performance was only tested on a segmentation task, we would also expect a higher performance gain in related tasks as we are able to match higher order statistics of the target domain images.
R2 mentioned that the “SVDNA evaluation in this study applies only to known domains’’. We would like to clarify that indeed in Unsupervised Domain Adaptation, the assumption is that there is an unlabelled dataset (separate from a final testset) of a target domain, which can be used for unsupervised training, which is the case here: no labels were shown to the SVDNA algorithm from the target domains during training. Attempting domain transfer to a totally unknown domain would be zero-shot domain adaptation, which we are not attempting here. R2 suggested additional comparisons to simple domain adaptation algorithms. In our work we investigated the feasibility of replacing current approaches that are using deep learning by a minimal method to reduce complexity. We did not aim to benchmark simple methods for unsupervised domain adaptation, but agree that other methods can work well too and make for interesting future studies.
We agree with R3 regarding the limitations concerning application to less noisy domains and also discuss this in the manuscript. However, as pointed out in the discussion part and similarly shown by other work, if the source domain is more noisy then the target domain, one could use SVDNA to add noise at test time instead. For the sake of scope this is mentioned as possible future work in this manuscript.
We thank R3 for raising that the variables of the algorithm need to be introduced. We revised our manuscript accordingly, and now show the whole algorithm, which is beneficial for the understanding of SVDNA and added two more lines to algorithm 1.
R2 mentions that “SVDNA to Bioptigen in Figure 2(b) appears clearly separate from Bioptigen, for example, and SVDNA to Cirrus/Topcon do not appear to be shown.” We agree with R2 that Bioptigen / SVDNA to Bioptigen is clearly separate, which we address in the discussion of the manuscript. However, in both, figure 2a and figure 2b we demonstrate style transfer to all three domains including Cirrus and Topcon. The legend of the UMAP plot shows the labeling of SVDNA to Topcon/Cirrus as purple/yellow. We thank R5 for pointing to issues with references to the supplemental figures. We also revised the caption of figure 3, eliminating the mismatch to supplemental table 1.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The reviewers highlight the novelty of the idea that uses singular value decomposition-based noise adaptation. They also highlight that the idea is well validated in the context of domain adaptation between OCT devices. Finally, the criticisms of R2 have been partly addressed in the rebuttal, lifting the paper over the acceptance threshold.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have clearly clarified the issues raised by the reviewers. The explanations in the rebuttal look reasonable and correct to me.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
8
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
concerns nicely adressed, borderline paper in my stack.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
mid range