List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Kris K. Dreher, Leonardo Ayala, Melanie Schellenberg, Marco Hübner, Jan-Hinrich Nölke, Tim J. Adler, Silvia Seidlitz, Jan Sellner, Alexander Studier-Fischer, Janek Gröhl, Felix Nickel, Ullrich Köthe, Alexander Seitel, Lena Maier-Hein
Abstract
Synthetic medical image generation has evolved as a key technique for neural network training and validation. A core challenge, however, remains in the domain gap between simulations and real data. While deep learning-based domain transfer using Cycle Generative Adversarial Networks and similar architectures has led to substantial progress in the field, there are use cases in which state-of-the-art approaches still fail to generate training images that produce convincing results on relevant downstream tasks. Here, we address this issue with a domain transfer approach based on conditional invertible neural networks (cINNs). As a particular advantage, our method inherently guarantees cycle consistency through its invertible architecture, and network training can efficiently be conducted with maximum likelihood training. To showcase our method’s generic applicability, we apply it to two spectral imaging modalities at different scales, namely hyperspectral imaging (pixel-level) and photoacoustic tomography (image-level). According to comprehensive experiments, our method enables the generation of realistic spectral data and outperforms the state of the art on two downstream classification tasks (binary and multi-class). cINN-based domain transfer could thus evolve as an important method for realistic synthetic data generation in the field of spectral imaging and beyond.
The code is available at https://github.com/IMSY-DKFZ/UDT-cINN.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43907-0_73
SharedIt: https://rdcu.be/dnwdU
Link to the code repository
https://github.com/IMSY-DKFZ/UDT-cINN
Link to the dataset(s)
N/A
Reviews
Review #4
- Please describe the contribution of the paper
The authors introduce a learnable method to perform domain adaptation between synthetic (physics model generated) and real data. They base the methods on invertable networks allowing to simultaneously optimize the maximum likelihood and adversarial losses. The method is tested on two spectral imaging types (PAT and HSI). Improvement on downstream classification task is achieved by using the proposed methods
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
An important problem of bridging the gap between model-based and learnable-based paradigms is addressed. A novel methodology based on invertible nets is proposed. Two tests are performed to assess how far from real data distribtuion the generated by the proposed method samples are.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Even though for PAT images the improvement is notable compared to the synthetic data, for HSI the improvment is ~ one order less. This is however not a negative point, as this is due to rather a good match of the synthetic HSI data to real ones.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Only a general description of the network is given. The lack of architectural details makes reproducibility somewhat challenging. Hope the code will be released to resolve this. Same applies to the data, as I understand the method is tested on two in-house datasets. So reproducibility is not possible. By default, I am biased to trust authors. Still, without providing access to the method and data, nothing stops any authors from potentially putting arbitrary numbers in the result section.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- I could no get from the description whether the adversarial part of the method contains a discriminator network or only a discriminator loss is used for the output of the cINN. Please explain. Also if the former, what is the argumentation to believe that the proposed setup is less prone to typical adversarial issues (e.g. mode collapse, difficulty of training)?
- as a downstream task for PAT a classification between veins/arteries was used. In the data description section, however, the authors mention that the dataset comes with semantic segmentation labeling. Did the authors means pixel-wise classification (i.e. segmentation) for the downstream task? If yes, why random forest was used and not e.g. U-net - the main segmentation tool?
- in the discussion section, the paragraph that starts with “The only similar work on domain” might better fit the related work section in the introduction
- please provide reference to the physics model used to generate synthetic HSI
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
It is a well-written work, with a novel methodology addressing an important synthetic-real data domain adaptation. The experimental results are convincing. I have some concerns though to reproducibility.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
The rebuttal addresses my concerns, so I raise my score to accept.
Review #5
- Please describe the contribution of the paper
This work proposes a domain transfer method, cINN, that enables the generation of realistic spectral data. The invertible architecture can transfer both simulated and real data into a shared latent space to bridge the domain gap. cINN outperforms the state-of-the-art on two downstream classification tasks.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The organization and writing of the paper are very clear and accessible.
- The images in this article are aesthetically pleasing and clearly describe the author’s work and contributions.
- The paper clearly demonstrates the research goal (two hypotheses) and verifies it with experiments on large data (hundreds of thousands).
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The proposed method is just a little innovation, considering previous work had already proposed the cINN method (Conditional invertible neural networks for guided image generation, Lynton Ardizzone et al.).
- The experiments in this paper are not rich enough, considering one plot only demonstrates one aspect of the method’s property.
- The authors only compare cINN with one method (UNIT, 2017). Is there any newer SOTA in the field? By the way, cINN does not consistently perform better than UNIT in Fig. 7.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Although the key hyperparameters are stated in the supplementary material, I recommend that the authors release the codes and data for the method.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- Reduce the description of methods and experimental settings, and increase the richness of experiments, such as supplementing several SOTA methods.
- Put important and intuitive experimental results into the main text (e.g. Table S2 in Supplementary Material).
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The research in this article is complete, and the description is clear and beautiful, but the innovation and experiment volume are slightly insufficient.
- Reviewer confidence
Somewhat confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #7
- Please describe the contribution of the paper
The main contribution of this paper is the introduction of a novel domain transfer method based on conditional invertible neural networks (cINNs) for addressing the domain gap between simulated and real spectral imaging data. The proposed method ensures cycle consistency and enables maximum likelihood learning while maintaining high visual quality. The paper demonstrates the generic applicability of the method by applying it to two spectral imaging modalities: photoacoustic tomography (PAT) at the image level and hyperspectral imaging (HSI) at the pixel level. The comprehensive validation studies confirm that the cINN-based models effectively bridge the domain gap and improve downstream task performance without relying on labeled real data.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper addresses the challenge of the lack of labeled reference data for training and validating neural networks in the medical domain, focusing on spectral imaging.
- The proposed method, based on conditional invertible neural networks (cINNs), offers a new sim-to-real transfer approach with inherent cycle consistency and the ability to conduct maximum likelihood learning while maintaining high visual quality.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The proposed method is based on the prior work (Ardizzone et al.) and yet the difference between the proposed network and the prior work has not been clearly discussed.
- The paper briefly mentions potential spectral inconsistency in previous domain transfer work but does not elaborate on how the proposed method specifically addresses this issue.
- The specific challenges and limitations of working with high-dimensional data are not further discussed, leaving room for potential concerns in practical applications with complex datasets.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The reproducibility of the paper is not clearly mentioned in the paper
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- The paper mentions that network training can be efficiently conducted with maximum likelihood training. It would be useful to provide more details on the scalability and efficiency of the proposed method, especially when dealing with large-scale datasets or high-dimensional input data.
- It would be beneficial to provide more details on how the invertible architecture achieves this and why cycle consistency is important for effective domain transfer.
- The paper briefly mentions the potential spectral inconsistency in previous domain transfer work. It would be beneficial to delve into this issue further and explain how the proposed cINN-based approach specifically addresses and mitigates spectral inconsistency. Providing a detailed discussion and analysis would strengthen the paper’s contribution and highlight the advantages of the proposed method.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- Lack of technical novelty: the proposed method is based on a prior method and yet the technical novelty related to this is not clearly articulated. Also
- Limited comparative analysis: The paper lacks a thorough comparison with existing state-of-the-art domain transfer methods, specifically in the context of spectral imaging. Without a broader comparative analysis, it may be difficult to fully assess the superiority or novelty of the proposed method.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
• The paper proposes a learnable method for domain adaptation using invertible networks and simultaneous optimization of maximum likelihood and adversarial losses. It demonstrates improvements in downstream classification tasks for spectral imaging and addresses the gap between model-based and learnable-based paradigms. Reviewer 4 rates it as a weak accept, mentioning its novelty and convincing results, but raises concerns about reproducibility. Reviewer 5 also gives a weak accept rating, highlighting its clear organization but suggesting improvements in innovation and experiment volume. Reviewer 7 rates it as a weak reject due to the lack of differentiation from prior work, limited comparative analysis, and insufficient discussion on addressing spectral inconsistency. Overall, the paper has interesting aspects but has weaknesses in novelty and thorough analysis. Therefore, this paper is invited for rebuttal.
Author Feedback
We thank the reviewers for the constructive feedback!
We think there was a miscommunication regarding the relation of our method to prior work. While phrasings like “The proposed cINN (cf. Fig. 2) is roughly based on the work of Ardizzone et. al. [3]” may have been misleading (apologies!), we want to point out that we are proposing an entirely new approach to domain transfer, specifically designed for medical imaging data with many spectral channels (e.g. HSI/PAT) and, importantly, not requiring paired images. This approach inherently addresses weaknesses of the state of the art with respect to the preservation of spectral consistency.
Response to main criticism:
Novelty: While invertible neural networks (INNs) have been proposed in prior work, the architecture by [3] is not directly applicable to sim-to-real transfer. We will clarify that our method specifically addresses the fact that [3] need paired images for training. With “roughly based on”, we were referring to the basic invertible building blocks which we use as well. We will further highlight the novelty and benefits of our key design choices:
(1) Domain-conditioned INN: We are the first to condition an INN on the domain to achieve sim-to-real transfer. As a key theoretically-motivated advantage compared to the state-of-the-art Cycle GAN-based approach, this approach offers inherent cycle consistency.
(2) GAN-enhanced INN: To achieve high visual quality beyond spectral consistency, we included two separate discriminators for the domains. As a key theoretical advantage, we avoid mode collapse with maximum likelihood optimization. We propose adding the performance of our architecture without GAN to Table S2, to demonstrate the substantial gain in performance (up to 9% for PAT with GAN).
(3) Organ conditioning for spectral consistency: To leverage the fact that different tissue types feature characteristic spectral signatures, we condition the image transfer method on the semantic segmentation maps/organ labels. This key novelty gives a substantial qualitative improvement and boosts performance on the downstream task (6-20%; cf. Table S2).
IMPORTANTLY: With these three novel design choices, we are contributing a new method, specifically designed for image analysis problems that are based on spectral data.
Spectral consistency: We agree with the reviewer that we failed to make sufficiently explicit how we achieve spectral consistency. We will clarify that it can be attributed to the conditioning on the target organ (see above) and theoretically-guaranteed cycle consistency through invertible components.
Comprehensiveness of experiments: The results for Hypothesis 2 were only represented by one figure in the main paper (rest in appendix). We are happy to switch Table S2 with Fig. 7 to clearly show in the main paper that our experiments were actually comprehensive (2 modalities based on 162 PA images / ~920,000 spectra; ablation experiments; comparison to state of the art).
Comparison to further methods: We should have mentioned explicitly that most recent sim-to-real transfer methods are not directly applicable to spectral data, e.g. as they inherently rely on the availability of pre-trained (3-channel) networks (e.g. VGG for feature extraction). As we did not manage to obtain high-quality results with the Alignflow method (Grover et al., 2020 [13]), the only open-source INN-based image transfer method, we put significant effort into optimizing the performance of the popular UNIT (cited by 2,658) on our data. To provide more evidence for the relevance of the method, we would be happy to complement Table S2 with results on the original Cycle GAN (cited by 17,530) compared to which we obtained improvements in the ranges of 30-300% (HSI) and 82-388% (PAT).
Reproducibility: We are happy to open-source all the code for our models, training, testing and visualization of results with pre-trained models including the simulation test sets for both PAT and HSI.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This work aims to address the domain gap in synthetic medical image generation by introducing a domain transfer approach based on conditional invertible neural networks. The method guarantees cycle consistency and utilizes maximum likelihood training, providing realistic spectral data for hyperspectral imaging and photoacoustic tomography. The rebuttal has adequately addressed the major concerns of the three reviewers, including the clarification of technical novelty and details on spectral consistency, reproducibility, and comparison against SOTA methods. Thus, this paper is recommended for acceptance.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
In general, this paper is clear written and easy to understand. the method is technical sound and does have some novelty for this problem. The rebuttal successfully addresed several concerns raised by the reviewers.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have satisfactorily addressed some of reviewers’ comments in the rebuttal, leading to reviewers upgrading their scores. I therefore recommend an acceptance at this stage.