Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews Back to top
List of Papers By topics Author List
Paper Info Reviews Meta-review Author Feedback Post-Rebuttal Meta-reviews

Authors

Qikui Zhu, Lei Yin, Qian Tang, Yanqing Wang, Yanxiang Cheng, Shuo Li

Abstract

Existing tumor augmentation methods cannot deal with both domain and content information at the same time, causing a content distortion or domain gap (distortion problem) in the generated tumor. To address this challenge, we propose a Domain-aware and Content-consistent Cross-cycle Framework, named DCAug, for tumor augmentation to eliminate the distortion problem and improve the diversity and quality of synthetic tumors. Specifically, DCAug consists of one novel Cross-cycle Framework and two novel contrastive learning strategies: 1) Domain-aware Contrastive Learning (DaCL) and 2) Cross-domain Consistency Learning (CdCL), which disentangles the image information into two solely independent parts: 1) Domain-invariant content information; 2) Individual-specific domain information. During new sample generation, DCAug maintains the consistency of domain-invariant content information while adaptively adjusting individual-specific domain information through the advancement of DaCL and CdCL. We analyze and evaluate DCAug on two challenging tumor segmentation tasks. Experimental results (10.48% improvement in KiTS, 5.25% improvement in ATLAS) demonstrate that DCAug outperforms current state-of-the-art tumor augmentation methods and significantly improves the quality of the synthetic tumors.

Link to paper

DOI: https://doi.org/10.1007/978-3-031-43904-9_33

SharedIt: https://rdcu.be/dnwHc

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #2

  • Please describe the contribution of the paper

    This paper proposes a tumor augmentation method which disentangle domain and context information to eliminate the distortion problem. The proposed framework incorporates two techniques of 1) contrastive learning which picks samples up from different domains, and 2) consistency learning to obtain domain invariant information. The method’s efficacy is evaluated with a combined dataset of CT and MRI each of which images contain tumors.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • This is an early work that aims to exploit multiple disease datasets across different domains (CT and MRI). A consistency learning approach across different domains is also newly proposed.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    • In the comparative studies, MixUp and CutMix is not suitable for an alternative augmentation method. The performance improvement of the DICE metric compared to a latest augmentation (StyleMix) is pretty small, and it will not show statistically significant results.
    • It is unclear that what kind of the proposed method is able to augment on generated tumor images.
  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    All data and codes are made available to public.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

    It is not easy to follow the contents. I think that the term “Domain” represents CT and MRI, but we will notice it at latter part of the paper. Similarly, the main object of the paper will be utilizing tumor images of different modalities, but it is not clearly mentioned. How about to explicitly describe the purpose of the method and application. -The font sizes in the figures are too small to read. Also the figures contain so many information, and readers cannot easy to understand.

    • Comparison with other generative approach like StyleMix is expected.
    • Readers expect that the method can generate diverse tumor samples. More detailed evaluation or discussion should be done.
  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    4

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The entire context is not easy to follow. In the current form, contributions of this paper seems unclear.

  • Reviewer confidence

    Confident but not absolutely certain

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    5

  • [Post rebuttal] Please justify your decision

    If the clarity of this paper’s contribution are included in the final paper, I consider this is worthy of acceptance.



Review #3

  • Please describe the contribution of the paper

    The paper proposes a new framework for generating synthetic tumors, called DCAug, which addresses the challenge of preserving both domain and content information in the generated tumors. The authors introduce a Cross-cycle Framework and two novel contrastive learning strategies, DaCL and CdCL, to disentangle image information into domain-invariant content information and individual-specific domain information. DCAug maintains consistency in domain-invariant content information while adapting individual-specific domain information during new sample generation. The authors evaluate DCAug on two tumor segmentation tasks and show that it outperforms current state-of-the-art tumor augmentation methods and significantly improves the quality of synthetic tumors. The proposed method has the potential to enhance the accuracy of medical image analysis and improve diagnosis and treatment of cancer.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1. Novel framework: The paper proposes a new framework, DCAug, for generating synthetic tumors that can preserve both domain and content information in the generated tumors. This is a significant challenge in tumor augmentation that has not been effectively addressed by previous methods.

    2. Novel contrastive learning strategies: The authors introduce two novel contrastive learning strategies, DaCL and CdCL, to disentangle image information into domain-invariant content information and individual-specific domain information. This approach has not been explored in previous studies on tumor augmentation.

    3. Strong evaluation: The authors evaluate DCAug on two challenging tumor segmentation tasks and demonstrate that it outperforms current state-of-the-art tumor augmentation methods with a 10.48% improvement in KiTS and a 5.25% improvement in ATLAS. They also perform extensive ablation studies to demonstrate the effectiveness of the proposed methods.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    Although the paper provides a well-written introduction, I have some questions that I would like to raise. Firstly, the ablation experiments for each component are missing. While the paper proposes one novel Cross-cycle Framework and two novel learning strategies (DaCL, CdCL) to solve the content and domain distortion problem in generated tumors, it does not verify the necessity of each part through ablation experiments.

    Secondly, the article does not clearly describe how to combine data augmentation strategies with the underlying segmentation model, and there are some unclear processes. For instance, it is unclear whether networks such as generators in DCAug serve as initializations for nnUNet components. Additionally, it is unclear whether the augmented data is synthesized in advance in an offline manner or in an online manner during the process of training the segmentation network.

  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    This paper uses several public datasets, and will provide training code, which is highly reproducible.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

    Firstly, this paper did not perform ablation experiments for each component to verify the necessity of each part. This is a critical point that authors should consider because it will help in identifying the specific contribution of each component to the overall performance of DCAug. Therefore, I recommend conducting such experiments and including the results in the paper.

    Secondly, the paper does not clearly describe how to combine data augmentation strategies with the underlying segmentation model. Specifically, it is unclear whether networks such as generators in DCAug serve as initializations for nnUNet components, and whether the augmented data is synthesized in advance in an offline manner or in an online manner during the process of training the segmentation network. I suggest that you provide a more detailed explanation of this process to clarify these aspects.

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    6

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    1. Novel method: The paper presents a novel approach for tumor augmentation, which combines a cross-cycle framework and two contrastive learning strategies (DaCL and CdCL) to address the content and domain distortion problem in synthetic tumors.

    2. Improved performance: The proposed method outperforms current state-of-the-art tumor augmentation methods in two challenging tumor segmentation tasks (KiTS and ATLAS), demonstrating its effectiveness in improving the quality of synthetic tumors.

    3. Contribution to medical imaging research: The paper addresses a critical problem in medical imaging research, which is the limited availability of annotated data. The proposed method can generate high-quality synthetic tumors, which can be used to augment the limited dataset for training deep learning models.

  • Reviewer confidence

    Confident but not absolutely certain

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #4

  • Please describe the contribution of the paper

    In this paper, the authors propose a tumor augmentation method in terms of avoiding content and domain distortions. Experiments were conducted on two publicly available datasets.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    +This paper demonstrates that the quality of sample generations is related to content and domain distortions. The proposed method learns both domain-specific information and domain-invariant information to improve the dice coefficients of tumor segmentation. +Experimental results show that this method achieves higher segmentation performance and also performs well on a small training set.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    -This paper lacks sufficient analysis to characterize the tumor image data itself, although experiments are conducted on the tumor dataset. -The writing of the article needs to be improved, there are some repetitive statements. For example, for DaCL and CdCL and description and some sentences in section 3.2. There are also some grammatical errors such as misuse of singular and plural. -The author claimed that the training/test split of ATLAS is identical with the study in [17], however, the results of CarveMix [17] reported in this paper are much lower than that reported in [17]. -About KiTS19 dataset, the results reported in this paper are significantly different from that reported in [19], (for some same works, the reported results in this paper are all over 72%, while the reported results in [19] are lower than 60%) it’s better to explain how those results of related works are obtained? -According to the rule of Supplementary Material, “Authors should not submit text materials beyond figure and table captions, definition of variables in equations, or detailed proof of a theorem.”

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    Datasets are public. Codes are not provided, it seems the code could be reproduced.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

    I think many parts of this could be improved. Some of them could be: 1) the writing could be concise and highlighted, for instance, the summarized contributions are a bit long. 2) the introduction of experiments setting could be more detailed, please refer to weakness part.

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    5

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Overall, I think this work is effective in experiments, however, the experiment setting is not that clear, and the reported results are significant to the results in previous works. So my current score is 5, I may change my score according to the authors’ response.

  • Reviewer confidence

    Somewhat confident

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #5

  • Please describe the contribution of the paper

    This paper proposes a cross-cycle tumor augmentation framework that utilizes two pairs of brain images and their tumor segmentation masks. To overcome domain and content distortion, they devise Domain-aware contrastive learning and cross-domain consistency learning, which distengles domain-information and content-information. Experiment on two datasets under limited annotation setting show that the proposed method has better performance than other augmentation methods.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper is well-organized and is easy to follow, especially considering that the authors address their contributions for several times.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work. 
    The manuscript fails to adequately explain the rationale behind the proposed method. Specifically, according to the definition of DaCL, the alignment is conducted on the image-level features, how is it supposed to be aware of the tumor localization and then make the generator only modify the tumor area?  •	The evaluation is insufficient, especially for ablation studies. Experiment The proposed cross-cycle framework is really complex, and the ablation study is missed to validate the effectiveness of each component in the framework. Also, I would like to see how the weight values for the total loss influence the performance, and why they are set as (1, 1, 1) ?
  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    There is nothing of the paper I’m concerned about in terms of reproducibility.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html

    • The idea of cross-domain consistency learning is similar with CycleGan paper “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks” J. Zhu, et al, I recommend that the author mention it as well as other related papers and discuss the difference. • Some core concepts in the paper is very ambiguous, like “individual-specific domain information” and “domain-invariant content information”, but “individual-specific” is related to “content”, it is hard for me to distinguish between them, it is better to give visual example of these two “information”; • What is i in equation (3) & (5)? • For the formula of L_{total}, I suggest to split it into two lines to avoid exceeding the pagewidth. • The standard deviation (64.64+/-29.91, for 100% Atals) is too large compared with the gap (0.64 = 64.64 – 64.00, DCAug and StleMix) between the mean dice scores of different methods, based on which we can hardly draw a conclusion that the DCAug is better than other methods. • Can the author explains why the proposed framework can improve Mixup and CutMix but hardly boost StyleMix?

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    3

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    I have concerns about the coherence and feasibility of the proposed method. The method presented in the manuscript is not clearly articulated and lacks sufficient evaluations to support its effectiveness.

  • Reviewer confidence

    Confident but not absolutely certain

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    3

  • [Post rebuttal] Please justify your decision

    According to the conference rules, no further experiment results should be provided or promised in the rebuttal. With that in mind, I think the original evaluation is insufficient to support the effectiveness of the proposed methods, especially when the results on ATLAS dataset is not convincing (The author only provides p-value for KiTS dataset).



Primary Meta-Review

  • Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The paper received mixed reviews with two (weak) reject and two (weak) accept recommendations. The area chairs considered the paper and the reviewers’ comments and agreed with the following strengths of the paper, such as the idea of using domain-specific information and domain-invariant information for tumor augmentation, the use of public datasets for evaluations and the agreement of releasing the code and data in the future. At the same time, there are several concerns with the papers (1) violation of the rules for supplementary material; (2) lacking evaluations in ablation study about the key modules, and comparisons with SOTA augmentation methods, (3) non-consistent results about baseline w.r.t. the reported results in published papers, (4) not clear writhing for some important parts of the paper. Based on the reviews, the authors are suggested to provide a rebuttal to address the reviewers’ concerns.



Author Feedback

We sincerely thank R2,3 for the acknowledgment of innovations (like “Novel framework”, “Novel strategies”, “Contribution to medical imaging”) and thank R2,3,4,5 for rating the effectiveness (like “outperforms state-of-the-art”, “significantly improves”).

Q1- Improvement is small over StyleMix (R2,5)
The significant difference analysis, p-Values (0.004, 0.03, 0.01 < 0.05) of paired t-tests between our model and StyleMix, proves the improvement is significant with 25%, 50%, 100% training data in KiTS19.

Q2- Model’s purpose and augmented data usage in segmentation (R2,3,5) 1) Generating diverse tumors for images by transforming various tumors from cross-modality or same-modality images (Sec.2.1 and Fig.1 of Supplementary), while overcoming content and domain distortion. 2) The augmented data is synthesized before segmentation model training, which overcomes the tumor scarcity challenge in tumor segmentation.

Q3-We have 3 contributions as stated by R3,4,5 (R2) 1) For the first time, addressing distortion challenge and improving the diversity and quality of synthetic tumor; 2) Two novel contrastive learning methods, DaCL and CdCL, enable separating tumor into content and domain;3) One novel cross-cycle architecture that enables learning cross-domain consistent features.

Q4- Recommended ablation studies on KiTS19 (R3,5) Results show that DaCL and CdCL improved the segmentation results by 1.53 and 7.06, respectively. And DaCL and CdCL together bring more significant improvement (10.48, p-Values<0.05). These improvements clearly demonstrate the advantage of DaCL, CdCL in tumor generation and improving segmentation performance. The performances (mean dice ± std (%)) are: without augmentation: 72.63±24.40, augment tumors by CdCL: 74.16±23.12, augment tumors by DaCL: 79.19±19.82; augment tumors by CdCL+DaCL: 83.11±14.15.

Q5- Characterizing tumor image (R2,4) Fig.1(1)(t-SNE) and Table.1 (Supplementary) characterize the feature distribution and quantitative difference between true and synthetic tumors, which shows our model addresses the distortion problem that existing methods cannot.

Q6-Results are different in published papers (R4) The distribution of num, position, size, and shape of generated tumors make the performance difference. But the same distribution setting in all methods and the independent segmentation model (nnUnet) ensured a fair comparison (Supplementary).

Q7-Localizing and modifying tumor(R5) The tumor mask (segmentation Ground Truth) is used (Eq.1,2,3,4 and Fig.2) to localize the tumor, which enables the model to focus on the tumor.

Q8-Influence of weights (α,β,γ) (R5) α,β,γ influence global, domain, and content information. Addressing the domain distortion (β) brings 3.12 improvement, and reducing the content distortion (γ) obtains 1.15 improvements. This clearly shows the necessity to address the distortion in the generated tumor. Experiments on KiTS19 are:α,β,γ=1,0.5,1:79.98±19.35;α,β,γ=1,1,0.5: 81.96±16.75; α,β,γ=1,1,1: 83.11±14.15.

Q9-Our cross-cycle has more advantages than CycleGAN(R5) 1)Maintaining cross-domain content consistency through a newly cross-interaction struct, CycleGAN cannot; 2) Improving domain representation ability of discriminator by constructing the novelty of cross-domain interaction data, CycleGAN cannot.

Q10-individual-specific domain, domain-invariant content (R5) The tumor’s domain is related to individual visual properties. It is specific and influenced by the feature distribution of the image and needs to be adjusted accordingly to prevent domain distortions. The tumor’s content should keep consistent and unaffected even be adapted to other domains to avoid content distortions.

Q11-Model improved Mixup and CutMix but hardly boost StyleMix(R5) Our model addresses domain distortion, the primary factor limiting the quality of generated tumors, unaddressed in Mixup and CutMix. StyleMix avoids this, but the improvement by our model over StyleMix is still significant (Ref Q1).



Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    This paper studied the tumor segmentation task and proposed a tumor augmentation method consisting of domain-aware and content-consistent constraints. The method is evaluated on two datasets to verify its effectiveness. The rebuttal provided additional details about the method differences w.r.t. previous methods, more experiment results, and ablation studies. The rebuttal address a number of concerns in the reviews. The remaining concern is the insufficient evaluations to support the claimed contributions.



Meta-review #2

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    This paper presents a novel approach for tumor augmentation called DCAug based on a novel Cross-cycle Framework and two novel contrastive learning strategies (DaCL and CdCL) to address the content and domain distortion challenges. Extersive experimental results on KiTS and ATLAS datasets demonstrate the proposed DCAug significantly outperforms the state-of-the-art methods and show improved quality of the synthetic tumors. The rebuttal has provided satisfactory responses and addressed the reviewers’ concerns regarding the extensive experiments, motivations, and experimental details.



Meta-review #3

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    As mentioned in the reviews, this paper is difficult to follow and the rationales of the proposed method are unclear. First of all, the problem is not well defined in the introduction. There are different types of augmentation methods such as manipulating a single sample, using generative models, or combining multiple samples. The problem is unclear until eq (1) and (2). The “domain gap” is the main issue to be addressed but it is not properly defined, and Fig. 1 is not helpful. Furthermore, the “domain” and “content” are not properly defined. Does the proposed method aim at combining images of different modalities or the same modality? For the same modality, how is the domain gap defined? Without proper definitions and rationales, it can be difficult to understand the advantages of the proposed strategies.

    In the rebuttal, the authors mention that “1) Generating diverse tumors for images by transforming various tumors from cross-modality or same-modality images (Sec.2.1 and Fig.1 of Supplementary)”. Such contents are not found in Section 2.1 and important definitions should not be put in the supplementary material. In fact, the word “modality” does not exist in the paper.



back to top