List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Kai Packhäuser, Sebastian Gündel, Florian Thamm, Felix Denzinger, Andreas Maier
Abstract
Robust and reliable anonymization of chest radiographs constitutes an essential step before publishing large datasets of such for research purposes. The conventional anonymization process is carried out by obscuring personal information in the images with black boxes and removing or replacing meta-information. However, such simple measures retain biometric information in the chest radiographs, allowing patients to be re-identified by a linkage attack. Therefore, there is an urgent need to obfuscate the biometric information appearing in the images. We propose the first deep learning-based approach (PriCheXy-Net) to targetedly anonymize chest radiographs while maintaining data utility for diagnostic and machine learning purposes. Our model architecture is a composition of three independent neural networks that, when collectively used, allow for learning a deformation field that is able to impede patient re-identification. Quantitative results on the ChestX-ray14 dataset show a reduction of patient re-identification from 81.8% to 57.7% (AUC) after re-training with little impact on the abnormality classification performance. This indicates the ability to preserve underlying abnormality patterns while increasing patient privacy. Lastly, we compare our proposed anonymization approach with two other obfuscation-based methods (Privacy-Net, DP-Pix) and demonstrate the superiority of our method towards resolving the privacy-utility trade-off for chest radiographs.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43898-1_26
SharedIt: https://rdcu.be/dnwAY
Link to the code repository
https://github.com/kaipackhaeuser/PriCheXy-Net
Link to the dataset(s)
https://nihcc.app.box.com/v/ChestXray-NIHCC
Reviews
Review #1
- Please describe the contribution of the paper
The authors present a deep learning based approach to anonymize (remove biometric linkages) chest x-rays without significant loss in diagnostic performance – at least compared other other popular approaches. On the Chest X-ray dataset there is a 24% drop in re-identification with only a 4% loss in diagnostic performance.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors point out that merely putting black masking boxes around identifier information on a chest x-ray is insufficient. It leaves the image prone to “linkage attack” – this part is not clearly defined for the lay reader. Noise introducing methods that are typically used are effective but also impact diagnostic ML training ability/performance. They authors propose a suite of three DL networks that ensure that linkability to patient identifier information is reduced without significant performance loss.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Not clear if the work is limited to chest x-rays. What about DICOM data / hounsfield units? Images were resized - what if they were not? How would performance deviate? What if the images were to be deidentified without a built-in diagnostic classifier?
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper is reproducible to the extent that sufficient detail is provided for someone to repeat the experiment if they are knowledgeable in the background. However, code is desirable with comments. Datasets used are publicly accessible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Some more detail about disambiguating the networks and their interplay with each other is necessary to see how your excellent work could be translated into general practice. What is the sensitivity to image size? What is the sensitivity to image format / compression? You have tested classification - what about segmentation? What is the loss to subtle / early stage disease?
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The authors address an important topic. They present fairly compelling results. The paper is well written. The supplementary images are helpful.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The paper introduced a Deep Learning based approach that is used to anonymize chest radiographs while maintaining data utility for diagnostic and machine-learning purposes. The proposed architecture contains three independent neural networks that are trained for learning a deformation field that is able to impede patient re-identification. Experiments on the ChestX-ray14 dataset showed that the approach reduced the rate of patient re-identification by >20% (in AUC) for the task of chest X-ray classification. In addition, this approach surpassed two other obfuscation-based methods (Privacy-Net, DP-Pix) in resolving the privacy-utility trade-off for chest radiographs.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The work aims to resolve the privacy-utility trade-off by proposing the first adversarial image anonymization approach for chest radiography data. The deep learning system is well-designed.
The main contribution is the use of a U-Net network to generate a flow field used to deform the original image. This flow field is the key to the robust and reliable anonymization of chest X-ray scans.
The performance of the proposed system was much better than the baseline model.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Major concerns:
-
The proposed approach did not well, comprehensively validated since it only was tested on Chest X-ray 14. To evaluate the effectiveness of the method, the authors are highly recommended to validate the approach on more CXR datasets, e.g. CheXpert, MINIC-CXR, and VinDr-CXR.
-
There are no extensive experiments and ablation studies show that the learned flow field F performs well and better than other noise distributions.
-
The patient re-identification performance was not comparable with PD-Pix
-
Can we train the whole network in an end-to- end manner?
-
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The dataset used in this study is publicly available. The author did not mention about the code of this work.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Bellow are some key concerns about your work:
-
The proposed approach did not well, comprehensively validated since it only was tested on Chest X-ray 14. To evaluate the effectiveness of the method, the authors are highly recommended to validate the approach on more CXR datasets, e.g. CheXpert, MINIC-CXR, and VinDr-CXR.
-
There are no extensive experiments and ablation studies show that the learned flow field F performs well and better than other noise distributions.
-
The patient re-identification performance was not comparable with PD-Pix
-
Can we train the whole network in an end-to- end manner?
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
These key factors include the novelty of the approach and the correctness of experimental settings as well as the reported results.
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The paper introduces PriCheXy-Net, a novel deep-learning-based approach for anonymizing chest radiograph data using an adversarial framework. The proposed model includes three independent neural networks that learn a deformation field to prevent patient re-identification. The authors evaluate their approach on the ChestX-ray14 dataset and report a reduction in patient re-identification from 81.8% to 57.7% after re-training. The proposed approach is compared against Privacy-Net and DP-Pix as baselines.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The proposed PriCheXy-Net approach effectively reduces patient re-identification from 81.8% to 57.7% (AUC) after re-training, outperforming comparison methods such as Privacy-Net and DP-Pix. The authors provide a comprehensive introduction to clearly explain the motivation for the study. PriCheXy-Net is a new deep learning-based approach for anonymizing chest radiographs while maintaining data utility for diagnostic purposes.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The proposed PriCheXy-Net method has only been evaluated on a single dataset, the ChestX-ray14 dataset, which may limit the generalizability of the results to other datasets or imaging modalities.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper is well written and seems reproducible. Also, the paper used chestX-ray14, a publicly available dataset. I will suggest releasing the code.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
The paper provides an approach for preserving privacy in chest radiographs while maintaining their diagnostic utility, which can be useful in preserving patients’ privacy.
It might be useful to evaluate the performance on the approach on multiple datasets.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper presents both clinical and technical contributions, which could benefit the research community. However, a concern is the reproducibility of the results. It would be beneficial if the authors could release the code to enable others to replicate the study. Also, the approach was evaluated on a single dataset.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The work presents a new deep learning based method for anonymising chest radiograph data using an adversarial framework. The method is well-designed and shows promising performance. The main concern of the work lies in its insufficient validation on dataset - only single dataset was evaluated on, which weakens the contribution of the work. Please clarify on this point. In addition, please also provide clarifications on the performance of the learnt flow field compared to other noise distributions and provide some details on the experiment settings as questioned by the reviewers in the rebuttal.
Author Feedback
We thank all reviewers and the meta-reviewer (MR) for the valuable comments on our manuscript. We appreciate the overall positive assessment and the agreement on the novelty and importance of our work. Moreover, we acknowledge the recognition of our clinical and technical contributions in tackling the privacy-utility trade-off for chest X-ray data. The main points to address - as noted by the MR - include the evaluation design and some clarifications on experimental settings. These will be resolved in our response below.
Evaluation (R2, R3): R2 and R3 raised a concern regarding the sole evaluation of our proposed anonymization approach on the ChestX-ray14 benchmark dataset. While we acknowledge the significance of testing our method on additional datasets, the primary focus of our study remains on comparing with relevant baseline methods. We deem the direct comparison with DP-Pix and Privacy-Net as most pertinent to assess the performance of our method. Thus, we conclude that our evaluation design is valid to show that our proposed approach yields superior results than the current state-of-the-art. We agree with R2 and R3 that validating our method on additional datasets would further enhance its evidence. However, these experiments are subject to future work and will be incorporated in an extended follow-up article.
Clarifications (R1, R2): R2 asked about the feasibility of training the entire network in an end-to-end fashion. In our study, we follow a strategy to pre-train the individual network components prior to the actual training run to enhance training stability and convergence speed. This pre-training is particularly important for the incorporated auxiliary classifier and the verification network, as they play integral roles as loss terms and provide guidance to the flow field generator during optimization. Once initialized with pre-trained weights, our proposed architecture undergoes end-to-end training using the objective function presented in the paper. These details will be explicitly clarified in the final version.
As noted by R2, we also aim to clarify the performance of the learned flow field compared to other noise distributions. Our study demonstrates that the proposed method surpasses both learning-based (Privacy-Net) and non-learning-based (DP-Pix) baselines. While DP-Pix internally employs Laplace noise, it is theoretically feasible to utilize alternative noise distributions as well, e.g., Gaussian noise. However, employing any noise distribution without domain-specific mechanisms significantly impairs image utility by erasing important diagnostic patterns. In contrast, our approach precisely addresses this concern by leveraging the flow field and its imposed constraints to achieve minimal deformation while effectively obfuscating patient-specific biometrics in the images.
To address the concerns of R1, we hypothesize that our method is robust to variations in image size or compression rate. The inclusion of additional layers could potentially compensate for higher resolutions, which may only impact network capacity and training speed. Furthermore, we presume our proposed system extends beyond chest radiographs to other imaging modalities. However, confirmation of these hypotheses requires further investigation, which falls outside the scope of this study. We will include these points in the discussion section of our paper.
Reproducibility (R1, R2, R3): As we fully share the idea of open research and aim to advance this - until now underexplored - research area, we will release our source code (alongside all the trained models) once the paper is accepted. A link to our GitHub repository will be added in the final version, making our study fully reproducible and transparent to the public.
Overall, we are convinced to have successfully dispelled all major concerns. We are confident that the requested clarifications and further work on minor comments will lead to a valuable contribution to MICCAI 2023.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The rebuttal is clear in responding to the concerns that have been raised. Though the rebuttal has highlighted that ‘the primary focus of the study is on comparing with relevant baseline methods’, which is true for most of the work, comprehensive validation would still be necessary to realise that. Though evaluating on single dataset could be fine, some experiments such as ablation studies would then be desired to get some insights on the work. However both are lacking from the existing work and more comprehensive validation is therefore suggested.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The proposed adversarial framework for deep-learning-based anonymizing chest radiograph data appears to be novel and outperforms two baseline methods on a publicly available benchmark. While the its technical novelty of the approach is limited (a deformation field is learned to destroy patient identification while preserving and classification accuracy) and only evaluated on a single database, the work is interesting and the paper does not contain any major flaws. Moreover, the authors do a good job in their rebuttal to clarify concerns raised by the reviewers.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper proposes a method to reduce the risk of re-identification of patients from their chest X-rays. The method is relatively straight-forward. My main objection to the paper is its importance. Specifically, I think the risk of patient re-identification, and in general, adversarial attacks in medical imaging have been over-stressed. I think in practice these risks are not really serious, unlike security-critical applications. Second, I think the results are not very good. Basically the main result is that there will be 4% drop in disease classification accuracy, while reducing the risk of re-identification by less than half. If I was the patient, would I prefer this? No.
Overall, well-written manuscript that does not meet the quality/importance level of the main conference.