List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Kimihiro Yamazaki, Yuichiro Wada, Atsushi Tokuhisa, Mutsuyo Wada, Takashi Katoh, Yuhei Umeda, Yasushi Okuno, Akira Nakagawa
Abstract
Structural analysis by cryo-electron microscopy (Cryo-EM) has become well-established in the field of structural biology. Recently, cutting-edge methods have been proposed for the purpose of reconstructing either a small set of structures or a conformational pathway (continuous structural change), where a 3D density map represents the structure. However, we usually perform heavy manual labor to define the plausible pathway related to biological significance. In this study, for automatizing such manual labor, we propose a deep Auto-Encoder (AE) with a trainable prior. The AE is trained using only a set of single particle Cryo-EM images. The trained AE reconstructs the corresponding structures for the latent variables of the Cryo-EM images. The latent distribution can not only be theoretically proportional to a distribution of the structure but also consistent with the trained prior. Taking advantage of this property, we can automatically compute the pathway by only accessing the latent space as follows: i) generating a ridgeline on the latent distribution and ii) defining the conformational pathway as a sequence of the reconstructed structures along the ridgeline using the trained decoder. In our numerical experiments, we evaluate the computed pathways by comparing them with existing ones that were manually determined by other researchers, and confirm that they are sufficiently consistent.
Link to paper
DOI: https://doi.org/10.1007/978-3-031-43907-0_38
SharedIt: https://rdcu.be/dnwcP
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
Cryo-Electron Microscopy (Cryo-EM) in is establishing its position as a major structural analysis technique along with X-ray crystal structure. The analysis of protien structures is quite difficult and the authors propose a method based on an Autoencoder.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The Cryo-Electron Microscopy technology is interesting, the analysis of these data is difficult. The application of the autoencoder is an interesting approach which has been exploited by other works such as RaDOGAGA.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The evaluation and and the dataset appears to be quite reasonable. There are some clear performance imrpovements.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Reproducibility is fine
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
Overall ths is a well-written and interesting paper with quite good results in comparison to SOTA methods.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This is solid work which could be of great interest to a few readers.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
Conformational pathways represent changes in protein’s structure. A single structure is represented by a spatial 3D electron density volume of a protein. In existing approaches, coformational pathway identification is tedious and involves manual work.
In this work, the authors aim at automating this process, leading to two contributions. First, they augment an existing generative model of density volumes to promote isometric mappings between the latent space and the space of density volumes. Under an isometric mapping, probabilities of a path in the latent space and in the volume space become proportional to each other. This justifies the second contribution – an algorithm to search for conformational pathways in the latent space only.
Both theoretical and empirical results support the claim that the proposed mapping is indeed more isometric than the existing one. Furthermore, the authors compare automatically identified pathways to the ones manually found in the literature.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- the use of isometric mappings for representation learning is novel in medical domain
- while the derivations for isometry are borrowed from existing work, they are adapted to the problem at hand (which is not trivial)
- assumptions in the theoretical claims are checked empirically
- the authors show theoretically and empirically that the proposed mapping is more isometric than the existing one
- automatically found pathways correspond to the ones found in the literature
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
While the isometry is well-founded and analyzed, the emerging pathways leave some open questions:
- emerging pathways are subject to selection of starting and end points, which the authors select based on the learned latent prior distribution. Fig. 2 suggests that many start/end points can be found, leading to many possible paths (in particular, many more than the ones listed in Fig. 7 in [4] for example). Are these paths “false positives”, or is their existence justified? Maybe the starting points can be derived based on some biological principles?
- while intuitively it makes sense that an isometric mapping leads to better paths than a non-isometric mapping, unfortunately no empirical evidence for that is provided
- emerging pathways are also subject to labeling, which is done via visual inspection. To a non-expert in protein volume density maps like myself, it is unclear, how subjective this process is. For example, differences between E2 and E4 are hard to identify (even after inspection of Fig. 4 and 7 in [4])
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The work is reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- provide some context for the 1. start/end point selection problem and 2. subjectivity of the labeling process in the rebuttal
- for a given fixed set of starting and end points from related work, the authors could compare the emerging paths from Alg.1 under cryoTWIN vs cryoDRGN, and how these paths compare to the ones from [4]. This would further empirically support claim that isometric mappings are important for path search.
- please make the problem statement more understandable to non-experts in cryo-EM imaging. [26] is a very good example. I am aware of the page limit, however a clearer presentation will allow a larger audience to take away the advantages of isotropic mappings.
- polish the text with a native speaker, if possible.
Minor remarks:
- I would refrain from using the term “autoencoder”. An autoencoder typically reconstructs its input, which is the case in [8], but not in your work. This term led to an initial confusion in my case.
- minor typo in caption of Figure 3: “B4” is not present in the figure.
- please make sure to publish the code upon acceptance
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The isometric mappings for representation learning provide an interesting way to reason about the probability density of the data. I see the benefit for the community from its demonstration in this work. I am just not entirely convinced that the chosen application of automated path detection is treated sufficiently.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
I do agree with the authors that automatic extraction of paths from cryoDRGN’s latent space (as a baseline) will most likely lead to implausible paths. The provided experiments sufficiently illuminate the isometry of the learned mapping (compared to cryoDRGN) in my opinion. Therefore, I upgrade my score to “accept”.
Review #5
- Please describe the contribution of the paper
The authors present an auto encoder model suitable for Cryo-Electron Microscopy data.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- new method for computing the conformational pathways for Cryo-EM Images
- the paper takes the right approach to suggesting a new method, starting from theoretical guarantees and motivations to the architecture
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Introduction
-what is a ‘conformational pathway’ —> Definition or earlier explanation -Make even clearer why this is an important problem and come back to it in the conclusion
Related Work
2.2 -Define the latent variable z -an earlier illustration would help understanding the method better in this part of the paper
“To understand the difference with RaDOGAGA, let V (resp. I) denote a 3D density map (resp. the projected image). Suppose that both RaDOGAGA and cryoTWIN are trained by a set of the images. Then, RaDOGAGA only theoretically guarantees that the latent space is isometric with a space of I. By contrast, cryoTWIN can theoretically guarantee that the latent space is isometric with a space of V . “ Can you elaborate on this difference. As far as I understand only the input/output space is different. The arguments (Nash embedding theorem, manifold hypothesis, cf. Intro RaDOGAGA paper) in the RaDOGAGA paper are very general and hold for different spaces (also for a space representing a 3D density map).
Method
- it is not clear to me whether Theorem 1 is a contribution of the authors or a derivative of another theorem to this special case.
- the paper is not always self-contained and therefore hard to read. E.g. Fig. 2 “The red point expresses one of means with Pψ∗ , and the “B” to “E” express the structural label defined in [4]. “
- Fig 2 does not show the mentioned red point.
-Algo 1: as far as I understand you mainly show the idea of decoding discrete points on a geodesic in the latenspace here. I think this is very intuitive and easy to understand and you could save space to fill it up with experiments. If I have not overseen anything here, the reader does not get much out of the algorithmic expression here.
Results
-There are very few experimental results shown in the paper. I also suggest structuring this part with Tables and Figures to make in more clear and easy to grasp for the reader.
-There is no comparison to benchmark models? Is this because there has been no architecture suggested for this task? Are there models that can be easily adapted for the same task to which you could compare?Conclusion
“Our model has several beneficial properties for the computation. “ - In my opinion the paper would benefit from removing such sentences that carry basically no information.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- code was provided
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
- please refer to the comments on strength and weaknesses
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
In my opinion the paper needs a restructuring, including a stronger results section and a clearer delimitation to other works. It would also benefit from an even clearer description of its Figures. From my perspective it is not in a state to be published in the current version.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #6
- Please describe the contribution of the paper
This paper incorporated rate distortion optimization guided autoencoder to facilitate study of structural heterogeneity in cryo-EM. The latent distribution of such autoencoder is theoretically proportional to the structure distribution and consistent with the trained prior. With this unique property, this paper proposed a method to automatically compute a conformational pathway for structural change.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- While rate distortion optimization guided autoencoder is not new, the application of this autoencoder to solve heterogeneity problem in cryo-EM is novel. It is able to leverage the unique property of this autoencoder to facilitate and with the potential to automate the investigation of structural conformation pathway, which is a critical problem in structural biology and can have significant impact.
- The proposed method is theoretically sound.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- I am not sure if Theorem 1 proof is necessary. In the original RaDOAGAGA paper, the proof for theoretical guarantee is not restricted to 2D, instead, it is a more generalized proof.
- The quality of 3D reconstruction in cryo-EM is heavily dependent upon the estimation of orientations. Here, estimation of these angles are done through a pre-processing steps. While the authors performed experimental comparisons with cryoDRGN, I’m not sure if the particle image used and the orientation angles are the same as cryoDRGN. These can easily affect the performance of the algorithm.
- In Figure 2, there are clusters labeled as B,C,D,E. However, in Figure 3, the conformation pathway begins with C and there is no B. Why is this? I want to make sure I am not misinterpreting the figures.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Somewhat reproducible
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2023/en/REVIEWER-GUIDELINES.html
I would like to give comments based on the weaknesses section.
- Overall, I found the paper a little bit hard to read, mostly due to notation reasons. There are too many variables names and many of them are concentrated in one short paragraph. It will be better if some of them can be break into multiple parts.
- It will be better to add a section on how orientation estimation (preprocessing step) can affect the performance of the proposed method.
- I would like some clarifications on Figure 2 and 3.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper is solving an important problem in structural biology and the way it is incorporating rate distortion optimization guided autoencoder and leverage its unique properties is indeed novel.
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
The authors addressed most of my concerns. one small thing, is there a typo in Figure 3 legend - it says C2 and B4.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper suggests using an auto-encoder for Cryo-EM data with an isometry assumption that ensures that the modelling of conformational pathways can be done using the latent space.
The reviewers have mixed feelings about this paper – on the one hand, they find the modelling nice and the results convincing; on the other hand, they also have a range of concerns, which will have to be addressed in the rebuttal:
- Reviewer 2 lists (under “weaknesses”) a number of concerns regarding modelling assumptions as well as our ability to validate the quality of paths.
- Reviewer 3 has concerns regarding the experimental validation
- Several reviewers are concerned regarding the clarity of the paper
Author Feedback
We thank all reviewers for their valuable comments. We will carefully improve the readability based on all the concerns raised by the reviewers, including clarification, fixing typos, and native-speaker checks.
Q1: Contribution of Theorem 1 (R5, R6) A1: Theorem 1 shows how to provide the latent space which is isometric with a space of the 3D density map, even though the input and output of the auto-encoder during the training are not a 3D density map but a Fourier image. RaDOGAGA only guarantees the isometric embedding in the given metric space. Theorem 1 shows the metric defined by radius-weighted squared L2 with a Fourier image (distortion in Eq.(3)) corresponds to a squared L2 metric of the 3D density map. We will add this explanation to Sec.2.3.
Q2: Biological principles for start/end points / Justification of created paths (R3) A2:The start/end points of the emerging paths in Fig.3 were selected based not only on the higher Gaussian weight but also on the biological principles. About the justification, first, the number of the possible paths via Alg.1 in our experiments is 100 choose 2; namely 4950 where 100 is the number of Gaussian components. Among them, we confirmed the existence of the paths whose start/end points are based on the principles, via visual inspection using [4]. For the remaining paths, we either did not or could not evaluate the existence; some of the unevaluated paths may lead to novel biological findings. The reason for “could not” is that sufficient supervised data do not exist. We at last remark that, in our preliminary experiments, we confirmed the existence of paths by Alg.1 whose start/end are only selected via the higher Gaussian weight.
Q3: Comparison between cryoTWIN and cryoDRGN as empirical evidence (R3) A3: Alg.1 is not directly applicable to cryoDRGN since the prior is a standard normal distribution N(0, I) with a single peak, and both the starting and ending Gaussian peaks within GMM cannot be provided to Alg.1. Furthermore, the evaluations of isometricity and KL divergence show that the prior of cryoDRGN does not match the empirical distribution of the conformation, implying that paths using cryoDRGN prior are implausible. That’s why we did not compare them as empirical evidence. We will add this explanation to Sec.4.
Q4: How subjective our annotation process is (R3) A4: For the ribosomal dataset, we understand the expert tends to annotate the structural labels objectively. In the case of a) & b) in Fig.3, the annotation was performed by our experts using the known structural labels of [4] and PyMOL. We will add this explanation to Sec.4. Q5: Visibility of Fig. 2 (R5) A5: We will make the red dots recognizable in black and white printing.
Q6: Comparison with benchmark models (R5) A6: In Sec.4, we will emphasize that cryoDRGN [25] is compared with cryoTWIN as a benchmark.
Q7: Need stronger results (R5) A7: Due to limited space, we could not show sufficient results. If accepted, we’ll further submit our method to a journal with more persuasive results.
Q8: Orientation angles in the experiments (R6) A8: CryoTWIN and cryoDRGN use the same particle images and angles. We will emphasize this point in Sec.4. We will also discuss in Sec.5 how important the orientation estimation is in the preprocessing.
Q9: Conformation pathway starting from B (R6) A9: In Fig.3, we aim to check whether the sub-paths of the four paths in [4] can be generated by Alg.1. Given B and E as start and end points, Alg.1 estimates only one path in Fig.7 of [4] (B-C2-E1-E2-E4-E5) as the most probable ridgeline. To reproduce the four paths, our strategy is to aggregate sub-paths generated by Alg.1, whose start/end points are significant; thus we demonstrated the reproducibility of the two sub-paths (both not starting at B) by Alg.1 in Fig.3. Note that all four paths were reproduced by the aggregation in our preliminary experiments. We will add this explanation to Sec.4.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper suggests using an auto-encoder for Cryo-EM data with an isometry assumption that ensures that the modelling of conformational pathways can be done using the latent space.
The reviewers found the modelling nice and the results convincing; but they also had concerns, in particular pertaining to the validity of the modelling assumptions, and our ability to validate the quality of paths. These concerns were met in the rebuttal. While the paper remains borderline – in particular as concerns regarding the limited experimental validation remain – I believe it is sufficient to recommend acceptance.
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper presented an auto-encoder method for analyzing the conformational pathways for Cryo-EM Images. In general, the presented method is effective as shown from the experiments.
While R#5 raised several concerns from a few different aspects, the rebuttal has largely addressed them. Considering the strengths as listed by all the reviewers, this paper can be accepted by the conference. Please the authors check R#5’s review comments and take into account them when preparing the final paper.
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper uses an auto-encoder in handling Cryo-EM data while making an isometry assumption that enables the exploration of conformational pathways within the latent space. The reviewers were generally impressed with the modeling techniques and found the results to be compelling. However, they did express some reservations regarding the clarity of certain aspects and the presentation of results. In response, the authors have adequately addressed these concerns in their rebuttal, leading to an improvement in the reviewers’ evaluation, with two of them upgrading their score to “accept.” Considering the rigorous methodology, the intriguing nature of the problem tackled, and the incorporation of suggested changes and clarifications from the reviewers, the paper appears to be suitable for publication at MICCAI.