Paper Info Reviews Meta-review Author Feedback Post-Rebuttal Meta-reviews

Authors

Haomiao Ni, Yuan Xue, Kelvin Wong, John Volpi, Stephen T.C. Wong, James Z. Wang, Xiaolei Huang

Abstract

Accurate infarct segmentation in non-contrast CT (NCCT) images is a crucial step toward computer-aided acute ischemic stroke (AIS) assessment. In clinical practice, bilateral symmetric comparison of brain hemispheres is usually used to locate pathological abnormalities. Recent research has explored asymmetries to assist with AIS segmentation. However, most previous symmetry-based work mixed different types of asymmetries when evaluating their contribution to AIS. In this paper, we propose a novel Asymmetry Disentanglement Network (ADN) to automatically separate pathological asymmetries and intrinsic anatomical asymmetries in NCCTs for more effective and interpretable AIS segmentation. ADN first performs asymmetry disentanglement based on input NCCTs, which produces different types of 3D asymmetry maps. Then a synthetic, intrinsic-asymmetry-compensated and pathology-asymmetry-salient NCCT volume is generated and later used as input to a segmentation network. The training of ADN incorporates domain knowledge and adopts a tissue-type aware regularization loss function to encourage clinically-meaningful pathological asymmetry extraction. Coupled with an unsupervised 3D transformation network, ADN achieves state-of-the-art AIS segmentation performance on a public NCCT dataset. In addition to the superior performance, we believe the learned clinically-interpretable asymmetry maps can also provide insights towards a better understanding of AIS assessment. Our code is available at https://github.com/nihaomiao/MICCAI22_ADN.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16452-1_40

SharedIt: https://rdcu.be/cVVpV

Link to the code repository

https://github.com/nihaomiao/MICCAI22_ADN

Link to the dataset(s)

N/A


Reviews

Review #1

  • Please describe the contribution of the paper

    This work develops a segmentation method for stroke infarcts in brain CT based on asymmetry. It differs from previous works on asymmetry by developing a method to differentiate between 2 asymmetry types: Pathological (stroke) and non-pathological (anatomical). The steps are: 1. A registration net learns to align the CT with mid-sagittal. 2. Asymmetry map made via CT’s reflection along mid-sagittal. 3. A second net takes the CT and predicts only the pathological asymmetries. This is learned via an intuitive regularizer designed based on domain-knowledge (main technical contribution). 4. Using output of (2) and (3), a synthetic “asymmetry compensated” image is made, where non-pathological asymmetries are removed. This is beneficial cause it can serve as human-interpretable side-output (secondary contribution). 5. A third network learns to segment given the “asymmetry compensated” image (4). Evaluated is done on a public database, AISD, outperforming 3D (Res)Unet and DeepMedic.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The main premise of the work, developing a method for leveraging asymmetry to detect pathology, is a very good one. It could potentially be useful for the detection not just of stroke, but others as well. And developing such methods could be useful to enhance the strong general-purpose learning-based models (networks) commonly used nowadays.

    • The method is well thought, well-designed, and well implemented. For this, I identify 2 components: 1) the regularizer for learning to predict a pathology-asymmetry is intuitive and nicely-designed for the purpose. 2) The framework for combining image X, all-asymmetries map A, and pathology-asymmetries map P, to get a get a synthetic image (asymmetry compensated X_comp) where the non-pathology-asymmetries are (pseudo-)removed (X_comp = X+A-P) is a well-designed one for the purpose.

    • The method gives a secondary output, the asymmetry-compensated image, where only the pathology-asymmetries should be visible, not the non-pathology ones. This is the image that is given to the segmentation network (primary output being the segmentation). This image makes the segmentation model more interpretable by humans. (shows why the segmenter predicts what it does). I find this a very nice advantage.

    • The results seem ok, outperforming 2 standard general-purpose segmentation models, the 3D CNN DeepMedic and the 3D (Res)Unet, and some stroke-specific segmentation models [9,18].

    • I liked that the authors re-implemented and adapted some of the previous methods [9,18] to make the comparison fair (e.g. [9] from 2D to 3D). Makes me more confident that they put some effort to make good baselines.

    • The paper is very well written and explains the method clearly. It was a pleasant read.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    • The method is task (Stroke segm) and modality (non-contrast CT) specific. This limits the audience of interest. It is based on (at least) 2 assumptions: 1) All the asymmetries of interest are dark. This is mentioned briefly in method section and discussion (doesn’t find bright blood). 2) The pathologies appear as “additive” intensity changes (X_comp=X+A-P). For example, it cannot deal with pathologies that create morphological changes (e.g. tumor). The work would benefit if a) task-specific assumptions were discussed very explicitly, b) if it would discuss how and to what tasks it could be potentially extended. E.g. multiple-sclerosis / leucoaraiosis in Flair MRI (though bright).

    • Evaluation is rather limited. Only 1 experiment per method. No runs with multiple seeds. So we aren’t sure whether improvements are by a “lucky seed” or consistent. No train/val/test setup, only train/test. So it’s not clear whether improvements are “best run” with optimally configured hyper-parameters on test set. This could be improved by averaging over multiple seeds per method, and by discussing how hyper-parameters were found for the method and the compared methods.

    • It is unclear whether some components of the method are adopted or proposed. In good scientific writing, this should be crystal clear. This needs fixing. Examples: In Sec.2, Transformation Network: Eq.1 is adopted from [18] but this is not explained. The text only mentions the difference that the model here is 3D whereas [12] is 2D. Please state explicitly that Eq.1 is adopted from [18]. Also, in rebuttal, please state explicitly if any other parts of the method are adopted (Eq.3? Parts of regularizer?), and same in text.

    • The regularizer has multiple parts. There is no ablation study whether each individual part is important. Knowing which part is important would help extend this work in the future. For example, one could see if the parts that are most important are applicable to another task and perhaps employ them for a different pathology/modality.

    • Parts of the regularizer could be straightforwardly applicable as a regularizer of the segmenter, and not just the asymmetry-predictor D. In fact, some of them may be the cause of segmentation improvement, instead of the asymmetry map generation. Example: For example, penalizing the pathology when it appears on CSF (1st part), or the regularization of the average size (2nd part). This has not been studied. There could be a comparison/ablation where those parts are tried on 3D ResUnet to see if they improve it, to identify whether the improvements come from the asymmetry-image generation or the regularization of the size/location of pathology. The authors & article could explicitly discuss this.

  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    (based on the article alone)

    I think the article describes the method sufficiently clear, and gives some of the hyper-parameters, so that a reader can sufficiently reimplement the core method. Although the 3D ResUnet itself may not be able to be replicated (not all hyperparameters are here such as width of layers etc).

    Less reproducible are likely the results themselves, as they are only 1 run per experiment, which means improvements may be subject to the initialization seed, etc.

    Overall, average.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

    In the rebuttal, the authors should aim answering primarily about the weaknesses I raised in previous section. I below provide some additional points, that they should also clarify at least within the article in case of publication, if they do not find space to clarify them in the rebuttal.

    Sec 2, “However, this can lead to a trivial solution… network.”: I dont think that this would happen, because (I think) gradients of D are not a function of A (because of the addition). So D does not receive information about the shape of A. It just processes X and learns to “highlight” specific parts of it (pathology assymetries) to enable segmentation. E.g. if output of D is 0, then X+A-0 = an image without asymmetries, so segmentation is impossible. Therefore to enable segmentation, D learns to “highlight” (P) the pathological-assymetries. It would be good to have an experiment to study this (perhaps I am wrong), so that the reader understands the method’s behaviour: Just run the whole framework with regularizer = 0. I believe it would still predict the pathology-assymmetries, but with more false positives, which the regularizer resolves via the size/location penalties. If the authors agree that this sentence of the article is not 100% true, then consider replacing this statement.

    Sec 3, it is unclear how the tanh activations predict rotation and translation degrees, and how rotation is restricted to 60 degrees. Do you associate -1 and +1 of the output tanh to correspond to e.g. -60 and +60 degrees rotation, and something similar for translation? Please clarify.

    “warm-start strategy … for training D.”: Please clarify this a bit more. Do you add supervised cross-entropy loss with G at the very output of D? Which output of D do you penalise to look as G? P? E.g. via L_bce(P,G)? If so, please clarify within text, e.g. by writing “L_bce(P,G)”. Will help reproducibility.

    In the regularizer of “mean size”, is mean(T(G)) computed for each case (G) separately at each iteration, or is this mean computed over the whole database (mean of all subjects)? Please clarify (also in article). Note that if it is over whole database, it biases the network, which is less likely to predict extreme cases (too small or too big infarcts), in which case it should be discussed.

    (minor, in case you find this helpful) I think the title is a bit confusing. Most readers, and this reviewer included, when they see “Disentanglement Network”, they will assume it is a paper focusing on methodology how to learn to disentangle in a data-driven manner. Here, the method does not really learn how to disentangle one different asymmetries. Instead, the asymmetries are actually modelled (total asymmetry = X-reflect(x), pathological asymmetry via the regulariser, non-pathological via A-P) and the model learns to predict pathological asymmetry P. A less confusing title could be something like “Pathological Asymmtry Prediction Network for…” or something like that.

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    6

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper has significant strengths, an intuitive idea, clear presentation. It has weaknesses, but the strengths outweigh them I believe.

  • Number of papers in your stack

    5

  • What is the ranking of this paper in your review stack?

    1

  • Reviewer confidence

    Very confident

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    6

  • [Post rebuttal] Please justify your decision

    The authors have answered some of my questions. They clarified that most parts are their own contributions (as per the original text), and they commited to add clarification about Eq1 and differences with [18] in the camera ready, if accepted.

    They also clarified about the non-existence of a validation split, and they explained metaparameters were not really configured, but rather mostly adopted from previous works. I hope and will urge them to provide such clarifications in the revised version of the text as this is important to the reader.

    They also provided examples of applications where their method could be of interest. I hope such examples will be added to the revised version, if accepted, as they will increase the value of the work.

    Overall, the rebuttal has re-assured me about certain reservations that I had about the work. I still think that the paper has some weaknesses (e.g. limited experiments, no significant ablation/sensitivity study, only 1 seed per experiment which makes significance of results uncertain, etc), but I am confident that the work is of quality good enough for publication.



Review #2

  • Please describe the contribution of the paper

    the authors presented an asymmetry disentanglement network to provide extra supervision for the stroke segmentation on non-contrast-enhanced CT images.
    the authors demonstrate the effectiveness of the disentanglement module.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. an asymmetry disentanglement network based on domain knowledge to provide extra supervision.
    2. improved numbers by comparing with existing methods.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    1. Narrative application Although the proposed method is interesting, aiming to disentangle two kinds of asymmetries and reduce the artefacts of input images, this approach might be narrative to stroke segmentation on noisy CT images, which rely on such a prior on asymmetry to improve the input images.

    2. Potential issues: The authors propose to modify the input image Xˆ by Xˆ = X + Q = X + A - P. This might be problematic as it might generate false positives due to the intensity range difference between X and Q (or X, A and P). A clarification or discussion on this might be needed.

    3. Results and Clarifications. x. The author mentioned that the transformation network T can be unsupervised. I am missing the details of such a training strategy. From my understanding, this is similar to regress the rotation angle of an object in an images. One could simulate an angle (transformation matrix in the authors’ case). However, this would require some ‘object-centered’ images. Essentially, a question would be: how good the network is to regress the parameters? The transformation module is very important in the proposed framework as it generate A, which is base for next steps. x. For training D, is there any weighting strategy for four loss terms? x. How much does the segmentation maps (WM, GM, CSF) help as a regularization? x. For self-mirrored version X’, which plane/view is used?

  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    seems reproducibable.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
    1. Applications Not sure if it is relavant to include MR image for stroke segmentation such as ISLES-2018/2016. More datasets on demonstrating the idea would be helpful.

    2. Potential issues: The authors propose to modify the input image Xˆ by Xˆ = X + Q = X + A - P. This might be problematic as it might generate false positives due to the intensity range difference between X and Q (or X, A and P). A clarification or discussion on this might be needed.

    3. Clarifications. x. Details of the training strategy of the transformation network T. Essentially, a question would be: how good the network is to regress the parameters? The transformation module is very important in the proposed framework as it generate A, which is base for next steps. A clarification on this would be needed. x. For training D, is there any weighting strategy for four loss terms? x. For self-mirrored version X’, which plane/view is used?

    4. Presentation X in second row of Fig. 4 is not transformed while X+Q is transformed.

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    4

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Good presentations of a sophisticated approach. However, some details and potential issues need clarifications.

  • Number of papers in your stack

    3

  • What is the ranking of this paper in your review stack?

    2

  • Reviewer confidence

    Confident but not absolutely certain

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    4

  • [Post rebuttal] Please justify your decision

    I thank the authors for the rebuttal which addressed a part of the comments. I would like to keep the rating. The main reasons are:

    1. The applicability of the method is limited (segmentation of stoke in CT) as it only compared methods on one specific application. Although in the rebuttal the authors claimed that the new method can be used for many applications, I just could not simply believe that it would work well for other applications, such as stoke segmentation in MRI.
    2. Some comments are not addressed. For example, a) The training of transformation network T, if it is unsupervised, how can it predict the parameters? b) why X+Q in second row is transformed while the others are not?



Review #3

  • Please describe the contribution of the paper

    Developed a novel network to disentangle asymmetries from non-contrast CT images for AIS segmentaion, trained with a tissue-awareness loss function to make the model more interpretable.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    Overall a nice paper to read some aspects that were nice to see:

    1. Tissue type aware regularisation term
    2. Different asymmetry maps to try and seperate/disentangle pathologies - like this concept, so not focusing on one aspect but potentially have the ability to highlight a few pathologies as one tries to make the model more explainable
    3. Nice detail in their methodology and explanation of the parts of the network
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    The idea I think is novel for the application, I think a weakness is potentially the lack of significance testing between the methods - which I think is nice to utilise in the evaluation, especially as you are comparing your method to current state of art methods.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
    1. Dataset available, parameters chosen given, liekly it can be impelemented whilst following the methodology.
  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
    1. I was interested to find out how the parameters were set - I would like to see some strategy performed to find lambda - unless this was done, if so how?
    2. If the work aims to improve AIS assesment - if it can’t detect the right bleeding spots, do you think that the outcomes are overconfident?
    3. I would like to see it perform on a larger dataset, is that int eh scope for future work?
  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    5

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Overall a nice paper, the idea to utilise dientanglement to separate symmetries is interesting, I think its a paper that looks to exploit the available data and at the same time improve current models so I did enjoy reading it.

  • Number of papers in your stack

    5

  • What is the ranking of this paper in your review stack?

    3

  • Reviewer confidence

    Somewhat Confident

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Not Answered

  • [Post rebuttal] Please justify your decision

    Not Answered




Primary Meta-Review

  • Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The reviewers all insisted on the relevance of the idea and the elegance of the method. A few points require clarification in the rebuttal regarding possible generalisability as well as detailed description of the separation between contributions and application of existing solutions

  • What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    6




Author Feedback

We thank reviewers and AC for their time and valuable comments. We are grateful that all reviewers and AC found our paper to present relevant and elegant ideas. We will release codes upon acceptance.

AC (and R1, R2): Generalisability and description of contributions A1: We propose new and effective ways of leveraging asymmetry for pathology segmentation. We work on NCCT because it is noisier and more challenging when dealing with soft tissues. Besides CT, our ADN can be extended to other tasks/modalities that leverage bilateral asymmetries to identify abnormalities. Such tasks include but are not limited to stroke/MS in brain MRI [4; R1; R2], fractures in pelvic X-Rays [6], and infiltration in chest X-Rays [Kim et al.].

We propose novel ideas for incorporating clinical observations into the design of ADN for 3D asymmetry disentanglement and achieve high interpretability by directly producing pathological and anatomical asymmetry maps. All asymmetry detection and regularization terms are our original contributions. We will revise the description of Eq. 1 as mentioned by R1 to better differentiate ours from [18], which only performed 2D asymmetry detection.

Ref: Kim et al., Learning visual context by comparison. ECCV, 2020.

R1

  1. Assumptions A1: 1) Our ADN can be easily adapted to handle bright asymmetry changes; see R3 A2 about bleeding spot detection. 2) We assume most morphological changes due to pathology would also lead to asymmetry. Because we separate pathological (P) and anatomical (Q) asymmetries, morphological changes should be captured by P and/or Q. If there happen to be lesions in symmetric locations, intensity changes would still be preserved in the network input, X_comp, and be recognized by ADN.

  2. Evaluation and hyperparameters A2: We follow the train/test split of AISD dataset. We did not use the test set when setting hyperparameters; we only train models using the training set until convergence. To show improvements are consistent and significant, we ran additional statistical significance tests comparing Dice scores of ADN and SOTAs; compared to the second-best model Clerigues [9], the p-value is 0.015. For hyperparameters, we followed settings in closely related works [7,17,19] for ADN and strictly followed settings in the published works/code for all baselines.

  3. Contributions A3: All regularization terms are originally proposed. Please refer to AC A1.

  4. Ablation on regularizations A4: During rebuttal, we removed the L_tissue and retrained ADN. This resulted in lower Dice 0.5090 and HD 39.66 compared to 0.5245 and 39.18 in original ADN. We expect ablation for other regularizations to yield similar results. We will conduct more experiments in future work.

R2

  1. Issue regarding modified inputs A1: We would like to clarify that the modified input is generated to suppress anatomical asymmetry (Q) and enhance pathological asymmetry (P) so that segmentation can be easier and more accurate (as demonstrated). X, Q, P, and A intensities are all normalized within the range [0, 1], and Q and P are learned to be meaningful via implicit supervision from segmentations so they should not affect pathology information presented in X^ or lead to false positives.

  2. Clarifications A2: 1) Transformation network T predicts rotation and translation parameters, so “object-centered” is not needed. Fig. 4, showing promising asymmetry detection results, proves that T was effective; 2) No weighting was used for losses; 3) Please refer to R1 A4 for info on additional ablation. 4) The mid-sagittal plane was used as shown in Fig. 1(a).

R3

  1. Comparison, hyperparameter, datasets A1: Please refer to AC A1; R1 A2 and A4 for details.

  2. Bleeding spots detection A2: Currently, we are not considering bleeding spots. With available annotations, bleeding spots would be detected by a network with inverse intensity change using the same architecture. We will mention how to handle bright asymmetry changes in the revised version.




Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    The authors adequately address the concerns relative to generalisability and contribution. With such clarifications included in the revised versions, the paper would be a relevant addition to the MICCAI programme. I therefore recommend acceptance

  • After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

    Accept

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    5



Meta-review #2

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    The rebuttal partially addressed generalisability and contributions and other concerns. Overall the merits slightly weigh over weakness.

  • After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

    Accept

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    6



Meta-review #3

  • Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

    Both meta reviewer and reviewers agreed on “the relevance of the idea and the elegance of the method”. In the rebuttal the authors also addressed the remaining concerns (generalisability and contribution in relation to existing work) to an extent. I’d like accept this paper for MICCAI 2022.

  • After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

    Accept

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    6



back to top