List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Bilha Githinji, Lei Shao, Lin An, Hao Zhang, Fang Li, Li Dong, Lan Ma, Yuhan Dong, Yongbing Zhang, Wen B. Wei, Peiwu Qin
Abstract
Vision-threatening pathological myopia presents several lesions affecting various retinal anatomical structures. Detection approaches, however, either focus on one anatomical feature or are not intentional. This study uses hypergraph learning to modulate delineated retinal anatomical features from fundus images and capitalize on hidden associations between these features. Experiments are conducted to assess prediction performance when targeting a particular anatomical trait versus using a mixture of select anatomical features, and in comparison to a ResNet34-based convolutional neural network classifier. Results indicate better prediction with hypergraph learning on a mix of the delineated features (F1 score 89.75%, AUC score 95.39%). A choroid tessellation segmentation method is also included.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16434-7_53
SharedIt: https://rdcu.be/cVRsl
Link to the code repository
For choroid tessellation code: https://github.com/qin-lab-tbsi/fundus_choroid_tessellation/
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
In this work, the authors developed a multimodal hypergraph learning approach for identifying pathological myopia using features from different retinal structures of fundus images and utilizing the associations between the hidden features. The authors have demonstrated that the combination of these features provide better prediction performance than other approaches, which focus, at most, on only one target structure.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors propose a hypergraph learning to detect early or mild pathological myopia. • This approach extracts features from several prominent retinal structures, instead of the conventional methods, which focus on only one structure. • The results from hypergraph learning on a mixture of retinal structure provides highest performance, which is statistically significant than conventional CNN based method.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
• This paper has limited technical novelty, because the only contribution is using the hypergraph learning on the features extracted from other machine learning and deep learning techniques. • The authors use ResNet34 encoder to compare with the performance of hypergraph learning. What would be the performance using other state-of-the-art CNN, for example ResNeXt-50? ResNeXt-50 provides better performance in comparison with ResNet
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
• The hyper-parameters for the proposed approach has been described, and the details of the training/testing set has been provided.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
This is an interesting work to provide better prediction performance of the degree of pathological myopia, using multimodal hypergraph learning model. It would be interesting to see the performance of the approach in comparison with other CNN based methods on RGB images.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This work has applications for predicting the degree of PM by utilizing the relationship between different retinal anatomical features using graph learning. However, most of the extracted features used are obtained from machine learning and deep learning methods.
- Number of papers in your stack
4
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The authors introduced a novel multimodal hypergraph learning technique to learn higher-order associations and modulate delineated retinal features. Their experimental results demonstrate the potential of the model to improve prediction performance, in addition, an intensity thresholding approach was proposed to extract choroid tubular patterns.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The idea of modulating delineated retinal anatomical features from fundus images using a multimodal hypergraph learning technique is an important task. In this sense, the paper is attempting an important problem in a timely manner.
- The authors used a hypergraph to learn higher-order associations, which is novel and smart way to solve the overall problem.
- The proposed method is validated on a real dataset (I the paper and the supplementary material) and show promising results.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The itroduction section needs to be more elaborated by discussing better the limitation of different related works. I see that it does not cover well papers related to the problem to solve.
- The authors mentioned that the use of CNN embeddings to generate statistical features is limited to the loss of interpretability and they did not explain that. I expected more discussion as well as mentioning related works about this point.
- The choice of the loss term R_emp(M) is not explained. What is the reason of suming Frobenius norm and L1 norm, why not one simply using one of them?
- The edges weights were defined as the similarity between retinal characteristiques. How such similarity was computed? The definition of the hypergraph should be better explained in the method section. If an euclidean distance was used, what is the dimension of its input variables? How do the authors define mathematically the retinal carachteristiques?
- To evaluate their method, the authors used a 80/20 split strategy for training and testing on the dataset. I expected a cross-validation evaluation strategy. How are they sure their model is not overfitting?
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The checked points in the reproducibilty checklist match perfectly the information provided in the paper.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
The paper is well structured and well written. I suggest that the authors consider answering my questions in the weaknesses part mentioned above, in order to improve this and future submission.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I see that this work tackled an interesting problem in a novel way. However, with the limitations I mentioned above I rate this paper with “Weak accept”.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #4
- Please describe the contribution of the paper
The authors develop an algorithm to classify pathological myopia (PM) from retinal fundus images, based on hypergraph learning. Specifically, the authors extract multiple anatomical features from retinal fundus images (including choroid tessellation, for which they present a sub-algorithm in the manuscript), and build a hypergraph where each node is an image, and images with similar retinal features are connected by an edge (the edge weight is the similarity score). They argue that this approach can learn to associate multiple interpretable features, and they compare the predictive performance of this model against a hypergraph trained on a reduced set of anatomical features, and a traditional CNN-based approach. Overall, they demonstrate that the hypergraph trained on multiple anatomical features slightly outperforms the other two approaches, on a range of different inputs.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Main strengths of the paper:
-
The authors present a timely and interesting application of hypergraphs to the problem of classifying pathological myopia
-
They also present sub-algorithms for segmenting choroid tubular patterns, and make the code publicly available on github
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Main weaknesses of the paper:
-
Overall, the performance improvement of the proposed approach is minor. One potential advantage of the proposed algorithm is improved interpretability, which the authors discuss; however, they don’t investigate interpretability of the method on this dataset or compare interpretability against the standard CNN-based approach.
-
Figure 1 caption: there is no explanation of what the symbols (circle, x, square) correspond to in the data. From the text, I assume it is the level of pathologic myopia, but this should be clearly labeled in a legend.
-
p-values should be reported as, for example, p-value < 10^-4. They should not be reported as p-value = 0.000
Minor weaknesses:
-
Why is AUC score missing for RGB CNN in Table 1?
-
page 6 type: “modals” should be “models”
-
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The reproducibility checklist was accurately filled out. One thing that could improve reproducibility of the paper is to make the full code available after publication.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Given that the performance between the standard CNN-based approach and the proposed hypergraph learning approach is so similar, I think it would strengthen the paper to investigate the interpretability of the hypergraph, to show that it is indeed an advantage of this method.
Also, I found the reference to “multimodal” to be a bit confusing. Specifically, the proposed algorithm only processes retinal fundus images – it does not take into account different modalities during the prediction. I appreciate that the authors extract multiple anatomical features, and input this to their hypergraph model; however, I would not refer to this as multimodal. Perhaps “multi-dimensional” would be more accurate.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The authors evaluate a novel algorithmic approach on a novel dataset, and show competitive results. Supporting algorithms were developed and made available to the community. This is an interesting direction that I think would spur further research.
- Number of papers in your stack
6
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposes a novel multimodal hypergraph learning technique to learn higher-order associations and modulate delineated retinal features for detecting early or mild pathological myopia.
It is an interesting work, and the paper is well structured and written. I recommend accepting this submission. The authors should address the detailed comments from the reviewers in the camera-ready manuscript.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3
Author Feedback
We appreciate the feedback provided, thank you. We have grouped the feedback as recommended and below is further clarification
Graph estimation and hypergraph learning: For the graph construction task, we use Euclidean distance between two features to estimate edge weights. As for the loss term, Frobenius norm (F-norm) loss is sensitive to outliers and given our relatively small dataset we want to tamper that down and have more consistent results across different test samples. The general intuition for it is that since F-norm is a square, the percentage update in the combined loss is larger for small losses than for already large losses. We also empirically compare Frobenius norm loss and L1 loss separately during earlier hyperparameter exploration and find that their combination has better performance and with less variability across random samples (data are not shown here).
The constructive comments on exploring interpretability of the hypergraph resonate. We do see building in interpretability in an end-to-end deep learning fashion as a separate and larger problem that warrants its own study and hope to do more in future works. To the best of our knowledge, this study is the first trial of adopting hypergraph learning on a PM problem. In this work, we focus on the pipeline and the idea of having a different take on the features, and hope to encourage exploration using non-conventional approaches with regard to how we learn using the various anatomical information that retinal fundus images can present. Additionally, decomposition of complex fundus image into independent anatomical features seems to enhance information distillation with hypergraph learning, as the results suggest.
Regarding the choice of CNN model (e.g. use of ResNeXt-50), the setup is such that we have comparably capable pipelines for the baseline case, and so we use an encoder from the same CNN for offline encoding scheme in the hypergraph pipeline (changing to ResNeXt-50, for instance, would update the pipeline’s offline encoder as well). This comparable baseline condition is evident in the results, allowing us to focus on the presented idea.
Additional material for introduction section, formatting and presentation related comments: We acknowledge the suggestions on including descriptions on a couple of the figures, fixing a typo, and properly reporting on p-values. We’ll make updates on the camera copy accordingly.
Multimodal Vs multi-dimensional: We see how using ‘multimodal’ can be confusing and the suggestion to consider ‘multi-dimensional’ is much welcomed.