Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews Back to top
List of Papers By topics Author List
Paper Info Reviews Meta-review Author Feedback Post-Rebuttal Meta-reviews

Authors

Zixu Zhuang, Sheng Wang, Liping Si, Kai Xuan, Zhong Xue, Dinggang Shen, Lichi Zhang, Weiwu Yao, Qian Wang

Abstract

Magnetic resonance imaging (MRI) has become necessary in clinical diagnosis for knee osteoarthritis (OA), while deep neural networks can contribute to the computer-assisted diagnosis. Recent works prove that instead of only using a single-view MR image (e.g., sagittal), integrating multi-view MR images can boost the performance of the deep network. However, existing multi-view networks typically encode each MRI view to a feature vector, fuse the feature vectors of all views, and then derive the final output using a set of shallow computations. Such a global fusion scheme happens at a coarse granularity, which may not effectively localize the often tiny abnormality related to the onset of OA. Therefore, this paper proposes a Local Graph Fusion Network (LGF-Net), which implements graph-based representation of knee MR images and multi-view fusion for OA diagnosis. We first model the multi-view MR images to a unified knee graph. Then, the patches of the same location yet from different views are encoded to one-dimensional features and are exchanged mutually during fusing. The local fusion of the features further propagates following edges by Graph Transformer Network in the LGF-Net, which finally yields the grade of OA. The experimental results show that the proposed framework outperforms state-of-the-art methods, demonstrating the effectiveness of local graph fusion in OA diagnosis.

Link to paper

DOI: https://link.springer.com/chapter/10.1007/978-3-031-16437-8_53

SharedIt: https://rdcu.be/cVRuE

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

  • Please describe the contribution of the paper

    This paper proposed a local graph fusion network (LGF-Net) to extract features from multi-view MR images for knee osteoarthritis (OA) diagnosis. Specifically, a knee graph is first constructed based on the segmentation of sagittal-view MR images and the intersection of the multi-view MR images. After that, a local graph fusion (LGF) module is devised for the fusion of multi-view patches. Moreover, a graph transformer network (GTN) is developed to aggregate the features among different patches and to predict the grading of OA. Experimental results demonstrate the effectiveness of the proposed LGF-Net.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper is well organized and clearly clarified. It proposed a novel LGF-Net to gradually fuse the multi-view MR images for knee OA grading. Experimental results show the best performance of the proposed LGF-Net and the effectiveness of GTN. This work can benefit the clinic diagnosis of OA based multi-view MR images.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    Several sections are not well illustrated. Please see as below,

    1. the selection of slices from the three views for intersecting is not well illustrated. It is the basis for the construction of graph.
    2. the number of vertices N is not displayed.
    3. What is the meaning of the sentence “ Ni =9 can include the i-th vertex itself” ? Doesn’t the i-th vertex is the nearest vertex of vi?
    4. It would be better if the meanings of S, C, and A in Table 1 can be displayed. Additionally, there is another S at the last sentence of the second paragraph in Page 4.
    5. Section 3.3 is kinda of redundant with Section 3.2. I suggest to merge the two sections. Moreover, the specific pre-training strategy is not well illustrated. Do you pre-train the LGF module using an existing dataset or just load weights from an existing model?
  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    I am not sure whether the propsed method can be re-implemented because some points are not well illustrated.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

    Some details are required to display. Please see the the main weakness.

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    6

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper is well organized and propose a novel framework for multi-view MR image fusion to diagnose knee OA. The experimental results are the best.

  • Number of papers in your stack

    4

  • What is the ranking of this paper in your review stack?

    1

  • Reviewer confidence

    Very confident

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #2

  • Please describe the contribution of the paper

    In this work, the authors propose a graph fusion network to fuse local patch information from multi-view MR images for knee OA classification. The proposed framework includes a knee graph construction step using knee segmentation labels and interpolation points of multi-view slices, and a local fusion network (LFN) to encode each vertex’s local patch information, and then a graph transformer network (GTN) to aggregate multi-view patch features along the edges of generated graph in the first module. The proposed framework has obtained higher performance by comparing with some related papers.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1. I agree that fusing local patch from multi-view images could benefit the knee OA diagnosis problem, and I think the local information can better reflect the degree of OA than the global information.

    2. Combining the local patch information from multi-view slices and the GTN obtains good performance increase.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    1. The whole proposed framework and its application of this article is highly similar to this work: Knee Cartilage Defect Assessment by Graph Representation and Surface Convolution. The authors should give the discussion and comparison with the above work.

    2. I also find a previous paper called: Graph Transformer Networks, it also proposed a “Graph Transformer layer”. But the authors did not cite this work, and there has no discussion or comparison with this work.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    I think the reproducibility of this work is ok.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

    1. The authors should do a more comprehensive related paper review. If possible, I hope I could get a very detailed explanation in the rebuttal.

    2. When the “PD” abbreviation first appears, you should give it the full name (proton-density-weighted sequence). And, in table 1, in this sentence “LGF-Net (No PT)”, does the “PT” mean “PD”?

    3. Some specific implementation details of the “Projection” step in the “Knee Graph Construction” could be introduced. Dose the projection or “multi-view slices alignment” utilize slice registration or an external positioning hardware during data sampling?

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    4

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Effectiveness and some possible innovations of method for the knee OA problem.

    But the article lacks discussion and comparison with high similarity articles, which is a big flaw.

  • Number of papers in your stack

    4

  • What is the ranking of this paper in your review stack?

    2

  • Reviewer confidence

    Confident but not absolutely certain

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #3

  • Please describe the contribution of the paper

    The authors used multi-contrast MR images of the knee to identify WORMS of the knee for osteoarthritis. Unlike previous approaches of fusing the information from multi-contrast/multiview images at later stages of a network, the authors proposed to use a local fusion using a graph transformer network. The paper demonstrated that the local fusion with registered MR images using bone segmentation and a graph construction.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The authors proposed a novel method to fuse images from multi-view knee images using graph transformer network and local information. Previous approaches of fusing multi-contrast MR images relied only on late fusion strategies. The authors evaluate their approach in a comparative study with the current available approaches and presented that the proposed approach provided superior performance. The proposed approach can be extended to other MRI relevant tasks where multi-contrast images are used.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    None

  • Please rate the clarity and organization of this paper

    Excellent

  • Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

    It will be great to have the code availability included within the paper for reproducibility.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html

    The proposed approach clearly outperforms the current available approaches for grading OA using multi-contrast knee MR images.

    Please clarify why did the authors defined crop size of the knee region by given fixed mms? is it related to the details of WORMS?

    For the future work, it would be interesting to see the effect of individual pipelines within the knee graph construction. for example, how crucial is to have an accurate segmentation or how bad bone segmentations could be and the approach will be still performing well.

  • Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making

    7

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The presented approach is novel and it is of community interest as multi-contrast MR images are used for diagnosis. Comparative and ablation studies are performed and explained in great detail.

  • Number of papers in your stack

    4

  • What is the ranking of this paper in your review stack?

    1

  • Reviewer confidence

    Confident but not absolutely certain

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Primary Meta-Review

  • Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The paper presents a local graph fusion network to extract features from multi-view MRI for knee osteoarthritis diagnosis. The method is interesting, and experiments and results are convincing.

  • What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    NR



Author Feedback

We would like to thank the editors and reviewers for providing professional and helpful comments on Submission #578. We respond to the comments and queries raised by the reviewers here.

R1

  1. The selection of slices from the three views for intersecting is not well illustrated. It is the basis for the construction of graph. Given three sagittal, coronal, and axial slices, their intersection yields a point in the coordinate system of the MRI volume. In this way, we can get a set of intersection points P from many slices of different views, which is shown in the gray part of Fig. 2.

  2. The number of vertices N is not displayed. The number of vertices N is automatically determined across subjects, depending on the inter-slice spacing and the number of slices in different views. In our experiment N is typically 300~500. We note that the floating N is an advantage of our method, as it can adaptively handle heterogeneous knee sizes.

  3. What is the meaning of the sentence “|Ni|=9 can include the i-th vertex itself”? Doesn’t the i-th vertex is the nearest vertex of vi? Here we aim to state that each vertex is self-connected and counted as a neighbor of itself. Thus, the size of the neighborhood is 9 (including the center vertex of the neighborhood). We will revise the relative sentence to make it clear in the final paper.

  4. It would be better if the meanings of S, C, and A in Table 1 can be displayed. Additionally, there is another S at the last sentence of the second paragraph in Page 4. We will change them to “Sagittal”, “Coronal”, and “Axial” in the final paper.

  5. Section 3.3 is kind a of redundant with Section 3.2. I suggest to merge the two sections. Moreover, the specific pre-training strategy is not well illustrated. Do you pre-train the LGF module using an existing dataset or just load weights from an existing model? We will follow the suggestion to merge the two sections in the final. The LFN module in LGF is pre-trained, using labeled lesions on our training data. In our current model, the pre-training is necessary to attain high performance. However, we are working on to lower the data requirement on the pre-training, i.e., to attain the same classification performance without using lesion labeling that is tedious to attain.

R2

  1. The whole proposed framework and its application of this article is highly similar to this work: Knee Cartilage Defect Assessment by Graph Representation and Surface Convolution. The authors should give the discussion and comparison with the above work. The Paper Submission Guideline points out that “arXiv papers are not considered prior work since they have not been peer-reviewed.” Thus, we choose to skip this discussion or comparison.

  2. I also find a previous paper called: Graph Transformer Networks, it also proposed a “Graph Transformer layer”. But the authors did not cite this work, and there has no discussion or comparison with this work. The work proposed by Yun et al. above has very different goals and network structures from our work. They aim to address the limitation of learning representations on the misspecified or heterogeneous graph, which is not related to our work.

  3. Some specific implementation details of the “Projection” step in the “Knee Graph Construction” could be introduced. Dose the projection or “multi-view slices alignment” utilize slice registration or an external positioning hardware during data sampling? Please refer to R1.1. We do not use slice registration or external positioning hardware for vertex extraction. Instead, given three sagittal, coronal, and axial slices, their intersection yields a point in the coordinate system of the MRI volume. We can easily transform such points to the world coordinate system with header information in DICOM files.



back to top