List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Siyuan Dong, Gilbert Hangel, Wolfgang Bogner, Georg Widhalm, Karl Rössler, Siegfried Trattnig, Chenyu You, Robin de Graaf, John A. Onofrey, James S. Duncan
Abstract
Magnetic Resonance Spectroscopic Imaging (MRSI) is a valuable tool for studying metabolic activities in the human body, but the current applications are limited to low spatial resolutions. The existing deep learning-based MRSI super-resolution methods require training a separate network for each upscaling factor, which is time-consuming and memory inefficient. We tackle this multi-scale super-resolution problem using a Filter Scaling strategy that modulates the convolution filters based on the upscaling factor, such that a single network can be used for various upscaling factors. Observing that each metabolite has distinct spatial characteristics, we also modulate the network based on the specific metabolite. Furthermore, our network is conditioned on the weight of adversarial loss so that the perceptual sharpness of the super-resolved metabolic maps can be adjusted within a single network. We incorporate these network conditionings using a novel Multi-Conditional Module. The experiments were carried out on a 1H-MRSI dataset from 15 high-grade glioma patients. Results indicate that the proposed network achieves the best performance among several multi-scale super-resolution methods and can provide super-resolved metabolic maps with adjustable sharpness.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16446-0_39
SharedIt: https://rdcu.be/cVRTy
Link to the code repository
https://github.com/dsy199610/Multiscale-SR-MRSI-adjustable-sharpness
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
To achieve multi-scale super-resolution, the author adopted a Filter Scaling strategy to obtain resolved results with different upscaling factors. The proposed network was conditioned on the weight of GAN loss and instance normalization to adjust the sharpness of the image. The experiments were evaluated on various metabolite types of 3D brain images from 15 high-grade glioma patients. And, given methods could achieve the best performance and have flexible operations in sharpness of the image.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The organization of this paper is good, and the idea of this work is novel. Proposed model could dynamically update the network with the different metabolites and be conditioned on the weight of adversarial loss to adjust the sharpness of the resolved image. Extensive experiments demonstrate that the proposed network could realize efficiency in training time and the number of parameters, and verify the effectiveness of adjustable sharpness. Besides, it achieves competitive performance compared with Hypernetworks and AMLayer.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Fig. 1 (b) is not clear for me. The function c(n) that outputs scaling factors(C_in x C_out) from n (input resolution) and E(m) should be deeply discussed. For instance, what is the dim of inputs of fc? Why outputs of fc could be acted as the upscale factors for filters (seems there missed a element-wise product.). Then, MCM conducted the next fc to represent \lambda into 2 x C_out in CIN. However, the \lambda is numerical scalar. The part of MCM is hard to follow and are not sufficiently motivation.These should be included in the corresponding ablation. What dose upscale factors for filters do? I don’t find upsampling or deconvolution operations. What does “as ground truth images contain missing pixels due to quality-filtering, we remove those pixels in network outputs …” mean? What the differences between quality-filtering ahead of network inputs and quality-filtering after outputs?
- It is strange for embedding 7 metabolites (text data) as the auxiliary variables in MCM. In Sec 2.2, E(m) is a trainable embedding layer that converts words to numerical vectors. The analysis on E(m) is excessively brief, not providing the complete description on specific operations, effects, sufficient discussions. This affects the quality of writing and the clarity of the presentation, also, scores. 3.“The adversarial loss uses a discriminator (4-layer CNN)” , Whether Wasserstein GAN will be trained alternatively together with proposed networks. If not trained, there should lead to a mistake. In this time, how Wasserstein GAN train a 4-layer CNN without a fc layer.
- “statistically insignificant p-value “ don’t have illustration and reflect on the specific results. I can’t follow its meanings.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Experimental data process depends on the professional tools (LCModel v6.3-1 and FSL v5.0) so that maybe missing the required details for reproducibility of the paper.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
Please see Sect. 5. Besides, there are a few typos and errors:
1) Introduction states “Furthermore, the adversairal loss was incorporated to ….”, was -> is. The tense of the related work.should be consistent. 2) “such that S best approximates the high resolution ground truth … ”, the text below the fig. 1 could’t be followed. 3) In Sec. 2.2, “where m is the type of metabolite” should be “… types …”. 4) In Sec. 3.3, “To ensure fairness, we set the same layer number and latent size” should be “… numbers …”, and “… sizes …”. … Please check the overall paper carefully.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Based on the novelty and good organization, also, the insufficient comparisons and lacking of some details.
- Number of papers in your stack
3
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
6
- [Post rebuttal] Please justify your decision
Novelty, good presentation and clarification to some technical points.
Review #2
- Please describe the contribution of the paper
This work develop a multi-conditional module to incorporate multiple conditions into a MRSI super-resolution network that can avoid training a separate network for each combination of hyperparameters.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The proposed framework can achieve comparable performance as the networks are trained under a single-scale setting.
- The developed model can provide multiple levels of sharpness for each super-resolved metabolic map learn the super-resolution process based on the specific metabolite.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1: The idea of the paper that super-resolve the image in multi-scale manner is not sufficiently new. There even have been some works arbitrary scale super-resolution, such as “Learning A Single Network for Scale-Arbitrary Super-Resolution”, “Arbitrary Scale Super-Resolution for Brain MRI Images”.
2: The experiments are performed only on H-MRSI dataset, but the number of images are too small to obtain meaningful training results. There are only 15 3D samples. The authors are encourage to estimate the proposed model on another dataset.
3: From Table 1, the proposed method doesn’t improve significantly than the unconditioned method but consumes more time and takes more memory for the parameters.
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
I am not so sure for the re-producibility of the paper. The code and the data are not available to aid reproducibility. However, the convergence of the proposed model is not discussed.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
1: The writing of the paper can be improved. The description of how the proposed method can deal with the multi-scale input is unclear.
- : The utility of the T1 and FLAIR images is confusing. The author doesn’t demonstrate the effect of the T1 and FLAIR images. Have the authors studied on using only LR Met images?
- The ablation studies are insufficient. For example, the ablation study of the pixel loss, structural loss and adversarial loss to explore the effect of different loss function.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
3
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This work is similar with published work.
- Number of papers in your stack
3
- What is the ranking of this paper in your review stack?
4
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
This work is similar with published work. Learning A Single Network for Scale-Arbitrary Super-Resolution”, “Arbitrary Scale Super-Resolution for Brain MRI Images”.
Review #5
- Please describe the contribution of the paper
This paper proposed a blind super-resolution method for Magnetic Resonance Spectroscopic Imaging. It applied the filter scaling strategy based on the upscaling factor to realize the multi-scale super-resolution within a single network. Extensive experimental results demonstrated that the performance of the proposed model was comparable to the single SR model.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The idea of applying the filter scaling strategy for adaptive super-resolution was well-motivated.
- The proposed method considered the distinct spatial characteristics of different metabolites and proved to improve the performance of the model.
- The experiments were extensive and proved the effectiveness of each modulating module of the proposed model.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The numerical results are not so significant compared to the existing multi-scale methods.
- The visual comparison to existing blind SR methods should be also included.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The paper gives a clear and detailed description to the experimental details.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
- As the proposed method was a multi-modal based super-resolution model, the author could discuss the influence brought by the different images in the proposed method.
- While Fig. 3 and 4 showed the relation of the sharpness to the parameter lambda, the effect of Equation (5) was still unknown.
- The visual comparison to the current blind SR method could be provided.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
6
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I suggest acceptance to this paper. The idea of using the scaling filter for blind SR is well motivated. The overall approach is clear and easy to understand. The experiment is aslo extensive to prove the effectiveness of the proposed method.
- Number of papers in your stack
1
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
A blind super-resolution approach for Magnetic Resonance Spectroscopic Imaging was proposed in this study. To achieve multi-scale super-resolution inside a single network, it used a filter scaling technique based on the upscaling factor. Extensive experimental results demonstrated that the proposed model’s performance was comparable to that of a single SR model. The author used a Filter Scaling approach to generate resolved findings with multiple upscaling factors to achieve multi-scale super-resolution. To change the sharpness of the picture, the suggested network was conditioned on the weight of GAN loss and instance normalisation. The investigations were performed on several metabolite kinds of 3D brain pictures from 15 individuals with high-grade glioma. Furthermore, the suggested approaches could reach the optimum performance and had flexible operations in image sharpness.
First, the work is indeed an interesting topic. Experimental results could provide some insights for this field. However, three reviewers have diverse review comments: Proposed model could dynamically update the network with the different metabolites and be conditioned on the weight of adversarial loss to adjust the sharpness of the resolved image (R1). The developed model can provide multiple levels of sharpness for each super-resolved metabolic map learn the super-resolution process based on the specific metabolite (R2). The proposed method considered the distinct spatial characteristics of different metabolites and proved to improve the performance of the model and the experiments were extensive and proved the effectiveness of each modulating module of the proposed model (R3). However, R2 mentioned the idea of the paper that super-resolve the image in multi-scale manner is not sufficiently new. The experiments are performed only on H-MRSI dataset, but the number of images are too small to obtain meaningful training results. There are only 15 3D samples.
Some of my review comments:
- I admire the authors efforts working on MRSI, which is a very tricky sequence.
- I understand the author have shown improved quantitative measurements, e.g., PSNR, SSIM and LPIPS. However, I am not quite convinced if the super-resolved data shown benefits for clinical studies. Compared to zero-filled “baseline”, which might be too basic, there are clear improvement. However, compared to the “ground truth”, I can’t really see if the super-resolved data is qualified. Maybe it is because the “ground truth” itself is hard to justify. Are there any way to indirectly quantify the results with for example tumor progression, diagnosis/prognosis or death rate?
- I must admit I have limited knowledge in MRS(I).
Reviewers are somewhat confident about their concerns. In the rebuttal, the authors may need to highlight:
- The limited novelty of the current study.
- The limited data problem and reproducibility of the current work.
- Address other negative comments from all the reviewers.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
8
Author Feedback
We thank the constructive feedback from all reviewers. We appreciate the positive feedback on the novelty of our method (R1) and the comprehensiveness of our experiments (R1, R3). Here we would like to address the raised issues.
Novelty (R2, Meta-R) We agree that multi-scale super-resolution (SR) is an existing topic, but our Filter Scaling strategy that scales the convolution filters based on the upscaling factor is entirely new. The Filter Scaling can achieve performance comparable to the single-scale SR without significantly increasing the network size. Our novelty also includes metabolite-specific modulation and sharpness adjustability, neither of which has been explored in previous works. Our MCM efficiently integrates multiple functionalities mentioned above into a single network. Furthermore, as noted by Meta-R, our work tackles a very tricky sequence MRSI, and our 7T high-resolution (HR) MRSI dataset from a large patient cohort is unique.
Limited data (R2, Meta-R) Due to the low concentration of metabolites, acquiring HR MRSI with acceptable SNR is always a challenge. We used a 7T scanner to obtain HR (64x64x39) and high SNR ground truth, and there is no other dataset like this at all. From those 15 3D samples, we obtained 1484 2D metabolic maps, which were substantially augmented via random transformations (section 3.2). We used the validation set to ensure that trainings converge without overfitting. All experiments were conducted with 5-fold cross-validation, so the evaluations were performed on all 15 patients, which we believe makes our results reliable.
Reproducibility (R1, R2, Meta-R) We will release the code after the review process. We also provided implementation details in section 3.2. The data-preprocessing tools, LCModel and FSL, are both publicly available [22][25].
Improvement given by our method in Table 1 (R2 and R3) To show that our improvement is statistically significant, we performed t-test on PSNR and SSIM for each pair of methods shown in Table 1. “Filter Scaling with Met (ours)” outperforms all other methods with p-value < 0.05 (described on page 6). We will include the corresponding visual comparisons in the final version (R3).
Benefits for clinical studies (Meta-R) Our method resolves some high-frequency details not available in low-resolution images. For example, in the last row of Fig. 3, we can observe a tumor in the FLAIR image, which is shown as a hotspot in the GT at the corresponding location. However, this hotspot is blurred in the zero-filled image, which may lead to misdiagnosis. Our method successfully reconstructs this hotspot. In future works, we can compare our results with tumor segmentation maps or histopathology to see if our method can help to better identify molecular markers for tumors.
Effects of multi-modal MRI and each component of the loss function (R2, R3) These ablation studies were already documented in previous literature [8], which are beyond the scope of our paper.
More details on MCM in Fig. 1(b) (R1, R2) The embedding layer E(m) learns a transformation matrix that maps each metabolite name to a vector of length 3. This vector is concatenated with the scalar n (input resolution) and fed into the FC layers c(n, E(m)), which has an input dimension of 4. The output of c(n, E(m)) has a dimension of C_in x C_out, of which each element is multiplied with a k x k conv filter (there are C_in x C_out of such filters in each conv layer), so the filters are “scaled” based on n and E(m). The next FC layers (described as s(λ) in Equation 5) take the scalar λ as the input and output both modulated mean and SD for each feature map, so the output dimension of s(λ) is 2C_out.
Discriminator (R1) The discriminator was trained alternatively with the proposed network. We forgot to mention that the discriminator does have FC layers after 4 conv layers.
In the final version, we will describe our method in more detail and correct the grammatical errors.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
After rebuttal, one of the reviewers raised his/her score and brought up the scores to 6/3/6. Although there is a reviewer who didn’t change his/her score as it remains low, I didn’t receive further feedback from this reviewer. Regarding the work, I can see there are some limitations of the technology, but the proposed method may propel the development in this field. I am therefore inclined to give acceptance to this work. As the overall scores are not high-ranked in my stack, I couldn’t recommend oral presentation for this work.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
5
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
A multi-scale super-resolution method for Magnetic Resonance Spectroscopic Imaging (MRSI) was proposed in this paper. The topic is interesting and of high clinical relevance, and the method demonstrates certain degree of innovation including Filter Scaling strategy, metabolite-specific modulation, and sharpness adjustability. There are divided reviews comments, especially from R2, on similar work published “Learning A Single Network for Scale-Arbitrary Super-Resolution”, “Arbitrary Scale Super-Resolution for Brain MRI Images”. Inspite of this, the overall reviews are positive, and I would recommend the acceptance of this but suggest authors include a discussion or comparison with the above two paper.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
6
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper proposed a model to handle multi-scale super-resolution of MRSI, conditioned on upscaling factors, which also includes metabolite-specific modulation and sharpness adjustability to improve the performance. Compared with other multi-scale super-resolution networks, the proposed model shows to be a lightweight model with good performance. However, the performance improvement in Table 1 may not be significant given the large variation, especially when comparing the proposed model with AMLayer[14], a model with less parameters and very close performance. In this case, statistic test is important. However, no exact p-values were given and it is unclear what type of student t-test was conduct, making the claim of significance less convincing. Meanwhile, the authors did not explicitly respond to the difference of their work from the two reference papers given by Reviewer#2, and did not remove the major concerns about novelty from this reviewer.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
9