List of Papers By topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Xingran Xie, Yan Wang, Qingli Li
Abstract
Benefited from the rich and detailed spectral information in hyperspectral images (HSI), HSI offers great potential for a wide variety of medical applications such as computational pathology. But, the lack of adequate annotated data and the high spatiospectral dimensions of HSIs usually make classification networks prone to overfit. Thus, learning a general representation which can be transferred to the downstream tasks is imperative. To our knowledge, no appropriate self-supervised pre-training method has been designed for HSIs. In this paper, we introduce an efficient and effective Self-supervised Spectral Regression (S$^3$R) method, which exploits the low rank characteristic in the spectral domain of HSI. More concretely, we propose to learn a set of linear coefficients that can be used to represent one band by the remaining bands via masking out these bands. Then, the band is restored by using the learned coefficients to reweight the remaining bands. Two pre-text tasks are designed: (1) S$^3$R-CR, which regresses the linear coefficients, so that the pre-trained model understands the inherent structures of HSIs and the pathological characteristics of different morphologies; (2) S$^3$R-BR, which regresses the missing band, making the model to learn the holistic semantics of HSIs. Compared to prior arts, i.e., contrastive learning and masked image modeling methods, which focuses on natural images, S$^3$R converges at least 3 times faster, and achieves significant improvements up to 14\% in accuracy when transferring to HSI classification tasks.
Link to paper
DOI: https://link.springer.com/chapter/10.1007/978-3-031-16434-7_5
SharedIt: https://rdcu.be/cVRq2
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #2
- Please describe the contribution of the paper
This paper presents a self-supervised spectral regression (S3R) to address the problem of self-supervised pre-training for hyperspectral histopathology image classification. Specifically, S3R consists of two pretext tasks (BR and CR) to learn a general representation for down-stream tasks. Experimental results on PDAC and PLGC datasets evaluated the effectiveness of S3R.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
a. They designed a self-supervised method tailored for microscopy HSI classification, which had not been concerned before. It’s a novel idea to utilize the properties of hyperspectral images that one band can be represented as a linear combination of the remaining bands. b. The experiments results show up to 14% improvements.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The author’s introduction and visualization of the data structure of HSI images is not very clear. Compared with regular RGB pathological images, is it just a few more channels? What are the specific challenges for pathologists to analyze HSI images in human vision?
- For S3R-CR, the necessity of band dropping and is debatable, since the regression targets is the coefficients of the bands that were not dropped.
- Similarly, for S3R-CR, the necessity of spatial masking is debatable, since the regression target is the missing band.
- Overall, the masking operation is far fetched. The relationship between CR and BR is not direct.
- Moreover, the ablation study is insufficient. Image masking and band dropping are not the premise of CR. Image masking is not the premise of BR. So what would the performance of the model be without image masking?
- Missing standard deviation of ACC.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The obtained results can, in principle, be reproduced if given access to the missing resources (code, data).
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
BR and CR are two reasonable pretext tasks for HSI image. However, the relationship between image masking and these two tasks is not direct, so the introduction of MIM model is abrupt. More theoretical analysis and experiments on the necessity of MIM for BR and CR are welcome.
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The novelty and the completeness of experiments.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
3
- Reviewer confidence
Very confident
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #3
- Please describe the contribution of the paper
The authors have conceptualized an efficient and effective self-supervised spectral regression method which proposes to understand the inherent structures of hyperspectral images and pathological characteristics of different morphologies. The paper is well-motivated, has novelty, and is clinically relevant.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The idea of characterizing spatiospectral information from hyperspectral images using self-supervised spectral regression of histopathology images is a novel application.
- Hyperspectral images contain rich information and processing them for accurate diagnosis is challenging. The authors have done a good job in providing sufficient background information, related work, and motivated the study in the right direction.
- Further, the authors have compared their strategy to the emergent contrastive learning approaches with greater performance and faster convergence which is interesting.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The authors highlight that S3R method can help explore the morphological characteristics of tissue images. However, no illustration or intuitive explanation is provided to address this contribution.
- The results from all the experiments having missing error bars to understand the stability of proposed algorithm.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors have provided sufficient implementation details for reproducibility of the paper.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
-
Could the authors provide qualitative assessment of the proposed method? For example, how does S3R method help in exploiting the morphological characteristics in a sample tissue image. Highlighting the spectral bands and showing the result of this approach can help better understand the learning strategy.
-
Next, the authors claim that S3R forces the network to understand the inherent structures of HSIs. Could the authors expand on this idea?
-
Can the authors provide an insight into the failure modes and if it could be expanded to datasets from other modalities?
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
5
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper presents a novel approach that is shown to provide a superior classification performance and training efficiency compared to contrastive learning. However, the paper has some limitations detailed above which needs to be addressed.
- Number of papers in your stack
5
- What is the ranking of this paper in your review stack?
2
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Not Answered
- [Post rebuttal] Please justify your decision
Not Answered
Review #4
- Please describe the contribution of the paper
This paper added prior knowledge that each band of the hyperspectral images (HSI) could be represented by a linear mixture of the remaining bands in self-supervised learning to learn the features efficiently and effectively. The proposed Coefficients Regression (S3R-CR) encourages the deep learning models to regress the linear coefficients among multiple bands, while the Band Regression (S3R-BR) converges the pixel-wise differences of the selected band by re-weighting the remaining bands. Masked image modeling was used during self-supervise learning. By exploring the low rankness in the spectral domain of an HSI, The proposed methods achieved better downstream classification accuracy compared with other contrastive learning approaches.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
This paper used the intrinsic relationship among different bands of HSI in self-supervised pre-training to extract the low-ranking contexts in the spectral domain.
-
The design of loss is straightforward and concise to the characteristic of HSI.
3.The proposed methods provide an efficient and effective pre-training strategy for unlabeled HSIs with high spatial-spectral dimensions.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
This paper lacks the literature reviews on unsupervised / self-supervised methods for HSIs[1].
-
The baseline models are insufficient to demonstrate the improvement. This paper did not evaluate the performance from classical unsupervised-learning methods since the linear coefficients among different bands from HSI can be closed-form solutions, and the features could be mathematically formed. For example: [2]. Meanwhile, this paper did not compare the SOTA deep learning based models for spectral-spatial feature learning of hyperspectral images. For example: [3,4,5].
-
This paper did not provide the 3-band results of the proposed methods, which is not fair to the baseline contrastive learning models since the selection of bands might highly influence the performance.
[1] Ortega, Samuel, Martin Halicek, Himar Fabelo, Gustavo M. Callico, and Baowei Fei. “Hyperspectral and multispectral imaging in digital and computational pathology: a systematic review.” Biomedical Optics Express 11, no. 6 (2020): 3195-3233. [2] Li, Kun, Yao Qin, Qiang Ling, Yingqian Wang, Zaiping Lin, and Wei An. “Self-supervised deep subspace clustering for hyperspectral images with adaptive self-expressive coefficient matrix initialization.” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 14 (2021): 3215-3227. [3] Mou, Lichao, Pedram Ghamisi, and Xiao Xiang Zhu. “Unsupervised spectral–spatial feature learning via deep residual Conv–Deconv network for hyperspectral image classification.” IEEE Transactions on Geoscience and Remote Sensing 56, no. 1 (2017): 391-406. [4] Yue, Jun, Leyuan Fang, Hossein Rahmani, and Pedram Ghamisi. “Self-supervised learning with adaptive distillation for hyperspectral image classification.” IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-13. [5] Hong, Danfeng, Lianru Gao, Jing Yao, Naoto Yokoya, Jocelyn Chanussot, Uta Heiden, and Bing Zhang. “Endmember-guided unmixing network (EGU-Net): A general deep learning framework for self-supervised hyperspectral unmixing.” IEEE Transactions on Neural Networks and Learning Systems (2021).
-
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Regarding the reproducibility of this work, the description in the method section is sufficient to understand. However, some of the detailed implementations are missing for reproduction.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://conferences.miccai.org/2022/en/REVIEWER-GUIDELINES.html
-
In the implementation details, some parameters, such as batch sizes of all methods, are missing, and the details of the fine-tuning stage are missing (freezing parts of parameters, the structures of the classifier, etc.).
-
In Fig. 3, it would be nice to see the “ground-truth” coefficient Visualizations and then compare the similarities between the two proposed methods.
-
It would be interesting to see the quantitative results and illustration about “S3R converges at least 3 times faster” from the abstract.
-
- Rate the paper on a scale of 1-8, 8 being the strongest (8-5: accept; 4-1: reject). Spreading the score helps create a distribution for decision-making
4
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The proposed strategies from this paper seem interesting in Spectral–Spatial Feature Learning for HSIs, but more evaluation results need to be provided to demonstrate the improvement.
- Number of papers in your stack
1
- What is the ranking of this paper in your review stack?
1
- Reviewer confidence
Confident but not absolutely certain
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
5
- [Post rebuttal] Please justify your decision
I appreciate all the clarifications and additional information provided by the authors about my questions. I have changed my rating to reflect the effort carried out by the authors.
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper utilizes the internal correlation of hyperspectral histopathology image to guide self-supervised learning. all reviewers unanimously expressed their concerns on the rigor of the evaluation and confusion about how the results are presented, which need to be carefully addressed before the paper can be further considered. Please see the reviewer comments for further details. Here are important points to address in the rebuttal:
The concerns about the fair comparison with baseline methods
The reviewer’s concern about details in methodology
The concerns about the metrics and visualization
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
7
Author Feedback
We thank reviewers and the AC for constructive feedback. First, we claim that we conducted fair comparisons with SOTA SSL methods [9,4,20]. There is no SSL method specifically designed for hyperspectral histopathology images. HSIs in remote sensing and medical areas differ a lot. Besides, the only two papers which match our setting do not have codes (ref2 and ref3 provided by @R4). We then follow popular SSL methods to report metrics. Masking operation is necessary to prevent the model from obtaining trivial solutions. In the following, we will address the major concerns one by one. 1) Comparison with baseline Remote sensing HSIs have very low spatial resolution, whose morphology is usually paid less attention to, and the classification task in that area refers to per-pixel classification rather than per-image classification. In response to @R4’s request, among ref2-ref5, ref4 and ref5 released codes. But neither of them is suitable for our task, since ref4 is SSL trained with few labeled data, and ref5 treats SSL as an auxiliary task for spectral unmixing. We implement ref3 and ref4 but their fine-tuned classification accuracy is even 3% and 1% lower than 1N1K baseline in our paper. 2.1) Necessity of band dropping & masking @R2 questioned the necessity of band dropping in CR and spatial masking in BR. The regression target is the coefficients of the bands that are not dropped w.r.t. the dropped band. In HSIs, the appearances in adjacent bands are very similar, with slight intensity difference w.r.t. a pixel in the same spatial location. For CR, without masking, SSL model can easily learn a short-cut from intensity to coefficient beta, without considering pathological characteristics of different morphologies. For BR, masking aims to learn semantics as suggested in MIM methods [1,19,20]. Without masking, the SSL model can converge rapidly without learning the holistic semantics. Thus, band dropping and masking are necessary. 2.2) Qualitative analysis As mentioned in Sec 1, S3R acquires an adequate understanding of complex spectral structures and pathological characteristics of different morphologies. First, beta’s Gaussian distribution (Fig 3a) reflects the spectral characteristic of HSIs, which makes CR focus on learning inherent structures. Second, we find that positive and negative pixels in HSIs have different beta curves, which suggests that regressing beta can help the model to obtain discriminability for classification. As for BR, it could be regarded as a novel MIM method designed for learning the semantic information of HSIs. 3) Metrics and visualization We followed current popular self-supervised algorithms [4,9,11] and reported the accuracy metric. As for stability, the std of ResNet18 on PLGC is 0.21 (CR), 0.42 (BR), 0.63 (SimSiam) (@R2 & R3). Our proposed method considers the low-rank properties of hyper spectra. Thus, experimental results for 3-band S3R are not doable, and S3R could only be applied to HSIs (@R3). In our 3-band baseline experiments, we randomly select 3 bands (various in each iteration) which still considers all bands’ information to avoid unfair comparisons (@R4). For regression-related visualization, the average L1 loss in both BR and CR during pretraining is under 0.01, which indicates the regressed coefficient and band are similar with groundtruth (@R4). Minor: @R2: Characteristics of HSI Instead of assigning RGB colors to each pixel, detailed spectral information is presented in hundreds of narrow spectral bands in HSIs. Since different molecules’ responses to light are different, utilizing HSI, spectral information across bands acquires biochemical properties invisible to the naked eye from both stained and unstained histological specimen. @R4: Details in fine-tuning The batch size was set as 8 (all bands) or 32 (3-band) in contrastive baselines (dual networks) and 16 in ours and MIM baselines. The classification head is a simple MLP and we didn’t freeze any modules during fine-tuning.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper presents a self-supervised learning method for hyperspectral images. In the first-round of review, two reviewers provided negative recommendation. But one has been updated as a positive final recommendation. I think the rebuttal have addressed the concerns about the original contributions, rigor of the experiments, the metrics and visualization. The method is one of the early works of applying self-supervised learning to hyperspectral images. For these reasons, the recommendation is toward acceptance.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
9
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper proposes a self-supervised pretraining method by constructing regression tasks regarding hyperspectral images, which shows better performance when compared with commonly used contrastive learning strategies. The proposed method is a novel and promising direction for self-supervised learning. There are some concerns regarding the details of technical implementations and results discussions. The authors have addressed most of them in the rebuttal. It is suggested that the authors provide more clarifications in the final version considering the reviewers’ comments.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
9
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper presents a self-supervised learning method for Hyperspectral histopathology Image Classification. The rebuttal has provided satisfactory response to concerns over performance evaluation and some method details. Overall, this seems an interesting study with some good results. The final version should carefully address the reviewers’ comments.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
3