Hyperspectral super-resolution combining multi-receptive field features with spectral-spatial attention

doi:10.6046/zrzyyg.2021115

Abstract
Figures/Tables
References
Related Articles
Metrics

Download: PDF(4409 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

To address the problem that image details are liable to be lost in the process of hyperspectral super-resolution, this study proposed a hyperspectral super-resolution algorithm that combines multi-receptive field features and spectral-spatial attention. By fully using the high- and low-frequency information in hyperspectral images, this algorithm reduces the loss of image details and improves the hyperspectral super-resolution effects. First, in the feature extraction stage, convolution with different sizes of convolutional kernels is used to obtain multi-scale receptive field features. This assists in extracting more high- and low-frequency information from low-resolution images, thus retaining the features of original images. Then, the acquired image features are enhanced by the spatial-spectral attention mechanism, and the reconstruction of spatial-dimension features is conducted using spectral-dimension information. Finally, the features of various groups are fused, and the checkerboard pattern is relieved by applying the pixel deconvolution layer. As a result, clear and high-resolution images can be produced. The proposed super-resolution algorithm that combines multi-receptive field features with spectral-spatial attention was applied to two public datasets Chikusei and Pavia Center Scene, achieving peak signal-to-noise ratios of 39.869 7 and 31.942 2, respectively and structural similarity of 0.937 6 and 0.878 6, respectively. Therefore, the super-resolution algorithm enjoys obvious performance advantages compared to the latest super-resolution algorithms. Overall, the algorithm proposed in this study integrates the advantages of the multi-receptive field feature extraction module and the spatial-spectral attention module and can significantly improve image details.

Keywords hyperspectral image image super resolution multi-receptive field feature extraction attention mechanism

ZTFLH:

TP751.1

Corresponding Authors: WAND Yaxuan E-mail: quhaicheng@lntu.edu.cn;940556702@qq.com

Issue Date: 14 March 2022

	Service

	E-mail this article
	E-mail Alert
	RSS
	Articles by authors

	Haicheng QU
	Yaxuan WAND
	Lei SHEN

Cite this article:

Haicheng QU,Yaxuan WAND,Lei SHEN. Hyperspectral super-resolution combining multi-receptive field features with spectral-spatial attention[J]. Remote Sensing for Natural Resources, 2022, 34(1): 43-52.

URL:

https://www.gtzyyg.com/EN/10.6046/zrzyyg.2021115 OR https://www.gtzyyg.com/EN/Y2022/V34/I1/43

Fig.1 High and low frequency information on the Chikusei and the Pavia Centre scene dataset

Fig.2 Overall network structure

Fig.3 Multi-receptive field feature extraction attention block

Fig.4 Multi-receptive field feature extraction module

Fig.5 Space spectrum combined with attention module

Tab.1 Experimental comparison of Chikusei dataset in different modules

Fig.6 Module comparison of the Chikusei dataset

Fig.7 Module comparison of the Pavia Centre scene dataset

Fig.8 Effect of overlap factor on MPSNR

Fig.9 Number of spectral channels per group

Tab.2 Comparison results of different algorithms on the Chikusei dataset

Tab.3 Comparison results of different algorithms on Pavia Centre scene dataset

Tab.4 Comparison results of different algorithms on CAVE dataset

[1]	Muhammad U, Arif M, Ajmal M. Hyperspectral face recognition using 3D-DCT and partial least squares[C]// Burghardt: BMVA Press.UK:University of Bristol,2013:57.1- 57.10.
[2]	Lowe A, Harrison N, French A P. Hyperspectral image analysis techniques for the detection and classification of the early onset of plant disease and stress[J]. Plant Methods, 2017, 13(1):80-92. doi: 10.1186/s13007-017-0233-z url: http://plantmethods.biomedcentral.com/articles/10.1186/s13007-017-0233-z
[3]	Lin J, Clancy N T, Qi J, et al. Dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks[J]. Medical Image Analysis, 2018, 48:162-176. doi: 10.1016/j.media.2018.06.004 url: https://linkinghub.elsevier.com/retrieve/pii/S1361841518303736
[4]	Akhtar N, Shafait F, Mian A S. Sparse spatio-spectral representation for hyperspectral image super-resolution[C]// European Conference on Computer Vision, 2014:63-78.
[5]	Dian R, Li S, Guo A, et al. Deep hyperspectral image sharpening[J]. IEEE Transactions on Neural Network Learning Systems, 2018, 29(11):5345-5355. doi: 10.1109/TNNLS.2018.2798162 url: https://ieeexplore.ieee.org/document/8295275/
[6]	Huang H, Yu J, Sun W. Super-resolution mapping via multi-dictionary based sparse representation[C]// IEEE International Conference on Acoustic,Speech and Signal Processing, 2014:3523-3527.
[7]	练秋生, 张钧芹, 陈书贞. 基于两级字典与分频带字典的图像超分辨率算法[J]. 自动化学报, 2013, 39(8):1310-1320.
[7]	Lian Q S, Zhang J Q, Chen S Z. Image superresolution algorithm based on two-level dictionary and subband dictionary[J]. Acta Automatica Sinica, 2013, 39(8):1310-1320. doi: 10.3724/SP.J.1004.2013.01310 url: http://www.aas.net.cn/cn/article/doi/10.3724/SP.J.1004.2013.01310
[8]	Yang J, Wright J, Huang T S, et al. Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19(11):2861-2873. doi: 10.1109/TIP.2010.2050625 url: http://ieeexplore.ieee.org/document/5466111/
[9]	Li X, Wei H W, Zhang H Q. Super-resolution reconstruction of single remote sensing image combined with deep learning[J]. Journal of Image and Graphics, 2018, 23(2):209-218.
[10]	Qiao J, Song H, Zhang K, et al. Image super-resolution using conditional generative adversarial network[J]. IET Image Processing, 2019, 13(14):2673-2679. doi: 10.1049/ipr2.v13.14 url: https://onlinelibrary.wiley.com/toc/17519667/13/14
[11]	Pan J T, Kevin M G, Elisa S, et al. Shallow and deep convolutional networks for saliency prediction[C]// IEEE Conference on Computer Vision and Pattern Recognition, 2016:598-606.
[12]	Li K, Dai D X, Ender K, et al. Hyperspectral image super-resolution with spectral mixup and heterogeneous datasets[EB/OL].(2021-1-19)[2021-3-8].https://arxiv.org/pdf/2101.07589.htm. url: https://arxiv.org/pdf/2101.07589.htm.
[13]	Aggarwal H K, Majumdar A. Hyperspectral image denoising using spatio-spectral total variation[J]. IEEE Geoscience and Remote Sensing Letters, 2016, 13(3):442-446.
[14]	Loncan L, De Almeida L B, Briottet X, et al. Hyperspectral pansharpening[J]. IEEE Geoscience and Remote Sensing Magazine, 2015, 3(3):27-46. doi: 10.1109/MGRS.2015.2440094 url: http://ieeexplore.ieee.org/document/7284770/
[15]	Wald L. Data fusion:Definitions and architectures:Fusion of images of different spatial resolutions[M]. Presses des MINES,Computer Science, 2002.
[16]	Peng J Y, Shi C Y, Laugeman E, et al. Implementation of the structural SIMilarity (SSIM) index as a quantitative evaluation tool for dose distribution error detection[J]. Medical Physics, 2020, 47(4):1907-1919. doi: 10.1002/mp.v47.4 url: https://onlinelibrary.wiley.com/toc/24734209/47/4
[17]	Mei S, Yuan X, Ji J, et al. Hyperspectral image spatial super-resolution via 3D full convolutional neural network[J]. Remote Sensing, 2017, 9(11):1139. doi: 10.3390/rs9111139 url: http://www.mdpi.com/2072-4292/9/11/1139
[18]	李成轶, 田淑芳. 基于字典学习的遥感影像超分辨率融合方法[J]. 自然资源遥感, 2017, 29(1):50-56.doi: 10.6046/gtzyyg.2017.01.08. doi: 10.6046/gtzyyg.2017.01.08
[18]	Li C Y, Tian S F. Super-resolution fusion method for remote sensing image based on dictionary learning[J]. Remote Sensing for Land and Resource, 2017, 29(1):50-56.doi: 10.6046/gtzyyg.2017.01.08. doi: 10.6046/gtzyyg.2017.01.08
[19]	Kim J, Lee J K, Lee K M. Accurate image super-resolution using very deep convolutional networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2016:1646-1654.
[20]	Yuan Y, Zheng X, Lu X. Hyperspectral image superresolution by transfer learning[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2017, 10(5):1963-1974. doi: 10.1109/JSTARS.4609443 url: https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=4609443
[21]	Lim B, Son S, Kim H, et al. Enhanced deep residual networks for single image super-resolution[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017:136-144.
[22]	Zhang Y, Li K P, Li K, et al. Image super-resolution using very deep residual channel attention networks[C]// Proceedings of the European Conference on Computer Vision (ECCV), 2018:286-301.
[23]	Dai T, Cai J, Zhang Y, et al. Second-order attention network for single image super-resolution[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2019:11065-11074.
[24]	Mei S, Yuan X, Ji J, et al. Hyperspectral image spatial super-resolution via 3D full convolutional neural network[J]. Remote Sensing, 2017, 9(11):1139. doi: 10.3390/rs9111139 url: http://www.mdpi.com/2072-4292/9/11/1139
[25]	Li Y, Zhang L, Dingl C, et al. Single hyperspectral image super-resolution with grouped deep recursive residual network[C]// Proceedings of the IEEE International Conference on Multimedia Big Data, 2018:1-4.
[26]	Sidorov O, Hardeberg J Y. Deep hyperspectral prior:Single-image denoising,inpainting,super-resolution[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop, 2019:3844-3851.

[1]	NIU Xianghua, HUANG Wei, HUANG Rui, JIANG Sili. A high-fidelity method for thin cloud removal from remote sensing images based on attentional feature fusion[J]. Remote Sensing for Natural Resources, 2023, 35(3): 116-123.
[2]	ZHENG Zongsheng, LIU Haixia, WANG Zhenhua, LU Peng, SHEN Xukun, TANG Pengfei. Improved 3D-CNN-based method for surface feature classification using hyperspectral images[J]. Remote Sensing for Natural Resources, 2023, 35(2): 105-111.
[3]	ZHANG Guojian, LIU Shengzhen, SUN Yingjun, YU Kaijie, LIU Lina. Hyperspectral anomaly detection based on the weakly supervised robust autoencoder[J]. Remote Sensing for Natural Resources, 2023, 35(2): 167-175.
[4]	JIN Yuanhang, XU Maolin, ZHENG Jiayuan. A dead tree detection algorithm based on improved YOLOv4-tiny for UAV images[J]. Remote Sensing for Natural Resources, 2023, 35(1): 90-98.
[5]	SHEN Jun’ao, MA Mengting, SONG Zhiyuan, LIU Tingzhou, ZHANG Wei. Water information extraction from high-resolution remote sensing images using the deep-learning based semantic segmentation model[J]. Remote Sensing for Natural Resources, 2022, 34(4): 129-135.
[6]	ZHANG Pengqiang, GAO Kuiliang, LIU Bing, TAN Xiong. Classification of hyperspectral images based on deep Transformer network combined with spatial-spectral information[J]. Remote Sensing for Natural Resources, 2022, 34(3): 27-32.
[7]	SUN Xiao, XU Linlin, WANG Xiaoyang, TIAN Ye, WANG Wei, ZHANG Zhongyue. Mineral identification from hyperspectral images based on the optimized K-P-Means unmixing method[J]. Remote Sensing for Natural Resources, 2022, 34(3): 43-49.
[8]	WANG Yiru, WANG Guanghui, YANG Huachao, LIU Huijie. A method for color consistency of remote sensing images based on generative adversarial networks[J]. Remote Sensing for Natural Resources, 2022, 34(3): 65-72.
[9]	KONG Ailing, ZHANG Chengming, LI Feng, HAN Yingjuan, SUN Huanying, DU Manfei. Knowledge-based remote sensing image fusion method[J]. Remote Sensing for Natural Resources, 2022, 34(2): 47-55.
[10]	LIU Guangjin, WANG Guanghui, BI Weihua, LIU Huijie, YANG Huachao. Cloud detection algorithm of remote sensing image based on DenseNet and attention mechanism[J]. Remote Sensing for Natural Resources, 2022, 34(2): 88-96.
[11]	XUE Bai, WANG Yizhe, LIU Shuhan, YUE Mingyu, WANG Yiying, ZHAO Shihu. Change detection of high-resolution remote sensing images based on Siamese network[J]. Remote Sensing for Natural Resources, 2022, 34(1): 61-66.
[12]	GAO Wenlong, ZHANG Shengwei, LIN Xi, LUO Meng, REN Zhaoyi. The remote sensing-based estimation and spatial-temporal dynamic analysis of SOM in coal mining[J]. Remote Sensing for Natural Resources, 2021, 33(4): 235-242.
[13]	LIU Yongmei, FAN Hongjian, GE Xinghua, LIU Jianhong, WANG Lei. Estimation accuracy of fractional vegetation cover based on normalized difference vegetation index and UAV hyperspectral images[J]. Remote Sensing for Natural Resources, 2021, 33(3): 11-17.
[14]	WANG Hua, LI Weiwei, LI Zhigang, CHEN Xueye, SUN Le. Hyperspectral image classification based on multiscale superpixels[J]. Remote Sensing for Natural Resources, 2021, 33(3): 63-71.
[15]	LIU Wanjun, GAO Jiankang, QU Haicheng, JIANG Wentao. Ship detection based on multi-scale feature enhancement of remote sensing images[J]. Remote Sensing for Natural Resources, 2021, 33(3): 97-106.

Viewed

Full text

Abstract

Cited

Shared

Discussed