A pansharpening algorithm for remote sensing images based on high-frequency domain and multi-depth dilated network

doi:10.6046/zrzyyg.2023133

Abstract
Figures/Tables
References
Related Articles
Metrics

Download: PDF(5310 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

The pansharpening of remote sensing images is a process that extracts the spectral information of multispectral images and the structural information of panchromatic images and then fuse the information to form high-resolution multispectral remote sensing images, which, however, suffer a lack of spectral or structural information. To mitigate this problem, this study proposed a pansharpening algorithm for remote sensing images based on a multi-depth neural network. This algorithm includes a structural protection module and a spectral protection module. The structural protection module extracts the high-frequency information of panchromatic and multispectral images through filtering and then extracts the multi-scale information of the images using a multi-depth neural network. The purpose is to improve the spatial information extraction ability of the model and to reduce the risk of overfitting. The spectral protection module connects the upsampled multispectral images with the structural protection module through a skip connection. To validate the effectiveness of the new algorithm, this study, under consistent experimental conditions, compared the new algorithm with multiple pansharpening algorithms for remote sensing images and assessed these algorithms from the angles of subjective visual effects and objective evaluation. The results indicate that the algorithm proposed in this study can effectively protect the spectral information of multispectral images and the structural information of panchromatic images in solving the lack of structural information in current algorithms for image fusion.

Keywords remote sensing pansharpening multi-depth network multi-scale learning skip connection spatial and spectral fusion

ZTFLH:

TP751.1

Issue Date: 03 September 2024

	Service

	E-mail this article
	E-mail Alert
	RSS
	Articles by authors

	Penghao GUO
	Jianlin QIU
	Shunan ZHAO

Cite this article:

Penghao GUO,Jianlin QIU,Shunan ZHAO. A pansharpening algorithm for remote sensing images based on high-frequency domain and multi-depth dilated network[J]. Remote Sensing for Natural Resources, 2024, 36(3): 146-153.

URL:

https://www.gtzyyg.com/EN/10.6046/zrzyyg.2023133 OR https://www.gtzyyg.com/EN/Y2024/V36/I3/146

Fig.1 Workflow of the proposed method

Fig.2 High-frequency domain input module

Fig.3 MDS block of network

Fig.4 Comparisons of reduce scale experiment

Tab.1 Quality metircs of reduce scale dataset

Fig.5-1 Comparisons of original-scale experiment

Fig.5-2 Comparisons of original-scale experiment

算法	$D s$	$D λ$	QNR
IHS	0.141 6	0.255 3	0.639 6
BDSD	0.065 4	0.179 9	0.767 1
PRACS	0.087 9	0.228 2	0.713 9
MTF-GLP	0.121 5	0.220 7	0.685 5
Indusion	0.113 9	0.603 0	0.350 6
PNN	0.068 5	0.603 5	0.368 3
DMDP	0.085 2	0.168 8	0.761 1
GND	0.073 8	0.110 5	0.822 7
本文算法	0.050 6	0.121 6	0.833 7
理想值	0	0	1

Tab.2 Quality metircs of original scale dataset

Fig.6 Original-scale experiment on an IKONOS image

Tab.3 Quality metircs of Ablation experiment

[1]	Carper W R, Lillesand T M, Kiefer R. The use of intensity-hue-saturation transformations for merging SPOT panchromatic and multispectral image data[J]. Photogrammetric Engineering and Remote Sensing, 1990, 56:457-467.
[2]	Laben C A, Brower B V. Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening:US6011875[P]. 2000-01-04.
[3]	Nason G P, Silverman B W. The stationary wavelet transform and some statistical applications[M]//Antoniadis A,Oppenheim G.Wavelets and Statistics. New York: Springer, 1995:281-299.
[4]	Hallabia H, Hamam H, Ben Hamida A. A context-driven pansharpening method using superpixel based texture analysis[J]. International Journal of Image and Data Fusion, 2021, 12(1):1-22.
[5]	Ballester C, Caselles V, Igual L, et al. A variational model for P+XS image fusion[J]. International Journal of Computer Vision, 2006, 69(1):43-58.
[6]	Fang F, Li F, Shen C, et al. A variational approach for pan-sharpening[J]. IEEE Transactions on Image Processing:A Publication of the IEEE Signal Processing Society, 2013, 22(7):2822-2834.
[7]	Fu X, Lin Z, Huang Y, et al. A variational pan-sharpening with local gradient constraints[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Long Beach,CA,USA.IEEE, 2019:10257-10266.
[8]	Masi G, Cozzolino D, Verdoliva L, et al. Pansharpening by convolutional neural networks[J]. Remote Sensing, 2016, 8(7):594.
[9]	Wang T T, Fang F M, Li F, et al. High-quality Bayesian pansharpening[J]. IEEE Transactions on Image Processing:a Publication of the IEEE Signal Processing Society, 2019, 28(1):227-239.
[10]	Yang Y, Lu H, Huang S, et al. An efficient and high-quality pansharpening model based on conditional random fields[J]. Information Sciences, 2021, 553:1-18.
[11]	Ye F, Guo Y, Zhuang P. Pan-sharpening via a gradient-based deep network prior[J]. Signal Processing:Image Communication, 2019, 74:322-331.
[12]	Guo P, Zhuang P, Guo Y. Bayesian pan-sharpening with multiorder gradient-based deep network constraints[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13:950-962.
[13]	Wang D, Li Y, Ma L, et al. Going deeper with densely connected convolutional neural networks for multispectral pansharpening[J]. Remote Sensing, 2019, 11(22):2608.
[14]	Fang F, Zhang G, Li F, et al. Framelet based pan-sharpening via a variational method[J]. Neurocomputing, 2014, 129:362-377.
[15]	Scarpa G, Vitale S, Cozzolino D. Target-adaptive CNN-based pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(9):5443-5457.
[16]	Yuan Q, Wei Y, Meng X, et al. A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(3):978-989.
[17]	Liu L, Wang J, Zhang E, et al. Shallow-deep convolutional network and spectral-discrimination-based detail injection for multispectral imagery pan-sharpening[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13:1772-1783.
[18]	Hu J, Hu P, Kang X, et al. Pan-sharpening via multiscale dynamic convolutional neural network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(3):2231-2244.
[19]	Fu X, Wang W, Huang Y, et al. Deep multiscale detail networks for multiband spectral image sharpening[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 32(5):2090-2104.
[20]	Choi J, Yu K, Kim Y. A new adaptive component-substitution-based satellite image fusion by using partial replacement[J]. IEEE Transactions on Geoscience and Remote Sensing, 2011, 49(1):295-309.
[21]	谭海, 张荣军, 樊文锋, 等. 融合多尺度特征的国产光学影像辐射异常分类检测[J]. 自然资源遥感, 2022, 34(4):97-104.doi:10.6046/zrzyyg.2021377.
[21]	Tan H, Zhang R J, Fan W F, et al. Classification and detection of radiation anomalies in Chinese optical satellite images by integrating multi-scale features[J]. Remote Sensing for Natural Resources, 2022, 34(4):97-104.doi:10.6046/zrzyyg.2021377.
[22]	Garbin C, Zhu X, Marques O. Dropout vs.batch normalization:An empirical study of their impact to deep learning[J]. Multimedia Tools and Applications, 2020, 79(19):12777-12815.
[23]	Nair V, Hinton G E. Rectified linear units improve restricted boltzmann machines[C]// Proceedings of the 27th International Conference on International Conference on Machine Learning.ACM, 2010:807-814.
[24]	韩彦岭, 崔鹏霞, 杨树瑚, 等. 基于残差网络特征融合的高光谱图像分类[J]. 国土资源遥感, 2021, 33(2):11-19.doi:10.6046/gtzyyg.2020209.
[24]	Han Y L, Cui P X, Yang S H, et al. Classification of hyperspectral image based on feature fusion of residual network[J]. Remote Sensing for Land and Resources, 2021, 33(2):11-19.doi:10.6046/gtzyyg.2020209.
[25]	Qiu D, Zheng L, Zhu J, et al. Multiple improved residual networks for medical image super-resolution[J]. Future Generation Computer Systems, 2021, 116:200-208.
[26]	Garzelli A, Nencini F, Capobianco L. Optimal MMSE pan sharpening of very high resolution multispectral images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2008, 46(1):228-236.
[27]	Alparone L, Wald L, Chanussot J, et al. Comparison of pansharpening algorithms:Outcome of the 2006 GRS-S data-fusion contest[J]. IEEE Transactions on Geoscience and Remote Sensing, 2007, 45(10):3012-3021.
[28]	Khan M M, Chanussot J, Condat L, et al. Indusion:Fusion of multispectral and panchromatic images using the induction scaling technique[J]. IEEE Geoscience and Remote Sensing Letters, 2008, 5(1):98-102.

[1]	ZHANG Chungui, PENG Jida. Air freshness monitoring technology based on meteorology and remote sensing[J]. Remote Sensing for Natural Resources, 2024, 36(3): 163-173.
[2]	LIU Yongxin, ZHANG Siyuan, BIAN Peng, WANG Pijun, YUAN Shuai. Exploring the spatio-temporal evolution of land cover types in the Bayannur section of the Yellow River basin from 1989 to 2020[J]. Remote Sensing for Natural Resources, 2024, 36(2): 207-217.
[3]	LI Lei, SUN Xiyan, JI Yuanfa, FU Wentao. Hyperspectral image classification based on superpixel segmentation and extended multi-attribute profiles[J]. Remote Sensing for Natural Resources, 2023, 35(4): 114-121.
[4]	DENG Dingzhu. Deep learning-based cloud detection method for multi-source satellite remote sensing images[J]. Remote Sensing for Natural Resources, 2023, 35(4): 9-16.
[5]	LONG En, LYU Shouye, QIAN Guodong, LIAN Cuiping, YANG Yuke, CHEN Lingyan. A space-based remote sensing service mode based on cloud+ terminals[J]. Remote Sensing for Natural Resources, 2023, 35(1): 258-264.
[6]	LIU Guangjin, WANG Guanghui, BI Weihua, LIU Huijie, YANG Huachao. Cloud detection algorithm of remote sensing image based on DenseNet and attention mechanism[J]. Remote Sensing for Natural Resources, 2022, 34(2): 88-96.
[7]	QU Haicheng, WAND Yaxuan, SHEN Lei. Hyperspectral super-resolution combining multi-receptive field features with spectral-spatial attention[J]. Remote Sensing for Natural Resources, 2022, 34(1): 43-52.
[8]	LIU Wanjun, GAO Jiankang, QU Haicheng, JIANG Wentao. Ship detection based on multi-scale feature enhancement of remote sensing images[J]. Remote Sensing for Natural Resources, 2021, 33(3): 97-106.
[9]	QIU Yifan, CHAI Dengfeng. A deep learning method for Landsat image cloud detection without manually labeled data[J]. Remote Sensing for Land & Resources, 2021, 33(1): 102-107.
[10]	WEI Hongyu, ZHAO Yindi, DONG Jihong. Cooling tower detection based on the improved RetinaNet[J]. Remote Sensing for Land & Resources, 2020, 32(4): 68-73.
[11]	LIU Shangwang, CUI Zhiyong, LI Daoyi. Multi-task learning for building object semantic segmentation of remote sensing image based on Unet network[J]. Remote Sensing for Land & Resources, 2020, 32(4): 74-83.
[12]	Kaixuan LIANG, Guifang ZHANG, Haoran ZHANG. A method for extracting alluvial fan based on DEM and remote sensing data[J]. Remote Sensing for Land & Resources, 2020, 32(2): 138-145.
[13]	Renbo SONG, Yuxin ZHU, Shangshan DING, Qiaoning HE, Xiyuan WANG, Yuexiang WANG. An automatic method for extracting skeleton lines from arbitrary polygons based on GIS spatial analysis[J]. Remote Sensing for Land & Resources, 2020, 32(1): 51-59.
[14]	Liuqing WU, Xiangyun HU. Automatic building detection of high-resolution remote sensing images based on multi-scale and multi-feature[J]. Remote Sensing for Land & Resources, 2019, 31(1): 71-78.
[15]	Yongmei ZHANG, Haiyan SUN, Yulong XU. An improved multispectral image segmentation method based on super-pixels[J]. Remote Sensing for Land & Resources, 2019, 31(1): 58-64.

Viewed

Full text

Abstract

Cited

Shared

Discussed