A pansharpening algorithm for remote sensing images based on high-frequency domain and multi-depth dilated network
GUO Penghao1(), QIU Jianlin2, ZHAO Shunan3
1. School of Computer and Information Engineering, Nantong Institute of Technology, Nantong 226001, China 2. School of Information Science and Technology, Nantong University, Nantong 226001, China 3. North Night Vision Science and Technology (Nanjing) Research Institute Co., Ltd., Nanjing 211100, China
The pansharpening of remote sensing images is a process that extracts the spectral information of multispectral images and the structural information of panchromatic images and then fuse the information to form high-resolution multispectral remote sensing images, which, however, suffer a lack of spectral or structural information. To mitigate this problem, this study proposed a pansharpening algorithm for remote sensing images based on a multi-depth neural network. This algorithm includes a structural protection module and a spectral protection module. The structural protection module extracts the high-frequency information of panchromatic and multispectral images through filtering and then extracts the multi-scale information of the images using a multi-depth neural network. The purpose is to improve the spatial information extraction ability of the model and to reduce the risk of overfitting. The spectral protection module connects the upsampled multispectral images with the structural protection module through a skip connection. To validate the effectiveness of the new algorithm, this study, under consistent experimental conditions, compared the new algorithm with multiple pansharpening algorithms for remote sensing images and assessed these algorithms from the angles of subjective visual effects and objective evaluation. The results indicate that the algorithm proposed in this study can effectively protect the spectral information of multispectral images and the structural information of panchromatic images in solving the lack of structural information in current algorithms for image fusion.
Carper W R, Lillesand T M, Kiefer R. The use of intensity-hue-saturation transformations for merging SPOT panchromatic and multispectral image data[J]. Photogrammetric Engineering and Remote Sensing, 1990, 56:457-467.
[2]
Laben C A, Brower B V. Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening:US6011875[P]. 2000-01-04.
[3]
Nason G P, Silverman B W. The stationary wavelet transform and some statistical applications[M]//Antoniadis A,Oppenheim G.Wavelets and Statistics. New York: Springer, 1995:281-299.
[4]
Hallabia H, Hamam H, Ben Hamida A. A context-driven pansharpening method using superpixel based texture analysis[J]. International Journal of Image and Data Fusion, 2021, 12(1):1-22.
[5]
Ballester C, Caselles V, Igual L, et al. A variational model for P+XS image fusion[J]. International Journal of Computer Vision, 2006, 69(1):43-58.
[6]
Fang F, Li F, Shen C, et al. A variational approach for pan-sharpening[J]. IEEE Transactions on Image Processing:A Publication of the IEEE Signal Processing Society, 2013, 22(7):2822-2834.
[7]
Fu X, Lin Z, Huang Y, et al. A variational pan-sharpening with local gradient constraints[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Long Beach,CA,USA.IEEE, 2019:10257-10266.
[8]
Masi G, Cozzolino D, Verdoliva L, et al. Pansharpening by convolutional neural networks[J]. Remote Sensing, 2016, 8(7):594.
[9]
Wang T T, Fang F M, Li F, et al. High-quality Bayesian pansharpening[J]. IEEE Transactions on Image Processing:a Publication of the IEEE Signal Processing Society, 2019, 28(1):227-239.
[10]
Yang Y, Lu H, Huang S, et al. An efficient and high-quality pansharpening model based on conditional random fields[J]. Information Sciences, 2021, 553:1-18.
[11]
Ye F, Guo Y, Zhuang P. Pan-sharpening via a gradient-based deep network prior[J]. Signal Processing:Image Communication, 2019, 74:322-331.
[12]
Guo P, Zhuang P, Guo Y. Bayesian pan-sharpening with multiorder gradient-based deep network constraints[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13:950-962.
[13]
Wang D, Li Y, Ma L, et al. Going deeper with densely connected convolutional neural networks for multispectral pansharpening[J]. Remote Sensing, 2019, 11(22):2608.
[14]
Fang F, Zhang G, Li F, et al. Framelet based pan-sharpening via a variational method[J]. Neurocomputing, 2014, 129:362-377.
[15]
Scarpa G, Vitale S, Cozzolino D. Target-adaptive CNN-based pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(9):5443-5457.
[16]
Yuan Q, Wei Y, Meng X, et al. A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(3):978-989.
[17]
Liu L, Wang J, Zhang E, et al. Shallow-deep convolutional network and spectral-discrimination-based detail injection for multispectral imagery pan-sharpening[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13:1772-1783.
[18]
Hu J, Hu P, Kang X, et al. Pan-sharpening via multiscale dynamic convolutional neural network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(3):2231-2244.
[19]
Fu X, Wang W, Huang Y, et al. Deep multiscale detail networks for multiband spectral image sharpening[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 32(5):2090-2104.
[20]
Choi J, Yu K, Kim Y. A new adaptive component-substitution-based satellite image fusion by using partial replacement[J]. IEEE Transactions on Geoscience and Remote Sensing, 2011, 49(1):295-309.
Tan H, Zhang R J, Fan W F, et al. Classification and detection of radiation anomalies in Chinese optical satellite images by integrating multi-scale features[J]. Remote Sensing for Natural Resources, 2022, 34(4):97-104.doi:10.6046/zrzyyg.2021377.
[22]
Garbin C, Zhu X, Marques O. Dropout vs.batch normalization:An empirical study of their impact to deep learning[J]. Multimedia Tools and Applications, 2020, 79(19):12777-12815.
[23]
Nair V, Hinton G E. Rectified linear units improve restricted boltzmann machines[C]// Proceedings of the 27th International Conference on International Conference on Machine Learning.ACM, 2010:807-814.
Han Y L, Cui P X, Yang S H, et al. Classification of hyperspectral image based on feature fusion of residual network[J]. Remote Sensing for Land and Resources, 2021, 33(2):11-19.doi:10.6046/gtzyyg.2020209.
[25]
Qiu D, Zheng L, Zhu J, et al. Multiple improved residual networks for medical image super-resolution[J]. Future Generation Computer Systems, 2021, 116:200-208.
[26]
Garzelli A, Nencini F, Capobianco L. Optimal MMSE pan sharpening of very high resolution multispectral images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2008, 46(1):228-236.
[27]
Alparone L, Wald L, Chanussot J, et al. Comparison of pansharpening algorithms:Outcome of the 2006 GRS-S data-fusion contest[J]. IEEE Transactions on Geoscience and Remote Sensing, 2007, 45(10):3012-3021.
[28]
Khan M M, Chanussot J, Condat L, et al. Indusion:Fusion of multispectral and panchromatic images using the induction scaling technique[J]. IEEE Geoscience and Remote Sensing Letters, 2008, 5(1):98-102.
2024, Vol. 36 
), 邱建林2, 赵淑男3
