With the fast development and wide application of remote sensing technology, remote sensing images with higher quality are needed. However, it is difficult to directly acquire high-resolution, multispectral remote sensing images. To obtain high-quality images by integrating the information from different imaging sensors, pansharpening technology emerged. Pansharpening is an effective method used to obtain multispectral images with high spatial resolution. Many scholars have studied this method and obtained fruitful achievements. In recent years, deep learning theory has developed rapidly and has been widely applied in pansharpening. This study aims to systematically introduce the progress in pansharpening and promote its development. To this end, this study first introduced the traditional, classical pansharpening methods, followed by commonly used remote sensing satellites. Then, this study elaborated on the pansharpening methods based on deep learning from the perspective of supervised learning, unsupervised learning, and semi-supervised learning. After that, it described and analyzed loss functions. To demonstrate the superiority of the pansharpening methods based on deep learning and analyze the effects of loss functions, this study conducted remote sensing image fusion experiments. Finally, this study presented the future prospects of the pansharpening methods based on deep learning.
胡建文, 汪泽平, 胡佩. 基于深度学习的空谱遥感图像融合综述[J]. 自然资源遥感, 2023, 35(1): 1-14.
HU Jianwen, WANG Zeping, HU Pei. A review of pansharpening methods based on deep learning. Remote Sensing for Natural Resources, 2023, 35(1): 1-14.
Xiao L, Liu P F, Li H. Progress and challenges in the fusion of multisource spatial-spectral remote sensing images[J]. Journal of Image and Graphics, 2020, 25(5):851-863.
[2]
Ghassemian H. A review of remote sensing image fusion methods[J]. Information Fusion, 2016, 32:75-89.
doi: 10.1016/j.inffus.2016.03.003
Yang L P, Ma M, Xie W, et al. Fusion algorithm evaluation of Landsat8 panchromatic and multispetral images in arid regions[J]. Remote Sensing for Land and Resources, 2019, 31(4):11-19.doi:10.6046/gtzyyg.2019.04.02.
doi: 10.6046/gtzyyg.2019.04.02
[5]
Weiss M, Jacob F, Duveiller G. Remote sensing for agricultural applications:A meta-review[J]. Remote Sensing of Environment, 2020, 236:111402-111420.
doi: 10.1016/j.rse.2019.111402
Liu W J, Gao J K, Qu H C, et al. Ship detection based on multi-scale feature enhancement of remote sensing images[J]. Remote Sensing for Natural Resources, 2021, 33(3):97-106.doi:10.6046/zrzyyg.20200372.
doi: 10.6046/zrzyyg.20200372
Wang L, Li X, Bao Y X, et al. Research progress of remote sensing application on transportation meteorological disasters[J]. Remote Sensing and for Land and Resources, 2018, 30(4):1-7.doi:10.6046/gtzyyg.2018.04.01.
doi: 10.6046/gtzyyg.2018.04.01
Li S T, Li C Y, Kang X D. Development status and future prospects of multi-source remote sensing image fusion[J]. National Remote Sensing Bulletin, 2021, 25(1):148-166.
Zhang L F, Peng M Y, Sun X J, et al. Progress and bibliometric analysis of remote sensing data fusion methods (1992—2018)[J]. Journal of Remote Sensing, 2019, 23(4):603-619.
[12]
Meng X, Shen H, Li H, et al. Review of the pansharpening methods for remote sensing images based on the idea of meta-analysis:Practical discussion and challenges[J]. Information Fusion, 2019, 46:102-113.
doi: 10.1016/j.inffus.2018.05.006
[13]
Javan F D, Samadzadegan F, Mehravar S, et al. A review of image fusion techniques for pan-sharpening of high-resolution satellite imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 171:101-117.
doi: 10.1016/j.isprsjprs.2020.11.001
[14]
Vivone G, Mura M D, Garzelli A, et al. A new benchmark based on recent advances in multispectral pansharpening:Revisiting pansharpening with classical and emerging pansharpening methods[J]. IEEE Geoscience and Remote Sensing Magazine, 2020, 9(1):53-81.
doi: 10.1109/MGRS.6245518
[15]
Huang W, Xiao L, Wei Z, et al. A new pan-sharpening method with deep neural networks[J]. IEEE Geoscience and Remote Sensing Letters, 2015, 12(5):1037-1041.
doi: 10.1109/LGRS.2014.2376034
[16]
Masi G, Cozzolino D, Verdoliva L, et al. Pansharpening by convolutional neural networks[J]. Remote Sensing, 2016, 8(7):594-615.
doi: 10.3390/rs8070594
[17]
Shahdoosti H R, Ghassemian H. Combining the spectral PCA and spatial PCA fusion methods by an optimal filter[J]. Information Fusion, 2016, 27:150-160.
doi: 10.1016/j.inffus.2015.06.006
[18]
Laben C A, Brower B V. Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening:U.S.,US09/069232[P]. 2000-01-04.
[19]
Ghahremani M, Ghassemian H. Nonlinear IHS:A promising method for pan-sharpening[J]. IEEE Geoscience and Remote Sensing Letters, 2016, 13(11):1606-1610.
doi: 10.1109/LGRS.2016.2597271
[20]
Garzelli A, Nencini F, Capobianco L. Optimal MMSE pan sharpening of very high resolution multispectral images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2007, 46(1):228-236.
doi: 10.1109/TGRS.2007.907604
[21]
Aiazzi B, Alparone L, Baronti S, et al. MTF-tailored multiscale fusion of high-resolution MS and pan imagery[J]. Photogrammetric Engineering and Remote Sensing, 2006, 72(5):591-596.
doi: 10.14358/PERS.72.5.591
[22]
Vivone G, Restaino R, Chanussot J. Full scale regression-based injection coefficients for panchromatic sharpening[J]. IEEE Transactions on Image Processing, 2018, 27(7):3418-3431.
doi: 10.1109/TIP.2018.2819501
pmid: 29671744
[23]
Khan M M, Chanussot J, Condat L, et al. Indusion:Fusion of multispectral and panchromatic images using the induction scaling technique[J]. IEEE Geoscience and Remote Sensing Letters, 2008, 5(1):98-102.
doi: 10.1109/LGRS.2007.909934
[24]
Dong L, Yang Q, Wu H, et al. High quality multi-spectral and panchromatic image fusion technologies based on curvelet transform[J]. Neurocomputing, 2015, 159:268-274.
doi: 10.1016/j.neucom.2015.01.050
[25]
El-Mezouar M C, Kpalma K, Taleb N, et al. A pan-sharpening based on the non-subsampled contourlet transform:Application to WorldView-2 imagery[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2014, 7(5):1806-1815.
doi: 10.1109/JSTARS.4609443
Wu Y Q, Wang Z L. Multispectral and panchromatic image fusion using chaotic Bee Colony optimization in NSST domain[J]. Journal of Remote Sensing, 2017, 21(4):549-557.
[28]
Li S, Yang B. A new pan-sharpening method using a compressed sensing technique[J]. IEEE Transactions on Geoscience and Remote Sensing, 2010, 49(2):738-746.
doi: 10.1109/TGRS.2010.2067219
[29]
Yin H. PAN-guided cross-resolution projection for local adaptive sparse representation-based pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(7):4938-4950.
doi: 10.1109/TGRS.36
Li C Y, Tian S F. Super-resolution fusion method for remote sensing image based on dictionary learning[J]. Remote Sensing and for Land and Resources, 2017, 29(1):50-56.doi:10.6046/gtzyyg.2017.01.08.
doi: 10.6046/gtzyyg.2017.01.08
[31]
Yin H. Sparse representation based pansharpening with details injection model[J]. Signal Processing, 2015, 113:218-227.
doi: 10.1016/j.sigpro.2014.12.017
[32]
Li S, Yin H, Fang L. Remote sensing image fusion via sparse representations over learned dictionaries[J]. IEEE Transactions on Geoscience and Remote Sensing, 2013, 51(9):4779-4789.
doi: 10.1109/TGRS.2012.2230332
[33]
Scarpa G, Vitale S, Cozzolino D. Target-adaptive CNN-based pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(9),5443-5457.
doi: 10.1109/TGRS.36
[34]
Wei Y, Yuan Q, Shen H, et al. Boosting the accuracy of multispectral image pansharpening by learning a deep residual network[J]. IEEE Geoscience and Remote Sensing Letters, 2017, 14(10),1795-1799.
doi: 10.1109/LGRS.2017.2736020
[35]
Yang J, Fu X, Hu Y, et al. PanNet:A deep network architecture for pan-sharpening[C]// Proceedings of the IEEE International Conference on Computer Vision.IEEE,Venice,Italy, 2017:5449-5457.
[36]
Zhang H, Ma J. GTP-PNet:A residual learning network based on gradient transformation prior for pansharpening[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 172:223-239.
doi: 10.1016/j.isprsjprs.2020.12.014
[37]
Yang Y, Tu W, Huang S, et al. PCDRN:Progressive cascade deep residual network for pansharpening[J]. Remote Sensing, 2020, 12(4):676.
doi: 10.3390/rs12040676
[38]
Huang W, Feng J, Wang H, et al. A new architecture of densely connected convolutional networks for pan-sharpening[J]. ISPRS International Journal of Geo-Information, 2020, 9(4):242.
doi: 10.3390/ijgi9040242
[39]
Peng J, Liu L, Wang J, et al. PSMD-Net:A novel pan-sharpening method based on a multiscale dense network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(6):4957-4971.
doi: 10.1109/TGRS.2020.3020162
[40]
Luo S, Zhou S, Qi Y. CSAFNET:Channel similarity attention fusion network for multispectral pansharpening[J]. IEEE Geoscience and Remote Sensing Letters, 2020.
Kong A L, Zhang C M, Li F, et al. Knowledge-based remote sensing imagery fusion method[J]. Remote Sensing for Natural Resources, 2022, 34(2):47-55.doi:10.6046/zrzyyg.2021179.
doi: 10.6046/zrzyyg.2021179
[42]
Jiang M, Shen H, Li J, et al. A differential information residual convolutional neural network for pansharpening[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 163:257-271.
doi: 10.1016/j.isprsjprs.2020.03.006
[43]
Lei D, Chen H, Zhang L, et al. NLRNet:An efficient nonlocal attention resnet for pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:5401113.
[44]
Shao Z, Cai J. Remote sensing image fusion with deep convolutional neural network[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(5),1656-1669.
doi: 10.1109/JSTARS.4609443
[45]
Liu X, Liu Q, Wang Y. Remote sensing image fusion based on two-stream fusion network[J]. Information Fusion, 2020,55,1-15.
[46]
Fu S, Meng W, Jeon G, et al. Two-path network with feedback connections for pan-sharpening in remote sensing[J]. Remote Sensing, 2020, 12(10),1674.
[47]
He L, Xi D, Li J, et al. A spectral-aware convolutional neural network for pansharpening[J]. Applied Sciences, 2020, 10(17),5809.
[48]
Zhang Y, Liu C, Sun M, et al. Pan-sharpening using an efficient bidirectional pyramid network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(8),5549-5563.
doi: 10.1109/TGRS.36
[49]
Fang S, Wang X, Zhang J, et al. Pan-sharpening based on parallel pyramid convolutional neural network[C]// IEEE International Conference on Image Processing (ICIP), 2020:453-457.
Fang S, Fang S H, Yao H L. Pan-sharpening based on a deep pyramid network[J]. Journal of Computer-Aided Design and Computer Graphics, 2019, 31(10):1831-1837.
[51]
Ronneberger O, Fischer P, Brox T. U-Net:Convolutional networks for biomedical image segmentation[C]// International Conference on Medical Image Computing and Computer-Assisted Intervention.Springer,Cham, 2015:234-241.
[52]
Yao W, Zeng Z, Lian C, et al. Pixel-wise regression using U-Net and its application on pansharpening[J]. Neurocomputing, 2018, 312:364-371.
doi: 10.1016/j.neucom.2018.05.103
[53]
Wang W, Zhou Z, Liu H, et al. MSDRN:Pansharpening of multispectral images via multi-scale deep residual network[J]. Remote Sensing, 2021, 13(6):1200.
doi: 10.3390/rs13061200
[54]
Lai Z, Chen L, Jeon G, et al. Real-time and effective pan-sharpening for remote sensing using multi-scale fusion network[J]. Journal of Real-Time Image Processing, 2021:1-17.
[55]
Benzenati T, Kallel A, Kessentini Y. Two stages pan-sharpening details injection approach based on very deep residual networks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(6),4984-4992.
doi: 10.1109/TGRS.2020.3019835
[56]
He K, Zhang X, Ren S, et al. Deep residual learning for image reco-gnition[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016:770-778.
[57]
Hu J, Du C, Fan S. Two-stage pansharpening based on multi-level detail injection network[J]. IEEE Access, 2020, 8:156442-156455.
doi: 10.1109/Access.6287639
[58]
Zhang L, Zhang J, Ma J, et al. SC-PNN:Saliency cascade convolutional neural network for pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021:1-19.
[59]
Li W, Liang X, Dong M. MDECNN:A multiscale perception dense encoding convolutional neural network for multispectral pan-sharpening[J]. Remote Sensing, 2021, 13(3):535.
doi: 10.3390/rs13030535
[60]
Yuan Q, Wei Y, Meng X, et al. A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(3):978-989.
doi: 10.1109/JSTARS.4609443
[61]
Hu J, Hu P, Kang X, et al. Pan-sharpening via multiscale dynamic convolutional neural network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(3):2231-2244.
doi: 10.1109/TGRS.36
[62]
Guo Y, Ye F, Gong H. Learning an efficient convolution neural network for pansharpening[J]. Algorithms, 2019, 12(1):16.
doi: 10.3390/a12010016
[63]
Hu J, Hu P, Wang Z P, et al. Spatial dynamic selection network for remote-sensing image fusion[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19:8013205.
[64]
Hu J, He Z, Wu J. Deep self-learning network for adaptive pansharpening[J]. Remote Sensing, 2019, 11(20):2395.
doi: 10.3390/rs11202395
[65]
Liu J, Feng Y, Zhou C, et al. PWnet:An adaptive weigh network for the fusion of panchromatic and multispectral images[J]. Remote Sensing, 2020, 12(17):2804.
doi: 10.3390/rs12172804
[66]
Deng L J, Vivone G, Jin C, et al. Detail injection-based deep convolutional neural networks for pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020:1-16.
[67]
Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11):139-144.
doi: 10.1145/3422622
[68]
Ma J, Yu W, Chen C, et al. Pan-GAN:An unsupervised pan-sharpening method for remote sensing image fusion[J]. Information Fusion, 2020, 62:110-120.
doi: 10.1016/j.inffus.2020.04.006
[69]
Zhou C, Zhang J, Liu J, et al. PercepPan:Towards unsupervised pan-sharpening based on perceptual loss[J]. Remote Sensing, 2020, 12(14):2318.
doi: 10.3390/rs12142318
[70]
Xiong Z, Guo Q, Liu M, et al. Pan-sharpening based on convolutional neural network by using the loss function with no-reference[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 14:897-906.
doi: 10.1109/JSTARS.4609443
[71]
Luo S, Zhou S, Feng Y, et al. Pansharpening via unsupervised convolutional neural networks[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13:4295-4310.
doi: 10.1109/JSTARS.4609443
Du C G, Hu J W, Hu P. Semi-supervised convolutional neural network remote sensing image fusion[J]. Journal of Electronic Measurement and Instrumentation, 2021, 35(6):63-70.
[73]
Vitale S, Scarpa G. A detail-preserving cross-scale learning strategy for CNN-based pansharpening[J]. Remote Sensing, 2020, 12(3):348.
doi: 10.3390/rs12030348
Huang S S, Jiang Q, Jin X, et al. Semi-supervised remote sensing image fusion method combining siamese structure with generative adversarial networks[J]. Journal of Computer-Aided Design and Computer Graphics, 2021, 33(1):92-105.
doi: 10.3724/SP.J.1089.2021.18227
[75]
Arjovsky M, Chintala S, Bottou L. Wasserstein generative adversarial networks[C]// International Conference on Machine Learning.PMLR, 2017:214-223.
[76]
Mirza M, Osindero S. Conditional generative adversarial nets[J]. Computer Science, 2014:2672-2680.
[77]
Liu X, Deng C, Zhao B, et al. Feature-level loss for multispectral pan-sharpening with machine learning[C]// IEEE International Geoscience and Remote Sensing Symposium(IGARSS).IEEE, 2018:8062-8065.
[78]
Xu H, Ma J, Shao Z, et al. SDPNet:A deep network for pan-sharpening with enhanced information representation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(5):4120-4134.
doi: 10.1109/TGRS.2020.3022482
[79]
Choi J S, Kim Y, Kim M. S3:A spectral-spatial structure loss for pan-sharpening networks[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 17(5):829-833.
doi: 10.1109/LGRS.8859
[80]
Eghbalian S, Ghassemian H. Multi spectral image fusion by deep convolutional neural network and new spectral loss function[J]. International Journal of Remote Sensing, 2018, 39(12):3983-4002.
doi: 10.1080/01431161.2018.1452074
[81]
Johnson J, Alahi A, Li F F. Perceptual losses for real-time style transfer and super-resolution[C]// European Conference on Computer Vision.Springer,Cham, 2016:694-711.
[82]
Bello J L G, Seo S, Kim M. Pan-sharpening with color-aware perceptual loss and guided re-colorization[C]// IEEE International Conference on Image Processing (ICIP).IEEE, 2020:908-912.
[83]
Vivone G, Alparone L, Chanussot J, et al. A critical comparison among pansharpening algorithms[J]. IEEE Transactions on Geoscience and Remote Sensing, 2014, 53(5):2565-2586.
doi: 10.1109/TGRS.2014.2361734
[84]
Alparone L, Aiazzi B, Baronti S, et al. Multispectral and panchromatic data fusion assessment without reference[J]. Photogrammetric Engineering and Remote Sensing, 2008, 74(2):193-200.
doi: 10.14358/PERS.74.2.193
2023, Vol. 35 
), 汪泽平, 胡佩
