A two-stage remote sensing image inpainting network combined with spatial semantic attention
LIU Yujia1(), XIE Shizhe2, DU Yang3, YAN Jin4,5(), NAN Yanyun4, WEN Zhongkai3,6
1. School of Geomatics, Liaoning Technical University, Fuxin 123000, China 2. School of Information Engineering, China University of Geosciemces(Beijing), Beijing 100083, China 3. College of Astronautics, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China 4. National Earthquake Response Support Service, Beijing 100049, China 5. College of Resource Environment and Toursim, Capital Normal University, Beijing 100048, China 6. Institute of Remote Sensing Satellite, CAST,Beijing 100094, China
In high-resolution remote sensing images, missing areas feature intricate surface features and pronounced spatial heterogeneity, causing the image inpainting results to suffer texture blurring and structural distortion, particularly for boundaries and areas with complex textures. This study proposed a two-stage remote sensing image inpainting network combined with spatial semantic attention (SSA). The network comprised two networks in series: one for coarse image inpainting and one for fine-scale image inpainting (also referred to as the coarse and fine-scale networks, respectively). This network was designed to guide the fine-scale network to restore the missing areas using the priori information provided by the coarse network. In the coarse network, a multi-level loss structure was constructed to enhance the stability of network training. In the fine-scale network, a novel SSA mechanism was proposed, with SSA being embedded differentially in the encoder and decoder based on the distribution of network features. This ensured the continuity of local features and the correlation of global semantic information. The experimental results show that the network proposed in this study can further improve the image inpainting effects compared to other existing algorithms.
Chen R X, Li X H. Restoring lost information on remote sensing images based on accessorial GIS data[J]. Geomatics and Information Science of Wuhan University, 2008, 33(5):461-464.
Zhang Q. Research on missing information reconstruction of remote sensing imagery employing temporal-spatial-spectral deep feature learning[D]. Wuhan: Wuhan University, 2019.
[5]
Dong F, Chen Z, Wang J. A new level set method for inhomogeneous image segmentation[J]. Image and Vision Computing, 2013, 31(10):809-822.
doi: 10.1016/j.imavis.2013.08.003
[6]
Zhang Y, Fan Y, Xu M. A background-purification-based framework for anomaly target detection in hyperspectral imagery[J]. IEEE Geoscience and Remote Sensing Letters, 2020, 17(7):1238-1242.
doi: 10.1109/LGRS.8859
[7]
Wang L, Wang Y, Zhao Y, et al. Classification of remotely sensed images using an ensemble of improved convolutional network[J]. IEEE Geoscience and Remote Sensing Letters, 2021, 18(5):930-934.
doi: 10.1109/LGRS.2020.2988934
Wang H Y, Li H Y, Gao X J. Research on image restoration method based on structure embedding[J]. Computer Engineering and Applications, 2021, 57(22):241-246.
doi: 10.3778/j.issn.1002-8331.2007-0279
Zhou X C, Xu Y. Adaptive image inpainting based on structural correlation[J]. Computer Science, 2020, 47(4):131-135.
[12]
Shen J, Chan T F. Mathematical models for local nontexture inpaintings[J]. SIAM Journal on Applied Mathematics, 2002, 62(3):1019-1043.
doi: 10.1137/S0036139900368844
Zhang G M, Li Y B. Image inpainting of fractional TV model combined with texture structure[J]. Journal of Image and Graphics, 2019, 24(5):700-713.
[14]
Criminisi A, Pérez P, Toyama K. Region filling and object removal by exemplar-based image inpainting[J]. IEEE Transactions on Image Processing:a Publication of the IEEE Signal Processing Society, 2004, 13(9):1200-1212.
doi: 10.1109/TIP.2004.833105
[15]
Barnes C, Shechtman E, Finkelstein A, et al. PatchMatch:A randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics, 2009, 28(3):24-34.
[16]
Pathak D, Krhenbühl P, Donahue J, et al. Context encoders:Feature learning by inpainting[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Las Vegas,NV,USA.IEEE, 2016:2536-2544.
Xu H M, Cheng H, Jiang H D, et al. Research on image restoration methods based on deep leaming[J]. Journal of Natural Science of Heilongjiang University, 2022, 39(1):114-120.
[18]
Zhang H, Goodfellow I, Metaxas D, et al. Self-attention generative adversarial networks[EB/OL]. 2018:arXiv:1805.08318. https://arxiv.org/abs/1805.08318.pdf.
[19]
Wang N, Li J, Zhang L, et al. MUSICAL:Multi-scale image contextual attention learning for inpainting[C]// Proceedings of the 28th International Joint Conference on Artificial Intelligence.2019,Macao,China.ACM, 2019:3748-3754.
[20]
Zeng Y, Fu J, Chao H, et al. Aggregated contextual transformations for high-resolution image inpainting[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(7):3266-3280.
doi: 10.1109/TVCG.2022.3156949
Liu Q, Zhang D C. Dense convolution generative adversarial network image inpainting method with SENet[J]. Journal of Chinese Computer Systems, 2022, 43(5):1056-1060.
[22]
唐骞. 基于对抗学习的图像修复[J]. 计算机产品与流通, 2019(1):210.
Tang Q. Image restoration based on antagonistic learning[J]. Computer Products and Circulation, 2019(1):210.
Sun J G, Yang Z W, Huang S. Image inpainting model with consistent global and local attributes[J]. Journal of Image and Graphics, 2020, 25(12):2505-2516.
Lin Z, Wang M. Image inpainting based on new encoder and similarity constraint[J]. Computer Systems & Applications, 2021, 30(1):122-128.
[25]
Yu J, Lin Z, Yang J, et al. Generative image inpainting with contextual attention[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Salt Lake City,UT,USA.IEEE, 2018:5505-5514.
[26]
Nazeri K, Ng E, Joseph T, et al. EdgeConnect:Generative image inpainting with adversarial edge learning[EB/OL]. 2019:arXiv:1901.00212. https://arxiv.org/abs/1901.00212.pdf.
[27]
Liu H, Jiang B, Xiao Y, et al. Coherent semantic attention for image inpainting[C]// 2019 IEEE/CVF International Conference on Computer Vision (ICCV).IEEE, 2019:4169-4178.
[28]
Wu H, Zhou J, Li Y. Deep generative model for image inpainting with local binary pattern learning and spatial attention[C]// IEEE Transactions on Multimedia.IEEE, 2021:4016-4027.
[29]
Ronneberger O, Fischer P, Brox T. U-net:Convolutional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham:Springer, 2015:234-241.
[30]
Isola P, Zhu J Y, Zhou T, et al. Image-to-image translation with conditional adversarial networks[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).IEEE, 2017:5967-5976.
[31]
Maggiori E, Tarabalka Y, Charpiat G, et al. Can semantic labeling methods generalize to any city? the inria aerial image labeling benchmark[C]// 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).IEEE, 2017:3226-3229.
[32]
Zhou B, Lapedriza A, Khosla A, et al. Places:A 10 million image database for scene recognition[C]// IEEE Transactions on Pattern Analysis and Machine Intelligence.IEEE, 2018:1452-1464.
2024, Vol. 36 
), 谢诗哲2, 杜阳3, 严瑾4,5(
