Please wait a minute...
 
Remote Sensing for Land & Resources    2020, Vol. 32 Issue (4) : 74-83     DOI: 10.6046/gtzyyg.2020.04.11
|
Multi-task learning for building object semantic segmentation of remote sensing image based on Unet network
LIU Shangwang1,2(), CUI Zhiyong1,2, LI Daoyi1,2
1. College of Computer and Information Engineering, Henan Normal University, Xinxiang 453007, China
2. “IntelligentBusiness and Internet of Things Technology” Henan Engineering Laboratory, Xinxiang 453007, China
Download: PDF(6948 KB)   HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks    
Abstract  

In order to accurately segment the building object of high-resolution remote sensing image, this paper proposes a multi-task learning method based on Unet network. Firstly, boundary distance map is generated from the ground-truth map of the building object remote sensing image; the boundary distance map, original remote sensing image and ground-truth map together are regarded as the input of Unet network. Then, based on the ResNet network, a multi-task network is built by adding the building object prediction layer and the boundary distance prediction layer at the end of the Unet network. Finally, the loss function of the multi-task network is defined, and the network is trained by using Adam optimization algorithm. Experiments on the Inria aerial remote sensing image building object dataset show that, compared with the full convolutional network combined with the multi-layer perceptron method, the intersection-over-unions of VGG16, VGG16+boundary prediction, ResNet50 and this method have been increased by 5.15, 6.94, 6.41, and 7.86 percentage points, and the accuracy has been increased to 94.71%, 95.39%, 95.30%, and 96.10% respectively,which ensures that the building object of high-resolution remote sensing image can be segmented effectively.
Keywords Unet network      multi-task learning      remote sensing image      semantic segmentation      ResNet network     
:  TP751.1  
Issue Date: 23 December 2020
Service
E-mail this article
E-mail Alert
RSS
Articles by authors
Shangwang LIU
Zhiyong CUI
Daoyi LI
Cite this article:   
Shangwang LIU,Zhiyong CUI,Daoyi LI. Multi-task learning for building object semantic segmentation of remote sensing image based on Unet network[J]. Remote Sensing for Land & Resources, 2020, 32(4): 74-83.
URL:  
https://www.gtzyyg.com/EN/10.6046/gtzyyg.2020.04.11     OR     https://www.gtzyyg.com/EN/Y2020/V32/I4/74
Tab.1  Visualization of training data
Fig.1  Framework of the multi-tasking network
城市 FCN+MLP VGG16 VGG16+边界预测 ResNet50 本文方法
IoU Acc IoU Acc IoU Acc IoU Acc IoU Acc
Austin 61.20 94.20 70.66 95.28 72.81 95.82 72.38 95.79 74.41 96.09
Chicago 61.30 90.43 66.37 91.44 67.38 91.92 66.12 91.50 67.76 92.02
Kitsap Co. 51.50 98.92 57.55 98.19 57.54 98.90 58.68 98.95 60.19 98.63
West Tyrol 57.95 96.66 67.82 95.35 67.18 97.01 67.32 97.07 69.09 97.74
Vienna 72.13 91.87 77.01 93.28 77.19 93.31 76.86 93.21 78.21 93.63
均值 64.67 94.42 69.82 94.71 71.61 95.39 71.08 95.30 72.53 96.10
Tab.2  Experimental results of different methods(%)
Tab.3  Building object segmentation results of remote sensing image by using different methods
Tab.4  Visualization of the boundary distance prediction layer output
Fig.2  Line chart of training period and loss value
Fig.3  Building object segmentation results of a factual remote sensing image by using different methods
[1] Zhang B, Wang C, Shen Y, et al. Fully connected conditional random fields for high-resolution remote sensing land use/land cover classification with convolutional neural networks[J]. Remote Sensing, 2018,10(12):1889-1903.
[2] Li W, Dong R, Fu H. Large-scale oil palm tree detection from high-resolution satellite images using two-stage convolutional neural networks[J]. Remote Sensing, 2019,11(1):11-31.
[3] 张永宏, 夏广浩, 阚希, 等. 基于全卷积神经网络的多源高分辨率遥感道路提取[J]. 计算机应用, 2018,28(7):2070-2075.
[3] Zhang Y H, Xia G H, Kan X, et al. Road extraction from multi-source high resolution remote sensing image based on fully convolutional neural network[J]. Journal of Computer Applications, 2018,28(7):2070-2075.
[4] Demir I, Koperski K, Lindenvaum D, et al. DeepGlobe 2018:A challenge to parse the earth through satellite images[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).Salt Lake City:IEEE, 2018:17201-17209.
[5] Li L, Liang J, Weng M, et al. A multiple-feature reuse network to extract buildings from remote sensing imagery[J]. Remote Sensing, 2018,10(9):1350-1368.
doi: 10.3390/rs10091350 url: http://www.mdpi.com/2072-4292/10/9/1350
[6] 施文灶, 刘金清. 基于邻域总变分和势直方图函数的高分辨率遥感影像建筑物提取[J]. 计算机应用, 2017,37(6):1787-1792.
[6] Shi W Z, Liu J Q. Building extraction from high-resolution remotely sensed imagery based on neighborhood total variation and potential histogram function[J]. Journal of Computer Applications, 2017,37(6):1787-1792.
[7] Sun X, Lin X, Shen S, et al. High-resolution remote sensing data classification over urban areas using random forest ensemble and fully connected conditional random field[J]. ISPRS International Journal of Geo-Information, 2017,6(8):245-271.
[8] Jabri S, Zhang Y, Suliman A. Stereo-based building detection in very high resolution satellite imagery using IHS color system[C]// 2014 IEEE Geoscience and Remote Sensing Symposium.Quebec City:IEEE, 2014:2301-2304.
[9] Garcia-Garcia A, Orts-Escolano S, Oprea S, et al. A review on deep learning techniques applied to semantic segmentation[EB/OL]. (2017-04-22) [2019-02-05]. http://arxiv.org/abs/1704.06857.
url: http://arxiv.org/abs/1704.06857
[10] Yuan J. Automatic building extraction in aerial scenes using convolutional networks.[EB/OL]. (2016-02-21) [2019-02-20]. http://arxiv.org/abs/1602.06564.
url: http://arxiv.org/abs/1602.06564
[11] Zhang Q, Wang Y, Liu Q, et al. CNN based suburban building detection using monocular high resolution Google Earth images[C]// 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).Beijing:IEEE, 2016:661-664.
[12] Zhou B, Zhao H, Puig X, et al. Scene parsing through ade20k dataset[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Hawaii, 2017:633-641.
[13] Long J, Shelhaner E, Darrell T. Fully convolutional networks for semantic segmentation[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Boston, 2015:3431-3440.
[14] Badrinarayanan V, Kendall A, Cipolla R. SegNet:A deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017,39(12):2481-2495.
doi: 10.1109/TPAMI.2016.2644615 pmid: 28060704 url: https://www.ncbi.nlm.nih.gov/pubmed/28060704
[15] Yu F, Koltun V. Multi-scale context aggregation by dilated convolutions[EB/OL]. (2016-04-30)[2019-03-25]. http://arxiv.org/abs/1511.07122.
url: http://arxiv.org/abs/1511.07122
[16] Chen L C, Papandreou G, Schroff F, et al. Rethinking atrous convolution for semantic image segmentation[EB/OL]. (2017-12-05)[2019-03-26]. http://arxiv.org/abs/1706.05587.
url: http://arxiv.org/abs/1706.05587
[17] Maggiori E, Tarabalka Y, Charpiat G, et al. Can semantic labeling methods generalize to any city? The inria aerial image labeling benchmark[C]// 2017 IEEE International Geoscience and Remote Sensing Symposium(IGARSS).Fort Worth:IEEE, 2017:3226-3229.
[18] Maggiori E, Tarabalka Y, Charpiat G, et al. High-resolution semantic labeling with convolutional neural networks[EB/OL]. (2016-11-07)[2019-03-26]. http://arxiv.org/abs/1611.01962.
url: http://arxiv.org/abs/1611.01962
[19] Marmanis D, Schindler K, Wegner J D, et al. Classification with an edge:Improving semantic image segmentation with boundary detection[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018,135:158-172.
[20] Peng C, Zhang X, Yu G, et al. Large kernel matters:Improve semantic segmentation by global convolutional network[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Hawaii, 2017:4353-4361.
[21] Huang Z, Cheng G, Wang H, et al. Building extraction from multi-source remote sensing images via deep deconvolution neural networks[C]// 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).Beijing:IEEE, 2016:1835-1838.
[22] Bischke B, Helber P, Folz J, et al. Multi-task learning for segmentation of building footprints with deep neural networks[EB/OL]. (2017-09-18) [2019-03-26]. http://arxiv.org/abs/1709.05932.
url: http://arxiv.org/abs/1709.05932
[23] Ronneberger O, Fischer P, Brox T. U-net:Convolutional networks for biomedical image segmentation[C]// International Conference on Medical Image Computing and Computer-Assisted Intervention.Springer,Cham, 2015:234-241.
[24] Iglovikov V, Shvets A. Ternausnet:U-net with VGG11 encoder pre-trained on ImageNet for image segmentation[EB/OL]. (2018-03-29) [2019-03-29]. http://arxiv.org/abs/1801.05746.
url: http://arxiv.org/abs/1801.05746
[25] Xu Y, Wu L, Xie Z, et al. Building extraction in very high resolution remote sensing imagery using deep learning and guided filters[J]. Remote Sensing, 2018,10(1):144-162.
[26] He K, Zhang X, Ren S, et al. Deep residual learning for image reco-gnition[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.San Francisco, 2016:770-778.
[27] Kendall A, Gal Y, Cipolla R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Reco-gnition.Salt Lake City, 2018:7482-7491.
[28] Russakovsky O, Deng J, Su H, et al. Imagenet large scale visual recognition challenge[J]. International Journal of Computer Vision, 2015,115(3):211-252.
[29] Hayder Z, He X, Salzmann M. Boundary-aware instance segmentation[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Hawaii, 2017:5696-5704.
[30] Kingma D P, Ba J. Adam:A method for stochastic optimization[EB/OL]. (2017-01-30) [2019-04-11]. http://arxiv.org/abs/1412.6980.
url: http://arxiv.org/abs/1412.6980
[1] ZHANG Daming, ZHANG Xueyong, LI Lu, LIU Huayong. Remote sensing image segmentation based on Parzen window density estimation of super-pixels[J]. Remote Sensing for Natural Resources, 2022, 34(1): 53-60.
[2] XUE Bai, WANG Yizhe, LIU Shuhan, YUE Mingyu, WANG Yiying, ZHAO Shihu. Change detection of high-resolution remote sensing images based on Siamese network[J]. Remote Sensing for Natural Resources, 2022, 34(1): 61-66.
[3] SONG Renbo, ZHU Yuxin, GUO Renjie, ZHAO Pengfei, ZHAO Kexin, ZHU Jie, CHEN Ying. A method for 3D modeling of urban buildings based on multi-source data integration[J]. Remote Sensing for Natural Resources, 2022, 34(1): 93-105.
[4] LIU Zhizhong, SONG Yingxu, YE Runqing. An analysis of rainstorm-induced landslides in northeast Chongqing on August 31, 2014 based on interpretation of remote sensing images[J]. Remote Sensing for Natural Resources, 2021, 33(4): 192-199.
[5] ZHANG Chengye, XING Jianghe, LI Jun, SANG Xiao. Recognition of the spatial scopes of tailing ponds based on U-Net and GF-6 images[J]. Remote Sensing for Natural Resources, 2021, 33(4): 252-257.
[6] LI Yikun, YANG Yang, YANG Shuwen, WANG Zihao. A change vector analysis in posterior probability space combined with fuzzy C-means clustering and a Bayesian network[J]. Remote Sensing for Natural Resources, 2021, 33(4): 82-88.
[7] SANG Xiao, ZHANG Chengye, LI Jun, ZHU Shoujie, XING Jianghe, WANG Jinyang, WANG Xingjuan, LI Jiayao, YANG Ying. Application of intensity analysis theory in the land use change in Yijin Holo Banner under the background of coal mining[J]. Remote Sensing for Natural Resources, 2021, 33(3): 148-155.
[8] WANG Yiuzhu, HUANG Liang, CHEN Pengdi, LI Wenguo, YU Xiaona. Change detection of remote sensing images based on the fusion of co-saliency difference images[J]. Remote Sensing for Natural Resources, 2021, 33(3): 89-96.
[9] LIU Wanjun, GAO Jiankang, QU Haicheng, JIANG Wentao. Ship detection based on multi-scale feature enhancement of remote sensing images[J]. Remote Sensing for Natural Resources, 2021, 33(3): 97-106.
[10] GUO Wen, ZHANG Qiao. Building extraction using high-resolution satellite imagery based on an attention enhanced full convolution neural network[J]. Remote Sensing for Land & Resources, 2021, 33(2): 100-107.
[11] LU Qi, QIN Jun, YAO Xuedong, WU Yanlan, ZHU Haochen. Buildings extraction of GF-2 remote sensing image based on multi-layer perception network[J]. Remote Sensing for Land & Resources, 2021, 33(2): 75-84.
[12] QIU Yifan, CHAI Dengfeng. A deep learning method for Landsat image cloud detection without manually labeled data[J]. Remote Sensing for Land & Resources, 2021, 33(1): 102-107.
[13] HU Suliyang, LI Hui, GU Yansheng, HUANG Xianyu, ZHANG Zhiqi, WANG Yingchun. An analysis of land use changes and driving forces of Dajiuhu wetland in Shennongjia based on high resolution remote sensing images: Constraints from the multi-source and long-term remote sensing information[J]. Remote Sensing for Land & Resources, 2021, 33(1): 221-230.
[14] LIU Zhao, ZHAO Tong, LIAO Feifan, LI Shuai, LI Haiyang. Research and comparative analysis on urban built-up area extraction methods from high-resolution remote sensing image based on semantic segmentation network[J]. Remote Sensing for Land & Resources, 2021, 33(1): 45-53.
[15] CAI Xiang, LI Qi, LUO Yan, QI Jiandong. Surface features extraction of mining area image based on object-oriented and deep-learning method[J]. Remote Sensing for Land & Resources, 2021, 33(1): 63-71.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
京ICP备05055290号-2
Copyright © 2017 Remote Sensing for Natural Resources
Support by Beijing Magtech