Multi-task learning for building object semantic segmentation of remote sensing image based on Unet network
LIU Shangwang1,2(), CUI Zhiyong1,2, LI Daoyi1,2
1. College of Computer and Information Engineering, Henan Normal University, Xinxiang 453007, China 2. “IntelligentBusiness and Internet of Things Technology” Henan Engineering Laboratory, Xinxiang 453007, China
In order to accurately segment the building object of high-resolution remote sensing image, this paper proposes a multi-task learning method based on Unet network. Firstly, boundary distance map is generated from the ground-truth map of the building object remote sensing image; the boundary distance map, original remote sensing image and ground-truth map together are regarded as the input of Unet network. Then, based on the ResNet network, a multi-task network is built by adding the building object prediction layer and the boundary distance prediction layer at the end of the Unet network. Finally, the loss function of the multi-task network is defined, and the network is trained by using Adam optimization algorithm. Experiments on the Inria aerial remote sensing image building object dataset show that, compared with the full convolutional network combined with the multi-layer perceptron method, the intersection-over-unions of VGG16, VGG16+boundary prediction, ResNet50 and this method have been increased by 5.15, 6.94, 6.41, and 7.86 percentage points, and the accuracy has been increased to 94.71%, 95.39%, 95.30%, and 96.10% respectively,which ensures that the building object of high-resolution remote sensing image can be segmented effectively.
刘尚旺, 崔智勇, 李道义. 基于Unet网络多任务学习的遥感图像建筑地物语义分割[J]. 国土资源遥感, 2020, 32(4): 74-83.
LIU Shangwang, CUI Zhiyong, LI Daoyi. Multi-task learning for building object semantic segmentation of remote sensing image based on Unet network. Remote Sensing for Land & Resources, 2020, 32(4): 74-83.
Zhang B, Wang C, Shen Y, et al. Fully connected conditional random fields for high-resolution remote sensing land use/land cover classification with convolutional neural networks[J]. Remote Sensing, 2018,10(12):1889-1903.
[2]
Li W, Dong R, Fu H. Large-scale oil palm tree detection from high-resolution satellite images using two-stage convolutional neural networks[J]. Remote Sensing, 2019,11(1):11-31.
Zhang Y H, Xia G H, Kan X, et al. Road extraction from multi-source high resolution remote sensing image based on fully convolutional neural network[J]. Journal of Computer Applications, 2018,28(7):2070-2075.
[4]
Demir I, Koperski K, Lindenvaum D, et al. DeepGlobe 2018:A challenge to parse the earth through satellite images[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).Salt Lake City:IEEE, 2018:17201-17209.
[5]
Li L, Liang J, Weng M, et al. A multiple-feature reuse network to extract buildings from remote sensing imagery[J]. Remote Sensing, 2018,10(9):1350-1368.
doi: 10.3390/rs10091350
Shi W Z, Liu J Q. Building extraction from high-resolution remotely sensed imagery based on neighborhood total variation and potential histogram function[J]. Journal of Computer Applications, 2017,37(6):1787-1792.
[7]
Sun X, Lin X, Shen S, et al. High-resolution remote sensing data classification over urban areas using random forest ensemble and fully connected conditional random field[J]. ISPRS International Journal of Geo-Information, 2017,6(8):245-271.
[8]
Jabri S, Zhang Y, Suliman A. Stereo-based building detection in very high resolution satellite imagery using IHS color system[C]// 2014 IEEE Geoscience and Remote Sensing Symposium.Quebec City:IEEE, 2014:2301-2304.
[9]
Garcia-Garcia A, Orts-Escolano S, Oprea S, et al. A review on deep learning techniques applied to semantic segmentation[EB/OL]. (2017-04-22) [2019-02-05]. http://arxiv.org/abs/1704.06857.
[10]
Yuan J. Automatic building extraction in aerial scenes using convolutional networks.[EB/OL]. (2016-02-21) [2019-02-20]. http://arxiv.org/abs/1602.06564.
[11]
Zhang Q, Wang Y, Liu Q, et al. CNN based suburban building detection using monocular high resolution Google Earth images[C]// 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).Beijing:IEEE, 2016:661-664.
[12]
Zhou B, Zhao H, Puig X, et al. Scene parsing through ade20k dataset[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Hawaii, 2017:633-641.
[13]
Long J, Shelhaner E, Darrell T. Fully convolutional networks for semantic segmentation[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Boston, 2015:3431-3440.
[14]
Badrinarayanan V, Kendall A, Cipolla R. SegNet:A deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017,39(12):2481-2495.
doi: 10.1109/TPAMI.2016.2644615
pmid: 28060704
[15]
Yu F, Koltun V. Multi-scale context aggregation by dilated convolutions[EB/OL]. (2016-04-30)[2019-03-25]. http://arxiv.org/abs/1511.07122.
[16]
Chen L C, Papandreou G, Schroff F, et al. Rethinking atrous convolution for semantic image segmentation[EB/OL]. (2017-12-05)[2019-03-26]. http://arxiv.org/abs/1706.05587.
[17]
Maggiori E, Tarabalka Y, Charpiat G, et al. Can semantic labeling methods generalize to any city? The inria aerial image labeling benchmark[C]// 2017 IEEE International Geoscience and Remote Sensing Symposium(IGARSS).Fort Worth:IEEE, 2017:3226-3229.
[18]
Maggiori E, Tarabalka Y, Charpiat G, et al. High-resolution semantic labeling with convolutional neural networks[EB/OL]. (2016-11-07)[2019-03-26]. http://arxiv.org/abs/1611.01962.
[19]
Marmanis D, Schindler K, Wegner J D, et al. Classification with an edge:Improving semantic image segmentation with boundary detection[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018,135:158-172.
[20]
Peng C, Zhang X, Yu G, et al. Large kernel matters:Improve semantic segmentation by global convolutional network[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Hawaii, 2017:4353-4361.
[21]
Huang Z, Cheng G, Wang H, et al. Building extraction from multi-source remote sensing images via deep deconvolution neural networks[C]// 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).Beijing:IEEE, 2016:1835-1838.
[22]
Bischke B, Helber P, Folz J, et al. Multi-task learning for segmentation of building footprints with deep neural networks[EB/OL]. (2017-09-18) [2019-03-26]. http://arxiv.org/abs/1709.05932.
[23]
Ronneberger O, Fischer P, Brox T. U-net:Convolutional networks for biomedical image segmentation[C]// International Conference on Medical Image Computing and Computer-Assisted Intervention.Springer,Cham, 2015:234-241.
[24]
Iglovikov V, Shvets A. Ternausnet:U-net with VGG11 encoder pre-trained on ImageNet for image segmentation[EB/OL]. (2018-03-29) [2019-03-29]. http://arxiv.org/abs/1801.05746.
[25]
Xu Y, Wu L, Xie Z, et al. Building extraction in very high resolution remote sensing imagery using deep learning and guided filters[J]. Remote Sensing, 2018,10(1):144-162.
[26]
He K, Zhang X, Ren S, et al. Deep residual learning for image reco-gnition[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.San Francisco, 2016:770-778.
[27]
Kendall A, Gal Y, Cipolla R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Reco-gnition.Salt Lake City, 2018:7482-7491.
[28]
Russakovsky O, Deng J, Su H, et al. Imagenet large scale visual recognition challenge[J]. International Journal of Computer Vision, 2015,115(3):211-252.
[29]
Hayder Z, He X, Salzmann M. Boundary-aware instance segmentation[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Hawaii, 2017:5696-5704.
[30]
Kingma D P, Ba J. Adam:A method for stochastic optimization[EB/OL]. (2017-01-30) [2019-04-11]. http://arxiv.org/abs/1412.6980.
2020, Vol. 32 
), 崔智勇1,2, 李道义1,2
