针对深层卷积较难兼顾全局特征与局部特征从而导致提取建筑物边缘不准确和微小建筑物丢失的问题,以注意力机制和跳跃连接为基础提出SER-UNet算法。SER-UNet算法在编码器阶段耦合SE-ResNet和最大池化层,在解码器阶段关联SE-ResNet与反卷积层,通过跳跃连接将编码器提取的浅层特征和解码器提取的深层特征进行融合后输出特征图。验证SER-UNet算法的有效性,在MAP-Net网络并行多路径特征提取阶段使用SER-UNet算法替换原网络中的特征提取结构,分别在WHU数据集和Inria数据集上进行评估,IoU与精度分别达91.46%、82.61%和95.67%、92.75%,对比UNet、PSPNet、ResNet101、MAP-Net网络,IoU分别提高0.49%、0.14%、1.89%、1.57%,精度分别提高0.14%、1.06%、2.42%、1.09%。分析SER-UNet算法的泛化能力,将级联SER-UNet的MAP-Net网络在AerialImage数据集上进行提取验证,IoU与精度分别达85.32%和94.13%。结果表明,结合SER-UNet算法的MAP-Net并行多路径网络表现出较好的提取精度与泛化能力。此外,SER-UNet算法可以有效地嵌入PSPNet、ResNet101、HRNetv2等网络中,提升网络特征表示能力。

Aiming at the problems of inaccurate edges and loss of small buildings in the extracted buildings due to the inability of deep convolution to take into account global features and local features, the SER-UNet algorithm is proposed based on attention mechanism and skip connection. SER-UNet algorithm couples SE_ResNet and max pooling layers in the encoder stage, and the SE_ResNet structure and deconvolution are used in the decoder stage. The feature map is output after fusing the shallow features extracted by the encoder and the deep features extracted by the decoder through skip connections. In order to analyze the effectiveness of the method, the SER-UNet is used to replace the feature extraction structure in the original network in the parallel multi-path feature extraction stage of the MAP-Net network. Finally, the method proposed is experimentally evaluated on the WHU dataset and the Inria dataset, and the IoU and precision reach 91.46%, 82.61% and 95.67%, 92.75%, compared with UNet, PSPNet, ResNet101, and MAP-Net Networks, the IoU is increased by 0.49%, 0.14%, 1.89%, and 1.57%, and the precision is increased by 0.14%, 1.06%, 2.42% and 1.09%, respectively. To further analyze the validity of the SER-UNet algorithm, the edge integrity and small extraction verification IoU and precision reached 85.32% and 94.13% on the AerialImage dataset. The experiment results show that the MAP-Net parallel multipath network combined with SER-UNet algorithm shows good generalization ability. In addition, the SER-UNet algorithm can be effectively embedded in PSPNet, ResNet101, HRNetv2 and other Networks to improve the ability of Network feature representation.

Deep learning has become popular and the mainstream technology in many researches related to learning, and has shown its impact on photogrammetry. According to the definition of photogrammetry, that is, a subject that researches shapes, locations, sizes, characteristics and inter-relationships of real objects from optical images, photogrammetry considers two aspects, geometry and semantics. From the two aspects, we review the history of deep learning and discuss its current applications on photogrammetry, and forecast the future development of photogrammetry. In geometry, the deep convolutional neural network (CNN) has been widely applied in stereo matching, SLAM and 3D reconstruction, and has made some effects but needs more improvement. In semantics, conventional methods that have to design empirical and handcrafted features have failed to extract the semantic information accurately and failed to produce types of “semantic thematic map” as 4D productions (DEM, DOM, DLG, DRG) of photogrammetry. This causes the semantic part of photogrammetry be ignored for a long time. The powerful generalization capacity, ability to fit any functions and stability under types of situations of deep leaning is making the automatic production of thematic maps possible. We review the achievements that have been obtained in road network extraction, building detection and crop classification, etc., and forecast that producing high-accuracy semantic thematic maps directly from optical images will become reality and these maps will become a type of standard products of photogrammetry. At last, we introduce our two current researches related to geometry and semantics respectively. One is stereo matching of aerial images based on deep learning and transfer learning; the other is precise crop classification from satellite spatio-temporal images based on 3D CNN.

Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixels-to-pixels, improve on the previous best result in semantic segmentation. Our key insight is to build "fully convolutional" networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, explain their application to spatially dense prediction tasks, and draw connections to prior models. We adapt contemporary classification networks (AlexNet, the VGG net, and GoogLeNet) into fully convolutional networks and transfer their learned representations by fine-tuning to the segmentation task. We then define a skip architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations. Our fully convolutional networks achieve improved segmentation of PASCAL VOC (30% relative improvement to 67.2% mean IU on 2012), NYUDv2, SIFT Flow, and PASCAL-Context, while inference takes one tenth of a second for a typical image.

We present a novel and practical deep fully convolutional neural network architecture for semantic pixel-wise segmentation termed SegNet. This core trainable segmentation engine consists of an encoder network, a corresponding decoder network followed by a pixel-wise classification layer. The architecture of the encoder network is topologically identical to the 13 convolutional layers in the VGG16 network [1]. The role of the decoder network is to map the low resolution encoder feature maps to full input resolution feature maps for pixel-wise classification. The novelty of SegNet lies is in the manner in which the decoder upsamples its lower resolution input feature map(s). Specifically, the decoder uses pooling indices computed in the max-pooling step of the corresponding encoder to perform non-linear upsampling. This eliminates the need for learning to upsample. The upsampled maps are sparse and are then convolved with trainable filters to produce dense feature maps. We compare our proposed architecture with the widely adopted FCN [2] and also with the well known DeepLab-LargeFOV [3], DeconvNet [4] architectures. This comparison reveals the memory versus accuracy trade-off involved in achieving good segmentation performance. SegNet was primarily motivated by scene understanding applications. Hence, it is designed to be efficient both in terms of memory and computational time during inference. It is also significantly smaller in the number of trainable parameters than other competing architectures and can be trained end-to-end using stochastic gradient descent. We also performed a controlled benchmark of SegNet and other architectures on both road scenes and SUN RGB-D indoor scene segmentation tasks. These quantitative assessments show that SegNet provides good performance with competitive inference time and most efficient inference memory-wise as compared to other architectures. We also provide a Caffe implementation of SegNet and a web demo at http://mi.eng.cam.ac.uk/projects/segnet.

从遥感图像中自动化地检测和提取建筑物在城市规划、人口估计、地形图制作和更新等应用中具有极为重要的意义。本文提出和展示了建筑物提取的数个研究进展。由于遥感成像机理、建筑物自身、背景环境的复杂性，传统的经验设计特征的方法一直未能实现自动化，建筑物提取成为30余年尚未解决的挑战。先进的深度学习方法带来新的机遇，但目前存在两个困境：①尚缺少高精度的建筑物数据库，而数据是深度学习必不可少的“燃料”；②目前国际上的方法都采用像素级的语义分割，目标级、矢量级的提取工作亟待开展。针对于此，本文进行以下工作：①与目前同类数据集相比，建立了一套目前国际上范围最大、精度最高、涵盖多种样本形式（栅格、矢量）、多类数据源（航空、卫星）的建筑物数据库（WHU building dataset），并实现开源；②提出一种基于全卷积网络的建筑物语义分割方法，与当前国际上的最新算法相比达到了领先水平；③将建筑物提取的范围从像素级的语义分割推广至目标实例分割，实现以目标（建筑物）为对象的识别和提取。通过试验，验证了WHU数据库在国际上的领先性和本文方法的先进性。

Automatic extraction of buildings from remote sensing images is significant to city planning, popular estimation, map making and updating.We report several important developments in building extraction. Automatic building recognition from remote sensing data has been a scientific challenge of more than 30 years. Traditional methods based on empirical feature design can hardly realize automation. Advanced deep learning based methods show prospects but have two limitations now. Firstly, large and accurate building datasets are lacking while such dataset is the necessary fuel for deep learning. Secondly, the current researches only concern building's pixel wise semantic segmentation and the further extractions on instance-level and vector-level are urgently required. This paper proposes several solutions. First, we create a large, high-resolution, accurate and open-source building dataset, which consists of aerial and satellite images with both raster and vector labels. Second,we propose a novel structure based on fully neural network which achieved the best accuracy of semantic segmentation compared to most recent studies. Third, we propose a building instance segmentation method which expands the current studies of pixel-level segmentation to building-level segmentation. Experiments proved our dataset's superiority in accuracy and multi-usage and our methods' advancement. It is expected that our researches might push forward the challenging building extraction study.