A method for remote sensing image semantic segmentation based on deep fusion convolutional networks and conditional random field is proposed. First, the fully convolutional networks framework with deconvolutional fusion structure is utilized to integrate the pooling-layer results at different levels. The low-level features with rich detail information are combined with the high-level features via deconvolutional fusion module. At the same time, the parameter inference process of conditional random field is embedded in the network architecture by adding recurrent neural networks iteration layers. In addition, the deep fusion convolutional networks and conditional random field model is established. Then, in the model training stage, the abundant detail information and context information in the image are introduced simultaneously to the positive and negative propagation. And lastly, the remote sensing image semantic segmentation is accomplished by the end-to-end framework. Semantic segmentation experiments were performed on the high-resolution optical remote sensing images, and the results show that, with the increase of the depth of deconvolution fusion layer in the model, semantic segmentation results are more refined, and the contour of terrestrial object is more accurate. The introduction of context information also improves the accuracy of image semantic segmentation to a certain extent. It is concluded that the proposed method can better maintain the consistency of internal areas of semantic object and effectively improve the accuracy of semantic segmentation.