Deep learning-based scene analysis of remote sensing images serves as a critical means for post-earthquake damage assessment. Given scarce images of damaged buildings, constructing high-quality sample sets of remote sensing scene images holds crucial significance for improving the accuracy of scene recognition and classification. The proportion of contextual information in scene images, as a significant reference for remote sensing analysis, is a key factor affecting the construction effects of sample sets. Currently, the appropriate proportion of contextual information remains under-studied in the sample set construction method. Aiming to construct high-quality sample sets, this study designed a method for adjusting the proportion of contextual information in scene images. It investigated the impacts of different proportions of contextual information on the construction of scene sample sets, exploring the optimal proportion range of contextual information. This study constructed six sample sets of scene images under different proportions of contextual information for training and testing in five classic convolutional neural network (CNN) models. It analyzed the classification results of all the CNN models under different proportions of contextual information. The results indicate that with the proportion of contextual information being 80%, the classification accuracy of the CNN reached an optimal value of 92.22%, which decreased to 89.03% with the proportion of contextual information at 95%. Among all the CNN models, GoogLeNet exhibited superior classification performance with an average accuracy of 93.13%. This study enables the setting of proper proportion ranges of contextual information in scene sample sets, thus effectively improving the classification accuracy of remote sensing scene images, and guiding the construction of sample sets of remote sensing scene images for damaged buildings.