The extraction of information from any remote sensing imagery has its own unavoidable theoretical limitation. Facing this problem properly can make clear the direction of research on related algorithms and reduce the cost of application. It is the Spectral Confusion within the Same Object or Similar Objects (SCSO) and the related problem of image objects with heterogeneous spectra that seriously restrict the precision of image information extraction. To understand this is the key to the awareness of such a limitation.  There exist different spectral reflective rates in different roof materials, which results in the serious SCSO phenomenon for different architecture objects in remote sensing imagery of urban areas. The Gaussian-mixture-model-based EM (Expectation Maximization) estimate algorithm can provide a scientific basis for analyzing the degree of spectral heterogeneity within a building and in an object of architecture category and hence improve the classification precision. Taking the multi-spectral QuickBird image as the sample data, this paper introduces the basic principle of Gaussian-mixture-model-based EM estimate algorithm, by means of which the density distribution of the different material roofs can be imitated and sub-classes of architecture be recognized. It turns out that the EM algorithm is superior to the traditional supervised and unsupervised classification in terms of the classification result of architecture image objects.