The linear spectral mixture model (LSMM) is usually used to study the urban environmental biophysical composition. The development of high-quality fraction images depends greatly on the selection of suitable end-members, which constitutes a key step. Spectral variability is obvious in the heterogeneous urban area, especially for high albedo objects, and this phenomenon affects the accuracy of LSMM. The study in this paper is based on the key assumption that the same pure land cover types exhibit obvious spectral similarity. The objective of this study is to examine the applicability of optimizing end-members selection based on spectral similarity. Four end-members, namely, vegetation, soil, low albedo and high albedo, were selected to model urban land cover by using Landsat Thematic Mapper data. Impervious surface abundance was estimated by adding low albedo fraction and high albedo fraction. Quantitative validation using impervious surface abundance measurement derived from high resolution multispectral QuickBird imagery indicates that the optimized end-members can reduce the residual error caused by end-member spectral variability. The results of this study reveal that impervious surface abundance distribution can be derived with a promising accuracy.