An effective unmixing method of hyperspectral mixed pixels can improve the precision of mineral information extraction. To further study such unmixing methods, this study explained the imaging mechanism of hyperspectral images using a linear spectral mixing model. The linear combinations of different mineral endmembers were used to express mixed pixels. The expected maximum (EM) algorithm was used to estimate the endmembers and abundance of mixed pixels under the framework of maximum likelihood estimation. A robust K-P-Means algorithm based on a random sampling consensus algorithm was proposed to improve the endmember optimization process, aiming to resist the impacts of anomalies on endmember extraction. The spectral angular distance and the spectral information divergence were used to assess the consistency between the estimated endmembers and the real endmembers. To obtain the similarity between the image and the original image, the structural similarity and the peak signal-to-noise ratio were used to measure the estimated abundance and endmembers. Various simulation data indicators show that the optimized model can obtain more precise estimations of endmembers and abundance. The mineral types were determined by matching the extracted endmembers with the mineral spectrum curves provided by the USGS spectral library. The actual data originated from the Cuprite data set of the AVIRIS hyperspectral sensor for the Nevada copper mining area. The results of mineral extraction showed that the model proposed in this study yielded satisfactory recognition results for eight types of main minerals including chlorite, which showed significant mineral aggregation and were consistent with the actual situation. Therefore, the method proposed in this study can extract precise mineral information while effectively resisting the impacts of noise.