Urbanization has decreased the area of ecological land and deteriorated ecological environment in Zhanjiang City. Therefore, it is significant to quickly, comprehensively, and accurately monitor the changes the ecological environment quality in this city. Based on the Landsat images in 2000, 2005, 2009, 2015, and 2020, this study constructed the improved remote sensing ecological index (IRSEI) using six indicators, namely greenness (NDVI), humidity (WET), dryness (NDBSI), heatiness (LST), land use (LUI), and population distribution (POP). Using IRSEI, this study quantitatively analyzed the changes in the ecological environment quality in Zhanjiang during 2000—2020. The results are as follows: ① The mean IRSEI values of 2000, 2005, 2009, 2015, and 2020 are 0.18, 0.18, 0.35, 0.42, and 0.38, respectively, showing a first increasing and then decreasing trend. ② According to the difference processing on IRSEIs during 2000—2020, the proportions of ecological environment areas with significant improvement (dominant), improvement, no change, deterioration, and significant deterioration in the study area are 78.95%, 8.70%, 8.01%, 1.35%, and 2.99%, respectively. ③ The IRSEI can effectively reflect the poor urban environment along the coastal zone during 2000—2020, specifically manifested as a low IRSEI value of building land along the coastal zone. The results of this study can provide a theoretical and scientific basis for Zhanjiang’s ecological environment protection.