Abstract: The video synthetic aperture radar (VideoSAR) technology is widely used in military reconnaissance, geological exploration, and disaster prediction, among other fields. Owing to multiple interference factors in SAR videos, such as speckle noise, specular reflection, and overlay effect, moving targets are easily mixed with background or other targets. Therefore, this study proposed an effective VideoSAR target detection and tracking algorithm. Firstly, several features of VideoSAR were extracted to construct multichannel feature maps. Then, deeper features were extracted using the improved lightweight EfficientDet network, thus improving the accuracy of SAR target detection while considering algorithm efficiency. Finally, the trajectory association strategy based on bounding boxes was employed to associate the same target in VideoSAR. The experimental results show that the method proposed in this study is effective for SAR shadow target detection and tracking.