Research advances and challenges in multi-label classification of remote sensing images
LIN Dan1(), LI Qiucen1, CHEN Zhikui1,2(), ZHONG Fangming1,2, LI Lifang1
1. School of Software Technology, Dalian University of Technology, Dalian 116620, China 2. Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province, Dalian University of Technology, Dalian 116620, China
Multi-label classification of remote sensing images plays a fundamental role in remote sensing analysis. Parsing given remote sensing images to identify semantic labels can provide a significant technical basis for downstream computer vision tasks. With the continuously improved spatial resolution of remote sensing images, many remote sensing objects with different scales, colors, and shapes are distributed in various zones of images, posing high challenges to the multi-label classification task of remote sensing images. This study focuses on the multi-label classification of images in the field of remote sensing, summarizing and analyzing the frontier research advances in this regard. First of all, this study expounded the problem definition for the multi-label classification task of remote sensing images while generalizing the commonly used multi-label image datasets and model evaluation indicators. Furthermore, by systematically presenting the frontier progress in this field, this study delved into two key tasks in the multi-label classification of remote sensing images: feature extraction of remote sensing images and label feature extraction. Finally, based on the characteristics of remote sensing images, this study analyzed the current challenges of multi-label classification as well as subsequent research orientation.
林聃, 李秋岑, 陈志奎, 钟芳明, 李丽方. 多标签遥感图像分类研究现状与展望[J]. 自然资源遥感, 2024, 36(2): 10-20.
LIN Dan, LI Qiucen, CHEN Zhikui, ZHONG Fangming, LI Lifang. Research advances and challenges in multi-label classification of remote sensing images. Remote Sensing for Natural Resources, 2024, 36(2): 10-20.
Chen C F, Bian X Y, Fei X J, et al. Weakly supervised multi-instance sub-concept learning for remote sensing scene classification[J]. Journal of Chinese Computer Systems, 2022, 43(1):76-83.
[2]
Li K, Wan G, Cheng G, et al. Object detection in optical remote sensing images:A survey and a new benchmark[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 159:296-307.
[3]
Yuan X H, Shi J F, Gu L C. A review of deep learning methods for semantic segmentation of remote sensing imagery[J]. Expert Systems with Applications, 2021, 169:114417.
Zhao Z Y, Yang D, Liang X H, et al. Study on storage and migration strategy of massive remote sensing data[J]. Journal of Information Engineering University, 2020, 21(1):115-119.
Bai J L, Wang Z Q, Yan H T. A K-means clustering-guided threshold-based approach to classifying UAV remote sensed images[J]. Remote Sensing for Natural Resources, 2021, 33(3):114-120.doi:10.6046/zrzyyg.2020301.
Tu B, Zhang X F, Zhang G Y, et al. Hyperspectral image classification via recursive filtering and KNN[J]. Remote Sensing for Land and Resources, 2019, 31(1):22-32.doi:10.6046/gtzyyg.2019.01.04.
[7]
Wang T, Zhang H S, Lin H, et al. Textural- spectral feature-based species classification of mangroves in Mai Po Nature Reserve from WorldView-3 imagery[J]. Remote Sensing, 2015, 8(1):24-35.
[8]
Zhou W, Song Y Q, Pan Z K, et al. Classification of urban construction land with WorldView-2 remote sensing image based on classification and regression tree algorithm[C]// 2017 IEEE International Conference on Computational Science and Engineering(CSE) and IEEE International Conference on Embedded and Ubiquitous Computing(EUC), 2017:277-283.
[9]
Hamidishad N, Cesar R Jr. Object-based method for identifying new constructions around water reservoirs:Preliminary results[C]// Anais Estendidos da Conference on Graphics,Patterns and Images(SIBGRAPI), 2019, 32:172-175.
[10]
Yessou H, Sumbul G, Demir B. A comparative study of deep learning loss functions for multi-label remote sensing image classification[C]// 2020 IEEE International Geoscience and Remote Sensing Symposium(IGARSS), 2020:1349-1352.
[11]
Chaudhuri B, Demir B, Chaudhuri S, et al. Multilabel remote sensing image retrieval using a semisupervised graph-theoretic method[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(2):1144-1158.
[12]
Xia G S, Hu J W, Hu F, et al. AID:A benchmark data set for performance evaluation of aerial scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(7),3965-3981.
[13]
Hua Y, Mou L, Jin P, et al. MultiScene:A large-scale dataset and benchmark for multiscene recognition in single aerial images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021(60):5010213.
[14]
Qi X, Zhu P, Wang Y, et al. MLRSNet:A multi-label high spatial resolution remote sensing dataset for semantic scene understanding[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 169:337-350.
[15]
Sumbul G, Charfuelan M, Demir B, et al. Bigearthnet:A large-scale benchmark archive for remote sensing image understanding[C]// IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium.IEEE, 2019:5901-5904.
[16]
Sumbul G, Wall A D, Kreuziger T, et al. BigEarthNet-MM:A large scale multi modal multi label benchmark archive for remote sensing image classification and retrieval[J]. IEEE Geoscience and Remote Sensing Magazine, 2021, 9(3):174-180.
[17]
Cheng G, Xie X, Han J, et al. Remote sensing image scene classification meets deep learning:Challenges,methods,benchmarks,and opportunities[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13:3735-3756.
[18]
Blaschke T, Strobl J. What’s wrong with pixels? Some recent developments interfacing remote sensing and GIS[J]. Interfacing Remote Sensing and GIS, 2001, 6(1):12-17.
[19]
Chen T, Wang Z, Li G, et al. Recurrent attentional reinforcement learning for multi-label image recognition[C]// Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32(1).
Zeng L, Xu H Y, Chen X H, et al. Application of deep learning in scene classificationof high-resolution remote sensing images[J]. Computer Engineering and Applications, 2021, 57(10):57-64.
Cai Z L, Weng Q, Ye S Z, et al. Remote sensing image scene classification based on Inception-V3[J]. Remote Sensing for Land and Resources, 2020, 32(3):80-89.doi:10.6046/gtzyyg.2020.03.11.
[22]
Zeggada A, Melgani F, Bazi Y. A deep learning approach to UAV image multilabeling[J]. IEEE Geoscience and Remote Sensing Letters, 2017, 14(5):694-698.
[23]
Shendryk I, Rist Y, Lucas R, et al. Deep learning:A new approach for multi-label scene classification in planetscope and Sentinel-2 imagery[C]// IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium.IEEE, 2018:1116-1119.
[24]
Karalas K, Tsagkatakisg G, Zervakism M, et al. Land classification using remotely sensed data:Going multi-label[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016, 54:3548-3563.
[25]
Melo A, Paulheim H. Local and global feature selection for multi-label classification with binary relevance[J]. Artificial Intelligence Review, 2019, 51(1):33-60.
[26]
Fdez-Díaz M, Fdez-Díaz L, Mena D, et al. Improving the ε-approximate algorithm for probabilistic classifier chains[J]. Knowledge and Information Systems, 2020, 62(7):2709-2738.
[27]
Zhou C L, Yuan J S. Multi-label learning of part detectors for heavily occluded pedestrian detection[C]// Proceedings of the IEEE International Conference on Computer Vision(ICCV). 2017:3486-3495.
[28]
Tsagkatakis G, Tsakalides P. Deep feature learning for hyperspectral image classification and land cover estimation[J]. Environmental Science,Mathematics, 2016, 6:435-446.
[29]
Cevikalp H, Benligiray B, Gerek O N. Semi-supervised robust deep neural networks for multi-label image classification[J]. Pattern Recognition, 2020, 100:107164.
[30]
Zhang T, Yan W, Li J, et al. Multiclass labeling of very high-resolution remote sensing imagery by enforcing nonlocal shared constraints in multilevel conditional random fields model[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2016, 9(7):2854-2867.
[31]
Wang C, Zhang P, Zhang Y, et al. A multi-label hyperspectral image classification method with deep learning features[C]// Proceedings of the International Conference on Internet Multimedia Computing and Service, 2016:127-131.
[32]
Koda S, Zeggada A, Melgani F, et al. Spatial and structured SVM for multilabel image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(10):5948-5960.
Hua Y S. Research status of application of deep learning in multi-label remote sensing image classification[J]. Scientific and Technological Innovation, 2020(1):44-45.
[34]
Hua Y, Mou L, Zhu X X. Recurrently exploring class-wise attention in a hybrid convolutional and bidirectional LSTM network for multi-label aerial image classification[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 149:188-199.
doi: 10.1016/j.isprsjprs.2019.01.015
pmid: 31007387
[35]
Wang X, Xiong X, Ning C. Multi-label remote sensing scene classification using multi-bag integration[J]. IEEE Access, 2019(7):120399-120410.
[36]
Zhang J L, Zhang J Y, Dai T, et al. Exploring weighted dual graph regularized non-negative matrix tri-factorization based collaborative filtering framework for multi-label annotation of remote sensing images[J]. Remote Sensing, 2019, 11(8):922.
[37]
Khan N, Chaudhuri U, Banerjee B, et al. Graph convolutional network for multi-label VHR remote sensing scene recognition[J]. Neurocomputing, 2019, 357:36-46.
[38]
Hua Y S, Lobry S., Mou L C, et al. Learning multi-label aerial image classification under label noise:A regularization approach using word embeddings[C]// 2020 IEEE International Geoscience and Remote Sensing Symposium(IGARSS), 2020,525-528.
[39]
Hao Q B, Li S T, Kang X D. Multilabel sample augmentation-based hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58:4263-4278.
[40]
Yang Y, Newsam S. Bag-of-visual-words and spatial extensions for land-use classification[C]// ACM Sigspatial International Conference on Advances in Geographic Information Systems. 2010:270-279.
[41]
Song Z L, Zhang J Q. Image registration approach with scale-invariant feature transform algorithm and tangent-crossing-point feature[J]. Journal of Electronic Imaging, 2020, 29:023010.
Jiang Y N, Zhang X, Zhang C L, et al. Classification of remote sensing images based on multi-scale feature fusion using local binary patterns[J]. Remote Sensing for Natural Resources, 2021, 33(3):36-44.doi:10.6046/zrzyyg.2020303.
[43]
Vincent P, Larochelle H, Lajoie I, et al. Stacked denoising autoencoders:Learning useful representations in a deep network with a local denoising criterion[J]. Journal of Machine Learning Research, 2010, 11(12).
[44]
Hinton G E, Simon O, Teh Y W. A fast learning algorithm for deep belief nets[J]. Neural Computation, 2006, 18:1527-1554.
doi: 10.1162/neco.2006.18.7.1527
pmid: 16764513
[45]
Hinton G E, Salakhutdinov R. Reducing the dimensionality of data with neural networks[J]. Science, 2006, 313:504-507.
doi: 10.1126/science.1127647
pmid: 16873662
[46]
Fang S, Zhang K, Wang Y B, et al. Multi-label hyperspectral classification with discriminative features[C]// 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, 2021:3233-3236.
[47]
Zhang M L, Zhou Z H. ML-KNN:A lazy learning approach to multi-label learning[J]. Pattern Recognition, 2007, 40(7):2038-2048.
[48]
Wang J, Yang Y, Mao J H, et al. CNN-RNN:A unified framework for multi-label image classification[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016:2285-2294.
[49]
Li Y, Song Y L, Luo J B. Improving pairwise ranking for multi-label image classification[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017:3617-3625.
[50]
Yan Y F, Huang S J. Cost-effective active learning for hierarchical multi-label classification[C]// Proceedings of the 27th International Joint Conference on Artificial Intelligence, 2018:2962-2968.
[51]
Li Y S, Chen R X, Zhang Y J, et al. Multi-label remote sensing image scene classification by combining a convolutional neural network and a graph neural network[J]. Remote Sensing, 2020, 12(23):4003.
[52]
Aksoy A K, Ravanbakhsh M, Kreuziger T, et al. A novel uncertainty-aware collaborative learning method for remote sensing image classification under multi-label noise[J]. CoRR,vol. vol.abs/2105.05496,2021.
[53]
El-Khamy S E, Al-Kabbany A, El-Bana S. MLRS-CNN-DWTPL:A new enhanced multi-label remote sensing scene classification using deep neural networks with wavelet pooling layers[C]// 2021 International Telecommunications Conference(ITC-Egypt), 2021:1-5.
[54]
Ji J C, Jing W P, Chen G S, et al. Multi-label remote sensing image classification with latent semantic dependencies[J]. Remote Sensing, 2020, 12:1110.
[55]
Sumbul G, Demir B. A deep multi-attention driven approach for multi-label remote sensing image classification[J]. IEEE Access, 2020, 8:95934-95946.
[56]
Yang M H, Liu H, Gao L, et al. DCA-GCN:A dual-branching channel attention and graph convolution network for multi-label remote sensing image classification[J]. Journal of Applied Remote Sensing, 2021, 15:044519.
[57]
Diao Y, Chen J Z, Qian Y. Multi-label remote sensing image classification with deformable convolutions and graph neural networks[C]// IGARSS 2020—2020 IEEE International Geoscience and Remote Sensing Symposium, 2020:521-524.
[58]
Huang S J, Gao W, Zhou Z H. Fast multi-instance multi-label learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41:2614-2627.
[59]
Zhou Z H, Zhang M L. Multi-instance multi-label learning with application to scene classification[C]// Advances in Neural Information Processing Systems, 2006:1609-1616.
[60]
Mohamed D. Multi-instance multi-label with application to high resolution remote sensing images[J]. Journal of Physics:Conference Series, 2018, 18:614-623.
[61]
Aksoy A K, Ravanbakhsh M, Demir B. Multi-label noise robust collaborative learning for remote sensing image classification[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022:1-14.
[62]
Aksoy A K, Ravanbakhsh M, Kreuziger T, et al. A consensual collaborative learning method for remote sensing image classification under noisy multi-labels[C]// 2021 IEEE International Conference on Image Processing(ICIP), 2021:3842-3846.
[63]
Burgert T, Ravanbakhsh M, Demir B. On the effects of different types of label noise in multi-label remote sensing image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:5413713.
[64]
Li Q, Qiao M Y, Bian W, et al. Conditional graphical lasso for multi-label image classification[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016:2977-2986.
[65]
Hua Y S, Mou L C, Zhu X X. Label relation inference for multi-label aerial image classification[C]// IEEE International Geoscience and Remote Sensing Symposium(IGARSS), 2019:5244-5247.
[66]
Chen J Z, Qian Y. Hierarchical multi-label ship recognition in remote sensing images using label relation graphs[C]// 2021 IEEE International Geoscience and Remote Sensing Symposium(IGARSS), 2021:4968-4971.
[67]
Zhang J J, Wu Q, Shen C H, et al. Multi-label image classification with regional latent semantic dependencies[J]. IEEE Transactions on Multimedia, 2018, 20(10):2801-2813.
[68]
Zhu F, Li H S, Ouyang W L, et al. Learning spatial regularization with image-level supervisions for multi-label image classification[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2017:2027-2036.
[69]
Kipf T N, Welling M. Semi-supervised classification with graph convolutional networks[EB/OL]. arXiv.(2016-09-09) [2023-03-27]https://arxiv.org/abs/1609.02907.
[70]
Wang Y, He D L, Li F, et al. Multi-label classification with label graph superimposing[EB/OL]. arXiv.(2019-11-21) [2023-03-27]https://arxiv.org/abs/1911.09243.
[71]
Li Y S, Chen R X, Zhang Y J, et al. A CNN-GCN framework for multi-label aerial image scene classification[C]// IEEE International Geoscience and Remote Sensing Symposium(IGARSS), 2020:1353-1356.
Yang M H, Chen L, Liu H, et al. Multi-label remote sensing image classification based on graph convolutional network[J]. Application Research of Computer, 2021, 38(11):3439-3445.
[73]
Lin D, Lin J Z, Zhao L, et al. Multi-label aerial image classification with a concept attention graph neural network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:5602112.
[74]
Lin D, Lin J, Zhao L, et al. Multilabel aerial image classification with unsupervised domain adaptation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:5609613.
2024, Vol. 36 
), 李秋岑1, 陈志奎1,2(
