Search

A Study on Sentiment Analysis Algorithms for Multimodal Data Fusion

Author(s)

Shuo Wang

Affiliation(s)

Mathematics Department, University of California, Santa Barbara, California, USA Corresponding author

Abstract

This paper studies the sentiment analysis algorithm of multimodal data fusion. Aiming at the complementary characteristics of text, audio, video and other modalities in emotional expression, a deep learning algorithm based on a multi-layer fusion framework is proposed. Through the joint design of feature extraction, modal fusion and classifier optimization, this method effectively solves the information gap caused by the heterogeneity between modalities and significantly improves the accuracy and robustness of sentiment analysis. Experimental results show that the multimodal fusion strategy is superior to the single-modal method in all evaluation indicators, especially in complex emotional scenes, showing stronger classification ability and generalization performance. The research results of this paper provide important theoretical support and technical reference for the design and optimization of multimodal sentiment analysis algorithms.

Keywords

Multimodal Fusion; Sentiment Analysis; Deep Learning; Feature Extraction; Classification Algorithm

References

[1]Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017). Tensor Fusion Network for Multimodal Sentiment Analysis. Conference on Empirical Methods in Natural Language Processing, 1103-1114. [2]Hazarika, D., Zimmermann, R., & Poria, S. (2020). MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis. Proceedings of the 28th ACM International Conference on Multimedia. [3]Aggarwal, C. (2018). Opinion Mining and Sentiment Analysis. Proceedings of the International Conference on Computer Science, 413-434. [4]Mao, R., Liu, Q., He, K., Li, W., & Cambria, E. (2023). The Biases of Pre-Trained Language Models: An Empirical Study on Prompt-Based Sentiment Analysis and Emotion Detection. IEEE Transactions on Affective Computing, 14, 1743-1753. [5]Zhu, T., Li, L., Yang, J., Zhao, S., Liu, H., & Qian, J. (2023). Multimodal Sentiment Analysis With Image-Text Interaction Network. IEEE Transactions on Multimedia, 25, 3375-3385. [6]Pawłowski, M., Wróblewska, A., & Sysko-Romańczuk, S. (2023). Effective Techniques for Multimodal Data Fusion: A Comparative Analysis. Sensors, 23. [7]Ahmed, S. F., Bin Alam, M. S., Afrin, S., Rafa, S. J., Rafa, N., & Gandomi, A. H. (2023). Insights into Internet of Medical Things (IoMT): Data Fusion, Security Issues and Potential Solutions. Information Fusion, 102, 102060. [8]Alsaeedi, A., & Zubair, M. (2023). A Study on Sentiment Analysis Techniques of Twitter Data. International Journal of Advanced Computer Science and Applications. [9]Nsengiyumva, W., Zhong, S., Luo, M., Zhang, Q., & Lin, J. (2021). Critical Insights into the State-of-the-Art NDE Data Fusion Techniques for Structural Systems. Structural Control and Health Monitoring, 29. [10]Khare, S. K., March, S., Barua, P., Gadre, V., & Acharya, U. R. (2023). Application of Data Fusion for Automated Detection of Developmental and Mental Disorders. Information Fusion, 99. [11]Steyaert, S., Pizurica, M., Nagaraj, D., Khandelwal, P., Hernandez-Boussard, T., Gentles, A., & Gevaert, O. (2023). Multimodal Data Fusion for Cancer Biomarker Discovery with Deep Learning. Nature Machine Intelligence, 5, 351-362.