Improving the Cross-Domain Classification of Short Text Using the Deep Transfer Learning Framework

Document Type : Research Paper

Authors

1 MA., Department of Information Technology Management, Faculty of Social Sciences and Economics, Alzahra University, Tehran, Iran.

2 Associate Prof., Department of Management, Faculty of Social Sciences and Economics, Alzahra University, Tehran, Iran.

Abstract

With the advent of user-generated text information on the Internet, text sentiment analysis plays an essential role in online business transactions. The expression of feelings and opinions depends on the domains, which have different distributions. In addition, each of these domains or so-called product groups has its vocabulary and peculiarities that make analysis difficult. Therefore, different methods and approaches have been developed in this area. However, most of the analysis involved a single-domain and few studies on cross-domain mood classification using deep neural networks have been performed. The aim of this study was therefore to examine the accuracy and transferability of deep learning frameworks for the cross-domain sentiment analysis of customer ratings for different product groups as well as the cross-domain sentiment classification in five categories “very positive”, “positive”, “neutral”, “negative” and “very negative”. Labels were extracted and weighted using the Long Short-Term Memory (LSTM) Recurrent Neural Network. In this study, the RNN LSTM network was used to implement a deep transfer learning framework because of its significant results in sentiment analysis. In addition, two different methods of text representation, BOW and CBOW were used. Based on the results, using deep learning models and transferring weights from the source domain to the target domain can be effective in cross-domain sentiment analysis.

Keywords

Main Subjects

References

Abdullah, N. A., Feizollah, A., Sulaiman, A., & Anuar, N. B. (2019). Challenges and Recommended Solutions in Multi-Source and Multi-Domain Sentiment Analysis. IEEE Access, 7, 144957-144971. https://doi.org/10.1109/ACCESS.2019.2945340
Alami, N., Meknassi, M., & En-nahnahi, N. (2019). Enhancing unsupervised neural networks-based text summarization with word embedding and ensemble learning. Expert systems with applications, 123, 195-211. https://doi.org/10.1016/j.eswa.2019.01.037
Al-Moslmi, T., Omar, N., Abdullah, S., & Albared, M. (2017). Approaches to cross-domain sentiment analysis: A systematic literature review. IEEE access, 5, 16173-16192. https://doi.org/10.1109/ACCESS.2017.2690342
Babu, N. V., & Kanaga, E. G. M. (2022). Sentiment analysis in social media data for depression detection using artificial intelligence: a review. SN Computer Science, 3, 1-20. https://doi.org/10.1007/s42979-021-00958-1
Başarslan, M. S., & Kayaalp, F. (2023). Sentiment analysis with ensemble and machine learning methods in multi-domain datasets. Turkish Journal of Engineering, 7(2), 141-148. https://doi.org/10.31127/tuje.1079698
Bengio, Y. (2012, June). Deep learning of representations for unsupervised and transfer learning. JMLR: Workshop and Conference Proceedings 27, 17–37. http://proceedings.mlr.press/v27/bengio12a/bengio12a.pdf
Bhavitha, B. K., Rodrigues, A. P., & Chiplunkar, N. N. (2017, March). Comparative study of machine learning techniques in sentimental analysis. In 2017 International conference on inventive communication and computational technologies (ICICCT) (pp. 216-221). IEEE. https://doi.org/10.1109/ICICCT.2017.7975191
Blei, D. M. (2012). Probabilistic topic models. Communications of the ACM, 55(4), 77-84. https://doi.org/10.1145/2133806.2133826
Chen, J., Feng, J., Sun, X., & Liu, Y. (2020). Co-Training Semi-Supervised Deep Learning for Sentiment Classification of MOOC Forum Posts. Symmetry, 12(1), 8-32. https://doi.org/10.3390/sym12010008
Chen, T., Xu, R., He, Y., & Wang, X. (2017a). Improving sentiment analysis via sentence type classification using BiLSTM-CRF and CNN. Expert Systems with Applications, 72, 221-230. https://doi.org/10.1016/j.eswa.2016.10.065
Chen, L., Guan, Z., He, J., & Peng, J. (2017b). A survey on sentiment classification. J. Comput. Res. Dev, 54(6), 1150-1170.
Dahal, B., Kumar, S. A., & Li, Z. (2019). Topic modeling and sentiment analysis of global climate change tweets. Social Network Analysis and Mining, 9(24), 1-20.
Dang, N. C., Moreno-García, M. N., & De la Prieta, F. (2020). Sentiment analysis based on deep learning: A comparative study. Electronics, 9(3), 483-512. https://doi.org/10.3390/electronics9030483
Gulati, K., Kumar, S. S., Boddu, R. S. K., Sarvakar, K., Sharma, D. K., & Nomani, M. Z. M. (2022). Comparative analysis of machine learning-based classification models using sentiment classification of tweets related to COVID-19 pandemic. Materials Today: Proceedings, 51, 38-41 https://doi.org/10.1016/j.matpr.2021.04.364
Haque, T. U., Saber, N. N., & Shah, F. M. (2018, May). Sentiment analysis on large scale Amazon product reviews. In 2018 IEEE International Conference on Innovative Research and Development (ICIRD) (pp. 1-6). IEEE. https://doi.org/10.1109/ICIRD.2018.8376299
Huq, M. R., Ahmad, A., & Rahman, A. (2017). Sentiment analysis on Twitter data using KNN and SVM. International Journal of Advanced Computer Science and Applications, 8(6),19-25. https://pdfs.semanticscholar.org/05a8/78000170abcd0c6f8208080470858422e17c.pdf
Inzalkar, S., & Sharma, J. (2015). A survey on text mining-techniques and application. International Journal of Research in Science & Engineering, 24, 1-14. http://www.ttcenter.ir/ArticleFiles/ENARTICLE/3771.pdf
Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. https://doi.org/10.48550/arXiv.1412.6980
Li, Z., Wei, Y., Zhang, Y., & Yang, Q. (2018, April). Hierarchical attention transfer network for cross-domain sentiment classification. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 32, No. 1). https://doi.org/10.1609/aaai.v32i1.12055
Li, Z., Zhang, Y., Wei, Y., Wu, Y., & Yang, Q. (2017, August). End-to-End Adversarial Memory Network for Cross-domain Sentiment Classification. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17) (pp. 2237-2243). https://hsqmlzno1.github.io/assets/publications/AMN2017.pdf
Liddy, E. D. (2001). Natural language processing. In Encyclopedia of Library and Information Science, 2nd Ed. NY. Marcel Decker, Inc. https://surface.syr.edu/cgi/viewcontent.cgi?article=1043&context=istpub
Lim, S., & Lee, D. (2017). Stable improved softmax using constant normalization. Electronics Letters, 53(23), 1504-1506. https://doi.org/10.1049/el.2017.3394
Lin, C., & He, Y. (2009, November). Joint sentiment/topic model for sentiment analysis. In Proceedings of the 18th ACM conference on Information and knowledge management (pp. 375-384). https://doi.org/10.1145/1645953.1646003
Lin, S., Xie, H., Yu, L. C., & Lai, K. R. (2017, December). SentiNLP at IJCNLP-2017 Task 4: Customer feedback analysis using a Bi-LSTM-CNN model. In Proceedings of the IJCNLP 2017, Shared Tasks (pp. 149-154). https://aclanthology.org/I17-4025/
Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167. http://proceedings.mlr.press/v37/ioffe15.html
Liu, Y., Jin, J., Ji, P., Harding, J. A., & Fung, R. Y. (2013). Identifying helpful online reviews: a product designer’s perspective. Computer-Aided Design, 45(2), 180-194. https://doi.org/10.1016/j.cad.2012.07.008
McTear, M., Callejas, Z., & Griol, D. (2016). The dawn of the conversational interface. In The Conversational Interface (pp. 11-24). Springer, Cham. https://doi.org/10.1007/978-3-319-32967-3_2
Medhat, W., Hassan, A., & Korashy, H. (2014). Sentiment analysis algorithms and applications: A survey. Ain Shams engineering journal, 5(4), 1093-1113. https://doi.org/10.1016/j.asej.2014.04.011
Monali, P., & Sandip, K. (2014). A concise survey on text data mining. International Journal of Advanced Research in Computer and Communication Engineering, 3(9), 8040-8043.
Pal, S., Ghosh, S., & Nag, A. (2018). Sentiment analysis in the light of LSTM recurrent neural networks. International Journal of Synthetic Emotions (IJSE), 9(1), 33-39. https://doi.org/10.4018/IJSE.2018010103
Pandey, A. C., Rajpoot, D. S., & Saraswat, M. (2017). Twitter sentiment analysis using hybrid cuckoo search method. Information Processing & Management, 53(4), 764-779. https://doi.org/10.1016/j.ipm.2017.02.004
Peng, M., Zhang, Q., Jiang, Y. G., & Huang, X. J. (2018, July). Cross-domain sentiment classification with target domain specific information. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 2505-2513). Melbourne, Australia. https://doi.org/10.18653/v1/P18-1233
Piryani, R., Madhavi, D., & Singh, V. K. (2017). Analytical mapping of opinion mining and sentiment analysis research during 2000–2015. Information Processing & Management, 53(1), 122-150. https://doi.org/10.1016/j.ipm.2016.07.001
Qi, J., Zhang, Z., Jeon, S., & Zhou, Y. (2016). Mining customer requirements from online reviews: A product improvement perspective. Information & Management, 53(8), 951-963. https://doi.org/10.1016/j.im.2016.06.002
Salas-Zárate, M. D. P., Medina-Moreira, J., Lagos-Ortiz, K., Luna-Aveiga, H., Rodriguez-Garcia, M. A., & Valencia-Garcia, R. (2017). Sentiment analysis on tweets about diabetes: an aspect-level approach. Computational and mathematical methods in medicine, 2017. https://doi.org/10.1155/2017/5140631
Severyn, A., & Moschitti, A. (2015, August). Twitter sentiment analysis with deep convolutional neural networks. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 959-962). https://doi.org/10.1145/2766462.2767830
Sivic, J., & Zisserman, A. (2008). Efficient visual search of videos cast as text retrieval. IEEE transactions on pattern analysis and machine intelligence, 31(4), 591-606. https://doi.org/10.1109/TPAMI.2008.111
Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems (pp. 2951-2959). https://proceedings.neurips.cc/paper_files/paper/2012/hash/05311655a15b75fab86956663e1819cd-Abstract.html
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1), 1929-1958. https://www.jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf?utm_content=buffer79b43&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer,
Sun, S., Luo, C., & Chen, J. (2017). A review of natural language processing techniques for opinion mining systems. Information fusion, 36, 10-25. https://doi.org/10.1016/j.inffus.2016.10.004
Sundström, J. (2018). Sentiment analysis of Swedish reviews and transfer learning using convolutional neural networks. UPTEC STS. Retrieved from http://urn.kb.se/resolve?urn=urn:nbn:se:uu:diva-339066.
Tang, D., Qin, B., & Liu, T. (2015). Deep learning for sentiment analysis: successful approaches and future challenges. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 5(6), 292-303. https://doi.org/10.1002/widm.1171
Vidya, S. (2018). Cross Domain Sentiment Classification Using Natural Language Processing.‏ International Journal of Scientific Research in Computer Science, Engineering and Information Technology. 3(3), 348-353. https://www.researchgate.net/profile/Dr-Svidya/publication/361983484_Cross_Domain_Sentiment_Classification_Using_Natural_Language_Processing/links/62cfc802e2a5013989058038/Cross-Domain-Sentiment-Classification-Using-Natural-Language-Processing.pdf
Wei, X., Lin, H., Yang, L., & Yu, Y. (2017). A convolution-LSTM-based deep neural network for cross-domain MOOC forum post classification. Information, 8(3), 92-108. https://doi.org/10.3390/info8030092
Yadav, A., & Vishwakarma, D. K. (2023). A deep multi-level attentive network for multimodal sentiment analysis. ACM Transactions on Multimedia Computing, Communications and Applications, 19(1), 1-19. https://doi.org/10.3390/info8030092
Yang, L., Li, Y., Wang, J., & Sherratt, R. S. (2020). Sentiment analysis for E-commerce product reviews in Chinese based on sentiment lexicon and deep learning. IEEE access, 8, 23522-23530.  https://doi.org/10.1109/ACCESS.2020.2969854
Yang, X., Zhang, T., Xu, C., & Yang, M. H. (2015). Boosted multi-feature learning for cross-domain transfer. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 11(3), 1-18. https://doi.org/10.1145/2700286
Yuan, Z., Wu, S., Wu, F., Liu, J., & Huang, Y. (2018). Domain attention model for multi-domain sentiment classification. Knowledge-Based Systems, 155, 1-10. https://doi.org/10.1016/j.knosys.2018.05.004
Yuan, Y., & Zhou, Y. (2015). Twitter sentiment analysis with recursive neural networks. CS224D course projects. https://cs224d.stanford.edu/reports/YuanYe.pdf
Zhang, L., Wang, S., & Liu, B. (2018). Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4), e1253. https://doi.org/10.1002/widm.1253
Zhang, X., & Zheng, X. (2016, July). Comparison of text sentiment analysis based on machine learning. In 2016 15th international symposium on parallel and distributed computing (ISPDC) (pp. 230-233). IEEE. https://doi.org/10.1109/ISPDC.2016.39s
Zhou, G., Zhou, Y., Guo, X., Tu, X., & He, T. (2015). Cross-domain sentiment classification via topical correspondence transfer. Neurocomputing, 159, 298-305. https://doi.org/10.1016/j.neucom.2014.12.006
Zhao, C., Wang, S., & Li, D. (2017, September). Deep transfer learning for social media cross-domain sentiment classification. In Chinese National Conference on Social Media Processing (pp. 232-243). Springer, Singapore. https://doi.org/10.1007/978-981-10-6805-819
Zhu, F., Dong, X., Song, R., Hong, Y., & Zhu, Q. (2017, November). A multiple learning model based voting system for news headline classification. In National CCF Conference on Natural Language Processing and Chinese Computing (pp. 797-806). Springer, Cham. https://doi.org/10.1007/978-3-319-73618-1_69
Ziser, Y., & Reichart, R. (2018, June). Pivot based language modeling for improved neural domain adaptation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) (pp. 1241-1251). https://aclanthology.org/N18-1112/
Zola, Paola, Paulo Cortez, Costantino Ragno, and Eugenio Brentari. (2019). Social Media Cross-Source and Cross-Domain Sentiment Classification. International Journal of Information Technology & Decision Making (IJITDM). 18(5), 1469-1499. https://doi.org/10.1142/S0219622019500305
Journal of Information Technology Management
Volume 16, Issue 2
2024
Pages 16-34

Files

Share

How to cite

Statistics