RES2NET-MAXOUT: Image-Based Multi-Class Malware Classification with Deep Learning Models

Document Type : Research Paper

Authors

1 Associate prof., Kallam HaranadhaReddy Institute of Technology (KHIT) (AUTONOMOUS), Chowdavaram, Andhra Pradesh, India.

2 Assistant Prof., Department of Computer and Engineering- Artificial intelligence & Machine learning, Kallam HaranadhaReddy Institute of Technology (KHIT) (AUTONOMOUS), Chowdavaram, Andhra Pradesh, India.

3 Associate Prof., Department of Computer and Engineering- Artificial intelligence & Machine learning, Kallam Haranadhareddy Institute of Technology (KHIT) (AUTONOMOUS), Chowdavaram, Andhra Pradesh, India.

4 Associate prof., Computer and Engineering- Artificial intelligence & Machine learning, Kallam HaranadhaReddy Institute of Technology (KHIT) (AUTONOMOUS), Chowdavaram, Andhra Pradesh, India.

5 Assistant prof., Computer and Engineering- Artificial intelligence & Machine learning, Kallam HaranadhaReddy Institute of Technology (KHIT) (AUTONOMOUS), Chowdavaram, Andhra Pradesh, India.

10.22059/jitm.2026.388431.3967

Abstract

The dynamic nature of the Internet has resulted in an increase in network security threats, primarily caused by the growing incidence of malware and the continuous emergence of its many forms. Conventional methods for detecting malware involve complex feature engineering, which considerably reduces detection effectiveness. Furthermore, machine learning algorithms, which were once a mainstay in the fight against cyber threats, are no longer able to accurately detect every new and complex malware variation. In contrast, deep learning approaches have the potential to be transformative. Their inherent flexibility and capacity for pattern recognition present a viable solution to the ever changing challenge of detecting a wide range of malware variations more reliably and effectively. For this reason, a unique deep learning based architecture is developed in this study to categorize malware types and protect networks. Three components comprise the most significant parts of this work: feature extraction, classification, and malware visualization. The suggested method first converts malware binary data into two dimensional images. Secondly, we introduce an automatic feature extractor that utilizes the Res2Net model to extract shared patterns among family members from the visualized images. Lastly, we propose a Deep Maxout Network designed to categorize malware using the extracted attributes. Moreover, data augmentation based on Conditional Generative Adversarial Networks (CGAN) addresses the issue of imbalanced data. The proposed technique is evaluated using the MaleVis, Microsoft Malware Classification Challenge (MMCC), and Drebin datasets. The experimental findings demonstrate that the proposed method can successfully and accurately classify malware compared to current state of the art techniques.

Keywords

Main Subjects

References

Ahmed, I. T., Jamil, N., Din, M. M., & Hammad, B. T. (2022). Binary and Multi-Class Malware Threads Classification. Applied Sciences, 12(24), 12528.
Alamro, H., Mtouaa, W., Aljameel, S., Salama, A. S., Hamza, M. A., & Othman, A. Y. (2023). Automated android malware detection using optimal ensemble learning approach for cybersecurity. IEEE Access.
Al-Andoli, M. N., Sim, K. S., Tan, S. C., Goh, P. Y., & Lim, C. P. (2023). An ensemble-based parallel deep learning classifier with PSO-BP optimization for malware detection. IEEE Access.
Alnajim, A. M., Habib, S., Islam, M., Albelaihi, R., & Alabdulatif, A. (2023). Mitigating the risks of malware attacks with deep learning techniques. Electronics, 12(14), 3166.
Alomari, E. S., Nuiaa, R. R., Alyasseri, Z. A. A., Mohammed, H. J., Sani, N. S., Esa, M. I., & Musawi, B. A. (2023). Malware detection using deep learning and correlation-based feature selection. Symmetry, 15(1), 123.
Alzubi, O. A., Alzubi, J. A., Alzubi, T. M., & Singh, A. (2023). Quantum Mayfly optimization with encoder-decoder driven LSTM networks for malware detection and classification model. Mobile Networks and Applications, 1–13.
Aslan, Ö., & Yilmaz, A. A. (2021). A new malware classification framework based on deep learning algorithms. IEEE Access, 9, 87936–87951.
Atacak, İ., Kılıç, K., & Doğru, İ. A. (2022). Android malware detection using hybrid ANFIS architecture with low computational cost convolutional layers. PeerJ Computer Science, 8, e1092.
Atitallah, S. B., Driss, M., & Almomani, I. (2022). A novel detection and multi-classification approach for IoT-malware using random forest voting of fine-tuning convolutional neural networks. Sensors, 22(11), 4302.
Aurangzeb, S., & Aleem, M. (2023). Evaluation and classification of obfuscated Android malware through deep learning using ensemble voting mechanism. Scientific Reports, 13(1), 3093.
Chaganti, R., Ravi, V., & Pham, T. D. (2022). Image-based malware representation approach with EfficientNet convolutional neural networks for effective malware classification. Journal of Information Security and Applications, 69, 103306.
Chen, Z., & Cao, J. (2023). VMCTE: Visualization-based malware classification using transfer and ensemble learning. Computers, Materials & Continua, 75(2).
Das, S., Garg, A., & Kumar, S. (2024). Stacking ensemble-based approach for malware detection. SN Computer Science, 5(1), 185.
Demirkıran, F., Çayır, A., Ünal, U., & Dağ, H. (2022). An ensemble of pre-trained transformer models for imbalanced multiclass malware classification. Computers & Security, 121, 102846.
Dhalaria, M., & Gandotra, E. (2024). MalDetect: A classifier fusion approach for detection of Android malware. Expert Systems with Applications, 235, 121155.
El-Ghamry, A., Gaber, T., Mohammed, K. K., & Hassanien, A. E. (2023). Optimized and efficient image-based IoT malware detection method. Electronics, 12(3), 708.
Eslavath, R., & Mummadi, U. K. (2024). ENSIC: Feature selection on Android malware detection attributes using an enhanced nonlinear SVM integrated with cross validator. International Journal of Intelligent Systems and Applications in Engineering, 12(2), 495–504.
Falana, O. J., Sodiya, A. S., Onashoga, S. A., & Badmus, B. S. (2022). Mal-Detect: An intelligent visualization approach for malware detection. Journal of King Saud University – Computer and Information Sciences, 34(5), 1968–1983.
Gao, X., Hu, C., Shan, C., Liu, B., Niu, Z., & Xie, H. (2020). Malware classification for the cloud via semi-supervised transfer learning. Journal of Information Security and Applications, 55, 102661.
Hemalatha, J., Roseline, S. A., Geetha, S., Kadry, S., & Damaševičius, R. (2021). An efficient DenseNet-based deep learning model for malware detection. Entropy, 23(3), 344.
Hosseini, S., Nezhad, A. E., & Seilani, H. (2021). Android malware classification using convolutional neural network and LSTM. Journal of Computer Virology and Hacking Techniques, 17(4), 307–318.
Jiang, J., & Zhang, Y. (2023). A pyramid stripe pooling-based convolutional neural network for malware detection and classification. Journal of Ambient Intelligence and Humanized Computing, 14(3), 2785–2796.
Kim, J., Paik, J. Y., & Cho, E. S. (2023). Attention-based cross-modal CNN using non-disassembled files for malware classification. IEEE Access, 11, 22889–22903.
Kumar, S. (2021). MCFT-CNN: Malware classification with fine-tune convolution neural networks using traditional and transfer learning in Internet of Things. Future Generation Computer Systems, 125, 334–351.
Kumar, S., Janet, B., & Neelakantan, S. (2024). IMCNN: Intelligent malware classification using deep convolution neural networks as transfer learning and ensemble learning in honeypot-enabled organizational network. Computer Communications.
Liu, Y., Yang, P., Jia, P., He, Z., & Luo, H. (2022). MalFuzz: Coverage-guided fuzzing on deep learning-based malware classification model. PLOS ONE, 17(9), e0273804.
Maray, M., Alqahtani, H., Alotaibi, S. S., Alrayes, F. S., Alshuqayran, N., Alnfiai, M. M., & Al Duhayyim, M. (2023). Optimal bottleneck-driven deep belief network enabled malware classification on IoT-cloud environment. Computers, Materials & Continua, 74(2).
Nikam, U., & Deshmukh, V. M. (2024). Hybrid feature selection technique to classify malicious applications using machine learning approach. Journal of Integrated Science and Technology, 12(1), 702–702.
Othman, H., AlHija, M. A., & Alsharaiah, M. A. (2024). Toward enhancing malware detection using practical swarm optimization in honeypot. International Journal of Intelligent Engineering & Systems, 17(1).*
Palma Salas, M. I., De Geus, P., & Botacin, M. (2023, October). Enhancing malware family classification in the Microsoft challenge dataset via transfer learning. In Proceedings of the 12th Latin-American Symposium on Dependable and Secure Computing (pp. 156–163).
Palma, C., Ferreira, A., & Figueiredo, M. (2024). Explainable machine learning for malware detection on Android applications. Information, 15(1), 25.
Parihar, A. S., Kumar, S., & Khosla, S. (2022). S-DCNN: Stacked deep convolutional neural networks for malware classification. Multimedia Tools and Applications, 81(21), 30997–31015.
Ravi, V., & Alazab, M. (2023). Attention-based convolutional neural network deep learning approach for robust malware classification. Computational Intelligence, 39(1), 145–168.
Sharma, O., Sharma, A., & Kalia, A. (2024). MIGAN: GAN for facilitating malware image synthesis with improved malware classification on novel dataset. Expert Systems with Applications, 241, 122678.
Son, T. T., Lee, C., Le-Minh, H., Aslam, N., & Dat, V. C. (2022). An enhancement for image-based malware classification using machine learning with low dimension normalized input images. Journal of Information Security and Applications, 69, 103308.
Sun, J., Luo, X., Wang, W., Gao, Y., & Zhao, W. (2023). Robust malware identification via deep temporal convolutional network with symmetric cross entropy learning. IET Software, 17(4), 392–404.
Tang, Y., Qi, X., Jing, J., Liu, C., & Dong, W. (2023). BHMDC: A byte and hex n‑gram‑based malware detection and classification method. Computers & Security, 128, 103118.
Tekerek, A., & Yapici, M. M. (2022). A novel malware classification and augmentation model based on convolutional neural network. Computers & Security, 112, 102515.
Yadav, B., & Tokekar, S. (2023). Malware multi-class classification based on malware visualization using a convolutional neural network model. International Journal of Information Engineering and Electronic Business, 13(2), 20.
Zhan, D., Hu, Y., Li, W., Chen, J., Guo, S., & Pan, Z. (2023). Towards robust CNN‑based malware classifiers using adversarial examples generated based on two saliency similarities. Neural Computing and Applications, 1–18.
Zhang, D., Wu, X., He, E., Guo, X., Yang, X., Li, R., & Li, H. (2023). Android malware detection based on hypergraph neural networks. Applied Sciences, 13(23), 12629.
Zou, B., Cao, C., Wang, L., Fu, S., Qiao, T., & Sun, J. (2024). FACILE: A capsule network with fewer capsules and richer hierarchical information for malware image classification. Computers & Security, 137, 103606.
Journal of Information Technology Management
Volume 18, Issue 3
2026
Pages 1-24

Files

Share

How to cite

Statistics