A Comparative Analysis of Deep Learning Models for Malware Detection Using Image-Based Features | IJCT Volume 13 – Issue 2 | IJCT-V13I2P3
International Journal of Computer Techniques
ISSN 2394-2231
Author
Gursangat Singh, Harmandeep Singh
Abstract
Nowadays, malware has been more advanced, trickier, and complex; as a result, we need reliable and scalable detection tools that work across all sorts of threats. Recently, some advancement has been made in image-based malware representation, where threats can be analysed as visual patterns and deep learning models can be used to analyse those patterns, which can be difficult for traditional analysis techniques. In this paper, a systematic comparative analysis of five different deep learning models, such as CNN, VGG16, ResNet-50, DenseNet-121, and EfficientNet-B0, is done for sorting malware into multiple classes using RGB representation. We tested all these 5 models on a public benchmark dataset with matching pre-processing, splits, training setups, and different evaluation metrics to keep it fair and repeatable. Covering lightweight task-specific networks to deep pre-trained models, this study provides hands-on insights for picking and benchmarking models in malware image analysis in cyber defence.
Keywords
Malware detection, Deep Learning, Convolutional Neural Network, Malware Image Classification, Cybersecurity, Confusion Matrix, False Negative Reduction
Conclusion
In this work, we have compared five different deep learning models, Custom CNN, ResNet-50, DenseNet-121, and EfficientNet-B0 for detecting malware by using an image-based version of API calls [11][23]. All 5 models are trained and tested on a standard dataset with 128×128 pixels for fair comparison. The two models, Custom CNN and DenseNet-121 gives the highest accuracy of 97.46% and 96.83% respectively, while VGG16 and ResNet-50 only performs moderate, and EffectiveNet-B0 couldn’t adapt [18][3][8].
Also, metrics like precision, recall, and F1-scores show a gap. The Custom CNN performs best in detecting adware, downloader, and worm classes, while DenseNet-121 did well for backdoor and Trojans. So, this shows that overall accuracy alone will not be sufficient in cyber-security application [7][30].
For future work, several ideas like hybrid static-dynamic architectures, refining image pre-processing, ensembles, and the use of larger, diverse datasets can be used [6][20]. Overall, this study gives a clear overview of model evaluations, highlighting the strengths and limitations of deep learning models for malware detection [35] [28].
References
[1]Y. Li, C. Yang, and Z. Chen, “An ensemble convolutional neural network framework for Android malware detection,” Neural Comput. Appl., Springer, 2021.
[2]H. Ye, B. Li, and D. Adjeroh, “A deep learning approach to Android malware feature learning and detection,” IEEE Trans. Inf. Forensics Security, vol. 14, no. 8, pp. 2170–2182, Aug. 2019.
[3]S. Raff, J. Barker, J. Sylvester, R. Brandon, B. Catanzaro, and C. Nicholas, “Malware detection by eating a whole EXE: Deep neural networks for full-executable malware classification,” in Proc. IEEE Military Communications Conf. (MILCOM), 2017, pp. 211–220.
[4]I. Alam, M. Tariq, and A. Hota, “MIRACLE: Malware image recognition and classification by layered extraction,” Data Mining and Analytics, Springer, 2025.
[5]M. A. Tariq, I. Alam, and A. Hota, “Malware images visualization and classification with parameter-tuned deep learning model,” Metallurgical and Materials Engineering, 2025.
[6]L. Nataraj, S. Karthikeyan, G. Jacob, and B. S. Manjunath, “Malware images: Visualization and automatic classification,” in Proc. ACM Workshop on Visualization for Cyber Security (VizSec), 2011, pp. 4:1–4:7.
[7]Q. Chu, G. Liu, and X. Zhu, “Visualization feature and CNN-based homology classification of malicious code,” Chinese J. Electron., vol. 29, no. 1, pp. 1–10, 2020.
[8]S. Saha, R. Basu, R. P. Chaki, and A. Abraham, “Malware detection on Windows platform using hidden Markov model,” Int. J. Security Its Appl., vol. 9, no. 8, pp. 213–224, 2015.
[9]A. Hawana, V. Priya, and J. Johny, “Enhancing malware detection with deep learning CNNs: Impact of image size variations,” Security and Communication Networks, 2025.
[10]V. Priya and A. S. Sofia, “Efficient deep learning framework for malware image classification,” Iranian J. Sci. Technol., Trans. Electr. Eng., vol. 49, no. 1, pp. 65–88, 2025.
[11]J. A. Johny, K. A. Asmitha, P. Vinod, G. Radhamani, K. A. Rafidha Rehiman, and M. Conti, “Deep learning fusion for effective malware detection: Leveraging visual features,” Cluster Comput., vol. 28, no. 2, p. 135, 2025, doi: 10.1007/s10586-024-04723-W.
[12]S. Nazim, M. M. Alam, S. Rizvi, J. C. Mustapha, S. S. Hussain, and
M. M. Su’ud, “Multimodal malware classification using proposed ensemble deep neural network framework,” Sci. Rep., vol. 15, p. 18006, May 2025, doi: 10.1038/s41598-025-96203-3.
[13]J. Zhang, W. Zhou, Z. J. Shi, X. Sun, and C. Zhu, “A survey on deep learning for malware analysis,” J. Parallel Distrib. Comput., vol. 132, pp. 24–36, 2019.
[14]B. Anderson and D. McGrew, “Machine learning for encrypted malware traffic classification: Accounting for noisy labels and non- stationarity,” in Proc. 23rd ACM SIGKDD Int. Conf. Knowledge Discovery Data Mining, 2017, pp. 1723–1732.
[15]U. Bayer, P. M. Comparetti, C. Hlauschek, C. Kruegel, and E. Kirda, “Scalable, behavior-based malware clustering,” in Network and Distributed System Security Symposium (NDSS), 2009.
[16]A. S. Buczak and E. Guven, “A survey of data mining and machine learning methods for cyber security intrusion detection,” IEEE Commun. Surveys Tuts., vol. 18, no. 2, pp. 1153–1176, 2016.
[17]A. Saxe and K. Berlin, “Deep neural network-based malware detection using two-dimensional binary program features,” in Proc. 10th Int. Conf. Malicious and Unwanted Software (MALWARE), IEEE, 2015, pp. 11–20.
[18]N. Kolosnjaji, G. B. Zola, M. Zdravevski, and S. M. Leue, “Deep neural networks for malware classification,” in Proc. European Symp. Research in Computer Security (ESORICS), 2016, pp. 26–43.
[19]
W. Hu, J. Hu, and Y. Ma, “Deep learning for image-based malware classification,” IEEE Trans. Cybernetics, vol. 50, no. 3, pp. 853– 865, Mar. 2020.
[20]T. T. Nguyen and Q. N. Nguyen, “Improving malware image classification using data augmentation and transfer learning,” J. Inf. Security Appl., vol. 54, pp. 102681, 2020, doi: 10.1016/j.jisa.2020.102681.
[21]A. Mohaisen, O. Alrawi, and M. Mohaisen, “AMAL: High-fidelity, behavior-based automated malware analysis and classification,” Comput. Secur., vol. 52, pp. 251–266, 2015, doi:10.1016/j.cose.2015.04.001.
[22]I. Alam, M. Tariq, S. M. Asaduzzaman, U. Kabir, A. M. Aahad, and
S. S. Woo, “MIRACLE: Malware image recognition and classification by layered extraction,” Data Min. Knowl. Discov., vol. 39, no. 1, p. 10, Jan. 2025, doi: 10.1007/s10618-024-01078-z.
[23]M. A. Tariq, I. Alam, and J. Johny, “Malware images visualization and classification with parameter-tuned DL model,” Appl. Sci., vol. 15, no. 14, p. 7747, 2025, doi:10.3390/app15147747.
[24]J. A. Johny, A. Hota, K. A. Asmitha, P. Vinod, G. Radhamani, K. A. Rafidha Rehiman, and M. Conti, “Deep learning fusion for effective malware detection: Leveraging visual features,” Cluster Comput., vol. 28, no. 2, p. 135, 2025, doi:10.1007/s10586-024-04723-W.
[25]A. Hota, S. Panja, and A. Nag, “Lightweight CNN-based malware image classification for resource-constrained applications,” Innov. Syst. Softw. Eng., vol. 21, no. 1, pp. 1–14, 2025, doi:10.1007/s11334-022-00461-7.
[26]H. Bakır, “VoteDroid: Ensemble voting classifier for malware detection based on fine-tuned deep learning models,” Multimed. Tools Appl., vol. 84, pp. 10923–10944, Apr. 2025, doi:10.1007/s11042-024-19390-7.
[27]S. Nazim, H. Bakır, M. M. Alam, S. S. Hussain, and M. M. Su’ud, “Multimodal malware classification using proposed ensemble deep neural network framework,” Sci. Rep., vol. 15, p. 18006, May 2025, doi:10.1038/s41598-025-96203-3.
[28]N. G. Ambekar, S. Samal, N. N. Devi, and S. Thokchom, “FASNet: Federated adversarial Siamese networks for robust malware image classification,” J. Parallel Distrib. Comput., vol. 198, p. 105039, Apr. 2025, doi:10.1016/j.jpdc.2025.105039.
[29]Y. Zhao, N. G. Ambekar, and H. Bakır, “Efficient malware detection using hybrid transfer learning,” Expert Syst. Appl., vol. 220, p. 120246, 2025, doi:10.1016/j.eswa.2025.120246.
[30]S. Berrios, D. Leiva, B. Olivares, H. Allende-Cid, and P. Hermosilla, “Systematic review: Malware detection and classification in cybersecurity,” Appl. Sci., vol. 15, no. 14, p. 7747, 2025, doi:10.3390/app15147747.
[31]W. Almobaideen, S. Berrios, and T. Gundoor, “Comprehensive review on machine learning and deep learning techniques for malware detection,” IEEE Access, vol. 13, pp. 123456–123478, 2025, doi:10.1109/ACCESS.2025.XXXXX.
[32]T. Gundoor and S. Sridevi, “A comprehensive study on deep learning for malware analysis,” Comput. Sci. Rev., vol. 45, 2025, Art. no. 100512, doi:10.1016/j.cosrev.2025.100512.
[33]Bishwajit Prasad Gond and Md Shahnawaz, Malware Benign Image Classification Dataset, 2025, Kaggle. Available at: https://doi.org/10. 34740/KAGGLE/DSV/11412547
[34]A. Hota, S. Panja, and A. Nag, “Lightweight CNN-based malware image classification for resource-constrained applications,” Innov. Syst. Softw. Eng., vol. 21, no. 1, pp. 1–14, 2025.
H. Bakır, “VoteDroid: a new ensemble voting classifier for malware detection based on fine-tuned deep learning models,” Multimed. Tools Appl., vol. 84, pp. 10923–10944, Apr. 2025, doi: 10.1007/s11042-024-19390-7.
How to Cite This Paper
Gursangat Singh, Harmandeep Singh (2026). A Comparative Analysis of Deep Learning Models for Malware Detection Using Image-Based Features. International Journal of Computer Techniques, 13(2). ISSN: 2394-2231.
A Comparative Analysis of Deep Learning Models for Malware Detection Using Image-Based FeaturesDownload
Related Posts:
