Exploiting Vision Transformer and Ensemble Learning for Advanced Malware Classification

ABSTRACT Malware remains a significant concern for modern digital systems, increasing the need for reliable and scalable detection methods. This work proposes an ensemble method that combines a random forest (RF) with a vision transformer (ViT). The approach exploits complementary feature spaces, including bag‐of‐words (BoW) and image representations, to enhance multi‐class malware classification. We also evaluate traditional machine learning models (Naïve Bayes, Support Vector Machine, and RF) and deep learning (DL) models (ResNet50 and ViT) using the Microsoft Malware and Dike datasets. The proposed ensemble model achieves 99.32% accuracy and 98.11% F1 score on the Malware dataset, outperforming individual models and recent state‐of‐the‐art studies. While ViT captures spatial and sequence dependencies via attention mechanisms, RF captures textual and byte‐level frequency patterns. Their combination, through a product rule, enhances robustness and reliability in multi‐class cybersecurity tasks.