Andi Novianto, Fatchul Arifin, Herman Dwi Surjono
Malware detection using deep learning has gained increasing attention due to its ability to generalize to previously unseen threats. However, the vulnerability of deep neural networks to adversarial perturbations poses a significant risk in real-world cybersecurity systems. This study investigates the robustness of convolutional neural network (CNN)-based malware detection models against white-box adversarial attacks, particularly the Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD). We propose a defense-enhanced architecture incorporating dropout, batch normalization, Gaussian noise injection, and adversarial training to improve model stability. Additionally, we apply gradient clipping and a cosine annealing learning rate scheduler to control the training dynamics better. Experiments are conducted on a labeled Windows Portable Executable (PE) file dataset, and robustness is evaluated based on accuracy degradation and the perturbation threshold required to cause misclassification. Results show that our defense strategies significantly improve resilience against adversarial attacks with minimal performance trade-offs on clean data. These findings highlight the importance of integrating defensive mechanisms into CNN-based malware classifiers to ensure reliability under adversarial conditions. © 2025 IEEE.
State University of Yogyakarta, Engineering Science, Yogyakarta, Indonesia
