| | | |

## A Study on the Efficacy of Deep Reinforcement Learning for Intrusion Detection

#### Halim Görkem GÜLMEZ [1] , Pelin ANGIN [2]

The world has witnessed a fast-paced digital transformation in the past decade, giving rise to all-connected environments. While the increasingly widespread availability of networks has benefited many aspects of our lives, providing the necessary infrastructure for smart autonomous systems, it has also created a large cyber attack surface. This has made real-time network intrusion detection a significant component of any computerized system. With the advances in computer hardware architectures with fast, high-volume data processing capabilities and the developments in the field of artificial intelligence, deep learning has emerged as a significant aid for achieving accurate intrusion detection, especially for zero-day attacks. In this paper, we propose a deep reinforcement learning-based approach for network intrusion detection and demonstrate its efficacy using two publicly available intrusion detection datasets, namely NSL-KDD and UNSW-NB15. The experiment results suggest that deep reinforcement learning has significant potential to provide effective intrusion detection in the increasingly complex networks of the future.
security, deep reinforcement learning, intrusion detection
• V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra and M. Riedmiller, “Playing Atari with Deep Reinforcement Learning,” pp. 1–9, 2013. Retrieved from http://arxiv.org/abs/1312.5602
• M. Lai, “Giraffe: Using Deep Reinforcement Learning to Play Chess,” September, 2015. Retrieved from http://arxiv.org/abs/1509.01549
• L. Wang, D. Zhang, L. Gao, J. Song, L. Guo and H. T. Shen, “MathDQN: Solving arithmetic word problems via deep reinforcement learning,” 32nd AAAI Conference on Artificial Intelligence, pp. 5545–5552, 2018.
• S. Nemati, M. M. Ghassemi and G. D. Clifford, “Optimal medication dosing from suboptimal clinical examples: A deep reinforcement learning approach,” Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2016.
• Z. Zhou, X. Li and R. N. Zare, “Optimizing Chemical Reactions with Deep Reinforcement Learning,” ACS Central Science, vol. 3, no. 12, pp. 1337–1344, 2017.
• M. Mahmud, M. S. Kaiser, A. Hussain and S. Vassanelli, “Applications of Deep Learning and Reinforcement Learning to Biological Data,” IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 6, pp. 2063–2079, 2018.
• T. Yen, A. Oprea and K. Onarlioglu, "Beehive: large-scale log analysis for detecting suspicious activity in enterprise networks,". Proc. 29th Annual Computer Security Applications Conference, pp. 199–208, 2013.
• A. Razaq, H. Tianfield and P. Barrie, "A big data analytics based approach to anomaly detection," Proc. - 2016 IEEE/ACM 3rd International Conference on Big Data Computing Applications and Technologies (BDCAT), pp. 187–193, 2016.
• A. O. Balogun and R. G. Jimoh, "Anomaly intrusion detection using a hybrid of decision tree and K-nearest neighbor," Journal of Advances in Scientific Research & Applications (JASRA), vol. 2, no. 1, pp. 67-74, 2015.
• A. Hariharan, A. Gupta and T. Pal, "CAMLPAD: Cybersecurity Autonomous Machine Learning Platform for Anomaly Detection," Proc. Future of Information and Communication Conference (FICC), San Francisco, CA, USA, pp. 705-720, 2020.
• M. S. Pervez and D. M. Farid, “Feature selection and intrusion classification in NSL-KDD cup 99 dataset employing SVMs,” SKIMA 2014 - 8th International Conference on Software, Knowledge, Information Management and Applications, pp. 1–6, 2014.
• B. Ingre and A. Yadav, "Performance analysis of NSL-KDD dataset using ANN," 2015 International Conference on Signal Processing and Communication Engineering Systems, pp. 92-96, 2015.
• S.O.M. Kamel, N. Hegazi, H. Harb, A. ElDein and H. ElKader, "AdaBoost Ensemble Learning Technique for Optimal Feature Subset Selection," International Journal of Computer Networks and Communications Security vol. 4, no. 1, pp. 1–11, 2016.
• W. Hu, W. Hu, and S. Maybank, "AdaBoost-Based Algorithm for Network Intrusion Detection," IEEE Transactions on Systems, Man, and Cybernetics - Part B: Cybernetics, vol. 38, no. 2, pp. 577-583, 2008.
• A. H. Engly, A. R. Larsen, and W. Meng, "Evaluation of Anomaly-Based Intrusion Detection with Combined Imbalance Correction and Feature Selection," Proc. 14th International Conference on Network and System Security, Melbourne, Australia, pp. 277-291, 2020.
• N. Moustafa and J. Slay, "A hybrid feature selection for network intrusion detection systems: central points and association rules," arXiv:1707.05505, (2017) [cs.CR].
• J. Kim and H. Kim, “Applying Recurrent Neural Network to Intrusion Detection with Hessian Free Optimization,” In: Kim H., Choi D. (eds) Information Security Applications. WISA 2015. Lecture Notes in Computer Science, vol. 9503, 2016, Springer, Cham.
• Y. Chuan-long, Z. Yue-fei, F. Jin-long and H. Xin-zheng, “A Deep Learning Approach for Intrusion Detection using Recurrent Neural Networks,” IEEE Access, vol. 5, pp. 21954 - 2196, 2017.
• C. Yin, Y. Zhu, J. Fei, and X. He, ‘‘A deep learning approach for intrusion detection using recurrent neural networks,’’ IEEE Access, vol. 5, pp. 21954–21961, 2017.
• Z. Li, A. L. G. Rios, G. Xu, and L. Trajkovic, ‘‘Machine learning techniques for classifying network anomalies and intrusions,’’ in Proc. IEEE Int. Symp. Circuits Syst. (ISCAS), pp. 1–5, 2019.
• S. Behera, A. Pradhan, and R. Dash, “Deep Neural Network Architecture for Anomaly Based Intrusion Detection System,” 5th International Conference on Signal Processing and Integrated Networks (SPIN 2018), pp. 270– 274, 2018.
• Z. Li, Z. Qin, K. Huang, X. Yang, and S. Ye, ‘‘Intrusion detection using convolutional neural networks for representation learning,’’ in Proc. Int. Conf. Neural Inf. Process. pp. 858–866, 2017.
• M. Lopez-Martin, B. Carro, A. Sanchez-Esguevillas, and J. Lloret, ‘‘Conditional variational autoencoder for prediction and feature recovery applied to intrusion detection in IoT,’’ Sensors, vol. 17, no. 9, p. 1967, Aug. 2017.
• F. A. Khan, A. Gumaei, A. Derhab, and A. Hussain, ‘‘TSDL: A twostage deep learning model for efficient network intrusion detection,’’ IEEE Access, vol. 7, pp. 30373–30385, 2019.
• T. Ma, F. Wang, J. Cheng, Y. Yu, and X. Chen, ‘‘A hybrid spectral clustering and deep neural network ensemble algorithm for intrusion detection in sensor networks,’’ Sensors, vol. 16, no. 10, p. 1701, Oct. 2016.
• R. Vinayakumar, M. Alazab, K. P. Soman, P. Poornachandran, A. Al-Nemrat, and S. Venkatraman, ‘‘Deep learning approach for intelligent intrusion detection system,’’ IEEE Access, vol. 7, pp. 41525–41550, 2019.
• N. Gao, L. Gao, Q. Gao, and H. Wang, "An Intrusion Detection Model Based on Deep Belief Networks," Proc. 2nd International Conference on Advanced Cloud and Big Data, Huangshan, China, pp. 247-252, 2014.
• B. Deokar and A. Hazarnis, “Intrusion Detection System using Log Files and Reinforcement Learning,” International Journal of Computer Applications, vol. 45, no. 1919, pp. 28–35, 2012.
• A. Servin and D. Kudenko, “Multi-agent reinforcement learning for intrusion detection: A case study and evaluation,” Frontiers in Artificial Intelligence and Applications, vol. 178, pp. 873–874, 2008.
• R. Elderman, L. J. J. Pater, A. S. Thie, M. M. Drugan and M. A. Wiering, “Adversarial reinforcement learning in a cyber security simulation,” ICAART 2017- Proceedings of the 9th International Conference on Agents and Artificial Intelligence, pp. 559–566, 2017.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Cambridge, MA, USA: MIT Press, 2016.
• L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement Learning: A Survey,” Journal of Artificial Intelligence Research, vol. 4, 1996.
• N. Moustafa, J. Slay, "UNSW-NB15: A Comprehensive Data Set for Network i Intrusion Detection Systems (UNSW-NB15 Network Data Set)," Proceedings of the 2015 IEEE Military Communications and Information Systems Conference (MilCIS), pp. 1–6, 2015.
• KDD Cup 1999. Avaliable online: https://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html (Accessed on 20 November 2020).
• 1998 DARPA Intrusion Detection Evaluation Dataset. Available online: https://www.ll.mit.edu/r-d/datasets/1998-darpa-intrusion-detection-evaluation-dataset (Accessed on 20 November 2020).
• M. Tavallaee, E. Bagheri, W. Lu, and A.A. Ghorbani, “A detailed analysis of the KDD CUP 99 data set,” IEEE Symposium on Computational Intelligence for Security and Defense Applications, pp. 1–6, 2009.
 Bibtex @araştırma makalesi { saucis834048, journal = {Sakarya University Journal of Computer and Information Sciences}, issn = {}, eissn = {2636-8129}, address = {}, publisher = {Sakarya Üniversitesi}, year = {2021}, volume = {4}, pages = {11 - 25}, doi = {10.35377/saucis.04.01.834048}, title = {A Study on the Efficacy of Deep Reinforcement Learning for Intrusion Detection}, key = {cite}, author = {Gülmez, Halim Görkem and Angın, Pelin} } APA Gülmez, H , Angın, P . (2021). A Study on the Efficacy of Deep Reinforcement Learning for Intrusion Detection . Sakarya University Journal of Computer and Information Sciences , 4 (1) , 11-25 . DOI: 10.35377/saucis.04.01.834048 MLA Gülmez, H , Angın, P . "A Study on the Efficacy of Deep Reinforcement Learning for Intrusion Detection" . Sakarya University Journal of Computer and Information Sciences 4 (2021 ): 11-25 Chicago Gülmez, H , Angın, P . "A Study on the Efficacy of Deep Reinforcement Learning for Intrusion Detection". Sakarya University Journal of Computer and Information Sciences 4 (2021 ): 11-25 RIS TY - JOUR T1 - A Study on the Efficacy of Deep Reinforcement Learning for Intrusion Detection AU - Halim Görkem Gülmez , Pelin Angın Y1 - 2021 PY - 2021 N1 - doi: 10.35377/saucis.04.01.834048 DO - 10.35377/saucis.04.01.834048 T2 - Sakarya University Journal of Computer and Information Sciences JF - Journal JO - JOR SP - 11 EP - 25 VL - 4 IS - 1 SN - -2636-8129 M3 - doi: 10.35377/saucis.04.01.834048 UR - https://doi.org/10.35377/saucis.04.01.834048 Y2 - 2020 ER - EndNote %0 Sakarya University Journal of Computer and Information Sciences A Study on the Efficacy of Deep Reinforcement Learning for Intrusion Detection %A Halim Görkem Gülmez , Pelin Angın %T A Study on the Efficacy of Deep Reinforcement Learning for Intrusion Detection %D 2021 %J Sakarya University Journal of Computer and Information Sciences %P -2636-8129 %V 4 %N 1 %R doi: 10.35377/saucis.04.01.834048 %U 10.35377/saucis.04.01.834048 ISNAD Gülmez, Halim Görkem , Angın, Pelin . "A Study on the Efficacy of Deep Reinforcement Learning for Intrusion Detection". Sakarya University Journal of Computer and Information Sciences 4 / 1 (Nisan 2021): 11-25 . https://doi.org/10.35377/saucis.04.01.834048 AMA Gülmez H , Angın P . A Study on the Efficacy of Deep Reinforcement Learning for Intrusion Detection. SAUCIS. 2021; 4(1): 11-25. Vancouver Gülmez H , Angın P . A Study on the Efficacy of Deep Reinforcement Learning for Intrusion Detection. Sakarya University Journal of Computer and Information Sciences. 2021; 4(1): 11-25. IEEE H. Gülmez ve P. Angın , "A Study on the Efficacy of Deep Reinforcement Learning for Intrusion Detection", Sakarya University Journal of Computer and Information Sciences, c. 4, sayı. 1, ss. 11-25, Nis. 2021, doi:10.35377/saucis.04.01.834048