STAP Journal of Security Risk Management

ISSN: 3080-9444 (Online)

A Multi-Layered Adaptive Cybersecurity Framework for the Banking Sector Integrating Next-Gen Firewalls with AI-Driven IDPS

by 

Sokroeurn Ang ;

Mony Ho ;

Sopheatra Huy ;

Midhunchakkaravarthy Janarthanan

PDF logoPDF

Published: 2026

Abstract

The accelerated digital transformation of the banking sector has enhanced the delivery of financial services but simultaneously expanded the cyberattack surface, exposing institutions to advanced persistent threat (APT), zero-day exploit, and obfuscated malware. Conventional perimeter defenses, primarily Layer 3 and 4 firewalls and signature-based intrusion detection systems (IDS), offer insufficient protection against encrypted, evasive, and previously unknown cyberattacks, and frequently generate high false-positive rates that burden Security Operations Center (SOC). This study proposes a multilayered adaptive cybersecurity framework that integrates Layer 7 Next Generation Firewall (NGFW), hybrid Network and Host-based Intrusion Detection and Prevention System (NIDPS/HIDPS), and an AI-driven analysis engine. The framework employs a dual-stage detection architecture, combining Convolutional Neural Network (CNN) for spatial representation learning and Random Forest (RF) classifiers for anomaly decisioning. The model was evaluated using a strategically consolidated dataset derived from CIC-IDS-2017 and UNSW-NB15, specifically isolating cyberattack vectors prevalent in financial infrastructures (e.g., SQL Injection, DDoS, and Brute Force). The model achieves 99.65% detection accuracy and a reduced false-positive rate of 0.35%, significantly outperforming classical SVM and standalone signature-based systems. The results demonstrate that the proposed architecture aligned with NIST and PCI-DSS standard as well as defense-in-depth mechanism suitable for real-time, high-frequency financial environments.

Keywords

Cybersecurity in BankingMulti layered security frameworkAI-driven IDPSNext-Generation Firewall (NGFW)Anomaly detectionZero-day attacksDeep Learning

A Multi-Layered Adaptive Cybersecurity Framework for the Banking Sector Integrating Next-Gen Firewalls with AI-Driven IDPS is licensed under CC BY 4.0CCBY

References

  1. A. S. Dina and Manivannan, D. (2021). Intrusion detection based on machine learning techniques in computer networks. Internet of Things, 16, 100462. https://doi.org/10.1016/j.iot.2021.100462
  2. Waleed, A., Jamali, A. F., & Masood, A. (2022). Which open-source IDS? Snort, Suricata or Zeek. Computer Networks, 213, 109116. https://doi.org/10.1016/j.comnet.2022.109116
  3. Heino, J., Hakkala, A., & Virtanen, S. (2022). Study of methods for endpoint aware inspection in a next generation firewall. Cybersecurity, 5, 25. https://doi.org/10.1186/s42400-022-00127-8
  4. Singh, A. P., & Singh, M. D. (2014). Analysis of host-based and network-based intrusion detection system. International Journal of Computer Network and Information Security, 6(8), 41–47. https://doi.org/10.5815/ijcnis.2014.08.06
  5. Scarfone, K., & Mell, P. (2007). Guide to intrusion detection and prevention systems (IDPS) (NIST Special Publication 800-94). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-94
  6. Otoum, Y., & Nayak, A. (2021). AS-IDS: Anomaly and signature-based IDS for the Internet of Things. Journal of Network and Systems Management, 29, 28. https://doi.org/10.1007/s10922-021-09589-6
  7. Scarfone, K., & Hoffman, P. (2009). Guidelines on firewalls and firewall policy (NIST Special Publication 800-41 Rev. 1). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-41r1
  8. Patel, M., Amritha, P. P., Sudheer, V. B., & Sethumadhavan, M. (2024). DDoS attack detection model using machine learning algorithm in next generation firewall. Procedia Computer Science, 233, 175–183. https://doi.org/10.1016/j.procs.2024.03.207
  9. Sichkar, M., & Pavlova, L. (2023). A short survey of the capabilities of Next Generation firewalls. Computer Science and Cybersecurity, 1, 28–33. https://doi.org/10.26565/2519-2310-2023-1-02
  10. Liang, J., & Kim, Y. (2022). Evolution of firewalls: Toward securer network using next generation firewall. In 2022 IEEE 12th Annual Computing and Communication Workshop and Conference (CCWC) (pp. 752–759). https://doi.org/10.1109/CCWC54503.2022.9720435
  11. Lee, J.-K., Hong, T., & Lee, G. (2024). AI-based approach to firewall rule refinement on high-performance computing service network. Applied Sciences, 14(11), 4373. https://doi.org/10.3390/app14114373
  12. Jemili, F., Zaghdoud, M., & Ben Ahmed, A. (2018). Intelligent firewall based on machine learning: A survey. In 2018 International Conference on Smart Communications in Network Technologies (SaCoNeT) (pp. 136–141). https://doi.org/10.1109/SaCoNeT.2018.8585424
  13. Gordon, A. (Ed.). (2015). Official (ISC)² Guide to the CISSP CBK (4th ed.). Auerbach Publications. https://doi.org/10.1201/b18257
  14. Sharafaldin, I., Lashkari, A. H., & Ghorbani, A. A. (2018). Toward generating a new intrusion detection dataset and intrusion traffic characterization. In Proceedings of the 4th International Conference on Information Systems Security and Privacy (ICISSP) (pp. 108–116). https://doi.org/10.5220/0006639801080116
  15. Seo, W., & Pak, W. (2021). Real-time network intrusion prevention system based on hybrid machine learning. IEEE Access, 9, 46387–46393. https://doi.org/10.1109/ACCESS.2021.3066620
  16. Ring, M., Wunderlich, S., Scheuring, D., Landes, D., & Hotho, A. (2019). A survey of network based intrusion detection data sets. Computers & Security, 87, 101600. https://doi.org/10.1016/j.cose.2019.06.005
  17. Ahmad, Z., Khan, A. S., Shiang, C. W., Abdullah, J., & Ahmad, F. (2021). Network intrusion detection system: A systematic study of machine learning and deep learning approaches. Transactions on Emerging Telecommunications Technologies, 32(1), e4150. https://doi.org/10.1002/ett.4150
  18. Vanin, P., et al. (2022). A study of network intrusion detection systems using artificial intelligence/machine learning. Applied Sciences, 21, 111752. https://doi.org/10.3390/app122211752
  19. Moustafa, N., & Slay, J. (2015). UNSW-NB15: A comprehensive data set for network intrusion detection systems. In 2015 Military Communications and Information Systems Conference (MilCIS) (pp. 1–6). https://doi.org/10.1109/MilCIS.2015.7348942
  20. Chung, C.-J., Khatkar, P., Xing, T., Lee, J., & Huang, D. (2013). NICE: Network intrusion detection and countermeasure selection in virtual network systems. IEEE Transactions on Dependable and Secure Computing, 10(4), 200–210. https://doi.org/10.1109/TDSC.2013.8
  21. Kumar, S., Gupta, S., & Arora, S. (2021). Research trends in network-based intrusion detection systems: A review. IEEE Access, 9, 157761–157774. https://doi.org/10.1109/ACCESS.2021.3129775
  22. Samrin, R., & Vasumathi, D. (2017). Review on anomaly-based network intrusion detection system. In 2017 International Conference on Electrical, Electronics, Communication, Computer and Optimization Techniques (ICEECCOT) (pp. 142–145). https://doi.org/10.1109/ICEECCOT.2017.8284655
  23. Ferrag, M. A., Maglaras, L., Moschoyiannis, S., & Janicke, H. (2020). Deep learning for cyber security intrusion detection: Approaches, datasets, and comparative study. Journal of Information Security and Applications, 50, 102419. https://doi.org/10.1016/j.jisa.2019.102419
  24. Satilmiş, H., Akleylek, S., & Tok, Z. Y. (2024). A systematic literature review on host-based intrusion detection systems. IEEE Access, 12, 27237–27266. https://doi.org/10.1109/ACCESS.2024.3367004
  25. Omitola, A. O., et al. (2022). A comprehensive review of the state-of-the-art in host-based intrusion detection systems. IEEE Access, 10, 116281–116306. https://doi.org/10.1109/ACCESS.2022.3218702
  26. Nallakaruppan, M. K., et al. (2024). Enhancing security of host-based intrusion detection systems for the Internet of Things. IEEE Access, 12, 31788–31797. https://doi.org/10.1109/ACCESS.2024.3355794
  27. Moskovitch, R., et al. (2007). Host based intrusion detection using machine learning. In 2007 IEEE Intelligence and Security Informatics (pp. 107–114). https://doi.org/10.1109/ISI.2007.379542
  28. Idrissi, S. E., Bouhadi, A. E., & Habbani, A. (2023). Performance analysis of machine learning algorithms for intrusion detection system using CICIDS2017 dataset. International Journal of Information Security, 22, 1365–1377. https://doi.org/10.1007/s10207-023-00683-x
  29. Panigrahi, R., & Borah, S. (2021). A consolidated decision tree-based intrusion detection system for binary and multiclass attacks. Mathematics, 7, 751. https://doi.org/10.3390/math9070751
  30. Kwon, H.-Y., Kim, T., & Lee, M.-K. (2022). Advanced intrusion detection combining signature based and behavior-based detection methods. Electronics, 6, 867. https://doi.org/10.3390/electronics11060867
  31. Garcia, C. F. J., & Blandon, T. E. G. A. (2022). A deep learning-based intrusion detection and prevention system for detecting and preventing denial-of-service attacks. IEEE Access, 10, 83044–83055. https://doi.org/10.1109/ACCESS.2022.3196642
  32. Sajid, M., et al. (2024). Enhancing intrusion detection: A hybrid machine and deep learning approach. Journal of Cloud Computing, 13, 6. https://doi.org/10.1186/s13677-024-00685-x
  33. Pinto, A., Herrera, L.-C., Donoso, Y., & Gutierrez, J. A. (2023). Survey on intrusion detection systems based on machine learning techniques for the protection of critical infrastructure. Sensors, 5, 2415. https://doi.org/10.3390/s23052415
  34. Aburomman, A. A., & Reaz, I. B. M. (2016). Review of IDS development methods in machine learning. International Journal of Electrical and Computer Engineering, 6(6), 2432–2434. https://doi.org/10.11591/ijece.v6i6.12478
  35. Andresini, G., Appice, A., & Malerba, D. (2022). A deep learning-based approach for intrusion detection in encrypted traffic. Neurocomputing, 486, 145–154. https://doi.org/10.1016/j.neucom.2022.02.046
  36. Thakkar, A., & Lohiya, R. (2022). A survey on intrusion detection system: Feature selection, model, performance measures, application perspective, challenges, and future research directions. Artificial Intelligence Review, 55, 453–563. https://doi.org/10.1007/s10462-021-10037-9
  37. Moustafa, N., Creech, G., & Slay, J. (2018). A new big data analytics framework for flow-based network intrusion detection systems. PLOS ONE, 5, e0196810. https://doi.org/10.1371/journal.pone.0196810
  38. Kikissagbe, B. R., & Adda, M. (2024). Machine learning-based intrusion detection methods in IoT systems: A comprehensive review. Electronics, 18, 3601. https://doi.org/10.3390/electronics13183601
  39. Jayalaxmi, P. L. S., et al. (2022). Machine and deep learning solutions for intrusion detection and prevention in IoTs: A survey. IEEE Access, 10, 121173–121192. https://doi.org/10.1109/ACCESS.2022.3220622
  40. Sarker, I. H. (2021). Machine learning: Algorithms, real-world applications and research directions. SN Computer Science, 2, 160. https://doi.org/10.1007/s42979-021-00592-x
  41. Yadav, S., & Saxena, M. (2020). Reducing false positives in intrusion detection systems using hybrid machine learning algorithms. Future Generation Computer Systems, 107, 107–115. https://doi.org/10.1016/j.future.2019.12.033
  42. Ahmad, A., & Khan, M. K. (2019). Machine learning-based IDS for reducing false positives in network security. Computers & Security, 87, 101557. https://doi.org/10.1016/j.cose.2019.101557
  43. Wang, F., & Yao, L. (2020). A deep learning approach for intrusion detection system with reduced false positives. Journal of Network and Computer Applications, 155, 102530. https://doi.org/10.1016/j.jnca.2020.102530
  44. Sowmya, T., & Mary Anita, E. A. (2023). A comprehensive review of AI-based intrusion detection system. Measurement: Sensors, 28, 100827. https://doi.org/10.1016/j.measen.2023.100827
  45. Rose, S., Borchert, O., Mitchell, S., & Connelly, S. (2020). Zero Trust Architecture (NIST Special Publication 800-207). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-207
  46. Al-Garadi, M. A., et al. (2020). A survey of machine and deep learning methods for Internet of Things (IoT) security. IEEE Communications Surveys & Tutorials, 22(3), 1646–1685. https://doi.org/10.1109/COMST.2020.2988293
  47. Islam, S. R., Eberle, W., Ghafoor, S. K., & Ahmed, A. A. (2023). Explainable Artificial Intelligence in cybersecurity: A comprehensive survey. IEEE Access, 11, 32152–32182. https://doi.org/10.1109/ACCESS.2023.3263521
  48. Kou, G., & Lu, Y. (2025). FinTech: A literature review of emerging financial technologies and applications. Financial Innovation, 11, 1. https://doi.org/10.1186/s40854-024-00668-6
  49. Rehman, M. H. U., et al. (2022). Trustable and explainable AI for FinTech: A survey. IEEE Access, 10, 100782–100806. https://doi.org/10.1109/ACCESS.2022.3208035
  50. National Institute of Standards and Technology (NIST). (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0) (NIST AI 100-1). https://doi.org/10.6028/NIST.AI.100-1