SECURE CLOUD-NATIVE DATA ENGINEERING FOR NEXT-GENERATION DIGITAL ENTERPRISES
Main Article Content
Abstract
The rapid proliferation of cloud-native architectures, microservices, containerisation, and distributed data pipelines has fundamentally transformed how modern digital enterprises design, deploy, and govern their data infrastructure. As organisations migrate mission-critical workloads to multi-cloud and hybrid-cloud environments, the intersection of data engineering and cybersecurity has emerged as a strategic imperative. This research paper presents a comprehensive examination of secure cloud-native data engineering, systematically analysing how contemporary security frameworks—including zero-trust architecture, encrypted stream processing, DevSecOps pipelines, and machine learning-driven threat detection—are being integrated into next-generation data engineering ecosystems. Through a rigorous mixed-methods approach encompassing systematic literature synthesis, quantitative performance benchmarking, and four empirical case studies spanning financial technology, healthcare, retail, and manufacturing, this study demonstrates that organisations adopting mature cloud-native security data engineering practices achieve security incident reductions of 25–32%, improve data pipeline availability by 18–29%, and reduce mean time to detect (MTTD) cybersecurity threats by up to 67%. The paper further examines persistent challenges including shared responsibility ambiguity, data sovereignty conflicts, multi-cloud governance complexity, and the computational overhead of in-flight encryption. A forward-looking framework for AI-augmented cloud security, data mesh governance, and quantum-resilient encryption is proposed. The findings underscore the critical need for integrated, policy-driven, and developer-centric security frameworks that treat data protection not as an afterthought but as a foundational pillar of cloud-native data engineering.
Downloads
Article Details
Section
References
1. Armbrust, M., Fox, A., Griffith, R., & Joseph, A. D. (2010). A view of cloud computing. Communications of the ACM, 53(4), 50–58.
2. Burns, B., Grant, B., Oppenheimer, D., Brewer, E., & Wilkes, J. (2016). Borg, Omega, and Kubernetes: Lessons learned from three container-management systems over a decade. ACM Queue, 14(1), 70–93.
3. Cloud Security Alliance. (2023). Cloud threats and vulnerabilities report 2023. CSA.
4. Dehghani, Z. (2022). Data mesh: Delivering data-driven value at scale. O'Reilly Media.
5. Dwork, C., & Roth, A. (2014). The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3–4), 211–407.
6. Goodarzi, F., & Karimpour, J. (2022). Security challenges in cloud-native microservices: A systematic review. Journal of Cloud Computing, 11(1), 1–29.
7. IBM Security. (2023). Cost of a data breach report 2023. IBM Corporation.
8. International Organization for Standardization. (2022). ISO/IEC 27017:2015 – Code of practice for information security controls based on ISO/IEC 27002 for cloud services. ISO.
9. Kindervag, J. (2010). No more chewy centers: Introducing the zero trust model of information security. Forrester Research.
10. Konda, S. R., & Kumar, P. (2023). Federated learning for privacy-preserving cloud data analytics. IEEE Transactions on Cloud Computing, 11(3), 1524–1538.
11. Li, J., Chen, X., Li, M., & Li, J. (2021). Crowdsourcing in cloud-native security: A review of collaborative threat intelligence platforms. ACM Computing Surveys, 54(6), 1–35.
12. National Institute of Standards and Technology. (2020). NIST special publication 800-207: Zero trust architecture. U.S. Department of Commerce.
13. Newman, S. (2021). Building microservices: Designing fine-grained systems (2nd ed.). O'Reilly Media.
14. Rountree, D., & Castrillo, I. (2020). The basics of cloud computing: Understanding the fundamentals of cloud computing in theory and practice. Elsevier.
15. Sharma, A., Singh, R., & Gupta, M. (2022). DevSecOps in cloud-native pipelines: A systematic analysis of security automation practices. Computers and Security, 118, 102748.
16. Singh, P., & Agrawal, R. (2023). Machine learning approaches for cloud intrusion detection: A comparative study. Journal of Information Security and Applications, 72, 103397.
17. Verizon. (2023). Data breach investigations report 2023. Verizon Communications.
18. Wang, Q., Li, J., & Zhao, Y. (2022). Encrypted stream processing for real-time financial data analytics: Performance and security trade-offs. IEEE Access, 10, 42813–42827.
19. Janakiraman, A., & Ghoraani, B. (2025). An empirical comparison of text summarization: A multi-dimensional evaluation of large language models. arXiv preprint arXiv:2504.04534.
20. Janakiraman, A. (2025). AI Agents for Synthetic Data Generation in Finance: Enhancing Security, Privacy, and Predictive Analytics. In The Impact of Artificial Intelligence on Finance: Transforming Financial Technologies (pp. 33-51). Cham: Springer Nature Switzerland.
21. Zaharia, M., Chowdhury, M., Franklin, M. J., Shenker, S., & Stoica, I. (2012). Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation, 15–28.
22. Zhang, Y., & Patras, P. (2023). Adversarial robustness of machine learning-based network intrusion detection systems. IEEE Transactions on Network and Service Management, 20(1), 712–726.