Anomaly detection in kernel-level process events using machine learning-based context analysis

Abstract

The limitation of the use of input-validation approach in anomaly detection at the application layer is that a malicious software like Stuxnet worm can successfully return looped expected values to a monitoring application while executing the attack. To overcome these application layer-based anomaly detector limitations, we focus on anomaly detection at the kernel layer. Anomaly detection using kernel-level traces has unique advantages in detecting security threats early, but challenges associated with understanding the patterns for online and offline monitoring are enormous. However, the problem of the intricate pattern of the kernel-level events can be solved with effective machine learning approaches but requires a deep understanding of the data in the application domain. In this thesis, we design and implement machine learning frameworks based on deep learning (DL) and clustering that capture the context of a process in both inter- process and intra-process interactions via kernel-level event profile analysis. Since the context is learned from the kernel-level events, we provide cybersecurity solutions without compromising the privacy of the software applications because there is no one-to-one mapping between the kernel events and the source code of the process. During operation, we label patterns deviating from the benign context as malicious and we can kill or restart the process when we detect that it has been compromised. Software applications have a high degree of reliability. Therefore, data collected from these applications to create machine learning models is not balanced and introduces a bias in the machine learning model. We solve this challenge with a novel generative adversarial network (GAN)-based oversampling technique that inherently removes noise and outliers from the data without the use of the computationally expensive strategy of input sample inversion. We test the proposed frameworks with several publicly available benchmark anomaly datasets of Unmanned Aerial Vehicle (UAV), network logs, and images with varying profiles that impact the order, distribution, and execution time contexts of the applications. In all the test cases, the results of the frameworks in this thesis show a 3% - 13% improvement in the Precision, Accuracy, and Recall over the benchmark approaches used for comparison.