Matching expectations and reality in AI systems - cybersecurity use case

Abstract

Artificial intelligence (AI) is a growing field in computer science which develops intelligent systems capable of performing things that a human mind can do. The manufacturers of security systems integrate AI capabilities into their systems for threat hunting, and market them with an emphasis on AI used to provide security features. This study evaluates the expectations of marketed AI features with reality in a use case of a cybersecurity system. To this end, we evaluated a system in a real-live environment with huge amount of data sent to it for analysis. Our evaluation demonstrates that, first, the virtual security analyst feature provided by the system cannot replace a human security analyst as it can only perform 3 amongst the 8 tasks of a human security analyst. Secondly, marketing claims exaggerate regarding the features provided by AI in the system.