Can Complex Event Processing be used for Cyber Threat Analysis?

Can Complex Event Processing (CEP) be used for cyber threat analysis? It sure can.

So what is Complex Event Processing (CEP)¹? It's a technique more so than a technology. At its base concept, CEP is a means for near real-time analysis of streams of data. Streams of data can be virtually anything: log data, financial data, call data, information system alerts, etc. While CEP is a concept, there are many vendors, commercial and open source, who implement the CEP concept. The basic features of what a CEP 'product' should strive for includes:

• Analyzing massive amounts of data at high rates of speed
• Integration with traditional database systems
• A query language which allows for the creation of expressive and complex analytical statements
• Pattern detection
• Time series analysis

You'll notice one of the bullet points describes a query language. Most, if not all CEP engines are implemented so that the queries against the streams of data are all done in memory. This allows for high performance throughput. It may sound like CEP engines are databases, but they are not. The best way I've heard the CEP concept compared to relational databases is as follows: databases execute queries against a one or more tables of data. CEP engines stream data through thousands of queries. And the streamed data can match multiple queries or none at all. It's like a database turned on its head.

Esper is a popular open source CEP implementation. The figure below shows a basic architecture based on Esper.

Esper is a framework and is implemented in 100% pure Java (it can be embedded in any Java application). You build your application using the framework. Let's look at quick example²:

select * from pattern [every LoginFailure -> (timer:interval(10 sec) having count(*) >= 5 and LoginSuccess)]

This is a query written in the Esper Query Language (EQL). As you can see it is similar to traditional database SQL, but there are some differences. I'm not going to go into detail about the query (you can read the documentation on the Esper Web site). The pseudo-code for the above query can be written as followed:

If 5 or more LoginFailures occur in 10 seconds followed by a LoginSuccess then alert.

What we have done is created a query to detect a possible brute-force login attempt. This examples shows the following features of CEP:

• Stateful behavior
• Pattern detection
• Windowing for detection over a time frame
• Threshold detection

This is actually a very basic example of what you can do with a CEP framework like Esper. Much more complex pattern and stateful queries can be crafted to detect all kinds of threats.

Alright, so what about the DoD? Well, in doing some research for this post I came across something called Complex Event Detection, Analysis and Ranking (CEDAR)³. It appears to be a Navy funded project. The Abstract for project CEDAR is

Complex Event Discovery, Analysis, and Ranking (CEDAR) leverages mature complex event processing (CEP) technology to automatically monitor large streams of intelligence, surveillance, and reconnaissance (ISR) data for combinations of activities indicative of threats. Developing proof-of-concept demonstration with operationally relevant data.

The following high-level diagram is also part of the abstract.

The company which built CEDAR was at the 2013 Vendor Acquisition Showcase (VAS)  for the US Navy. You should follow the link in the References section if you want to learn more about it.

What is the future of CEP? Today companies are combing the analytic results they are gaining from big data (data mining large sets of data in Hadoop, etc.) to discover behavioral patterns. These patterns can be fed right back into a CEP system for real time detection and alerting. IT and security operations are also using machine learning to "learn" from the behavior of their operators. Once a certain resolution is learned from a person or set of people, this can be generalized and converted into queries which can be fed back into the CEP system so you no longer have to have a human deal with that situation again.

References
Image: http://www.espertech.com/products/esper.php
¹  http://en.wikipedia.org/wiki/Complex_event_processing
²   http://store.elsevier.com/Logging-and-Log-Management/Anton-Chuvakin/isbn-9781597496353/
³  http://www.navyopportunityforum.com/mobile/vas_popup.php?abstract_num=147