On (Complex) Event Processing

I'm sure this has been said before, but I had cause to think again today on what is actual "complex" about CEP. I've never been particularly comfortable with the term CEP (of course, my personal comfort is not required ...) as it suggests that CEP is in some ways "hard". Well, those of us in the business of building CEP technologies know that the challenges of processing 10s of 000s of events per second with sub-millis latency against 000s of event patterns with temporal and logical constraints introduces a shed-load of complexity under the hood - but that's the point really - it is (or should be) all under the hood; we don't call databases "tricky databases" just because they have some fancy background indexing and schema evolution capabilities inside them.

Anyway, that ship has sailed - and I digress. The thing that got me thinking about this (again) was the recent press release regarding our launch - with our partner Detica - of a Solution Accelerator for Market Surveillance. That press release gives some examples of the kinds of surveillance strategies - "front running", "washing" ec. - that the Solution Accelerator provides (along with a handy definition of what this jargon actually means). The point is that the event processing logic required for these types of applications is not in any way "complex" - e.g. "detect a spike in trading volume or rapid price move within 10 minutes prior to a news release regarding a particular instrument" - despite the machinations under the hood that might be required to do this in real-time for all trades and news articles for all listed instruments on an exchange.

No, what is key in supporting these types of applications or strategies is the *lack* of complexity - and the provision of tools allowing strategy builders - here, operational teams at the exchange - to quickly express the business logic, generate a Dashboard to visualise the alerts it generates, and get all this into production before it becomes irrelevant. As markets and traders become ever more sophisticated it is the ease with which strategies can be modified and new strategies deployed that determines whether CEP is the right technology or not, not how clever it might be under the hood.

The term CEP seems to be here to stay, and I'm certainly not volunteering to be the flag carrier for a terminology battle ("Event Processing"? Anyone?). But let's be clear that neither the use case nor the hoops that need to be jumped through to deploy it need be complex for "CEP" to be an effective technology solution.

The key to effective CEP technology is to keep as much of the complexity as possible away from the people who have to use it.

Comments

[Opher Etzion]

There is a common confusion about the term CEP -- in this Blog posting you have referred to it as "complex processing of events" and tried to understand where the complexity is -- while the original meaning is "processing of complex events" - where complex event is an event that consists of possibly multiple events - the event is complex and not the processing. I admit that many people are having this confusion, and this is certainly a shortcoming of the CEP term.

cheers,

Opher

[Richard Bentley]

I understand the distinction but I'm not convinced by it; events do not need to be "complex" - and most often aren't. In the Capital Markets space I am most familiar with the event (expressions) you are interested in are often trivial - what complexity that does exist is usually found in:

a) the domain knowledge needed to define the expressions (or the specific parameters for them) - and what to do when you see them

b) the technology which allows 000s of these patterns to be evaluated in real-time against fast moving market data

Wherever you put the parentheses, I find the term "complex event processing" something of a misnomer.

[Opher Etzion]

I think that "misnomer" is not a bad term to describe CEP, I find other terms like "real-time" also as a misnomers - but the trouble is that sometimes they catch in industry. Personally I prefer EP without the "C" for the general set of capabalities, where I see "CEP" as a subset of that - as you mentioned in many applications there is very little need for "complex events" processing, while in others it is more purvasive. Satisfying non-functional requirements like: sclability, throughput, latency etc - requires an appropriate infrastructure, but this has nothing to do with the "complexity" either of events or of processing.

cheers,

Opher