Event Processing blog

Complex Event Processing: Don't Get Caught Watching the Wrong Launch Pad

James Kobielus with Network World wrote a recent article about CEP called:  "Complex Event Processing:  Still on the Launching Pad."  The author seems to equate being "off the launching pad" to being adopted by business intelligence vendors. The article made me wonder why anyone would equate success of a new technology by it's adoption and usage in a style it's designed to improve.  It's like saying that the internet will be successful once you flop down on your couch and watch the evening news on a web browser.

CEP's strongest use is for computer-based automation; not human decision support.   CEP's success should not be measured by it's use as human decision support.  CEP often replaces manual human decision making - as a result, it empowers a new class of knowledge worker and application metaphor. 

Take algorithmic trading, for example.  Trading is a business "decision making" function that has been performed manually for years by MBA's with a flair for identifying opportunities and acting quickly.  Traders apply the highest-end "business intelligence" there is.  CEP is being used to automate trading because much of what these MBAs used to do can be automated.  The head of FX trading at a large investment bank I recently visited, for example, told me that in the past 4 years their trading volume has increased by over 10X;  at the same time, their trading staff has been reduced by 80%.  Mind you, traders are not going away. Their role is changing from the old days of relationship-based, click-based trading to more of mission-control responsibility, monitoring and optimizing algorithms as market conditions change.  Similarly, CEP is being used to automate fraud detection.  Instead of having loads of surveillance staff visually  watching for suspicious patterns of activity, a computer can monitor activity for fraudulent patterns of behavior, and alert staff when suspicious activity is detected - in real-time.

And CEP can help do these kinds of automated systems more quickly and at greater scale.  Even the best traders, for example, can only monitor between 5 and 10 "spreads" (the difference in price between two instruements) in their head at a time.  A CEP engine can monitor hundreds of them.  Decision can be made and executed in less time than it takes a neuron to fire in the human brain, and for the trader's finger to be moved to press the "buy" button (about 80 milliseconds).  Finally, CEP can be used to build these applications more quickly than backward-looking BI programming styles, because CEP contains built-in notion of time, and the built-in ability to detect patterns of events (A followed by B, then C).  BI isn't designed for any of this.

So industry observers should watch for applications that replace large banks of analysts using BI tools;  not for BI vendors cannibalizing their own market with new technology that supplants the old.

Understanding Event Driven Architecture by Schulte / Chandy

Roy Schulte and Dr. Mani Chandy got together on this nice article on CEP and EDA, called Understanding Event Driven Architecture.  It succinctly describes complex event processing, event driven architecture, and business activity monitoring, three topics we're primarily concerned with in this blog.   

One of the sections of the article that I think is important and relatively new is their breakdown of "pull-based" versus "push-based" applications as one of the distinguishing characteristics of event processing applications versus more traditional applications.

The conclusion of this article provides a concise summary:

"EDA is under-utilized because architects, software engineers and business analysts   sometimes fall back on the familiar pull and scheduled models as a matter of   habit. This results in business processes that are slower and less responsive   than they should be. Businesses need to make more of their processes event-driven.   The push concept and the desirability of running some business processes straight-through   are not difficult to grasp and they are being used more frequently as business   pressures grow and as developers get more comfortable with using EDA. Some parts   of all new business systems should use EDA, while other parts should still use   pull and scheduled patterns. The key is to understand the advantages of each."

Read the entire article at the source here:  Understanding Event Driven Architecture

BAM Myth #4: Limit BAM to Monitoring Simply KPIs

Here’s another in our “BAM Myths” series, exploring some of the preconceptions behind BAM.

An uncontroversial definition of BAM’s role is “to provide real-time business visibility into important business data and processes”. Take the example of the monitoring of client behaviour on a Web site. Perhaps we wish to understand how end-users are interacting with the Web site and also aim for a certain service level to be delivered. A candidate KPI (key performance indicator) to measure is the average response time between a request being received and the response dispatched back to the client.

This is certainly pretty straightforward to measure and put on a dashboard. We could also have a graph that goes red when the average response time goes above 1s. All very useful, but we should be able to go much, much further to give more relevant visibility. What about, for example, if we could predict when our service level might be exceeded in the future based upon current trends and therefore give us time to provide more computing resource? And if we could also use past activity levels at the relevant time of day to determine when the response time goes beyond two standard deviations from the historical average? We could also start correlating response times and an increase in clickstreams which failed to go all the way through to order placement. Lots of sophistication is possible by having the capability to properly correlate and analyse multiple streams of information coming from our underlying systems. Very few BAM projects get anywhere near delivering this though.

By not taking this approach, valuable business context is lost. Instead, simple, technical, KPIs are monitored which are probably only interesting and suitable for IT. It is surely preferable that the people who are responsible for business performance should have a dashboard in front of them that gives them the information directly.

Such requirements are required throughout an organization. Therefore organisations should ensure they use technology which can cope with a wide variety of different situations and which is agile enough so the BAM rules which are being applied can evolve as the organisation evolves.

Managers need to take their decisions faster with trust, consistency and depth. There is often no time for analysis of historical data to find out what happened. The decisions must be taken now, with a clear assessment of their potential organisational or business impact. This is why solutions that goes beyond simple monitoring with real time analysis and action capacities are required. And this is also why solutions that are supposedly BAM oriented, but are in fact just capable of simple KPI analysis and alerting, fall short.