Event Processing blog

In Complex Event Processing, is “Un Oeuf Enough?”

(co-written by contributor Dr. Bart Schouw, Business Development Manager, EMEA)

Why do French recipes commonly prescribe to use only one egg in a recipe?  Because “un oeuf is enough.”  The same applies for event driven applications.  While many complex event processing (CEP) vendors harp on the ability to process 100’s of thousands of events per second (EPS),  the main value of CEP lies in an effective approach to event driven application architecture, not just the ability to handle large volumes.  It’s true that there are high-end CEP applications out there: in the financial services world of options pricing, the OPRA (Options Price Reporting Authority) penny-price feed will deliver 716,000 events per second (EPS) by January 8, 2008, and some of the CEP engines, like Apama, are built to handle these kinds of event rates.  But such EPS rates are the exception, rather than the rule, and the real value of CEP is much deeper than processing speed.

Outside of capital markets, the EPS demands for important event driven applications are often more modest.  For example, an Apama retail customer, in a relatively large automated warehouse management, has 75 pickers picking 500 items an hour, yielding about 40,000 order “events” an hour and 40,000 response “events” an hour (“done” or “unable to pick”).  That’s a total of 80,000 events an hour, or about 22 events a second.

22 events a second can realistically be handled by a traditional RDBMS solution, if the only technical challenge was about handling event volume. The leading CEP platform allow event-driven business logic to be expressed with a concise, powerful language that requires a fraction of the code required by traditional, static programming techniques. CEP languages are built on event-driven constructs (a “WHEN (event) >> THEN (logic)” metaphor), concisely identify event sequences (A followed by B, then C), and natively understand temporal constraints (with 2 hours).   So CEP concisely models the event-driven world – something static computing metaphors can’t easily do.

So back to our supply chain example, if real-time traffic and routing status and picker actions can be correlated, CEP rules can intelligently optimize picking to load a truck that’s at risk of on time and therefore must leave sooner because of traffic congestion and expected delivery times.  This customer wanted to express a complex correlation from its retail operation in Istanbul.  The logistics challenge was to replenish their outlets by automating a new centralized warehouse that replaced six small warehouses.  The new, massive warehouse they built was far from the center of Istanbul, which was far away from the most profitable shops were on the European side of Istanbul.  Out-of-stock situations at the European side would have a big impact on the profitability of the retailer. To make matters worse, the only way to get from the Asian side to the European side was via two bridges, which frequently had traffic jams.  To add to the complexity, in order to ease traffic burden, the government ordered that trucks were not allowed after 7:00 AM, making it crucial to get trucks over the bridge at the right time.  Previously, the picker’s plans were printed and handed to pickers, and couldn’t be changed. 

Correlating stock, picker schedules, traffic congestion, and truck availability helps automate the picking process with hand-held devices that dynamically displayed the next pick on the floor.  A central CEP application provides the manager visibility into the picking process to see if, for example, a delay was imminent due to traffic jams at the bridge, and gives him the real-time ability to optimize pick lists by picker on-line, re-directing resources. Extensions of this approach allow new CEP scenarios to rebalance operations intelligently and predictively, based on history and real-time inputs such as weather and traffic.

The value event-driven applications like these isn’t the ability to process hundreds of thousands of events a second, it is to express event-driven business logic quickly and easily (read on CEP and empowering business users), and to evolve event rules quickly as conditions change.  Some CEP tools allows business users to participate in this process as well, enabling domain experts that, in this example, understand supply chain. 

So CEP can quickly solve event-driven business problems, and provides value far beyond the ability to crunch numbers.   So one egg, the egg of performance, is often a key ingredient, but is not fully sufficient to create a full meal.  To provide complete business value, a comprehensive approach to event processing is required to rapidly compose, deploy, and evolve event driven logic.

CEP in Retail - Retail Systems Magazine Coverage

Retail system wrote a nice article that includes coverage of CEP in Retail Systems Magazine, it came in part from an interview I had with Penelope Ody in London last month.  Here's the section that talks about CEP, please do read the whole article to get all the context.   The section that discusses CEP is called "Dealing with Complexities."  (FYI, we are working with the publication to correct the error about David doing his work at Cambridge, they clearly caught that wrong!).

Dealing with Complexities

Extracting something useful from such vast amounts of data usually means exception reporting and selected alerts but these still happen after the event. In a real-time world something else will be needed, which is where complex event processing (CEP) comes in. “CEP really derives from the need to manage growing volumes of data combined across distributed computing systems,” says Mark Palmer, general manager of Progress Software’s Apama Division. “By the 1990s the data emphasis was on business intelligence while SOA was transforming distributed operations; CEP has followed from these developments.”

Work on CEP began in the academic arena back in the mid-1990s both in the US, at Stanford and MIT, as well as here at Cambridge under David Luckham, whose work led ultimately to the development of Apama in 1999. Other players drawing on this academic background include Tibco, which launched its BusinessEvents application in 2005, and Streambase – founded in 2003 by Mike Stonebreaker.

A (perhaps unlikely) retail leader here is Dutch book chain Boekhandels Groep Nederlands (BGN), which is using CEP systems from Apama in conjunction with item-level RFID for monitoring what is on the shelves. Key to the business case has been that vital reduction in labour cost. Last April the company started using RFID at its Almere store followed in September by Maastricht.

“We know from Almere that we can now check goods in at the branch in 12 minutes with a high degree of accuracy and we expect to rollout RFID to a further 15 or 16 stores during 2007,” says Jan Vink, IT director at Boekhandels Groep. Results from Almere and Maastricht also suggest a reduction in out-of-stocks by up to 20 per cent as well as stock records that are totally accurate, instead of the less than two-thirds accuracy levels of pre-RFID days. This real-time information is also enabling a ‘click and collect’ approach to buying online and picking up books in the store.

“The cool thing about CEP is that once the tools are in place you can start to see patterns and trends that were not visible before, and it is easy to add another rule and look for new events,” says Progress Software’s Palmer. “You can’t easily do that with conventional databases or traditional data warehouses.”

CEP is already widely adopted in the financial sector but retailers will not be far behind. Teradata, for example, already has it on the agenda and discussions with global players like Metro, Wal-Mart and Tesco on CEP/RFID solutions are already underway.

EDA and Event Processing in Retail and Supply Chain - A Case Study

Applications in the retail and supply chain can gain benefit from event driven architecture (EDA), event processing (EP), and complex event processing (CEP).  Many in the blogosphere have asked for more detail on use cases to better understand these claims.  At Progress Apama, we have been working with our customer BGN as a vehicle to educate on the application of event processing in this industry.

ROI is gained by optimizing processes at multiple levels - large and small.  BGN's use of event-driven business processes has many benefits, and here's one example, described by Jan Vink, the IT Director of BGN, from the whitepaper listed below:

“With this RFID solution in store, you can perform an inventory in less than two hours with two FTEs. In the existing stores you have to close down the whole store and execute the inventory with 20 to 25 employees. This will save us 260,000 euros on a yearly basis.” — Jan Vink, IT Director, BGN

Jan is describing how BGN has changed the way they manage inventory in their operations, and the manual labor that saves.  He goes on in the case study to describe how the increased visibility into their stock provides better customer visibility and satisfaction, as well has helping control their inventory.  Here are the detailed resources to find out more about BGN:

The BGN case study.

A video that describes the BGN system, the software components, and the the business value of this event-driven supply chain and retail operation.

Press release:  Progress Software Customer BGN Wins Prestigious RFID Visionary Award at 2006 RFID Breakthrough Awards

An overview of the Apama architecture that powers the event processing elements of BGN.

Selexyz store web site that describes their automated stores.

Article on CEP in Manufacturing

Good article in Manufacturing Computer Solutions by Brian Tinham, who I met this week in London. 

This was his take on things: Progress Apama, part of application infrastructure and database developer Progress Software, says its high end event correlation software has a significant role to play in bigger ticket manufacturing.

Its systems, which sample any form of data and can show event patterns and correlations at any level, have seen adoption in the financial services sector, but are now ready for use in production.

CEP In Manufacturing

We have experienced a recent rise in CEP use cases far afield from capital markets and trading, and we're finding them useful to describe to the community as illustrations of how CEP applies to many domains.

Surely there are enough monitoring and measuring technologies in the manufacturing fields that provide real-time alerting.  Right?  Perhaps not.  There are three recurring patterns that we see around the interest and introduction of Apama into manufacturing (and maybe more broadly issues of logistics, distribution and retail, like our customer BGN, who have automated their bookstore operations with Apama.  Read the BGN case study here). 

1.  Correlate events from multiple streams - sensors from plant floor devices, human input devices, ERP transaction systems - in real time

2.  Monitor scenarios across sliding time windows.  This has only been possible thru constant database polling. Hence its inherent latency (store-query-analyze, store-query-analyze etc etc) and is again post-facto.

3. The need to act, not just monitor.  As the need to tighten efficiencies and eliminate waste from manufacturing cycles gets more and more vital, Apama allows manufacturers to gain real-time insight into the impact of one cause (one or more events) on different processes.  Most importantly it allows them to ACT as the situation is developing rather than just use data in a post-mortem.

A vendor of plant monitoring software is using Apama to implement the following specific time-based logic:

In a plant producing a particular household product in containers, a container filling line is being monitored for fill weights within a +/- range of acceptance. The filling lines have in-line scales that monitor the fill weights. Containers that are not within this range are ejected from the filling process and recorded within the Apama correlator. If the frequency exceeds a user-defined tolerance an event or multiple events may be triggered such as an audible alarm, and/or a signal light is turned on, or the line itself could be stopped.

These examples are just a few of the things we're seeing in manufacturing.  Relative to trading, CEP adoption is early, but as we discover new use cases, we'll keep posting.