Event Processing blog

BAM Myth #2: SAM = BAM

Some CIOs feel that they have tried and failed with BAM by aggregating business views their technical components or alerts. Unfortunately, software vendors aggravate this myth because they sell their old network system management (NSM) tools as “business activity monitoring.” The problem is, it’s not easy to put a system monitor on the IT infrastructure and gain business insight from technical activity. That’s not business activity monitoring; it’s system activity monitoring. It’s not BAM, it’s SAM.

Without tools that help apply business context to technical events, SAM tools are IT operations tools. SAM tools are important, but only show that computers are working, or anticipate hardware failures (unexpected volume of data, wrong data processed, concurrent events that together are causing a business loss).

But just because the hardware is working, how do you know that your business is running well? Without the language of business built on top of system monitoring, you’re not doing BAM. BAM development tools help business users express their business commitments in business terms, taking into account technical alerts, transactions and customer behavior. They help a business user express key performance indicators (KPIs) that express SLAs with customers, key business metrics to track, and quality indicators like availability, response time, and error rates.

Complex Event Processing (CEP) is the language of BAM. CEP helps the business users simply express questions such as: are we ready to take actions? Are my service levels met now and are they trending towards non compliance? What actions could I take now to change that trend? Will I meet my deadline?

CEP provides a common language between IT and the business user, and allows business departments and technology business processes to interact – BAM and SAM – working together to provide more agility and faster problem resolution.

Welcome BEA. Seriously (explained).

My last post "Welcome, BEA.  Seriously." has drawn both chuckles, and blank looks.  I realized that the blank looks were because the source of the "joke" is pretty old.  This post was modeled after the welcome letter / ad campaign from Apple, directed at IBM when IBM entered the personal computer race at the same time Apple launched the Macintosh.   Apple ran the ad, at left, in full-page newspaper glory.  Posting a spoof of this ad in a blog is not quite so bold, but it was fun.

We don't know much about BEA's CEP product yet - it's still in beta.  They cite algorithmic trading as a use case, yet we've never heard of them in this space, and as well documented in this blog, capital markets is mature in its adoption of CEP.  BEA talks about innovation in optimizing garbage collection and performance in Java, and we're interested to hear about their approach (Apama has had a CEP Java interface for some time now).  We're also interested to hear about how the app server approach will scale and be open to support today's heterogeneous IT landscape, something most of the pure-play CEP engines focus on.  Their entry also poses some interesting questions about standards, as we've discussed in the past here, and the role of Java in those standardization talks.

So, seriously, welcome to BEA, and we look forward to getting a little deeper into the new CEP product on the block in the coming months and years.

Welcome, BEA. Seriously.

Welcome to the most exciting software marketplace - complex event processing (CEP) - since application servers in the late 1990's, about 12 years ago.

And congratulations on your first CEP product.

Putting real programming power in the hands of the business users is already improving the way people imagine, compose, develop, back test, and deploy event-driven applications today.

CEP is fast becoming as fundamental a computing model as client / server.

When we were among the first to invent the first CEP engine in the late 90's we estimated that most applications in the world would eventually be developed this way, if only they understood the benefits.

CEP has already become an established mainstay in applications in the capital markets, fraud detection, and RFID-enabled supply chains.  Over the next decade, the growth of CEP will continue in logarithmic leaps.

We look forward to responsible competition in the massive effort to distribute this technology to the world.  And we appreciate your magnitude of commitment.

Because what we are doing is providing an innovative way of developing applications that can monitor, analyze, and act on flowing event data, as opposed to the traditional, static computing model that an application server typically supports.

Welcome to the task.

In Complex Event Processing, is “Un Oeuf Enough?”

(co-written by contributor Dr. Bart Schouw, Business Development Manager, EMEA)

Why do French recipes commonly prescribe to use only one egg in a recipe?  Because “un oeuf is enough.”  The same applies for event driven applications.  While many complex event processing (CEP) vendors harp on the ability to process 100’s of thousands of events per second (EPS),  the main value of CEP lies in an effective approach to event driven application architecture, not just the ability to handle large volumes.  It’s true that there are high-end CEP applications out there: in the financial services world of options pricing, the OPRA (Options Price Reporting Authority) penny-price feed will deliver 716,000 events per second (EPS) by January 8, 2008, and some of the CEP engines, like Apama, are built to handle these kinds of event rates.  But such EPS rates are the exception, rather than the rule, and the real value of CEP is much deeper than processing speed.

Outside of capital markets, the EPS demands for important event driven applications are often more modest.  For example, an Apama retail customer, in a relatively large automated warehouse management, has 75 pickers picking 500 items an hour, yielding about 40,000 order “events” an hour and 40,000 response “events” an hour (“done” or “unable to pick”).  That’s a total of 80,000 events an hour, or about 22 events a second.

22 events a second can realistically be handled by a traditional RDBMS solution, if the only technical challenge was about handling event volume. The leading CEP platform allow event-driven business logic to be expressed with a concise, powerful language that requires a fraction of the code required by traditional, static programming techniques. CEP languages are built on event-driven constructs (a “WHEN (event) >> THEN (logic)” metaphor), concisely identify event sequences (A followed by B, then C), and natively understand temporal constraints (with 2 hours).   So CEP concisely models the event-driven world – something static computing metaphors can’t easily do.

So back to our supply chain example, if real-time traffic and routing status and picker actions can be correlated, CEP rules can intelligently optimize picking to load a truck that’s at risk of on time and therefore must leave sooner because of traffic congestion and expected delivery times.  This customer wanted to express a complex correlation from its retail operation in Istanbul.  The logistics challenge was to replenish their outlets by automating a new centralized warehouse that replaced six small warehouses.  The new, massive warehouse they built was far from the center of Istanbul, which was far away from the most profitable shops were on the European side of Istanbul.  Out-of-stock situations at the European side would have a big impact on the profitability of the retailer. To make matters worse, the only way to get from the Asian side to the European side was via two bridges, which frequently had traffic jams.  To add to the complexity, in order to ease traffic burden, the government ordered that trucks were not allowed after 7:00 AM, making it crucial to get trucks over the bridge at the right time.  Previously, the picker’s plans were printed and handed to pickers, and couldn’t be changed. 

Correlating stock, picker schedules, traffic congestion, and truck availability helps automate the picking process with hand-held devices that dynamically displayed the next pick on the floor.  A central CEP application provides the manager visibility into the picking process to see if, for example, a delay was imminent due to traffic jams at the bridge, and gives him the real-time ability to optimize pick lists by picker on-line, re-directing resources. Extensions of this approach allow new CEP scenarios to rebalance operations intelligently and predictively, based on history and real-time inputs such as weather and traffic.

The value event-driven applications like these isn’t the ability to process hundreds of thousands of events a second, it is to express event-driven business logic quickly and easily (read on CEP and empowering business users), and to evolve event rules quickly as conditions change.  Some CEP tools allows business users to participate in this process as well, enabling domain experts that, in this example, understand supply chain. 

So CEP can quickly solve event-driven business problems, and provides value far beyond the ability to crunch numbers.   So one egg, the egg of performance, is often a key ingredient, but is not fully sufficient to create a full meal.  To provide complete business value, a comprehensive approach to event processing is required to rapidly compose, deploy, and evolve event driven logic.

Complex Event Processing as SaaS

Today I'm going to take a new approach: I'm going to denigrate our own press release. 

Yesterday, we issued a press release announcing the availability of Apama as a platform for Software as a Service (SaaS). I hated doing this release because I think the term SaaS is a horrible example of what the software industry "marketing experts" (an oxymoron) do all the time.  SaaS is a new industry label for application service provider, or ASP - ASP was over-hyped as an idea a long time ago, and when the term tuckered out, someone decided to call it SaaS.  So, the reason I hate "SaaS" is that SaaS isn't "new." The technologies associated with SaaS are simply evolutionary improvements over stuff we've been working on for years to support remote deployment of software, remote end-user connectivity through web-based interfaces, and remote manageability of infrastructure.

OK, now I'll turn to the reason I agreed, and indeed helped write, this press release:  deploying CEP as a service (e.g., as an algorithmic trading platform), is a good idea, and it provides an innovative way to help sell-side (and exchanges or providers of algorithmic trading as a service) to help buy-side build trading strategies based on their own intellectual property.  This gives the buy-side control over algorithmic trading, while leveraging the connectivity and services of their broker - a win-win. 

But even that isn't truly "unique" about this release.  What I find interesting about this release is how it reveals something deeper:  the changing relationship between the sell-side and buy-side.  As the growth of hedge funds, in number, assets, and power, continues, and the continued electronification and fragmentation of the markets continues, the sell-side is finding increasing pressure to differentiate themselves with their buy-side customers.  They need better techniques to encourage flow.  Apama and CEP provides a platform upon which intelligent, customizable CEP algorithms can be deployed.  This provides value to the buy side, and channels differentiation from the sell-side to the buy side. 

The reason for this heavy adoption isn't because the buy-side wants low-level techie tools based on UML or a proprietary StreamSQL, its because the graphical CEP tools allow business users (traders) to build their own algorithms, and customize those algorithms quickly to get into market quickly with innovative algorithmic trading (CEP), and continually back test and evolve them so that they retain that advantage.

Is that SaaS?  Who cares.  But what is interesting is what this trend represents in terms of the evolution of business models - in the case of trading, SaaS represents a new way to win and keep customers, and that is something to care about.

BAM Myth #1: Use BI as BAM

Many organizations use business intelligence (BI) solutions as part of their strategic decision-making process - to understand their strengths and weaknesses from a competitive standpoint.  BI tools provide historical insight from the past week, month, or years in order to design future strategies that can improve competitiveness.   But using only historical BAM is like a general who only relies on history to fight the next battle.  As all historians are fond to say, past is prologue - but to what future?  What's missing from history is relevant context, and that context is essential to understand the most appropriate reaction. Event-based BAM is an infrastructure that fuses history with context, and turns the historian into a general who relies on experience and history to make decisions on current conditions - as they change.

So the first worst practice of BAM is to think only like a historian, and not like a general. 

BI-based BAM is historical BAM, and provides crucial historical insight.  But once historical insight is gained, how is action taken in the moment, as business conditions change?  Event-driven BAM helps proactively manage crisis in the moment to seize fleeting opportunties, avoid losses of clients and revenues, control operational risks, and monitor how and why the business is impacted by key performance indicators (KPIs), as those factors change.  In the world of business, "right time" decision making is important because competition and organizations don't sit still.  People, processes, partners, and technology are fluid and dynamic, and it's not enough to model a perfect, neat process flow based on what is "supposed" to happen, and then to sit back and watch the business go. 

The best practice of BAM is to fuse historical perspective with a fluid, adaptable model of what's happening now. 

The proactive, in-the-now style of BAM is event-driven BAM;  a BAM style that rests on modern complex event processing (CEP) technologies - a technology that can monitor, analyze and act on what's actually happening in an organization, rather than simply looking at what happened in the past.

Therefore, the best BAM practice is to tap into your business and operational realities, and apply historical AND event-driven BAM solutions that can both monitor the current operational realities of the business and compare them to the expectations or business models from the past.  The fusion of historical and event-based BAM enables faster decisions to be made, and competitive advantage to be gained by adding pragmatic context to the decision-making process.

 

The Myths of BAM, Featuring Complex Event Processing

Now that we've completed blogging on the "10 Imperatives of Algorithmic Trading," we're ready to share another list with you in a field that is closely related to complex event processing (CEP) - the field of business activity monitoring, or BAM.

BAM is a field loaded with opinions and perceptions.  Some think of BAM as a dashboard, some think of it as business intelligence (BI), and some think of it as an event-driven approach to not just business activity monitoring, but also business activity monitoring, management, and action.  The latter, of course, is the perspective we get from our customers, and it's from that perspective that we offer observations on the "worst practices" of BAM that we encounter in working with practitioners as the space of event processing evolves. 

So without further delay let's get right to the first worst practice of BAM:  to think of BI as BAM.  Hope you enjoy our latest list.

Don't Believe Everything You Read

The integration of news with algorithmic trading is something that we've written a lot about recently, and will be writing more on soon.  I thought this article called Don't Believe Everything Your Read by Basab Pradhan was very good, lays our a number of issues around the recent developments to elementize news so that it can be read by algorithmic trading systems.

Complex Event Processing 2.0? Try 4.0

The problem with a fast-growing technology field like complex event processing is the confusion caused by rapid expansion.  John Morrell from Coral8 recently published an article about the 5 things that are "coming next" for CEP - or "CEP 2.0," as he calls it. Unfortunately, some of the events John predicts for the future of CEP happened 5 years ago.  Here are some of John's future predictions, with specific facts that prove that these events, indeed, have already happened:

1. Enterprise-class, scalable deployments.  In 2003, JP Morgan, Deutsche Bank, and ABN Amro went public with their use of Apama. Other uses have published from other applications in the capital markets, such as exchange surveillance (e.g., Kaskad's at the Boston Stock Exchange), and in smart order routing (SRO), market making, bond pricing, and more. Outside of capital markets, CEP has been applied to a wide range of other applications, including TIBCO's BusinessEvents for airlines operations monitoring at Air France / KLM, Apama's use at BGN, a large European bookseller that has deployed  complex event processing to process events from their supply chain and retail operations.

The "future" of mission-critical CEP deployments, for some, started in 2003.

2. Further Simplify Complex Problems.  The trend cited here is that more and more applications will use CEP.  Again, this began in 2003 and continues today; it seems every few weeks there's a new, public use case for CEP.

The "future" of simplifying complex problems began in 2003.

3. Enabling Best Practices.  I honestly don't know why this is a future item either.  Best practices are  uncovered every day in every industry.

4. Involving the Business User.  The article states: "Although some CEP vendors attempt to make their tools usable by business users, there is still a long way to go to make CEP truly a business user tool."  This is the most egregiously inaccurate claim of the bunch.  In the case of Apama, we released version 1.0 of the "Scenario Modeler" tool (at right) in 2002, have re-written it, and are now on version three.  This is a business user-oriented CEP development tool, or "CEP 4GL," and Apama can cite a long list of customers who cite its flexibility as a key competitive advantage in terms of time-to-market and empowerment of business analyst users. Aptsoft also features a high-level graphical application development environment. 

Indeed, I blogged about the best practices on such tools weeks ago, including screen shots of Scenario Modeler and some of its design principles.

I'm not sure what other proof John needs but screen shots over version three of this tool, published best practices, and referencable customers should be enough.

Again, the "future" began in 2002.

5. Packagable Application Frameworks.  Frameworks and analytic packages in CEP engines first came to light around 2004, as the early adopter experience from the pioneers of CEP were distilled into best practices and deployed as frameworks.  One example is Apama's framework for algorithmic trading, which includes: over 30 adapters to market data, news, OMS, EMS, and middleware; built-in, customizable "SmartBlocks," or algorithms like VWAP, spread trading, market participation, iceberging, P&L calculators; and pre-built dashboards for visualization and control of CEP logic.

So this "future" state began in 2004, although more frameworks are to be discovered. 

So although CEP is evolving quickly, and some tools are farther behind than others, the future of the CEP industry as a whole is now.

Complex Event Processing at CERN

This week I visited CERN in Switzerland, the European Organisation for Nuclear Research, who is a customer of Progress. It was an astonishing and inspiring visit. CERN is in the final stages of building the Large Hadron Collider (LHC) which is due to go into production late this year. The LHC consists of a 27km loop in which protons will be accelerated and collided at unprecedented power levels to give us new insights into the building blocks of matter. In particular the search is on for the Higg's Boson, predicted originally in a paper dating from the 1960s. Finding this will fill a gap in the Standard Model of elementary particles and forces, and will help in furthering a "theory of everything". A particular highlight was to go down nearly 100m underneath the ground to look at the ATLAS experiment - a truly massive particle detector. Its enormous size consists of a number of different elements which detect different types of particles - muons, gluons and many others. The huge magnets which form part of the detector are cooled with liquid helium down to -269 degrees C to make them superconducting (and therefore more powerful). Viewing all this brought home what a remarkable engineering effort it all is.

Anyway, what has all this got to do with events? Well, through a number of presentations that CERN staff were kind enough to give us throughout the day it became apparent that their whole world is to do with events and the processing of them. The term "event" is one which they used often, to describe the information gathered by the detectors which sit around the collider. Every time a set of protons collides sets of events are created which need to be analysed and filtered to determine which are of real interest. For example, there are two ways in which a proton can decay to produce two Z particles (check). One is predicted to involve a Higg's Boson so the set of events to look for is something like "proton collision followed by a Higg's Boson followed by two Z particles". To identify such sets of temporally correlated events the raw events are propagated up through three levels of filter to be finally sent through to a central computing resource for further research and analysis. Up to 40 million collisions per second take place. These are firstly analysed in FPGA hardware reducing the 40 million collisions to a few thousand of interest. These are further filtered in software to produce finally a few hundred. These few hundred are then sent to other computing systems for further analysis.

It's not only collider events that CERN needs to handle. CERN also has a newly built central control centre, part of which is used to monitor CERN's technical infrastructure. About 35,000 separate sensors exist to monitor everything from fire, to electricity substations, to coolant plants. All these sensors are currently producing about 1.6M events per day all of which have to propagated to a central point for analysis. In turn these 1.6M are reduced to 600K events which are overviewed by human operators. Most are inconsequential (for example the 18KeV power supply is still producing 18KeV) but some will require attention. By appropriately analysing these CERN can ensure that the colliders are running as smoothly and as safely as possible. With billions of euros invested so far in the LHC, keeping the collider up and running as continually as possible is a top priority.

The visit proved a fascinating insight into the world of particle physics and the data processing challenges it produces. It really showed event processing at its most extreme.