Event Processing blog

EDA and SOA Myth #5: Do BPM Before BAM

In a recent tour of Asia Pacific, I met with the lead Gartner Analyst in Japan, Kimihiko Iijima, who told me that, in Japan, companies were "bypassing BPM and moving directly to BAM" as a way to demonstrate the business value of their investment in service oriented architecture (SOA).  Rather than unify their business process management in a traditional BPM tool, in Japan, companies are directly implementing event processing (CEP) tools that are integrated with BAM dashboards to process events from a variety of sources, including processes that represent a structured business process. 

Why skip the BPM?  Because BAM tools process events from a variety of sources, not just a business process flow.  Business process flow is one thing, but the integration and analysis of events from applications, technical infrastructure, databases, and even automation devices like kiosks or RFID readers is intrinsic to pure BAM tools, and is not common with BPM.

One example of such an application is shown here, where a traditional business process flow that manages mortgages is shown.  CEP allows this application to monitor, analyze, and act on this flow and ask questions like: "when the time between the "assign coordinator" and "acknowledge" phase of the business process is greater than 10 minutes, then alert the coordination manager with an alert."  CEP provides a flexible model to express these rules on top of a business process flow.

Some argue that BAM is simply an extension of BPM, or, worse, that BAM and BPM are converging (Myth #3).  In Japan, Gartner's seeing this another way, and we see it differently as well - that CEP and BAM, when combined, can provide a higher level of intelligence, more rich and complete than a unified BPM environment where input comes from a variety of sources, not just a BPM tool.

EDA and SOA Myth #4: Think SOA or EDA Architecture First

Todd Biske made some interesting observations about Myth #3:  BAM and BPM are Converging.  Here's what he wrote:

I think that the most important takeaway on this is that you have to be thinking from an architectural standpoint as you build these things out. This isn’t about running out and buying a BAM tool, a BPM tool, a CEP tool, or anything else. What metrics are important? How will the metrics be collected? How do you want to perform analytics (is static analysis against a centralized store enough, or do you need dynamic analysis in realtime driven by changing business rules)? What do you want to do with the results of that analysis? Establishing a management architecture will help you make the right decisions on what products you need to support it.

I agree with 105 of the 114 words he writes.  I violently agree that the most essential element to consider on any project is the business requirements ("what metrics are important," "how to perform analytics," "Do you need dynamic rules management," etc.).  But the 9 words I don't like, and another great myth of the SOA versus EDA debate is:  you have to be thinking from an architectural standpoint.  Perhaps he didn't mean it this way, but it sounds like he may be advocating beginning by choosing the right architecture like BAM, BPM, and CEP, then consider the requirements the technology is supposed to fill.

Todd may not have meant it that way, but many people do it this way, every day.  They buy technologies without careful consideration of what they are trying to achieve.   Now I can hear you grown:  "Great insight - gather good requirements first - duh."  Simple idea, hard to follow.  As a software vendor, we encounter a steady stream of technology investigations that have lost sight of this simple truism.  Technology RFIs, proof of concepts, benchmarks, programming language evaluations, philisophical debates over what language syntax is better - SQL or rules languages or Java or visual for CEP logic.  Blaa, blaa, blaa, blaa.  The architecture and technology debates go on forever. 

Software vendors are guilty of fanning the techno-babble flame.  Wild claims about broad support for one vendor's proprietary language approach, or declarations of performance barriers broken by "new, revolutionary" approaches.  Here's a quick sampling of best hyperbole from the past few months:

• Skyler manages rates as high as 200,000 messages/second
• Key feature:  Coral8 handles thousands to millions of events per second
• StreamBase extends performance leadership by processing over one million events per second with near zero latency
• Aleri Labs breaks sub-millisecond latency barrier

Can't the marketing vice presidents of the world come up with something new rather than droaning on about feeds and speeds? 

Fortunately, there are cagey technology professionals out there.  The cagey ones never stop asking the question:  "What are we trying to achieve?"  The cagey ones constantly ask, answer, and re-ask these questions.  The cagey ones care about thoughtful product functionality that solves specific problems and provides specific value.  The cagey ones think about the problem first, and architecture, features, quality, reputation, languages, feeds and speeds next.

So this myth, myth #4 on our list, is to NOT consider architecture first;  instead, spend more time measuring twice, three, and four times before choosing the right architectural cut to make in your enterprise architecture fabric.

EDA and SOA Myth #3: BAM and BPM are Converging

When all you have is a hammer, everything looks like a nail.  When all you have is a business process management (BPM) view of the world, then everything looks like something that converges with it.  For example, business activity monitoring, or BAM, often gets "lumped with" BPM – after all, what’s more event-driven than a business process?   Events come in, a process ensues, and events are emitted as a result.  Gartner has spoken the convergence of BAM and BPM, so has IBM, so have respected academics like Mani Chandy from Cal Tech. If you think the whole world is using BPM tools, then why not assume they all need a dashboard?  And therefore why not conclude that the raison d'être of BAM is to monitor a BPM tool?  BPM tools have many great applications, but they are hardly THE dominant force in the automation of processes.  In fact, most of the world's most sophisticated automated systems don't use BPM at all (for example, algorithmic trading in capital markets).

In the event driven architecture (EDA) community, many view BAM converging with EDA, not BPM.  David Luckham wrote a nice article on “Basic BAM,” and later on "Why BAM must use CEP."    Since then, CEP vendors have continued to simply see BPM as yet another source of events.  For event processing platforms, state transitions in a BPM tool generate events that can be used as source events that trigger a CEP scenarios.  CEP adds the ability to ask and answer critical business questions about key performance indicators like: "when the time between receipt of a mortgage application and the first processing step is 2 hours greater than our acceptable duration, then alert the mortgage claims officer immediately to mitigate the issue."  In this way, CEP adds intelligence to BPM.  And, as an event sink, a business process engine can execute sophisticated actions that are triggered by a CEP scenario - for example, when the mortgage claims process exceeds the allowable threshold, then trigger a new business process that can alleviate the backlog.

Part of this issue was articulated by Gartner's lead BAM analyst, Bill Gasman, when he said with dissapointment at lunch a year ago (paraphrasing): "unfortunately the term BAM has de-evolved to be a synonym for 'dashboards,' and everything in software nowadays has a dashboard.  Our original vision of BAM included CEP capabilities as an element of BAM."  (full disclosure:  although Bill said this to me in so many words, I have not validated this quote with him;  I will soon).

So the modern definition of BAM is deeper than "a dashboard" and certainly more expansive than "a dashboard for BPM."  The latest definition of BAM includes CEP, event data management and dashboards, which is why CEP can be thought of as: “The Brains Behind BAM," and why Gartner includes CEP as part of their BAM definition in their 2006 BAM MarketScope.   Where does BPM fit?  Events can be one of many sources of events, although events can just as easily come from a database or piece of middleware.

So BAM is an application paradigm that includes business logic, data management, and GUIs.  The underlying architecture for BAM is fundamentally event-driven - that is, it's an EDA; BPM plays a role, but it’s not the epicenter of the BAM universe.

EDA and SOA Myth #2: EDA is "SOA 2.0"

Myth #2 takes a swipe at an easy target - Oracle.  Oracle has been a primary purveyor of the myth that EDA is "version 2.0" of SOA.  The fact that Oracle has taken this position is loaded with irony:  EDA is rooted in the idea of processing events in the now, rather than by looking in the rear-view mirror - that is, into a database.   No wonder Oracle has it all wrong - their company was built around technologies designed to analyze the past.  Naturally they look at the world from a database perspective.  Hopefully nobody is listening when it comes to their views on EDA.

So when Oracle describes EDA as version 2.0 of SOA they are even further from the mark.  EDA is not version 2.0 of SOA, it's not even a type of SOA.  By contrast, I think Brenda Michelson got it right - that SOA and EDA are peers and complements to one another. 

So if EDA is not "SOA 2.0," then what is it?  Myth #1 described one thing that EDA is not: asychronous software design, and describes the critical elements of an EDA:  CEP, BAM dashboards, and event data management.  EDA requires event-driven data sources of all types - including pub / sub, messaging, SOAP / HTTP, an enterprise service bus (ESB), email events, events from business process management (BPM) tools, SMS events, and yes - Oracle will be thrilled - events that originate from a database - perhaps database triggers, for example.  Joe McKendrick, in his article Is EDA the ‘new’ SOA? presented a fair view of what EDA is, and Rich Seeley did a nice job describing CEP in his article EDA + CEP:  A New Physics of Computing.

CEP, as a "new physics of computing," is central to this idea that EDA is an independent, and unique, architecture.  The idea behind CEP was captured by Rich Seeley when he quoted John Bates, one of the inventors of CEP:

In the traditional system the "data is static and the queries are dynamic," [Bates] said. "In event-based systems it's different. The rules that you're using to monitor the data and take action are fairly static, and it's the data that's dynamic. The data is continuously changing. So you have to structure your software to take into account that paradigm shift."

CEP's unique technical capability lies in its ability to identify patterns of events and apply temporal, or spatial, constraints to streaming data, at tremendous scale and low latency.  It adds dynamic intelligene to any source of events, whether they come from a SOA or a database, it doesn't matter.  So EDA is agnostic to the existence of SOA.  In my sampling of a few hundred EDA systems I've been involved with in the past five years, over 80% of them don't have a SOA anywhere in the system architecture.  If a SOA is there, great.  If a SOA is not there, fine.  EDA is indifferent to the presence of SOA.

All that said, a SOA is a great source of events!  But just because SOAs can be event-driven, that doesn't mean it's an event driven architecture.   

So, in summary:

Is EDA indifferent to the presence of SOA?  Yes.

Are EDA and SOA are complementary?  Yes.

Will EDA "replace" SOA?  No. 

Is EDA "SOA 2.0" - No.

That's why the claim that EDA is "SOA 2.0," is myth #2 on our list.

CEP, as a "new physics of computing," is central to this idea that EDA is an independent, and unique, architecture.  The idea behind CEP was captured by Rich Seeley when he quoted John Bates, one of the inventors of CEP:

In the traditional system the "data is static and the queries are dynamic," [Bates] said. "In event-based systems it's different. The rules that you're using to monitor the data and take action are fairly static, and it's the data that's dynamic. The data is continuously changing. So you have to structure your software to take into account that paradigm shift."

CEP's unique technical capability lies in its ability to identify patterns of events and apply temporal, or spatial, constraints to streaming data, at tremendous scale and low latency.  It adds dynamic intelligene to any source of events, whether they come from a SOA or a database, it doesn't matter.  So EDA is agnostic to the existence of SOA.  In my sampling of a few hundred EDA systems I've been involved with in the past five years, over 80% of them don't have a SOA anywhere in the system architecture.  If a SOA is there, great.  If a SOA is not there, fine.  EDA is indifferent to the presence of SOA.

All that said, a SOA is a great source of events!  But just because SOAs can be event-driven, that doesn't mean it's an event driven architecture.   

So, in summary:

Is EDA indifferent to the presence of SOA?  Yes.

Are EDA and SOA are complementary?  Yes.

Will EDA "replace" SOA?  No. 

Is EDA "SOA 2.0" - No.

That's why the claim that EDA is "SOA 2.0," is myth #2 on our list.

EDA and SOA Myth #1: EDA is just asynchronous software design

Let's begin the 10 myths of EDA and SOA by attacking a statement from an excellent article by Jack van Hoof on “How EDA extends SOA and why it’s important.” Jack’s article both nails it and perpetuates some myths, at the same time.  He begins by writing that “reading records from a file is like consuming from a subscription topic.  This is asynchronous design.” While this is true, asynchronous design is not the same as event driven architecture.  Later, in a blog post, we see this:

I have my roots in the old IBM370 JCL. What I like about EDA is that designing an EDA is very similar to designing batch job flows. Where you write a record to a file, you publish an event.  Reading from a batch file is consuming from a topic. Where you read multiple files in balance, you come to complex event processing. Batch programs are decoupled. Programs write records and others read records, without any knowledge of each other's existence.

Simply said: Just take out the read/write-to-end-of-file loop from a batch program and the program is modernized to play its real-time role in an EDA.

Although I remember the days of JCL, just because a computer has events flying around on it doesn’t mean it’s an event driven architecture. Yes, events have existed since the beginning of time. JCL can emit events. X-windows event loops emit events.  Database stored procedures can emit events.  TCP/IP sockets can emit events.  Yes, events are as old as the hills.

But the technologies needed to build a proper event driven architecture are new.  EDA has been at the heart of innovative academic research and commercial innovation for 10 years. Analysts have grouped these technologies into the catagory of “event processing." 

Modern event processing grew its roots in the mid 1990’s, when academic research began at Cal Tech (Mani Chandy), Cambridge University (John Bates), and Stanford University (David Luckham).  The focus of this research was to develop a new approach to identify complex sequences of events (event A followed by B, then C), with temporal constraints (within 20 minutes), spatial constraints (within 5 miles of each other), and to control complex actions as a result of these patterns (begin warning an operator at code yellow until another event, D, is detected within 5 seconds).  In 2002, Luckham published the seminal book in this field called: "The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems" that helped popularize the term “complex event processing,” or CEP.

Commercial software companies were created based on this research.  Chandy’s work spawned the iSpheres (acquired by Avaya last year), and Bates co-founded Apama with Giles Nelson in 1999.  In 2003, two more vendors were founded - Coral8 (Stanford) and StreamBase (MIT / Brown / Brandeis; Michael Stonebraker / Stan Zdonick / Mitch Cherniack).  TIBCO BusinessEvents entered the market in 2004 as well.  Other CEP firms include Kaskad and Aptsoft.

Software technology typically has a gestation period of about 10 years from academic research to broad commercial adoption, and that’s where we are now. It's also typical for the capital markets to be on the bleeding edge of new technologies, and for CEP it’s no exception. CEP forms the basis of algorithmic trading EDAs in top firms like Deutsche Bank, ABN Amro, and JP Morgan.  For a technical tutorial on the application of CEP to algorithmic trading, read this article in Doctor Dobb's by John Bates.  Recently, CEP’s use has broadened in telecommunications, healthcare, retail, supply chain, fraud detection, compliance, and more. 

As a result, the event processing market now includes commercial software for complex event processing (CEP), event driven GUIs (sometimes called BAM dashboards), and event data management (aka data stream management). 

What's the point?  All these event processing technologies are designed to monitor, analyze, and act on events, not just transport them (as pub/sub middleware does), or emit them (as a sensor, batch file reader, or an x-windows event loop would).  Event processing is about adding intelligence and action to events. 

So back to the question of EDA and SOA.  Event processing technology must connect to all sources of events.  But it doesn't matter where the events come from.  As a result, EDA is not a form of SOA;  it's agnostic to the existence of SOA.  Events come from SOA?  Fine.  Events come from publish / subscribe middleware, such as TIBCO’s Rendezvous?  FIne. An RFID reader?  Fine.   TCP/IP socket?  FIne.  Sensors on a piece of manufacturing equipment?  Fine.  A business process management (BPM) tool?  Fine.  Database?  Fine.  Or, indeed, the source of events could be a batch job flow on a mainframe computer, each event emitted when an record has been read from a file.  Fine.

Yes, events are as old as the hills. But EDA, and event processing, is a fresh approach to processing events of all types.  That's why EDA is emerging as an important, modern enterprise IT architecture.

But it’s not SOA.   

10 Myths about SOA and EDA

Over the past several months, the debate about EDA and SOA has heated up.  Unfortunately, much of this debate has been driven by the “SOA crowd,” who portray everything new as an extension of what they know. As a result, there have been a series of absurd assertions that EDA is an “advanced” form of SOA, or, worse, that EDA is “SOA 2.0.”  To say that EDA is version 2.0 of SOA is like saying that I am version 2.0 of my father.  I love my dad, but I’m not an upgrade for him, just as EDA is not an "upgrade" for SOA 1.0.

I’m a big fan of lists - in the past 6 months, I have noticed at least 10 major misperceptions of EDA and SOA, so I’m going to commit to provoke a discourse about some of these myths, and try to rattle at least 10 cages that I hope get me in trouble with my friends, colleagues, and employers.

The first among a long list starts at the very base of software architecture: the notion that EDA is equivalent to asynchronous design, or publish / subscribe middleware.