EDA and SOA Myth #1: EDA is just asynchronous software design

Let's begin the 10 myths of EDA and SOA by attacking a statement from an excellent article by Jack van Hoof on “How EDA extends SOA and why it’s important.” Jack’s article both nails it and perpetuates some myths, at the same time.  He begins by writing that “reading records from a file is like consuming from a subscription topic.  This is asynchronous design.” While this is true, asynchronous design is not the same as event driven architecture.  Later, in a blog post, we see this:

I have my roots in the old IBM370 JCL. What I like about EDA is that designing an EDA is very similar to designing batch job flows. Where you write a record to a file, you publish an event.  Reading from a batch file is consuming from a topic. Where you read multiple files in balance, you come to complex event processing. Batch programs are decoupled. Programs write records and others read records, without any knowledge of each other's existence.

Simply said: Just take out the read/write-to-end-of-file loop from a batch program and the program is modernized to play its real-time role in an EDA.

Although I remember the days of JCL, just because a computer has events flying around on it doesn’t mean it’s an event driven architecture. Yes, events have existed since the beginning of time. JCL can emit events. X-windows event loops emit events.  Database stored procedures can emit events.  TCP/IP sockets can emit events.  Yes, events are as old as the hills.

But the technologies needed to build a proper event driven architecture are new.  EDA has been at the heart of innovative academic research and commercial innovation for 10 years. Analysts have grouped these technologies into the catagory of “event processing." 

Modern event processing grew its roots in the mid 1990’s, when academic research began at Cal Tech (Mani Chandy), Cambridge University (John Bates), and Stanford University (David Luckham).  The focus of this research was to develop a new approach to identify complex sequences of events (event A followed by B, then C), with temporal constraints (within 20 minutes), spatial constraints (within 5 miles of each other), and to control complex actions as a result of these patterns (begin warning an operator at code yellow until another event, D, is detected within 5 seconds).  In 2002, Luckham published the seminal book in this field called: "The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems" that helped popularize the term “complex event processing,” or CEP.

Commercial software companies were created based on this research.  Chandy’s work spawned the iSpheres (acquired by Avaya last year), and Bates co-founded Apama with Giles Nelson in 1999.  In 2003, two more vendors were founded - Coral8 (Stanford) and StreamBase (MIT / Brown / Brandeis; Michael Stonebraker / Stan Zdonick / Mitch Cherniack).  TIBCO BusinessEvents entered the market in 2004 as well.  Other CEP firms include Kaskad and Aptsoft.

Software technology typically has a gestation period of about 10 years from academic research to broad commercial adoption, and that’s where we are now. It's also typical for the capital markets to be on the bleeding edge of new technologies, and for CEP it’s no exception. CEP forms the basis of algorithmic trading EDAs in top firms like Deutsche Bank, ABN Amro, and JP Morgan.  For a technical tutorial on the application of CEP to algorithmic trading, read this article in Doctor Dobb's by John Bates.  Recently, CEP’s use has broadened in telecommunications, healthcare, retail, supply chain, fraud detection, compliance, and more. 

As a result, the event processing market now includes commercial software for complex event processing (CEP), event driven GUIs (sometimes called BAM dashboards), and event data management (aka data stream management). 

What's the point?  All these event processing technologies are designed to monitor, analyze, and act on events, not just transport them (as pub/sub middleware does), or emit them (as a sensor, batch file reader, or an x-windows event loop would).  Event processing is about adding intelligence and action to events. 

So back to the question of EDA and SOA.  Event processing technology must connect to all sources of events.  But it doesn't matter where the events come from.  As a result, EDA is not a form of SOA;  it's agnostic to the existence of SOA.  Events come from SOA?  Fine.  Events come from publish / subscribe middleware, such as TIBCO’s Rendezvous?  FIne. An RFID reader?  Fine.   TCP/IP socket?  FIne.  Sensors on a piece of manufacturing equipment?  Fine.  A business process management (BPM) tool?  Fine.  Database?  Fine.  Or, indeed, the source of events could be a batch job flow on a mainframe computer, each event emitted when an record has been read from a file.  Fine.

Yes, events are as old as the hills. But EDA, and event processing, is a fresh approach to processing events of all types.  That's why EDA is emerging as an important, modern enterprise IT architecture.

But it’s not SOA.   

Comments

[Jack van Hoof]

Hi Mark,

You are completely right that asynchronous design is not the same as event driven architecture. Your attack is valid. I wanted the illustrate the pattern and how easily batch programs can be modified to play a role in an EDA, just because both share the same pattern; designing batch jobs needs the same mindset of the designer as designing event driven architectures. Well defined batch job flows may even be EDA's. But you are right, my statement lacked some nuances. And I agree that batch jobs are (mostly) not EDA's.

Accidentally I was writing a posting related to this subject today. I just published it:

http://soa-eda.blogspot.com/2007/03/publish-and-subscribe-is-not-eda.html

I'm looking forward to the other 9 myths.

Jack

[Lars Toomre]

Mark,

I could not agree more with this post. CEP and EDA are so much more than asynchronous software design. You have really hit the nail on the head with the statement "Event processing is about adding intelligence and action to events."

The intelligence can be expressed in just about any way of relating one or more events either to each other or other data stored in traditional data bases. In the algo trading world of pairs trading, the intelligence might be if the price ratio of two securities breaks certain pre-defined bands while the MACD indicator is below some value. If this "complex event" is "identified" in real-time, the EDA application might raise an event message that could be consumed by some other process like a SOA process that recalculates portfolio risk parameters.

Alternatively, the raised event might be consumed by another EDA application that checks that a recommended buy Security A and sell Security B pair of transactions meets various compliance checks. Or the original EDA process might perform an "action" like checking that the price quotes are still valid and then electronically submitting trade instructions.

The key point though is that EDA adds intelligence and/or action to the processing of events. In the financial markets, an event can be defined in a myriad of ways. To some it might be if the price of the last trade in a stock changes by some amount like a cent, or the volume in the last period of time exceeds some percentage of the average trading volume over the last number of trading sessions. To others, it might be if a change in portfolio positions or risk limits exceed pre-set parameters.

The value prop of the Apama CEP software and other EDA vendors is that once such a "complex event" is identified, intelligence and/or action can be applied in real-time. And these software packages can process literally a hundred thousand plus of these events in a second. Hence, CEP has been rapidly adopted in the financial sector by firms engaged in algorithmic trading and other quantitative strategies that involve electronic trading.

It will be quite interesting to see how CEP/EDA applications expand into other areas of financial firm operations as more people come to understand this new way of processing events (or data changes). One of the key challenges will be how easy will it be for the "informed" business "owner" to define what "complex events" to search for in the incessant world of streaming data.

I too look forward to your future posts about the other nine myths of EDA and SOA. This one was quite good.

Lars