Event Processing blog

BAM Myth #3: BAM Works Bottom-Up

When most large enterprises engage software technology vendors, they send in the wrong troops:  they send in IT.  Although IT is an absolutely critical stakeholder in an technology decision, too often the entire decision-making process is delegated to technologists.   With BAM, this approach is a recipe for disaster.

Here's a typical approach.  When organizations consider looking at BAM, they start by an inventory of all their hardware and software and try to associate business processes to this inventory.   Out of the gate, this process is difficult to get right, and requires a lot of estimation and guessing, becuase most infrastructure, in some way, is shared.  And, with the introduction of service oriented architecture, it is the goal of IT today to share assets.  So allocation schemes and decisions based upon them, begin in a flawed way.

Next, the key performance indicators (KPIs) from an IT point of view usually don't map to the KPIs of the from a business point of view.  Technology-oriented KPIs yield technology-oriented BAM dashboards, and technology-oriented views of business information.  Too often, a business user wants to see how many orders have been processed, and is shown a tree that displays the technical components supporting the ordering application. To find the answer to a simple question requires 100 clicks to discover an aggregated view of orders.  This is not the objective of BAM.

IT-driven software development also tends to be driven by software development methodologies that are designed to build software products, not answer business questions.  Successful BAM products utilize a heavy dose of rapid propotyping, high-level description of business processes, and isolation layers between BAM dashboards and the low-level IT infrastructure.

The output of bottom-up BAM is complex requirements expressed in technology terms, wasted time, and mixed results.  The output of top-down BAM is an engaged business, an application that fits the need, and the ability to rapidly evolve and expand the system as requirements change. 

The first word in BAM is the word "Business" for a reason - projects should begin with, always include, and always be measured by, the business. 

Why use SQL?

SQL is certainly one of the successes of the computing industry. It all started with the much cited and seminal paper of Codd in 1970 which first described the relational model. Over the next few years and after efforts by both IBM and Relational Software (which later became Oracle Corp) SQL was launched into the commercial domain in the late 1970s. Standardisation then followed in the mid-1980s and further support for more modern trends such as XML added more recently. Database management systems now have matured into highly sophisticated environments for the storage, manipulation and retrieval of enterprise information. SQL is the standard language of use in the database world. Attempts to move this on and break with the Codd paradigm, such as the move towards object oriented databases in the 80s and 90s have, apart from in niche areas, failed.

We now see a trend by a number of event processing vendors to represent SQL as the language of choice for building CEP applications. For example, Streambase, Coral8 and now BEA. Why is this? Well, Streambase is simple to explain. Michael Stonebraker is one of the key forces behind Streambase and his background in the database industry is second to none. He was involved in Ingres in the 1970s and also behind some of the work integrating object-oriented principles with relational databases with Illustra in the 1990s. Databases, and therefore SQL, is part of Streambase’s DNA. In comparison, BEA’s use of SQL is harder to understand. BEA’s business is built (still) upon their application server technology and they are strongly going to market with an enterprise integration offering – Aqualogic. Databases haven’t really formed part of BEA’s background. The use of Xquery would have been more obvious.

Perhaps these vendors have concluded rightly that SQL is actually the right way of doing things in an event processing world. I don’t believe it is. It’s certainly a way, but it’s not the best. I believe it can confuse people as to what event processing is all about and can serve to inhibit adoption. SQL is certainly well understood, but by providing an SQL interface to event processing products, practitioners assume that an SQL way of thinking will be appropriate. It isn’t. By thinking of event processing as actually a real-time database you get stuck in a database-centric design pattern.

When John Bates and I were doing some of the academic research which resulted in the formation of Apama in 1999, we were looking at how to support effectively applications which were powered by high-volume streams of data – stock tick analysis, location-aware telco applications and others. We looked at using database technologies to support this, but had to rip the work up. Not only did these technologies not perform, we realised we were force fitting one paradigm into another. Taking a clean sheet of paper we came up with the beginnings of a much more elegant, performant architecture. It was data, not query driven. The use of SQL to be a language interface to this just didn’t seem to be appropriate.

It seems that others agree. I was at a conference at CITT in Germany recently where CEP formed a major topic of discussion. In particular we talked about some of the challenges of using SQL to build event processing applications underpinned by practical implementations of use cases using a variety of event processing technologies. What became apparent was that the SQL approaches appeared to hinder, not help, the developer. The baggage that SQL brought made it difficult for people to get their heads around the thinking required to implement event processing use cases. The resulting SQL was clunky and difficult to follow.

So, am I going to conclude by saying that SQL should be shunned? Well no, I’m not. As a vendor, Progress Software is all too well aware that its products exist only to help organisations solve their business problems. Partly this is allowing problems to be solved that could not be solved previously. Partly, this is also to enable productive development. Giving a choice of a development environment with which many technologists are familiar is important and SQL can provide some of this familiarity. We therefore are observing and listening closely to the opinion of the wider market and to our prospects and customers. SQL may be one of the ways in which organisations should be able to interact with an event processing system.

However I maintain that it is certainly not the best choice, nor should it be the only one.