Event Processing blog

Reflections on the Gartner Conference and EPTS4

Like many of my colleagues in the event processing community, I thought I would share a few reflections on the recent happens at the two back-to-back technology conferences of the past week. Gartner sponsored their annual vendor-fest known as the Event Processing Summit, and the EPTS had their fourth annual symposium. This being my first EPTS, I've had some initial thoughts and reactions which I've shared over the weekend.  For this, I'll delve more into the conference's content.

I attended a number of the sessions at the Gartner conference. I did not have any set agenda so I picked the sessions more on a personal appeal rather than some well thought out plan. While I do work in an engineering team, I have a customer focus so I attended all the customer sessions. I always find it valuable to understand how customers are deploying event processing technology in real-world use cases. Their efforts clearly infiltrate the product roadmap of vendors.

 
• Lou Morgan of HG Trading, a lively speaker described his use of event processing technology in high frequency trading. Lou has been an Apama user for quite a few years and we've invited him to speak on our behalf on a number of occasions. He's an entertaining soul with a clear understanding of the Capital Markets business. We're delighted he presented his use of Apama at this conference.

 
• Albert Doolittle of  George Weiss Associates Inc. gave a talk on using event processing technologies in this firm.  Albert described his technique to pick a vendor for his CEP project, which if I were to paraphrase was a coin flip.  Towards the end of his talk, he digressed from CEP technologies to present a short discourse on high performance computing (HPC). The idea of leveraging supercomputing-like technologies and FPGA's for compute intensive operations like Black-Sholes Options pricing certainly has caught Mr. Doolittle's attention. Typically CEP and compute intensive tasks don't mix well because of latency considerations. However, a marriage of CEP and HPC is possibly one made in heaven. I was intrigued.

 
• The ebullient Marc Alder gave his brusque, no-holds-barred perspective on the CEP project he embarked on at Citi. Marc did a great job of explaining the challenges of introducing a new technology at a large corporation, one with a well entrenched bureaucratic IT organization.  I think most of us have faced the bureaucratic fortress at some time or another in our careers. Knowing how to play the game is a skill only a few master well, kudos to Marc for his successful venture.  As Marc unfolded his project's architecture he wisely chose a course to prevent vendor lock-in.

The juxtaposition of these three use-cases was most curious. Lou Morgan jumped deep into CEP technology and bet-the-ranch on it. Albert Doolittle took a gamble with a coin flip in choosing a vendor and Marc Alder kept his choice of a CEP product isolated and contained within his overall system architecture. A safeguard in case he felt the need to replace it.  Nonetheless all great examples of how CEP is gaining momentum in main stream business.

One session I thoroughly enjoyed was Don  DeLoach's "Extending the range of CEP". Don is the CEO of Aleri. I'm not sure I enjoyed this session more for its content or for Don's presentation skills. As is usually the case at technology conferences, it's death-by-Powerpoint. Slideware is typically jammed with an overabundance of barely readable text and dazzling graphics.  Don's slides however had a clear minimalist slant. A plain monotone background with either a single word or a (very) short phase well choreographed with his oration. He spoke of CEP as an evolving technology from the simple ability to filter streaming data to managing complex application state. He used an example that has become the Pièce de résistance of Aleri, order book consolidation.

There were many sessions on SOA and Event Driven Architectures - so many I lost count. 

I attended the panel discussion on low-latency messaging protocols. This was a Q&A session moderated by Roy Schulte of Gartner. The panelists were the crop of high-speed/low-latency message vendors. TIBCO-killers as I've affectionately referred to them. Vendors such as 29West, RTI, Solace Systems, IBM and even TIBCO themselves (apologies to those vendors I've not mentioned). Each described how they have defied physics to achieve incredible speeds yet still provide reliable delivery, management tools and even application level services (i.e. RTI's last value cache).  However, its noteworthy to contrast these low-latency vendors, all focused on shaving microseconds off message delivery via proprietary, even hardware-based schemes, to the many standard-based messaging systems trumpeted in other sessions. Those SOA and EDA sessions paraded a whole barrage of Web Services based standards models (i.e. WSDL, WS-Eventing, WS-Notification, WSDM, the list goes on and on) as the right way to build applications. These certainly seem like opposing forces that will only foster confusion in the eyes of those who have a clear business need for low-latency yet desire to adhere to a standards approach.

The EPTS Symposium began its first day with a keynote address from a VC which had funded Event Zero.  I had first met with Event Zero about a year ago, they have appeared to recast themselves from an adapter/connectivity vendor to one delivering an Event Processing Network (EPN). An EPN can be defined as an infrastructure platform for event processing agents or services. Those CEP agents performing both independently and in concert with other agents (or services) act upon streaming data sources. Together the whole becomes greater than the sum of the parts. Such is the grandiose vision of an EPN.  SRI was also promoting a similar notion of event processing as a service, which I would argue is a variation on this same theme.  Unfortunately, I think there is trouble ahead. The problem is simply timing, maturity and standards (or lack thereof).  I don’t think customers will buy into EPN's or Event Zero's vision until there is a clear establishment of standards for CEP. As a perspective, Application Server vendors tried this and failed (anyone remember SilverStream? Apptivity?). It was not until the J2EE specification established a uniform model that created true viability for a network or service infrastructure platform for AppServers.  Until we see the formation of CEP standards for interoperability and integration, the appeal of CEP will remain as basically a standalone application platform and vendors will continue to market a solutions approach, just look at any CEP vendor's website for proof of this. Nonetheless, Event Zero has embarked on a bold initiative and I wish them all the best.

Speaking of standards, moving slightly up the stack one could clearly detect the prevailing wind blowing against streaming SQL as the language of choice for CEP.  Going back to the Gartner conference there were a few noticeable comments to that effect. Marc Adler, described streaming SQL as making the simple things difficult to do.  Don DeLoach, downplayed the SQL language in Aleri in favor of the SPLASH enhancements. The renowned Dr. Luckham in his closing keynote address outlined Holistic Event Processing as the future implied it required a language beyond streaming SQL. 

At the EPTS Alex Koslenkov from Betfair castigated the streaming SQL approach for his use case in managing complex long-running state. Alex is an advocate of the RuleML approach to CEP languages, as such it stands to reason he doesn't have a high regard for streaming SQL and it showed.

Susan Urban from Texas Tech University presented a research project on a language they've dubbed StreamCEDL. Susan denounced streaming SQL as lacking the algebraic expressiveness necessary to move beyond simple stream processing to true complex event processing. One example, she mentioned in the description of StreamCEDL is its support of an APERIODIC operator.  The intent is to process irregular or out-of-order data streams.

Lastly, Chris Ferris from IBM presented on Industry Software Standards. This was a great session that portrayed the far reaching impact of adopting standards across our industry.  He stressed the importance in making every attempt to get broad vendor agreement, customer validation and to be sure the adopted technology serves the needs of the community because you'll have to live with it for years to come.  This is such an important message in the quest for standardization of CEP. Open, widely accepted standards are exactly what the CEP community needs; the sooner we embark on this journey the better.

A Truce at the CEP Front

                                                       

I am a bit of a history buff and often times I'm reminded of some historical event when reading about current events. This inclination I have can easily be applied to the meltdown of the global financial markets we see happening all around us. The lessons of the past should be constant reminders of how we should behave now and in the future. I've always thought a degree in history should be a prerequisite to a political life, armed with such knowledge would clearly provide guidance to govern wisely. Maybe our business leaders should follow a similar career path.

I've just attended my first EPTS Symposium. It was the Technical Society's 4th annual get together. If you're unfamiliar with this organization, it's purpose is to promote event processing technologies through academic research and industry participation. The organization has a number of working groups that have contributed greatly to the overall awareness of event processing.  You can read more about the EPTS at their website.

The symposium was well attended by members of both academia and industry. All the major CEP vendors were there and it was the first time I've been in a setting where the atmosphere was completely non-competitive. It was a truce of sorts. While we typically wage war in the virtual battlefield in a land-grab for customers, for 2 days we discussed vision, standards and use-cases. We debated ideas, but we also laughed, ate and drank together. It was general camaraderie. As I mentioned, history is one of my interests and these two days reminded me of the 1914 Christmas Truce where the Germans and the Brits crawled out of their trenches and met in the no man's land to celebrate Christmas together. The guns fell silent that night in 1914 and for the 2 days of the symposium the virtual guns of competition also fell silent.

Come Monday we'll all be back at the war again. But for a short while it was fun. To see the face of the enemy unmasked, to get to know him, to share an idea and a drink was genuinely uplifting. We found common ground in our desire to see event processing become a main stream technology.

SQL Standards - an impedance mismatch with reality

Well the hype train has left the station. As I'm sure the whole of the CEP community knows by now StreamBase has teamed up with Oracle announcing a Streaming SQL standard. I am certainly in favor of standards in software technology, they clearly represent the tide that raises all boats. Customers and vendors alike are benefactors from communal standards. From ANSI standard programming languages like C and C++  to open standards like XML. Many a consortium of vendors and customers have labored arduously to define well-known technology standards for the collective benefit of the greater worldwide community.    However, this recent announcement by StreamBase and Oracle is nothing more than the practice of the crafty art of deception and diversion.  While I see nothing wrong with StreamBase and Oracle teaming up to work on enhancing the streaming SQL language for their CEP products, to tout it as representing an emerging industry standard is simply brazen.

The streaming SQL language in today's CEP products finds its roots in academia.  The Aurora project is one such academic endeavor. SQL was the language of choice for this project for good reason. Streaming data shares a number of common attributes with static data, why not use a well known data access, data filtering, data manipulation language. The Auoroa authors clearly had this in mind when they chose SQL. I'm sure they also had an expectation that streamingSQL and the future products based on it would evolve in a manner similar to database or other backend data service technology.

However, CEP platforms have matured into application platforms. This in no small measure is due to Progress Apama and our solutions approach to the market.  The Apama stack easily lends itself to the rigors and demands of the solutions or application environment. The Apama EPL, Monitorscript has the expressiveness of syntax to describe the semantics of  complex logic in today's CEP applications.  As the saying goes, imitation is the sincerest form of flattery, many of our competitors have followed our lead by introducing a solutions approach themselves. But as a result, they've faced a challenge with SQL being the underpinnings of their EPL.  SQL was never intended to be an application language, therefore they've chosen either to build application solutions in the mixed-language environment or extend their base EPL to include procedural constructs to support the needs of application semantics. In either case, something has to give.  Reading the fine print of a StreamBase solutions datasheet: "Incorporates algorithms as Java or C++ plugins" is an indication of the inefficacy of streamSQL for the intended purpose.  With each new release of Coral8 and Aleri they announce features in their SQL-based EPL adding procedural constructs and imperative scripting constructs similar to Apama Monitorscript.  These language enhancements or mixed-mode development requirements clearly validate that CEP has evolved into an application platform and not just a back-end data service engine.  From a language standards viewpoint this has only served to fracture.  Each vendor has carved their own course in this brave new world.

As a cautionary note, standards can be the opiate of the masses. They give customers a sense that they are protected against vendor lock-in.  Even the perception of an emerging standard can be hypnotic. This is all under false pretense.  Real standards provide benefits to customers and vendors alike covering a broad swathe not just a select few.  As the CEP community ventures into the standards world we should focus in those same areas where standardization has a proven track record in other technologies, interoperability and integration.  There is plenty of fodder here and I'm sure it will unfold in the coming months.

SIFMA Retrospective

As a follow-up to my colleague Louis' report on last week's SIFMA show, I thought I'd add some thoughts of my own. My conclusion is that it was the most exciting SIFMA show I have experienced.  While I think attendance was down from previous years, I also think the quality of attendees was up. And for me personally, the excitement of being involved in some industry-moving announcements as well as meeting up with many of my colleagues from capital markets firms, vendors, press and analysts was highly invigorating.

So what were the highlights and take-aways for me?

1. CEP is clearly a theme that is getting a lot of mindshare. So many people said that CEP was a key theme of the show – which is great to hear after many years of working to help define the market. It’s also great to see this add to the momentum so soon after the Event Processing Technical Society was launched. The use cases of CEP are many and varied – and there was a lot of interest and questions around this at SIFMA. We demonstrated on our SIFMA booth 5 different CEP use cases on 5 different pods - algorithmic trading, smart order routing, managing FX market fragmentation, market surveillance and real-time bond pricing. Also the demands of CEP applications continue to make demands on the technology, and we were thrilled to demonstrate Apama 4.0 – which extends performance and user experience of CEP to new levels. Another supporting factor in the maturing of CEP is that there are starting to be very senior people in Capital Markets firms focusing on CEP as an enabling technology. Marc Adler from Citigroup   is a key example. He’s active in the community and on the STAC CEP committee, helping to define benchmarks. It was great to meet Marc at SIFMA and also to catch up with many  other esteemed colleagues from the CEP space.

2. The liquidity wars are hotting up. It was our pleasure to be involved in a press release with NYSE-Euronext which was certainly one of the big releases of the conference. Progress Apama will be hosted in the NYSE Euronext as part of the exchange's Advanced Trading Solutions offering. Traders will be able to download custom logic for algorithmic trading, risk management and smart order routing into the NYSE itself - with low latency connectivity to other trading venues via Wombat and Transact Tools. This arrangement turns NYSE into a technology provider as well as a one-stop-shop liquidity provider. This announcement was picked up by major press, including the Financial Times - in Europe, America and Asia -- see the article here.

3. Hardware is important – and so is “green”. The increase of capital markets data volumes require completely new software architectures – like CEP. But software is not always enough to support the low latency transport , processing and storage requirements. Many firms are turning to specialized hardware, combined with software – to create high performance solutions. Vhayu, for example, launched Squeezer – which combines hardware and software to supercharge their tick data offering. Also, Progress Apama were pleased to put out a joint announcement with Sun on a collaboration for end-to-end CEP solutions – combining Sun hardware and operating systems with Apama’s CEP platform and solutions. We demonstrated an end-to-end bond pricing application using the whole stack. Sun was one of the vendors who have a “green” aspect to their hardware – for example on a major CEP deployment, the hardware can be scoped for peak throughput – but can selectively shut down capacity to save power when event throughput is reduced. In this era of high energy costs and global warming there seems to be a lot of interest in this approach.

4. I love partying on the trading floor. Progress Apama were honored to be invited to a party at the NYSE to celebrate the latest developments at NYSE-Euronext (see picture at the top). It was a great pleasure to speak with our friends at NYSE-Euronext and to meet many of our old friends from the capital markets industry there – while sipping some delicious wine in that amazing place. In a way it is a shame that electronic trading is making the traditional trading floor a thing of the past – but there is something amazing about that place and I hope it stays just the way it is – even if it becomes a cool venue for other purposes. Thanks to NYSE-Euronext for inviting us – we had a great time.

I’m sure I’ve neglected a load of other trends and themes – but there’s my brain dump for the day. I’m interested to hear if you all agree.

John

Taking CEP to The Next Step

Standards are an area of growing interest in the complex event processing market. What will be the role of standards.  In what aspects of CEP will they emerge? When? 

With the premise that “in the future, it will be necessary for heterogeneous CEP products to interoperate across organizational boundaries” the Object Management Group is hosting a Technical Meeting in Washington DC to address some of the issues. As part of the agenda, fellow Apama colleague and blog contributor, Matt Rothera, will be speaking on How CEP Takes the Next Step. So if you are going to be in the DC area on March 13, and interested in this topic, you definitely should check it out.

Why use SQL?

SQL is certainly one of the successes of the computing industry. It all started with the much cited and seminal paper of Codd in 1970 which first described the relational model. Over the next few years and after efforts by both IBM and Relational Software (which later became Oracle Corp) SQL was launched into the commercial domain in the late 1970s. Standardisation then followed in the mid-1980s and further support for more modern trends such as XML added more recently. Database management systems now have matured into highly sophisticated environments for the storage, manipulation and retrieval of enterprise information. SQL is the standard language of use in the database world. Attempts to move this on and break with the Codd paradigm, such as the move towards object oriented databases in the 80s and 90s have, apart from in niche areas, failed.

We now see a trend by a number of event processing vendors to represent SQL as the language of choice for building CEP applications. For example, Streambase, Coral8 and now BEA. Why is this? Well, Streambase is simple to explain. Michael Stonebraker is one of the key forces behind Streambase and his background in the database industry is second to none. He was involved in Ingres in the 1970s and also behind some of the work integrating object-oriented principles with relational databases with Illustra in the 1990s. Databases, and therefore SQL, is part of Streambase’s DNA. In comparison, BEA’s use of SQL is harder to understand. BEA’s business is built (still) upon their application server technology and they are strongly going to market with an enterprise integration offering – Aqualogic. Databases haven’t really formed part of BEA’s background. The use of Xquery would have been more obvious.

Perhaps these vendors have concluded rightly that SQL is actually the right way of doing things in an event processing world. I don’t believe it is. It’s certainly a way, but it’s not the best. I believe it can confuse people as to what event processing is all about and can serve to inhibit adoption. SQL is certainly well understood, but by providing an SQL interface to event processing products, practitioners assume that an SQL way of thinking will be appropriate. It isn’t. By thinking of event processing as actually a real-time database you get stuck in a database-centric design pattern.

When John Bates and I were doing some of the academic research which resulted in the formation of Apama in 1999, we were looking at how to support effectively applications which were powered by high-volume streams of data – stock tick analysis, location-aware telco applications and others. We looked at using database technologies to support this, but had to rip the work up. Not only did these technologies not perform, we realised we were force fitting one paradigm into another. Taking a clean sheet of paper we came up with the beginnings of a much more elegant, performant architecture. It was data, not query driven. The use of SQL to be a language interface to this just didn’t seem to be appropriate.

It seems that others agree. I was at a conference at CITT in Germany recently where CEP formed a major topic of discussion. In particular we talked about some of the challenges of using SQL to build event processing applications underpinned by practical implementations of use cases using a variety of event processing technologies. What became apparent was that the SQL approaches appeared to hinder, not help, the developer. The baggage that SQL brought made it difficult for people to get their heads around the thinking required to implement event processing use cases. The resulting SQL was clunky and difficult to follow.

So, am I going to conclude by saying that SQL should be shunned? Well no, I’m not. As a vendor, Progress Software is all too well aware that its products exist only to help organisations solve their business problems. Partly this is allowing problems to be solved that could not be solved previously. Partly, this is also to enable productive development. Giving a choice of a development environment with which many technologists are familiar is important and SQL can provide some of this familiarity. We therefore are observing and listening closely to the opinion of the wider market and to our prospects and customers. SQL may be one of the ways in which organisations should be able to interact with an event processing system.

However I maintain that it is certainly not the best choice, nor should it be the only one.

Oracle, Databases, and Complex Event Processing: An Oxymoron?

Well this is going to be fun.  Oracle, who joined the event processing technical society group last year, is beginning to be public about their work in complex event processing (CEP) in the blogosphere.   One of their architects, Khanderao, provided an introduction to Oracle's thoughts on CEP, and listed some of the common use cases of CEP, including algorithmic trading.  Here's a link to the Khandero's blog - the entry about Oracle, EDA, and CEP.

Although the entry of Oracle validates that CEP is a fashionable term, the fact that they actually don't have a product is something to ponder - is this simply a hijacking of a cool new term?  Or are they serious about it?  I can tell you they seem serious - Oracle hosted the EPTS meeting last year, and we had about 3 hours of presentations from Oracle on CEP.  The presentations were excellent, informative, and looked nice - unfortunately, there was one thing missing:  they didn't discuss CEP.  There was all sorts of talk about, surprise, surprise, extensions to the Oracle database and SQL to support events.

So I'm sure Oracle will provide plenty of strange perspectives to spark debate in the coming months, since using the "database" and "CEP" in the same sentence without the word "antithetical" is difficult.  We use databases (not just Oracle) as a muse to explain why new approach - CEP - was necessary.  See the complex event processing tutorial discussion on this topic,  which compares "static data processing," to CEP; they are diametrically opposed topics.  Architecturally speaking, Oracle's philosophy is a classic case of "when all you have is a hammer, everything looks like a nail." 

I hope the CEP community will encourage Oracle to release, document, and publish hard customer use case information about a product, and the customer problem it solves, so we can understand their CEP approach.  Apama has done this, TIBCO has done this, Kaskad has done this, Aleri has done this, Coral8 did this - even StreamBase did one!

In related news, if you recall myth #2 in the top 10 Myths of SOA and EDA, that "EDA is SOA 2.0," I lashed out at Oracle about a lack of perspective on EDA and SOA in calling it "SOA 2.0."  About a month later, at JavaOne, folks are reporting that Oracle has stopped talking about SOA 2.0.  Let's see if their vision is similarly flawed about CEP.  So far, so flawed.

Sentient CEP and the Rights of Algorithms

I have just returned from the UK where, as part of my duties for the week, I spoke at a conference on Fixed Income and another on Foreign Exchange. At both these events, the delegates were interested to hear of the latest trends in the industry – which include aggregating multiple market data streams from disparate trading venues into a single view and using rules-based trading techniques to rapidly detect complex patterns and make and execute trading decisions. In these areas and beyond, Complex Event Processing is being successfully used by Investment Banks and Hedge Funds to enable such complex and low latency requirements.

While I was in the UK, one of the items in the news was the marking of the 200^th anniversary of the abolition of the slave trade by Britain with a service in Westminster Abbey, attended by the Queen. I hope none of my comments seem to in any way belittle this significant incident, but it did rekindle some thoughts about “will algorithms ever have rights?”. We look back on the past and find it inconceivable that any class of people could be considered as less human than others. Society has adopted that principle, quite rightly, as part of our programming. However, we’re quite happy to turn a blind eye to the suffering of animals in unnecessary cosmetic testing and in horrific factory farm conditions. In the same way that most people in the 18^th Century were quite happy to receive cheap cotton and sugar, now we are quite happy to receive cheap cosmetics and food. History suggests, however, that this will change eventually.

So what of the rights of the algorithm? Now you may think this is totally mad – and you’d probably be right. However, consider for a moment the field of algorithmic trading. While speaking at both of the conferences this week, I illustrated the changing role of the trader. Rather than the trader watching the markets for key trading opportunities and then manually executing, he/she now can deploy an army of algorithms to look for and respond to the key opportunities. The algorithms are the trader’s slaves. As a customer of Apama’s, Lou Morgan of HG Trading, put it recently – “…. They don’t need a lunch-break and they don’t need a bonus…!”. Of course these algorithms are not sentient, and therefore they don’t have any rights – but what if they were!?

Together with my colleague and mentor at Cambridge - Professor Andy Hopper, I ran a research group that looked into what we termed “Sentient Computing”. This is a very grandiose title for what Gartner might describe in commercial terms as the “Enterprise Nervous System”. It was all about how complex asynchronous stimuli in a wide-area computing environment could be detected and used to automate responses. There were lots of fun hardware in this environment, like “Active Badges” - that could detect your location to within 2 centimeters in 3 dimensions, “Active surfaces” – that provided a high bandwidth network to your wearable computers when in contact with the surface, and a variety of other ubiquitously deployed sensors, actuators, compute terminals and devices.

But the whole glue that made Sentient Computing possible was Complex Event Processing and Event-Driven Architectures – as they are called today. Sensors generated events – that needed to be analyzed; Actuators could be triggered – but needed something to do the triggering. Event-based rules provided the glue to enable complex circumstances to be modeled. For example “When John Trigg and Chris Martins are within 10 feet of each other  and there is a video terminal in their view, then play message ‘good morning’ on the nearest video terminal”. Some people described this kind of environment as an “Exploded Robot” – because rather than a single object having various sensors and actuators attached to it, the network cloud is the medium to which they are attached – and CEP and EDA are the mechanism through which the “neuron firing” is wired together. Nowadays, we are familiar with how CEP and EDA are enabling all sorts of “exploded robot” applications – such as logistics companies that monitor the wide-area movements of their fleets of trucks, and optimize based on the perishability of their goods, weather, traffic conditions etc.

Although we are at an early stage with Sentient Computing, the technology approaches of CEP and EDA definitely provide a great framework. An event is a nerve impulse from the body. CEP provides the brain – which can be changed with new rules (new thinking?) at any time – rather than being hardwired. The next stage of course is rules that are written and refined by the system itself. But the exciting thing is that there doesn’t have to be one brain; the powerful thing about events is they can flow on to another brain that can be processed and analyzed in a different way (a different perspective on the information; a different opinion?). And then events resulting from decisions can be routed to actuators/services/humans to ultimately cause things to happen – the resulting nerve impulse. All the time, EDA is providing a framework to bring these various nerve inputs, brains and nerve outputs together. Clearly we’re not there yet with EDA – and CEP is being used mainly in individual systems unconnected – but it is coming to offer an exciting “event cloud”, enabling various CEP systems to interoperate.

So should algorithms have rights? Well of course not – and it is April 1^st as I’m writing this. But I do ask you to think about what you would have been like it you’d be alive in the 18^th Century and enjoyed cheap tobacco, sugar and cotton. Would you have “thought differently”? I remember an episode of “Star Trek – The Next Generation” in which a scientist wanted to take Mr Data apart to see how he worked. Captain Picard successfully proved to Star Fleet that Mr Data was sentient and thus an individual, rather than being property. I don’t think you necessarily have to be sentient to have rights – there’s no reason to pull the wings off a butterfly just because it isn’t familiar with Descartes. I used to comment to one of my friends – who is a Maths Professors – that his computer programs were so badly written that they were like strange mutated creatures that probably suffered. Of course this is very silly!

Anyway, I leave you with the thoughts that perhaps Event-driven Architectures offer us the ability to have a truly global “information nervous system”. We are asynchronous; the world is asynchronous – so computing paradigms should support this. Of course realistically this is just a set of inputs, rules and outputs. However, as we add more intelligence to the “brains” then we must be careful this system doesn’t become too autonomous. After all – you know what happened with "Skynet" in Terminator!

On StreamBase, Algorithmic Trading and MiFID

(NOTE:  This post has been softened in tone from its original form, but carries the same message)

StreamBase recently announced "groundbreaking algorithmic trading and MiFID solutions."   As for this latest release, I'm left wondering what ground has been broken.  That claim, by my read, fails to be substantiated.  I'm curious to know: has StreamBase benchmarked their algorithmic technics and what can they say about them?  What general class of algorithms have they developed?  VWAP?  TWAP?  Market participation?  Spread trading?  MACD?  Iceberg? SOR?  What new techniques have they been developed and on what academic or research foundation?  Were the algorithms co-developed with a specific firm?  Sell-side firms are proud of and promote their proprietary algorithms, so it's odd that the techniques developed were not described.

Just like their press release about a ground swell of support for proprietary SQL-ish CEP standards, these tactics continue to do a disservice to the event processing market.  Despite claims of "broad market acceptance," they have yet to announce news about customers.  Despite claims of "broad support for CEP / SQL standards," no other complex event processing vendor supports their approach in public.  Despite claims of a "ground breaking" algorithmic trading solution, they fail to reveal even a general sense for what kind of ground has been broken.

In this blog we'll continue to promote substance-based marketing tactics by anyone in the event processing industry.  For example, Coral8 published an article on RFID Journal yesterday about Patient Care Technology Systems and Kaskad recently announced their customer, the Boston Stock Exchange.  Coral8 and Kaskad's releases raise awareness of important use cases for CEP technology, and that's good for the whole event processing market.

On CEP and Standardization

Last week Anita Howard wrote an article about the adoption of event processing in capital markets.  The first part was great - she reports that event processing is being broadly adopted in the capital markets (and other markets as well).  She also suggests that SQL is an approach to addressing the complex event processing (CEP) technology issue.  But then the article implied, via quotes from StreamBase, that SQL is the "only candidate" for a standard event processing language and that SQL is "widely used," and there was follow up in the blog from StreamBase marketing people that there was "broad support" for SQL as a standard for event processing.  Although it's accurate to say that a SQL based approach is a viable, and in some ways effective, approach to CEP, there are lots of other language-based approaches that are, arguably,  more effective, and, arguably, more "widely used." I'm waiting for the next claim that SQL has been adopted for CEP on Jupiter, indicating that SQL is, indeed, an intergalactic standard. 

But the real issue I have is that StreamBase continues to polarize the industry and provoke CEP-by-SQL religious language wars.  Naturally software vendors will emphasize that their approach is better than the next guy's, but tactics that divide early stage technology markets are not healthy and should be stopped immediately.

Research in CEP began simultaneously in the mid 1990's at Stanford University (David Luckham), Cal Tech (Mani Chandy), and Cambridge University (John Bates).  This work, and subsequent commercial work, has resulted in a range of language approaches to CEP, including:

• Graphical point-and-click environments
• CEP rules
• CEP languages (e.g., David's RAPIDE, and Apama's EPL)
• Java
• SQL-like syntax

Vendors, of course, are a proxy for customer demand and preference, and so far, there is no "intergalactic CEP standard." Progress Apama supports graphical tools, CEP rules, a CEP language, and Java; I believe Aptsoft has visual tools;  I believe TIBCO BusinessEvents is Java based, and has graphical tools; I believe Kaskad has a language based approach; and I believe Aptsoft has a visual environment.  In other words, these vendors, and by proxy their customers, represent that there is strong support for visual, Java, rules, and language-based approaches to CEP.

At Apama, we will continue to avoid polarizing discussions about standards, and participate in the nascent standards discussions that are inclusive of all approaches.  We'll also continue to advocate other important issues like customer use cases, the business value of those use cases, architectural discussion about the relationship between EDA and SOA, continued work on the event processing glossary, and more.

Constructive discourse that unites, not divides, will help the tide rise for event processing, and lift all boats along with it - customers, vendors, and more.