Event Processing blog

Sibos 2008 - the event processing angle

I am writing this at the end of the Sibos financial services show in Vienna. Sibos is the biggest global banking event of the calendar with pretty much everyone involved in the core banking area present including commercial banks, central banks, regulators, vendors and consultancies. It couldn’t, of course, have take place at a more interesting time. The extraordinary events we have witnessed this week in financial markets permeated every panel session, presentation and informal discussion held.

Event processing is big in financial services but, so far, it has generally only penetrated the front-office and middle-office risk management functions for use cases related to trading. There are good reasons for this: in general terms the front-office uses technology to directly enable more money to be made. Core banking on the other hand is about doing what banks were set up to do – to take deposits, to give credit and to be an arbiter of financial transactions. The reasons for technology investment are quite different and are driven by increasing operational efficiency, lowering costs and improving customer service. It’s more conservative and as yet event processing has not penetrated this area to any significant extent.

There’s no lack of use cases, I believe. Here are a couple of examples around the processing of payments. Earlier this year the UK launched its Faster Payments Initiative. Finally in the UK (the Netherlands, for example, have had this for 10 years) you can now pay a counterparty who banks with another UK bank in real-time, rather than waiting 3 days for the payment to clear (it’s remarkable it’s taken so long to fix such a, frankly rubbish, state of affairs and indeed it took the regulator itself to force change). As an end-user of this I am delighted with the results. I can now make an electronic payment using Web-based banking and it all happens immediately – the transaction occurs in the way one feels in the modern Internet era that it should. However this does raise a problem: how does a bank do its anti-money laundering checks, its comparison with counterparty blacklists and all the other fraud checks in the 12 seconds it has for the payment to go through? The answer is – currently with enormous difficulties.  Event processing is surely part of the answer.

Here’s another example. Currently the European payments industry is going through a lot of regulatory change to bring about lower prices and more competition for cross-border Euro payments (PSD and SEPA are the relevant acronyms if you’re interested). This will force technology investment because consolidation will mean a smaller number of banks will have to process more payments at lower cost. Furthermore competition will increase and, for example, a business in France will be able to use a bank in Germany to deal with its payments. Now, I reckon that having insight into what is going on with my payment systems, being able to identify processing exceptions, being able to identify when my customer SLAs are being exceeded and so on in real-time will be a crucial part of ensuring a world-class operation. Payment systems will continue to use many different types of technology, from mainframe to modern SOA environments,  so you need something to logically sit above this, extracting relevant real-time information and analysing and correlating it appropriately. There are offerings from conventional BAM vendors that address some of this now but I think they won't be performant or flexible enough to deal with future needs. Some customer engagements support this.

All of this is really about risk management and it seems inevitable that this area is going to be a booming area of investment in the next few years. Knowing what is going on in your business transactions, as they occur, will become more and more important. For example, in electronic trading it is becoming vital to not only regulators and trading venues (such as our customer Turquoise who we announced went live with Apama this week) but also to brokers. They want to know what their customers and they themselves are up to.

I think Richard Oliver, Executive Vice President at the Federal Reserve summed it up well. When asked about the future role of technology in core banking and payment systems he responded that the “immediacy of information is going to be vital” and that it was going to be all about “getting value from the information flows”. I think that provides a pretty good fit for event processing.

SIFMA 2008 - Hot and Busy

This week is the SIFMA (Securities Industry and Financial Markets Association) technology management conference in NYC.  The city has been brutally hot and the NYC Hilton exhibit hall, at least where Apama is, has not been measurably cooler. Those $5.00 bottles of water that they sell in the exhibit hall might represent opportunism - or capitalism - at its best. Or maybe it’s just NYC. :-)

First day attendance has been quite good and I’d encourage any attending to stop by the Apama exhibit (Grand Ballroom, Level 2, #2415). You’ll see a range of demos, including our new 4.0 release, and see some of the “magic” of CEP in action (you’ll have to come by to understand the magic reference). A lot has been happening with Apama and we’ve used this show to make some significant announcements. 

• A partnership with NYSE Euronext Advanced Trading Solutions that enables NYSE Euronext ATS to offer its customers Apama-based CEP capabilities for algorithmic trading, real-time risk management, and smart order routing. The announcement has received significant press pickup, including a nice story in the Financial Times.
• A technology development collaboration with Sun Microsystems involving support of the Apama platform on Sun x64 systems, running the Solaris 10, as well as the launch of a new Apama “Accelerator”, the Apama Real-time Pricing Accelerator. Apama continues to focus on application templates that enhance the speed with which solutions can be deployed and this Accelerator extends that theme into the area of market-making for bonds and other instruments.
• Announcement of the upcoming availability of above-mentioned Apama 4.0, which introduces a new development environment, Apama Studio. Studio brings all the Apama development tools together within a single, integrated environment, complete with tutorials, sample applications and other aids. You might consider Apama 4.0 as, at its heart, a CEP Accelerator. 4.0 also includes significant enhancements to Apama’s execution performance, partially achieved via a new event messaging transport.

So a lot is happening both in NYC and at Apama.

Now if the Celtics could only close out the Lakers.

Event Processing Technical Society

Another milestone in the evolution of the CEP market (I'll purposely not use the word "maturity", given the consternation that term seems to generate) was yesterday's announcement of the Event Processing Technical Society.  This consortia of vendors, academia, and other industry participants has formed to "facilitate the adoption and  effective use of event processing."  As one of the founding members, Progress is happy to see this effort get going in a more public manner, after several years of meetings and online discussion.

Not a standards body, the EPTS will otherwise work to try to advance an understanding of what event processing is and how it can be used, via promotion of use cases, best practices, academic research and other initiatives.   One factoid in announcement was a Gartner citation that a typical  large company deals with 100,000 to 1 million business events per second.”  The value of harnessing that information suggests a vibrant market opportunity that hopefully all market participants can get behind.

Taking CEP to The Next Step

Standards are an area of growing interest in the complex event processing market. What will be the role of standards.  In what aspects of CEP will they emerge? When? 

With the premise that “in the future, it will be necessary for heterogeneous CEP products to interoperate across organizational boundaries” the Object Management Group is hosting a Technical Meeting in Washington DC to address some of the issues. As part of the agenda, fellow Apama colleague and blog contributor, Matt Rothera, will be speaking on How CEP Takes the Next Step. So if you are going to be in the DC area on March 13, and interested in this topic, you definitely should check it out.

Unus per utilitas ut fulsi perspicuus ero laurifer

The Gartner CEP show in Orlando last week demonstrated a number of interesting things about CEP platforms

1. The story of core CEP is common across all vendors

2. Aside from the nature of how CEP is expressed within a platform, true differentiation starts to emerge in tooling and user constituencies that are using CEP

3. Future state as described by Dr Luckham indicates that CEP logic becomes available inside reusable libraries/repositories that can be folded into applications.

 

The premise of Apama has always been that you must embrace the full spectrum of users – encompassing IT, business analyst, end user, and senior management, to gain true productivity and application acceleration with CEP development.  Perhaps that is true for any application development platform.  But this has been a core tenet of the Apama platform since day 1 when the idea of having a platform that incorporates both a core event language aimed at programmers, as well as a metaphor for expressing & implementing event logic for the non-programmer who owns the core IP of the process.  Being able to express this information graphically - thus making it accessible and understandable, and then take action, completes the user stack we aim for with our Apama platform.

 

We have blogged about this before here and here, but what got my interest this time is the pitching of pure CEP programming approaches that are supposedly open to the core business user.  To construct a complete CEP application within a comprehensive platform you should be able to exercise the skills and knowledge of different users in a collaborative environment. Iterative and componentized development of interfaces, business logic, presentation and action come from the minds of many and in CEP all 4 elements are core to rapid, real time execution and adaptiveness.

 

The point that Dr Luckham makes about the accessibility and reuse of CEP components in CEP application construction as a future state is one which can be realized now.  The use of Smartblocks within Apama allows for the encapsulation of reusable logic that can subsequently be incorporated in other CEP processing.  For instance, logic to represent common trading algorithms or known air traffic congestion patterns or network intrusion patterns or supply chain metrics or internally developed analytic - all can be expressed as a CEP pattern.  Creating any of these as an Apama Smartblock, organizing them into meaningful catalogs for analysts to interrogate and select from, speeds application development and eliminates replication of core CEP logic within a larger implementation.

 

The mainstream adoption of CEP won't be based solely on it being cool technology (it is, but then again so was this). Being able to accelerate and redefine application construction in a real time world will further its adoption.  And maybe so will a solid and well thought out set of standards.  But that is for another time …

Thank You Gartner - Event Processing Conference #1 In the Books

Between September 19^th and 21^st, Gartner held its first conference on Complex Event Processing and Business Activity Monitoring in Orlando, Florida.  Some 200 people (by my estimation) came together to meet others interested in these technologies, as well as see and hear presentations from a range of Gartner analysts, CEP vendors, educators and thought leaders, and most importantly users of CEP.  The conference was bookended by impressive presentations from Roy Schulte and Bill Gassman on Day One setting out the current state of the CEP and BAM market, and by Dr David Luckham who closed the conference with a thoughtful and insightful look at the future of CEP. 

 

We’ll blog entries about different aspects of the conference over the coming days and weeks.  But for now it is important to stress how vital the timing of this conference was and how its attendees have shown that the principles of CEP are beginning to take hold in a wide array of industries and solutions.  Between the 3 conference tracks organized by Gartner (Event Processing, Business Activity Monitoring and Event Processing in Banking and Capital Markets) and the vendor sponsored sessions, we heard descriptions of applications of CEP in a variety of scenarios ranging from algo trading to clickstream analysis to content distribution to manufacturing and many more.

 

Architectural presentations were also prevalent with many sound ideas being put forward on the relationship between the ever evolving alphabet soup of CEP, BAM, SOA, EDA, BPM, OLTP, ESB and I am sure, many others.  Bringing together an audience such as this to discuss both practical implementations and more theoretical research allows insight to flow around the CEP community and to understand the ramifications for when CEP is seen as more than just event feeds and event processing speeds. For true application infrastructures to be built on the principles and technologies of CEP, a wide understanding of how we can evolve the relationships between these disciplines will be key.  And that understanding will come from the continued holding of conferences such as this one (already looking forward to next year in
New York) and interplay between the many disciplines, vendors and consumers of these technologies.

 

Dr Luckham posited that CEP will become a ubiquitous infrastructure technology in some 30 years.  For that to be true - indeed for it to happen sooner - we all have a lot of work to put in … but you can be sure that it will be worth it.

 

Apama SIFMA - II

As promised in the last post, the enclosed picture is the final mural, completed on Thursday of SIFMA.   All in all, a nice artistic tribute to what's been happening with Apama.  And given the amount of interest spawned by our SIFMA presence, we'll be needing a bigger canvas next year.   

Apama at SIFMA

This week has proven to be a big one for Progress Apama at the SIFMA (formerly SIA) show in NYC.  As of end of Wednesday – with one day remaining - we can already categorize the event as a big success with lots of visitors to the Apama exhibit at #2117, in the “Grand Ballroom” of the New York Hilton.  At times, there's been little room to maneuver, as people have come to hear about and see demonstrations of our recent announcements - including market surveillance (FSA),  integration of news in trading strategies (the Dow Jones Elementized News Feed), and the Apama FX Market Aggregation Accelerator. With a number of other demonstrations showcasing the application of Apama's CEP platform to algorithmic trading, it has been a challenge to accommodate all the demonstrations, while we also highlight the capabilities of fellow Progress product line, Sonic, and partners ULLINK and Microsoft, who are also demonstrating with us.

One of the areas we sought to illustrate at this year's event is the ever-expanding Apama ecosystem of customers and partners.  We’ve done this via a mural that is being created live by a local NY artist during the show. What began as a blank wall is evolving into a colorful and eye catching array of logos from many of Apama's customers and partners. The mural is still a work–in-progress, but we thought we'd provide an interim view, with a complete one to follow, once it is complete.

While a visually interesting, it's also proven an enlightening one as people see the number of organizations who are using Apama or partnering with Apama to deliver sophisticated algorithmic trading solutions.

 

 

 

Complex Event Processing at CERN

This week I visited CERN in Switzerland, the European Organisation for Nuclear Research, who is a customer of Progress. It was an astonishing and inspiring visit. CERN is in the final stages of building the Large Hadron Collider (LHC) which is due to go into production late this year. The LHC consists of a 27km loop in which protons will be accelerated and collided at unprecedented power levels to give us new insights into the building blocks of matter. In particular the search is on for the Higg's Boson, predicted originally in a paper dating from the 1960s. Finding this will fill a gap in the Standard Model of elementary particles and forces, and will help in furthering a "theory of everything". A particular highlight was to go down nearly 100m underneath the ground to look at the ATLAS experiment - a truly massive particle detector. Its enormous size consists of a number of different elements which detect different types of particles - muons, gluons and many others. The huge magnets which form part of the detector are cooled with liquid helium down to -269 degrees C to make them superconducting (and therefore more powerful). Viewing all this brought home what a remarkable engineering effort it all is.

Anyway, what has all this got to do with events? Well, through a number of presentations that CERN staff were kind enough to give us throughout the day it became apparent that their whole world is to do with events and the processing of them. The term "event" is one which they used often, to describe the information gathered by the detectors which sit around the collider. Every time a set of protons collides sets of events are created which need to be analysed and filtered to determine which are of real interest. For example, there are two ways in which a proton can decay to produce two Z particles (check). One is predicted to involve a Higg's Boson so the set of events to look for is something like "proton collision followed by a Higg's Boson followed by two Z particles". To identify such sets of temporally correlated events the raw events are propagated up through three levels of filter to be finally sent through to a central computing resource for further research and analysis. Up to 40 million collisions per second take place. These are firstly analysed in FPGA hardware reducing the 40 million collisions to a few thousand of interest. These are further filtered in software to produce finally a few hundred. These few hundred are then sent to other computing systems for further analysis.

It's not only collider events that CERN needs to handle. CERN also has a newly built central control centre, part of which is used to monitor CERN's technical infrastructure. About 35,000 separate sensors exist to monitor everything from fire, to electricity substations, to coolant plants. All these sensors are currently producing about 1.6M events per day all of which have to propagated to a central point for analysis. In turn these 1.6M are reduced to 600K events which are overviewed by human operators. Most are inconsequential (for example the 18KeV power supply is still producing 18KeV) but some will require attention. By appropriately analysing these CERN can ensure that the colliders are running as smoothly and as safely as possible. With billions of euros invested so far in the LHC, keeping the collider up and running as continually as possible is a top priority.

The visit proved a fascinating insight into the world of particle physics and the data processing challenges it produces. It really showed event processing at its most extreme.

An Overview and History of Complex Event Processing

Modern event processing grew its roots in the mid 1990’s, when academic research began at Cal Tech (Mani Chandy), Cambridge University (John Bates), and Stanford University (David Luckham).  The focus of this research was to develop a new approach to process streaming event data by identifying complex sequences of events (event A followed by B, then C), with temporal constraints (within 20 minutes), spatial constraints (within 5 miles of each other), and to control complex actions as a result of these patterns (begin warning an operator at code yellow until another event, D, is detected within 5 seconds).  An example of a CEP logic, written in a high-level representation of Apama's CEP programming language is shown at left.

Commercial software companies were created based on this research.  Chandy’s work spawned the iSpheres (acquired by Avaya last year), and Bates co-founded Apama with Giles Nelson in 1999.  In 2002, Luckham published a book called: "The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems" that helped popularize the term “complex event processing,” or CEP.  In 2003, two more vendors were founded - Coral8 (Stanford) and StreamBase (MIT / Brown / Brandeis; Michael Stonebraker / Stan Zdonick / Mitch Cherniack).  TIBCO BusinessEvents entered the market in 2004 as well.  Other CEP firms include Kaskad and Aptsoft.

Recently CEP has experienced rapid growth and adoption, and has publically recognized in a variety of industries with documented use cases.  Capital markets was the first market to heavily utilize CEP it forms the basis of algorithmic trading (read this tutorial on algorithmic trading) applications in top firms like Deutsche Bank, ABN Amro, and JP Morgan (read this tutorial on CEP and algorithmic trading in Doctor Dobb's).  Apama's CEP engine has been customized as an algorithmic trading platform through the addition of market connectivity, algorithms, and pre-packaged user interfaces that implement user interfaces such as algorithmic order management, market aggregation, and trading surveillence (details here).  Other uses have published from other applications in the capital markets, such as exchange surveillance (e.g., Kaskad's at the Boston Stock Exchange), and in smart order routing (SRO), market making, MiFID compliance (Microsoft and Apama and Aleri Labs), bond pricing, and more. Outside of capital markets, CEP has been applied to a wide range of other applications.  Another significant area of adoption for CEP in in the travel industry, where TIBCO's BusinessEvents was selected for Airlines Operations Monitoring by Air France / KLM, and many other airlines have announced projects to automate operations.  TIBCO has announced the use of their CEP product in telecommunications, used by Telecom Italia for their entertainment-on-demand portal.  Coral8 is being used for patient care in the health care industry,  StreamBase is being used in the gaming industry, and Apama is used to manage real-time supply chain and retail operations in a deployed item-level RFID tagging systems like BGN in Holland.

As a result, the event processing market is now considered an independent software market, and includes commercial software for complex event processing (CEP), business activity monitoring, and event data management (aka data stream management). Gartner announced its inaugural event processing industry event which will cover event processing and BAM, firms like AITE group and Bloor have published reports on the event processing industry and software vendors and applications. 

So CEP is now being labelled as a new paradigm of computing as being compared to business intelligence in terms of potential impact on the market as a way to monitor, analyze, and act on events of any type, as a way to add intelligence and action to events in any kind of application.