Event Processing blog

Real-Time Temperature Monitoring

In an interesting demonstration of real-time monitoring, Apama participated in a recent study that was conducted by Radboud University as part of the Dutch "Four Days March".  The march is an annual event in which tens of thousands of participants walk daily distances of 30-50 KMs for four days in succession.  Unfortunately, past marches have had some unfortunate instances where participants have been overcome by heat - including a couple of deaths.

In a pilot study this year, volunteers took a pill embedded with an RFID chip and thermometer, which sent signals every 10 seconds to a Bluetooth-enabled GPS phone.  The phone in turn sent the information to an implementation of Apama, which correlated the information about the volunteer, their temperature and their location, all plotted on an implementation of Google Maps. Leveraging the capabilities of Apama, monitors could track the progress of volunteers and identify those volunteers whose temperatures exceeded certain thresholds.  And with the ability to correlate that information with the GPS data, they could tell where those volunteers were on the route, thus delivering a CEP-driven infrastructure for real-time monitoring and, if needed, a pre-emptive reaction to help somebody who might shortly be in distress.

In the image below, one can see the purple "pins", illustrating where on the route the volunteers are.

Rendering Unto Caesar - The Role of the Business User in CEP

"Render unto Caesar the things which are Caesar's
and unto God the things that are God's"

A recent posting in the Enterprise Decision Management Blog entitled "Can we trust business users" addresses a topic that seems equally pertinent to the CEP market. I think there is a tendency to become so enamored with the technical virtuosity of new technology that we may lose sight of who the real users are. In terms of the development of CEP applications, an understanding of the prospective roles of business users vs. that of IT developers seems to be an evolving thing. Apama has long been a proponent of the participation of the business user in the process of building CEP-driven applications. In Capital Markets, the notion of "empowering the trader" has been a key element of our strategy and the Apama product offers a graphical development tool, Event Modeler, that focuses on that constituency. We have another RAD tool that is intended for developers who can create applications in our event processing language, as well.

We are beginning to see third party validation of the value of this approach from outside of Capital Markets, as well. For example, a report from Forrester Research published earlier this year indicated that early adopters of CEP have tended to come from the line of business rather than IT because "developers and architects often know painfully little about these [non-traditional, non-IT] events and how they are used to run a business." That Forrester quote is certainly not intended to diss IT. It just recognizes that there are lots of different kinds of events that are important to a business and not all those events are traditional IT-aware or IT-generated events. In order to make sense and respond to such events, it seems quite logical that providing tools that are amenable to a more "business" oriented audience is important.

But I would argue that it is not just the nature of events - and their varied sources - that suggests a strong correlation between CEP and business users. It is also the nature of the CEP-driven applications themselves. CEP applications are not "one and done", they tend to be iterative and evolving, because they are crafted to respond to what is happening and what is happening is often a frequently changing set of conditions. Or, if the conditions are not changing, how you choose to respond to them may be changing. So you need to continually calibrate and revise.

In another Forrester report published earlier this year, this characteristic was noted within the context of a review of Business Activity Monitoring best practices. Event-driven BAM is a particularly strong use case for CEP and the report stated that BAM efforts "are typically never-ending projects" with a "high degree of change." That makes sense since BAM monitors 'business activities' and the nature of most businesses will change over time. So to support BAM applications, it seems perfectly logical to provide tools for business users who can take on some role in the initial development and ongoing operational calibration of these applications. There is clearly an important role for developers in building these applications – no one would suggest otherwise, but we best not forget what the “B” in BAM refers to.

What seems to be emerging is the notion that we are should not look at CEP and/or BAM deployments as discrete, finite projects with clearly prescribed end dates. They are continuously iterative projects that must evolve to remain effective. That’s the environment in which they operate. Given that, perhaps we should not see the roles of business users and IT as fitting within well prescribed boundaries. The development and ongoing management of these applications will have evolving roles for the line of business user and for IT over time. We might expect IT-centric development to have a more dominant role in the initial deployment, but over time the goal might be to have the line-of-business assume greater and greater roles - because the business will be the dominant user and best positioned to react to changing circumstances.

Perhaps the EDM blog posting says it best, though it expresses it within a "business rules" context. “Too many rules specialists focus on rules that will get executed and not enough on the people that will maintain those rules, although this is where the success of the project resides.” That is quite the same for CEP and BAM. There is a role for business users, driven by the nature of events and the continuously evolving nature of the applications that are “event-driven.” And it is incumbent on the technology provider to offer tools that will facilitate that evolution. All the CEP performance in the world will be of little use, unless that performance is well-aligned with the needs of the business.

So we might debate who is the metaphorical Caesar and who is God in CEP development, but the success may well rest on giving each their due.

Apama's Fit in the Future of FX

It seems like only yesterday that Atriax (2002) died leaving FXall and FXConnect the only multibank portals offering the Buy Side “competitive” FX prices via RFQ pricing from the banks.

In the past few years we’ve seen exponential growth in the number of FX trading venues and the services that they offer. No longer are those venues which provide streaming prices and anonymous matching exclusive to the banks, with even the electronic FX trading duopoly of the 80’s & 90’s forced to allow Buy Side participants a seat at the table (via Prime Brokerage) because of the competition.

The changes have blurred the roles between the Buy & Sell Sides, primarily benefiting the Buy Side through greater transparency, enhanced prices and real-time market access. To the Sell Side, ECN may no longer be a term that evokes hatred but it’s difficult for them not to remember the “good ole days” before paper thin spreads and anonymity.

In today’s ever fragmenting environment Complex Event Processing (CEP) has already solved many problems for both the Buy & Sell Side. We have customers who’ve automated a variety tasks, produced both Trading and Execution Algorithms and claim that they’re doing twenty times the number of trades with the same number of dealers.

In the past year FX Aggregation has been “the flavour of the month.” I particularly liked it when a senior executive who saw our system and simply said “There is no fragmentation. With this system my dealers can access the whole market through just one screen.”  (learn more)

The May Edition of Profit and Loss (P&L) contained an article called the “Shifting Landscape: DMA, BANKS AND ECNS” which said “many refer to today as the Golden Age of ECNs.” It pointed out how the “Credit Crisis” and the events of last August forced many in the Buy Side to rethink the importance of having a relationship with banks to ensure access to the liquidity banks is maintained when other pools start drying up. 

Although P&L doesn’t see the death of ECN’s, the article made a number of observations and predictions on how the banks shall reassert themselves in the ever changing landscape of FX. I’m writing to comment about CEP’s role in what’s being predicted.

The cornerstone to P&L’s future lies in banks offering Direct Market Access (DMA) to their customers to guarantee transparency and best execution. DMA in FX of course will have its own intricacies, but this is an evolution not a revolution as DMA has been part of the Equity, Futures and Options markets for years.

Through FX Aggregators banks are already merging the fragmented FX market to guarantee their own access to best prices and deeper liquidity. The extension of this to their customers for DMA is relatively straight forward in much the same way that the Sell Side currently offers DMA via Smart Order Routers in the Equity space.

The next prediction is that banks will add their own liquidity to the aggregated pool and, when adequate numbers of customers are connected, to allow them to transact with each other. This sounds very much like a Crossing Network and a perfect home for a very fast CEP engine that can evaluate streams or orders, looking for matches.

Finally, P&L sees the banks interconnecting their systems to extend the liquidity pool, thus becoming ECN’s in their own right. Will this lead to the Buy Side then Aggregating the Aggregators?

Sprinkle in a few trading, execution and liquidity seeking algorithms and what P&L is predicting makes a lot of sense.

So what does this mean and what problems does it pose? It means that trading banks must be capable to quickly produce and deploy new services and be proficient in adapting these to address ever evolving requirements from the market (i.e. be swift & agile). The problem is many organisations aren’t particularly good at rapid development and deployment.  Those who are shall hold a real competitive advantage going forward.

Complex Event Processing fits the infrastructural needs for DMA, Aggregation & Crossing Networks well, but it’s not good enough to simply have a very fast engine. The engine must readily connect to a disparate world, provide tools for rapid development and back testing facilities to ensure successful deployments.

Apama fits all these requirements as it’s not only a very, very fast engine but has graphical tools that enable both business and IT staff to collaborate and rapidly produce solutions, back testing facilities to mitigate deployment risk and an open integration framework that includes many existing “off the shelf” adaptors.

It’ll be five years before my 20/20 hindsight enables me to recall what our future was. For now the only thing that I can say with certainty is Complex Event Processing and Apama will be playing a major role in whatever the future of FX holds.

SIFMA Retrospective

As a follow-up to my colleague Louis' report on last week's SIFMA show, I thought I'd add some thoughts of my own. My conclusion is that it was the most exciting SIFMA show I have experienced.  While I think attendance was down from previous years, I also think the quality of attendees was up. And for me personally, the excitement of being involved in some industry-moving announcements as well as meeting up with many of my colleagues from capital markets firms, vendors, press and analysts was highly invigorating.

So what were the highlights and take-aways for me?

1. CEP is clearly a theme that is getting a lot of mindshare. So many people said that CEP was a key theme of the show – which is great to hear after many years of working to help define the market. It’s also great to see this add to the momentum so soon after the Event Processing Technical Society was launched. The use cases of CEP are many and varied – and there was a lot of interest and questions around this at SIFMA. We demonstrated on our SIFMA booth 5 different CEP use cases on 5 different pods - algorithmic trading, smart order routing, managing FX market fragmentation, market surveillance and real-time bond pricing. Also the demands of CEP applications continue to make demands on the technology, and we were thrilled to demonstrate Apama 4.0 – which extends performance and user experience of CEP to new levels. Another supporting factor in the maturing of CEP is that there are starting to be very senior people in Capital Markets firms focusing on CEP as an enabling technology. Marc Adler from Citigroup   is a key example. He’s active in the community and on the STAC CEP committee, helping to define benchmarks. It was great to meet Marc at SIFMA and also to catch up with many  other esteemed colleagues from the CEP space.

2. The liquidity wars are hotting up. It was our pleasure to be involved in a press release with NYSE-Euronext which was certainly one of the big releases of the conference. Progress Apama will be hosted in the NYSE Euronext as part of the exchange's Advanced Trading Solutions offering. Traders will be able to download custom logic for algorithmic trading, risk management and smart order routing into the NYSE itself - with low latency connectivity to other trading venues via Wombat and Transact Tools. This arrangement turns NYSE into a technology provider as well as a one-stop-shop liquidity provider. This announcement was picked up by major press, including the Financial Times - in Europe, America and Asia -- see the article here.

3. Hardware is important – and so is “green”. The increase of capital markets data volumes require completely new software architectures – like CEP. But software is not always enough to support the low latency transport , processing and storage requirements. Many firms are turning to specialized hardware, combined with software – to create high performance solutions. Vhayu, for example, launched Squeezer – which combines hardware and software to supercharge their tick data offering. Also, Progress Apama were pleased to put out a joint announcement with Sun on a collaboration for end-to-end CEP solutions – combining Sun hardware and operating systems with Apama’s CEP platform and solutions. We demonstrated an end-to-end bond pricing application using the whole stack. Sun was one of the vendors who have a “green” aspect to their hardware – for example on a major CEP deployment, the hardware can be scoped for peak throughput – but can selectively shut down capacity to save power when event throughput is reduced. In this era of high energy costs and global warming there seems to be a lot of interest in this approach.

4. I love partying on the trading floor. Progress Apama were honored to be invited to a party at the NYSE to celebrate the latest developments at NYSE-Euronext (see picture at the top). It was a great pleasure to speak with our friends at NYSE-Euronext and to meet many of our old friends from the capital markets industry there – while sipping some delicious wine in that amazing place. In a way it is a shame that electronic trading is making the traditional trading floor a thing of the past – but there is something amazing about that place and I hope it stays just the way it is – even if it becomes a cool venue for other purposes. Thanks to NYSE-Euronext for inviting us – we had a great time.

I’m sure I’ve neglected a load of other trends and themes – but there’s my brain dump for the day. I’m interested to hear if you all agree.

John

Successful languages - show me the code please

The SIFMA show has just ended. It was a great success, we announced a few new partnerships and our latest version, Apama 4.0. We had a constant flow of people coming by our booth which of course increased dramatically at 3:00 pm each day when we opened the bar. We also had a magician who mesmerized the crowd with his sleight-of-hand exploits.  While our fearless leader John Bates had a seemingly endless stream of journalist and analyst interviews, myself and my colleagues did a three day tour of (booth) duty.  In addition to showing off the new Apama 4.0 Studio, we had demonstrations of our various Accelerators in Pricing, Smart Order Routing, Market Surveillance, FX Aggregation and good ol' Algo Trading.

SIFMA is a true technology show, vendors large and small had dazzling displays of their wares - hard and soft. For the average attendee it was the quintessential kid in a candy store experience. It was truly geek heaven.  My compatriots and I fielded a wide range of questions about the Apama platform, from the basic explanation of CEP in Capital Markets to how Apama is deployed in a wide range of asset classes.  A consistent theme I heard from many of the attendees coming by our booth was  "show me some code". The challenge of course was explaining a programming language in five minutes. It made me think of a recent blog by Mark Tsimelzon of Coral8 on what makes a programming language successful.

One of the most prominent characteristics of CEP, yet one of the most contentious is language. Mark's reference to slashdot links to a well written tutorial by Daniel Pietraru on the success of general-purpose programming languages and the likelihood of newcomers (i.e. Ruby, Python) unseating the mainstays (i.e. Java, C++). In a nutshell the answer to that question is an emphatic no. Interestingly, there are numerous aspects of this research on language that are applicable to CEP.

Event Processing Languages come in many shapes and sizes. They build upon prior art deriving their base syntax and semantic logic from a variety of older languages, whether that's SQL or general purpose languages like Java or C++. The marketing departments of all vendors trumpet the merits of their chosen course. In the past, I certainly have not been too quiet about this particular topic myself (although I have soften a bit lately).  The very fact that the language of CEP is so fractured among vendors is a clear sign of CEP's lack of maturity.  Yet it's what makes us unique, it's our special sauce and quite frankly gives this blog community the level of interest it enjoys. Standardization and commoditization has that Borg sense of sameness that I find a bit dull and boring (oh I've probably just incited a riot by making that statement).

There are a number of attributes that make for a successful programming language. The mainstays of C, C++, Java and (now) C# hold a commanding 49.915% popularity rating. Daniel Pietraru ascribes this success to a number of rationale, the first being similar syntax. A recognizable syntax is what gives languages that sense of familiarity as Mark Tsimelzon so thoughtfully pointed out.  But I believe there is one aspect of a programming language that is perhaps the most important of all, that of readability. Readability plays a huge role in the long term survival of not only a language but the software projects built in it.  The Java language, as shown in the popularity chart of this tutorial,  holds a commanding lead (20.176%) over all other general purpose languages. This success is arguably due to a natural evolution, a Darwinian survival of the fittest so to speak (or most popular as the case may be). It's authors wisely pruned the obtuse and wildly unreadable aspects of C++.  It's interesting to note where SQL (or PL/SQL to be specific) ranks, you'll have to check that for yourself.

How does all this relate to the language of CEP? The same set of characteristics still ring true. A familiar recognizable syntax is important yet readability is vitally important. This is the approach we at Apama have taken. Our EPL, MonitorScript is quite purposed for the event processing paradigm yet has the familiar readability of those mainstay languages like Java.  MonitorScript is predominately an imperative programming language with declarative constructs purposed for the event paradigm. Yet even the declarative on all Tick(symbol="IBM") is an easily understood concept.  In a short five minute session a Java, C++ or C# programmer will be able to not only see a familiar syntax but also a readable semantic of even a reasonably complex MonitorScript application.  This makes the first impression of Apama and our EPL a good one.

 

SIFMA 2008 - Hot and Busy

This week is the SIFMA (Securities Industry and Financial Markets Association) technology management conference in NYC.  The city has been brutally hot and the NYC Hilton exhibit hall, at least where Apama is, has not been measurably cooler. Those $5.00 bottles of water that they sell in the exhibit hall might represent opportunism - or capitalism - at its best. Or maybe it’s just NYC. :-)

First day attendance has been quite good and I’d encourage any attending to stop by the Apama exhibit (Grand Ballroom, Level 2, #2415). You’ll see a range of demos, including our new 4.0 release, and see some of the “magic” of CEP in action (you’ll have to come by to understand the magic reference). A lot has been happening with Apama and we’ve used this show to make some significant announcements. 

• A partnership with NYSE Euronext Advanced Trading Solutions that enables NYSE Euronext ATS to offer its customers Apama-based CEP capabilities for algorithmic trading, real-time risk management, and smart order routing. The announcement has received significant press pickup, including a nice story in the Financial Times.
• A technology development collaboration with Sun Microsystems involving support of the Apama platform on Sun x64 systems, running the Solaris 10, as well as the launch of a new Apama “Accelerator”, the Apama Real-time Pricing Accelerator. Apama continues to focus on application templates that enhance the speed with which solutions can be deployed and this Accelerator extends that theme into the area of market-making for bonds and other instruments.
• Announcement of the upcoming availability of above-mentioned Apama 4.0, which introduces a new development environment, Apama Studio. Studio brings all the Apama development tools together within a single, integrated environment, complete with tutorials, sample applications and other aids. You might consider Apama 4.0 as, at its heart, a CEP Accelerator. 4.0 also includes significant enhancements to Apama’s execution performance, partially achieved via a new event messaging transport.

So a lot is happening both in NYC and at Apama.

Now if the Celtics could only close out the Lakers.

Event Processing Technical Society

Another milestone in the evolution of the CEP market (I'll purposely not use the word "maturity", given the consternation that term seems to generate) was yesterday's announcement of the Event Processing Technical Society.  This consortia of vendors, academia, and other industry participants has formed to "facilitate the adoption and  effective use of event processing."  As one of the founding members, Progress is happy to see this effort get going in a more public manner, after several years of meetings and online discussion.

Not a standards body, the EPTS will otherwise work to try to advance an understanding of what event processing is and how it can be used, via promotion of use cases, best practices, academic research and other initiatives.   One factoid in announcement was a Gartner citation that a typical  large company deals with 100,000 to 1 million business events per second.”  The value of harnessing that information suggests a vibrant market opportunity that hopefully all market participants can get behind.

CEP Maturity Models

                                                                                                                                               

With the contentious debate on CEP maturity brewing, Tim Bass is using the Garner Hype Cycle to indicate that CEP is in the Technology Trigger phase - certainly an arguable point. Considering the next two phases in the Gartner Hype Cycle, are "Peak of Inflated Expectations" and "Trough of Disillusionment" I'm not so sure we are at such an early phase. Those two terms give the illusion CEP is very nascent and I personally don't think so. One's view of the maturity of a software platform is predicated on past experiences and use cases. In my most recent blog on high availability the idea of a mandate on maturity was a point I was attempting to convey in the notion of Lost Opportunity vs. Loss-Less.  A Loss-Less use case clearly requires a mature platform to support such a business critical function.

As with most software infrastructure platforms, CEP being no exception, one can describe or categorize multiple maturity models. What a platform has and what a platform does.   

A CEP platform has (or should have) development tools, deployment & management tools, connectivity adapters, database adapters, dashboards, a robust architecture for reliability, scalability and high availability.  All of these infrastructure capabilities have a maturity life cycle within a CEP platform and are paramount to customer IT organizations responsible for the care and feeding of applications. What a CEP platform does refers to what sort of applications one can build with the technology. Can one just do simple pattern detection or infinitely more complex analytics?

These two categories have independent maturity life cycles but are inter-related. To illustrate consider the following fictitious example. Say we have an application whose sole purpose is to count. What this application does is count an integer number continually for each connected client. It simply needs to count upwards - 7x24 without missing a beat, failure to count means catastrophic business failure with financial and legal ramifications. It starts out counting for just a few clients but over time grows to support thousands or tens of thousands of clients.   In order to support the deployment of our application, we need a platform with the reliability demanded of a 7x24 operation. A high availability architecture is mandated in the event of a failure and it must be able to scale as the business grows.  This simple example illustrates the point that even the simplest application, if critical to the business needs a mature platform.  Just substitute a counting application for any more complex use-case - Smart Order Routing, Dark Pool trading, Fraud detection, etc. 

What a CEP platform has tracks independently of what it is capable of doing. The maturity of what a CEP platform has is much more quantifiable, we just need to look at other platforms such as enterprise class Application Servers, Message Systems and Database systems and for the most part follow suit. What CEP does, is likely what Tim is referring to when he states we're in the Technology Trigger phase.  This maturation process will drive much more slowly and will expand as new use cases are discovered.

Apama Wins FX Algo Award

The Apama platform notched another award last week, this time at Forex Network New York, an event hosted by Profit & Loss Magazine on May 22nd. Apama was selected to receive a Digital Market Award as the Best Algorithmic Trading Platform for its ability to power unique FX trading strategies.  FX has proven to be a particularly strong market opportunity for the Apama platform's CEP capabilities. With its breadth of connectivity to FX liquidity venues and rich set of tools, Apama is quite attractive to organizations who want to quickly capitalize on FX trading opportunities with their own unique strategies.   

Waters Survey - Pick Your CEP Vendor

While it may not have the excitement of Hillary vs. Barack (or David vs David for you American Idol fans), we would nevertheless encourage all those eligible (investment firms, hedge funds, brokerages and exchanges) to vote in the Annual Waters Survey, currently being conducted.  Progress is a contestant in two places - for Data Stream Management (#11) and CEP (#20).  Not all the products included in those categories really belong there, but we'll leave it to the voters to reach those conclusions.  Voting ends midnight, May 28th.