Event Processing blog

The Financial Meltdown and the impact on CEP

Watching, waiting, wondering are the catch phrases of late. What will be the eventual conclusion of the meltdown and subsequent bailout of our financial system? Like many of you, I've been steadfast in my search for information, watching the news, reading blogs and op-ed's about how we got into this mess. I anxiously await our government's response. As a homeowner and family man with a retirement plan I am keen to understand this predicament. I also have a vested interest from a professional sense. The event processing technology in which we immerse ourselves is well entrenched in this same financial community.  Below I share a few thoughts and perspectives on our troubled financial times and how it might impact event processing technology.

Short Selling induced volatility

The idea behind short-sell trading is to sell an asset (i.e. shares of stock) that you the seller don't really own.  Then with luck the price drops, you buy the shares at the new lower price return the shares you borrowed and pocket the difference. Short selling strategies are common place in equities, currencies and futures markets and have been in use for a number of years. Which of course came to a complete halt when the FSA and the SEC banned short selling in an effort to stave off market instability. The FSA's clampdown gaged only a small number financial instruments, whereas the SEC's action was more punitive, setting restrictions on 800 companies. Once those regulatory bodies announced the crackdown the investment bank community, hedge funds and asset managers reluctantly scaled back their short selling strategies to comply. Short selling is a common use-case for CEP.  These recent restrictions are a clear indication of the need for algo strategies to be dynamically adaptable to market or regulatory conditions, whether in short selling or other types of strategies. It's imperative that the development tools provide the means to swiftly enable changes to strategies.

All told, the shorting restrictions were applied to those instruments that had massive impact on the market's stability. Just determining the market impact, the instability or general volatility was an investigative research project where CEP could have played a significant role. Volatility is a statistical measure of the scale of fluctuations in a price or index. By looking at historical norms, the FSA concluded 29 stocked exceeded those historical norms and the SEC determined it was 800. While I don't know if either regulatory body used a CEP product they clearly could have. Used in conjunction with a tick database for the historical market data, a CEP product like Apama provides the tool set and language to rapidly construct a market impact analysis solution with the ability to carry-out a multi-year analysis. 

Leveraged induced risk

Investment banks rely heavily on borrowed money, so much so the typical "leverage ratio" is 30 to 1. That means for every tangible dollar held, 30 are borrowed. While leverage can and does create the opportunity for huge windfall profits, it can also incur massive risk and huge losses. As long as markets are reasonably stable it allows for predictable behavior of investment strategies thus permitting banks to make money on borrowed money and continue to leverage.  However, as instability invades it erodes many aspects of investment strategies. Losses begin to mount -  on the investment itself, on the borrowed amount and the interest payment of the borrowed sum.

So how can CEP play a more active role in this highly leveraged world? With risk mitigation.  CEP is emerging as a key constituent in real-time risk management systems. One example is in deal-monitoring / position keeping systems. Auto-hedging is also a technique employed by risk systems to keep positions within the bounds of tolerable risk limits. One form of hedging is short-selling (see above). Risk management systems define allowable tolerances for trading, positions and P&L. With a 30 to 1 leverage ratio those tolerance limits seem terribly high. So when the market takes a turn for the worse, losses become astronomical. This is not the fault of the risk systems themselves but the allowable limits established by the bank's business then coded into the risk systems that permit highly leverage positions.   One can only image that investment banks will be adjusting the limits on their risk downward a few notches. It will also spark the opportunity for them to invest in newer more flexible real-time risk solutions. In volatile, ever changing market conditions risk systems need to be dynamically adaptable. CEP is an ideal technology to fill that need.

Reclassified Investment Banks

The last two major investment banks, Goldman Sachs and Morgan Stanley, have recently changed status from an investment bank to a commercial bank. In doing so they can now take deposits in the form of checking and savings accounts as a source of funds to shore up losses created by their highly leveraged positions. But commercial bank status means they face a litany of Federal regulation. For one, the 30 to 1 leverage ratio will be prohibited. The days of windfall profits in good times (and catastrophic losses in bad times) are over.  The Fed's also provide a safety net in the form of an emergency loan program for commercial banks. Goldman and Morgan now qualify to be protected by this net, thus preventing them from suffering the same fate as Lehman. I'm sure they had this in mind when they requested the change of status.  I wonder if Goldman and Morgan will be buying new name plates for the front entrances to signify their new commercial designation?

"In every crisis, opportunity" could be an appropriate catch phrase for the eternal optimist. While this meltdown certainly has the appearance of doom and gloom, in the end banks are in the business to make money and they will find a way. Event processing technology will be at the fore-front of that endeavor. Regulations will simply draw a box around which they operate. It will stir the creative spirit to achieve within these new boundaries. New forms of algo strategies, real-time risk monitoring and surveillance will be required as a result of the imposed bans and regulations. CEP technology - the Apama platform, will be at the forefront of this new (banking) world order.

Reflections on the Gartner Conference and EPTS4

Like many of my colleagues in the event processing community, I thought I would share a few reflections on the recent happens at the two back-to-back technology conferences of the past week. Gartner sponsored their annual vendor-fest known as the Event Processing Summit, and the EPTS had their fourth annual symposium. This being my first EPTS, I've had some initial thoughts and reactions which I've shared over the weekend.  For this, I'll delve more into the conference's content.

I attended a number of the sessions at the Gartner conference. I did not have any set agenda so I picked the sessions more on a personal appeal rather than some well thought out plan. While I do work in an engineering team, I have a customer focus so I attended all the customer sessions. I always find it valuable to understand how customers are deploying event processing technology in real-world use cases. Their efforts clearly infiltrate the product roadmap of vendors.

 
• Lou Morgan of HG Trading, a lively speaker described his use of event processing technology in high frequency trading. Lou has been an Apama user for quite a few years and we've invited him to speak on our behalf on a number of occasions. He's an entertaining soul with a clear understanding of the Capital Markets business. We're delighted he presented his use of Apama at this conference.

 
• Albert Doolittle of  George Weiss Associates Inc. gave a talk on using event processing technologies in this firm.  Albert described his technique to pick a vendor for his CEP project, which if I were to paraphrase was a coin flip.  Towards the end of his talk, he digressed from CEP technologies to present a short discourse on high performance computing (HPC). The idea of leveraging supercomputing-like technologies and FPGA's for compute intensive operations like Black-Sholes Options pricing certainly has caught Mr. Doolittle's attention. Typically CEP and compute intensive tasks don't mix well because of latency considerations. However, a marriage of CEP and HPC is possibly one made in heaven. I was intrigued.

 
• The ebullient Marc Alder gave his brusque, no-holds-barred perspective on the CEP project he embarked on at Citi. Marc did a great job of explaining the challenges of introducing a new technology at a large corporation, one with a well entrenched bureaucratic IT organization.  I think most of us have faced the bureaucratic fortress at some time or another in our careers. Knowing how to play the game is a skill only a few master well, kudos to Marc for his successful venture.  As Marc unfolded his project's architecture he wisely chose a course to prevent vendor lock-in.

The juxtaposition of these three use-cases was most curious. Lou Morgan jumped deep into CEP technology and bet-the-ranch on it. Albert Doolittle took a gamble with a coin flip in choosing a vendor and Marc Alder kept his choice of a CEP product isolated and contained within his overall system architecture. A safeguard in case he felt the need to replace it.  Nonetheless all great examples of how CEP is gaining momentum in main stream business.

One session I thoroughly enjoyed was Don  DeLoach's "Extending the range of CEP". Don is the CEO of Aleri. I'm not sure I enjoyed this session more for its content or for Don's presentation skills. As is usually the case at technology conferences, it's death-by-Powerpoint. Slideware is typically jammed with an overabundance of barely readable text and dazzling graphics.  Don's slides however had a clear minimalist slant. A plain monotone background with either a single word or a (very) short phase well choreographed with his oration. He spoke of CEP as an evolving technology from the simple ability to filter streaming data to managing complex application state. He used an example that has become the Pièce de résistance of Aleri, order book consolidation.

There were many sessions on SOA and Event Driven Architectures - so many I lost count. 

I attended the panel discussion on low-latency messaging protocols. This was a Q&A session moderated by Roy Schulte of Gartner. The panelists were the crop of high-speed/low-latency message vendors. TIBCO-killers as I've affectionately referred to them. Vendors such as 29West, RTI, Solace Systems, IBM and even TIBCO themselves (apologies to those vendors I've not mentioned). Each described how they have defied physics to achieve incredible speeds yet still provide reliable delivery, management tools and even application level services (i.e. RTI's last value cache).  However, its noteworthy to contrast these low-latency vendors, all focused on shaving microseconds off message delivery via proprietary, even hardware-based schemes, to the many standard-based messaging systems trumpeted in other sessions. Those SOA and EDA sessions paraded a whole barrage of Web Services based standards models (i.e. WSDL, WS-Eventing, WS-Notification, WSDM, the list goes on and on) as the right way to build applications. These certainly seem like opposing forces that will only foster confusion in the eyes of those who have a clear business need for low-latency yet desire to adhere to a standards approach.

The EPTS Symposium began its first day with a keynote address from a VC which had funded Event Zero.  I had first met with Event Zero about a year ago, they have appeared to recast themselves from an adapter/connectivity vendor to one delivering an Event Processing Network (EPN). An EPN can be defined as an infrastructure platform for event processing agents or services. Those CEP agents performing both independently and in concert with other agents (or services) act upon streaming data sources. Together the whole becomes greater than the sum of the parts. Such is the grandiose vision of an EPN.  SRI was also promoting a similar notion of event processing as a service, which I would argue is a variation on this same theme.  Unfortunately, I think there is trouble ahead. The problem is simply timing, maturity and standards (or lack thereof).  I don’t think customers will buy into EPN's or Event Zero's vision until there is a clear establishment of standards for CEP. As a perspective, Application Server vendors tried this and failed (anyone remember SilverStream? Apptivity?). It was not until the J2EE specification established a uniform model that created true viability for a network or service infrastructure platform for AppServers.  Until we see the formation of CEP standards for interoperability and integration, the appeal of CEP will remain as basically a standalone application platform and vendors will continue to market a solutions approach, just look at any CEP vendor's website for proof of this. Nonetheless, Event Zero has embarked on a bold initiative and I wish them all the best.

Speaking of standards, moving slightly up the stack one could clearly detect the prevailing wind blowing against streaming SQL as the language of choice for CEP.  Going back to the Gartner conference there were a few noticeable comments to that effect. Marc Adler, described streaming SQL as making the simple things difficult to do.  Don DeLoach, downplayed the SQL language in Aleri in favor of the SPLASH enhancements. The renowned Dr. Luckham in his closing keynote address outlined Holistic Event Processing as the future implied it required a language beyond streaming SQL. 

At the EPTS Alex Koslenkov from Betfair castigated the streaming SQL approach for his use case in managing complex long-running state. Alex is an advocate of the RuleML approach to CEP languages, as such it stands to reason he doesn't have a high regard for streaming SQL and it showed.

Susan Urban from Texas Tech University presented a research project on a language they've dubbed StreamCEDL. Susan denounced streaming SQL as lacking the algebraic expressiveness necessary to move beyond simple stream processing to true complex event processing. One example, she mentioned in the description of StreamCEDL is its support of an APERIODIC operator.  The intent is to process irregular or out-of-order data streams.

Lastly, Chris Ferris from IBM presented on Industry Software Standards. This was a great session that portrayed the far reaching impact of adopting standards across our industry.  He stressed the importance in making every attempt to get broad vendor agreement, customer validation and to be sure the adopted technology serves the needs of the community because you'll have to live with it for years to come.  This is such an important message in the quest for standardization of CEP. Open, widely accepted standards are exactly what the CEP community needs; the sooner we embark on this journey the better.

A Truce at the CEP Front

                                                       

I am a bit of a history buff and often times I'm reminded of some historical event when reading about current events. This inclination I have can easily be applied to the meltdown of the global financial markets we see happening all around us. The lessons of the past should be constant reminders of how we should behave now and in the future. I've always thought a degree in history should be a prerequisite to a political life, armed with such knowledge would clearly provide guidance to govern wisely. Maybe our business leaders should follow a similar career path.

I've just attended my first EPTS Symposium. It was the Technical Society's 4th annual get together. If you're unfamiliar with this organization, it's purpose is to promote event processing technologies through academic research and industry participation. The organization has a number of working groups that have contributed greatly to the overall awareness of event processing.  You can read more about the EPTS at their website.

The symposium was well attended by members of both academia and industry. All the major CEP vendors were there and it was the first time I've been in a setting where the atmosphere was completely non-competitive. It was a truce of sorts. While we typically wage war in the virtual battlefield in a land-grab for customers, for 2 days we discussed vision, standards and use-cases. We debated ideas, but we also laughed, ate and drank together. It was general camaraderie. As I mentioned, history is one of my interests and these two days reminded me of the 1914 Christmas Truce where the Germans and the Brits crawled out of their trenches and met in the no man's land to celebrate Christmas together. The guns fell silent that night in 1914 and for the 2 days of the symposium the virtual guns of competition also fell silent.

Come Monday we'll all be back at the war again. But for a short while it was fun. To see the face of the enemy unmasked, to get to know him, to share an idea and a drink was genuinely uplifting. We found common ground in our desire to see event processing become a main stream technology.

Another CEP TLA - BOCI

Colleague Giles Nelson recently commented on the fixation on TLAs and their meanings - a fixation that has seemed to dominate the CEP market's online discussion of late. He suggests that it obscures what is important in terms of coming to appreciate what is really the value of CEP. I tend to agree, though it can be entertaining to monitor the lively - but not necessarily illuminating - debate about internet routers and what kind of router best serves as a metaphor for CEP.

But rather than dwell on real or metaphorical routers, I'll attempt here to interject a bit more substantive news - another Apama customer win, this time the Bank of China.  For purposes of this posting lets call them BOCI (Bank of China International Holdings) to ensure this posting has enough TLA critical mass - though admittedly, BOCI is really an FLA. BOCI has chosen the Progress Apama CEP platform to support algorithmic trading in equities, futures, futures indices, warrants and bonds. With Apama they can receive market data concurrently from both SEHK and HKFE (the Hong Kong Stock and Futures Exchanges respectively) and algorithmically place orders into different sub-markets on these exchanges.

Now, there may be some debate somewhere as to whether this is really CEP. We cannot always provide details regarding the specifics of what a client does; in some instances the client does not share that detail, as they deem it too proprietary and valuable to their business. But the intricacies of trading in such complex markets (with transient liquidity and the need to act quickly in order to capitalize on that liquidity) make these applications excellent CEP use cases. Not all trading applications may represent CEP, but these applications indeed qualify.

TLAs vs. TLIs

On a side note, kudos to Giles Nelson for his posting's reference to "initialisms" vs acronyms.  It prompted a bit of research on the differences between acronyms and initialisms. As it turns out, BOCI is an initialism, not an acronym.  Likewise, so is CEP and perhaps many of the common shorthand abbreviations that dominate technology. If BOCI were an acronym, you would say it like the Italian game: bocce.

So, as far as Apama is concerned, the choices are

Our BOCI is on the far left. :-)

SQL Standards - an impedance mismatch with reality

Well the hype train has left the station. As I'm sure the whole of the CEP community knows by now StreamBase has teamed up with Oracle announcing a Streaming SQL standard. I am certainly in favor of standards in software technology, they clearly represent the tide that raises all boats. Customers and vendors alike are benefactors from communal standards. From ANSI standard programming languages like C and C++  to open standards like XML. Many a consortium of vendors and customers have labored arduously to define well-known technology standards for the collective benefit of the greater worldwide community.    However, this recent announcement by StreamBase and Oracle is nothing more than the practice of the crafty art of deception and diversion.  While I see nothing wrong with StreamBase and Oracle teaming up to work on enhancing the streaming SQL language for their CEP products, to tout it as representing an emerging industry standard is simply brazen.

The streaming SQL language in today's CEP products finds its roots in academia.  The Aurora project is one such academic endeavor. SQL was the language of choice for this project for good reason. Streaming data shares a number of common attributes with static data, why not use a well known data access, data filtering, data manipulation language. The Auoroa authors clearly had this in mind when they chose SQL. I'm sure they also had an expectation that streamingSQL and the future products based on it would evolve in a manner similar to database or other backend data service technology.

However, CEP platforms have matured into application platforms. This in no small measure is due to Progress Apama and our solutions approach to the market.  The Apama stack easily lends itself to the rigors and demands of the solutions or application environment. The Apama EPL, Monitorscript has the expressiveness of syntax to describe the semantics of  complex logic in today's CEP applications.  As the saying goes, imitation is the sincerest form of flattery, many of our competitors have followed our lead by introducing a solutions approach themselves. But as a result, they've faced a challenge with SQL being the underpinnings of their EPL.  SQL was never intended to be an application language, therefore they've chosen either to build application solutions in the mixed-language environment or extend their base EPL to include procedural constructs to support the needs of application semantics. In either case, something has to give.  Reading the fine print of a StreamBase solutions datasheet: "Incorporates algorithms as Java or C++ plugins" is an indication of the inefficacy of streamSQL for the intended purpose.  With each new release of Coral8 and Aleri they announce features in their SQL-based EPL adding procedural constructs and imperative scripting constructs similar to Apama Monitorscript.  These language enhancements or mixed-mode development requirements clearly validate that CEP has evolved into an application platform and not just a back-end data service engine.  From a language standards viewpoint this has only served to fracture.  Each vendor has carved their own course in this brave new world.

As a cautionary note, standards can be the opiate of the masses. They give customers a sense that they are protected against vendor lock-in.  Even the perception of an emerging standard can be hypnotic. This is all under false pretense.  Real standards provide benefits to customers and vendors alike covering a broad swathe not just a select few.  As the CEP community ventures into the standards world we should focus in those same areas where standardization has a proven track record in other technologies, interoperability and integration.  There is plenty of fodder here and I'm sure it will unfold in the coming months.

Acronym irrelevance

There’s been lots of discussion very recently (and lots of discussion about the discussion) about how CEP is related to other software acronyms and what constitutes a CEP use case or not. See here and here.

This kind of debate depresses me in two different ways. Firstly it displays symptoms of a more general software malaise – the wish to group and pigeon hole things into software classes which then confuse people who are not in the in-crowd. Once a name is agreed upon, let’s say Complex Event Processing, it then gets reduced to an acronym - CEP (excuse the pedantry but this is actually an initialism, not an acronym but that's enough of that). People feel a little sense of achievement. “We’ve made it – we’ve got a TLA!”. Debates then rage about how CEP relates to BRMS, BAM, ESB, BPM, ESP, BEP, EDA and EAI. Dreadful stuff, but, yes I know, we’re all guilty of it at times; it does give others the impression that the IT industry has its head up its own back passage.

The second reason this debate depresses me is that I really don’t understand this constant wish to class things as problems which fit into the "CEP class" or not (just see all the nonsense, albeit amusing, around routers of various types voiced by Messrs Bass and Palmer). Software is ultimately a productivity tool and what end-users really want to know is whether a product will help them achieve something they would be unable to do so by other means. End-users are using and considering using event processing products for a whole variety of purposes – trading, order routing, travel information dissemination, exception management in long-lived IT processes, detection of aberrant conditions in groups of pressure sensors, detection of fraud at retail point-of-sale terminals… the list could go on. People who have a problem to solve might think that event processing technology could help them. It is their responsibility, together with a vendor who may hope to sell something to them, to determine whether a product would help them or not. Were people doing event processing before off-the-shelf products came along? Yes. Do you need a CEP product to do event processing? No. Does everything you do with a CEP product have to involve complex, multiple streams of perhaps temporally related information existing in some distributed computing data cloud? No, of course it bloody doesn’t. I note that Opher Etzion seems similarly turned off by this type of debate.

And before I go I’m quite aware that I’ve used the CEP initialism eight times in this posting. I think I can be excused - this is a Complex Event Processing blog after all.

CEP - Some Applications within Capital Markets

There is occasionally discussion in the blogosphere around the role of CEP within Capital Markets.  Beyond the online chatter, we also hear via offline discussions that Apama comes up in that context, given our strong (arguably dominant) presence in that vertical.  I'd rather not get embroiled in a debate about what is or is not CEP or what are the historical antecedents of CEP.  Actually I would, but I won't here. Let's just say that there have been suggestions that Apama is really not CEP, because it is an algorithmic trading platform.  Or, on occasion, there is the corollary assertion that algorithmic trading is not CEP and since Apama does algorithmic trading, therefore it is not a CEP product, which is false both in terms of the facts and the logical structure of the argument.  And it goes on - and on.

John Bates recently conducted a series of "audio interviews" that talk about some of the different usages of Apama and CEP within Capital Markets.  They might be illustrative to those who see Apama and CEP solely in terms of algorithmic trading or don't really understand algorithmic trading.  The information is not intended to be deeply technical from either a CEP or Cap Markets perspective, but hopefully provides some introductory context for understanding the real potential for CEP in delivering value within that market - and beyond.

Download Rogue Trading >

http://apama.typepad.com.nyud.net:8090/podcast/1_rogue_trading.mp3

Download Early Adoption of Complex Event Processing >

http://apama.typepad.com.nyud.net:8090/podcast/2_cep_early_adoptions.mp3 

Download Risk Management and Market Surveillance >

http://apama.typepad.com.nyud.net:8090/podcast/3_cep_and_risk_management_V2.mp3
 

Real-Time Temperature Monitoring

In an interesting demonstration of real-time monitoring, Apama participated in a recent study that was conducted by Radboud University as part of the Dutch "Four Days March".  The march is an annual event in which tens of thousands of participants walk daily distances of 30-50 KMs for four days in succession.  Unfortunately, past marches have had some unfortunate instances where participants have been overcome by heat - including a couple of deaths.

In a pilot study this year, volunteers took a pill embedded with an RFID chip and thermometer, which sent signals every 10 seconds to a Bluetooth-enabled GPS phone.  The phone in turn sent the information to an implementation of Apama, which correlated the information about the volunteer, their temperature and their location, all plotted on an implementation of Google Maps. Leveraging the capabilities of Apama, monitors could track the progress of volunteers and identify those volunteers whose temperatures exceeded certain thresholds.  And with the ability to correlate that information with the GPS data, they could tell where those volunteers were on the route, thus delivering a CEP-driven infrastructure for real-time monitoring and, if needed, a pre-emptive reaction to help somebody who might shortly be in distress.

In the image below, one can see the purple "pins", illustrating where on the route the volunteers are.

Rendering Unto Caesar - The Role of the Business User in CEP

"Render unto Caesar the things which are Caesar's
and unto God the things that are God's"

A recent posting in the Enterprise Decision Management Blog entitled "Can we trust business users" addresses a topic that seems equally pertinent to the CEP market. I think there is a tendency to become so enamored with the technical virtuosity of new technology that we may lose sight of who the real users are. In terms of the development of CEP applications, an understanding of the prospective roles of business users vs. that of IT developers seems to be an evolving thing. Apama has long been a proponent of the participation of the business user in the process of building CEP-driven applications. In Capital Markets, the notion of "empowering the trader" has been a key element of our strategy and the Apama product offers a graphical development tool, Event Modeler, that focuses on that constituency. We have another RAD tool that is intended for developers who can create applications in our event processing language, as well.

We are beginning to see third party validation of the value of this approach from outside of Capital Markets, as well. For example, a report from Forrester Research published earlier this year indicated that early adopters of CEP have tended to come from the line of business rather than IT because "developers and architects often know painfully little about these [non-traditional, non-IT] events and how they are used to run a business." That Forrester quote is certainly not intended to diss IT. It just recognizes that there are lots of different kinds of events that are important to a business and not all those events are traditional IT-aware or IT-generated events. In order to make sense and respond to such events, it seems quite logical that providing tools that are amenable to a more "business" oriented audience is important.

But I would argue that it is not just the nature of events - and their varied sources - that suggests a strong correlation between CEP and business users. It is also the nature of the CEP-driven applications themselves. CEP applications are not "one and done", they tend to be iterative and evolving, because they are crafted to respond to what is happening and what is happening is often a frequently changing set of conditions. Or, if the conditions are not changing, how you choose to respond to them may be changing. So you need to continually calibrate and revise.

In another Forrester report published earlier this year, this characteristic was noted within the context of a review of Business Activity Monitoring best practices. Event-driven BAM is a particularly strong use case for CEP and the report stated that BAM efforts "are typically never-ending projects" with a "high degree of change." That makes sense since BAM monitors 'business activities' and the nature of most businesses will change over time. So to support BAM applications, it seems perfectly logical to provide tools for business users who can take on some role in the initial development and ongoing operational calibration of these applications. There is clearly an important role for developers in building these applications – no one would suggest otherwise, but we best not forget what the “B” in BAM refers to.

What seems to be emerging is the notion that we are should not look at CEP and/or BAM deployments as discrete, finite projects with clearly prescribed end dates. They are continuously iterative projects that must evolve to remain effective. That’s the environment in which they operate. Given that, perhaps we should not see the roles of business users and IT as fitting within well prescribed boundaries. The development and ongoing management of these applications will have evolving roles for the line of business user and for IT over time. We might expect IT-centric development to have a more dominant role in the initial deployment, but over time the goal might be to have the line-of-business assume greater and greater roles - because the business will be the dominant user and best positioned to react to changing circumstances.

Perhaps the EDM blog posting says it best, though it expresses it within a "business rules" context. “Too many rules specialists focus on rules that will get executed and not enough on the people that will maintain those rules, although this is where the success of the project resides.” That is quite the same for CEP and BAM. There is a role for business users, driven by the nature of events and the continuously evolving nature of the applications that are “event-driven.” And it is incumbent on the technology provider to offer tools that will facilitate that evolution. All the CEP performance in the world will be of little use, unless that performance is well-aligned with the needs of the business.

So we might debate who is the metaphorical Caesar and who is God in CEP development, but the success may well rest on giving each their due.

Apama's Fit in the Future of FX

It seems like only yesterday that Atriax (2002) died leaving FXall and FXConnect the only multibank portals offering the Buy Side “competitive” FX prices via RFQ pricing from the banks.

In the past few years we’ve seen exponential growth in the number of FX trading venues and the services that they offer. No longer are those venues which provide streaming prices and anonymous matching exclusive to the banks, with even the electronic FX trading duopoly of the 80’s & 90’s forced to allow Buy Side participants a seat at the table (via Prime Brokerage) because of the competition.

The changes have blurred the roles between the Buy & Sell Sides, primarily benefiting the Buy Side through greater transparency, enhanced prices and real-time market access. To the Sell Side, ECN may no longer be a term that evokes hatred but it’s difficult for them not to remember the “good ole days” before paper thin spreads and anonymity.

In today’s ever fragmenting environment Complex Event Processing (CEP) has already solved many problems for both the Buy & Sell Side. We have customers who’ve automated a variety tasks, produced both Trading and Execution Algorithms and claim that they’re doing twenty times the number of trades with the same number of dealers.

In the past year FX Aggregation has been “the flavour of the month.” I particularly liked it when a senior executive who saw our system and simply said “There is no fragmentation. With this system my dealers can access the whole market through just one screen.”  (learn more)

The May Edition of Profit and Loss (P&L) contained an article called the “Shifting Landscape: DMA, BANKS AND ECNS” which said “many refer to today as the Golden Age of ECNs.” It pointed out how the “Credit Crisis” and the events of last August forced many in the Buy Side to rethink the importance of having a relationship with banks to ensure access to the liquidity banks is maintained when other pools start drying up. 

Although P&L doesn’t see the death of ECN’s, the article made a number of observations and predictions on how the banks shall reassert themselves in the ever changing landscape of FX. I’m writing to comment about CEP’s role in what’s being predicted.

The cornerstone to P&L’s future lies in banks offering Direct Market Access (DMA) to their customers to guarantee transparency and best execution. DMA in FX of course will have its own intricacies, but this is an evolution not a revolution as DMA has been part of the Equity, Futures and Options markets for years.

Through FX Aggregators banks are already merging the fragmented FX market to guarantee their own access to best prices and deeper liquidity. The extension of this to their customers for DMA is relatively straight forward in much the same way that the Sell Side currently offers DMA via Smart Order Routers in the Equity space.

The next prediction is that banks will add their own liquidity to the aggregated pool and, when adequate numbers of customers are connected, to allow them to transact with each other. This sounds very much like a Crossing Network and a perfect home for a very fast CEP engine that can evaluate streams or orders, looking for matches.

Finally, P&L sees the banks interconnecting their systems to extend the liquidity pool, thus becoming ECN’s in their own right. Will this lead to the Buy Side then Aggregating the Aggregators?

Sprinkle in a few trading, execution and liquidity seeking algorithms and what P&L is predicting makes a lot of sense.

So what does this mean and what problems does it pose? It means that trading banks must be capable to quickly produce and deploy new services and be proficient in adapting these to address ever evolving requirements from the market (i.e. be swift & agile). The problem is many organisations aren’t particularly good at rapid development and deployment.  Those who are shall hold a real competitive advantage going forward.

Complex Event Processing fits the infrastructural needs for DMA, Aggregation & Crossing Networks well, but it’s not good enough to simply have a very fast engine. The engine must readily connect to a disparate world, provide tools for rapid development and back testing facilities to ensure successful deployments.

Apama fits all these requirements as it’s not only a very, very fast engine but has graphical tools that enable both business and IT staff to collaborate and rapidly produce solutions, back testing facilities to mitigate deployment risk and an open integration framework that includes many existing “off the shelf” adaptors.

It’ll be five years before my 20/20 hindsight enables me to recall what our future was. For now the only thing that I can say with certainty is Complex Event Processing and Apama will be playing a major role in whatever the future of FX holds.