Event Processing blog

High Frequency Trading driving the need to build quickly, run fast

In just about any race there is usually a starting point and a finish line, unless of course you are in an arms race.  For that sort of race there may have been some nebulous beginning in the distant past, but there is no finish line. The race just keeps sprinting along, each competitor angling for an edge, regularly recharging their ammunition supply with some new weaponry to get ahead however slight or temporary.

I recently read an interesting article describing High Frequency Trading as embroiled in an arms race. I certainly believe it's well entrenched in such a conflict, but frankly this combat has arguably had a beneficial net effect especially in that it's contributed to the wellspring of invention, inspiring the creative spirit in all the supporting attributes that make High Frequency Trading a reality. Behind any trader (and trading firm) is an entire armada including the vendors supplying the underlying hardware, networks, software platforms and trading applications.  They are all immersed in the war.  As new hardware, software and/or algo's are deployed it allows the trader to do battle and speed ahead even if it's just for a short while.  Competitive pressures, increasing market volatility, regulatory imperatives, risk mitigation and a host of other challenges are the land mines and roadside bombs on the long and winding road that stall and slow causing re-tooling and re-stocking the ammunition (i.e. algo strategies). There is no time to stop and catch your breath or stand on the roadside.

Sang Lee from the Aite Group reports that High Frequency Trading has had a significant impact on the overall market, providing greater liquidity, tighter spreads and overall improving the quality of the market.  At the macro level these are great advancements and mark a natural evolutionary step due to so many market changes in recent years (i.e. electronic trading venues, adoption of CEP platforms for algo trading, etc.) in Equities and beyond (i.e. FX and Futures & Options).  Down in the trenches, the battles rage on day by day as a multitude of traders and an untold number of algo strategies provide the market liquidity by moving in and out of positions in milliseconds (or even less time). The trading firms engaged in this never ending conflict drive a set of imperatives on software infrastructures for building and deploying algos in the High Frequency battlefield:

Rapid development and customization of algo's

Algo strategies in the High Frequency world have a limited life time. They soon become obsolete (i.e. whatever alpha they took advantage of has disappeared due to the competition, economic changes, or other situations).  To react and respond to this inevitability, having the right sort of tooling to recalibrate strategies is a necessity. This includes graphical modeling tools for Quants to prototype ideas quickly, backtest with historic data, test in a scalable manner to instill confidence prior to production rollout and lastly dynamic parameterization of strategies from graphical dashboards. Not forgetting the code-slinging types, an Integrated Development Environment (IDE) for support of event processing language (EPL) development for more low-level tasks.

Abstracting over increasingly complex strategy logic

Supporting Quants with a rich and robust set of functionality from the basics (connectivity to markets) to the advanced (Linear Algebra, Black Scholes, and other statistical functions).

Support for the 'ilities (availability, security, reliability, ...) to manage the mundane

Deploy with confidence. An important role of software infrastructure is to instill confidence that deployed strategies are always available, securely accessed and run without failure.

Support for scalable performance, providing high throughput and low latency

This is probably the most paramount requirement in the arms race of High Frequency Trading.  The race to the microsecond is pushing both hardware and software vendors alike. Parallelism in CEP engines like Apama's Correlator can leverage multi-core processor architectures like the Intel Nehalem. 

Along with my colleague Dan Hubscher,  I have recorded a 30 minute webinar that describes how the Apama platform along with the Apama Algorithmic Trading Accelerator meet these imperatives.

The pre-recorded webinar, is available here:  Apama Algorithmic Trading Accelerator, Build Quickly, Run Fast.

Once again thanks for reading (plus watching and listening to the webinar in this case), you can also follow me at twitter, here.
Louie

We're going on Twitter

Louis Lovas and myself, Giles Nelson, have started using Twitter to comment and respond to exciting things happening in the world of CEP (and perhaps beyond occasionally!).

The intent is to complement this blog. We'll be using Twitter to, perhaps, more impulsively report our thinking. We see Twitter as another good way to communicate thoughts and ideas.

We would be delighted if you chose to follow our "twitterings" (to use the lingo), and we'll be happy to follow you too.

Click here to follow Louis and here to follow Giles (you'll need to signup for a Twitter account).

CEP - Some Applications within Capital Markets

There is occasionally discussion in the blogosphere around the role of CEP within Capital Markets.  Beyond the online chatter, we also hear via offline discussions that Apama comes up in that context, given our strong (arguably dominant) presence in that vertical.  I'd rather not get embroiled in a debate about what is or is not CEP or what are the historical antecedents of CEP.  Actually I would, but I won't here. Let's just say that there have been suggestions that Apama is really not CEP, because it is an algorithmic trading platform.  Or, on occasion, there is the corollary assertion that algorithmic trading is not CEP and since Apama does algorithmic trading, therefore it is not a CEP product, which is false both in terms of the facts and the logical structure of the argument.  And it goes on - and on.

John Bates recently conducted a series of "audio interviews" that talk about some of the different usages of Apama and CEP within Capital Markets.  They might be illustrative to those who see Apama and CEP solely in terms of algorithmic trading or don't really understand algorithmic trading.  The information is not intended to be deeply technical from either a CEP or Cap Markets perspective, but hopefully provides some introductory context for understanding the real potential for CEP in delivering value within that market - and beyond.

Download Rogue Trading >

http://apama.typepad.com.nyud.net:8090/podcast/1_rogue_trading.mp3

Download Early Adoption of Complex Event Processing >

http://apama.typepad.com.nyud.net:8090/podcast/2_cep_early_adoptions.mp3 

Download Risk Management and Market Surveillance >

http://apama.typepad.com.nyud.net:8090/podcast/3_cep_and_risk_management_V2.mp3
 

SIFMA Retrospective

As a follow-up to my colleague Louis' report on last week's SIFMA show, I thought I'd add some thoughts of my own. My conclusion is that it was the most exciting SIFMA show I have experienced.  While I think attendance was down from previous years, I also think the quality of attendees was up. And for me personally, the excitement of being involved in some industry-moving announcements as well as meeting up with many of my colleagues from capital markets firms, vendors, press and analysts was highly invigorating.

So what were the highlights and take-aways for me?

1. CEP is clearly a theme that is getting a lot of mindshare. So many people said that CEP was a key theme of the show – which is great to hear after many years of working to help define the market. It’s also great to see this add to the momentum so soon after the Event Processing Technical Society was launched. The use cases of CEP are many and varied – and there was a lot of interest and questions around this at SIFMA. We demonstrated on our SIFMA booth 5 different CEP use cases on 5 different pods - algorithmic trading, smart order routing, managing FX market fragmentation, market surveillance and real-time bond pricing. Also the demands of CEP applications continue to make demands on the technology, and we were thrilled to demonstrate Apama 4.0 – which extends performance and user experience of CEP to new levels. Another supporting factor in the maturing of CEP is that there are starting to be very senior people in Capital Markets firms focusing on CEP as an enabling technology. Marc Adler from Citigroup   is a key example. He’s active in the community and on the STAC CEP committee, helping to define benchmarks. It was great to meet Marc at SIFMA and also to catch up with many  other esteemed colleagues from the CEP space.

2. The liquidity wars are hotting up. It was our pleasure to be involved in a press release with NYSE-Euronext which was certainly one of the big releases of the conference. Progress Apama will be hosted in the NYSE Euronext as part of the exchange's Advanced Trading Solutions offering. Traders will be able to download custom logic for algorithmic trading, risk management and smart order routing into the NYSE itself - with low latency connectivity to other trading venues via Wombat and Transact Tools. This arrangement turns NYSE into a technology provider as well as a one-stop-shop liquidity provider. This announcement was picked up by major press, including the Financial Times - in Europe, America and Asia -- see the article here.

3. Hardware is important – and so is “green”. The increase of capital markets data volumes require completely new software architectures – like CEP. But software is not always enough to support the low latency transport , processing and storage requirements. Many firms are turning to specialized hardware, combined with software – to create high performance solutions. Vhayu, for example, launched Squeezer – which combines hardware and software to supercharge their tick data offering. Also, Progress Apama were pleased to put out a joint announcement with Sun on a collaboration for end-to-end CEP solutions – combining Sun hardware and operating systems with Apama’s CEP platform and solutions. We demonstrated an end-to-end bond pricing application using the whole stack. Sun was one of the vendors who have a “green” aspect to their hardware – for example on a major CEP deployment, the hardware can be scoped for peak throughput – but can selectively shut down capacity to save power when event throughput is reduced. In this era of high energy costs and global warming there seems to be a lot of interest in this approach.

4. I love partying on the trading floor. Progress Apama were honored to be invited to a party at the NYSE to celebrate the latest developments at NYSE-Euronext (see picture at the top). It was a great pleasure to speak with our friends at NYSE-Euronext and to meet many of our old friends from the capital markets industry there – while sipping some delicious wine in that amazing place. In a way it is a shame that electronic trading is making the traditional trading floor a thing of the past – but there is something amazing about that place and I hope it stays just the way it is – even if it becomes a cool venue for other purposes. Thanks to NYSE-Euronext for inviting us – we had a great time.

I’m sure I’ve neglected a load of other trends and themes – but there’s my brain dump for the day. I’m interested to hear if you all agree.

John

TradeTech Recap

Colleague, Dr. Giles Nelson, CTO of Progress Software EMEA and a co-founder of Apama, took some time to share his thoughts about this year's TradeTech, Paris, which we have captured in the audio file below.

An Apama Hat Trick

Last week proved to be a busy one for Apama on the marketing front as we issued three separate announcements in conjunction with our presence at the TradeTech show in Paris.  Two of the announcements focused on customers, while the third focused on work that a partner is jointly doing with Apama in the area of market surveillance.

• ING Wholesale Banking announced that it is expanding its use of Apama, previously focused on algorithms for Benelux Small and Mid-Caps.  ING has re-engineered those algorithms to address markets in Hungary and Poland with a variety of features that include hybrid cross asset algorithms that leverage ING’s direct access within those emerging markets.

• SEB, the Scandinavian financial group, announced it will expand its use of Apama to deliver advanced order flow monitoring services within a compliance application.  This was a second SEB announcement, following one last year regarding the SEB deployment of Apama to support client trading in Exchange-traded equities and futures.

• And lastly, but certainly not least, together with Detica, a key partner, we jointly announced a Market Surveillance Accelerator.  Accelerators are extensions to the core Apama platform that help our customers jumpstart their deployments, incorporating business logic and other components like sample dashboards and adapters for connectivity.  In this instance, we are combining the technical know-how and experience of Detica and Apama – both of which are now supporting projects at the FSA and Turquoise - to address the growing demand for real-time market surveillance capabilities.  We’ve previously announced Accelerators for Market Aggregation and Smart Order Routing.   And there'll be more to come.
  

These three announcements collectively illustrate that a key part of the value of the Apama CEP platform is its versatility.  Apama may initially be deployed in support of a specific application like algorithmic trading or a specific asset class, but upon experience with the product, many of our customers expand their use to different asset classes, different geographic markets and/or entirely different applications – like compliance or risk or market surveillance.

Asia Report: Fighting White Collar Crime

Hello from Hong Kong. As always it is fascinating to see how CEP is evolving in Asia. One trend I am observing is the huge interest in Hong Kong in rogue traders and white collar crime – and how CEP can be used to detect and prevent this – before it moves the market. Obviously the original rogue trader, Nick Leeson, is well known here. But there has been a great deal of interest in more recent goings-on, at firms such as SocGen. Amazingly, until a couple of years ago, insider trading was not illegal in Hong Kong! Now we have a highly volatile market, with a lot of uncertainty, huge event volumes and a real problem of seeking out and preventing rogue trading activities, as well as managing risk exposure proactively.

Of course CEP provides a compelling approach. In market surveillance - the ability to monitor-analyze and act on complex patterns that indicate potential market abuse or potential dangerous risk exposure can allow a regulator, trading venue or bank to act instantly. Banks want the reassurance that they are policing their own systems. Regulators need to protect the public. The media and public here find this fascinating.

On the topic of a different kind of white collar crime – consider using CEP to detect abuse in the gaming industry. The gambling phenomenon that has propelled Macau to overtake Las Vegas as the world’s biggest gambling hub is also an exciting opportunity for CEP. We have customers using CEP to monitor and detect various forms of potential abuse in casinos. Events that are analyzed to find these patterns include gamblers and dealers signing on at tables, wins and losses, cards being dealt etc. It is possible to detect a range of potential illegal activities, ranging from dealer-gambler collusion to card counting.

As a final thought - having met with some of our customers that operate both in Hong Kong and mainland China, it is clear that China is a massive market opportunity for CEP. Exciting times ahead for CEP in Asia.

CEP down under

I’m sitting here at Melbourne Airport in Australia on my way to Hong Kong. I’ve been delayed by a typhoon – probably a good reason to delay. After a very successful week in Sydney and Melbourne visiting customers, I thought I’d report that the CEP market is hotting up down under! As you would expect financial services is an early adopter and Apama has had several customers in Australia in this space for a few years now. But the demand is increasing. This is driven by factors such as increasing competitive pressures in the trading space and the impending fragmentation of the Australian market. Just like in Europe and North America, it is likely that several new trading venues will join the Australian Stock Exchange in offering liquidity in Australia. My diagram shows some of these in the form of Chi-X, AXE and Liquidnet.

Complex Event Processing offers a powerful way of monitoring, aggregating and analyzing the liquidity across all of these markets, as well as making real-time routing decisions. This of course can work in parallel with traders and algorithms. In fact it is becoming very interesting to see trading decision algorithms routing messages to execution algorithms, routing messages to liquidity tracking algorithms, routing trades to the market, which are being checked by market surveillance algorithms -- and all part being implemented in CEP. I am biased of course, but what other technology can offer the seamless federation of such systems. Events provide a powerful and low latency mechanism for such interoperation. Each component can be built independent of the other - but yet they can work together seamlessly. But I am getting off topic!

Over the last few years Australia has mainly been interested in equities algorithms, but now the interest in FX, futures, bonds and commodities is growing. While I was in Sydney, I was pleased to deliver the keynote address at the Trading Technology conference and met many interesting sellside and buyside participants with a variety of trading interests. It was fascinating to see how the market is developing.

And it is not just financial services where CEP is being applied down under. I also met with organizations in a number of other spaces including travel, transportation and location-based services. I hope to report more on these in the near future.

And now I look forward to finding out what is happening in Hong Kong and Asia beyond. Hopefully I can avoid the typhoon!

John

CEP and Real-Time Risk – “The Dog Whisperer”

This week’s Financial Times published an interesting article by Ross Tieman on technology’s role as a “scapegoat” (his term) for some of the problems in financial markets. With its both evocative and provocative title, “Algo Trading: the dog that bit its master”, you can get a sense of the article theme, though a complete reading of the piece is worthwhile.

One of the challenges noted by the author is in the area of risk management. Too often existing risk processes - and their supporting technology - focus upon performing end-of-day assessments. But when much of the trading activity is quantitative, that can be much too late to detect when positions are careening out of control and risk thresholds have been breached. There seems to be growing recognition for the need for continuous calibration of positions – what amounts to real-time risk management – in order to keep pace. Perhaps the notion of a “daily VAR” may well become an artifact of the 1990’s. 

Just as it powers a number of real-time trading deployments, CEP can be an equally rich technology foundation for building the kind of real-time visibility that is needed by modern risk management systems. The same technology that drives quantitative trading can equally be applied to the task of monitoring that trading and keeping it in check, if necessary.

Now risk management is an extremely complex endeavor, so I would not argue that CEP alone is the answer. But rightly implemented, CEP clearly offers the low latency infrastructure that can help drive the real-time calculations that are needed. Market regulators (e.g. FSA) and trading exchanges (e.g. Turquoise) have begun to recognize the potential of CEP to monitor market behavior. It is likely only a matter of time before trading firms awaken to the possibilities of CEP driving real-time risk systems that monitor behavior within the firms themselves.

So, if you’re concerned about the technology “dog” biting its master, perhaps it’s time to consider CEP as a prospective “dog whisperer” that can help manage the risk. 

Apama Wins Market Surveillance Deal at Turquoise

Progress Apama, along with our system integration partner, Detica, has been chosen by Turquoise, the European multi-lateral trading facility, to deliver a real-time market surveillance system. Turquoise is funded by nine of Europe's largest banks and is essentially a new electronic stock exchange that, when it goes live this year, will compete with traditional equities venues like the London Stock Exchange.

Explicit in the announcement is how important market surveillance can be in helping to ensure an orderly market that traders can trust to be fair to all participants. Such confidence helps promote liquidity, which is the lifeblood of any trading venue. Implicit in the announcement is the value of CEP-powered, real-time market surveillance in driving the rapid detection of potential patterns of abuse or other questionable behavior. As Apama observers will note, this is the second selection of Apama and Detica for a real-time market surveillance system.  Last year, the same vendor team was chosen by the British regulator, the FSA.

There has been much press in the last week regarding the problems that can occur when proper procedures – and technology – are not in place to monitor trading behavior.  The situations are different - the problems at the French bank Société Genéralé appear to involve the Futures trades of a single trader, while Turquoise will be an equities exchange. But billions in losses certainly shine a bright light on the value of being able to monitor activities and respond quickly, regardless of the financial instruments involved. And though the motivations might differ, the value is there whether the monitoring is done by individual firms, by the trading exchanges, or by regulatory authorities.