Event Processing blog

In defence of high frequency trading

The high frequency trading (HFT) debate seems to have entered a new and worrying phase in the UK. On Tuesday this week in an interview with the BBC, Lord Myners, the UK’s financial services minister, warned that high frequency trading had “gone too far” and that share ownership had “now lost its supporting function for the provision of capital to business”. (You can find the original interview here and reports of it in the Financial Times and The Independent yesterday).

 

 Mary Schapiro, head of the SEC, signalled at the end of October that a number of electronic trading areas were going to be looked into – naked access (where a broker sponsors a firm to have direct electronic access to an exchange), dark pools and high frequency trading.

 

It does seem now that on both sides of the Atlantic, governments and regulators are steeling themselves to act and softening the markets up to be able to accept the fact that electronic trading might have some limits.

 

The concern is that governments and regulators are going to come down too hard on electronic trading and the benefits that it gives investors will be damaged.

 

It all started with the flash order issue in the US a few months ago. Commentators were linking together various different, although related issues, in an inappropriate way. Flash orders seemed to be viewed sometimes as being synonymous with HFT, both of which were sometimes reported as forms of market abuse. All three topics are quite different. In my opinion, there are legitimate questions over the use of flash orders and a proposal to ban them is now being considered.

 

Dark pools, where large blocks of stock are traded off exchange to minimise market impact, have been the next targets. There are, again, legitimate issues. Dark pools, by their very nature, do not have good price transparency. Regulators have become concerned with their use because more and more trading is going through dark pools. Some estimates put this at between 10% and 30% in Europe and the US. This lack of knowledge about what exactly is the proportion is part of the problem itself. No one really knows what proportion of trading dark pools is taking. If a significant proportion of the market has no price transparency then this undermines the notion of a fair market for all. Regulators are looking at this and its likely that they will force dark pool operators to disclose far more information about what is being traded than they do currently. The SEC is considering limiting the proportion of a stock that can be traded through dark pools to a small percentage.

 

These legitimate issues however risk skewing the whole HFT debate to one where people will conclude that “HFT is bad”.

 

What people are now describing as HFT – the very fast and frequent, computer assisted trading of, usually, equities – is an evolution of something that has been happening in the market place for at least the last 10 years. In this time electronic trading has proliferated, not just in equities but also in all asset classes such as derivatives, bonds and foreign exchange. Far more venues for trading have been created. There are now many places where a company’s stock can be traded both in the US and Europe. This has brought competition and choice. Prices have been lowered, improving access to retail investors. Spreads have narrowed. Arbitrage opportunities are harder to find, which mean that market information is disseminating faster which, in turn, means that price transparency has improved. Because there is more trading going on, there is more liquidity available, which also means keener prices.

 

A key part of the HFT trend has been the use of algorithmic trading (the most prevalent use of complex event processing technology). Algo trading models fall broadly into one of two camps: alpha seeking, where market prices are examined to find a trading opportunity that will make money, and execution where orders are, usually, split up into smaller parts and then traded automatically in the market in an intelligent way to find good prices and to ensure those prices are not overly influenced by the trades being made themselves. For each type of model it can be very useful to react very quickly to market information, either to take advantage of a price discrepancy or to quickly pickup liquidity at a good price. Algorithmic trading is enormously beneficial for those who use it and its use is not limited to specialist hedge funds. Most algorithmic trading uses execution models that find liquidity, good prices, help minimise market impact and, lastly, increase significantly a trader’s productivity. Instead of wasting time executing several simple orders in the market over the course of many minutes or hours, the trader can simply ask a machine to do it. The trader can then spend time either covering more of the market (useful in straitened economic times) or spend more time actually delivering real value to a client.

 

Algorithmic trading and HFT have brought very significant benefits. It is these benefits that must not be threatened.

 

Trading has always involved cunning and guile, whether human or computer based. Competition has always existed in who’s got the best traders and trading systems. Organisations investing in ultra low-latency infrastructure to ensure orders arrive at an exchange in microseconds (not nanoseconds as sometimes claimed by the way – light travels 30cm in 1 nanosecond which isn’t far enough to be very useful) are part of this competitive world. Competition leads to innovation and it is this innovation that has brought so many of the benefits described above. Computer-based models can somtimes be used abusively. There are many forms of market abuse that regulators and exchange operators look for. Some exchanges and regulators have been investing in real-time surveillance technology (Progress counts Turquoise and the UK Financial Services Authority as customers using Apama) to ensure that they can spot abusive patterns of behaviour quickly.

 

We can’t start slowing trading down. We can’t go backwards and put the electronic trading genie back in the bottle. We don’t want to lose all the benefits that have come. Rather, regulators and exchanges should concentrate on ensuring maximum transparency in how markets operate and ensure that those attempting to maliciously abuse the markets are dissuaded or caught.

 

Apama 4.2 release - Cruising in the fast lane

The Apama engineering team has done it once again. True to our record of releasing significant new features in the Apama product every 6 months, the v4.2 release is hot off the presses with major new functionality. The Apama roadmap is driven by a keen sense of our customer requirements, the competitive landscape and an opportunistic zeal. The engineering team is a dedicated R&D team driven to excellence and quality. We are dedicated to delivering value to our customers. A consistent comment we've heard from analysts and customers alike is the maturity of the Apama product.  

The current v4.2 release, the third in the v4.x family adds significant enhancements in three concurrent themes - Performance, Productivity and Integration. This consistent thematic model is one we've held for a number of years. Below I've touched upon the highlights of the current release along these themes:

• Performance

High Performance Parallelism for Developers.  The Apama Event Processing Language (EPL) provides a set of features uniquely suited to build scalable event-driven applications.  The language natively offers capabilities for event handling, correlating event streams, pattern matching and defining temporal logic, etc. Equally important, the language provides a flexible means to process events in parallel.  For this we provide a context model and a new high performance scheduler. Contexts can be thought of as silos of execution, where CEP applications run in parallel. The scheduler's role is to manage the runtime execution in an intelligent high-performance way, and to leverage the underlying operating system threading model. It’s via the context architecture that the Apama Correlator squeezes the most out of operating system threads to achieve maximum use of multi-core processors for massive vertical scalability. For IT developers, this is a effective and efficient means to build high performance, low latency CEP applications without the pitfalls of thread-based programming, such as deadlocks and race conditions.

High Performance Parallelism for Business Analysts.  Not to be left out of the race, we've also ensured the scalable parallelism provided in the Apama CEP engine is available through our graphical modeling tool, the Event Modeler. We've had this graphical modeling capability since the very first release of Apama. This tool designed for analysts, quantitative researchers and of course developers, allows you to design and build complete CEP applications is a graphical model.  Parallelism is as easy as an automatic transmission, simply select P for parallel.

• Productivity

Real men do use Debuggers (and Profilers too). The Apama Studio now sports major new functionality for development, a source level debugger and a production profiler. Building applications for an event-driven world presents new programming challenges. Having state-of-the-art development tools for this paradigm is a mandate. The Apama EPL is the right language for building event-driven applications - now we have a source-level debugger designed for this event paradigm. Available in the Eclipse-based Apama Studio it provides breakpoints to suspend applications at specific points, examine contents of program variables and single stepping. It works in concert with our parallelism as well. Profiling is a means to examine deployed Apama applications to identify possible bottlenecks in CPU usage.

Jamming with Java. We've enhanced our support for Java for building CEP applications. The Apama Studio includes a complete set of wizards for creating monitors, listeners, and events to improve the development process when building java-based CEP applications in Apama.

• Integration

The (relational) world plays the event game. While we have provided connectivity to relational databases for many years we've made a significant re-design in the architecture of how we do it with the new Apama Database Connector (ADBC). The ADBC provides a universal interface to any database and includes standard connectors to ODBC and JDBC.  Through the ADBC, Apama applications can store and retrieve data in standard database formats using general database queries, effectively turning these relational engines into timeseries databases. The data can be used for application enrichment and playback purposes. To manage playback the Apama Studio includes a new Data Player that enables back-testing and event playback from a range of data sources via the ADBC. One can replay at varying speeds event data and time itself. The tested CEP applications behaves temporally consistent even as data is replayed at lightening speed.

Cruising at memory speed with MemoryStore. The MemoryStore is a massively scalable in-memory caching facility with in-built navigation,  persistence and visualization functionality.  This allows CEP applications, which typically scan, correlate and discard data very quickly to retain selected portions in memory for later access at extreme speed. This could be for managing a financial Order Book, Payments or other data elements that the application needs to be able to access at user’s requests quickly. Furthermore, if required the in-memory image can be persisted to a relational database for recovery or other retrieval purposes, and lastly the MemoryStore allows selected portions of the in-memory cache to be automatically mapped to dashboards.

Well that's the highlights. There were also about a dozen other features within each of these three themes, just too numerous to mention.

We are committed to improving the Apama product by listening to our many customers, paying close attention to the ever-changing competitive landscape and researching new opportunities.

Again thanks for reading, you can also follow me at twitter, here.
Louie

Progress Apama Capital Markets in Brazil

In recent years, the Brazilian market has grown stronger, and become very aggressive with algorithmic trading. Just back from a conference in Brazil, listen to this podcast where Dan Hubscher shares insight into the current state of Brazil’s market, and what the people down there are buzzing about.

Apama SIFMA 09 Announcements

SIFMA is happening this week in NY and this typically sparks a flurry of announcements in Capital Markets, as this event can be seen as a benchmark for what is happening in the industry, similar to TradeTech Paris for Europe earlier in the Spring.  Apama has made a series of significant announcements that capture some of the scope of the Apama platform and what we believe is required to be successful in this market.  In different niches of the blogosphere you’ll find some proponents of fairly narrow definitions of what “CEP” is or how one measures CEP “leadership” or “maturity”.  I think the Apama announcements illustrate a different perspective, reflective of the breadth of the Apama platform and the possibilities available to such a platform in terms of building new event-driven solutions in Capital Markets.

Market Surveillance and Monitoring Accelerator

Apama has previously announced market surveillance customers, FSA and Turquoise, and with our new enhanced Solution Accelerator we are capturing our capabilities in a way that allows regulators, exchanges/MTFs, and trading firms to further jumpstart deployment of monitoring applications.  The idea that it would be efficacious to have real-time monitoring of trading is becoming better understood in the indusry.  If you read the release, you’ll note that the targets are not just regulators or exchanges, but firms themselves, who would benefit from detecting potentially damaging activities prior to that activity hurting firm reputations or profits.

Apama Solution Accelerators provide a set of core functions that focus on specific domain areas, but they still retain the flexibility to evolve and adjust as circumstances require.  That is key to keeping pace with very dynamic market conditions.  These Accelerators have proven a key driver to our recent success as they marry the power of the underlying platform with the real “end game” of providing customers with solutions that deliver value.

UniCredit Customer Win

As an example of the power of Solution Accelerators, Apama also announced that UniCredit is using the Apama FX Market Aggregator (another of our roster of Accelerators) to give their FX traders access to prices from a number of FX liquidity venues.  UniCredit is also using the Accelerator to publish FX prices to its eFX downstream channels.   Again, a key point here is that the use of an Accelerator in no ways constrains a client from using the Apama tools that are part of the underlying platform to build out other capabilities that complement the Accelerator.  For some the Accelerator is close to what is needed, but for others it is an attractive launching point.  For further discussion of this, you might want to check out a recent Webinar on “FX Aggregation and Beyond” that talks to this issue.

Lime Brokerage

We added to our broad range of connectivity adapters with  an important connection to Lime Trading System’s Citrius market data feed and FIX order placement services.  This give our customers access to Lime’s trade execution capabilities for equities, derivatives, ETFs, futures and options.  Apama customers can also look to Apama applications via Lime Trading System’s collocation facilities.  Lime is a really cool firm and we see this as a great opportunity for our common customers – both those now and to come.

Connectivity is the lifeblood of CEP and other event-driven applications.  Apama has a broad range of adapters, and we have an engineering team whose specific focus is on this aspect of the product platform.  Integration adapters aren’t “sexy” and they don’t get a lot of attention in marketing literature, but they are vital and deserve a bit of spotlight.

BondDesk

BondDesk provides 2,000 broker-dealers with access to 35,000 live and executable offerings from 120 premier fixed income dealers.  In an announcement this week, we announced that BondDesk ATS (alternative trading system) customers will be able to register their interest in the availability of fixed income securities that meet certain criteria and Apama will monitor the inbound data and provide real-time notification when a matching offering becomes available on the ATS.

So four announcements this week, and more upcoming.

So stay tuned. 

.

We're going on Twitter

Louis Lovas and myself, Giles Nelson, have started using Twitter to comment and respond to exciting things happening in the world of CEP (and perhaps beyond occasionally!).

The intent is to complement this blog. We'll be using Twitter to, perhaps, more impulsively report our thinking. We see Twitter as another good way to communicate thoughts and ideas.

We would be delighted if you chose to follow our "twitterings" (to use the lingo), and we'll be happy to follow you too.

Click here to follow Louis and here to follow Giles (you'll need to signup for a Twitter account).

Adaptive CEP - A Prerequisite for CEP Success

Further affirmation that low latency execution is not the only motivator for decisions to deploy the Apama CEP platform is found in a recent quote from Yann L'Huillier, the CTO of Turquoise.  Turquoise is the London-based alternative trading venue that is using Apama for real-time market surveillance in a project that was managed by Progress Apama partner, Detica. 

In a quite interesting interview in InformationAge that speaks to the role of technology in jumpstarting the Turquoise effort to challenge the incumbents, L'Huillier explains "With regard to Apama and Detica, our thinking is that using a streaming engine with complex-event processing gives us a competitive edge, because we can change the way we read the market and adapt to the always-changing market conditions."

That has proven to be a common - and key - reason why organizations chose the Apama CEP platform.  CEP systems generally respond to conditions that they don't control, as the events are generated by external sources and the CEP system is monitoring what is happening via the events.  A successful CEP system must be adaptive, as the conditions that spawn the events are assuredly going to change.

The word "agility" has become hackneyed in the technology lexicon, but that really does describe the requirement.  As the Turquoise article suggests, it can be fundamental to competitive differentiation.  As a colleague recently noted to me, accuracy is as important as speed because without accuracy, technology merely accelerates the speed with which you get the wrong answer.  With the world changing as fast as it is, accuracy will be quite fleeting unless you can adapt quickly.

CEP - Some Applications within Capital Markets

There is occasionally discussion in the blogosphere around the role of CEP within Capital Markets.  Beyond the online chatter, we also hear via offline discussions that Apama comes up in that context, given our strong (arguably dominant) presence in that vertical.  I'd rather not get embroiled in a debate about what is or is not CEP or what are the historical antecedents of CEP.  Actually I would, but I won't here. Let's just say that there have been suggestions that Apama is really not CEP, because it is an algorithmic trading platform.  Or, on occasion, there is the corollary assertion that algorithmic trading is not CEP and since Apama does algorithmic trading, therefore it is not a CEP product, which is false both in terms of the facts and the logical structure of the argument.  And it goes on - and on.

John Bates recently conducted a series of "audio interviews" that talk about some of the different usages of Apama and CEP within Capital Markets.  They might be illustrative to those who see Apama and CEP solely in terms of algorithmic trading or don't really understand algorithmic trading.  The information is not intended to be deeply technical from either a CEP or Cap Markets perspective, but hopefully provides some introductory context for understanding the real potential for CEP in delivering value within that market - and beyond.

Download Rogue Trading >

http://apama.typepad.com.nyud.net:8090/podcast/1_rogue_trading.mp3

Download Early Adoption of Complex Event Processing >

http://apama.typepad.com.nyud.net:8090/podcast/2_cep_early_adoptions.mp3 

Download Risk Management and Market Surveillance >

http://apama.typepad.com.nyud.net:8090/podcast/3_cep_and_risk_management_V2.mp3
 

SIFMA Retrospective

As a follow-up to my colleague Louis' report on last week's SIFMA show, I thought I'd add some thoughts of my own. My conclusion is that it was the most exciting SIFMA show I have experienced.  While I think attendance was down from previous years, I also think the quality of attendees was up. And for me personally, the excitement of being involved in some industry-moving announcements as well as meeting up with many of my colleagues from capital markets firms, vendors, press and analysts was highly invigorating.

So what were the highlights and take-aways for me?

1. CEP is clearly a theme that is getting a lot of mindshare. So many people said that CEP was a key theme of the show – which is great to hear after many years of working to help define the market. It’s also great to see this add to the momentum so soon after the Event Processing Technical Society was launched. The use cases of CEP are many and varied – and there was a lot of interest and questions around this at SIFMA. We demonstrated on our SIFMA booth 5 different CEP use cases on 5 different pods - algorithmic trading, smart order routing, managing FX market fragmentation, market surveillance and real-time bond pricing. Also the demands of CEP applications continue to make demands on the technology, and we were thrilled to demonstrate Apama 4.0 – which extends performance and user experience of CEP to new levels. Another supporting factor in the maturing of CEP is that there are starting to be very senior people in Capital Markets firms focusing on CEP as an enabling technology. Marc Adler from Citigroup   is a key example. He’s active in the community and on the STAC CEP committee, helping to define benchmarks. It was great to meet Marc at SIFMA and also to catch up with many  other esteemed colleagues from the CEP space.

2. The liquidity wars are hotting up. It was our pleasure to be involved in a press release with NYSE-Euronext which was certainly one of the big releases of the conference. Progress Apama will be hosted in the NYSE Euronext as part of the exchange's Advanced Trading Solutions offering. Traders will be able to download custom logic for algorithmic trading, risk management and smart order routing into the NYSE itself - with low latency connectivity to other trading venues via Wombat and Transact Tools. This arrangement turns NYSE into a technology provider as well as a one-stop-shop liquidity provider. This announcement was picked up by major press, including the Financial Times - in Europe, America and Asia -- see the article here.

3. Hardware is important – and so is “green”. The increase of capital markets data volumes require completely new software architectures – like CEP. But software is not always enough to support the low latency transport , processing and storage requirements. Many firms are turning to specialized hardware, combined with software – to create high performance solutions. Vhayu, for example, launched Squeezer – which combines hardware and software to supercharge their tick data offering. Also, Progress Apama were pleased to put out a joint announcement with Sun on a collaboration for end-to-end CEP solutions – combining Sun hardware and operating systems with Apama’s CEP platform and solutions. We demonstrated an end-to-end bond pricing application using the whole stack. Sun was one of the vendors who have a “green” aspect to their hardware – for example on a major CEP deployment, the hardware can be scoped for peak throughput – but can selectively shut down capacity to save power when event throughput is reduced. In this era of high energy costs and global warming there seems to be a lot of interest in this approach.

4. I love partying on the trading floor. Progress Apama were honored to be invited to a party at the NYSE to celebrate the latest developments at NYSE-Euronext (see picture at the top). It was a great pleasure to speak with our friends at NYSE-Euronext and to meet many of our old friends from the capital markets industry there – while sipping some delicious wine in that amazing place. In a way it is a shame that electronic trading is making the traditional trading floor a thing of the past – but there is something amazing about that place and I hope it stays just the way it is – even if it becomes a cool venue for other purposes. Thanks to NYSE-Euronext for inviting us – we had a great time.

I’m sure I’ve neglected a load of other trends and themes – but there’s my brain dump for the day. I’m interested to hear if you all agree.

John

TradeTech Recap

Colleague, Dr. Giles Nelson, CTO of Progress Software EMEA and a co-founder of Apama, took some time to share his thoughts about this year's TradeTech, Paris, which we have captured in the audio file below.

An Apama Hat Trick

Last week proved to be a busy one for Apama on the marketing front as we issued three separate announcements in conjunction with our presence at the TradeTech show in Paris.  Two of the announcements focused on customers, while the third focused on work that a partner is jointly doing with Apama in the area of market surveillance.

• ING Wholesale Banking announced that it is expanding its use of Apama, previously focused on algorithms for Benelux Small and Mid-Caps.  ING has re-engineered those algorithms to address markets in Hungary and Poland with a variety of features that include hybrid cross asset algorithms that leverage ING’s direct access within those emerging markets.

• SEB, the Scandinavian financial group, announced it will expand its use of Apama to deliver advanced order flow monitoring services within a compliance application.  This was a second SEB announcement, following one last year regarding the SEB deployment of Apama to support client trading in Exchange-traded equities and futures.

• And lastly, but certainly not least, together with Detica, a key partner, we jointly announced a Market Surveillance Accelerator.  Accelerators are extensions to the core Apama platform that help our customers jumpstart their deployments, incorporating business logic and other components like sample dashboards and adapters for connectivity.  In this instance, we are combining the technical know-how and experience of Detica and Apama – both of which are now supporting projects at the FSA and Turquoise - to address the growing demand for real-time market surveillance capabilities.  We’ve previously announced Accelerators for Market Aggregation and Smart Order Routing.   And there'll be more to come.
  

These three announcements collectively illustrate that a key part of the value of the Apama CEP platform is its versatility.  Apama may initially be deployed in support of a specific application like algorithmic trading or a specific asset class, but upon experience with the product, many of our customers expand their use to different asset classes, different geographic markets and/or entirely different applications – like compliance or risk or market surveillance.