Event Processing blog

In defence of high frequency trading

The high frequency trading (HFT) debate seems to have entered a new and worrying phase in the UK. On Tuesday this week in an interview with the BBC, Lord Myners, the UK’s financial services minister, warned that high frequency trading had “gone too far” and that share ownership had “now lost its supporting function for the provision of capital to business”. (You can find the original interview here and reports of it in the Financial Times and The Independent yesterday).

 

 Mary Schapiro, head of the SEC, signalled at the end of October that a number of electronic trading areas were going to be looked into – naked access (where a broker sponsors a firm to have direct electronic access to an exchange), dark pools and high frequency trading.

 

It does seem now that on both sides of the Atlantic, governments and regulators are steeling themselves to act and softening the markets up to be able to accept the fact that electronic trading might have some limits.

 

The concern is that governments and regulators are going to come down too hard on electronic trading and the benefits that it gives investors will be damaged.

 

It all started with the flash order issue in the US a few months ago. Commentators were linking together various different, although related issues, in an inappropriate way. Flash orders seemed to be viewed sometimes as being synonymous with HFT, both of which were sometimes reported as forms of market abuse. All three topics are quite different. In my opinion, there are legitimate questions over the use of flash orders and a proposal to ban them is now being considered.

 

Dark pools, where large blocks of stock are traded off exchange to minimise market impact, have been the next targets. There are, again, legitimate issues. Dark pools, by their very nature, do not have good price transparency. Regulators have become concerned with their use because more and more trading is going through dark pools. Some estimates put this at between 10% and 30% in Europe and the US. This lack of knowledge about what exactly is the proportion is part of the problem itself. No one really knows what proportion of trading dark pools is taking. If a significant proportion of the market has no price transparency then this undermines the notion of a fair market for all. Regulators are looking at this and its likely that they will force dark pool operators to disclose far more information about what is being traded than they do currently. The SEC is considering limiting the proportion of a stock that can be traded through dark pools to a small percentage.

 

These legitimate issues however risk skewing the whole HFT debate to one where people will conclude that “HFT is bad”.

 

What people are now describing as HFT – the very fast and frequent, computer assisted trading of, usually, equities – is an evolution of something that has been happening in the market place for at least the last 10 years. In this time electronic trading has proliferated, not just in equities but also in all asset classes such as derivatives, bonds and foreign exchange. Far more venues for trading have been created. There are now many places where a company’s stock can be traded both in the US and Europe. This has brought competition and choice. Prices have been lowered, improving access to retail investors. Spreads have narrowed. Arbitrage opportunities are harder to find, which mean that market information is disseminating faster which, in turn, means that price transparency has improved. Because there is more trading going on, there is more liquidity available, which also means keener prices.

 

A key part of the HFT trend has been the use of algorithmic trading (the most prevalent use of complex event processing technology). Algo trading models fall broadly into one of two camps: alpha seeking, where market prices are examined to find a trading opportunity that will make money, and execution where orders are, usually, split up into smaller parts and then traded automatically in the market in an intelligent way to find good prices and to ensure those prices are not overly influenced by the trades being made themselves. For each type of model it can be very useful to react very quickly to market information, either to take advantage of a price discrepancy or to quickly pickup liquidity at a good price. Algorithmic trading is enormously beneficial for those who use it and its use is not limited to specialist hedge funds. Most algorithmic trading uses execution models that find liquidity, good prices, help minimise market impact and, lastly, increase significantly a trader’s productivity. Instead of wasting time executing several simple orders in the market over the course of many minutes or hours, the trader can simply ask a machine to do it. The trader can then spend time either covering more of the market (useful in straitened economic times) or spend more time actually delivering real value to a client.

 

Algorithmic trading and HFT have brought very significant benefits. It is these benefits that must not be threatened.

 

Trading has always involved cunning and guile, whether human or computer based. Competition has always existed in who’s got the best traders and trading systems. Organisations investing in ultra low-latency infrastructure to ensure orders arrive at an exchange in microseconds (not nanoseconds as sometimes claimed by the way – light travels 30cm in 1 nanosecond which isn’t far enough to be very useful) are part of this competitive world. Competition leads to innovation and it is this innovation that has brought so many of the benefits described above. Computer-based models can somtimes be used abusively. There are many forms of market abuse that regulators and exchange operators look for. Some exchanges and regulators have been investing in real-time surveillance technology (Progress counts Turquoise and the UK Financial Services Authority as customers using Apama) to ensure that they can spot abusive patterns of behaviour quickly.

 

We can’t start slowing trading down. We can’t go backwards and put the electronic trading genie back in the bottle. We don’t want to lose all the benefits that have come. Rather, regulators and exchanges should concentrate on ensuring maximum transparency in how markets operate and ensure that those attempting to maliciously abuse the markets are dissuaded or caught.

 

Apama 4.2 Deeper Exploration: Enhanced Support for Parallelism

In this podcast Louis Lovas, Apama Architect, discusses some of the details of the enhanced parallelism that is now available in Progress Apama 4.2.

.

Progress Apama and Revenue Management Podcast

Listen to this podcast to learn how organizations are using Progress Apama Business Event Processing for Revenue Management.

Business Event Processing Podcast Overview

Listen to this podcast to get an overview of how Progress Apama Business Event Processing helps organizations to monitor, analyze and act on information in real-time.

Progress Apama Announcing Latest Release 4.2

As a follow up to the Louie Lovas blog posting on October 16^th , this  podcast captures a discussion between David Olson and Giles Nelson on Apama 4.2 features.

Apama 4.2 release - Cruising in the fast lane

The Apama engineering team has done it once again. True to our record of releasing significant new features in the Apama product every 6 months, the v4.2 release is hot off the presses with major new functionality. The Apama roadmap is driven by a keen sense of our customer requirements, the competitive landscape and an opportunistic zeal. The engineering team is a dedicated R&D team driven to excellence and quality. We are dedicated to delivering value to our customers. A consistent comment we've heard from analysts and customers alike is the maturity of the Apama product.  

The current v4.2 release, the third in the v4.x family adds significant enhancements in three concurrent themes - Performance, Productivity and Integration. This consistent thematic model is one we've held for a number of years. Below I've touched upon the highlights of the current release along these themes:

• Performance

High Performance Parallelism for Developers.  The Apama Event Processing Language (EPL) provides a set of features uniquely suited to build scalable event-driven applications.  The language natively offers capabilities for event handling, correlating event streams, pattern matching and defining temporal logic, etc. Equally important, the language provides a flexible means to process events in parallel.  For this we provide a context model and a new high performance scheduler. Contexts can be thought of as silos of execution, where CEP applications run in parallel. The scheduler's role is to manage the runtime execution in an intelligent high-performance way, and to leverage the underlying operating system threading model. It’s via the context architecture that the Apama Correlator squeezes the most out of operating system threads to achieve maximum use of multi-core processors for massive vertical scalability. For IT developers, this is a effective and efficient means to build high performance, low latency CEP applications without the pitfalls of thread-based programming, such as deadlocks and race conditions.

High Performance Parallelism for Business Analysts.  Not to be left out of the race, we've also ensured the scalable parallelism provided in the Apama CEP engine is available through our graphical modeling tool, the Event Modeler. We've had this graphical modeling capability since the very first release of Apama. This tool designed for analysts, quantitative researchers and of course developers, allows you to design and build complete CEP applications is a graphical model.  Parallelism is as easy as an automatic transmission, simply select P for parallel.

• Productivity

Real men do use Debuggers (and Profilers too). The Apama Studio now sports major new functionality for development, a source level debugger and a production profiler. Building applications for an event-driven world presents new programming challenges. Having state-of-the-art development tools for this paradigm is a mandate. The Apama EPL is the right language for building event-driven applications - now we have a source-level debugger designed for this event paradigm. Available in the Eclipse-based Apama Studio it provides breakpoints to suspend applications at specific points, examine contents of program variables and single stepping. It works in concert with our parallelism as well. Profiling is a means to examine deployed Apama applications to identify possible bottlenecks in CPU usage.

Jamming with Java. We've enhanced our support for Java for building CEP applications. The Apama Studio includes a complete set of wizards for creating monitors, listeners, and events to improve the development process when building java-based CEP applications in Apama.

• Integration

The (relational) world plays the event game. While we have provided connectivity to relational databases for many years we've made a significant re-design in the architecture of how we do it with the new Apama Database Connector (ADBC). The ADBC provides a universal interface to any database and includes standard connectors to ODBC and JDBC.  Through the ADBC, Apama applications can store and retrieve data in standard database formats using general database queries, effectively turning these relational engines into timeseries databases. The data can be used for application enrichment and playback purposes. To manage playback the Apama Studio includes a new Data Player that enables back-testing and event playback from a range of data sources via the ADBC. One can replay at varying speeds event data and time itself. The tested CEP applications behaves temporally consistent even as data is replayed at lightening speed.

Cruising at memory speed with MemoryStore. The MemoryStore is a massively scalable in-memory caching facility with in-built navigation,  persistence and visualization functionality.  This allows CEP applications, which typically scan, correlate and discard data very quickly to retain selected portions in memory for later access at extreme speed. This could be for managing a financial Order Book, Payments or other data elements that the application needs to be able to access at user’s requests quickly. Furthermore, if required the in-memory image can be persisted to a relational database for recovery or other retrieval purposes, and lastly the MemoryStore allows selected portions of the in-memory cache to be automatically mapped to dashboards.

Well that's the highlights. There were also about a dozen other features within each of these three themes, just too numerous to mention.

We are committed to improving the Apama product by listening to our many customers, paying close attention to the ever-changing competitive landscape and researching new opportunities.

Again thanks for reading, you can also follow me at twitter, here.
Louie

Developing Event Processing Applications

Listen to this podcast to hear Chris Martins and Giles Nelson discuss development of event processing applications.

If You Build It They Will Come, An Apama Algo Webinar

My colleague Dan Hubscher and I just finished the first of a two part Webinar entitled "Build Quickly, Run Fast". In this Webinar we explained and demonstrated Apama as an Algo platform for high frequency and order execution algorithms.

As I've blogged in the recent past it is an arms race in High Frequency trading.  The need to build quickly is a demanding requirement to keep ahead in the race. Being armed with the right tools is paramount. Rapid development and customization of strategies using graphical modeling tools provides the leverage necessary to keep pace with fast moving markets.

To that point, in this webinar I demonstrated a couple of algo examples. The first was a complete strategy that incorporates an alpha element with multiple order execution options. In  designing and building strategies the trading signal detection is just the first part of the problem. This typically involves an analytic calculation over the incoming market data within some segment or window of time. For example a moving average calculation smooths out the peaks and valleys or the volatility of an instrument's price. Once the signal is detected it's time to trade and manage the order's executions. This is a key distinction between other CEP products and the Apama platform for building trading strategies. While it's possible to define an Event Flow in most or all CEP products for data enrichment and data analysis (i.e. the signal detection), for most other CEP products you have to switch out to some other environment & language to build the rules to manage the executions. The Apama platform is about building complete event-driven applications. So trade signal detection and order executions, whether it's a simple iceberg execution or something much more complex it can easily be designed, built and backtested in the same Apama graphical modeling environment (Of course for those more inclined to traditional development tools and methodologies, Apama offers a full suite of developer tools, an EPL, debugger, profiler and java support).




The second example in the Webinar demonstration was to build a small, but working strategy from scratch. I did this live in full view of the attendees. For this I did a basic price momentum strategy. This tracked the velocity of price movements. The trading signal was a parameterized threshold which indicated when that price moved up (or down) a specific amount for a specific duration.

This webinar is focused on highlighting the ever-present challenges investment firms face in high frequency trading:

• Fears of the Black Box
• The simple fact that markets are continually evolving
• First Mover Advantage
• Customization is king

Along with my colleague Dan Hubscher,  the Build Quickly webinar describes how the Apama platform delivers solutions to the Capital Markets industry to meeting these needs and challenges. 

Stay tuned for a link to the recording and don't forget to dial in to part II where we focus on performance requirements and characteristics. Again thanks for reading (plus watching the webinar), you can also follow me at twitter, here.

A follow up note, here's the link to the recordings for both part I and part II on Build Quickly Run Fast.

Louie

Business Events and Operational Responsiveness - our research

Yesterday, we published a press release on some research that we commissioned from a independent research firm. I wanted to give a bit more background to the research and how we intend to use it.

Our intent in doing this research was twofold:

(a) To discover something new about the markets that Progress operate in and validate some of our own beliefs about the market (or dispell them).

(b) To gather some interesting and relevant information to act as talking points around the things we think are important for our customers and prospective customers, as well, of course, as being commercially relevant to us.

We commissioned the research company Vanson Bourne to do this research and whilst we worked with them on the scoping of it, it was left entirely to them to execute on that scope.

We wanted to hear from end-users so a range of questions were posed to 400 organisations in Europe and the US in three industries - telecommunications, energy generation and logistics. No vendors, analysts or systems integrators were approached.

The questions were all around the theme of "operational responsiveness" - how good are firms at monitoring their operations, identifying issues with process execution, interacting with their customers, extracting and integrating information etc. In particular how good are firms at dealing with the business events which are flowing around, both internally and externally, and how good are they at acting on them in a timely fashion?

Why did we pick these three verticals? Firstly, we couldn't cover everybody and we wanted to go to more companies in a few verticals rather than go very broad. Secondly, we believe that these three verticals are the most interesting when it comes to the demands being placed upon them to cope with business events (Financial services is another obvious one but we know quite a lot about the demands in that industry already). Telecommunications firms are very dependent upon IT to differentiate their services; logistics companies are using more and more technology to track goods, trucks, ships etc. to streamline and automate their operations; energy producers are having to rapidly plan for the introduction of smart metering.

We're still digesting the results. But a few are worth highlighting here. Social networking is creating a significant challenge for all organisations in dealing with customer feedback - consumers expect instant feedback to their interactions. Organisations aspire to more dynamic and real-time pricing of goods and services to increase their competitiveness and maintain margins. And companies struggle with achieving a holistic view of how their processes are operating, both to operationally identify issues before they become expensive to fix or affect customer services, and to identify ways in which processes can be shortened.

We'll be talking more about the research results soon, both qualitatively and quantitatively.

Progress Apama Capital Markets in Brazil

In recent years, the Brazilian market has grown stronger, and become very aggressive with algorithmic trading. Just back from a conference in Brazil, listen to this podcast where Dan Hubscher shares insight into the current state of Brazil’s market, and what the people down there are buzzing about.