Event Processing blog

CEP and Real-Time Risk – “The Dog Whisperer”

This week’s Financial Times published an interesting article by Ross Tieman on technology’s role as a “scapegoat” (his term) for some of the problems in financial markets. With its both evocative and provocative title, “Algo Trading: the dog that bit its master”, you can get a sense of the article theme, though a complete reading of the piece is worthwhile.

One of the challenges noted by the author is in the area of risk management. Too often existing risk processes - and their supporting technology - focus upon performing end-of-day assessments. But when much of the trading activity is quantitative, that can be much too late to detect when positions are careening out of control and risk thresholds have been breached. There seems to be growing recognition for the need for continuous calibration of positions – what amounts to real-time risk management – in order to keep pace. Perhaps the notion of a “daily VAR” may well become an artifact of the 1990’s. 

Just as it powers a number of real-time trading deployments, CEP can be an equally rich technology foundation for building the kind of real-time visibility that is needed by modern risk management systems. The same technology that drives quantitative trading can equally be applied to the task of monitoring that trading and keeping it in check, if necessary.

Now risk management is an extremely complex endeavor, so I would not argue that CEP alone is the answer. But rightly implemented, CEP clearly offers the low latency infrastructure that can help drive the real-time calculations that are needed. Market regulators (e.g. FSA) and trading exchanges (e.g. Turquoise) have begun to recognize the potential of CEP to monitor market behavior. It is likely only a matter of time before trading firms awaken to the possibilities of CEP driving real-time risk systems that monitor behavior within the firms themselves.

So, if you’re concerned about the technology “dog” biting its master, perhaps it’s time to consider CEP as a prospective “dog whisperer” that can help manage the risk. 

The News on Elementized News: Apama Partners with Dow Jones

Big news continues to flow on the algorithmic trading for complex event processing (CEP).  Today, Dow Jones and Apama announced our partnership to deliver the first integrated algorithmic trading / elementized news platform.  This is a seminal news event, and marks the public beginning of a new frontier for algorithmic trading, one where news can be understood and leveraged by algorithmic trading platforms, and, in general, with CEP for a variety of applications.

What is “elementized news?”  DJ has added XML tags to their electronically distributed news stories that describe the semantics, or sentiment, of the news.  The tags describe things like: is positive news? Negative? Earnings up? Earnings Down? Better than analysts estimates? Worse? And so on.  Since trading began, news of economic and business trends have impacted behaviour; but, until now, it has been difficult to automate decision based on news while it occurs because state of the art, until now, has been to publish news information in an “unstructured” way – that is, in a human-readable form.

What's the role of Apama and CEP?  For years, we’ve been working with Apama customers and news providers like Dow Jones on bleeding-edge applications that process news;  we even blogged about it as one of our algorithmic trading imperatives (#7).  But integrating unstructured text in a reliable way is hard without semantic information.  Now,  DJ's  elementized feed makes it easier and more reliable to process news sentiment into CEP algorithms.

Fortunately, enough customers have been trying to work to gain more understanding within this flow of unstructured text that the news agencies themselves have decided to lend a helping hand. Dow Jones, a pioneer in this area, have released their elementized news feeds which tags each news item with information that describes the semantics, or meaning, of the news, so that more understanding can be gleaned from their news.

This is important for algorithmic trading because elementized news feeds allow computers to:

•  Trap each news item event, as it hits the wire

• Based on the tag elements, understand the semantics (e.g., positive earnings, negative analyst rating) of the news item

• Correlate the news to other news, or to trading activity

• Make automated decisions on this event, in the context other news and real-time trading activity

• Automate real-time advisory for human traders as a means to relate overall market sentiment in reaction to news, or to predict future market sentiment.

Complex event processing (CEP) engines is the underlying technology platform that enables effective elementized news feed processing.

These use cases will be explored in our Apama blog series on news in the coming days.

Algorithmic Trading Imperative #10: Learn from Experience

Today’s algorithmic trading tools identify the ‘cause and effect’ of trading techniques, learn from profit and loss, identify repeating market patterns and suggest new combinations of algorithms. Consistent use of these tools over time enables traders to ‘genetically tune’ algorithmic trading systems. Like

Darwin’s ‘survival of the fittest’ theory, algorithmic traders can run thousands of permutations of an algorithm, swap out the least profitable and replace them with more effective approaches. Analysis of recorded strategy behavior can be used to answer questions such as, “Why did I make $1 million today, but lose $1 million yesterday?”  Tools to replay algorithmic trading activity and examine both raw market data, as well as the resulting reactions and actions of automated systems is imperative to ensure not only that the algorithms worked they way they were supposed to, but also, how might they be improved.

By stepping through logs of strategy behavior with appropriate analysis tools, it is possible to determine, for example, that a firm was unsuccessful on a given day because ‘a trader modified the algorithm parameters, a position was taken, a news article moved the market and we didn’t have a rule to respond appropriately.’

Back testing, simulation, and root-cause analysis is the key to learning from past performance and improving the effectiveness of trading strategies in the future.

Algorithmic Trading Imperative #9: Research and Backtest Strategies

With firms continuously developing their own unique algorithmic trading strategies using complex event processing (CEP) technology, how can they ensure the strategies they feed into the markets are the best ones?  For the rapid development and deployment of new strategies, testing algorithms under a range of anticipated market conditions is critical. The latest techniques use back testing environments that enable the selection and naming of a library of market sequences, such as a ‘bull market’ or ‘bear market’. These sequences can be streamed through a strategy to test how the strategy performs.

Event processing platforms often contain event storage and management technologies, such as Apama's Research Studio, which provides back testing and analysis capabilities via a TiVo®-like event replay capability that allows CEP Scenario developers to interactively explore the prospective behavior of Apama Scenarios prior to deployment.  Event data management can also support "digital forensics" operations that allow users to audit the performance of Apama strategies already in deployment.

Algorithmic Trading Imperative #8: Design for Low Latency Decisions

In algorithmic trading, milliseconds matter. Minimizing the time between event detection (market data, news, requests for quotes) and action (placing an order) is critical. To do this, firms are using complex event processing (CEP) technology to implement their white-box algorithmic trading platforms. CEP is a new paradigm that allows organizations to identify patterns among streaming event data and respond to those patterns in microseconds (read a detailed overview of CEP and the history of its development here). Using a traditional database, you must store, index and retrieve the data – a very time-consuming process. CEP allows you to establish rules, or trading strategies, and ‘stream’ data through them, so the relevant data may be selected. This makes it possible to monitor, analyze and act on market data and respond immediately.

Many CEP engines were designed with unique in-memory architectures that ensure the continuous processing of events that can arrive in volumes of tens of thousands of events a second, process millions of concurrent CEP "rules," and make decisions in less than a millisecond.   The Apama "Correlator," depicted here, includes a patented technique of processing events based on an in-memory multi-dimensional indexing scheme called the HyperTree, in combination with a complex event sequencer, which optimizes Correlator performance by optimizing the processing of event scenarios that express the temporal and sequential event patterns that are expressed in CEP rules.  These rules can be expressed by using an eclipse-based development environment for the Apama CEP language or a high-level, graphical CEP Scenario Modeler that allows non-programmers (e.g., heads of desks) to "paint" strategies quickly and easily without programming expertise.

CEP correlation engines help fulfill imperative #8, to design algorithmic trading architectures for low-latency.

Algorithmic Trading Imperative #7: Integrate Real-time News into Algorithmic Trading

Today’s financial markets are moved by news, and firms are increasingly integrating electronic news into their algorithmic trading strategies. For example, news about US non-farm payroll numbers, global interest rate decisions or announcements associated with specific stocks all have an impact on the confidence in affected securities, and therefore prices. If a trading strategy can analyze and react to the news before a human trader, advantages can be realized. A complex event processing (CEP) algorithm can for example, contain the following rule: ‘Alert a trader if a news article is released on stock x, and is followed by a fall or a rise of greater than 5% in the value of that stock within five minutes.’

According to the WS&T article called "Trading Off News," many Wall Street firms appear to be proceeding with caution. "Machine interpretation of news is still beyond current science, and we're probably still waiting a few years before that's really going to evolve to fruition," says Carl Carrie, VP of JPMorgan Securities. "The classic challenge is interpreting the news."  The article continued to suggest that, "on the other hand, there are things you can do in the high-frequency sense without having in-depth interpretation involved, rather than interpret the direction of the stocks that's implied by the news, the heat indicator interprets if the news events impact that particular ticker by counting the number of news stories. For example, "If you are trading a stock like Red Hat and there's a whole bunch of news stories around Red Hat -- without having an interpretation of directional view -- you can assume that there's a higher level of risk that's associated with trading that stock," explains Carrie. The heat signal suggests there is an increase in volatility potentially. "So if that's the case, you can adapt that to your risk/trading models," says Carrie.

We experience this latter point - that increasingly, digitalized versions of news wire services, with meta data annotations are being used to algorithmically trade on news.  In fact, in a recent trip to Asia, we found that the localized news feeds in countries such as Korea are being used to trade locally on news.  So there's no doubt that imperative number #7, trading algorithmically on news, is happening.

Algorithmic Trading Imperative #6: Operate within Multiple Asset Classes

Algorithmic trading is gaining momentum in asset classes beyond its initial domain of equities, including derivatives, fixed income and FX. This is due in part to increased electronic access to liquidity sources via electronic APIs, such as EBS and Hotspot in FX. When a trading platform has electronic access to multiple asset classes, existing algorithmic strategies can be combined by operating within multiple assets simultaneously within a single strategy. For example, a firm might buy an equity and hedge it with a future, while taking out an FX position – all at the same time.

Algorithmic Trading Imperative #5: Gain Access to Multiple Liquidity Pools

With the rise of ECNs and DMA, the electronic markets are continuing to advance. Today, firms can gain advantage by spreading trading activity across these multiple pools, which differ in their strengths. For example, in the FX market, Currenex is similar to Hotspot, but it is not anonymous; EBS and Reuters Dealing 3000 are major players but they tend to be especially competitive in specific exchange rate pairs. Understanding the anomalies in the variety of liquidity pools can be a source for advantage, but the only way to gain this advantage is if your algorithmic trading platform can access multiple liquidity pools at the same time. Also, monitoring multiple pools in real time enables a strategy to route orders to the pool with, for example, the best price or the most available liquidity.

So an imperative #5 for algorithmic trading is to ensure your platform can connect to and operate on mutiple liquidity pools.

Algorithmic Trading Imperative #4: Evolve Algorithms Rapidly

As building and customizing algorithmic strategies is critical, so too is the rapid evolution of trading strategies. Markets are continually evolving and new opportunities, for example in the form of arbitrage, constantly emerge. If you do not develop strategies to capitalize on an opportunity quickly, then the competition will. Customization of trading strategies is not a ‘one-off’; strategies must be continuously and systematically evolved.  In the race for algorithmic supremacy, firms attempt to observe counter party trading activity and either automatically or manually ‘reverse engineer’ the strategies being used. As a result, firms must plan to rapidly evolve – or perish.

I have been presenting this week with Koscom, our partner in Korea - the screen shot here comes from their presentation, which illustrates the principle of following the imperative rapid evolution and of localize, localize, localize - at the same time.  Through localized rapid application development tools, Koscom delivers a differentiated algorithmic trading solution to the Korean market.

Algorithmic Trading Imperative #3: Localize, Localize, Localize

Today we issued a press release about the adoption of algorithmic trading in Asia.  In it we discussed algorithmic trading imperative number 3:  Localize, Localize, Localize.  Now it might appear obvious to say that localization of software in Asia is an imperative, but that's not the point.  Yes, language localization is critical, but more important in the field of trading is the ability to localize trading technologies for local strategies that can work in a specific market, for the local connectivity requirements the market requires.  For example,  Credit Suisse and Goldman Sachs have already localized their algorithmic trading strategies, and Sang Lee, Managing Partner, Aite Group comments: “Algorithmic trading in Asia Pacific began its development in a similar vein to Europe and North America with an initial focus on equities. Today, however, firms are rapidly adopting techniques that have taken longer to develop in the more established markets. These include the use of algorithms for foreign exchange and cross-asset class trading. As algorithmic trading in Asia continues to evolve, flexible, customizable technology that accounts for the unique characteristics of local markets is essential.”

Another element of localization in the Asian markets is around regulation.  In FTMandate, Andrew Freyre Sanders from JPMorgan said:

“You can quickly create a basic VWAP algorithm, cut up orders and send it into the market in Europe or the
US – with the resulting performance not being too bad. In certain markets in Asia, however, you cannot get away with slicing up orders and sending them to the market… due to the liquidity, big tick sizes, order books and a raft of differing regulations.”

An open, customizable, flexible "white box" algorithmic trading model, such Progress Apama's, facilitates the need to localize algorithmic strategies rapidly, which is also discussed at length in the press release.

So the need to localize, localize, localize is Algorithmic Trading Imperative #3.