Event Processing blog

How One Very Naughty Algorithm Ruined Everyone's Day....

The chaos theory says that small differences in initial conditions can yield widely diverging outcomes. Thus it was that on May 6th when a mutual fund in Kansas entered a rather large ($4.1bn) sell order in E-mini S&P 500 futures contracts on the CME, the reverberations were felt throughout the marketplace. The order sparked a totally human panic on a day when fear was in the air and sentiment was leaning toward the bearish. The fire was then fueled and fanned by algorithmic trading strategies and high frequency trading, causing an unprecedented drop within minutes.

 

As regulators said following the flash crash: "a complex web of traders and trading strategies" links the fragmented multitude of markets here in the U.S. And, like dominoes, when one goes the rest follow. That a large sell order for E-mini S&P 500 contracts on the CME, executed by an aggressive broker's algorithm, had such a devastating knock-on effect throughout the whole of the marketplace is telling. At first, high frequency trading programs absorbed the liquidity but, as prices dove, they changed their tack and began to sell as well. Liquidity was choked off. As traders tried to make sense of the situation, it because clear that the rest of the market was a-tumbling.

 

The futures contract - technically an ETF - is a stock market index futures contract based on the S&P 500. And the S&P 500 after all is made up of 500 individual equities shares. Panic, although a human emotion, can also spread in the electronic sense. In this case when the ETF algos ‘panicked’, the 'emotion' spread across other asset classes in an instant. Inconsistent and inhomogeneous trading rules across the various destinations worsened the effect. As trading halted on NYSE other destinations kept churning, but liquidity was already strangled.

 

The post-crash joint report by the SEC and CFTC notes that on the morning of the flash crash there was a decidedly unsettled feeling in the market. The European debt crisis was top of the list, and risk indicators included higher premiums on credit default swaps for debt from Greece, Portugal, Spain, Italy, and Ireland and a weak Euro. The VIX shot up by over 30%, the fourth largest single-day increase. Prices on gold futures rose 2.5%, while yields of ten-year Treasuries fell nearly 5% as investors engaged in a “flight to quality", said the report.

 

Clearly it wasn't a good day to sell; but we all make mistakes. The report highlights just how easy it is to make a well-intentioned trading faux pas that can wipe hundreds of billions of dollars off the market within minutes.

 

Lessons have been learned and they will help to prevent another flash crash. I have some to add:

 

• Perform comprehensive pre-trade analysis, including by backtesting algorithms under a wide range of circumstances using realistic market simulation – If the mutual fund in question or its executing broker - had done a thorough back-test of its trading strategy, using some of the dire indicators and conditions present, it might have thought twice about selling so aggressively - possibly preventing the crash.
• "Light up" the algorithmic trading process. Visibility during the trading process is crucial. Market monitoring and surveillance technology exists than can monitor the markets for anomalous behavior and alert the parties involved if it is spotted. Red alerts should have been going off in the broker, with real-time risk analytics highlighting impending problems. Also, the regulators should have been able to see an “early warning” and respond from a NORAD-style monitoring HQ.
• Homogenize trading rules across all exchanges and ECNs. When one halts trading they all halt - for the same reason and for the same amount of time.

 

Gary Gensler, chairman of the CFTC, said following the report that perhaps both brokers and customers need to be obligated  "to monitor and make non-disruptive trading judgments." As Mr. Gensler noted, high volume is not necessarily an indicator of real liquidity. The more visibility we have into trading, the more responsive we can be and the more likely we are to avoid another flash crash.

 

Mystery Solved, but Questions Remain

Scooby Doo and the gang have been investigating the spooky mystery of the "phantom orders" on CME and they found out that the culprit was... wait for it.... The CME! Turns out the exchange plugged in some new contracts to try them on for size and they mistakenly went live and bam! they got traded.

According to the FT, CME Group said it had “inadvertently” posted test orders intended for its quality assurance procedure on Globex (http://tinyurl.com/23bpfjj). CME said the company tests its systems as a matter of course and it had not determined whether human error or a computer glitch caused the mistake.

The FT said the mistaken order flow began at 3:38pm ET time on Monday, September 13th and lasted for six minutes. Futures brokers noticed oddly anomalous spread price activity during that period, when trading is usually slow.

CME said it was working with the customers that somehow managed to trade these contacts in those six minutes, and says it won't "bust" the transactions. Fair enough. But the mistake shows how the exchange lacked adequate monitoring technology to see that some test contracts had entered the live environment - and were being traded.

This six minute phantom orders mystery is solved, but the very fact that it occurred supports my argument that brokers, traders and exchanges need to have more advanced capabilities to detect problems and abuse as they happen, and recommend actions to take in response. The Mystery Machine got to the bottom of the case, but it was not fitted with the latest real-time surveillance equipment. It seems, like Velma, the CME temporarily lost their glasses!

I Would’ve Got Away With It Too - It If It Wasn’t for You Pesky Kids….

Strange things are happening in the markets! As you know if you’ve been following this blog, quite a few scares, horrors and apparitions have been sighted in equities, futures, FX and oil markets over the last few months. But fortunately the Mystery Machine has rolled into town and Scooby, Shaggy and the gang are investigating…

 

The latest spooky mystery is “phantom orders” that appeared to scare participants on the CME (read more about it here). For 6 minutes on Tuesday unexplained orders, that could have been caused by a “fat finger” error or an “algo gone wild”, caused CME to use Rule 587 which gives them powers “including without limitation, closing the market, deleting bids and offers, and/or suspending new bids and offers”. So although they weren’t sure what was going on – the CME were at least able to respond.

 

Another report this week shows how those involved in alleged market abuse are starting to be unmasked. Last week I wrote a blog posting called “Algos Can Get Stuffed” which was also featured on the Tabb Forum. In it I talked about the possibility of firing orders into the market with the intention of misleading the market or slowing the market down – and even how some link this phenomenon with the flash crash. This week a trading firm called Trillium was fined by FINRA for using illegal trading practices (read more about it here).  Trillium was fined $1m for sending orders aimed at deceiving investors. Nine traders entered buy and sell orders in patterns that aimed to manipulate the prices of instruments. And they did this 46,000 times! This “layering” enabled Trillium to trade at a profit once they’d manipulated the price.

 

These 2 incidents show that awareness of the problems we’ve been writing about on this blog have increased radically. Trading venues are more aware that algos gone wild and fat fingers can cause market panics and manipulate prices. Regulators are more aware that high frequency trading can be used as a weapon for market manipulation.

 

But we can’t rest on our laurels. Maybe we got lucky this time. As market data volumes continue to increase we need to have more advanced capabilities to detect problems and abuse as it’s happening, and recommend actions to take in response. Let’s ensure the Mystery Machine is fitted with the latest real-time surveillance equipment. Let’s enable the gang to unmask more villains at the haunted “high frequency manor” inspiring the legendary outburst of “I would’ve got away with it if it wasn’t for you pesky kids….”

Algos can get stuffed!!

As regulators continue to look into the causes of the May 6th flash crash, some high frequency trading approaches are coming under scrutiny. In particular the concept of “quote stuffing”, where algorithms send so many orders into the order book that the market cannot possibly respond, has come under fire from market participants and the press.

 

The SEC, having reportedly decided that quote stuffing probably did not have a major role in the flash crash, is now taking aim at the practice to see if it puts some investors at a disadvantage by distorting stock prices (http://tinyurl.com/264kr3o). The CFTC is looking at data from database developer Nanex and mulling how to address quote stuffing in futures markets (http://tinyurl.com/3a7w7sv).

 

Meanwhile, concerning incidents continue to happen in the market. As recently as last week there was an incident that caused Christopher Steiner at Forbes to write a story called “Did we dodge another flash crash on September 1st?” The story describes how on September 1^st at 10am quote volumes ballooned - as they did on May 6^th. In fact quotes reached 275,000 per second, as opposed to 200,000 on May 6^th. Unlike the flash crash though there wasn’t a dramatic fall in prices. However, the bids and offers did cross for a time – leading to high frequency traders taking advantage of arbitrage opportunities. This data was exposed again by firm Nanex – and left the market wondering if quote stuffing by high frequency traders was behind the spikes.

 

The world is waking up to the fact that high frequency and algorithmic trading have quietly become part of the market fabric, and the world does not seem to be too happy about it. HFT and algorithms are being "demonized" said the FT article, and I agree. I also think the hype is overblown.

 

Once trading became automated, trading strategies naturally morphed to take advantage of the available technology and higher speeds. High frequency statistical arbitrage techniques can also mean more order cancellations, some of which may - wittingly or unwittingly - fall into the quote stuffing category. Those involved in intentional quote stuffing as a strategy need to be held to task. But to demonize all strategies or call for banning them is a step backward. What is needed is a framework by which to police them - and to prevent them from going wrong. The technology to do this is already available. For example, an area I’ve had a lot of experience in is the use of complex event processing to provide a platform for high frequency, multi-venue market surveillance. With such a system quotes can be monitored to determine how many quotes per second there are on each ticker symbol, the ratio of quotes to trades, when large spikes are emerging and many other interesting real-time analytics and patterns that it’s useful to track in real-time.

 

But regulation of high speed trading practices has fallen short to date. Regulators have not had the funds, the technology, the power or the expertise to follow and control high speed trading. However, it is good to see that progress is now being made. CFTC commissioner Bart Chilton wrote last week in an article entitled Rein in the cyber cowboys: “There may be some cyber cowboys out there and they could be giving respectable traders a bad name”. His colleague CFTC commissioner Scott O'Malia told Reuters last week that, if traders are flooding the market with orders with the intention of slowing others down, the regulator would consider addressing quote stuffing under new rules in the financial regulation bill that deal with disruptive trading practices.

 

It is possible that quote stuffing is causing more problems that just slowing down the natural flow of trades. Trading behavior patterns suggest that these quotes are a distraction to other traders. There are patterns evident where the quote "stuffer" continuously traded first - possibly by distracting others. And the disruption can cause the bid and offer to cross – providing a nice arbitrage opportunity for those who are not distracted! All of this needs to be looked into further. I fully support the CFTC and SEC's efforts to get to the bottom of not just the flash crash, but HFT and algorithmic trading practices. They are now integral in the equities and futures markets, and increasingly so in FX, fixed income and energy. What we need is better policing of the markets to protect the honest ranchers from the cyber cowboys.

Taming the Wild Algos

"And now," cried Max, "let the wild rumpus start!"

— Maurice Sendak: Where the Wild Things Are

 

It’s not just equities and futures markets where strange stuff happens! An “algo gone wild” was spotted in the oil market (it actually happened earlier this year) and intrepid Reuters journalists got to the bottom of it.

 

High frequency trading firm Infinium Capital Management is at the center of a six-month probe by CME Group (and reportedly the CFTC) into why its brand new trading program malfunctioned and racked up a million-dollar loss in about a second, just before markets closed on Feb. 3. The algorithm, which was brand new, went live 4 minutes before the end of trading. It fired in 2000-3000 orders per second before being shut off. The oil price surged $1 then slid $5 over the course of the next two days. Read about the full story here:

http://www.reuters.com/article/idUSTRE67O2QQ20100825

 

I know the CEO of Infinium Chuck Whitman from the CFTC technology advisory committee – he’s a good guy and very knowledgeable. I believe him when he says his wild algos had no malicious intent – the algos were just broken and shouldn’t have been put live.

 

With algorithms and HFT comes the possibility of mistakes. Many more firms outside of the equities world are embracing HFT and their inexperience can cause market disruptions such as the Feb 3rd CME issue. A flash crash in oil or other commodities - or even foreign exchange - is not to be scoffed at. In fact, many commodities markets are much less liquid and homogenous than equities, and can be even more vulnerable to mistakes or manipulation. In the case of Infinium, the algo caused a spike in trading volumes by nearly eight times in less than a minute. It was a classic case of the algo running wild until it faltered and 'choked'. This is not how HFT strategies are supposed to work.

 

There are a number of best practices that can be used to mitigate against algos going wild:

 

The first best practice is diligent backtesting – using historic data and realistic simulation to ensure many possible scenarios have been accounted for. What does the algo do in a bull market, a bear market, at the open, at the close, when unexpected spikes occur, during a flash crash, when non-farm payrolls or other economic news is released etc. etc.? Of course there’s always the possibility of a “black swan” scenario – but then there’s always the possibility of an earthquake in London – but I bet the buildings aren’t built to withstand one – it’s a matter of covering likely possibilities as best you can. A backtesting process needs to be streamlined of course – as short time to market of new algos is key.

 

A second best practice is building a real-time risk firewall into your algo environment. Just like a network firewall stops anomalous network packets reaching your computer, so the risk firewall should stop anomalous trades getting to trading venues. These anomalous trades might be human or computer generated – such as “fat finger” errors, risk exposures (for a trader, a desk or an institution) being breached, or even algos gone wild (e.g. entering spurious loops and spitting out anomalous orders). Real-time risk monitoring is a second level protection for those problems you don’t catch in backtesting.

 

A third best practice is to use real-time market surveillance in your algo environment. Even if trades do not breach risk parameters, they may breach compliance rules, regulations or may be perceived by a regulator as market manipulation (by accident if not design). Detecting these patterns as they happen enables good internal policing by trading firms, rather than investigation or prosecution by regulators.

 

An algorithm is a tool in a trader's toolkit, and it needs to be taken care of as such. If it is well-oiled and the trader or quant or risk manager monitors its progress then the algo will do its job quickly and nicely. If the trader/quant/risk manager doesn’t properly prepare the algo or ignores the algo and lets it get rusty, so to speak, it could lose its edge and run amok. Algorithms must be monitored constantly for performance and for errors, and sometimes tweaked on-the-fly to ensure best results. A good algorithmic trading platform will enable trading firms to do just that.

 

Trading firms are not the only ones who need to be on guard for possible algos gone wild. In the case of Infinium, the regulators and the exchange were also slow on the uptake. This shows that everyone needs to be proactive in using the correct tools to monitor algorithmic trading. Sensing and responding to market patterns before the aberrations or errors have a chance to move prices is the right thing to do - in all asset classes. Be like Max and tame the wild things!

The Risks of Being Caught Naked

According to the FT, the Financial Industry Regulatory Association is undertaking a “sweep” of broker-dealers that offer direct market access to high-frequency traders to find out if they have pre-trade risk management controls in place for their algorithmic customers. (Read the full story here: http://tinyurl.com/28rg287). Here at Progress Software we have been advocating the use of pre-trade risk management tools for some time. The prospect of High Frequency Trading (HFT) firms running trades directly through broker algorithms (naked access) to exchanges/ECNs without adequate supervision has always concerned us. Brokerage firms simply give their clients an exchange ID number, which the clients then enter into their algorithms so that they can trade directly with the exchange.

Regulators are right to be concerned. Unfiltered access to trading destinations can end up causing trading errors or worse. Anything can happen with naked access; from fat fingered errors, trading limits being breached, even fraud - all of which can cost brokers, traders and regulators big money.

Although the SEC proposed banning naked access in January, and the Committee of European Securities Regulators (CESR) is likely to follow in its footsteps, there has been considerable pushback from brokers and trading firms. They say that adding a pre-trade risk management step would cause a "latency hop" in the trading process, claiming it would make them uncompetitive.  A layer of risk management often adds a third party into the mix, and - consequently - a delay.

But if everyone is required to add real-time, pre-trade risk management, then the competitive disadvantage is moot. The benefits of being able to pro-actively monitor trades before they hit an exchange or ECN far outweigh any microscopic latency hops in any event. The naked access playing field is hardly level. There are trading systems out there that claim to throughput a trade in 16 microseconds from outside an exchange's firewall, while exchange and ECN systems range anywhere from 250 to 700 microseconds (according to measurements by latency management firm Corvil).

Latency is clearly an important issue and complex event processing offers a solution. Brokers can achieve ultra-low-latency pre-trade risk management without compromising speed of access.  An option is a low latency "risk firewall" utilizing complex event processing as its core, which can be benchmarked in the low microseconds.  With a real-time risk solution in place, a message can enter through an order management system, be run through the risk hurdles and checks, and leave for the exchange a few microseconds later.

It is the ideal solution to a tricky question - how do I manage risk before my trade hits the exchange without compromising speed? The benefits are clear - a latency hop of a few microseconds not only saves money, it can also prepare your firm for the regulatory future.

Evacuate the Dancefloor

Looking for all the world like someone yelled "fire" in a crowded nightclub, prop and quant traders are stampeding out of investment banks and headed for the hedge fund world. Some, mainly the prop traders, are being pushed gently out the door as banks prepare for the Volcker Rule (http://tinyurl.com/39ap28d). Others, like the quants (http://tinyurl.com/23c5h6d), are in search of the mega-bonuses that their prop trader or hedge fund manager compatriots are (or were) getting.

 

Impending changes in regulation are prompting banks to spin off proprietary trading activities, many by expanding their operations overseas where Messieurs Dodd and Frank cannot reach them. I’m very concerned about this “regulatory arbitrage” in which firms may move away from the US to find less strict regulatory regions. We don’t want to lose the lead in this important area of the economy.

 

Spin offs and regulatory arbitrage may well leave a herd of US traders looking for work and many may end up working at - or starting - hedge funds. The quants, having slaved over hot computers for the last few years to line bankers' pockets, are forming their own trading companies or joining prop trading firms with a profit-sharing deal.

 

Most of these traders will be in for a rude awakening when they sit down to work. Prop traders joining hedge funds will find that the technology budgets may not be as generous as they were at their last bulge bracket employer's firm. The quants, who are essentially programmers, will face huge challenges in finding firms that have the kind of low latency, scalable architecture that they need to design, tweak and trade with their algorithms. The level of trading freedom is different, too. Hedge fund managers will have something to say about a trader's profits - or lack thereof. Quants may find that designing an algorithm and handing it over to the trading desk is not quite the same as being responsible for the profits that the algo makes - or doesn't make.

 

Make no mistake, these prop traders and quants are highly intelligent and adaptable people. There will be many challenges to face going forward, but technology need not be one of them. There are instantly useable, scalable platforms that quants and hedge funds can use to build and deploy algorithms. These platforms, such as Progress Apama's Complex Event Processing Platform, offer a robust technology infrastructure to successfully create, test, deploy and manage their algorithmic strategies.

 

Algorithmic trading software is constantly transforming. As the volume of real-time market data continues to increase, algorithmic trading solutions demand an infrastructure that can respond to market data with near zero latency. To trade effectively in competitive markets requires rapid, opportunistic response to changing market conditions before one's competition can seize those opportunities. The people that are running for the doors and into the arms of hedge funds or other trading firms, will need this advantage. Competition is fierce, and their previous employers already have the technology advantage.

The Busy "B" in BRICS

Here in the United States algorithmic and high frequency trading have been under fire lately, blamed for everything from the May 6th flash crash to ruining small investors' profit margins. In the rest of the world, however, algorithmic trading is being recognized as both an opportunity and a way to attract liquidity and it is starting to blossom.

 

Exchanges in several emerging nations are getting with the electronic trading program - many by doing cross-listing (and even cross-ownership) deals with some of the larger, more automated exchanges. As Jeremy Grant put it in an article for the FT: "You scratch my back, and I’ll scratch yours." (see link here: http://www.ft.com/cms/s/0/123ece18-99ad-11df-a852-00144feab49a.html). Eurex and the CME have done many such deals recently, spanning Brazil, Malaysia, India, South Korea and Mexico. It makes sense for both parties - the big exchanges get exposure to new markets, and the smaller ones get to take advantage of the electronic expertise of the larger ones.

 

Brazil was one of the first to get on board when Brazilian Securities, Commodities and Futures Exchange (BM&FBovespa) was created by the merger of Brazil’s derivatives and stock exchanges in May 2008. Of all the BRICs (Brazil, Russia, India and China) Brazil is the real busy bee. In June 2009 the BM&F closed its “open outcry” pits in favor of an all-electronic GTS trading system (already introduced by Bovespa in 2004). The same month, the exchange joined the global trend to “co-location”, allowing customers to install servers inside the exchange’s data centers, ensuring low-latency for high-speed automated trades. These two moves set up BM&FBovespa as a destination that attracted algorithmic trading like bees to honey. The big banks are flocking in - Bank of America, UBS and Goldman Sachs have already announced the availability of algorithmic trading in Brazil. And an avalanche of brokerages have geared up with technology to enable algorithmic trading for their customers.

 

Brazil’s appetite for automated trading – particularly high-speed, low-latency algorithmic trading – remains very keen. Progress Software is excited to be part of this growth story. In February 2009, Ágora Corretora, one of Brazil’s largest brokerages, offered custom algorithmic trading strategies to its buy-side clients, through Progress Software’s Apama algorithmic trading platform. Around the same time another Brazilian brokerage, Alpes Corretora, deployed the same Apama algorithmic trading to develop and deploy execution arbitrage strategies for futures and equities.

 

Since mid 2008, over twenty Brazilian banks and brokerages rushed to make the switch from manual to electronic trading using Progress Apama’s algorithmic trading platform. These also include FinaBank Corretora, Banco Fator and most recently Link Investmentos, which bought in Apama’s algorithmic trading platform in March 2010. And many of these large banks are now offering their buy-side clients the chance to fine-tune their already customized algorithmic trading strategies, as per in Europe and the US. (It’s worth noting Apama also has direct buy-side clients.) Our Brazil success story has much to do with the fact that Progress is still the only capital markets-focused technology provider with an established local presence in São Paulo, Brazil, offering customizable technology.

 

BM&FBovespa will doubtlessly continue to incentivize both local institutions and international investors. The exchange’s tie-ups with CME and Nasdaq OMX Group will continue to drive new business growth in Brazil this year and into 2011. And the introduction of co-location at the Bovespa will further boost algorithmic trading both locally and internationally.

 

Brazil has been an incredibly exciting success story for Progress and our customers over the last 2 years. Now other firms are beginning to sit up and take notice of the region! But Progress was there first; we’ve gathered huge experience in the region and we remain ready and willing to provide more new customers with the ability to deploy and customize algorithmic trading solutions.

 

So Brazil is busy ‘B’! I’m looking forward to reporting more on the RICs in BRICs soon.

Algorithmic Terrorism

At the CFTC's first Technology Advisory Council meeting on July 14, there was concern expressed around the concept of quote-stuffing. There was some evidence presented that the May 6th flash crash may have been caused by or exacerbated by this activity. While with regard to the flashcrash, other market experts I’ve spoken to know dispute this was the cause, quote-stuffing is a topic worthy of discussion

 

At the CFTC meeting, where I was an invited participant, data was presented from trade database development firm Nanex, which suggested quote stuffing contributed to the destabilization on May 6^th. In this case the data suggests huge numbers of quotes were fired into the market on particular symbols (as many as 5000 per second) and that many of these were outside the national best bid/offer (NBBO). So what’s the point of this? Well with latency as a key weapon, one possibility is that the generating traders can ignore these quotes while the rest of the market has to process and respond to them – giving an advantage to the initiator. Even more cynically one can consider these quotes misleading or even destabilizing the market. In fact, Nanex state in their paper: "What we discovered was a manipulative device with destabilizing effect". Quote stuffing may be innocent or an honest mistake, but Nanex's graphs tell a very interesting tale (http://www.nanex.net/FlashCrash/CCircleDay.html). There are patterns detected - on a regular basis - that one could conclude is quote stuffing for the purpose of market manipulation. There's a very good article by Alexis Madrigal that discusses the research and issues in more detail (http://www.theatlantic.com/science/archive/2010/08/market-data-firm-spots-the-tracks-of-bizarre-robot-traders/60829/).

 

At the extreme, quote-stuffing could operate like a “denial of service attack” – firing so many orders that the market can’t cope - and crippling the trading of certain symbols, certain exchanges or the whole market. An influx of orders in sudden bursts to one exchange on one stock can slow down that system as it tries to process these orders. Nanex notes that there are 4,000 stocks listed on the NYSE and nine other reporting exchanges in the U.S. If each reporting exchange for each stock quoted at 5,000 quotes per second it would equal 180.0 million quotes per second. A daunting task no matter how advanced their processing technology is.

 

Without trying to overstate the issue, in the most extreme circumstances these practices could be considered algorithmic terrorism. One can imagine how, at the extreme, it is potentially catastrophic. The concern is that a well-funded terrorist organization might use such tactics in the future to manipulate or cripple the market. So much of our economy is underpinned by electronic trading – so protecting the market is more important than guarding Fort Knox! Regulators, such as the CFTC and SEC are taking this seriously - and need to respond.

Smart - but is it Smart Enough?

Nasdaq liked the idea of real-time market surveillance so much that it bought one of the companies that specialize in it.

 

Nasdaq OMX announced this week that it will buy Smarts Group, an Australia-based market surveillance business that helps exchanges monitor compliance with trading rules.  You can read the full story here: http://online.wsj.com/article/BT-CO-20100727-712964.html.

 

The market moves a lot faster than it used to thanks to algorithmic trading. What has not kept pace is monitoring high speed trading. Smarts is one commercial approach that aims to enable such monitoring. However, there is a big problem with Smarts - the time it takes to develop a new surveillance scenario. I have spoken to a number of venues around the world, including the Australian Stock Exchange, who have told me they are totally dependent on Smarts to add new rules when they need one – and it takes 6 months to a year – if they’re lucky. In fast-moving markets we need to evolve in hours not years!!

 

Despite shortcomings with Smarts, the Nasdaq acquisition is an indicator of the importance of real-time surveillance in a post-flashcrash world. Maybe the flash crash of May 6th has a silver lining if lessons learned are leading exchanges to better use surveillance and monitoring. In the aftermath of the crash, exchanges scrambled to recover trading data and do some forensic investigation into the causes.  This proved extremely difficult probably because of inadequate analysis capabilities to pinpoint what had happened.

 

Exchanges, ECNs, brokers, traders and regulators all must take an intelligent approach to monitoring and surveillance in order to prevent rogue trades and fat fingers. Transparency is the key. Regulators in the US and Europe are concerned about the lack of transparency in markets where high frequency algorithmic trading takes place, as well as in dark pools.

 

We ran a survey at SIFMA this year where we asked 125 attendees about high frequency trading and market surveillance. A staggering 83 percent said that increased transparency is needed to effectively deal with market abuse and irregular market activity, such as the flash crash. However, only 53 percent of firms surveyed currently have real-time monitoring systems in place.

 

Nasdaq says that Smarts will be used to expand broker surveillance solutions, which I take to mean monitoring scenarios such as sponsored access. This would be a smart move (forgive the pun). With naked access, high frequency traders can plug straight into an exchange through their broker – and it’s critical that pre-trade risk and surveillance is in place to prevent a crisis in which wild algos could cause havoc.

 

The detection of abusive patterns or fat fingered mistakes must happen in real-time, ideally before it has a chance to move the market. This approach should be taken on board not just by the regulators, but by the industry as a whole. Only then can it be one step ahead of market abuse and trading errors that cause a meltdown (or up).

 

As many market participants have pointed out, technology can't solve all of the problems, but it can help to give much more market transparency. To restore confidence in capital markets, organizations involved in trading need to have a much more accurate, real-time view on what's going on. In this way, issues can be prevented or at least identified much more quickly.

 

While I applaud Nasdaq's initiative and dedication to improving market surveillance buying Smarts, I must point out that you don't have to go quite that far to get the same results. Progress provides market-leading real-time monitoring, surveillance and pre-trade risk – powered by Complex Event Processing – enabling complex real-time monitoring of the fastest moving markets. Unlike Smarts, Progress includes the ability for business users to customize and create new scenarios rapidly (in hours rather than Smart’s months). And you don’t have to buy and integrate our company to get access to it!!