Event Processing blog

SIFMA Retrospective

As a follow-up to my colleague Louis' report on last week's SIFMA show, I thought I'd add some thoughts of my own. My conclusion is that it was the most exciting SIFMA show I have experienced.  While I think attendance was down from previous years, I also think the quality of attendees was up. And for me personally, the excitement of being involved in some industry-moving announcements as well as meeting up with many of my colleagues from capital markets firms, vendors, press and analysts was highly invigorating.

So what were the highlights and take-aways for me?

1. CEP is clearly a theme that is getting a lot of mindshare. So many people said that CEP was a key theme of the show – which is great to hear after many years of working to help define the market. It’s also great to see this add to the momentum so soon after the Event Processing Technical Society was launched. The use cases of CEP are many and varied – and there was a lot of interest and questions around this at SIFMA. We demonstrated on our SIFMA booth 5 different CEP use cases on 5 different pods - algorithmic trading, smart order routing, managing FX market fragmentation, market surveillance and real-time bond pricing. Also the demands of CEP applications continue to make demands on the technology, and we were thrilled to demonstrate Apama 4.0 – which extends performance and user experience of CEP to new levels. Another supporting factor in the maturing of CEP is that there are starting to be very senior people in Capital Markets firms focusing on CEP as an enabling technology. Marc Adler from Citigroup   is a key example. He’s active in the community and on the STAC CEP committee, helping to define benchmarks. It was great to meet Marc at SIFMA and also to catch up with many  other esteemed colleagues from the CEP space.

2. The liquidity wars are hotting up. It was our pleasure to be involved in a press release with NYSE-Euronext which was certainly one of the big releases of the conference. Progress Apama will be hosted in the NYSE Euronext as part of the exchange's Advanced Trading Solutions offering. Traders will be able to download custom logic for algorithmic trading, risk management and smart order routing into the NYSE itself - with low latency connectivity to other trading venues via Wombat and Transact Tools. This arrangement turns NYSE into a technology provider as well as a one-stop-shop liquidity provider. This announcement was picked up by major press, including the Financial Times - in Europe, America and Asia -- see the article here.

3. Hardware is important – and so is “green”. The increase of capital markets data volumes require completely new software architectures – like CEP. But software is not always enough to support the low latency transport , processing and storage requirements. Many firms are turning to specialized hardware, combined with software – to create high performance solutions. Vhayu, for example, launched Squeezer – which combines hardware and software to supercharge their tick data offering. Also, Progress Apama were pleased to put out a joint announcement with Sun on a collaboration for end-to-end CEP solutions – combining Sun hardware and operating systems with Apama’s CEP platform and solutions. We demonstrated an end-to-end bond pricing application using the whole stack. Sun was one of the vendors who have a “green” aspect to their hardware – for example on a major CEP deployment, the hardware can be scoped for peak throughput – but can selectively shut down capacity to save power when event throughput is reduced. In this era of high energy costs and global warming there seems to be a lot of interest in this approach.

4. I love partying on the trading floor. Progress Apama were honored to be invited to a party at the NYSE to celebrate the latest developments at NYSE-Euronext (see picture at the top). It was a great pleasure to speak with our friends at NYSE-Euronext and to meet many of our old friends from the capital markets industry there – while sipping some delicious wine in that amazing place. In a way it is a shame that electronic trading is making the traditional trading floor a thing of the past – but there is something amazing about that place and I hope it stays just the way it is – even if it becomes a cool venue for other purposes. Thanks to NYSE-Euronext for inviting us – we had a great time.

I’m sure I’ve neglected a load of other trends and themes – but there’s my brain dump for the day. I’m interested to hear if you all agree.

John

TradeTech Recap

Colleague, Dr. Giles Nelson, CTO of Progress Software EMEA and a co-founder of Apama, took some time to share his thoughts about this year's TradeTech, Paris, which we have captured in the audio file below.

An Apama Hat Trick

Last week proved to be a busy one for Apama on the marketing front as we issued three separate announcements in conjunction with our presence at the TradeTech show in Paris.  Two of the announcements focused on customers, while the third focused on work that a partner is jointly doing with Apama in the area of market surveillance.

• ING Wholesale Banking announced that it is expanding its use of Apama, previously focused on algorithms for Benelux Small and Mid-Caps.  ING has re-engineered those algorithms to address markets in Hungary and Poland with a variety of features that include hybrid cross asset algorithms that leverage ING’s direct access within those emerging markets.

• SEB, the Scandinavian financial group, announced it will expand its use of Apama to deliver advanced order flow monitoring services within a compliance application.  This was a second SEB announcement, following one last year regarding the SEB deployment of Apama to support client trading in Exchange-traded equities and futures.

• And lastly, but certainly not least, together with Detica, a key partner, we jointly announced a Market Surveillance Accelerator.  Accelerators are extensions to the core Apama platform that help our customers jumpstart their deployments, incorporating business logic and other components like sample dashboards and adapters for connectivity.  In this instance, we are combining the technical know-how and experience of Detica and Apama – both of which are now supporting projects at the FSA and Turquoise - to address the growing demand for real-time market surveillance capabilities.  We’ve previously announced Accelerators for Market Aggregation and Smart Order Routing.   And there'll be more to come.
  

These three announcements collectively illustrate that a key part of the value of the Apama CEP platform is its versatility.  Apama may initially be deployed in support of a specific application like algorithmic trading or a specific asset class, but upon experience with the product, many of our customers expand their use to different asset classes, different geographic markets and/or entirely different applications – like compliance or risk or market surveillance.

On (Complex) Event Processing

I'm sure this has been said before, but I had cause to think again today on what is actual "complex" about CEP. I've never been particularly comfortable with the term CEP (of course, my personal comfort is not required ...) as it suggests that CEP is in some ways "hard". Well, those of us in the business of building CEP technologies know that the challenges of processing 10s of 000s of events per second with sub-millis latency against 000s of event patterns with temporal and logical constraints introduces a shed-load of complexity under the hood - but that's the point really - it is (or should be) all under the hood; we don't call databases "tricky databases" just because they have some fancy background indexing and schema evolution capabilities inside them.

Anyway, that ship has sailed - and I digress. The thing that got me thinking about this (again) was the recent press release regarding our launch - with our partner Detica - of a Solution Accelerator for Market Surveillance. That press release gives some examples of the kinds of surveillance strategies - "front running", "washing" ec. - that the Solution Accelerator provides (along with a handy definition of what this jargon actually means). The point is that the event processing logic required for these types of applications is not in any way "complex" - e.g. "detect a spike in trading volume or rapid price move within 10 minutes prior to a news release regarding a particular instrument" - despite the machinations under the hood that might be required to do this in real-time for all trades and news articles for all listed instruments on an exchange.

No, what is key in supporting these types of applications or strategies is the *lack* of complexity - and the provision of tools allowing strategy builders - here, operational teams at the exchange - to quickly express the business logic, generate a Dashboard to visualise the alerts it generates, and get all this into production before it becomes irrelevant. As markets and traders become ever more sophisticated it is the ease with which strategies can be modified and new strategies deployed that determines whether CEP is the right technology or not, not how clever it might be under the hood.

The term CEP seems to be here to stay, and I'm certainly not volunteering to be the flag carrier for a terminology battle ("Event Processing"? Anyone?). But let's be clear that neither the use case nor the hoops that need to be jumped through to deploy it need be complex for "CEP" to be an effective technology solution.

The key to effective CEP technology is to keep as much of the complexity as possible away from the people who have to use it.

Asia Report: Fighting White Collar Crime

Hello from Hong Kong. As always it is fascinating to see how CEP is evolving in Asia. One trend I am observing is the huge interest in Hong Kong in rogue traders and white collar crime – and how CEP can be used to detect and prevent this – before it moves the market. Obviously the original rogue trader, Nick Leeson, is well known here. But there has been a great deal of interest in more recent goings-on, at firms such as SocGen. Amazingly, until a couple of years ago, insider trading was not illegal in Hong Kong! Now we have a highly volatile market, with a lot of uncertainty, huge event volumes and a real problem of seeking out and preventing rogue trading activities, as well as managing risk exposure proactively.

Of course CEP provides a compelling approach. In market surveillance - the ability to monitor-analyze and act on complex patterns that indicate potential market abuse or potential dangerous risk exposure can allow a regulator, trading venue or bank to act instantly. Banks want the reassurance that they are policing their own systems. Regulators need to protect the public. The media and public here find this fascinating.

On the topic of a different kind of white collar crime – consider using CEP to detect abuse in the gaming industry. The gambling phenomenon that has propelled Macau to overtake Las Vegas as the world’s biggest gambling hub is also an exciting opportunity for CEP. We have customers using CEP to monitor and detect various forms of potential abuse in casinos. Events that are analyzed to find these patterns include gamblers and dealers signing on at tables, wins and losses, cards being dealt etc. It is possible to detect a range of potential illegal activities, ranging from dealer-gambler collusion to card counting.

As a final thought - having met with some of our customers that operate both in Hong Kong and mainland China, it is clear that China is a massive market opportunity for CEP. Exciting times ahead for CEP in Asia.

Real-Time Risk and Surveillance

Last week we heard of Société Générale and its $7B loss. Earlier this week, we at Progress announced that Turquoise, the European Multilateral Trading Facility (MTF), had selected Apama to support its market surveillance and anti-fraud operations (see the previous blog entry for more information on Turquoise). Following our Turquoise announcement therefore, a common question addressed to us has been: "how can technology, and more specifically Apama, be used for risk management and could it have prevented the SocGen loss?"

Specifically on SocGen, considering that the fraud was committed over an extended period of time and by someone well versed in the way that the bank's systems and processes operated, technology by itself certainly couldn't have prevented it. However fraud detection (as our Financial Services Authority and Turquoise customers show) and risk management is a key area where financial organisations have been investing in technology.

Trading and financial markets have accelerated substantially over the last few years. Take the New York Stock Exchange. Between 2001 and 2006 (the last year figures were available), the total number of transactions increased by nearly 400%. Such a change is happening elsewhere in the world. The total number of transactions on the London Stock Exchange for the same period increased by nearly 300%. These increases have been caused both by the total volume of shares traded going up and also by the average size per trade going down. Electronic trading in other products has also increased very substantially - foreign exchange for example.

With markets moving more quickly there is an increased need for organisations to have up-to-date information on their positions and risk exposure. By knowing when limits are breached or when markets have moved unfavourably positions can be adjusted or closed down quickly before things get out of hand. "Up-to-date" often has to mean "real-time" and with the right infrastructure, analysis and reporting technology then real-time really is possible. In particular this is where CEP technology, such as Apama, comes in.

The best known use for Apama in capital markets is algorithmic trading. Algo trading has been a cause, and a response to, the market velocity increases that have occurred. Organisations that have deployed algo trading systems are finding that their use is putting pressure on the middle-office risk function to have a real-time view on trading activity. This has led to the adoption of exactly the same CEP technology to monitor and analyse this trading activity and report in real-time on the associated risk positions. Many of our customers are using Apama for this purpose.

It is this real-time capability of CEP that appeals to both the FSA and Turquoise. As, respectively, a regulator and operator of a market they feel the pressure that comes from the increased velocity of market activity. Conventional market surveillance techniques, where perhaps activity is examined end-of-day or end-of-week are no longer viable. In fact, being able to detect abusive and fraudulent behaviour on a market is of particular importance to ensure the running of a fair market. Often such behaviour takes the form of planned, systematic and repeated interventions which are intended to move the market itself. If you can only find out what occurred days after the event then the markets have already been manipulated and the damage done. By detecting it as early as possible the behaviour can be investigated and stopped.

Can CEP technology used in this way actually create problems? A comment on the FinancialTech Insider blog makes the suggestion that having a real-time view on risk may throw up false positives and thus create more problems. Certainly this is possible, but it should be remembered that technology should always be supportive of a human-centred risk process. Only in the simplest cases will a person not be involved. In the majority of cases further analysis will need to be done and context brought to bear in order to reach a decision about what to do. The value of the technology is to identify issues and potential issues as soon as possible in order for them to be managed in a timely fashion.

Apama Wins Market Surveillance Deal at Turquoise

Progress Apama, along with our system integration partner, Detica, has been chosen by Turquoise, the European multi-lateral trading facility, to deliver a real-time market surveillance system. Turquoise is funded by nine of Europe's largest banks and is essentially a new electronic stock exchange that, when it goes live this year, will compete with traditional equities venues like the London Stock Exchange.

Explicit in the announcement is how important market surveillance can be in helping to ensure an orderly market that traders can trust to be fair to all participants. Such confidence helps promote liquidity, which is the lifeblood of any trading venue. Implicit in the announcement is the value of CEP-powered, real-time market surveillance in driving the rapid detection of potential patterns of abuse or other questionable behavior. As Apama observers will note, this is the second selection of Apama and Detica for a real-time market surveillance system.  Last year, the same vendor team was chosen by the British regulator, the FSA.

There has been much press in the last week regarding the problems that can occur when proper procedures – and technology – are not in place to monitor trading behavior.  The situations are different - the problems at the French bank Société Genéralé appear to involve the Futures trades of a single trader, while Turquoise will be an equities exchange. But billions in losses certainly shine a bright light on the value of being able to monitor activities and respond quickly, regardless of the financial instruments involved. And though the motivations might differ, the value is there whether the monitoring is done by individual firms, by the trading exchanges, or by regulatory authorities.