Event Processing blog

Real-Time Temperature Monitoring

In an interesting demonstration of real-time monitoring, Apama participated in a recent study that was conducted by Radboud University as part of the Dutch "Four Days March".  The march is an annual event in which tens of thousands of participants walk daily distances of 30-50 KMs for four days in succession.  Unfortunately, past marches have had some unfortunate instances where participants have been overcome by heat - including a couple of deaths.

In a pilot study this year, volunteers took a pill embedded with an RFID chip and thermometer, which sent signals every 10 seconds to a Bluetooth-enabled GPS phone.  The phone in turn sent the information to an implementation of Apama, which correlated the information about the volunteer, their temperature and their location, all plotted on an implementation of Google Maps. Leveraging the capabilities of Apama, monitors could track the progress of volunteers and identify those volunteers whose temperatures exceeded certain thresholds.  And with the ability to correlate that information with the GPS data, they could tell where those volunteers were on the route, thus delivering a CEP-driven infrastructure for real-time monitoring and, if needed, a pre-emptive reaction to help somebody who might shortly be in distress.

In the image below, one can see the purple "pins", illustrating where on the route the volunteers are.

SIFMA Retrospective

As a follow-up to my colleague Louis' report on last week's SIFMA show, I thought I'd add some thoughts of my own. My conclusion is that it was the most exciting SIFMA show I have experienced.  While I think attendance was down from previous years, I also think the quality of attendees was up. And for me personally, the excitement of being involved in some industry-moving announcements as well as meeting up with many of my colleagues from capital markets firms, vendors, press and analysts was highly invigorating.

So what were the highlights and take-aways for me?

1. CEP is clearly a theme that is getting a lot of mindshare. So many people said that CEP was a key theme of the show – which is great to hear after many years of working to help define the market. It’s also great to see this add to the momentum so soon after the Event Processing Technical Society was launched. The use cases of CEP are many and varied – and there was a lot of interest and questions around this at SIFMA. We demonstrated on our SIFMA booth 5 different CEP use cases on 5 different pods - algorithmic trading, smart order routing, managing FX market fragmentation, market surveillance and real-time bond pricing. Also the demands of CEP applications continue to make demands on the technology, and we were thrilled to demonstrate Apama 4.0 – which extends performance and user experience of CEP to new levels. Another supporting factor in the maturing of CEP is that there are starting to be very senior people in Capital Markets firms focusing on CEP as an enabling technology. Marc Adler from Citigroup   is a key example. He’s active in the community and on the STAC CEP committee, helping to define benchmarks. It was great to meet Marc at SIFMA and also to catch up with many  other esteemed colleagues from the CEP space.

2. The liquidity wars are hotting up. It was our pleasure to be involved in a press release with NYSE-Euronext which was certainly one of the big releases of the conference. Progress Apama will be hosted in the NYSE Euronext as part of the exchange's Advanced Trading Solutions offering. Traders will be able to download custom logic for algorithmic trading, risk management and smart order routing into the NYSE itself - with low latency connectivity to other trading venues via Wombat and Transact Tools. This arrangement turns NYSE into a technology provider as well as a one-stop-shop liquidity provider. This announcement was picked up by major press, including the Financial Times - in Europe, America and Asia -- see the article here.

3. Hardware is important – and so is “green”. The increase of capital markets data volumes require completely new software architectures – like CEP. But software is not always enough to support the low latency transport , processing and storage requirements. Many firms are turning to specialized hardware, combined with software – to create high performance solutions. Vhayu, for example, launched Squeezer – which combines hardware and software to supercharge their tick data offering. Also, Progress Apama were pleased to put out a joint announcement with Sun on a collaboration for end-to-end CEP solutions – combining Sun hardware and operating systems with Apama’s CEP platform and solutions. We demonstrated an end-to-end bond pricing application using the whole stack. Sun was one of the vendors who have a “green” aspect to their hardware – for example on a major CEP deployment, the hardware can be scoped for peak throughput – but can selectively shut down capacity to save power when event throughput is reduced. In this era of high energy costs and global warming there seems to be a lot of interest in this approach.

4. I love partying on the trading floor. Progress Apama were honored to be invited to a party at the NYSE to celebrate the latest developments at NYSE-Euronext (see picture at the top). It was a great pleasure to speak with our friends at NYSE-Euronext and to meet many of our old friends from the capital markets industry there – while sipping some delicious wine in that amazing place. In a way it is a shame that electronic trading is making the traditional trading floor a thing of the past – but there is something amazing about that place and I hope it stays just the way it is – even if it becomes a cool venue for other purposes. Thanks to NYSE-Euronext for inviting us – we had a great time.

I’m sure I’ve neglected a load of other trends and themes – but there’s my brain dump for the day. I’m interested to hear if you all agree.

John

Asia Report: Fighting White Collar Crime

Hello from Hong Kong. As always it is fascinating to see how CEP is evolving in Asia. One trend I am observing is the huge interest in Hong Kong in rogue traders and white collar crime – and how CEP can be used to detect and prevent this – before it moves the market. Obviously the original rogue trader, Nick Leeson, is well known here. But there has been a great deal of interest in more recent goings-on, at firms such as SocGen. Amazingly, until a couple of years ago, insider trading was not illegal in Hong Kong! Now we have a highly volatile market, with a lot of uncertainty, huge event volumes and a real problem of seeking out and preventing rogue trading activities, as well as managing risk exposure proactively.

Of course CEP provides a compelling approach. In market surveillance - the ability to monitor-analyze and act on complex patterns that indicate potential market abuse or potential dangerous risk exposure can allow a regulator, trading venue or bank to act instantly. Banks want the reassurance that they are policing their own systems. Regulators need to protect the public. The media and public here find this fascinating.

On the topic of a different kind of white collar crime – consider using CEP to detect abuse in the gaming industry. The gambling phenomenon that has propelled Macau to overtake Las Vegas as the world’s biggest gambling hub is also an exciting opportunity for CEP. We have customers using CEP to monitor and detect various forms of potential abuse in casinos. Events that are analyzed to find these patterns include gamblers and dealers signing on at tables, wins and losses, cards being dealt etc. It is possible to detect a range of potential illegal activities, ranging from dealer-gambler collusion to card counting.

As a final thought - having met with some of our customers that operate both in Hong Kong and mainland China, it is clear that China is a massive market opportunity for CEP. Exciting times ahead for CEP in Asia.

CEP down under

I’m sitting here at Melbourne Airport in Australia on my way to Hong Kong. I’ve been delayed by a typhoon – probably a good reason to delay. After a very successful week in Sydney and Melbourne visiting customers, I thought I’d report that the CEP market is hotting up down under! As you would expect financial services is an early adopter and Apama has had several customers in Australia in this space for a few years now. But the demand is increasing. This is driven by factors such as increasing competitive pressures in the trading space and the impending fragmentation of the Australian market. Just like in Europe and North America, it is likely that several new trading venues will join the Australian Stock Exchange in offering liquidity in Australia. My diagram shows some of these in the form of Chi-X, AXE and Liquidnet.

Complex Event Processing offers a powerful way of monitoring, aggregating and analyzing the liquidity across all of these markets, as well as making real-time routing decisions. This of course can work in parallel with traders and algorithms. In fact it is becoming very interesting to see trading decision algorithms routing messages to execution algorithms, routing messages to liquidity tracking algorithms, routing trades to the market, which are being checked by market surveillance algorithms -- and all part being implemented in CEP. I am biased of course, but what other technology can offer the seamless federation of such systems. Events provide a powerful and low latency mechanism for such interoperation. Each component can be built independent of the other - but yet they can work together seamlessly. But I am getting off topic!

Over the last few years Australia has mainly been interested in equities algorithms, but now the interest in FX, futures, bonds and commodities is growing. While I was in Sydney, I was pleased to deliver the keynote address at the Trading Technology conference and met many interesting sellside and buyside participants with a variety of trading interests. It was fascinating to see how the market is developing.

And it is not just financial services where CEP is being applied down under. I also met with organizations in a number of other spaces including travel, transportation and location-based services. I hope to report more on these in the near future.

And now I look forward to finding out what is happening in Hong Kong and Asia beyond. Hopefully I can avoid the typhoon!

John

CEP Use Case: Airlines Operations Monitoring

The recent "JetBlue disaster" in the United States has motivated many airlines to expand their usage of event processing to more effectively monitor and control their flight operations in real-time, and to apply intelligence with complex event processing (CEP) as part of an event driven architecture (EDA).  A public example of this trend is TIBCO's recent  announcement that Air France / KLM, who have chosen TIBCO BusinessEvents, their CEP product, for this kind of application as part of their SOA middleware infrastructure.

Event processing technologies - business activity monitoring (BAM) dashboards, CEP, and event data management can monitor an airline's operational events - from passenger check-in, baggage handling (bag on conveyor, bag off conveyor, bag loaded on ULD, ULD loaded on plane, etc.), and flight operations (flight leaves gate, flight lands, etc.).  Most airlines process "events" manually today, but the increasing desire for intelligent automation is driving the use of EDA to process them.  By applying CEP logic to these events, an airline can better ensure that the right bags get loaded into the right unit loading device (ULD), or ensure that flights leave the gate and take off on time.  CEP rules, or scenarios, can monitor for conditions such as:  "alert baggage handling operations for any bag that was checked in and not loaded onto a ULD within 15 minutes," or, "alert flight operations when a flight has departed the gate but not taken off in 20 minutes," these complex sequences of events can be used to manage airline operations in an intelligent, automated way.

Business activity monitoring (BAM) dashboards provide rich graphical views of baggage, flights, ULDs, even P&L's or important frequent flyers.  BAM dashboards can provide visibility for baggage handling, flight operations, gate agents, even airline executives a real-time and historical view of the business of travel.  They can even show of the resulting real-time P&L for operations.  The dashboard shown is one of many use cases that shows, in real-time, the airport P&L for the past week, flights that are behind schedule in the last 5 days, or last 24 hours, etc., exactly where bags are in the baggage handling process, and more.

Finally, the piece you don't see here in the event processing system is the event data management element, which stores all this event data for subsequent replay, root-cause analysis, and change-of-area views so users can change their scope of concern and look at events from other angles of analysis.

By applying CEP logic, BAM dashboards, and event data management, an airline can increase the intelligence in its operations and proactively detect and avoid disasters when exceptional conditions occur.

TIBCO Airs Out CEP at Air France / KLM

Big CEP use case news yesterday:  TIBCO announced a deal with Air France KLM for service oriented archiecture (SOA) and complex event processing (CEP). 

Way to go TIBCO.  This is exact the kind of continued great press the industry needs, and yet another important use case for the technology.  And, as we're discussion quite a bit on line, TIBCO did a nice job of separating the two concepts:  SOA and CEP, in their communication:

"[Air France] has chosen TIBCO's software as its enterprise backbone to integrate disparate IT assets under one open, scalable service-oriented architecture (SOA) framework. The seamless integration of AIR FRANCE KLM's legacy applications will help streamline business processes and improve efficiency. In addition, TIBCO's complex event processing software will provide real-time analysis and insight into operational performance and customer needs to help increase the airline carrier's market agility and competitive responsiveness."

Yes, Air France appears to be doing both SOA and CEP - TIBCO is not perpetuating the myth that SOA and EDA are the same thing.

Congratulations to TIBCO and the BusinessEvents team.