« April 2009 | Main | June 2009 »

May 2009

Wednesday, May 27, 2009

Location Tracking, fit-for-purpose design patterns in CEP

Posted by Louis Lovas

As the CEP community is well aware the technology often gets compared and contrasted to traditional data processing practices. It's a topic that ebbs and flows like the ocean's tides. It's a conversation that occurs regularly in various forums, with clients, analysts and of course within this community at large. Far be it from me to stir things up again but I believe there is some credible evidence the critics draw upon. This touched a nerve not too long ago when Curt Monash hammered the CEP community.  The response was swift, but frankly unconvincing.

In many respects, I believe this argument is rooted in how some CEP vendor's have architected their product. Many vendors have a focus of event stream processing as a variant of database processing. They see streaming data as just a database in motion and therefore have designed and built their products myopically around that paradigm. By doing so those vendors (possibly inadvertently) have plotted a course where the fit-for-purpose of their products is focused on use-cases that are data-intake dominate. They can consume the fire-hose of data, filter, aggregate and enrich it temporally but little else. What is missing in this equation is the definition of the semantics of the application. Whether that is a custom application such as algo trading or monitoring telco subscribers or some business intelligence (BI) dashboard. To those vendors, that is viewed as adjunct or external (i.e. the client) and solved with the typical complement of technologies; java, C++, .NET and considered outside of the direct domain of CEP. 

While this paradigm clearly does work, it incites the (CEP) critics; "where's the distinguishing characteristics? why can't I just do this with traditional data processing technologies?".

A challenging question when so many CEP products are stuck with that look and feel of a database, even a few of the academic projects I've reviewed seem to be myopically centered on this same paradigm. It reminds of that television commercial with the tag line:  "Stay Within the Lines. The Lines Are Our Friends." (I think it was for an SUV). Quite frankly such thinking does not represent the real world. Time to think outside the box (or table as the case may be).

Yes, the real world is full of in motion entities, most often interacting with each other in some way. Cars and trucks careening down the freeway zigzag from one lane to another at high speed with the objective of reaching a destination in the shortest possible time without a collision.  Would be an interesting CEP application to track and monitor the location and movement of such things.  

In fact, location tracking is beginning to show signs of being a common use-case with the Apama platform. Not long ago we announced a new customer, Royal Dirkzwager that uses Apama to track ship movements in sea lanes. My colleagues Matt Rothera and David Olson recently published a webinar on maritime logistics. This webinar follows the same basic premise as the Royal Dirkzwager use-case, that of  tracking the location of ships at sea.  In fact, we aren't the only one seeing activity in location tracking, here's a similar use-case for CEP in location-based defense intelligence.  The basic idea is the ability to track the movement of some entity, typically in relation to other entities, are they getting closer together (i.e. collision detection) or moving further apart (i.e. collision avoidance), are they moving at all? at what speed? will they reach a destination at an appropriate time? A CEP system needs, at it's core the ability to have both temporal and geospatial concepts to easily support this paradigm.  Here's a handful of examples where this applies:

  • Tracking ship movements at sea (as I mentioned with Royal Dirkzwager, and the Apama webinar on maritime logistics)
  • Airplanes moving into and out of an airspace
  • Baggage movement in an airport
  • Delivery trucks en route to destinations
  • Service-enabled mobile phones delivering content as people move through shopping and urban areas
  • Men, machines and material moving on the battlefield

These are just a handful of location tracking use-cases for which the Apama platform is well suited.

Another colleague, Mark Scannell has written a location tracking tutorial that is part of the Apama Studio product. This is a great example that exemplifies the power of the Apama EPL for building location tracking CEP applications. The tutorial provides a narrative description explaining it's purpose and the implementation. Below I've included a small snippet of that example to highlight the elegant simplicity, yet powerful  efficiency of the Apama EPL. If you're in need of a brief introduction to the Apama EPL, you can find that here, the beginning of a three part series on the language.

Location Tracking in the Apama EPL
// Track me - the tracked entity
action trackMe() {
  // Call self again when new location is detected
  on LocationUpdate( id = me.id ):me {

  // Detect a neighbor in our local area -
  // do this by excluding events that are for our id,
  // which will also cause us to reset this listener
  // through calling trackMe() again.
  LocationUpdate n;
  on all LocationUpdate( x in [ me.x - 1.0 : me.x + 1.0 ],
                         y in [ me.y - 1.0 : me.y + 1.0 ] ):n and
     not LocationUpdate( id = me.id ) {

     // Increment number of neighbors that have been spotted
     // and update the alarm
     spotted := spotted + 1;

     // Decrement count of spotted neighbors after one second
     on wait ( 1.0 ) {
       spotted := spotted - 1;

As a brief introduction, the Location Tracker tutorial is designed to track the movement of Entities (i.e. cars, ships, trucks, planes, or any of those things I listed above) in relation to other Entities within a coordinate system or grid. An entity is considered a neighbor if it is within 2 grid units (-1,+1) of any other entity. The grid and the units within the grid are largely irrelevantly for the syntactic definition of entity tracking. Their semantic meaning on the other hand, is within the context of a specific use-case (i.e. a shipping harbor, air space, battlefield, etc.).

From the tutorial I pulled a single action, trackMe, it contains the heart and soul of the tracking logic.  As entities move they produce LocationUpdate events. The event contains the entities unique id and the X,Y coordinate of the new location. This trackMe action is designed to track their movement by monitoring LocationUpdate events. For each unique entity there is a spawned monitor instance (a running micro-thread so to speak) of this trackMe action. 

The idea is that when an entity moves its new location is instantly compared against all other tracked entities (except of course itself, witness the recursive call to trackMe when id's match (id = me.id)) to determine if it has become a neighbor (remember the 2 grid units).  This is elegantly implemented with the listener "on all LocationUpdate( x in [me.x - 1.0 : me.x + 1.0], ...". In a narrative sense, this can be read as "If the X,Y coordinate of this entities new location is within 2 grid units (-1.0, + 1.0)  of me then identify it as a neighbor and update an alarm condition"  (via  a call to updateAlarm()).

This small bit of code (about 20 lines) exhibits an immensely powerful geospatial concept, the ability to track the movement of 100's, 1000's even 10,000's of entities against each other as they move, and of course this is accomplished with millisecond latency.


This small example demonstrates a few of characteristics of the Apama EPL, specifically that it is an integrated well-typed procedural language with event expressions. It allows you to code complex event conditions elegantly expressed in the middle of your procedural program. This allows you to focus on the logic of your application instead of just selecting the right event condition.

However to get a clear picture, the language of CEP is just one aspect of an overall platform. The Apama strategy has also been focused on a whole product principle, one where the whole is greater than the sum of the parts. As a mandate to our vision we have outlined four key defining characteristics: 1) A scalable, high performing Event Engine. 2) Tools for rapid creation of event processing applications supporting business and IT users. 3) Visualization technologies for rich interactive dashboards and 4) An Integration fabric for the rapid construction of high performance, robust adapters to bridge into the external world.

The possibility exists that CEP will struggle to break out as a truly unique technology platform when so many just see a variant of database technology.  It's time to break out of the box, drive over the lines and succinctly answer the critics questions. CEP is not about tables, rows and columns but events. Events that are often artifacts of the real world. A world that is in constant motion, be it ships, planes, car, trucks, phones, or you and me. Information flows from it all in many forms but that does mean we have squeeze it into the database paradigm.

Once again thanks for reading, 

Friday, May 22, 2009

Real-Time Risk – A Mandate for Trading in Volatile Markets

Posted by Dan Hubscher

Risk management in trading is now more important than ever, but we can't let that strangle the front office.  Though managing risk is about preventing loss, it is also about making money.  It is about knowing, across traders, desks, and asset classes, what the entire firm exposure is, at any point in time... and being able to do something about it, automatically, in real time.  With the confidence to sense and respond immediately, it is about allowing trading to go fast.

Apama is hosting a Webinar on Real-Time Risk Management.  You can see the details and register for free by clicking here.  It will take place on Wednesday, May 27th, at 10am EDT / 3pm BST, and will feature speakers from Buy Side Technology, the Tabb Group, Thomas Weisel Partners - and our very own Dr. John Bates, Founder and General Manager of Apama

Topics include:

  • How CEP can enable risk management to keep pace with high-frequency trading
  • How trading operations are optimized when they can offer greater visibility to risk
  • How real-time risk with CEP increases trading volumes, commissions and profits

Come join us on-line, you will be able to participate in a Q&A session at the end.  If you miss it, you can view the replay later on the same site.Race Car

Remember the old saying in racing: Never drive faster than your guardian angel can fly.  Do you want to slow down to the speed of your angel?  You want your angel to speed up.  You want to win the race, and take the winning prize.  That's how to win in trading.

Sunday, May 10, 2009

CEP deployed in 3 Italia, wireless telco

Posted by Giles Nelson

3 Italia is a third-generation mobile telco in Italy, a division of Hutchison Whampoa who hold a number of 3G licenses worldwide. 3 is in the process of deploying Apama to achieve better real-time analysis of network and business systems.


The first area where 3 Italia has deployed Apama is in the monitoring of core network elements and business support systems, such as customer relationship and billing systems. By correlating events from both network and business systems, 3 can better understand issues that are occurring which might affect customer service or 3’s ability to charge correctly for services used.


So why do they need Complex Event Processing to do this?


Firstly, telcos are highly sophisticated technical environments. IT is, of course, entirely at the centre of how a telco operates. Even 3, a “modern”, wireless-only telco with none of the complexity demanded of legacy, wired environments, has many different operational and business systems. Many of these are monitored on a continual basis to ensure that the network and services are running correctly. However the operational monitoring solutions used to do this work in a restrictive way. Typically they only monitor a specific network or business system (billing or CRM for example). Some offerings are able to monitor multiple systems but the analysis and the presentation of information from these systems is not correlated. So, a holistic view of what is happening as network and business services execute is simply not possible.


Secondly, there’s the issue of scale. The amount of traffic in a telco is enormous. A typical telco with a few million subscribers will generate billions of events per day. Yes, some of these events can be intepreted, analysed and aggregated, but only at low level within a telco’s architecture where limited value is gained. To be able to analyse this information in a flexible, general-purpose way in anything approaching real-time is impossible using conventional technologies such as databases or business intelligence systems.


So this is where CEP comes in at 3. It is only by being able to bring information from multiple different parts of the organisation together and to correlate and analyse this in real-time that 3 can obtain a more complete, customer-centric view. 3 has built a large number of end user dashboards, built in Apama, to monitor various aspects of their billing and service delivery. In total, several hundred key performance indicators are being tracked which allow deviations from norms to be identified and predictions made when service level agreements would be exceeded. It is the operational business users which monitor these dashboards to give them the personalised, real-time view of 3’s business that they need. Apama provides 3 with the flexibility to introduce new monitoring scenarios and enhance existing ones straightforwardly, so ensuring that the operations department stays on top of, and influences, changes and enhancements to the services that 3 offers.


Looking beyond the monitoring of the operational business systems, CEP can provide other benefits. Telcos operate within a very competitive, innovative business environment which is rapidly changing because of trends such as ubiquitous wireless access, smartphones and Web 2.0. Telcos do not want to be simple utility providers – giving network access and nothing more. Delivering value-add services and improving customer loyalty are key aims. By understanding how customers are using network services now provides benefits. Customers may change the way they use services over time. Consider a user who may start sending significant numbers of international SMS messages. By being able to detect this trend and be aware that they might benefit from discounts from a different tariff means that a telco can, by being real-time aware of this usage, interact with the user once behaviour patterns are detected to offer that user a better, more personalised tariff – perhaps to offer them a bundle of extra international SMS messaging for a small additional fee. By responding to their usage, not at the end of the month, or in some other arbitrary time period, but now, the telco can be seen to be responsive to their needs. The customer can receive immediately a benefit and so, customer loyalty is enhanced. This understanding of customers’ usage patterns is invaluable for a telco’s marketing department and makes them far more customer focussed.

3 provides an excellent example of how a telco can deploy CEP to give them insights that they have never previously had. Not only are operations in 3 being optimised, but new models of real-time interaction with customers are in sight. CEP promises to be an invaluable aid in the telcos’ fight to deliver higher-value, personalised services and to be better placed to retain customers.


Our press release on 3 can be found here.