« January 2008 | Main | March 2008 »

February 2008

Friday, February 29, 2008

Taking CEP to The Next Step

Posted by Chris Martins

Standards are an area of growing interest in the complex event processing market. What will be the role of standards.  In what aspects of CEP will they emerge? When? 

With the premise that “in the future, it will be necessary for heterogeneous CEP products to interoperate across organizational boundaries” the Object Management Group is hosting a Technical Meeting in Washington DC to address some of the issues. As part of the agenda, fellow Apama colleague and blog contributor, Matt Rothera, will be speaking on How CEP Takes the Next Step. So if you are going to be in the DC area on March 13, and interested in this topic, you definitely should check it out.

Wednesday, February 20, 2008

Apama Monitorscript by Example

Posted by Louis Lovas

There have been a couple recent posts on Apama's Monitorscript language, both here and here. To provide a bit more insight into the Apama EPL, below is a working sample that demonstrates a number of its capabilities. The language includes a declarative nature for defining and registering listeners for specific event types and it has a java-like syntax for imperative logic.  The language provides a balance between a recognizable vernacular and a purposed nature for event processing.

 

Example narrative

Prior to an annotated walk-thru of the code sample, I thought it would help to first explain its purpose and what event streams it's processing. This simple example defines a work dispatcher. It receives a request in the form of an event (AddSymbol) to dispatch a discrete listener against an event stream of market depth (bids and asks) events. This discrete listener processes the market depth for a specific symbol.  The actual work performed as it pertains to this example is inconsequential and is represented by an empty method (processDepth). Additionally, once a listener is dispatched it also listens for a request to terminate.

 

The subtleness of this example is its ability to leverage the simplicity of the Apama EPL and the power of the runtime engine wherein it executes. Thousands or even tens of thousands of listeners can be dispatched each running in its own independent context processing its unique slice of the streaming market data.

 

In reality there are a number of techniques that can be employed within the MonitorScript EPL to accomplish this sort of work dispatcher. The EPL includes a spawn operator which I've outlined in a previous blog. The spawn operator is the primary means for establishing independent worker threads and is the basis for instance creation. The example below focuses on event listeners to define discrete units of work.

 

1 package com.apamax.sample;

 

2 monitor ProcessMarket {

3  sequence <string> symbols; // contains list of symbols to process.

4   com.apama.marketdata.SubscribeDepth subDepth;

5   com.apama.marketdata.Depth adepth;

6  dictionary< string, string > emptyDict;

 

7  action onload {

 

8    // Listen for incoming AddSymbol events and

9    // add to symbols list if not already present

10    AddSymbol addSymbol;

11    on all AddSymbol(): addSymbol {

12     if symbols.indexOf(addSymbol.symbol) = -1 then {

13       string local_symbol := addSymbol.symbol;

14       symbols.append(local_symbol);


15        // Subscribe to this symbol

16       route com.apama.marketdata.SubscribeDepth("", "", local_symbol, emptyDict );

 

17       // wait for 20.0 seconds, if no depth event received, terminate

18       listener waitListener;

19       waitListener := on wait(20.0) {

20          route RemoveSymbol(local_symbol);

21       }

 

22       listener depthListener;

23       depthListener := on all com.apama.marketdata.Depth(symbol=local_symbol):adepth {

24           waitListener.quit();

25           processDepth(adepth);

26       }

 

27       // Listen for RemoveSymbol events and remove from symbols list,

28       // unsubscribe & quit

29       RemoveSymbol removeSymbol;

30       on RemoveSymbol(symbol=local_symbol): removeSymbol {

31           integer index := symbols.indexOf(removeSymbol.symbol);

32           if index != -1 then {

33              symbols.remove(index);

34             processRemove(removeSymbol.symbol);

35

36             // Unsubscribe to this symbol

37             route com.apama.marketdata.UnsubscribeDepth("", "",

                                                            removeSymbol.symbol,

                                                           emptyDict );

38             depthListener.quit();

39             }

40       }

41     }

42     else {

43       log "Debug: Ignored (existing) Add Symbol Event = " + addSymbol.symbol;

44     }

45    }

46 }

 

47    action processDepth(com.apama.marketdata.Depth d) {

48       // Do something

49    }

 

50    action processRemove(string s) {

51       // Do something.

52    }

53 }

 

 

Example Annotation

 

In describing this example, the first point to note is that the event definitions are not included. For the sake of brevity they're assumed to be defined elsewhere. Actually there are only a few anyway. They can be categorized into two logical groups; control events (AddSymbol, RemoveSymbol, SubscribeDepth, UnsubscribeDepth) and data events (Depth).  This categorization is only for a semantic understanding of the example, there is no such classification in the language. Additionally, Monitorscript has an easily recognizable syntax to anyone schooled in Java, C++ and other classic languages.

 

A monitor (line 2) defines the encapsulating block definition. Similar to a java class it is typically scoped to a package name space (line 1). Monitors are the main block scope and a typical Apama application is made up of many monitors that interact with one another by sending and receiving events. Within a monitor one can declare events, define actions (i.e. methods) and variables (integers, strings, floats, etc.). This example defines a handful of monitor-scoped variables. The language also supports a number of complex data types; the sequence and dictionary both use a C++ template style declaration to define array types and collection types respectively (lines 3 and 6).

 

The onload action (line 7) is the main entry point of a monitor. When a monitor is loaded into the runtime engine, it's onload action is immediately invoked. This work dispatcher example is entirely implemented within this action, for the sake of brevity it's a simple way to describe the language. Line 10 defines an instance of an AddSymbol event and declares a listener for all occurrences of this event type (line 11).  The remainder of the functionality of this example is scoped to the encapsulating block of this listener (lines 12 – 45). This is an important note, since the intent is to receive and process multiple AddSymbol events (potentially 1,000's) where each AddSymbol will cause the invocation (dispatch) of a discrete unit of work that is represented by this encapsulating block of code. Within this block of code we communicate with other monitors and establish a number of unique listeners for this unique symbol name.

 

The route statement (line 16) sends a SubscribeDepth event. Route is the standardized form of communication between monitors. Under the covers, the route statement causes the event to be routed to be placed at the head of the engine's input queue – thus become the next event to the processed by the engine.  Semantically, routing a SubscribeDepth event starts the flow of Depth events for this symbol (i.e. local_symbol). Lines 22-26 establish a listener to receive the stream of Depth events for this symbol, calling the action processDepth upon receipt of each one.

 

In addition to establishing a Depth listener, this block of code also creates a wait timer in lines 17-21. The purpose of this timer is to terminate this dispatched unit of work for this unique symbol if we do not receive an initial Depth event within 20 seconds. Line 24 kills that wait listener once the Depth events start flowing. Termination is handled by the RemoveSymbol listener declared at line 30. Note that since it will be executing within the context of a specific symbol's unit of work we're only interested in receiving a single occurrence of RemoveSymbol. This is specified in the on statement – sans the all modifier. Upon receipt of a RemoveSymbol event we unsubscribe, remove the symbol's entry from the list and terminate (i.e. quit) the Depth listener for this symbol. Like AddSymbol, RemoveSymbol control events can arrive from another monitor or a client connected to the runtime engine.

 

I hope this simple example sheds light on the simplicity, elegance and power of the Apama Montorscript EPL.

 

Post Script …

 

After posting this blog, one of my esteemed colleagues with a much better command of the Monitorscript language offered a few refinements to avoid the need to manually handle termination (i.e. lines 17 – 21 in the code snippet). It does add one new control event - Terminate, but it avoids the need to use listener variables.

 

 

on com.apama.marketdata.Depth(symbol=local_symbol):adepth and not wait (20.0) {

   on all com.apama.marketdata.Depth(symbol=local_symbol):adepth and not Terminate(local_symbol) {

   processDepth(adepth);

  }

}

 

...

 

 

on RemoveSymbol(symbol=local_symbol):removeSymbol {

     ...

  route Terminate(removeSymbol.symbol);

}

 

 

 

This enhancement shows the declaration of complex (or compound) listeners against multiple event streams (and a timeout condition) concurrently. This is a commonly used technique in MonitorScript – and clearly quite powerful.

 

Friday, February 15, 2008

CEP: What about the action?

Posted by Matt Rothera

In recent posts, much has been made of various language and technology choices for CEP (SQL versus an Imperative EPL, such as Progress Apama’s Monitor Script). As also was pointed out, the specific constructs of the CEP language have benefits for certain use cases. For example, the SQL variants of CEP focus on real-time event filtering . Use cases such as these put more emphasis on the “event streaming and filtering” piece of the CEP, but less on the “act” portion. There is typically little or no ability to perform complex actions in response to the events, outside of building these pieces of CEP applications using conventional languages.


So what are the drawbacks of the SQL based approach? Imperative based EPL provides additional value beyond the initial event detection, and provide the ability to solve more complex problems. For example, for certain applications, the actions themselves may incur the need to spawn new rules on the event streams. There may be a need to call out to an external system, loop through the results, and conditionally spawn new rules, each having additional cascading actions which require similar treatment. The actions themselves become new instances of event detection, and must operate in the context of the original initiating event.


Is a filtering-centric CEP approach potentially a “dead-end road”? The great promise of an event-driven enterprise is the ability to begin to automate business processes in response to potentially complex events. For this to become a reality, the CEP engine must be able to not only detect, filter, and analyze a high volume of events, but also be able to orchestrate complex actions within milliseconds from initial detection. Some of those actions may themselves involve complex correlation, as responses are coordinated between multiple systems to achieve the end result.


With an imperative EPL, not only can you perform the front-end event pattern detection, but you can also create arbitrarily complex actions that run in real-time. A CEP platform must strive to not only make the event detection performant, reliable, and easy from a development point of view, but also provide facilities to build the corresponding real-time actions (which could be arbitrarily complex). As was mentioned in previous posts, lower level facilities such as looping and conditional constructs are important, but also higher level facilities and constructs would be important. For example, the ability to model any arbitrary FSM (Finite State Machine) to handle the complex action portion of CEP is important, especially when higher level, visual modeling tools are available to help developers build these types of applications.


A related blog post entitled “CEP as the EDA Application Server”, the author refers to a CEP engine as a potential choice for an implementing an event-driven application. If this be the case, the CEP engine must have the characteristics described above. In the event where certain tasks are offloaded to other components such as a BPEL engine within an ESB environment, the CEP engine still must be able to correlate one or more responses in the context of the originating event, and conditionally launch additional actions based on the results. It seems an imperative EPL would be an absolute requirement to meet the needs of this use case.

At Progress Apama, we strive to create the complete complex event processing platform, which involves providing facilities to not only detect patterns in one or more high volume event streams, but also to respond with complex actions in milliseconds. One classic event driven application that demonstrates this is Algorithmic Trading, where the complex action is orchestrating a trade. However, other types of event driven applications are clearly emerging, such as:

  1. Operations Center Monitoring and Control: Monitor a vast gas pipeline, energy grid, or water treatment plant and provide reactive control to events in real-time.

  2. Communications Enabled Business Process: Based on complex events within the enterprise, setup voice or chat sessions between key parties to ensure that the issue is resolved quickly.

  3. Battlefield Operations: Monitor a vast number of objects (tanks, personnel, equipment), and take directed actions and orders within milliseconds based on the exact nature of the battlefield conditions.

These are just some of the examples. There are many others.

 

The point in calling them out is to suggest that we must not become so transfixed by the ability to tap into event streams that we presume that a real event-driven application ends there. Arguably more important – is what you can do with the information.

Wednesday, February 13, 2008

Thoughts on the Bitter Pill

Posted by Louis Lovas

In reading a recent CEP blog, a Bitter Pill To Swallow by Tim Bass and Opher's commentary it reminded me of a few thoughts I've had recently on the role of CEP in the modern trading platform for Capital Markets.

First, let me say that I believe Apama takes a much larger view and includes much more in the CEP stack than many other vendors. Naturally, we’ve turned this into a competitive advantage. This larger perspective encompasses many aspects beyond the core CEP engine. That includes dashboards, connectivity adapters, development and deployment tooling, vertical-focused application starter kits and of course language. That said, I still believe the CEP industry is still quite nascent and only loosely defined. However I think it is gaining clarity – from both an industry view and customer/prospect expectations. What’s the role of CEP in financial services and where does it fit in total software stack of fully deployed trading platforms? A few recent experiences have drawn me to the conclusion that CEP technology is just one of many elements in that infrastructure.

  • AITE Report on High Performance Infrastructure

The AITE report is a review of the infrastructure components that make up a trading platform, this included various sorts of connectivity (market data, order management, messaging), distributed cache and CEP. The report did not articulate the actual trading applications (i.e. strategies) themselves; those are unique to any one customer. The point being, those trading apps are viewed as independent and not defined as part of the trading infrastructure (a rather obvious point frankly). The technology used for strategy implementation is left as “an exercise for the user”.  My interpretation of this… CEP is part of the infrastructure and not necessarily the application. Without question, I clearly understand that the division between infrastructure and application can be quite gray, and with CEP lacking an absolute definition it's not all that surprising. Within Apama we are driven towards providing a total platform for our customers. As such, we undertake the task of providing both the infrastructure and a framework for applications. This framework takes the form of ready-to-use functions, components and starter kits or accelerators as we call them. Ideally, customers only need to inject their own IP. Our platform lends itself it that end – a one-stop-shop if you will. Our future plans are focused on this breadth in the platform because that's the sort of demand we face. While I don't think this necessarily broadens the definition of CEP, it simply means CEP as a technology cannot exist in a vacuum. It needs the standard complement of tooling both within itself and for outside integration to the IT-centric world at large. This is the stuff one comes to expect as maturity overtakes hype.

  • Standardization on EP benchmarking

There have been a few initiatives to define vendor-neutral benchmarks for CEP. Additionally, a few vendors have published benchmark studies, BEA in particular. Efforts to define independent benchmarks will inevitably do more that just define test cases for CEP, they will also shape the definition of what CEP is and does. I would expect any effort to benchmark CEP engines will be somewhat narrowly focused to just the core capability of doing something to or with streaming data. Many of the elements that make CEP truly commercially viable, from tools for development and visualization to deployment, runtime management and high availability will not be part of the equation. Practically speaking it stands to reason this is the case. It's impossible to create a neutral benchmark for these things. However, it would be ideal for benchmarks to include real-world usage. Some finance examples would be a Pricing engine, Index Arbitrage, Spread trading, Crossing and VWAP trading. I would expect not all CEP products to be fit-for-purpose to meet the needs of these examples. Furthermore, I'm not convinced one could say these were all CEP use cases either.

  • Discussions with prospects

Over the course of the past year I've met with numerous clients. There are a couple of observations I've made with respect to the growing understanding of CEP technology. First I've noticed an increasing number of them use the phrase "researching CEP platforms" in conversation on their next-generation trading platform. While this might appear innocuous, it represents a shift in thinking. CEP is gaining wider awareness and its general applicability to trading. It was not so long ago that the term CEP was not mentioned either by us or the client – it had no apparent meaning to the opportunity at hand. However, times are changing and CEP awareness is growing, unfortunately this does not necessarily mean there is complete clarity on its definition. This lack of definition, I believe has caused (future) adopters of CEP to look at the various vendors' products through different lenses. For example if a particular CEP product is not fit-for-purpose for implementing a Pricing Engine that does not necessarily mean it cannot provide a component of a Pricing Engine, with the remainder (or a significant portion) being implemented in more a traditional manner. In a competitive bid environment, I am convinced clients draw the line-in-the-sand at a different place for different vendors based on their fit-for-purpose. Unfortunately, this causes CEP products to be judged on unequal footing.

Another validation that shows CEP is gaining awareness but still immature is a recent report from the Forrester group on CEP Adoption. To paraphrase, CEP adoption is being driven by and used within the business side of organizations sans IT. Business tends to be on the bleeding edge of technology simply because they're looking for any nugget that can give them a competitive edge. IT on the other hand, has to face the challenging realities of daily care and feeding of software. Their priorities are in stability and manageability - attributes of mature technologies.

As the CEP hype curve starts to level off, it will follow the same path as all other hyped technology to eventual commoditization. Its usage will coalesce around a few paradigms and an industry definition will start to solidify. The items I mention above indicate that this is happening already.
 

Monday, February 11, 2008

Apama CEP Code Snippet

Posted by Chris Martins

We've posted examples of coding in alternative CEP languages in the past to illustrate how concise or verbose those approaches might be in expressing an event processing function.  An example of Apama's language has made an appearance in a blog posting by Lab49.  In the example, the code defines very crisply an operation in which the system responds to incoming price events, ensuring that you skip intermediate events and always process the latest event.