Posted by Apama Audio
Louis Lovas, Chief Architect of Progress Apama, discusses aspects of the Apama 4.2 release that focus on application developer productivity and how Apama enhances an organization’s ability to built event-driven applications.
Posted by Apama Audio
In this podcast Louis Lovas, Apama Architect, discusses some of the details of the enhanced parallelism that is now available in Progress Apama 4.2.
.
Posted by Apama Audio
As a follow up to the Louie Lovas blog posting on October 16th , this podcast captures a discussion between David Olson and Giles Nelson on Apama 4.2 features.
Posted by Louis Lovas
The
Apama engineering team has done it once again. True to our record of
releasing significant new features in the Apama product every 6 months,
the v4.2 release is hot off the presses with major new functionality.
The Apama roadmap is driven by a keen sense of our customer
requirements, the competitive landscape and an opportunistic zeal. The
engineering team is a dedicated R&D team driven to
excellence and quality. We are dedicated to delivering value to our
customers. A consistent comment we've heard from analysts and customers
alike is the maturity of the Apama product. Posted by Louis Lovas
My colleague Dan Hubscher and I just finished the first of a two part Webinar entitled "Build Quickly, Run Fast".
In this Webinar we explained and demonstrated Apama as
an Algo platform for high frequency and order execution algorithms. 
Posted by Louis Lovas
How
many angels can dance on the head of a pin? I suppose that was a
question debated at the Council
of Trent
that took place in Trento, Italy back in the 16th century. However, the
Event
Process Technical Society's (EPTS) annual symposium just last week took
up residence in Trento to discuss and debate a host of lofty
topics on
event processing. | Awareness | Provide commercial business and industry the necessary knowledge of event processing as a technology supported by numerous vendors with continuing research in academia. |
| Definition | Provide a concise and definitive meaning of event processing, a Taxonomy of Event Processing so to speak. This is both from the horizontal technology perspective and also a vertical focus for a handful of specific industries. It's often difficult for business people to understand technology without the context of a business or application focus. |
| Differentiation | Provide a clear distinction that defines event processing and distinguishes it from other technologies. Event processing is available is many forms, this symposium provided evidence of that. Much of it is available in specialized form as in IT Systems management. There are also pure play event processing (CEP) vendors, such as Progress/Apama. But there are also Rules engines, Business Intelligence platforms, Analytic platforms, etc. This easily presents a bewildering world filled for choice, conflicting and overlapping marketing messages. The EPTS is in the perfect position to provide that clarity behind defining what is CEP and what isn't. |
| Cooperative | Event Processing rarely operates in a vacuum. There are many synergistic technologies that closely pair with CEP. Often this can have a specific vertical business flavor, but often it's other platform technology such as BPM and temporal databases. |
Posted by Louis Lovas
In just a few short days of its announcement news of the Forrester CEP Wave has
spread to all corners of the globe. From trade magazines, online
journals and blogs to Facebook and Twitter, the headlines are
everywhere. A Google search yields thousands of hits. "Independent Research names Progress® Apama® as a Standout Leader in CEP ..."Posted by Louis Lovas
In just about any race there is usually a starting point and a finish line, unless of course you are in an arms race. For that sort of race there may have been some nebulous beginning in the
distant past, but there is no finish line. The race just keeps sprinting
along, each competitor angling for an edge, regularly recharging their ammunition supply with some new weaponry to get
ahead however slight or temporary.
I recently read an interesting article describing High Frequency Trading as embroiled in an arms race. I certainly believe it's well entrenched in such a conflict, but frankly this combat has arguably had a beneficial net effect especially in that it's contributed to the wellspring of invention, inspiring the creative spirit in all the supporting attributes that make High Frequency Trading a reality. Behind any trader (and trading firm) is an entire armada including the vendors supplying the underlying hardware, networks, software platforms and trading applications. They are all immersed in the war. As new hardware, software and/or algo's are deployed it allows the trader to do battle and speed ahead even if it's just for a short while. Competitive pressures, increasing market volatility, regulatory imperatives, risk mitigation and a host of other challenges are the land mines and roadside bombs on the long and winding road that stall and slow causing re-tooling and re-stocking the ammunition (i.e. algo strategies). There is no time to stop and catch your breath or stand on the roadside.
Sang Lee from the Aite Group reports that High Frequency Trading has had a significant impact on the overall market, providing greater liquidity, tighter spreads and overall improving the quality of the market. At the macro level these are great advancements and mark a natural evolutionary step due to so many market changes in recent years (i.e. electronic trading venues, adoption of CEP platforms for algo trading, etc.) in Equities and beyond (i.e. FX and Futures & Options). Down in the trenches, the battles rage on day by day as a multitude of traders and an untold number of algo strategies provide the market liquidity by moving in and out of positions in milliseconds (or even less time). The trading firms engaged in this never ending conflict drive a set of imperatives on software infrastructures for building and deploying algos in the High Frequency battlefield:
Rapid development and customization of algo's
Abstracting over increasingly complex strategy logic
Support for the 'ilities (availability, security, reliability, ...) to manage the mundane
Support for scalable performance, providing high throughput and low latency
Along with my colleague Dan Hubscher, I have recorded a 30 minute webinar that describes how the Apama platform along with the Apama Algorithmic Trading Accelerator meet these imperatives.
The pre-recorded webinar, is available here: Apama Algorithmic Trading Accelerator, Build Quickly, Run Fast.
Once again thanks for reading (plus watching and listening to the webinar in this case), you can also follow me at twitter, here.
Louie
Posted by Louis Lovas
As
the CEP community is well aware the technology often gets compared and
contrasted to traditional data processing practices. It's a topic that
ebbs and flows like the ocean's tides. It's a conversation that occurs
regularly in various forums, with clients, analysts and of course
within this community at large. Far be it from me to stir things
up again but I believe there is some credible evidence the critics draw
upon. This touched a nerve not too long ago when Curt Monash hammered the CEP community. The response was swift, but frankly unconvincing.
In many respects, I believe this argument is rooted in how some CEP vendor's have architected their product. Many vendors have a focus of event stream processing as a variant of database processing. They see streaming data as just a database in motion and therefore have designed and built their products myopically around that paradigm. By doing so those vendors (possibly inadvertently) have plotted a course where the fit-for-purpose of their products is focused on use-cases that are data-intake dominate. They can consume the fire-hose of data, filter, aggregate and enrich it temporally but little else. What is missing in this equation is the definition of the semantics of the application. Whether that is a custom application such as algo trading or monitoring telco subscribers or some business intelligence (BI) dashboard. To those vendors, that is viewed as adjunct or external (i.e. the client) and solved with the typical complement of technologies; java, C++, .NET and considered outside of the direct domain of CEP.
While this paradigm clearly does work, it incites the (CEP) critics; "where's the distinguishing characteristics? why can't I just do this with traditional data processing technologies?".
A challenging question when so many CEP products are stuck with that look and feel of a database, even a few of the academic projects I've reviewed seem to be myopically centered on this same paradigm. It reminds of that television commercial with the tag line: "Stay Within the Lines. The Lines Are Our Friends." (I think it was for an SUV). Quite frankly such thinking does not represent the real world. Time to think outside the box (or table as the case may be).
Yes, the real world is full of in motion entities, most often interacting with each other in some way. Cars and trucks careening down the freeway zigzag from one lane to another at high speed with the objective of reaching a destination in the shortest possible time without a collision. Would be an interesting CEP application to track and monitor the location and movement of such things.
In fact, location tracking is beginning to show signs of being a common use-case with the Apama platform. Not long ago we announced a new customer, Royal Dirkzwager that uses Apama to track ship movements in sea lanes. My colleagues Matt Rothera and David Olson recently published a webinar on maritime logistics. This webinar follows the same basic premise as the Royal Dirkzwager use-case, that of tracking the location of ships at sea. In fact, we aren't the only one seeing activity in location tracking, here's a similar use-case for CEP in location-based defense intelligence. The basic idea is the ability to track the movement of some entity, typically in relation to other entities, are they getting closer together (i.e. collision detection) or moving further apart (i.e. collision avoidance), are they moving at all? at what speed? will they reach a destination at an appropriate time? A CEP system needs, at it's core the ability to have both temporal and geospatial concepts to easily support this paradigm. Here's a handful of examples where this applies:
These are just a handful of location tracking use-cases for which the Apama platform is well suited.
Another colleague, Mark Scannell has written a location tracking tutorial that is part of the Apama Studio product. This is a great example that exemplifies the power of the Apama EPL for building location tracking CEP applications. The tutorial provides a narrative description explaining it's purpose and the implementation. Below I've included a small snippet of that example to highlight the elegant simplicity, yet powerful efficiency of the Apama EPL. If you're in need of a brief introduction to the Apama EPL, you can find that here, the beginning of a three part series on the language.
| // Track me - the tracked entity action trackMe() { // Call self again when new location is detected on LocationUpdate( id = me.id ):me { trackMe(); } // Detect a neighbor in our local area - // do this by excluding events that are for our id, // which will also cause us to reset this listener // through calling trackMe() again. LocationUpdate n; on all LocationUpdate( x in [ me.x - 1.0 : me.x + 1.0 ], y in [ me.y - 1.0 : me.y + 1.0 ] ):n and not LocationUpdate( id = me.id ) { // Increment number of neighbors that have been spotted // and update the alarm spotted := spotted + 1; updateAlarm(); // Decrement count of spotted neighbors after one second on wait ( 1.0 ) { spotted := spotted - 1; updateAlarm(); } } } |
This small example demonstrates a few of characteristics of
the Apama EPL, specifically that it is an integrated well-typed
procedural language with
event expressions. It allows you to code complex event conditions
elegantly
expressed in the middle of your procedural program. This allows you to
focus on the logic of your application instead of just selecting
the
right event condition.
Louie
Posted by Louis Lovas
This
is my final installment in a series devoted to a specific
example in the Apama EPL. I began this example by describing the basic design pattern of
a consumer/producer. Further enhancements enabled multiple
consumers and as a result the instance
idiom. Finally below, I will again enhance
this consumer/producer by illustrating how one can leverage multi-core
processors for massive scalability and parallelism.
As I have mentioned before, instances or 'sub-monitors' as they're often referred to in the Apama EPL define a discrete unit of work. That unit of work represents a set of business logic however large (a complete application scenario) or small (a simple analytic). Instances are created on demand using the spawn operator in the language. Each scenario instance is invoked with a unique set of input parameters that represent that occurrence. Each instance can then uniquely maintain its own reference data, timers and event streams, in effect its own state. In general programming patterns this is known as a factory behavioral model but we've extended it to include an execution model.
To provide a means to leverage multi-core processors, the Apama EPL provides a syntax and a simple semantic to allow those instances to execute in parallel. We do this with a language feature called contexts. These are silos of execution which take the factory model to the next level. A context defines a logical container that holds and executes instances of a scenario (of the same or differing types). The EPL provides a semantic for inter-context communication, there is no need for mutexes, semaphores or other locking schemes thus avoiding common deadlock code patterns typical of imperative languages such as java. Each context in effect has it's own logical input queue to which events are streamed from external sources or other contexts. Behind contexts our CEP engine squeezes the most out of operating system threads to leverage maximum use of multi-core processors.
The same CEP engine can create multiple contexts (a context pool as you'll soon see in the code example below), they can be used to hold and execute multiple scenario instances, additionally those instances can create sub-contexts for additional parallelism. If for example, these instances are an application for pricing Options and require a compute-intensive calculation such as Black Scholes, additional contexts can be spawned for these calculations. Furthermore, sub-contexts can be designed as shared compute services to be leveraged by multiple scenario instances running in different (parallel) contexts.
Contexts take the factory model and extend it to include a parallel execution model with a few simple keywords in the EPL as you'll soon see below.
The enhancements to the Item consumer/producer include a Context Pool which I've listed the code for below and the enhanced Item Producer that leverages it. The interface is unchanged except for one new event and the Consumer (client) has a minor revision (thus adhering to my belief that an EPL should follow the principles of structured programming of modularity and encapsulation that I've blogged on at the start of this series). The complete example for this revision is available here and requires Apama version 4.1 (or later of course).
package com.apamax.sample; event ContextPool { integer numContexts; sequence<context> contexts; integer idx; action create(integer nc, string name) { self.numContexts := nc; while(nc > 0) { contexts.append(context(name, false)); nc := nc - 1; } } action getContext() returns context { context c:= contexts[idx]; idx := idx + 1; if(idx=numContexts) then { idx := 0; } return c; } } |
The ContextPool
as implemented here is a general-purpose utility that provides a pool
of contexts via a create
method (i.e. action) and a means to distribute a workload across them
in a simple round-robining technique each time
the getContext
action is called.
| package com.apamax.sample; monitor ItemService { event ClearUserID { integer id; } integer count := 0; float price := 0.0; action onload { ContextPool cf:=new ContextPool; cf.create(4, "ClientService"); // list of subscriber (user) identifiers sequence<integer> ids := new sequence<integer>; SubscribeToItems s; on all SubscribeToItems():s { if ids.indexOf(s.subscriberId)= -1 then { context c:= cf.getContext(); ids.append(s.subscriberId); route SubscriptionResponse(s.subscriberId, c); on completed SubscriptionResponse() { spawn startSubscriptions(s.subscriberId, s.item_name, context.current()) to c; } } } ClearUserID c; on all ClearUserID():c { log "in " + c.toString(); integer index := ids.indexOf(c.id); if index != -1 then { ids.remove(index); } } } action startSubscriptions(integer this_subscriberId, string name, context mainContext) { log "in startSubscriptions"; on all wait(0.1) and not UnsubscribeFromItems(subscriberId = this_subscriberId) { route Item(this_subscriberId, name, count, price); count := count + 1; price := price + 0.1; } on UnsubscribeFromItems(subscriberId = this_subscriberId){ enqueue ClearUserID(this_subscriberId) to mainContext; } } } |
To get a general sense of what the multi-instance Item Producer code is intended to do, I
suggest a quick scan of my last installment,
this revision does not change that basic foundation it only
parallelizes it. It is worth pointing out how little the code and
design has changed yet this implementation has the ability to scale
massively to tens of thousands of instances across multiple
processor cores. Clearly this is just a simple example
that does very little real
work (producing Item events). However structurally, it's a model that
represents how one would design such a scalable service in the Apama
EPL.
Louie
