Event Processing blog

Apama Monitorscript by Example

There have been a couple recent posts on Apama's Monitorscript language, both here and here. To provide a bit more insight into the Apama EPL, below is a working sample that demonstrates a number of its capabilities. The language includes a declarative nature for defining and registering listeners for specific event types and it has a java-like syntax for imperative logic.  The language provides a balance between a recognizable vernacular and a purposed nature for event processing.

 

Example narrative

Prior to an annotated walk-thru of the code sample, I thought it would help to first explain its purpose and what event streams it's processing. This simple example defines a work dispatcher. It receives a request in the form of an event (AddSymbol) to dispatch a discrete listener against an event stream of market depth (bids and asks) events. This discrete listener processes the market depth for a specific symbol.  The actual work performed as it pertains to this example is inconsequential and is represented by an empty method (processDepth). Additionally, once a listener is dispatched it also listens for a request to terminate.

 

The subtleness of this example is its ability to leverage the simplicity of the Apama EPL and the power of the runtime engine wherein it executes. Thousands or even tens of thousands of listeners can be dispatched each running in its own independent context processing its unique slice of the streaming market data.

 

In reality there are a number of techniques that can be employed within the MonitorScript EPL to accomplish this sort of work dispatcher. The EPL includes a spawn operator which I've outlined in a previous blog. The spawn operator is the primary means for establishing independent worker threads and is the basis for instance creation. The example below focuses on event listeners to define discrete units of work.

 

1 package com.apamax.sample;

 

2 monitor ProcessMarket {

3  sequence <string> symbols; // contains list of symbols to process.

4   com.apama.marketdata.SubscribeDepth subDepth;

5   com.apama.marketdata.Depth adepth;

6  dictionary< string, string > emptyDict;

 

7  action onload {

 

8    // Listen for incoming AddSymbol events and

9    // add to symbols list if not already present

10    AddSymbol addSymbol;

11    on all AddSymbol(): addSymbol {

12     if symbols.indexOf(addSymbol.symbol) = -1 then {

13       string local_symbol := addSymbol.symbol;

14       symbols.append(local_symbol);

15        // Subscribe to this symbol

16       route com.apama.marketdata.SubscribeDepth("", "", local_symbol, emptyDict );

 

17       // wait for 20.0 seconds, if no depth event received, terminate

18       listener waitListener;

19       waitListener := on wait(20.0) {

20          route RemoveSymbol(local_symbol);

21       }

 

22       listener depthListener;

23       depthListener := on all com.apama.marketdata.Depth(symbol=local_symbol):adepth {

24           waitListener.quit();

25           processDepth(adepth);

26       }

 

27       // Listen for RemoveSymbol events and remove from symbols list,

28       // unsubscribe & quit

29       RemoveSymbol removeSymbol;

30       on RemoveSymbol(symbol=local_symbol): removeSymbol {

31           integer index := symbols.indexOf(removeSymbol.symbol);

32           if index != -1 then {

33              symbols.remove(index);

34             processRemove(removeSymbol.symbol);

35

36             // Unsubscribe to this symbol

37             route com.apama.marketdata.UnsubscribeDepth("", "",

                                                            removeSymbol.symbol,

                                                           emptyDict );

38             depthListener.quit();

39             }

40       }

41     }

42     else {

43       log "Debug: Ignored (existing) Add Symbol Event = " + addSymbol.symbol;

44     }

45    }

46 }

 

47    action processDepth(com.apama.marketdata.Depth d) {

48       // Do something

49    }

 

50    action processRemove(string s) {

51       // Do something.

52    }

53 }

 

 

Example Annotation

 

In describing this example, the first point to note is that the event definitions are not included. For the sake of brevity they're assumed to be defined elsewhere. Actually there are only a few anyway. They can be categorized into two logical groups; control events (AddSymbol, RemoveSymbol, SubscribeDepth, UnsubscribeDepth) and data events (Depth).  This categorization is only for a semantic understanding of the example, there is no such classification in the language. Additionally, Monitorscript has an easily recognizable syntax to anyone schooled in Java, C++ and other classic languages.

 

A monitor (line 2) defines the encapsulating block definition. Similar to a java class it is typically scoped to a package name space (line 1). Monitors are the main block scope and a typical Apama application is made up of many monitors that interact with one another by sending and receiving events. Within a monitor one can declare events, define actions (i.e. methods) and variables (integers, strings, floats, etc.). This example defines a handful of monitor-scoped variables. The language also supports a number of complex data types; the sequence and dictionary both use a C++ template style declaration to define array types and collection types respectively (lines 3 and 6).

 

The onload action (line 7) is the main entry point of a monitor. When a monitor is loaded into the runtime engine, it's onload action is immediately invoked. This work dispatcher example is entirely implemented within this action, for the sake of brevity it's a simple way to describe the language. Line 10 defines an instance of an AddSymbol event and declares a listener for all occurrences of this event type (line 11).  The remainder of the functionality of this example is scoped to the encapsulating block of this listener (lines 12 – 45). This is an important note, since the intent is to receive and process multiple AddSymbol events (potentially 1,000's) where each AddSymbol will cause the invocation (dispatch) of a discrete unit of work that is represented by this encapsulating block of code. Within this block of code we communicate with other monitors and establish a number of unique listeners for this unique symbol name.

 

The route statement (line 16) sends a SubscribeDepth event. Route is the standardized form of communication between monitors. Under the covers, the route statement causes the event to be routed to be placed at the head of the engine's input queue – thus become the next event to the processed by the engine.  Semantically, routing a SubscribeDepth event starts the flow of Depth events for this symbol (i.e. local_symbol). Lines 22-26 establish a listener to receive the stream of Depth events for this symbol, calling the action processDepth upon receipt of each one.

 

In addition to establishing a Depth listener, this block of code also creates a wait timer in lines 17-21. The purpose of this timer is to terminate this dispatched unit of work for this unique symbol if we do not receive an initial Depth event within 20 seconds. Line 24 kills that wait listener once the Depth events start flowing. Termination is handled by the RemoveSymbol listener declared at line 30. Note that since it will be executing within the context of a specific symbol's unit of work we're only interested in receiving a single occurrence of RemoveSymbol. This is specified in the on statement – sans the all modifier. Upon receipt of a RemoveSymbol event we unsubscribe, remove the symbol's entry from the list and terminate (i.e. quit) the Depth listener for this symbol. Like AddSymbol, RemoveSymbol control events can arrive from another monitor or a client connected to the runtime engine.

 

I hope this simple example sheds light on the simplicity, elegance and power of the Apama Montorscript EPL.

 

Post Script …

 

After posting this blog, one of my esteemed colleagues with a much better command of the Monitorscript language offered a few refinements to avoid the need to manually handle termination (i.e. lines 17 – 21 in the code snippet). It does add one new control event - Terminate, but it avoids the need to use listener variables.

 

 

on com.apama.marketdata.Depth(symbol=local_symbol):adepth and not wait (20.0) {

   on all com.apama.marketdata.Depth(symbol=local_symbol):adepth and not Terminate(local_symbol) {

   processDepth(adepth);

  }

}

 

...

 

 

on RemoveSymbol(symbol=local_symbol):removeSymbol {

     ...

  route Terminate(removeSymbol.symbol);

}

 

 

 

This enhancement shows the declaration of complex (or compound) listeners against multiple event streams (and a timeout condition) concurrently. This is a commonly used technique in MonitorScript – and clearly quite powerful.

 

Apama CEP Code Snippet

We've posted examples of coding in alternative CEP languages in the past to illustrate how concise or verbose those approaches might be in expressing an event processing function.  An example of Apama's language has made an appearance in a blog posting by Lab49.  In the example, the code defines very crisply an operation in which the system responds to incoming price events, ensuring that you skip intermediate events and always process the latest event.

Taking Aim

I am both humbled and appreciative of all the accolades, constructive comments (hey, fix that misspelled word) and yes, criticism on my latest blog about using SQL for Complex Event Processing. I was expecting some measure of response, as shown by this rebuttal given the somewhat polarizing nature of using the SQL language for CEP applications. For every viewpoint there is always an opposing, yet arguably valid outlook. I welcome any and all commentary. Again thanks all reading and commenting.

There were two main themes of the criticism that I received. One was on the viability of the SQL language, the other on my commentary on the use of Java and C++ for CEP applications. I would like to clarify and reinforce a few points that I made.

I chose the aggregation use-case as an example to highlight limitations of SQL because its one that I have recent experience with. I have been both directly and indirectly involved in six aggregation projects for our financial services customers over the past year. In those endeavors I've both learned much and leveraged much. As I tried to describe in a condensed narrative, aggregation is a challenging problem. One that is best solved by use of complex nested data structures and associated program logic. Trying to represent this in SQL is a tall order given the flat 2-dimensional nature of SQL tables and the limited ability for semantic expression. To try to further explain this, I have taken a snippet of a MonitorScript-based implementation, presenting it as a bit of pseudo code that describes the nested structure I am referring to. I've intentionally avoided any specific language syntax and I've condensed the structures to just the most relevant elements. But suffice to say, defining these structures is clearly possible in Java and C++ and Apama's MonitorScript. I would also like to give credit where it's due and acknowledge many of my colleagues for the (abridged) definition I'm using below from one of our customer implementations.

structure Provider {

string symbol; // symbol (unique to this provider)

string marketId; // the market identifier of the price point

integer quantity; // quantity the provider is offering

float timestamp; // time of the point

float cost; // transaction cost for this provider

hashmap<string,string> extraParams; // used for storing extra information on point

}

 

structure PricePoint

{

float price;// the price (either a bid or ask)

array<Provider> providers; // array of providers at this price

integer totalQty; // total quantity across all providers at this price

}

 

structure AggregatedOrderBook

{

integer sequenceId; // counter incremented each time the book is updated

integer totalBidQuantity; // total volume available on the bid side

integer totalAskQuantity; // total volume available on the ask side

integer totalProviders; // total number of providers

array<PricePoint> bids; // list of bids, sorted by price

array<PricePoint> asks; // list of asks, sorted by price

}

An aggregation engine would create an instance of AggregatedOrderBook for each symbol, tracking prices per market data Provider. As market data quotes arrive they are decomposed and inserted (sort/merged) into the appropriate PricePoint and total values are calculated. This is an oversimplification of what transpires per incoming quote, but the aim here is to provide a simplified yet representative example of the complexities in representing an Aggregated Order Book.

Furthermore, after each quote is processed and the aggregate order book is updated it's imperative that it be made available to trading strategies expeditiously. Minimizing the signal-to-trade latency is a key measure of success of algorithmic trading. Aggregation is a heavyweight, compute intensive operation. It takes a lot of processing power to aggregate 1,000 symbols across 5 Exchanges. As such, it is one (of many) opposing forces to the goal of minimizing latency. So this presents yet another critical aspect of aggregation, how best to design it so that is can deliver its content to eagerly awaiting strategies. One means of minimizing that latency is to have the aggregation component and trading strategies co-resident within the CEP runtime engine. Passing (or otherwise providing) the aggregated order book to the strategies becomes a simple 'tap-on-the-shoulder' coding construct. But it does imply the CEP language has the semantic expressiveness to design and implement both aggregation and trading strategies and then the ability to load and run them side-by-side within the CEP engine. Any other model implies not only multiple languages (i.e. java and streamSQL) but likely some sort of distributed, networked model. Separating aggregation from its consumers, the trading strategies will likely incur enough overhead that it impacts that all important signal-to-trade latency measure. I do realize that the CEP vendors using a streaming SQL variant have begun to add imperative syntax to support complex prodedural logic and "loop" constructs something I'm quite glad to see happening. It only validates the claim I've been making all along. The SQL language at its core is unsuitable for full-fledged CEP-style applications. The unfortunate side effect of these vendor-specific additions is that it will fracture attempts at standardization.

In my previous blog, I wanted to point out the challenges of the SQL language to both implement logic and manage application state. To that end, I provided a small snippet of a streamSQL variant. A criticism leveled against it states that it's an unnecessarily inefficient bit of code. I won't argue that point, and I won't take credit for writing it either. I simply borrowed it from a sample application provided with another SQL-based CEP product. The sample code a vendor includes with their product is all too often taken as gospel. A customer's expectation is that it represents best practice usage. Vendors should take great care in providing samples, portions of which inevitably end up in production code.

The second criticism I received was on a few unintentionally scathing comments I made against Java and C++. I stated that using C++ and/or Java "means you start an application's implementation at the bottom rung of the ladder". My intent was to draw an analogy to CEP with its language and surrounding infrastructure. All CEP engines provide much more than just language. They provide a runtime engine or virtual machine, connectivity components, visualization tools and management/deployment tools. CEP vendors like all infrastructure vendors live and die by the features, performance and quality of their product. All too often I've witnessed customers take a "not invented here" attitude. They may survey the (infrastructure) landscape and decide "we can do better". For a business' IT group chartered with servicing the business to think they can implement infrastructure themselves is a naïve viewpoint. Granted, on occasion requirements might be so unique that the only choice is to start slinging C++ code, but weighing the merits of commercial (and open source) infrastructure should not be overlooked.

My goal in this and past blogs is to provide concrete use-cases and opinions on CEP drawn from my own experiences with designing, building and deploying Apama CEP applications. In doing so I was quite aware that I am drawing a big red bulls-eye on my back making me an easy target for detractors to take aim. Surprisingly, I have received much more positive commentary than I ever expected and fully professional criticisms. I thank all that have taken the time to read my editorials, I am quite flattered.

Bending the Nail

In my recent blogs (When all you have is a hammer everything looks like a nail and Hitting the nail on the head) one could conclude that I've been overly inflammatory to SQL-based CEP products. I really have no intention to be seditious, just simply factual. I've been building and designing software far too long to have an emotional response to newly hyped products or platforms. Witnessing both my own handy work and many compelling technologies fade from glory all too soon has steeled me against any fervent reactions. I've always thought the software business is for the young. As with most professions one gets hardened by years of experience, some good, and some only painful lessons. Nonetheless, over time that skill, knowledge and experience condenses. The desire to share that (hard-won) wisdom is all too often futile. The incognizant young are too busy repeating the same mistakes to take notice. Funny thing is … wisdom is an affliction that inevitably strikes us all.

Well, enough of the philosophical meanderings just the facts please …

In a recent blog, I explored the need for CEP-based applications to manage state. As a representative example, I used the algo-trading example of managing Orders-in-Market. The need to process both incoming market data and take care of Orders placed is paramount to the goals of algorithmic trading. I'll delve a bit deeper into the state management requirement but this time focusing on the management of complex market data, the input if you will to the algorithmic engine. Aggregation of market data is a trend emerging across all asset classes in Capital Markets. Simply put, aggregation is the process of collecting and ordering quote data (bids & asks) from multiple sources of liquidity into a consolidated Order Book. In the end, this is a classic sort/merge problem. Incoming quotes are dissected and inserted into a cache organized by symbol, exchange and/or market maker and sorted Bid and Ask prices. Aggregation of market data is applicable to many asset classes (i.e. Equities, Foreign Exchange and Fixed Income). The providers of liquidity in any asset class share a number of common constructs but an equal number of unique oddities. For the aggregation engine, there are also common requirements (i.e. sorting/merging) and a few unique nuances. It's the role of the aggregation engine to understand each provider's subtleties and normalize them for the consuming audience. For example, different Equities Exchanges (or banks providing FX liquidity) can use slightly different symbol naming conventions. Likewise, transaction costs can (or should) have an influence on the quote prices. Many FX providers put a time-to-live (TTL) on their streaming quotes, which implies the aggregation engine has to handle price expirations (and subsequently eject them from its cache). In the event of a network (or other) disconnection, the cache must be cleansed of that provider's (now stale) prices. The aggregation engine must account for these (and a host of other needs) since its role is to provide a single unified view of an Order Book to trading applications. The trading applications can be on both sides of the equation. A typical Buy-side consumer is a Best Execution algo. Client orders or Prop desk orders are filled by sweeping the aggregate book from the top. For Market Makers, aggregation can be the basis for a Request For Quote (RFQ) system.

At first glance, one would expect that SQL-based CEP engines would be able to handle this use-case effectively. After all, sorting and merging (joining) is a common usage of SQL in the database world and streaming SQL does provide Join and Gather type operators. However, the complexities of an aggregation model quickly outstrip the use of SQL as an efficient means of implementation. The model requires managing/caching a complex multi-dimensional data structure. For each symbol, multiple arrays of a price structure are necessary, one for the bid side another for the ask side. Each element in the price structure would include total quantity available at this price and a list of providers. Each provider entry in turn, ends up being a complex structure in itself since it would include any symbol mapping, transaction costing, expiration and connectivity information. At the top level of the aggregate book would be a summation of the total volume available (per symbol of course). Algos more interested in complete order fulfillment (i.e. fill-or-kill) would want this summary view.

Using stream SQL to attempt to accomplish this would mean flattening this logical multi-dimension object into the row/column format of a SQL table. SQL tables can contain only scalar values; multidimensional-ness can only be achieved by employing multiple tables. I don't mean to imply this is undesirable or illogical. Initially it seems like a natural fit. However, an Aggregated Book is more than just it's structure, but as I mentioned above, a wealth of processing logic. In the end one would be bending the SQL language to perform unnatural acts in any attempt to implement this complex use-case.

To illustrate an unnatural act, here's a very simple streamSQL example. The purpose of this bit of code is to increment an integer counter, (TradeID = TradeID + 1) on receipt of every tick (TradesIn) event and produce a new output stream of ticks (Trades_with_ID) that now includes that integer counter - a trade identifier of sorts.

 

CREATE INPUT STREAM TradesIn (

Symbol string(5),

Volume int,

Price double

);

 

CREATE MEMORY TABLE TradeIDTable (

TradeID int,

RowPointer int,

PRIMARY KEY(RowPointer) USING btree

);

 

CREATE STREAM Trades_with_ID;

 

INSERT INTO TradeIDTable (RowPointer, TradeID)

SELECT 1 AS RowPointer, 0 AS TradeID

FROM TradesIn

ON DUPLICATE KEY UPDATE

TradeID = TradeID+1

RETURNING

TradesIn.Symbol

AS

Symbol,

TradesIn.Volume

AS

Volume,

TradesIn.Price

AS

Price,

TradeIDTable.TradeID

AS

TradeID

INTO Trades_with_ID;

 

The state to manage and the processing logic in this small stream SQL snippet is no more than incrementing an integer counter (i.e. i = i + 1). In order to accomplish this very simple task a memory table (TradeIDTable) is used to INSERT and then SELECT a single row (1 AS RowPointer) that contains that incrementing integer (ON DUPLICATE KEY UPDATE TradeID = TradeID + 1) when a new TradesIn event is received. In a way, a rather creative use of SQL don't you think? However, simply extrapolate the state requirements beyond TradeID int and the processing logic beyond TradeID = TradeID + 1 and you quickly realize you would be bending the language to the point of breaking.

In the commercial application world, relational databases are an entrenched and integral component. SQL is the language for applications to interact with those databases. As applications have grown in complexity, the data needs have also grown in complexity. One outgrowth of this is a new breed of application service known as Object-Relational (O/R) mapping. O/R mapping technologies have emerged to fill the impedance mismatch between an application's object view of data and SQL's flat two-dimensional view. A wealth of O/R products are available today so the need for such technologies clearly exists.

Why am I mentioning O/R technologies in a CEP blog? Simply to emphasize the point that the SQL language, as validated by the very existence of O/R technologies in the commercial space, is a poor choice for CEP applications. As I've mentioned in previous blogs, programming languages that provide the vernacular to express both complex structures (objects) and complex semantics (programming logic) are as necessary for aggregation as they are for Orders-in-Market or any CEP application.

So what sort of language is appropriate for CEP? Well, there is always the choice of Java or C++. Using traditional languages such as Java and C++ clearly provide this expressiveness and can be used to build applications in any domain. However, trailing along behind that expressiveness is also risk. Using these languages means you start an application's implementation at the bottom rung of the ladder. The risk associated with this is evident in many a failed project. A step up is domain-specific languages. For the domain of streaming data, Event Programming Languages (EPL's) are clear winners. Like C++ and Java they contain syntax for defining complex objects (like an Aggregated Order Book) and imperative execution but they also include a number of purposed declarative constructs specifically designed to process streaming data efficiently. Apama's MonitorScript is one such EPL.

Hitting the nail on the head

In my continuing research into CEP vendor's use of the SQL as the language of choice for CEP applications I am reminded of a past industry initiative – but only in reverse! Not so long ago (relatively speaking of course) a new technology platform was being promoted by a number vendors - both large and small. That new technology was Application Servers. The core message and supporting technology was a backlash to the prior de facto standards (i.e client/server, host-based, etc.). The message from App Server vendors was separate your user interface from your business logic from your data. This message resonated well in the industry for many reasons. As the dust settled on the AppServer landscape a number of leading vendors (commercial and opensource) emerged – still with that same message. The value of separating UI's from business logic from data was clear then and still is today.  User Interfaces are like the fashion and electronics industry. Everyone wants the coolest (UI) gadgets available (witness Adobe Flex and Microsoft WPF). Business logic is central to operational effectiveness of the Enterprise. As for the data, once it was de-coupled it meant that business logic components could break out of their silos and access data from all parts of an organization. SOA has now superseded AppServers as the leading edge technology for commercial applications – but that separation message still rings true.

Now a new paradigm enters the arena – Complex Event Processing. Numerous vendors are parading their platform in front of prospects, customers, analysts and the market at large. The dust is far from settled but a couple of paradigms are starting to emerge. One builds on the archetype SQL database syntax which I'll refer to as an Event Query Language (EQL). The other model is one that builds upon classic development languages, those used for building complete applications. I'll refer to that as an Event Programming Language (EPL).

What I've begun to see from the CEP EQL (SQL-based, remember from my definition) vendors is an attempt to convince the industry that SQL (with its streaming extensions) is a language for building CEP applications. As such they are violating the separation rules (of business logic and data) that I outline above. Furthermore, the mashing together of business logic and data makes building different iterations of such applications and evolving those iterations over time tremendously cumbersome. Business logic is ever evolving and must be easily adaptable to changing business climates, competitive pressures and regulatory agencies.  The semantics of this logic should be easily articulated using an appropriate metaphor. An EPL by definition provides the syntactic wealth of expression for this purpose. A Rules-style metaphor is also a viable alternative. Apama's Monitorscript, a mature EPL and the Apama Scenario Modeler which provides that Rules style metaphor are well suited for the purpose of building complete CEP applications.

Not to be too inflammatory, SQL is well suited for filtering or enriching data, whether that is from traditional relational databases or from streaming data sources (via the streaming extensions). However, it's no more suitable for the semantic expression of business logic for CEP than it is (or ever was) for traditional commercial applications (in any deployed form; host-based, client/server, AppServer or SOA).

Looking at a few EQL examples you'll begin to see the pattern that I'm referring to. First, a simple, easy to understand example:

SELECT symbol, VWAP(price) FROM Ticker [RANGE 15 minutes]

From this simple SQL statement it's easy to see basic filtering and enrichment of the raw Ticker data. The result set is temporally organized into 15 minute buckets and grouped by symbol with a calculated value – VWAP. However, once you move beyond simple enrichment into complex condition detection the language becomes horribly unwieldy. Furthermore, once you add the need to manage state (see When all you have is a hammer everything looks like a nail) you've moved beyond unwieldy to undecipherable or more likely – impossible to implement.

Here's one example published by an EQL vendor:

CREATE VIEW vwap_stream (vwap_price) AS RStream(SELECT symbol, VWAP(price) FROM ticker [RANGE 15 minutes]);

CREATE VIEW vwap_outside_price(vwap_outside_count) AS SELECT COUNT(*) AS price_outside_vwap FROM ticker, vwap_stream [range 15 minutes] WHERE price - vwap_price > 0.02*price AND symbol = "MSFT";

CREATE VIEW trade_cond_stream (matching_row_count) AS SELECT COUNT(*) FROM ticker [RANGE 2 minute] RECOGNIZE ONE ROW PER MATCH PATTERN [S T] DEFINE S AS |price - PREV(price)| <= .05*PREV(price) AND symbol = "S&P" DEFINE T AS (price >= 1.05*PREV(price) AND symbol = "IBM") OR (price <= 1.02*PREV(price) AND symbol = "MSFT");

Here's the narrative of what it's intended to accomplish:

IF MSFT price moves outside 2% of MSFT-15-minute-VWAP FOLLOWED BY S&P moving by 0.5% AND IBM's price moves up by 5% OR MSFT's price moves down by 2% ALL within a 2 minute time window THEN [Signal] BUY MSFT and SELL IBM;

One note, the final part "THEN BUY MSFT and SELL IBM" is an exception. In the EQL example, there is no provision to take the BUY/SELL action only a means to signal it. Implementing the action is left as an exercise for the user.

A bit of commentary…

• One minor point in the example (with respect to the narrative), there isn't a symbol called S&P on the U.S Equities market (assumed to be Ticker)   S&P is an acronym for the Standard and Poor's Index of 500 leading   companies in leading industries of the U.S. economy. One will not find that index on a Ticker;   it needs to be calculated at runtime like the VWAP value.

• The vendor in question highlights   the terseness of the syntax. While it's clearly terse it's arguable if   terseness is goodness. Perl is quite terse and I've rarely heard anyone   singing its praises. In fact overly terse languages are often referred to as "write-only code". Meaning someone wrote it but no one can read it.

• There are only a few recognizable idioms – most of the processing logic is pre-defined implicit behavior of the query processor. For an applications programmer who needs to both understand and control the code that implements true business logic this presents a most disconcerting situation. And certainly one that I would not feel comfortable owning.

• Lastly is the violation of separation, multiple streams of data – both raw (Ticker) and derived (vwap_stream, etc.) are inseparable from the semantics of detecting a "BUY/SELL condition". 

Engineers (also known as programmers but that term has long since lost its glamour) still prefer languages with well known, tried and true vernacular. It gives them the control and wide-ranging expressiveness necessary to implement business logic and to have confidence in its correctness and behavior. State of the art optimizers and just-in-time compilers far outweigh any presumed benefits of a terse syntax. 

To conclude, the vendors of Application Servers (and now SOA) learned early on that separation of user interfaces from business logic from data was essential. The purveyors of EQL languages are violating this cardinal rule of Separation. Unfortunately, it was inevitable given the restrictiveness of the SQL syntax.