The Unexpected Happens

When Complex Event Processing (CEP) emerged around ten years ago, one of the early applications was real-time risk management. In the financial sector, there was growing recognition for the need for real-time visibility - continuous calibration of positions – in order to keep pace with the emerging importance of algorithmic trading. This is now relatively well-established in banking and trading sectors; Chemitiganti argues that the insurance industry now faces similar requirements.

In 2008, Chris Martins, then Marketing Director for CEP firm Apama, suggested considering CEP as a prospective "dog whisperer" that can help manage the risk of the technology "dog" biting its master.

But "dog bites master" works in both directions. In the case of Eliot Spitzer, the dog that bit its master was the anti money-laundering software that he had used against others.

And in the case of algorithmic trading, it seems we can no longer be sure who is master - whether black swan events are the inevitable and emergent result of excessive complexity, or whether hostile agents are engaged in a black swan breeding programme.  One of the first CEP insiders to raise this concern was John Bates, first as CTO at Apama and subsequently with Software AG. (He now works for a subsidiary of SAP.)

from Dark Pools by Scott Patterson

And in 2015, Bates wrote that "high-speed trading algorithms are an alluring target for cyber thieves".

So if technology is capable of both generating unexpected events and amplifying hostile attacks, are we being naive to imagine we use the same technology to protect ourselves?

Perhaps, but I believe there are some productive lines of development, as I've discussed previously on this blog and elsewhere.


1. Organizational intelligence - not relying either on human intelligence alone or on artificial intelligence alone, but looking for establishing sociotechnical systems that allow people and algorithms to collaborate effectively.

2. Algorithmic biodiversity - maintaining multiple algorithms, developed by different teams using different datasets, in order to detect additional weak signals and generate "second opinions".

---

John Bates, Algorithmic Terrorism (Apama, 4 August 2010). To Catch an Algo Thief (Huffington Post, 26 Feb 2015)

John Borland, The Technology That Toppled Eliot Spitzer (MIT Technology Review, 19 March 2008) via Adam Shostack, Algorithms for the War on the Unexpected (19 March 2008)

Vamsi Chemitiganti, Why the Insurance Industry Needs to Learn from Banking’s Risk Management Nightmares.. (10 September 2016)

Theo Hildyard, Pillar #6 of Market Surveillance 2.0: Known and unknown threats (Trading Mesh, 2 April 2015)

Neil Johnson et al, Financial black swans driven by ultrafast machine ecology (arXiv:1202.1448 [physics.soc-ph], 7 Feb 2012)

Chris Martins, CEP and Real-Time Risk – “The Dog Whisperer” (Apama, 21 March 2008)

Scott Patterson, Dark Pools - The Rise of A. I. Trading Machines and the Looming Threat to Wall Street (Random House, 2013). See review by David Leinweber, Are Algorithmic Monsters Threatening The Global Financial System? (Forbes, 11 July 2012)

Richard Veryard, Building Organizational Intelligence (LeanPub, 2012)

Related Posts

Black Swans and Complex Systems Failure (April 2011)
The Shelf-Life of Algorithms (October 2016)
Robust Against Manipulation (July 2019)

Intelligent Business Operations

@opheretzion reposts a healthcare example from @jimsinur concerning resource allocation in surgeries.

The loop is as follows.

Decision / Planning	Scheduling and resource allocation for the following day, using simulation and optimization tools.
Information Gathering	Real-time tracking of selected "things" (physicians, nurses, equipment; monitor of procedure duration and status) using a range of devices (sensors and cameras).
Sensemaking	Detecting deviations from plan - "things going wrong"
Decision / Planning	Revising the plan in "near real time"

Obviously this is an impressive use of the relevant technologies, and it may well deliver substantial benefits in terms of supply-side cost-effectiveness as well as a safer and better experience for the patient. This is essentially a goal-directed feedback loop.

However, we may note the following limitations of this loop.

1. Decision / Planning is apparently based on a fixed pre-existing normative model of operations - in other words a standard "solution" that should fit most patients' needs. This may be a reasonable assumption for some forms of routine surgery, but doesn't seem to allow for the always-present possibility of surprise when you cut the patient open.

2. Information Gathering is based on a fixed set of things-to-be-monitored. Opher calls this "real-time tracking of everything" - but of course this is a huge exaggeration. Perhaps the most important piece of information that cannot be included in this rapid feedback loop is the patient outcome. We might think that this cannot be determined conclusively until much later, but there may be some predictive metrics (perhaps the size of the incision?) that may be strongly correlated with patient outcomes.

3. Sensemaking is extremely limited - there is no time to understand what is going wrong, or to carry out deeper root-cause analysis and learning. All we can do is to react according to previously established "best practice".

4. Replanning is limited to detecting and quick-fixing deviations from the plan. See my post on Real-Time Events.

Full organizational intelligence needs to integrate this kind of rapid goal-directed feedback loop with a series of deeper analytic sensemaking and learning loops. For example, we might want to monitor how a given surgical procedure fits into a broader care pathway for the patient. Real-time monitoring is then useful not only for near-real-time operational intelligence but also for longer-term innovation.

---

Jim Sinur Success Snippet (January 2012)

Opher Etzion Medical Use Case (January 2012)

And please see my draft Organizational Intelligence Primer, now available on LeanPub.

High Frequency Trading

Tim Bass upsets a few more people with his provocative posts arguing that authorities should Strongly Regulate High Frequency Trading (May 2010) and claiming that High Frequency Trading Destroys Market Integrity (May 2010).


Reading through the comments, it is necessary to separate three separate questions.

1. the potential value of certain classes of technology (notably CEP) to those using these technologies
2. the system-wide effects of using these technologies
3. the ethical consequences of these system-wide effects.

Taking the third question first, what does "market integrity" actually mean? The notion of "equal playing field" implies that certain forms of advantage are eliminated. The notion of "playing field" implies that certain forms of advantage may still exist - otherwise there would be no point playing.

For example, some sports are so expensive that only affluent people or well-funded organizations can compete. (Think F1 motor racing, yacht racing, anything with horses.) And in any sport, wealth can get you access to superior training facilities - tennis players with money can fly to Florida when it's raining at home. But there are still rules that prevent certain forms of unfair practices, or certain types of equipment, and there is still a certain amount of talent involved, so that the outcome cannot be simply predicted from the amount invested.

High frequency trading may indeed be directly available to some players and not others. Lots of people clearly believe this provides a genuine advantage, else they wouldn't invest in it. The effects of this are asymmetric - those possessing the technology may experience a detectable advantage, even if those lacking the technology may not experience a detectable disadvantage. Taking profits out of the system, without harming any individual investor, is a "victimless crime"; according to extreme libertarians, a "victimless crime" isn't a crime at all, and is therefore not an appropriate subject for regulation.

However, most people accept the need for some market regulation, not just to protect individual investors, but also to protect the integrity of the market. If only we knew what that actually meant in this case.

See Tim Harford, Algorithm and Blues (Financial Times Feb 2013)


Updated 3 Feb 2013

Two Second Advantage

There are two reasons I like TIBCO's new slogan "The Two-Second Advantage". (I have some reservations as well, but let's start with the good bits.)

advantage means something to business

The first reason is that it actually means something to the business, unlike the widely misunderstood technical term “real-time”. One company that is currently boasting “real-time” performance is SAP, which claims that its in-memory databases will result in analytics that are “really in real-time”. Larry Dignam quotes Hasso Plattner as follows. “In analytics, there’s theoretically no limitation on what you can analyze and at what level of detail. (In-memory databases) mean reports on a daily basis, hourly basis.” [ZDnet] @davidsprott calls this In-memory madness. Even if an hourly or daily report counted as “real-time” (which it doesn't), this kind of technical wizardry doesn't make any sense to the business.

On a Linked-In discussion thread recently, I've seen vendors excusing their misuse of the term "real-time" (to describe software that isn't strictly, or sometimes even remotely, real-time) by claiming that the meaning of technical terms evolve over time. Oh yeah, very convenient. But we shouldn't allow vendors to fudge perfectly good technical terms for their own marketing purposes, any more than we should tolerate car manufacturers self-interestedly redefining the word "friction".

(In The 2 second advantage...the 2 culture disadvantage? Vinnie Mirchandani praises TIBCO executives for being able to talk business, and contrasts them with the IT analysts in the expo hall, with a special dig at Gartner.)

Unfortunately, TIBCO is not content with the "two second advantage" slogan, and we find TIBCO CEO Vivek Ranadivé over-egging the pudding by introducing some additional notions including Enterprise 3.0. Transcript of HCL keynote by TIBCO CEO Vivek Ranadive (April 2010). For @neilwd "Enterprise 3.0 ... is the sound of a company trying too hard" (TIBCO, Enterprise 3.0 and the two-second advantage, May 2010. See also Tibco’s Hits and Misses (Ovum, May 2010).

advantage is relative

The second reason I like the phrase "Two Second Advantage" is that it focuses our attention on the business advantage - not of raw speed but of getting there first. If you are a speculator who judges that some asset is overvalued or undervalued, the way to make money from this judgement is to buy or sell and then wait for other speculators to arrive at the same judgement. Being just ahead of the pack is actually more profitable than being a long way ahead, because you don't have to wait so long.

And although simple decisions can be taken quickly, complex decisions need time for understanding. (See my presentation on Mastering Time.)With complex decision-making, it's about spending just enough time to process just enough information to make a good enough judgement.

Ranadivé also talks about two trends - the increasing volume of data and the diminishing half-life. (In physics, the concept of "half-life" suggests a long tail of residual value - just as a radioactive sample will always remain somewhat radioactive, so the value of data never reaches zero.)

But as @neilwd points out, these trends don't necessarily refer to the same kinds of data, especially if we measure data volumes in terms of physical storage, since these volumes are dominated by email attachments and rich media. Maybe we need to find a way of measuring data volumes in terms of information content ("a difference that makes a difference"): as the cost of data transmission and storage continues to get smaller, it is not the number of megabytes but the number of separate items (giving managers the experience of being overloaded with information) that really matters.

Even if we limit ourselves to traditional data, the relationship between data volumes and response speed is not as simple as all that. Let's look at a specific example.

If a retail store gives a hand-held scanning device to the customer and/or places electronic tags on all the goods, it can collect a much higher volume of data about the customer's behaviour - not merely the items that the customer takes to the check-out but also the items that the customer returns to the shelf. As technology becomes cheaper, this enables a huge increase in the volume and granularity of the available data, collected while the customer is still shopping, and therefore the retailer actually has more time to use the data before the customer leaves the store.

For example, you might infer from the customer's browsing behaviour that she is looking for her favourite brand of pasta sauce. The shelf is empty, but there's a new box just being unloaded from a truck at the back of the store. Find a way of getting a jar to the customer before she reaches the checkout, and there's your two second advantage.

Event-Driven Enterprise Architecture

@toddbiske responded to @jeffrschneider 's question about the trigger for EA services.

• What are your EA Services?
• EA Services: Part Two

 Todd distinguishes between two kinds of service, which he calls request/response-style and event-driven services.

"There are services that are explicitly invoked at the request of a consumer, and then there are services that are executed in response to some event. In the latter case, the team providing the service is the one monitoring for the event. If some other team was monitoring for it, and then told your team to do something, then we’re back to the request/response style with one team acting as consumer and the other acting as the provider."

But surely a request is an event? The EA team may have many conflicting demands on its time and attention, and so there is always a decision (perhaps driven by some policy) whether and how quickly to respond to a given request.

A kind of event that might trigger EA services is one that looks like "we think we might have a problem in such-and-such area". (This is what Philip Boxer calls a P-type service - rcKP - services at the edge.)

In this example, "we" could be the EA team or the EA customer or both together. The service might include clarifying whether there really is a problem at all, not just solving a known problem.

The event triggering this kind of work is typically a weak signal. There are usually more possible problems than we actually have time to work on, so the existence of a problem is not a sufficient trigger for activity.

One interesting question here is how the event-driven EA actually works. The EA team must have some view of a complex and dynamic world to which it is attempting to add value, that allows it to organize its work. In other words, it needs a model (possibly implicit, but preferably explicit). But what model is this? Not just the enterprise model itself, not just the enterprise architecture framework (TOGAF or what have you) but something else as well.

Towards the Intelligent Business

What do organizations need to be successful in good times and to survive in bad times?

It's not just about the people (a few talented individuals on large bonuses). And it's not just about the software systems (sophisticated real-time complex event processing or whatever). What matters is how the people and the systems collaborate effectively to address emerging business opportunities and threats, in an agile and innovative manner. I call this Organizational Intelligence.

Organizational intelligence is like human intelligence - it involves the ability to solve problems, to make good judgements, to deal with complex situations in an effective and uncomplicated manner, and to learn from experience.

Many of the technologies I've been talking about on this blog can be used to support Organizational Intelligence, including Business Intelligence (BI), Business Process Management (BPM), Complex Event Processing and Enterprise 2.0.

Conversely, the concept of Organizational Intelligence helps us to work out two things.

1. To identify the potential contribution of these technologies to business survival (business value).
   
2. To determine how these technologies can be combined to support the requirements of a specific organization (architecture).

Technologies for Organizational Intelligence

View more presentations from Richard Veryard.

Enterprise as a Network of Events 2

Following my post on Enterprise as a Network of Events, Nigel Green drew my attention to something he wrote a couple of years ago on the case for a clear distinction between events and content, in which he points out that "aggregated events become content over time".

As it happens, I touched on something like this in my 1992 book on Information Modelling. At that time, the specific question I was addressing was how to represent the history of organizational change in a data model. There are two basic options.

1. History is represented as a series of snapshots of the organization at specific times. If you want to know what the change was, you just have to compare two snapshots. (For example, this is how page history is represented on Wikipedia.)
   
2. History is represented as a series of changes (differences). Then if you want to know what the state of the organization was at a given time, you just have to aggregate the changes forwards or backwards from a fixed point.

From a purely logical point of view, it doesn't matter which of these two representations you adopt, because you can always derive one from the other. In practical terms, however, you may find that one representation is a lot easier to work with than the other.

What both representations have in common is a definition of information in terms of difference: a difference that makes a difference, as Bateson puts it.

Information as Difference

Hold your hand perfectly still, palm upwards and resting comfortably on a table. With your other hand, drop a small coin into the palm. You will feel the impact, and if the coin is cold, you will feel the coldness of the metal. Soon however, you will feel nothing. The nerve cells don't bother repeating themselves. They will only report to the brain when something changes. Information is difference.

A lizard hunting insects operates on the same principle. The lizard's eye only reports movement to the lizard's brain. If the hunted insect settles on a leaf, the lizard literally cannot see it. But the moment the insect starts to move, whop, the lizard can see it again, and the tongue flickers out and catches it.

But there are differences and differences. Information is difference that makes a difference. You were probably aware, as you dropped the coin into your palm, your eyes told you automatically, without your brain even asking, what the value of the coin was; but you were probably not aware what date it was minted. This is because (unless you are a numismatist) the value of the coin makes a difference to you whereas its date doesn't.

What is it that makes a difference to a lizard, to a numismatist, to you? Surely not the same things. What is information for the lizard is not information for you, and what is information for you is not information for the lizard.

This is why the perspective of information is important. Perspective defines what counts as information at all, perspective defines to whom the information makes a difference.

Enterprise Strategy and Difference

Now what's this got to do with the enterprise? There are two important differentiation questions for any enterprise: how do They differentiate Us, and how do We differentiate Them.

Firstly how do They differentiate Us. In other words what differences in our products and services or organization are (a) going to be noticed by external stakeholders such as customers and (b) going to affect the behaviour and attitudes of these stakeholders.

Sophisticated marketing organizations pay a lot of attention to this kind of thing. Does it make a difference if we put this in a blue envelope? Does it make a difference if we have a female voice here? Sometimes you have to experiment with differences that don't matter, in order to have a refined view of what differences do matter.

Secondly, how do We differentiate Them. What are the strong signals in the environment that we must respond to, and what are the weak signals we might respond to? An organization that responded to every weak signal would be impossibly unstable, so most organizations have operational processes that respond to strong signals, and some form of business intelligence that scans the environment for weak signals and identifies a subset of these signals demanding some kind of response.

Sometimes the appropriate response to some detected difference or discrepancy is some kind of business improvement, such as business process improvement and/or system engineering. Which means that the history of the organization is inscribed into its current structure and processes.

"What's the purpose of this process step?"

"We've done that extra check ever since we got badly burned in the XYZ deal five years ago."

If we knew enough (which of course we don't), we might conceivable infer the current state of the organization from a careful analysis of all the events that have ever happened to it. That would join this post back to where we started. Neat but impractical.

But that's not actually what I'm trying to do. I think understanding an organization as an information-processing system helps us to do two practical and concrete things.

• Diagnose the current problems and opportunities facing a complex organization.

• Design an event-driven business improvement process, based on business intelligence.

My next question is: what kind of business model supports these challenges.

Mollusc-Driven Architecture

Thanks to Brenda Michelson, I have picked up a wonderful example of Smart SOA from Sandy Carter of IBM.

"RFID tagging blue mussels to identify oil leaks -- mussels closes, alert sent, line stopped; no mussels harmed"

According to Steve Núñez of Illation, this is done by Norwegian oil company StatOil. See Benefits of Going Green: Who Would Have Thought? I hope the alert is not sent by snailmail.

(Aside: For their role in detecting leaks, blue mussels have been dubbed the canaries of the sea. And don't miss the Statoil recipe for Manhatten Mussel Chowder!)

For a longer description, plus links to Sandy Carter's presentation, see Brenda's post Sandy Carter on Smart SOA in Tough Economic Climate.

Enterprise as a Network of Events

From the early days of SOA, we've talked about understanding the enterprise as a network of services, but clearly this is not the only possible viewpoint. Can we understand the enterprise as a network of events?

In a recent post SOA and Events - Two Sides of BAM?, Ramesh Loganathan (Progress) explained how a process view of the enterprise could be flipped into an event-oriented view. Ramesh's example is in governance risk and compliance (GRC), but it looks as if the same idea could be applied to a much broader range of business requirements.

Meanwhile, in Decision Management and software development, James Taylor wants to model the enterprise as a series of decisions. But of course a decision can be understood as a special kind of event. See Model Driven Engineering and Event Processing by Opher Etzion (IBM). See also SOA is dead; long live Model-Driven SOA by Johan den Haan.

In his post Event, Rules, Processes, Decisions, Paul Vincent (TIBCO) identifies an important gap in the notations (BPMN, BMM) available for modelling these aspects of the enterprise. He talks about decisions in terms of rules, but I presume he is associating the rule with the decision process (this is how we authorize transactions) rather than with the specific instance of decision (I hereby authorize this transaction).

At the strategic level, we need to understand the relationship between an external set of possible events and an internal set of possible events. Each enterprise is capable of responding in certain ways to certain classes of external events. Making an enterprise more agile means enabling the enterprise to mobilize more appropriate responses to a greater range of external events, without proliferation of unnecessary complexity. In system thinking this is called Requisite Variety.

So if we think about the enterprise as a network of events, this gives us a direct way to reason about strategic business improvement. But what are the CEP vendors doing in this space? I can see a lot of talk about specific event-driven applications, but where is the architectural thinking about the event-driven enterprise?


Update: After I posted this, Nigel Green drew my attention to something he wrote a couple of years ago on the case for a clear distinction between events and content, which he describes in terms of the wave-particle metaphor.

Event Processing and Security

What is the relationship between Complex Event Processing (CEP) and Application Security (asks James McGovern)? My answer is that there is not one single relationship, but there may be relationships in both directions.

Perhaps the most obvious relationship is that event processing technologies can be used to support application security, helping to detect security threats and to monitor security compliance.

But there is also a reverse relationship. A CEP system may itself have vulnerabilities, and Application Security principles may be needed to address these vulnerabilities. For example, any event detection system is vulnerable to deceptive events, known as chaff.

Actually, there are two different concepts of chaff at play here. Some CEP experts use the term as it is used in agriculture, to refer to uninteresting or unwanted stuff. For example, see David Luckham and Mark Palmer, Separating Wheat from Chaff (RFID Journal, October 2004).

Meanwhile, security-conscious CEP experts, such as my friend Tim Bass, use the term as it is used in radar defence, to refer to decoy events. Chaff are pieces of metal designed to create misleading information for enemy radar systems. (Interestingly, in the second world war, each side invented this technique but hesitated to deploy it, for fear of revealing the technique to the other side.) For example, see Tim's post on Quintessential Event Processing (November 2008).

Both kinds of chaff increase the level of clutter, thus reducing the accuracy and reliability of event-driven systems. For example, see this article about the effect of clutter on security: Five ways to clean your firewall of clutter and stay secure.

David Luckham (again) mentions clutter as a user design issue (Dashboard Design) but even this kind of clutter has system consequences: it affects the effectiveness of a man-machine system; and this is not so very different from the consequences of (a different kind of) clutter on the effectiveness of a firewall.

From the accounts I have read of the Three Mile Island case, my understanding is that it was something akin to clutter (in David's sense) that made it almost impossible for the operators to respond intelligently and appropriately to a series of system alerts. The Three Mile Island case has important lessons for both event-processing systems and for security.

In terms of the manifold relationships between complex event processing and application security, which was where James' original question led us, I think that one of the possible relationships is that they both rest on the same set of fundamental concepts and theories - for example sociotechnical systems and cybernetics.

To understand these relationships more deeply, we need a general account of (secure event-driven) or (event-driven secure) systems, and this might involve more abstract but rigorous notions of chaff and clutter.

---

originally posted in the CEP discussion group on Linked-In

Event Reduction

Opher Etzion on Event Reduction

“in some cases, the business benefit of event processing system is actually reducing the number of events, and focus the decision makers on the important events”

An event processing system generally has some degree of event reduction and aggregation. There may be real-world events that are not represented as system events (for example, a continuous real-world process is reduced to a series of periodic instrument readings, and the intermediate points are not captured). And some events may be front-end captured but subject to some filtering so that not all of them are passed onto the business applications.

In both cases, the total behaviour of the enterprise system may be affected by the quantity and variety of events that get through to the business applications. Too many events, and the decision-makers may be unable to focus on the important ones. Too few events, and the enterprise may lack requisite variety. A good cybernetic systems design would include feedback loops, so that the quantity and variety of events can be adjusted until the enterprise works to the maximum efficiency and effectiveness.

So an event processing architecture needs some kind of variety regulator (“regulator” in the engineering sense, not the legal compliance sense) – some kind of monitoring and control. For example, changing the frequency of some measurement, or changing the filtering or aggregation rules.

In very simple event processing, the systems designers might possibly be clever enough to calculate the right measurement frequencies and event processing rules in advance, although I don’t know of any robust methodology for doing this. For complex event processing the mathematics start to get too difficult, and we need to rely on proper engineering and not just simple arithmetic.

Market Predictions - CEP

What lies in store?

Complex Event Processing (CEP) guru David Luckham asks What are your predictions for 2009?

What challenges?

Prediction is hard, especially for experts. What is easier is to identify some of the challenges that have to be addressed. For example:

• Detection (Tim Bass)
  
• Interoperability (Opher Etzion, Tim Bass)
• Scope and Terminology (Opher Etzion, Paul Vincent)
  

CEP Products or Services?

David's starting point was a Forrester estimate of the market for CEP software and services. But what kind of services are these? Is Forrester just talking about conventional systems integration services - e.g. paying consultants to build and install your CEP systems? Or are we starting to see a genuine service economy based around the trading and collaborative processing of events?

A certain amount of this kind of thing goes on in the security world, with specialist firms performing what is essentially collective event processing for a number of customers (Monitoring-as-a-Service). But apart from that, I haven't seen much evidence of a distributed economy of complex events.

But why might this kind of thing be particularly interesting in 2009? Because the prevailing economic environment may make it harder to justify people going it alone, building large and complex event-processing applications for their own use.

Shortening the lead

Most people are expecting 2009 to be a tough year. In such circumstances, there is a widespread reluctance to invest scarce resources on remote gains; any vendors trying to sell solutions to business will need to find innovative ways of shortening and tightening the lead between investment and return.

For example, instead of vendors offering an elaborate set of CEP products, together with consultants skilled in wiring them together, there may be demand for out-of-the-box hosted solutions.

Shared Event Semantics

But in order to share events and event processing, firms will need to have a common event model. Which brings us back to some of the challenges identified by Opher, Paul and Tim ...

And Not Just Complex Event Processing ...

These arguments don't just apply to complex event processing, but to many areas of new technology. I'll look at some other areas in subsequent posts ...

How many events?

Marco writes about Event Models - Identity, and points out that there may not be a simple mapping between events and the reporting of events. A single collision may result in multiple notifications. He concludes:

"Something as simple as a event id could have a rather complex semantics and you should be aware of this when designing your event model. Or is it just me complicating things?"

I agree that event semantics may be complex. If you have a three-car pile-up, does that count as one collision or two, given that the third car hits a few seconds after the first two? Or three collisions, if the third car hits both of the first two cars?

So you have to have some semantic model that provides a basis for counting how many collisions there are in the real world, let alone how many event reports are received. There is a semantic rule (sometimes called an identity rule, after Frege) as to when two things are the same, or the-same-again.

An identity rule (together with a membership rule - - which things count as collisions at all) should be part of your definition of “collision”.

See my earlier posts: How many products? and The Semantics of Security

From Complex Events to Predictive Analytics

Beth Gold-Bernstein (eBizQ) reckons that predictive analytics, which she describes as "a capability based on complex event processing (CEP)" is A Good Investment for a Down Economy.

Hans Gilde is scornful. In Predictive analytics, CEP and your local mechanic, he points out that the key capability for an effective predictive system is "a solid foundation for ongoing critical analysis of the effectiveness of your analytics".

I would extend his scorn to anyone who uses the term "real-time" in a sloppy and confused manner. I keep reading about "real-time complex event processing", but in many cases it looks as if much of the complex processing is actually done at design-time, leaving the real-time portion fairly quick and simple. And that's probably as much as the current state-of-the-art technology can manage.

To do predictive analytics properly, as Hans says, you need critical analysis - what is sometimes called double-loop learning. How does a CEP system learn new patterns? How does a CEP get recalibrated as the environment evolves? How do we control (and reduce) the level of false positives and false negatives? And how much of this analysis does it make sense to even attempt in real-time?

So I looked at a recent paper in the IBM Systems Journal, Generating real-time complex event-processing applications (Volume 47, Number 2, 2008).

"Complex event processing (CEP) enables the extraction of meaningful and actionable information from event streams. The CEP technology is intended to provide applications with a flexible and scalable mechanism for constructing condensed, refined views of the data. CEP correlates the data (viewed as event streams) in order to detect and report meaningful predefined patterns, thus supplying the application with an effective view of the accumulated incoming data (events), and allowing the application to react to the detections by executing actions."

The IBM paper talks about some of the challenges of achieving genuine real-time data extraction based on predefined patterns, and talks about the possibility of real-time CEP applications, but is careful not to make any larger claims.

Other writers are not so careful, and you can find lots of websites promising real-time predictive analytics or real-time decision-making. But there are not only technological but conceptual limits to what you can do in real-time.

Event-Driven Financial Catastrophe

Is Complex Event Processing (CEP) part of the problem or part of the solution?

On the one hand, apparently knowledgeable financial journalists tell us that automatic and globalized trading has contributed to an unprecedented level of global volatility. For example, the dramatic rise in Volkswagen shareprice yesterday appears to have been amplified by computer programs automatically reacting to a small price rise by attempting to buy shares (in order to unwind positions resulting from earlier short-selling by hedge funds), thus converting a small price rise into an extremely large one.

On the other hand, I am constantly seeing claims from vendors that their CEP products contain ways of dealing with market volatility, as if the volatility is merely caused by the failure of humans to react as quickly and as intelligently as their software.

But what I don't see in this kind of CEP material is a picture of how the whole system works: an evidence-based explanation as to what causes volatility, and what are the overall consequences of a particular kind of intervention. That kind of whole-system thinking is surely a key responsibility of the architect. By refusing to pay attention to the whole system, this kind of CEP discourse is almost anti-architectural. That's why I think we should call it JBOCE.

(Of course, a lot of so-called architects aren't very good at that kind of whole-system thinking, and indulge in what Nick Malik calls Bizarre Functional Decomposition, but that's another story.)

---

Speaking for one of the CEP vendors (Streambase), Mark Palmer (EDA and CEP: Where's the Beef?) protests against recent posts by Jack van Hoof (EDA versus CEP, Again...) and Joe McKendrick (Why 'Event Driven Architecture' is more than Complex Event Processing).

But I think the problem identified by Jack and Joe is not that vendors are asserting that CEP and EDA are the same thing. I am sure all the vendors, including Mark, can point to technical material that explains what these terms really mean. The problem is that the terms are being used in ordinary marketing material as if they were interchangeable - in other words, the vendors are not putting enough emphasis on the practical differences between CEP and EDA. I think the spate of claims about CEP "solving" market volatility is further evidence of this.

JBOCE - Just a Bunch of Complex Events

For quite a while now, we have been trying to explain to people that Service Oriented Architecture (SOA) is a lot more than Just A Bunch Of Web Services (JBOWS).

In his latest post, EDA versus CEP, Jack van Hoof reminds us that event processing alone, however complex, doesn't necessarily imply a proper architectural approach.

"Think of the architectural challenges of semantics mediation, extremely loosely coupled process flows and transaction control. And think of security in a extremely loosely coupled environment: authentication, authorization, encryption, credential assertion, non-repudiation. All of these aspects are not explicitly addressed in CEP (Complex Event Processing), but are in EDA."

So I plan to start using a new acronym - JBOCE - Just A Bunch Of Complex Events.

SOA Example - Real-Time Regulation

In a couple of recent posts on Turbulent Markets, I asked whether real-time profit and loss could tame turbulent markets (answer: No), and asked whether some other form of real-time event-driven system could perform a regulatory function (answer: Possibly).

Bloggers from some of the CEP vendors have been making similar suggestions for a while.

Back in March 2008, there was a flurry of interest in the strange fate of Eliot Spitzer, who was apparently exposed by the very regulatory technologies he himself had advocated.

• The Technology That Toppled Eliot Spitzer (Technology Review)
• Algorithms for the War on the Unexpected (Emergent Chaos)

Jesper Joergensen of BEA (BEA now part of Oracle, Jesper has now joined SalesForce, Jesper's BEA blog has disappeared) took the opportunity to put in a plug for BEA's event processing products. "If anyone working in a bank's anti-money laundering, compliance or fraud detection unit is reading this", he writes, "go check out [my company's products]. This is the technology you need to automate these compliance requirements."

There is obviously a need for systems to trap people like Eliot Spitzer. But I'm not convinced that simple compliance systems need to be real-time service-oriented event-driven systems. See my post on Real-Time Fraud Detection.

But a much stronger case can be made for real-time risk management. Chris Martins (Progress Apama) put the case for CEP and real-time risk in March 2008. More recently, Jeff Wotton (Aleri) has put the case for real-time risk consolidation, drawing on an interview with Nick Leeson. Meanwhile, Progress Actional has a product page on Real-Time Risk Profiling for Banks.

The point about real-time risk aggregation is that you need to produce a rapid and reliable picture of total risk, drawing on data from many heterogeneous sources. In a typical business environment, new types of risk and sources of data are constantly being added, and you want to be able to plug these into your risk consolidation straightaway. In this kind of scenario, it should be very easy to justify SOA.

Event Processing Example - Turbulent Markets 2

In my previous post on Turbulent Markets, I answered a rhetorical question posed by Bob Giffords and Mark Palmer ("Can Real-Time Profit and Loss Tame the Turbulent Markets?"). Mark has now followed up with a proposal to Regulate News Market Data Sources.

The story starts with a large fluctuation in the stock price of United Airlines (UAL), which dropped from $12 to $3 in just 15 minutes, apparently in an over-reaction to an incorrect news story. Some journalists blamed computers. In response, Brenda Michaelson pointed out that "Shiny technology alone doesn't guarantee better decisions. Just faster decisions."

Of course, as Mark points out, what counts as a "better decision" depends on your perspective. The people who reacted fastest to the incorrect news story probably made lots of money. In the very short term, a market trader doesn't care whether a news story is true or false, because a false news story can have just as strong an effect on prices as a true one. In the longer term, true (or at least well-corroborated) stories usually (but not always) win out over false (or poorly corroborated) stories; but by that time the market may be chasing a completely different rumour.

But Mark acknowledges that such wild fluctuations are "just not right" - in other words wrong for everyone else, and probably for the stability and integrity of the market as a whole. So Mark's proposed solution is to regulate rumour - "automated trading based on unregulated sources of information should be prohibited".

Apart from the fact that the control of rumour sounds like one of the twelve labours of Hercules ...

(in fact Agatha Christie adopted exactly that interpretation of the Lernaean Hydra when she wrote twelve stories for Hercule Poirot based on the twelve labours)

... the notion of regulating sources of information seems to inhibit many of the supposed benefits of real-time event processing in financial markets. Unless of course you plug some real-time regulation (in other words, a real-time event-driven information filter, directly controlled by the regulator) into an event-driven system architecture. Now that would be an interesting challenge.

Event Processing Example - Shoplifting

Perhaps shoplifting (or rather the prevention of shoplifting) is a good application of complex event processing (CEP).

• Marc Adler CEP and Shoplifting.
• Tim Bass Modelling Shoplifting

Shoplifting prevention has a level of complexity that is absent from something like air traffic control (the subject of Tim's next post). The behaviour of aircraft is subject to the laws of physics, which (at least for the purposes of air traffic control) are pretty well understood and unlikely to change. Whereas the behaviour of shoplifters is (i) subject to the observation and confirmation of shoplifting patterns and (ii) subject to learning and innovation by the shoplifters themselves. Therefore the two cases aren’t the same: there is a form of flexibility I’d want to build into a shoplifting system that I wouldn’t need in the Air Traffic Control system.

I think we need to distinguish between modelling shoplifting (constructing a theory about the observable patterns associated with shoplifting) and modelling shoplift-detection (designing a system that will observe or infer these patterns). In the comments to Tim's blog, there is a discussion on the relative merits and technical feasibility of two mechanisms: image processing and RFID-based merchandise-object tracking. I think these two mechanisms belong to a model of shoplift-detection rather than a model of shoplifting; in discussing them we are not saying anything about the phenomenon of shoplifting itself.

Perhaps the reason I think this distinction might be important is that I’m coming at this from a service-oriented perspective, and so I’m always looking for ways to build solutions in a loosely coupled way, therefore decoupling the models. I accept that other people may not see this as (i) relevant or (ii) technically feasible at the current state-of-the-art.

There is of course an arms race between shoplifters and shoplifting detection systems. Shops fit cameras and RFID tags, shoplifters adopt new evasion tactics and carry cheap devices to reprogram the RFID tags. So both the shoplifting model and the shoplifting-detection models evolve over time.

The evolution of the shoplifting model sometimes affects the evolution of the shoplifting-detection model. Suppose we discover that professional shoplifters work in teams - one distracts the shop assistant, a second one puts the item into a bag, then switches bags with a third team member before leaving the store. So even if the theft was spotted on the cameras, the thief no longer has the goods, goes protesting to the manager's office to be searched, and then leaves the store with an apology. How are you going to build a system to detect that? It's no good just tracking individual shoppers and trying to determine which of them might be shoplifters (which is the form of simplistic event-response I criticized in my earlier posts). The granularity of the event-tracking system has to change in response to a new theory about the behaviour of shoplifters.

However, the shoplifting-detection model may evolve as a consequence of technological change - improved image processing, more sophisticated or widespread RFID. This change doesn't immediately affect the behaviour of shoplifters, although shoplifters may change their behaviour over time to respond to these advances in technology. Loosely coupled, therefore.

Event Processing Example - Turbulent Markets

Can Real-Time Profit and Loss tame the turbulent markets? ask Bob Giffords (independent analyst) and Mark Palmer (Streambase).

The simple answer is No. Turbulence is a complex systems phenomenon. (Complex event processing is not primarily about complex systems, although some of the advocates of complex event processing might benefit from knowing a bit more about complex systems.)

If we want to know how turbulence can be tamed, we need to understand the root causes of turbulence, in terms of the non-linear effects of feedback loops. This is properly a job for market regulators. For example, central banks may try to reduce volatility in the money supply, and have sophisticated economic models to support their analysis. But the recent history of market regulation is a sorry one. Some analysts have argued that regulations such as Basel2 actually amplify volatility and turbulence in the system, because they force individual banks to execute transactions in response to market movements in order to maintain key ratios.

So it would be interesting to see an application of complex event processing in regulating a complex system. But this is not what the white paper is about. Perhaps wisely, it doesn't actually talk about taming the markets, merely about riding (=profiting from) the markets.

If some players have better tools, including CEP systems, this may give them an advantage in a competitive turbulent market. But this raises three important questions at the ecosystem level,

1. How does the use of these tools affect the market itself? Does the level of turbulence increase or decrease?

2. If the players with the best tools are those that profit the most from turbulence, then they possibly have an interest in promoting increased turbulence, even if this is damaging to everyone else.

3. What would happen to the ecosystem if these tools become commonplace? Would the advantages of these tools be reduced if everyone else had them?