The Unexpected Happens

When Complex Event Processing (CEP) emerged around ten years ago, one of the early applications was real-time risk management. In the financial sector, there was growing recognition for the need for real-time visibility - continuous calibration of positions – in order to keep pace with the emerging importance of algorithmic trading. This is now relatively well-established in banking and trading sectors; Chemitiganti argues that the insurance industry now faces similar requirements.

In 2008, Chris Martins, then Marketing Director for CEP firm Apama, suggested considering CEP as a prospective "dog whisperer" that can help manage the risk of the technology "dog" biting its master.

But "dog bites master" works in both directions. In the case of Eliot Spitzer, the dog that bit its master was the anti money-laundering software that he had used against others.

And in the case of algorithmic trading, it seems we can no longer be sure who is master - whether black swan events are the inevitable and emergent result of excessive complexity, or whether hostile agents are engaged in a black swan breeding programme.  One of the first CEP insiders to raise this concern was John Bates, first as CTO at Apama and subsequently with Software AG. (He now works for a subsidiary of SAP.)

from Dark Pools by Scott Patterson

And in 2015, Bates wrote that "high-speed trading algorithms are an alluring target for cyber thieves".

So if technology is capable of both generating unexpected events and amplifying hostile attacks, are we being naive to imagine we use the same technology to protect ourselves?

Perhaps, but I believe there are some productive lines of development, as I've discussed previously on this blog and elsewhere.


1. Organizational intelligence - not relying either on human intelligence alone or on artificial intelligence alone, but looking for establishing sociotechnical systems that allow people and algorithms to collaborate effectively.

2. Algorithmic biodiversity - maintaining multiple algorithms, developed by different teams using different datasets, in order to detect additional weak signals and generate "second opinions".

---

John Bates, Algorithmic Terrorism (Apama, 4 August 2010). To Catch an Algo Thief (Huffington Post, 26 Feb 2015)

John Borland, The Technology That Toppled Eliot Spitzer (MIT Technology Review, 19 March 2008) via Adam Shostack, Algorithms for the War on the Unexpected (19 March 2008)

Vamsi Chemitiganti, Why the Insurance Industry Needs to Learn from Banking’s Risk Management Nightmares.. (10 September 2016)

Theo Hildyard, Pillar #6 of Market Surveillance 2.0: Known and unknown threats (Trading Mesh, 2 April 2015)

Neil Johnson et al, Financial black swans driven by ultrafast machine ecology (arXiv:1202.1448 [physics.soc-ph], 7 Feb 2012)

Chris Martins, CEP and Real-Time Risk – “The Dog Whisperer” (Apama, 21 March 2008)

Scott Patterson, Dark Pools - The Rise of A. I. Trading Machines and the Looming Threat to Wall Street (Random House, 2013). See review by David Leinweber, Are Algorithmic Monsters Threatening The Global Financial System? (Forbes, 11 July 2012)

Richard Veryard, Building Organizational Intelligence (LeanPub, 2012)

Related Posts

Black Swans and Complex Systems Failure (April 2011)
The Shelf-Life of Algorithms (October 2016)
Robust Against Manipulation (July 2019)

Two Second Advantage

There are two reasons I like TIBCO's new slogan "The Two-Second Advantage". (I have some reservations as well, but let's start with the good bits.)

advantage means something to business

The first reason is that it actually means something to the business, unlike the widely misunderstood technical term “real-time”. One company that is currently boasting “real-time” performance is SAP, which claims that its in-memory databases will result in analytics that are “really in real-time”. Larry Dignam quotes Hasso Plattner as follows. “In analytics, there’s theoretically no limitation on what you can analyze and at what level of detail. (In-memory databases) mean reports on a daily basis, hourly basis.” [ZDnet] @davidsprott calls this In-memory madness. Even if an hourly or daily report counted as “real-time” (which it doesn't), this kind of technical wizardry doesn't make any sense to the business.

On a Linked-In discussion thread recently, I've seen vendors excusing their misuse of the term "real-time" (to describe software that isn't strictly, or sometimes even remotely, real-time) by claiming that the meaning of technical terms evolve over time. Oh yeah, very convenient. But we shouldn't allow vendors to fudge perfectly good technical terms for their own marketing purposes, any more than we should tolerate car manufacturers self-interestedly redefining the word "friction".

(In The 2 second advantage...the 2 culture disadvantage? Vinnie Mirchandani praises TIBCO executives for being able to talk business, and contrasts them with the IT analysts in the expo hall, with a special dig at Gartner.)

Unfortunately, TIBCO is not content with the "two second advantage" slogan, and we find TIBCO CEO Vivek Ranadivé over-egging the pudding by introducing some additional notions including Enterprise 3.0. Transcript of HCL keynote by TIBCO CEO Vivek Ranadive (April 2010). For @neilwd "Enterprise 3.0 ... is the sound of a company trying too hard" (TIBCO, Enterprise 3.0 and the two-second advantage, May 2010. See also Tibco’s Hits and Misses (Ovum, May 2010).

advantage is relative

The second reason I like the phrase "Two Second Advantage" is that it focuses our attention on the business advantage - not of raw speed but of getting there first. If you are a speculator who judges that some asset is overvalued or undervalued, the way to make money from this judgement is to buy or sell and then wait for other speculators to arrive at the same judgement. Being just ahead of the pack is actually more profitable than being a long way ahead, because you don't have to wait so long.

And although simple decisions can be taken quickly, complex decisions need time for understanding. (See my presentation on Mastering Time.)With complex decision-making, it's about spending just enough time to process just enough information to make a good enough judgement.

Ranadivé also talks about two trends - the increasing volume of data and the diminishing half-life. (In physics, the concept of "half-life" suggests a long tail of residual value - just as a radioactive sample will always remain somewhat radioactive, so the value of data never reaches zero.)

But as @neilwd points out, these trends don't necessarily refer to the same kinds of data, especially if we measure data volumes in terms of physical storage, since these volumes are dominated by email attachments and rich media. Maybe we need to find a way of measuring data volumes in terms of information content ("a difference that makes a difference"): as the cost of data transmission and storage continues to get smaller, it is not the number of megabytes but the number of separate items (giving managers the experience of being overloaded with information) that really matters.

Even if we limit ourselves to traditional data, the relationship between data volumes and response speed is not as simple as all that. Let's look at a specific example.

If a retail store gives a hand-held scanning device to the customer and/or places electronic tags on all the goods, it can collect a much higher volume of data about the customer's behaviour - not merely the items that the customer takes to the check-out but also the items that the customer returns to the shelf. As technology becomes cheaper, this enables a huge increase in the volume and granularity of the available data, collected while the customer is still shopping, and therefore the retailer actually has more time to use the data before the customer leaves the store.

For example, you might infer from the customer's browsing behaviour that she is looking for her favourite brand of pasta sauce. The shelf is empty, but there's a new box just being unloaded from a truck at the back of the store. Find a way of getting a jar to the customer before she reaches the checkout, and there's your two second advantage.

SOA Example - Real-Time Regulation

In a couple of recent posts on Turbulent Markets, I asked whether real-time profit and loss could tame turbulent markets (answer: No), and asked whether some other form of real-time event-driven system could perform a regulatory function (answer: Possibly).

Bloggers from some of the CEP vendors have been making similar suggestions for a while.

Back in March 2008, there was a flurry of interest in the strange fate of Eliot Spitzer, who was apparently exposed by the very regulatory technologies he himself had advocated.

• The Technology That Toppled Eliot Spitzer (Technology Review)
• Algorithms for the War on the Unexpected (Emergent Chaos)

Jesper Joergensen of BEA (BEA now part of Oracle, Jesper has now joined SalesForce, Jesper's BEA blog has disappeared) took the opportunity to put in a plug for BEA's event processing products. "If anyone working in a bank's anti-money laundering, compliance or fraud detection unit is reading this", he writes, "go check out [my company's products]. This is the technology you need to automate these compliance requirements."

There is obviously a need for systems to trap people like Eliot Spitzer. But I'm not convinced that simple compliance systems need to be real-time service-oriented event-driven systems. See my post on Real-Time Fraud Detection.

But a much stronger case can be made for real-time risk management. Chris Martins (Progress Apama) put the case for CEP and real-time risk in March 2008. More recently, Jeff Wotton (Aleri) has put the case for real-time risk consolidation, drawing on an interview with Nick Leeson. Meanwhile, Progress Actional has a product page on Real-Time Risk Profiling for Banks.

The point about real-time risk aggregation is that you need to produce a rapid and reliable picture of total risk, drawing on data from many heterogeneous sources. In a typical business environment, new types of risk and sources of data are constantly being added, and you want to be able to plug these into your risk consolidation straightaway. In this kind of scenario, it should be very easy to justify SOA.

Real-Time Events

Opher reports a car accident (unplanned events again) and concludes that people need to process events in real-time and not in batch.

Congratulations to Opher on his fast reactions, and commiserations on the slower reactions of the driver behind. Clearly there are some events you have to process in real time. But I hope he is not implying that all events must be processed in real-time. When the fuel tank indicator appears, do you refuel immediately or do you wait until you reach the next gas station? How often do you have the vehicle serviced?

I think the critical question for systems designers here is to determine which are the events that call for a real-time response, and which are the events where a batch response is more appropriate.

There are also events that call for a low-latency (extremely fast) response, but don't count as real-time. For example, in the financial markets, there may be short-lived arbitration opportunities, which means you can make a lot of money if you can react within a small number of milliseconds. This is not a real-time requirement, because nobody expects you to pick up every single opportunity - just catch a reasonable number of them. (The reactions of the frog should allow it to catch just enough insects to fill its belly - but some insects escape to breed more insects. The insects get faster, and so do the reactions of the frogs.)

Surely the event infrastructure should be capable of handling any of these patterns.

Real-Time Fraud Detection

Analytics vs. CEP

Alan Lundberg (TIBCO) notes the growing use of Complex Event Processing within the insurance industry, but is surprised not to find examples of CEP in real-time fraud detection. Fraud detection is often quoted by CEP vendors (including BEA, Progress and TIBCO itself) as one of the obvious applications of CEP. Although according to CEP evangelist Tim Bass, risk management experts in the banking sector remain sceptical about the benefits of CEP. Nevertheless, there do seem to be some interesting examples of CEP for real-time fraud detection, in the financial markets and in retail.

• FSA sharpens up Sabre II fraud-detection system. The Financial Services Authority (FSA) is updating its fraud-detection system with complex event processing (CEP) software to monitor transactions and detect insider trading and other market abuses as they occur. [Computer Weekly, June 22nd 2007] [Apama blog]

• CEP is also being promoted within retail systems, to protect against card fraud [SeeWhy]. Indeed, long before CEP became fashionable, Microsoft had an example of real-time fraud detection at Marks and Spencer, which I mentioned in my June 2003 article for CBDI on Service-Oriented Business Intelligence.

Fraud Detection in Insurance?

So why not insurance? I can think of several possible explanations for this. Firstly, insurance is not a particularly real-time business, and fraud detection within insurance does not need to be a real-time process. The insurance company generally takes the customer's money first, and then asks lots of questions before it pays any money back. If some elapsed time is needed to detect and investigate fraud, this may not be as critical in the insurance sector as it is in some other sectors. Secondly the identification of fraud in the insurance may be a matter of interpretation (semantics). Insurance experts worry about various types of moral hazard, and look for ways to reduce overall liability. Thirdly, insurance companies may wish to be secretive about how they define and detect fraud, to avoid giving ammunition to those they regard as potentially hostile.

Future opportunities for CEP in insurance

In the longer term, these explanations may well go away, as the insurance industry becomes more agile, open and collaborative. I think there are some really interesting opportunities for collaborative CEP in particular, but these will require a lot of work on the shared semantics and pragmatics of events.

Dynamic Pricing

In his marketing blog, Seth Godin asks about dynamic pricing.

Why doesn't fresh fish cost more than the same fish a day later? bowling a few cents less when it's not so crowded? movie tickets more on the day a movie debuts? why don't computers with a three-year obsolence cycle have predictable pricing that starts high and gets near cheap just before the new upgrades?

I first came upon this idea in Kevin Kelly's book New Rules for the New Economy (available online).

A head of lettuce today ... does not contain any financial information beyond a price sticker. Once applied, that price is fixed, too. It doesn’t change unless a human changes it. The economic consequences of lettuce sales elsewhere, or a change in the general global economy do not affect the head of lettuce itself. Instead, lettuce-related information flows through wholly separate channels—news programs or business newsletters—that are divorced from the lettuce itself. The lettuce is economically inert.

The realm of the animated is different. It’s vastly interconnected. In this coming world a head of lettuce carries its own identity and price, displayed perhaps on an LED slab nearby, or on a disposable chip attached to its stem. The price changes as the lettuce ages, as lettuce down the street is discounted, as the weather in California changes, as the dollar surges in relation to the Mexican peso. Traders back in supermarket headquarters manage the "yield" of lettuce prices using the same algorithms that airlines use to maximize their profits from airline seats. (An unsold seat on a 747 is as perishable as an unsold head of lettuce.)

[Source: Chapter 5]

Kelly describes this as animating the lettuce, following the principle of "Feed the Web First". (Presumably Webb's Wonder 2.0.)

This kind of animation calls for new and more complex kinds of system interconnection, and new and more complex kinds of commercial arrangement. Shifting the enterprise further into the real-time.

See also Dynamic Pricing Update (November 2023)