Complex Event Processing

Lots of people are talking about Complex Event Processing (CEP). As Opher Etzion points out, it's not always clear whether this means "Processing of Complex Events", or "Complex Processing of Events", or both.

Processing of Complex Events:

According to Wikipedia, the goal of CEP is to identify meaningful events from an event cloud. I interpret this as detecting complex events from a mass of atomic events.

Complex Processing of Events:

But in some of the examples cited by CEP experts, the events themselves don't seem to be particularly complex. The complexity comes from the need to mobilize an efficient and effective response to a large number of relatively simple events.

• Tiffin Box Delivery (Paul Vincent)

... which means what we are really talking about is ...

Complex Systems:

A complex system is buffeted by events from the environment, and must respond in efficient, effective and flexible ways to these events. This requirement is addressed by a considerable body of work on complex systems and cybernetics - including Stafford Beer's Viable Systems Model (VSM), which I believe to be of particular relevance to CEP - but there is surprisingly little reference to this work in the CEP field. (Searching the Internet for links between CEP and VSM, I only find one person aside from myself who has made this connection - one David HC Soul, who has posted a number of cybernetics reading lists and other material to Squidoo and elsewhere.)

If you just read the Wikipedia articles on CEP and VSM, you may not see much connection between them, so let me try and explain. In my view, the most useful contribution from VSM to CEP is the notion of a transducer. VSM describes a system of systems, with messages or signals passed between the systems. Some of the systems are relatively simple, while others are more complex, handling higher levels of variety. So you need some way of producing a high-variety stream of events from a low-variety stream of events - this is known as an amplifier. And you need some way of converting a high-variety stream of events to a low-variety stream of events - this is known as an attenuator. Both the amplifier and the attenuator convert a stream of events from one form to another - the general word for such a mechanism is transducer.

(In event-driven SOA, events are transmitted through message-oriented middleware or ESB. Some transduction can usually be performed directly by this platform; more complex transduction can be implemented by utility services sitting on the platform.)

Stafford Beer then proposes an elaborate architecture for distributing management functions (such as coordination and planning) between the systems. I don't think we necessarily have to adopt this architecture exactly as Beer lays it out, but we should certainly adopt some of Beer's architectural principles.

Notes:

Wikipedia: Complex Event Processing, Event Stream Processing, Event-Driven Architecture, Viable System Model