Clouds and Clocks 4

An interesting debate on SaaS and Control

• Cloudy Future for the Enterprise and most likely for ISVs too (Gianpaolo Carraro, Microsoft)
• Tradeoffs in SaaS (Charlie Bass, EDS)

Gianpaolo starts out by suggesting a simple trade-off between user control and the economics of scale. With SaaS, the user forgoes some control, in order to benefit from (some share of) the economics of scale achieved by the service provider. Gianpaolo divides control into two aspects: Control of Features and Control of SLA.

Charlie thinks it comes down to a perspective of what is "control". He suggests a different notion of control, whereby a contract with a service provider would give you more legal control than doing the work in-house.

But as far as I can see, both of them are talking about Command (setting the target conditions) rather than Control (regulating the system to satisfy the target conditions). See Alberts and Hayes, Understanding Command and Control (pdf).

Gianpaolo associates the Control of Features with who builds the service (Implementation) and the Control of SLA with who runs the service (Deployment). (This is a bit of a simplification; I shall ignore that for the moment.) But in Service-Oriented Architecture, there is a third and perhaps the most important role: who specifies the service. It is possible for the user (or a community of users, or an independent regulatory body) to provide the specification, and for the provider merely to deliver an acceptable implementation and deployment. Returning to the distinction between Command and Control, we may associate the Command of both Features and SLA with who specifies the service.

But what kind of service are we talking about here? The economics of scale are most obviously associated with simple one-size-fits-all services that are symmetric and replicable - what Philip Boxer calls r-type services. In most cases such services are unilaterally specified by the service provider, offer little if any contextual variety to the service user, and control tends to focus on quality control and cost control.

With more complex types of service, the service user typically demands a much greater fit to a specific use-context, and the question of control expands to include requisite variety and the economics of governance. But does this mean that the economics of scale go out of the window? The challenge for very large and complex service-based systems is to combine the economics of scale AND scope AND governance.

Clouds and Clocks 3

Some CEP vendors (Aleri, Coral8, RuleCore) are boasting that their CEP software is "deterministic" or "consistent".

In other words, "clockwork" rather than "cloudlike". Mark Tsimelzon, President & CTO of Coral8, insists that we shouldn't think of an event cloud at all - he prefers the concept of event stream.

But what do "determinism" and "consistency" actually mean here, and why are they important?

Both Aleri and Coral8 define “determinacy” to mean that a set of input data produces the same output data.

Aleri adds that the order in which the data arrives doesn’t matter. This only follows from the definition of determinism if we assume that "set" means a pure set, without sequence. Perhaps a better word for this property is "commutativity". There is also the related property of idempotence, in which it doesn't matter if the same event or message appears more than once. Aleri also defines “consistency” to mean that the output is predictable from the input - which sounds equivalent to their definition of "determinacy" - but they go on to say that consistency means that "the results are the ones we’d expect" - which sounds more like correctness to me.

Meanwhile, Mark of Coral8 introduces three degrees of determinism: non-deterministic, single-stream and multiple-stream determinism. He hopes his post has demystified the notion of determinism a little. Er, thanks Mark.

But why does determinism matter? Mark explains that this has to do with testing, especially regression testing. If we cannot compare results from two runs of the same stream of events, then our traditional approach to software testing breaks down. At least, that's the only reason he mentions in his blog.

This is rather like the value of the controlled experiment in science. A laboratory provides a controlled environment, in which most of the variables can be fixed, so that the cause-and-effect can be isolated. Software testing also requires a controlled environment, so that any unexpected variation in results can be reliably traced to a specific cause - such as a software bug.

But complex systems are not particularly amenable to laboratory experiments, because the requisite complexity cannot be properly contained and controlled. And complex internet systems raise a similar challenge.

Determinism may be useful from a software engineering perspective, but it seems to deny some of the technological potential of complex event processing, including machine learning. Surely the whole point of machine learning is that the system doesn't always produce the same results on the same input, but is capable of producing improved results on the same input.

So I'd like to see a wider debate about the limits of determinism. I've been talking about this for a while in an SOA context - see my earlier post on Determinism.

---

Consistency and Determinacy (Aleri March 2008)

Marco Seiriö, Context Management (RuleCore June 2009)

Mark Tsimelzon, Determinism in CEP (Coral8 October 2007)

Mark Tsimelzon, Unclouding and streamlining your thinking about CEP use cases (Coral8 October 2007)


Links via WayBack Machine. Updated 2 December 2014.

Clouds and Clocks 2

Pat Helland regrets ...

Pat Helland has resumed blogging, having recently returned to Microsoft from a sojourn at Amazon. In his latest post SOA and Newton's Universe, he renounces the quasi-Newtonian paradigm of distributed systems to which he adhered for most of his 30-year career, and outlines an alternative paradigm with some resemblance to the Special Theory of Relativity.

The Newtonian paradigm of distributed systems is that we are trying to make many systems appear as the One System. This paradigm may be linked with the idea of the Global Schema or Universal Ontology. Helland contrasts this with an alternative paradigm of distributed systems, in which the systems are entirely dependent on the point of view of the observer, and there is no Universal Ontology. (I'd have wanted to use the term relativistic semantics here, but it has already been bagsied for academic linguistics - see for example Catfood and Buses.)

Helland sees this in terms of a relaxation of consistency. I disagree. Distributed systems-of-systems (including SOA) must follow a consistent logic - but not necessarily a traditional two-valued logic. Flexibility comes from being underdetermined (clouds) rather than overdetermined (clocks).

See my earlier posts Beyond Binary Logic and On Clouds and Clocks. See also Philip Boxer on Modelling Structure-Determining Processes.

Clouds and Clocks

I've looked at clouds from both sides now ...

• Todd Biske: In the Clouds
• Dion Hinchcliffe: SOA will begin to open up more to the Internet cloud
• Joe McKendrick: SOA reaches out to the cloud - will business follow?
• Mark O'Neill: When Enterprise SOAs meet the cloud
• Phil Wainewright: The Internet cloud is one great global SOA

The classic distinction is between clouds and clocks. Clocks follow precise rules, they are under "iron control". Clouds are more complex, and are under "plastic control".

In SOA terms, the term "cloud" is generally used to describe what happens outside the enterprise boundaries, and outside the corporate firewall. SOA is commonly expected to start in areas that are under greater control, and then spread to more complex, less well-controlled areas. Numerous pundits are now predicting this trend for the current year.

But the contrast between clouds and clocks may be a misleading one. In physics, one of the central questions of the twentieth century was whether atoms behaved like clocks (predictable mechanisms) or like clouds (statistical mechanisms). In 1965, Karl Popper gave a lecture called "On Clouds and Clocks", in which he contrasted the assumption of classical atomism (even clouds are composed of tiny clock-like bits) with the modern alternative (even clock-like structures are composed of tiny cloud-like bits).

We can ask a similar question for SOA. Is there really such a strong distinction between (tightly controlled) behaviours inside the enterprise and (loosely controlled or uncontrolled) behaviours outside the enterprise? Or is the cloud (cloudiness) fractal?

Much of my consultancy time is spent with organizations that are just too large and complex to be able to draw simple boundaries. Especially in the public sector and defence sector. As soon as you get past tightly controlled pilot projects, everything typically becomes more cloud-like.

Obituary of Karl Popper, 1902–1994, by John Watkins
Printed in Proceedings of the British Academy, Volume 94, pp. 645–684. 1997

Technorati Tags: SOA service-oriented