SOA and Holism

Wikipedia's definition of holism traces back to Jan Smuts

Holism ... is the idea that all the properties of a given system (biological, chemical, social, economic, mental, linguistic, etc.) cannot be determined or explained by its component parts alone. Instead, the system as a whole determines in an important way how the parts behave.

To what extent does this concept apply to SOA? My own view is that SOA needs to be understood from the Systems-of-Systems Engineering paradigm rather than from the Systems Engineering or Software Engineering paradigm. This helps us to deal with a range of system level phenomena including Feature Interaction.

In my writings I've drawn on the recent work of Christopher Alexander, from A New Theory of Urban Design to The Nature of Order, where Alexander talks about something he calls structure-preserving transformations.

According to the New Theory (or at least my interpretation of it, which I call Organic Planning) each act of transformation should be a step within a larger and open-ended evolutionary development, and should have three aspects.

1. Produce something at some level
2. Complete something (larger) that was already part-developed - typically by linking smaller (lower-level) things and peer (same-level) things that already existed, or were created in previous steps.
3. Create new opportunities - Alexander calls this hinting-at.

For example, an information service called Product Catalogue might (1) establish a single point for accessing product information, (2) linking together data from multiple legacy data stores and third-party feeds, while (3) hinting towards a new business process (yet to be fully specified) in which the product catalogue becomes a dynamic object, using some kind of real-time business intelligence.

I published a very simplified version of this in the CBDI Journal in 2004, suggesting that the service designer needed to look in four directions (upwards, downwards, sideways, inwards). I understand that this was picked up and referenced by the ArchiMate people, for example in a 2005 book called Enterprise Architecture at Work, and it has been implemented in the Telelogic tool. My colleague Tony Bidgood has recently published an article on ArchiMate, with some further examples.

• Richard Veryard, Business Driven SOA (CBDI Journal, May 2004), Business Driven SOA Governance (CBDI Journal, June 2004)
• Tony Bidgood, Archimate Standard for Enterprise Service Modelling (CBDI Journal, August 2008)

Here's how I intended the four directions to map to the New Theory of Organic Planning

1. Inwards: Functional correctness
2a. Downwards: Integrating and composing smaller stuff
2b. Sideways: Interoperability
3. Upwards: Larger whole

Organic Planning is described in my 2001 book on the Component-Based Business, and there is a short version on my website, but I didn't make this explicit in my 2004 article.

My expectation has always been that a series of transformations (whether structure-preserving or otherwise) would be expressed as a series of model pairs, in which the nth TO-BE becomes the (n+1)th AS-IS. But some alternative modelling notations (such as Michael Bell's SOMF) allow both AS-IS and TO-BE to be expressed in a single model, so it would be very interesting to see how a series of transformations could be expressed and analyzed.

Organic Planning from Richard Veryard

 

 See also SOA and Holism 2 (February 2009)

Efficiency and Robustness - On Central Planning

The debate between Ian Welsh and Chandler Howell has continued since my previous post on Efficiency and Robustness, and the topic of Central Planning has appeared. In this post, I want to set aside the ideological aspects of central planning and discuss some of the practical aspects.

Central Planning - The Ideology

The question of central planning is widely seen as a political one, but it certainly isn't a simple matter of left and right. There are many right-wing libertarians who reject central planning as equivalent to Stalinism, the worst excesses of the Soviet experiment. (See this article Central Planning is Spontaneous Economic Order whose author complains that the Santa Fe Institute has betrayed the principles of complexity.) And there are left-wing libertarians who are puzzled or aghast at the centralizing tendencies of some of the right-wing authoritarians as well as corporate commercial interests.

Intelligent Design is of course a form of central planning - but executed by a perfect being rather than imperfect humans. (Boston Globe, via Snowdeal.) Robin Wilton asks whether it is possible for an atheist to believe in Intelligent Design. Well, it is certainly possible for atheists to believe in central planning. There are science fiction worlds in which central planning is carried out by a supreme computer, sometimes known as Multivac (or perhaps Spaghetti Monster). Science fiction writers often use technology as an oblique way of discussing serious philosophical and moral questions.

But if we put the politics and religion on one side, there is clearly a great deal of imperfect central planning that goes on within large organizations. We should be able to discuss this in pragmatic terms rather than ideological ones.

Central Planning in Practice

In a comment to Chandler Howell's previous post, Stu Berman advocates "horizontal integration rather than vertical" and rejects central planning. Chandler replies that, "Central Planning is the best analogy I have ever seen for how large corporations are run today. The only difference [with the Soviet Union] is that the corporations have a lot better computing power to manage their logistics." Chandler goes on to describe the fragility of systems (including large organizations) based on central planning, and suggests that H5N1 (aka Bird Flu) could have a more devastating effect than Hurricanes Katrina and Rita.

Chandler goes on to talk about coordination (what the military call C3I) in a crisis management context. "The disaster in Katrina may have started with the levees breaching, but lack of coordination at any level is what kept it going for days."

Discussion of central planning is relevant not just to the destruction of New Orleans, but to its subsequent reconstruction. The popular alternatives to central planning come under a range of labels including clusters (SwampFox) dynamic improvisation (Arnold Kling via OceanStateBlogger), organic growth (MCB) and private initiative (CapitalFreedom). These alternatives certainly don't all come from the same end of the political spectrum.

Advocacy for central planning also comes from various quarters, including Environmental Planning (WHY, WHAT, HOW and FOR WHOM).

In the real world, we are not likely to have either total central planning or total anarchy, but a complicated mixture of the two. To make a productive contribution to this debate, we have to be able to reason intelligently about outcomes (evidence-based policy) rather than ideological principles. (MCB also makes this point.) Furthermore, we have to connect outcomes with a specific context. (Not generic outcomes that would be exactly the same for New Delhi, New Orleans and New York.)

Interoperability

Central Planning is an attempt to produce all decisions from a single directing mind. To this extent that this attempt is successful, it puts the emphasis on endo-interoperability - coordination within the scope of a single plan. In contrast, exo-interoperability involves dynamic collaboration between autonomous agencies, whose plans may be formulated in entirely different terms.

Chander Howell, Surge Protectors (21 September 2005), Surge Protectors Part 2 (25 September 2005)

Ian Welsh, The Economics of a Flu Pandemic II (24 August 2005)

Related posts on Efficiency and Robustness

Technology Change and Business Change

The resemblance between technology and magic has been remarked by a number of writers, and provides a series of running jokes in the Harry Potter books. (Wizards cannot cope with the telephone, but communicate instead using owls and fireplaces.) For those who believe in technology as some kind of magic, the business benefits or social evils of a new technology appear automatically as soon as a new device is plugged in. The box in the corner of your office will magically cause your markets and profits to multiply, your customers to be eager and loyal, your employees to be happy and productive, your suppliers to be efficient and cheap, and your competitors to vanish in a puff of smoke.

Realistically, the connection between Web Services and the service-based business is neither necessary nor sufficient. It is theoretically possible to achieve the service-based business without using any technology at all - although in practice this option is likely to be uneconomic or otherwise unattractive. It is also possible to use the technology fully at the system level, without taking any steps towards the service-based business at the business level.

The relationship between technology and business is more subtle than this, but still important. Consider this: telephone, fax and email doesn't stop you writing old-fashioned letters to your friends - but as it happens, most people nowadays do most of their personal communication using telephone and email. Technology makes certain choices easier, and thus certain patterns of business and personal interaction are favored by technology, while others fall into relative disuse.

Incremental change can be very difficult to manage, especially when so many people are driven by short-term results. Large numbers of small steps may be taken, each one meaningful in isolation - but adding up to gross confusion and ultimately failing to meet the business requirements. In industry, longer-term strategic investment in organizational capability and learning may (at least theoretically, occasionally and approximately) be converted into tangible value by being reflected in the share price. In Government, there is no directly equivalent measure of tangible value, and the benefits of organizational learning can only be experienced indirectly, for example from the enhanced ability to develop and implement future policy.

But here is where the intelligent service-based approach potentially scores - by allowing large complex organizations to deliver short-term results while paying attention to longer-term issues of organizational adaptability and improvement. It supports organic planning, in which large-scale coherence can be developed without large-scale cost and disruption. While ill-considered and rigid service-based structures frequently lead to enormous problems, proper attention to flexible service-based structure can yield significant short-term and longer-term benefits.