Emergent Architecture

#entarch #emergence #systemsthinking What is emergent architecture, and what are the practical implications for enterprise architecture?

In August 2009, Gartner produced a definition of Emergent Architecture with two synonyms (Middle-Out EA and Light EA), two characteristic practices (modelling lines not boxes, modelling relationships as interactions) and seven characteristic properties (non-deterministic and decentralized; autonomous, rule-bound, goal-oriented and locally influenced actors; dynamic or adaptive systems; resource constraints) [Gartner Identifies New Approach for Enterprise Architecture, August 2009].

At the same time, Dion Hinchcliffe produced a similar list of properties and one of his trademark diagrams [Pragmatic new models for enterprise architecture take shape, August 2009]. Meanwhile Dion was also talking a lot about WOA (which he credits to Nick Gall of Gartner), and there was clearly a link in their thinking between WOA and some notion of emergence [A Web-Oriented Architecture (WOA) Un-Manifesto, December 2009].

Overall, the emergent and evolving definition of Emergent Architecture across the internet is pretty muddled: although other writers in the Enterprise Architecture world may reference Gartner and/or Hinchcliffe, they don't always pick up the full richness and power of their definitions. In addition, these writers may be influenced by the agile community, where emergent is contrasted with upfront and seems to mean something like making it up as you go along.

For example

The architect should collaborate with the development team to define and code higher-level contexts, responsibilities, interfaces, and interactions, as needed, and leave the details to the team. The development team, through the rigorous use of automated unit and story tests via continuous integration, is then able to improve the system design incrementally and continually—both within and across model-context boundaries— without compromising system functionality. Gartner uses the term Emergent Architecture to describe this practice. Keeping Architectures Relevant, Microsoft Architecture Journal

The practice described in that article can only be described as emergent on a fairly narrow interpretation of the word, presumably based on the rule-bound criterion and ignoring the other characteristics.

My own view is that all of the characteristics Gartner and Dion Hinchcliffe originally proposed (with the possible exception of resource constraints, which are nothing new) could be regarded as emerging, in the sense of being new and trendy and representing a departure from the modernist paradigm of traditional enterprise architecture. (See my post on Modernism and Enterprise Architecture.) But not all of the characteristics they proposed are directly associated with emergence, in the complex systems sense.

Wikipedia defines emergence as the way complex systems and patterns arise out of a multiplicity of relatively simple interactions [Wikipedia: Emergence]. As I see it, the notion of emergence leads to a key distinction for enterprise architecture, between a planned order (which Hayek called Taxis) and an emergent spontaneous order based on self-organization (which Hayek called Cosmos).

Although some writers like to use biological and ecological analogies when talking about emergence, the sociopolitical analogies are probably more relevant to enterprise architecture because they include the notion of human intentionality. (And some people use the terms top-down and bottom-up, but nowadays I try to avoid these terms because of their potential ambiguity. See my note on Service Planning.)

The distinction between planned and emergent architecture is akin to the distinction introduced by Henry Mintzberg between deliberate and emergent strategy, the latter referring to strategies that originate in the interaction of an organization with its environment [Wikipedia: Strategy dynamics].

This distinction also maps onto some recent work in the system-of-systems (SoS) domain. Mark Maier introduced a distinction between directed and collaborative systems (1998), and this has been developed by the U.S. Department of Defense (DoD) into a more complicated four-part schema (directed, acknowledged, collaborative and virtual). See for example System Engineering Guide for System-of-Systems Engineering (Version 1, August 2008). Planned architectures tend to assume directed or acknowledged systems-of-systems, while emergent architectures are associated with collaborative and virtual systems-of-systems.

Boxer and Garcia write:

In the complex systems-of-systems contexts supporting distributed collaboration, the architecture of the collaborative enterprise has to be approached as emergent, created through an alignment of individual architectures. [Enterprise Architecture for Complex System-of-Systems Contexts, 3rd Annual IEEE Systems Conference in Vancouver March 23-26 2009 (abstract) (pdf)] See also Type III Agility and Ideologies of Architecture.

The acknowledged category of systems of systems allows for some flexibility and autonomy for local development of component systems, while remaining under central oversight and direction. This is surely where rule-bound actors would fit, and would include the practices described in the Microsoft Architecture Journal article mentioned above (Keeping Architectures Relevant). There is also some scope in the acknowledged category for dynamic or adaptive systems and adaptive processes. But this is some way short of a genuinely emergent architecture.

Gartner is closer to the mark when it talks about decentralized decision-making (which it calls non-deterministic) involving autonomous goal-oriented actors, responsive to local influence. Similarly, Dion Hinchcliffe talks about community-driven architecture run by autonomous stakeholders, producing decentralized solutions with emergent outcomes.

I don't know whether Gartner is still promoting the idea of emergent architecture, or whether its definition has evolved since 2009, given that Nick Gall's recent work (such as his latest piece on Panarchy) doesn't seem to use the term at all. However, Gartner's original piece on emergent architecture remains available on its website, and continues to be referenced as one of the primary sources for the term.

---

What are the practical implications of emergence for enterprise architecture?

One of the most important practical differences between planned architecture and emergent architecture is that we tend to think of planned architecture as a design-time artefact - something that is envisioned, designed and then implemented by architects. So architecting is regarded as a special kind of designing. Implementation may be directly managed by the architects, or indirectly by means of a set of architectural policies to govern acquisition, development and use.

The implementation of a planned architecture always involves a gap between plan and reality. Plans take time to realise, some bits of the plan may never get realised. There is always a defacto architecture, which we may call architecture-in-use, which is a structural description of what-is-going-on, paying attention to certain structural and sociotechnical aspects of a (run-time) system of systems from a given observer position.

Devotees of planned architecture see this in terms of a simple transformation from AS-IS to TO-BE. Their attention to the existing defacto architecture is largely motivated by the need to define a business case and a technical transition strategy for replacing AS-IS with TO-BE, possibly in discrete phases or chunks.

But emergence tells us this: that the architecture-in-use emerges from a complex set of interactions between the efforts and intentions of many people. The architects cannot anticipate, let alone control, all of these interactions. There may be some areas in which the architects are able to carry out something that looks a bit like design, either autonomously or collaboratively, but there will be other areas in which the architects are simply trying to understand the structural implications of some messy situation and make some useful interventions. The primary activity of architects is therefore following not a design loop but an intelligence loop.

However that depends what we mean by design. There is a renewed interest in a generalized notion of design in the enterprise architecture world, especially with the current popularity of Design Thinking (and such derivatives as Hybrid Thinking).  But that's a subject for another post.

---

Sources

Gartner identifies new approach for enterprise architecture (Aug 2009)

Commentary on Gartner's emergent architecture concept by Abel Avram, Leo de Sousa, Adrian Grigoriu and Mike Rollings (then with Burton Group).

Dion Hinchcliffe, Pragmatic new models for enterprise architecture (Aug 2009)

Mark Maier, Architecting Principles for Systems of Systems (InfoEd June 1997)

See also

Peter Cripps, Enterprise Architecture is Dead. Long Live Interprise Architecture (Oct 2010)

Tom Graves, Hybrid-thinking, enterprise architecture and the US Army (May 2010) 

Three or Four Schools of Enterprise Architecture (December 2011)

Architecture and Logic

Jörgen Dahlberg suggests that "Enterprise Architects fail because of their reliance on logic as their single means of persuasion".

At a SCiO meeting in Manchester in 2010, Patrick Hoverstadt led an interesting discussion about logical levels, showing (among other things) how serious negotiation of critical issues often mixed logical levels. Sometimes you can "win" an argument by shifting to a higher logical level; but sometimes shifting to a lower logical level is the "winning" tactic.

There are several important points here. Firstly, "winning" doesn't necessarily mean winning. If you dig your heels in, you can win a local victory in a particular discussion, but you may damage your own longer-term interests. (Winning the battle but losing the war.) Secondly, shifting logical levels always has some emotional as well as logical content - it relieves some feelings as well as producing other feelings - for example, feelings of anger and frustration, especially in people who sense what is happening without being able to rationalize it.

@CarlChilley didn't find this surprising. "Decrease S/N ratio by changing the game and enter the emotional minefield that results." But what kind of people are going to win this kind of game? Presumably we need some form of emotional intelligence to successfully traverse an emotional minefield.

Just before writing this post, I happened to find my copy of Paul Watzlawick's book Ultrasolutions at the bottom of a box of old books. Watzlawick worked with Bateson on logical levels of communication, and includes some great examples of mixed logical levels in his book, including some heated exchanges between a fictional husband and wife.

Enterprise architects are typically adept at going up to a higher logical level (for example HOW - WHAT - WHY) regardless of whether this is appropriate in a particular context, and may not understand why this doesn't always automatically win the argument. Furthermore, if they lack emotional intelligence, they may fail to appreciate the emotional consequences of this tactic on the people they are hoping to influence.

So in what sense are enterprise architects good at logic?

SPARK workshop 3

Some more observations from the SPARK workshop ...

Network of centres

I really liked this diagram, which was produced by one of the other groups. I think it echoes some of the ideas of Christopher Alexander about order emerging from the progressive differentiation and completion of a network of centres. The diagram shows a network of services, each drawn as a hub surrounded by some critical Web 2.0 stuff: community, rich content, trust requirements, dynamic mediation. There are centripetal and centrifugal forces, which represent business advantage (outwards) and business constraint/cost (inwards).

Emergence

Nick Gall (Gartner) suggested a simple economic model of emergence, based on benefit to the direct and indirect user, as well as to the provider. This model could help to add richness to the notions of centripetal and centrifugal forces shown in the previous diagram. This reminded me of another model of emergence by Kevin Kelly, which he called Nine Laws of God, included in his great book Out of Control. Kelly's model includes many of the themes that are relevant to innovation in Web 2.0.

Distribute being, Control from the bottom up, Cultivate increasing returns, Grow by chunking, Maximize the fringes, Honor your errors, Pursue no optima; have multiple goals, Seek persistent disequilibrium, Change changes itself.

Emergent Dysfunction

Nick Malik asks an interesting question in his blog: Does SOA create a new class of defect: passive-aggressive behavior?

Something like this is certainly possible at the business level. Badly designed service-oriented relationships can produce frustration, anger and alienation, both for customers and for employees. In this weekend's Observer, Simon Caulkin has an excellent analysis of how this operates in off-shore (and for that matter on-shore) call centre operations: You call this best practice? (Observer, June 5th, 2005).

A simplistic analysis is that people in the US and UK are angry because jobs have been exported to India, and they take out their frustrations on the call centre staff. But this is confusing cause and effect. Customer service has been exported to India by companies that don't really care a fig about authentic customer service, and don't regard it as core to their business. No wonder customers are angry.

Nick continues: Can two systems behave in a manner that is counterproductive to both, but makes both of them look effective from the outside?

Quoting Professor Harry Scarbrough, Simon Caulkin says that given the low-skill, low-value of the jobs created in India on one side and the destructive hidden costs for customers and companies on the other, offshored call centres may be a remarkable example of an international exchange, freely entered into, that benefits neither party: 'That's quite rare.'

Technorati Tags: call+centre offshore organizational+behaviour SOA service-oriented

Emergent Intelligence

I have long regarded intelligence as an emergent property of a whole system rather than an attribute of a single component.

Two items crossed my screen this week. Firstly a debate between David Kennedy (Eurescom) and Martin Geddes (Telepocalypse). In a piece entitled It might be big but it will never be dumb, Kennedy argues that telecoms networks must contain a lot of intelligence. "A critical step for the immediate introduction of NGN services will be an open discussion on how much intelligence will actually be in the network – a big clever network – and where it should be located." In his response, entitled Welcome to Planet Zog, Geddes sarcastically points out the flaw in this argument - what if everyone will be using Skype instead?

Meanwhile, a number of bloggers (Bill, Sean, Stephen) have pointed at a good piece called Insects and Entropy. (Actually, I think entropy is the wrong word, but never mind.) This is an Aesop fable for our time, involving a complex hare and a simple tortoise - or is it the other way around? Complex and powerful behaviours can be produced by composing simple behaviours.

So the big clever network needs a big load of intelligence does it? Fine, but don't assume you can locate a big clever component that is providing the intelligence.

Architecture and Abstraction

SOA calls upon designers and architects to operate at a higher level of abstraction.

One mode of abstraction commonly associated with information architects is generalization - they are often thought to operate mainly with generalized business objects/processes, such as CUSTOMER and SALES.

But this is actually not where the real architectural challenges lie.

Instead, architects need to reason explicitly about system structure and dynamics - cohesion and coupling, composition and decomposition, change and emergence. They need to understand granularity and stratification as active choices, rather than text-book patterns.

But the generally available architectural practices and modelling notations simply do not support these challenges. How are architects supposed to pay attention to such concerns as coupling and flexibility, when they are working with models that don’t make these concerns visible, and with practices that don’t support reasoning about these concerns?

We are developing tools and techniques to support this reasoning, and the development of architectural knowledge and know-how.