Organizations as Brains

This post is based on Chapter 3 of Gareth Morgan's classic book Images of Organization (Sage 1986), which opens with the following question: "Is it possible to design organizations so that they have the capacity to be as flexible, resilient, and inventive as the functioning of a brain?"

To start with, Morgan makes two important distinctions. The first distinction is between two different notions of rationality, and the second involves two contrasting uses of the "brain" metaphor.

Mechanistic or bureaucratic organizations rely on what Morgan (following Karl Mannheim) calls "instrumental rationality", where people are valued for their ability to fit in and contribute to the efficient operation of a predetermined structure. Morgan contrasts this with "substantial rationality", where elements of organization are able to question the appropriateness of what they are doing and to modify their action to take account of new situations. Morgan states that the human brain possesses higher degrees of substantial rationality than any man-made system. (pp78-79)

Morgan also observes a common trend to use the term "brain" metaphorically to refer to a centralized planning or management function within an organization, the brain "of" the firm. Instead, Morgan wants to talk about brain-like capabilities distributed throughout the organization, the brain "as" the firm. Using the brain metaphor in this way leads to two important ideas. Firstly, that organizations are information processing systems, potentially capable of learning to learn. And secondly, that organizations may be holographic systems, in the sense that any part represents and can stand in for the whole. (p 80)

The first of these two ideas, organizations as information processing systems, goes back to the work of James March and Herbert Simon in the 1940s and 1950s. Simon's theory of decision-making leads us to understand organizations as kinds of institutionalized brains that fragment, routinize and bound the decision-making process in order to make it manageable. (p 81) According to this theory, the  organization chart does not merely define a structure of work activity, it also creates a structure of attention, interpretation and decision-making. (p 81) Later organization design theorists such as Jay Galbraith showed how this kind of decision-making structure coped with uncertainty and information overload, either by reducing the need for information or by increasing the capacity to process information. (pp 82-83)

Nowadays, of course, much of this information processing capacity is provided by man-made systems. Writing in the mid 1980s, Morgan could already see the emergence of the virtual organization, embedded not in human activity but in computer networks. If it wasn't already, the organization-as-brain is now indisputably a sociotechnical system. The really big question, Morgan asks, is whether such organizations will also become more intelligent. (p84)

The problem here is that man-made systems (bureaucratic as well as automatic) tend towards instrumental rationality rather than substantial rationality. Such systems can produce goal-directed behaviour under four conditions. (p87)

1. The capacity to sense, monitor and scan significant aspects of their environment
2. The ability to relate this information to the operating norms that guide system behaviour
3. Ability to detect significant deviations from these norms
4. Ability to initiate corrective action when discrepancies are detected.

But this is merely single-loop learning, whereas true learning-to-learn calls for double-loop learning.  Morgan identifies three factors that inhibit double-loop learning. (pp89-90)

1. Division of responsibilities cause a fragmentation of knowledge and attention.
2. Bureaucratic accountability and asymmetric information produce ethical problems such as deception. (This is a form of the principal-agent problem.) 
3. Organizations also suffer from various forms of collective self-deception, resulting in a gap between "espoused theory" and "theory-in-use".

and he goes on to identify four design principles that may facilitate double-loop learning. (pp 91-95)

1. Encourage and value openness and reflectivity. Accept error and uncertainty.
2. Recognize the importance of exploring different viewpoints. 
3. Avoid imposing structures of action. Allow intelligence and direction to emerge.
4. Create organizational structures and principles that help implement these principles.

The flexible, self-organizing capacities of a brain depend on four further design principles, which help to instantiate the notion of the "holographic" organization. (pp 98-103)

1. Redundancy of function - each individual or team has a broader range of knowledge and skills than is required for the immediate task-at-hand, thus building flexibility into the organization.
2. Requisite variety - the internal diversity must match the challenges posed by the environment. All elements of an organization should embody critical dimensions of the environment.
3. Minimal critical specification - allow each system to find its own form.
4. Learning to learn - use autonomous intelligence and emergent connectivity to find novel and progressive solutions to complex problems.

In conclusion, innovative organizations must be designed as learning systems that place primary emphasis on being open to enquiry and self-criticism. The innovative attitudes and abilities of the whole must be enfolded in the parts. (p 105) Morgan identifies two major obstacles to implementing this ideal.

1. The realities of power and control. (p 108)
2. The inertia stemming from existing assumptions and beliefs. (p 109)

Morgan says he favours the brain metaphor because of the fundamental challenge it presents to the bureaucratic mode of organization. (pp 382-3) Writing in the mid 1980s, Morgan noted that computing facilities were often used to increase centralization, and to reinforce bureaucratic principles and top-down hierarchical control, and expressed a hope that this was a consequence of the limited vision of system designers rather than a necessary consequence of the new technologies. "The principles of cybernetics, organizational learning, and holographic self-organization provide valuable guidelines regarding the direction [technology] change might take." (p 108) A quarter of a century later, let's hope we're finally starting to move in the right direction.

Selling Business Architecture

#entarch @alexcullen asks "How Would You Sell Business Architecture To Your CEO?" Alex's own answer to this question is in the form of a 15-minute pitch, based on the following three points.

1. Your business is complex, and consistency is a challenge.
2. IT is an enabler of business strategy. The effects of inconsistent strategies and priorities are more visible in IT than anywhere else in the business.
3. A business capability map enables you to have strategic discussion on business priorities, and to coordinate business planning and execution.

I'm sorry, but that just sounds like IT making the usual excuses for poor delivery. It's not our fault that IT is a mess, it's because the business can't get its act together. And we think we can do a better job if we have a different set of models.

Frankly, if I had to sell business architecture to the CEO, I wouldn't want to do it from an IT perspective at all. My pitch would be based on the following three points.

1. Let me show you how the structural complexities in your business may critically affect business performance. (Of course, the details of this argument will look different for different businesses. See my post on Enterprise Structure.)
2. Let me show you how you can manage these structural complexities more effectively by thinking  architecturally about your business.
3. An explicit business architecture will help you coordinate XXX (specific forms of congruence and requisite variety) across all your human activity systems, including management information systems (IT) and management reward systems (HR), and thus overcome YYYY (the structural inhibitors to business performance).

I should stress that the abstract jargon in this argument serve as placeholders. In putting a customized version of this argument to a real CEO, I'd want to be in a position to talk about the concrete structural problems facing his/her organization, rather than appealing to some general theory of structure. And if I didn't have much idea what these concrete structural problems might be, I probably wouldn't have much luck persuading a hard-nosed CEO that I would be much use solving them.

After all, practical business arguments are rarely based on simplistic deduction ("All businesses need X, therefore your business needs X"), so it's not a good idea for enterprise architects to rely on it either. See my post on Architecture and Logic.

Enterprise Structure

#entarch There are some critical structural issues for business organizations, which have critical implications for IT architecture, including generating new requirements for management information systems and collaboration platforms. However, these structural issues arise not from technology alone but from the structural complexity of markets and the business environment, and from the need to manage a complex organization as a viable and dynamic sociotechnical system.

It is perhaps curious that enterprise architects, even those who insist that EA is not just about IT, should be so quick to frame all structural issues in terms of IT architecture. The industry analyst firm Forrester got some stick recently for a list of "Top Ten Issues" that were all technologically oriented.

To understand the complex structural issues of business organizations, we need to look beyond the methods and models derived from computer science and database theory, and draw from such disciplines as systems thinking and management cybernetics. Here are some of the (overlapping) structural themes I have been looking at.

• Asymmetrical demand 
• Business as a platform
• Coupling (the enterprise as a loosely coupled network of sociotechnical components and services - as in my book on the Component-Based Business)
• Differentiation and integration
• Edge organization
• Enterprise tempo (different activities with different characteristic tempi)
• Ethical dilemmas (conflicts of interest, moral hazard)
• Multi-sided markets (different stakeholder classes with complementary demands)
• Organizational intelligence
• Viability (command and control, VSM)

 
 It's not easy to get one's head around some of these structural themes in the abstract, so I'm going to try and produce a series of concrete vignettes showing how these themes play out in specific industry sectors.

Enterprise Tempo

An enterprise operates at several different tempi. For example

• A retail chain has one tempo aligned to the customer visiting the store, a longer tempo for purchasing and logistics, and a longer one still for planning and establishing new stores

• A military organization operates a campaign tempo (perhaps measured in months) and an acquisition tempo (possibly measured in decades). See my post on the Economics of Agility.

• An operational control room can produce instant responses (akin to reflex action), considered responses (requiring human intelligence) and long-term improvement. See my post Organizational Intelligence in the Control Room.

• A consultancy firm has a job tempo (solving specific problems for its clients) and a capability tempo (developing the knowledge, skills and practices to maintain its competitive edge).

Differences in tempo also exist between an enterprise and key external stakeholders. For example, the customers of a retail grocery chain may wish to eat three meals a day, but usually visit the store on a more infrequent tempo. Meanwhile, some of the suppliers may base their production on an annual harvest.

Such differences in tempo raise some important structural and economic issues.

• Alignment - how are operations coordinated across different tempi, and what are the costs of alignment? (In their paper Agility and Value for Defence, Nicholas Whittall and Philip Boxer consider this question in relation to the military scenario outline above.)

• Resource bargaining - how do we divide scarce assets (time, money, management attention, capacity for risk-taking and innovation) between different tempi?

• Intelligence - how are knowledge flows and learning loops affected by the different tempi?

In ecology, it is usually thought that the slow-moving processes dominate the faster-moving processes. This is one of the axioms of the architectural theory of pace layering.

 

See also Beyond Bimodal (May 2016)

Wisdom of Confucius

#entarch My friend @taotwit appeals to a saying of Confucius in relation to enterprise architecture.

What is necessary is to call things by their right names. ... If names be not correct, language is not in accordance with the truth of things. If language be not in accordance with the truth of things, affairs cannot be carried on to success. When affairs cannot be carried on to success, proprieties and harmony will not flourish. When proprieties and harmony do not flourish, punishments will not be properly awarded. When punishments are not properly awarded, the people do not know how to move hand or foot. Therefore a superior man considers it necessary that the names he uses may be spoken appropriately, and also that what he speaks may be carried out appropriately. What the superior man requires, is that in his words there may be nothing incorrect.” [Analects of Confucius]

Many enterprise architects will strongly agree with this thought, and may even wish to apply it to the question of Defining Enterprise Architecture itself. However, we should remember that Confucius was a highly conservative thinker, who believed in a fixed body of knowledge. (His comment about names was apparently provoked by his disapproval of the Duke of Wei, who had usurped his father's title.)

In Daoism, of course, a name can be named, (but) this is not the constant naming. Thus, whatever one may say about the Dao, cannot but fall short of reality [ReligionFacts]. The same is almost certainly true of enterprise archtecture, as well as the most central topics addressed by enterprise architecture practice.

However, I can wholeheartedly agree with Confucius when he said

“To rule a country, there must be reverent attention to business, and sincerity; economy in expenditure, and love for men; and the employment of the people at the proper seasons.”

For country, read enterprise.

Coherence Premium

Interesting article by Paul Leinwand and Cesare Mainardi, The Coherence Premium. HBR June 2010 http://www.booz.com/media/uploads/HBR_Coherence_Premium.pdf

The authors define the "coherent company" as one that "has aligned its differentiating internal capabilities with the right external market position". They claim that most companies lack coherence because they pay too much attention to the external and not enough to the internal. "We are suggesting that companies start from the opposite direction, figuring out what they're really good at and then developing those capabilities (three to six at most) until they're best in class and interlocking. From there, strategy becomes a matter of aligning that distinctive capabilities system with the right marketplace opportunities."


We may observe in passing that enterprise architects are often criticized for the opposite tendency - paying too much attention to internal structure without knowing how to align this to external demand. 



The authors offer two examples of enterprise models that could be expressed as an interconnected set of capabilities, although the detail of the interconnections are not described in the paper.

Retail (based on Wal-Mart)

• vendor management
• point-of-sale data analytics
• logistics
• working capital management

Consumer healthcare (based on Pfizer)

• Science-based innovation (I interpret this to be "technology" rather than "product")
• Influencing regulation and government policy
• New product development (including licensing and acquisition)
• Claims-based marketing (in other words, giving the consumer verifiable and relevant chunks of knowledge)
• Channel management
• Portfolio management

The authors claim to have a procedure for quantifying something they call "Coherence Premium" for a company.

1. Define the segments the company serves
2. Identify the capabilities that drive value for the company in each segment
3. Determine the number of common capabilities across all the segments the company serves.

They have calculated the coherence premium for a number of large companies including Campbell's, Clorox, Coca-Cola, ConAgra, General Mills, Heinz, Kimberly-Clark, Kraft, Nestlé, PepsiCo, Procter&Gamble, Sara Lee, Unilever and Wrigley's. These are then mapped against EBIT margin%, showing a very rough correlation.

Unfortunately, however, this graph doesn't show the two companies that they examined in greatest detail earlier in the paper, namely Wal-Mart and Pfizer. This may be because these companies face complex multi-sided markets, and this would cause difficulties for their simple procedure. It would seem that the kind of coherence they praised in these two companies would not be adequately represented by their simplistic "coherence premium" metric.


There have been many previous attempts to address the issues raised by Leinwand and Cesare Mainardi. In her blogpost Coherence - the new alignment, commenting on a Booz presentation of their concepts, Naomi Stanford recalls an earlier book on The Power of Alignment by George Labovitz and Victor Rosansky. Also see the discussion of cohesion costing in Nicholas Whittall and Philip Boxer, Agility and Value for Defence (pdf).

Brownfield Requirements Engineering

Some partial and selective notes from the RESG meeting on Brownfield Requirements Engineering, held at BCS headquarters on October 12th 2010. By Richard Veryard.

---

Chairman's Introduction

Ian Alexander opened the meeting by pointing out the gulf between the demands of real engineering projects (95% brownfield) and the body of knowledge about requirements engineering offered by academics and authors (95% greenfield), and sketching some of the characteristic features of brownfield requirements. The importance of this topic is demonstrated by an excellent turnout for this meeting.

Brownfield Systems Engineering

Ian Gallagher of Altran Praxis presented some experience from several large systems projects, both military and civilian. The typical scenario is a major upgrade of existing systems to improve performance and deal with obsolescence; the upgrade may therefore include different types of requirement

• doing new things
• doing old things in new ways
• migrating away from ageing technologies, proprietary systems or restricted technologies (e.g. those subject to export licenses)

The upgrade may be regarded as a single large increment (from AS-IS to TO-BE) but is often managed as a series of smaller increments.

With greenfield engineering, the requirements of the engineering project are pretty much the same as the requirements of the TO-BE system, but brownfield engineering introduces a critical choice: whether to write total requirements for the TO-BE system or to write requirements for the project (or separate increments) based on the difference(s) between AS-IS and TO-BE. IanG expressed a strong preference for the former, but acknowledged that this isn't always acceptable to key stakeholders, especially if the effort would be perceived as excessive. But in any case it's obviously important to be clear what kind of requirements we are talking about.

Beyond the scope of the TO-BE system, there may be a larger system-of-systems context, in which other equally large systems are the subject of equally large brownfield engineering activity. This raises questions of the interoperability of the increments, and the coordination between autonomous brownfield engineering projects, which is another important issue for brownfield requirements engineering.

A key issue is the pressure to reuse and repurpose existing assets. There are three possible reasons for this.

1. Rational economics - genuine cost-saving based on lifetime exploitation of assets
2. Irrational inertia - desire to justify past investment decisions, resistance to change
3. Commercial - vested interests from key stakeholders (such as contractors) to maintain proprietary dependencies

Requirements such as migrating towards more "open" architectures are therefore not only relevant for an immediate brownfield project but also have important implications for the economics of future brownfield projects.

Managing Brownfield Financial Software

Phil Cantor of Smartstream then talked about his past and present experience managing and maintaining software products in the financial sector. (Most of his examples came from previous companies he had worked for.) The requirements of such products don't come exclusively (or even primarily) from the "users" but from a body of knowledge that the package vendor is expected to master. Phil gave the example of an obscure piece of financial calculation that is now only required in the Philippines: users elsewhere in the world may not know about this, but a package that doesn't cater for this requirement will be unacceptable to a global bank.

The package vendor then has to balance a wishlist of enhancement requests and bug reports from hundreds of user organizations, with the practical constraints of maintaining - and hopefully improving the quality of - an existing body of software.

For me, one of the most interesting issues raised by Phil was when he talked about the platform and its requirements. There is an internal platform supporting the user-facing components of the product suite, containing common services and suchlike, and the requirements for this platform cannot be derived purely from the requirements of the package as a whole. Furthermore, the package as a whole serves as a platform for the business processes of the user organizations (Phil's customers), so the same question arises at that level as well. However, this is not particularly a brownfield issue.

Phil also mentioned the challenges of training. From a sociotechnical perspective this is a major brownfield issue, since the performance of the TO-BE system will be affected by the user habits and expectations (e.g. knowing where to find things) carried forward from the AS-IS system. Training is a solution (and often not a very good one) to some set of sociotechnical requirements, and there may be better ways of conveying a new set of habits and expectations to the users and technical staff, but clearly we need to have an understanding of these requirements as well.

Discussion

Lots of further issues came up in the discussion, and I didn't manage to capture half of them. Someone raised the question of scale - what do you do with a brownfield situation with a complex network of interdependent pieces of hardware and software, with the need for upgrades constrained by small budgets and massive complexity. This led to the question of timing - how do you decide whether to upgrade something now or to defer the upgrade until next year. (Nobody actually mentioned the concept of technical debt, but that is clearly relevant here.)


Finally, my memory and notes can be supplemented by a few choice tweets.

@rmonhem much debate over how much to document requirements with significant re-use of existing applications

@rmonhem most difficult challenges have often been posed by cultural, commercial and legal factors.

@jiludvik Bit too much focus on techniques documenting and signing off detailed reqs. This is not specific for brownfield projects!

@jiludvik Phil Cantor's pres has been very entertaining and spot on for brownfield req's eng. I can certainly relate to many challenges he mentioned

Unstructured knowledge

@SFDreverman asks (how) does unstructured knowledge hinder the collective intelligence of a group?

To start with, I am slightly puzzled by the word "hinder". Surely unstructured knowledge is usually better than no knowledge at all, although sometimes too much knowledge produces confusion and procrastination. But is structured knowledge always better than unstructured knowledge?

There are different kinds of structure that may be relevant here. Some kinds of knowledge may be formally codified and classified, especially for the purposes of storage in a library (which may be paper-based or electronic or both). But although this kind of structure may be useful for storage and retrieval, what is important for decision-making and learning (in other words, the processes contributing to collective intelligence) is a completely different kind of structure, which we might call its logical structure - the links and dependencies and entailments and contradictions between different fragments of knowledge and would-be knowledge.

Where does this logical structure come from? From another of the processes contributing to collective intelligence, namely sense-making. We make sense of knowledge by discovering (or creating) a logical structure for it. If we fail to find a logical structure for some random collection of would-be knowledge then that clearly limits our ability to do anything intelligent with it.

The discovery of a logical structure by a group of people depends in part on the intelligence and motivation of the group, and is not just an inherent quality of the knowledge itself. So if the knowledge remains unstructured, it may be because the group cannot or cannot be bothered to make sense of it. In which case it is the (insufficient) collective intelligence of the group that hinders the collective intelligence of the group.

The relationship between the storage structure and the logical structure remains problematic however. Library systems and knowledge management systems are generally not designed to reflect the logical structure of the knowledge contained within them, and this may make it hard to navigate the logical structure, or to retrieve knowledge according to the logical demands of the situation. So what we are talking about here is not unstructured knowledge but poorly and inappropriately structured knowledge. So the architecture and usability and effective use of knowledge management systems clearly have an impact on organizational intelligence.

Architecture and Logic

Jörgen Dahlberg suggests that "Enterprise Architects fail because of their reliance on logic as their single means of persuasion".

At a SCiO meeting in Manchester in 2010, Patrick Hoverstadt led an interesting discussion about logical levels, showing (among other things) how serious negotiation of critical issues often mixed logical levels. Sometimes you can "win" an argument by shifting to a higher logical level; but sometimes shifting to a lower logical level is the "winning" tactic.

There are several important points here. Firstly, "winning" doesn't necessarily mean winning. If you dig your heels in, you can win a local victory in a particular discussion, but you may damage your own longer-term interests. (Winning the battle but losing the war.) Secondly, shifting logical levels always has some emotional as well as logical content - it relieves some feelings as well as producing other feelings - for example, feelings of anger and frustration, especially in people who sense what is happening without being able to rationalize it.

@CarlChilley didn't find this surprising. "Decrease S/N ratio by changing the game and enter the emotional minefield that results." But what kind of people are going to win this kind of game? Presumably we need some form of emotional intelligence to successfully traverse an emotional minefield.

Just before writing this post, I happened to find my copy of Paul Watzlawick's book Ultrasolutions at the bottom of a box of old books. Watzlawick worked with Bateson on logical levels of communication, and includes some great examples of mixed logical levels in his book, including some heated exchanges between a fictional husband and wife.

Enterprise architects are typically adept at going up to a higher logical level (for example HOW - WHAT - WHY) regardless of whether this is appropriate in a particular context, and may not understand why this doesn't always automatically win the argument. Furthermore, if they lack emotional intelligence, they may fail to appreciate the emotional consequences of this tactic on the people they are hoping to influence.

So in what sense are enterprise architects good at logic?

Defining Enterprise Architecture

#entarch Another big discussion on Twitter yesterday about the correct definition (yawn) of Enterprise Architecture (EA). This one was whether EA should be defined as "Architecture of Enterprise" or "Architecture of Enterprise Technology". I mostly saw Tweets voting for the former, so it looked to me like a pretty one-sided discussion but maybe that's just because of the selection of people I follow on Twitter.

Clearly the difference between these two definitions is in the word "Technology". I wonder how many people who contributed to this discussion thought much about this word.

A narrow definition of "technology" from an IT perspective might limit the term to computer hardware and software; a broader definition might include everything from production technologies (e.g. Kanban) to accounting technologies (e.g. double-entry book-keeping), covering the strategic choices made in everything from transportation (oil tanker versus pipeline) to warfare (air bombardment versus ground forces). Followers of Lewis Mumford might define technology even more broadly (see for example his "Myth of the Machine") while followers of Bruno Latour would have a more subtle take on the whole subject. For my part, I have no wish to produce a single definition of technology; I merely wish to point out that the boundaries of "technology" may be as debatable as the boundaries of "enterprise architecture". The game of definition has no end.

But in any case, I wonder about the purpose and usefulness of this kind of definition. People often put an extraordinary amount of energy into definitions as if they thought that the simple act of defining something made it true. There are different forms of authority underpinning such definitions - for example the reputation of a well-known writer or organization, the emerging consensus of a group or community, or the negotiated standards of some industry body - and we may sometimes be able to achieve some kind of convergence as to what some term is supposed to mean.

But even if we could achieve a universal definition of what the term "enterprise architecture" is supposed to mean, that would be a pretty empty victory if the definition failed to reflect reality - either what enterprise architects actually do, or what they are capable of doing.

The reason I think this kind of definition can never satisfactorily reflect reality is that it is monothetic - in other words, it defines a concept in terms of specific features it must or mustn't have. Inspired by Wittgenstein, the anthropologist Rodney Needham introduced the concept of polythetic definition - defining a concept in terms of characteristic features it might have. Thus instead of debating endlessly whether enterprise architecture should be either A or B, and whether to adopt a narrow or broad definition of technology, we can start to make useful (and hopefully less dogmatic) statements about enterprise architecture and technology in the real world and the relationship between them.

---

Rodney Needham, Polythetic Classification: Convergence and Consequences (Man, 10:3, September 1975), pp. 349-369. 

Richard Veryard, Information Modelling - Practical Guidance (Prentice-Hall 1992) pp 99-100

Stanford Encyclopedia of Philosophy: Wittgenstein on Family Resemblance

Wikipedia: Family Resemblance

Intelligence as Requirement

Let's suppose an organization suffers from a number of the symptoms of organizational stupidity and decides to do something about it. (Not an easy decision, but let's start from there.) How do we determine the detailed requirements for organizational and technical systems change?

Many brownfield requirements analysis projects start with negative requirements - we want to eliminate this bottleneck, reduce this wastage or variation, take out this cost and risk. This involves a combination of factual observation (some bottleneck exists in the target system, with such-and-such consequences) and a value judgement (the bottleneck and its consequences are undesirable relative to some value system).

These negative requirements then need to be converted into positive requirements - we want to improve certain whole-system properties (such as organizational intelligence), and we think we can achieve that by making certain changes to the whole enterprise (regarded as a sociotechnical system). These changes may include engineering new or improved mechanisms, as well as improved use of the existing mechanisms.

Typically, there are many different things we could consider doing to improve these whole-system properties, so we produce a wishlist of potential requirements. But such wishlists create all kinds of difficulties. It may not be feasible to do all of the items on the wishlist, firstly because of limited resources and secondly because some of the items may be mutually incompatible or redundant. So we try to create a meaningful subset of the wishlist, to produce a set of real requirements that is both feasible and affordable.

If the items on the wishlist were all completely independent of one another, then we could categorize each item into subjective categories (say "essential" and "desirable"). Or perhaps we could perform a cost-benefit analysis on each item, calculating a risk-weighted accounting value of the item in terms of its ROI or NPV. This would allow us to rank the wishlist with the most important, valuable or critical requirements at the top; the final set of "requirements" is then defined as that subset of the possible requirements above some arbitrary cut-off line.

Of course both the subjective sorting of requirements ("essential" versus "desirable") and the estimation of costs, benefits and risks are evaluated differently from different stakeholder positions, so the final list is typically fought over between stakeholders. Furthermore, things change over the duration of an engineering project, for various reasons, so the cut-off line comes under pressure.

But a more fundamental problem with this way of defining requirements is that the items on the wishlist are usually mutually dependent, at least to some extent. The costs, benefits and risks of each item may vary according to which other items are chosen, and the number of possible permutations is astronomically large. Therefore the challenge of defining a meaningful and feasible subset of the original wishlist becomes an architectural one - finding some underlying deep structure within the wishlist that permits stakeholders to make intelligent and informed choices.

Can requirements be "solved"?

Much writing on systems architecture, requirements engineering and related disciplines starts from the premise of a distinction between problem-space and solution-space. This is an extremely useful distinction under certain conditions, but we are increasingly facing challenges in which these conditions no longer hold valid. Here are some examples.

1. Where the goal is not to design some complex artefact but to deliver some set of outcomes. For example, to make an organization more efficient or intelligent, to make a tax system more equitable, or to make a community more self-sufficient.

2. Where the target system is complex and governed by conflicting value systems, so that the challenge is to formulate and negotiate a series of interventions, from which a useful and acceptable change to the target system might emerge.

3. Where the agency for change is embedded within the target system, rather than being external. Anyone who gets involved in the project becomes part of the target system, and it no longer makes sense to ask the popular question "are you part of the problem or part of the solution?"

---

One of the attractions of the problem/solution distinction is that it helps to build agreement around a statement of the problem, and focuses the debate on different ideas and preferences for solving the problem. But to see the limitations of this distinction, we only have to look at the political sphere. There is surprisingly little difference between the political parties in their statement of the fundamental problems facing the government; but there are wide differences in the policy instruments they advocate to solve these problems. Of course, these differences are not just alternative sociotechnical designs, but are produced by different belief systems (how the economy really works, what are the root causes of crime and terrorism, and so on) as well as different value systems (what kinds of prosperity and well-being matter most).

So is there a discontinuity between highly complex and messy sociopolitical problems on the one hand and relatively simple and self-contained engineering problems on the other? To answer this kind of question, some people appeal to a simplistic interpretation of the Cynefin framework, treating it as if it were a simple 2x2 matrix mandating different problem-solving or problem-tackling methodologies according to the degree of complexity in the problem domain - but of course this kind of answer begs lots of questions about the relationship between problem and solution which a more subtle and sophisticated application of Cynefin should expose.

Any human system (including organizations) will display some degree of complexity and messiness, and it is in this context that the concept of "requirements" needs to be understood. In a separate post, I shall explore how such a concept of requirements applies to organizational intelligence.

---

Requirements engineering is about the interaction between five kinds of system. In the problem-solving paradigm, these fit together as follows.

• A deliverable system is designed to be a "solution" to some "problem". (In a simple case, this could be a single complex artefact or a coordinated series of interventions.)
• A target system is where the "problem" is located, and where the "solution" must fit. (In a simple case, this could be an organization unit or operational business process.)
• A value system provides the criteria justifying why the problem is a problem, and why the solution would count as an improvement. (In a simple case, there is a single sponsor whose value system is paramount.)
• A belief system, an explanatory model or theory that explains the cause of the problem and the plausibility of the solution. (In a simple case, everyone broadly shares the same belief system.)
• An intervention system, a force of people and resources mobilized to do something relevant. (In a simple world, the intervention system is a project team under the exclusive governance of a project sponsor with a unified and dominant value system.)

Within the problem-solving paradigm, the relationships between these five are pretty straightforward, and it may not be worth articulating and analysing them separately. However, as we move beyond the problem-solving paradigm, these become plural rather than singular, and the relationships between them become far more interesting.

---

See also

• Niklas Björnerstedt, The obsolete idea of requirements (Feb 2010)
• Richard Veryard, Zero-Based Requirements (Sept 2010)
• Richard Veryard, Third-Order Requirements Engineering (Slideshare presentation based on a paper I wrote in 1995 with John Dobson).

Single cause of failure

In May 2010, there was a mysterious plunge in US share prices. At the time an SEC investigation found that no single cause was to blame [BBC News 12 May 2010]. Following further investigation, the SEC now believes that the "flash crash" did indeed have a single cause, which the SEC has traced to an algorithmic trade of around $4 billion, executed by a single trader's computer program [BBC News 1 October 2010]. The authorities have since introduced "circuit breakers", which may help to mitigate the effects of such algorithmic trades in future, and there are hints that further measures could be on the way.

I haven't looked at the detail of these circuit breakers, but my architectural instincts make me uncomfortable with the idea of bolting on an additional mechanism into an already complicated system, in order to mitigate a single point of failure. Surely these circuit breakers will themselves be subject to perverse consequences, as well as being anticipated (and perhaps even deliberately triggered) by ever more sophisticated algorithms.

One of the key principles of distributed systems architecture is the avoidance of a single point of failure. Technical architects tend to focus on technical failure, although security experts often remind them of the equal dangers of socially engineered points of failure in technical systems. Meanwhile, enterprise architects need to pay attention to the possible failure modes of the business and its ecosystem, from a business and sociotechnical perspective.

In the case of the "flash crash", the key question for market regulators and market players is about the resilience and intelligence of the market in the face of certain classes of activity. Although the finger of blame is now pointing to a piece of software, the architectural question here is not software architecture but market architecture - regarding the market as a complex sociotechnical system in which pieces of software interact with other social and economic actors. Architects should beware of thinking that "single point of failure" is merely a technical question.

Value of argument

#entarch #JohnSeddon People sometimes describe the value of enterprise architecture (EA) in terms of the battles of enterprise architects against narrow or short-term thinking from various stakeholders. (See my post on the Future of Enterprise Architecture.) Describing the value of EA in these terms largely consists of anecdotes - either actual money that was supposedly saved thanks to the timely intervention of enterprise architects, or hypothetical money that might have been saved if only the enterprise architects had had more power and proximity. This gives us the notion of the EA-as-realist, who earns his/her keep largely by stopping ill-conceived initiatives, saying No to pushy vendors, and producing economies of governance. (See my post on EA Archetypes.)

More generally, people may describe the value of enterprise architecture in terms of things that supposedly only EA can do - things like vision or business structure. But there are many organizations whose leaders are perfectly comfortable and competent in these things - for example, I imagine it would take some nerve for an enterprise architect within Microsoft to try teach Ray Ozzie something about structuring a collaborative and innovative organization. (See my post Who Architects the Organization?)

Alternatively, enterprise architecture simply takes up a different position of vision or business structure, and its value depends on being able to mobilize a healthy and constructive argument. So maybe, this is just a thought, you could offer Ray Ozzie a "different perspective" on business structure and see how far that gets you.

The implication always is that EA isn't a viable system in its own right, it merely contributes to the viability of some larger system by correcting some classes of failure or lack. In other words, the value proposition for enterprise architecture is driven by what John Seddon calls "failure demand". According to a Seddonite view, enterprise architecture probably seems like a kind of back office function, and it would make a lot more sense just to reengineer the larger process/system so that it doesn't generate this kind of failure demand in the first place.

(That last paragraphy may cause a bit of cognitive dissonance with my Seddonite friends in the enterprise architecture communuity. I look forward to some healthy and constructive argument here.)

And if there is a maturity path for EA, as some EA experts suggest, then the mature state may be one where "enterprise architecture" is practised throughout the organization, rather than located in a separate organization unit whose viability and value seems to be so problematic. In other words, formal enterprise architecture will wither away. Yes?