Wednesday, September 29, 2010

Value of models 2

#entarch What are enterprise models for? What is their purpose, status, perspective, scope, meaning?

Even the scope isn't as straightforward as you might think. Okay, the enterprise model is usually supposed to be "enterprise-wide" - but how wide is that exactly? Do customers and supply chain partners belong to the enterprise for modelling purposes, or not? (Don't tell me you can resolve such questions by looking at the organization chart, because that's just another enterprise model, so you're just shifting the problem.) And even if we know how wide, we still need to know how deep. For some purpose, from some perspective.

What about the status of an enterprise model - in other words, what kind of knowledge it claims to represent? Is it supposed to be a simplified description of the current state of the enterprise (AS-IS)? Or an idealized blueprint of some future state of the enterprise (TO-BE)?

Enterprise architects typically produce both AS-IS and TO-BE models, and then produce transition plans to get from AS-IS to TO-BE. This is therefore an exercise in requirements engineering at the enterprise level. The difference between AS-IS and TO-BE can be analysed from many perspectives including engineering (technical design and specification), accounting (cost estimation and financial justification) and sociotechnical (human systems management, sometimes disdainfully referred to as "implementation factors").

In the past I have talked a lot about this process, and I may have created the false impression that the sole purpose for modelling is to support this process.

But enterprise models have another very important purpose. There is a collective management need to make sense of the complexities of the business, and so an enterprise model is a sense-making tool as well as an engineering tool.

For example, an enterprise model might give us a picture of an organization as a set of conversations and collaborations, thus helping the participants in these conversations and collaborations to make them more systematic and focused,

Looking at this model from an engineering perspective, we might identify opportunities to install more sophisticated mechanisms if that's justified, but often it's about making better use of the existing mechanisms.

If the enterprise architect always looks at the enterprise and its models from an engineering perspective, then "better use of the existing mechanisms" may not seem very important.

But I see this as part of the job of enterprise architect – not just fixing the mechanisms but understanding and advising on their effective use. The enterprise architect must therefore be capable of looking at things from a sociotechnical or business perspective, not just an engineering perspective.

If our purpose here goes beyond conventional EA practice, then the requisite scope and focus of the enterprise models changes as well. Conventional enterprise models are dominated by the operational processes, with management information typically regarded as mere superstructure. If we want to improve the manageability and intelligence of the whole organization as a complex sociotechnical system of systems, we need to be interested in the management system - the information flows and coordination mechanisms and feedback loops - not just the operational system.

An enterprise model is therefore a map to support this kind of investigation, as well as a sense-making map to be used within the management system itself, not just a blueprint for reengineering the enterprise.


Thanks to Sally Bean and Jon Durrant for useful discussion.
See also Value of Models 1.

Tuesday, September 28, 2010

Who architects the organization?

#entarch @nickmalik challenged my post on The Future of Enterprise Architecture and referred to his earlier blogpost on The Mission, Capabilities, and Business Output of Enterprise Architecture.

According to Nick, the mission of EA involves strong influence over the structure of the business organization. Although he makes it clear that he is speaking generically and not about the mission of enterprise architecture within his own employer, Microsoft, he does claim to base his account of the mission of enterprise architecture on his own experience, and so it seems reasonable for us to ask how much influence EA has over Microsoft's organizational structure.

The first point to note is the critical relationship between Microsoft's organization structure and its product structure. (Popular mythology attributes these aspects of Microsoft's business not to enterprise architects but to the Chief Software Architect - Bill Gates and his successor Ray Ozzie.) The degree to which the organization units responsible for different product lines (say Windows and Office) are operated autonomously, and the amount of coordination and knowledge flows that pass between these units - these are surely critical structural elements of the Microsoft organization, to which an enterprise architecture should pay close attention. The organization structure affects product quality, above all interoperability.

Furthermore, the actual coordination and knowledge flows are not determined by the formal organization structure (org chart) but on the de facto structure. Enterprise architects who wish to monitor and control defacto organization structure will need to look at things like email traffic, not just formal information systems.

Microsoft aside, I wonder how many organizations genuinely allow their enterprise architects anywhere near such structural decisions


Nachiappan Nagappan, Brendan Murphy, and Victor Basili, The Influence of Organizational Structure On Software Quality: An Empirical Case Study. Microsoft Research, January 2008

Ding Zhou, Yang Song, and Hongyuan Zha, Towards Discovering Organizational Structure from Email Corpus (pdf). Microsoft Research, 2005.

Friday, September 17, 2010

Chinese Walls

In Joined up Daily Mail, @psbook via @chris_yapp points out a contradiction between the front page (attacking @stephenfry) and an advertisement on page 27 (featuring @stephenfry). "Perhaps the Daily Mail should try a little harder not to offend their advertisers?"

The Daily Mail is not my favourite newspaper (as readers of my posts on the POSIWID blog will know), but with this news my respect for the Daily Mail has gone up a notch - at least they aren't allowing their advertisers to influence the front page.

What's this got to do with architecture? We have here an example of a clash between economics (which apparently favours joined-up thinking) and ethics (which in this case apparently favours the opposite), typically resolved by the erection of an intra-organizational boundary known as a Chinese Wall. This is a structure within an enterprise intended to reduce conflicts of interest, asymmetrical information and moral hazard.

For example, financial institutions are supposed to have Chinese Walls, to prevent various patterns of inappropriate behaviour, including Insider Trading and Insider Recommendations. Among other things, the Chinese Wall is supposed to protect investment analysts from commercial pressure from other parts of the same organization. Recently, there has been much criticism of financial analysts (especially in America) who recommend the purchase of stocks simply because their colleagues have a commercial interest in promoting that stock.

Sometimes of course, the Chinese Wall is just a notional boundary, with little real effect - for symbolic or compliance purposes only. Although the actual information flows may be concealed, a strong correlation between activities on both sides of the wall may be sufficient evidence that some collusion has occurred. (Clearly architects need to appreciate the differences between official structure and defacto structure.)

In journalism, there is always supposed to be a Chinese Wall between editorial and advertising, to protect the objectivity of the journalists from commercial bias. This principle is often blatently breached, so it is pleasing to see the Daily Mail following the principle on this occasion. Although I disagree with their attack on Mr Fry, I respect the fact that the Mail has chosen to offend their advertisers rather than abandon its strongly held views.

Wednesday, September 15, 2010

The Future of Enterprise Architecture

Let's start with the positive.

Enterprise architects are mostly pretty skilled at identifying duplication, inconsistency and lack of interoperability across complex systems of systems, which they regard as fragmentation and waste. As a consequence, they typically set themselves a mission of simplification and unification. For some, this mission is largely confined to IT systems; for others, this mission may extend to business processes and the entire business operation.

In pursuing this mission, they find themselves opposed to narrow or short-term thinking from various stakeholders - line managers wanting quick and cheap solutions to local problems, project managers or external contractors trying to meet tight project goals, infrastructure managers wanting to protect established assets and arrangements. A committed enterprise architect will therefore get into a series of battles inside the organization, defending the principles of simplification and unification. (An uncommitted enterprise architect will withdraw from these battles and devote energy to producing abstract plans that will be ignored by everyone else - but it's difficult to see any value whatsoever in this kind of disengagement.)

The primary weapon in these battles is the Plan - an abstract blueprint that represents an idealized TO-BE architecture. This Plan needs to embody the principles of simplification and unification, not only because that's the mission, but also because that's the only way it's going to make sense to senior management, without whose support the battles cannot be won. The Plan is therefore not just a weapon but a story, perhaps even a myth.

But here's the problem: real business is inherently complex. Therefore battles against complexity (not always, but often enough to cause concern) are ultimately battles against the business. Or worse - against the customers of the business.

The mission to simplify and unify has undoubtedly had some successes, but it has also produced some massive failures - projects that have gone way over budget without coming anywhere close to delivering the promised business benefits. Some of the best-known examples can be found in the public sector because these are open to public scrutiny, but the private sector is not immune to this kind of failure. In the UK this week, the National Project for IT in the health service has been finally put out of its misery, following cancellation of a number of other projects that were supposed to deliver both improved outcomes and massive cost savings.

To the extent that projects like this appear to be following an old-style enterprise architecture agenda, these failures serve to undermine the credibility of enterprise architecture in the eyes of many stakeholders, and should pose a series of hard questions for enterprise architecture practice.

Of course enterprise architects could try to attribute all such failures to errors of execution rather than errors of intention or planning (in other words, the vision was okay, the overall plan was okay, but the project managers screwed up). Alternatively, they could try to find fault with each of the failed plans, identifying ways in which these plans disobeyed some of the core principles of enterprise architecture.

But I believe these examples force us to do something more fundamental - to rethink the mission of enterprise architecture. If the mission to simplify and unify conflicts with the mission to "align" (whatever that means) with the business, then perhaps enterprise architects need to find a new mission.

Simplification and unification is merely a strategy in the management of complexity. As I see it the enterprise architect's job is not to manage-away the complexity of business or IT, but to find ways to make the complexity more manageable.

More manageable for whom? Just about anybody. Business line managers need to be able to manage the complexity of their business operation, and the top management team needs to be able to manage the complexity of the whole organization. Meanwhile, business support managers (such as IT and HR) need to be able to manage the complexity of their own domain as well as providing specialist support for the rest of the organization.

Thus the enterprise architect should be helping the CIO answer the following two questions.
  • How can I manage the complexity of IT operations, systems and services?
  • How can IT contribute to managing the complexity of the rest of the organization (including other support functions)?
Similar questions apply to other support functions. Thus for example the enterprise architect should also be able to help the HR director answer the following two questions.
  • How can I manage the complexity of people management?
  • How can I help develop the capability of the whole organization to collectively manage complexity?

The IT department can support the management of complexity across the organization by providing a set of useful tools, including business intelligence tools, decision support tools, social networking tools and so on. (The provision of management information has always been a major responsibility of the IT department, and this cannot be done properly without understanding how and why management information is used.) The HR department has a different set of tools, such as training programmes, incentive packages, career paths, and so on. The enterprise architect's role here is not to manage the detailed implementation of all these different tools, but to understand the overall structural requirements and implications.


Architecture is basically about structure. Business is facing some increasing complex structural problems, and the primary task for enterprise architects is to tackle these structural problems. Tackle, not solve. The collective ability of the whole organization to manage the essential complexity of demand, from customers and the rest of the environment, depends strongly (although not solely) on the emergent structure of the organization and its systems, and this is where the enterprise architect's attention should be focused.

Monday, September 06, 2010

Generalization as Business Strategy

#entarch @BBCRadio4 Just listening to a business story on the radio: the music retailer HMV is now going to be selling clothes from its flagship Oxford Street store. Of course this kind of thing is nothing new - retailers have always tried to diversify, and there are clothes shops in Oxford Street that are trying to sell CDs, while large retail chains that started as grocery stores now sell books, clothing, computers, mobile phones and furniture, as well as financial services.

For many enterprise architects, this kind of diversification strategy seems like a "no-brainer". If you have a robust retail process, together with the assets and infrastructure to support the process, then surely it makes sense to make as much use of the process as you can.

But successful diversification is by no means a "no-brainer" (how I despise that term and its implications), but requires careful consideration and planning. This kind of planning should be meat and drink for enterprise architects, but not if they imagine that the decision follows automatically from some abstract process model.

So what models and techniques would be relevant to support an intelligent diversification strategy? And how comfortable are enterprise architects with using these models and techniques to support business decision-making?

These are questions I've looked at before on this blog (see the generic v specific category) and will certainly look at again. Meanwhile, I'd welcome your ideas.

Saturday, September 04, 2010

Making Conversations Visible

@sbskmi gave an excellent presentation at the RESG AGM yesterday evening, offering a survey of sensemaking tools.

Simon's research group at the Open University has a tool called Compendium, which belongs to a long and respectable line of issue and argument mapping tools, going back to IBIS (Horst Rittel, 1972). Simon showed a range of recent initiatives that demonstrate that issue and argument mapping has, as Simon puts it, "come of age".

Compendium itself has been designed to allow logical arguments to be supported by rich media evidence, such as video. So we can manipulate conflicting knowledge claims, together with the ethnographic material that might be relevant to resolving these claims.

The particular interest of tools like Compendium (and its Web 2.0 cousin Cohere) to the Requirements Engineering community is the desire to document design rationale, and these tools can certainly be used for this purpose. But they can equally be used to support real-time control systems, and Simon showed an emergency response coordination scenario, with web service links to various information feeds, as well as the ability to produce mashups of various kinds.

However the aspect of Simon's work that interested me most was the potential link to organizational intelligence. For example, he displayed a model of creative competences for complex challenges, based on the work of Palus and Horth, and showed how Compendium could use visual images to support the Palus and Horth methodology. (There are some parallels with the Repertory Grid technique, which I introduced into data modelling in the 1980s.) Simon also showed how Compendium could be used to compose machine intelligence with human intelligence. This helps to realise the original vision of Takehito Matsuda, who twenty years ago defined organizational intelligence as "the interactive-aggregative complex of human intelligence and artificial intelligence in an organization".

As Brenda Dervin argues, sense-making is triggered by anomalies and exceptions. The point about an exception is that it forces us to review and revise our pet models and theories, or at least it should do so. As Lakatos pointed out however, in his brilliant essay Proofs and Refutations, people typically deploy various tactics for dismissing exceptions in order to preserve their favourite models and theories, including Monster-Barring and Monster-Adjustment. (See my piece on Models and Monsters.) Thus the relationship between any given piece of evidence and a complex argument is an act of interpretation and is itself subject to argument.

To understand an argument or rationale, we need to pay attention not just to the domain (subject matter) of the argument but also the discourse or discursive practice. Within a large organization, there are several competing discourses, and the intelligence of the whole organization depends critically on a healthy interchange between different discourses. For example, arguments based on conventional accounting practice may bias the organization towards certain ways of solving problems, and make some kinds of innovation impossible; so sometimes management needs to be able to step away from the accounting paradigm and look at other kinds of rationale for organizational change. It would be interesting to see if models and software tools could support a conversation that straddled multiple discourses.

But in any case, these tools for collective sense-making and decision-making should fit nicely into an overall architecture for organizational intelligence.



Friday, September 03, 2010

Zero-Based Requirements

Brown-Field Requirements

One view of requirements engineering is that its purpose is to produce a complete and coherent statement of what some system-of-systems is required to do - in other words, its behaviour in the broadest sense, including "functional", "non-functional" and "commercial" requirements.

In a "Green-Field" scenario, we might imagine that this statement of requirements would result in the procurement and installation of a system of systems meeting the stated requirement. Requirements engineering is focused on understanding exactly what is required, and specifying it in an unambiguous and testable form.

But in almost all engineering projects, some system of systems - technical or sociotechnical - already exists, and the practical purpose is to make some planned changes to it. So people in the RE community are starting to talk more about "Brown-Field" requirements.

(RESG Event: Managing Brownfield Project Requirements, London, October 12th 2010)

Gap Analysis

An obvious starting point for brownfield requirements analysis would seem to be the identification of a gap between desire and reality. People often produce two models - an AS-IS model that describes the existing system, and a TO-BE model that describes its future replacement. The engineering requirements are then derived from the differences between the AS-IS model and the TO-BE model. This will typically result in a solution, possibly involving rebuilding some of the subsystems, replacing or upgrading some of the component parts, adding some new stuff or stripping out old stuff, rewiring the network, retraining the people, resetting system policies and parameters, and so on. In addition to a solution blueprint, showing how all these elements are to be configured, there will also be a transition strategy, indicating how (and in what sequence) all these changes will be installed. There are usually operational constraints - for example, a requirement to keep critical business processes running at an acceptable level during the transition period.

Imagine you want to rebuild your kitchen. You have to think about fitting new units into the existing space, or possibly moving a wall to give yourself more space. You have to decide whether you are going to keep the existing fridge (which you only bought last year) or buy a new one. And you have to think about how long you can manage without being able to cook. Moving the wall, or deciding to keep the old fridge, belong to the solution domain. But if you are going to do requirements analysis properly, there needs to be something in the statement of requirements that helps you determine these aspects of the solution.

In all but the smallest and most simple projects, there will be many solution variants. The decision to retain or replace a particular component may be based on a technical calculation of its likely performance and capacity within the new configuration, or may be based on a political calculation as to the most convenient budget from which to fund the replacement (in other words, preferably someone else's budget, if we can get away with not replacing it now).

Ten years ago this month, I wrote a piece about this in relation to Component-Based Software Engineering. Supply and Fit (CBDI Journal, September 2000).

But there is a more fundamental reason why there are many possible solutions - because making sustainable changes to complex systems is a tough challenge. Large and complicated change programmes aren't always the most effective; a small intelligent fix is often far better (and less risky) than any amount of optimistic meddle.

So before we can get to a solution blueprint and a transition strategy, we need an intervention strategy. This takes us out of the comfort zone of requirements engineering into general systems thinking.

Leverage Points

Donella Meadows identified twelve leverage points for making changes in complex systems, and suggested that these could be ranked according to their power. See original paper by Donella Meadows. A version is included in her posthumously published book Thinking in Systems (2008).


If the intervention strategy can be expressed as a combination of leverage points, then this raises the question for requirements engineering - how do we work through the requirements of changing a complex adaptive system in a way that could produce this kind of intervention strategy?

Zero Based Procurement

Finally, I wanted to make a comment about one of the (many) dysfunctional aspects of prevailing procurement practices. In his blogpost Was this NHS IT tender a stitch-up? (Computer World, September 2010), Tony Collins talks about the difficulties of referencing a specific product or system in a tender document. "If a user organisation has a system it’s happy with, and wants to keep and enhance, why would it want to go through the needless expense of an EC tendering, rather than simply renew the contract?"

Procurement rules may have been designed to prevent cosy and uncompetitive relationships between public sector organizations and their suppliers. They appear to have the effect of forcing each procurement to be treated as a separate exercise, starting each time from a blank sheet of paper, so that there is at least the theoretical possibility of giving new suppliers a chance. (This is similar to the principle of Zero-Based Budgeting.) Many people doubt that these mechanisms actually have any real effect on competition or value-for-money; but meanwhile, these mechanisms appear to have a strongly negative impact on through-life capability management. How can brownfield requirements engineering be done properly under these constraints?