Documenting Business Requirements

A couple of questions came in on my phone, and I thought I'd share my answers here.

When is a Business Requirements Document good enough?

 

Here are a few considerations to start with

Fit for purpose - What is the document going to be used for - planning, estimating, vendor selection, design or whatever? Does it contain just enough for this purpose, or does it dive (prematurely) into solution design or technical detail?

SMART - are the requirements specific, measurable, and so on, or are they just handwaving? How will you know when (if) the requirements have been satisfied?

Transparency and governance - is it clear whose requirements are being prioritized (FOR WHOM)? Who wrote it and what is their agenda? (Documents written by a vendor or consultancy may suit their interests more than yours.)

Future-proof - are these requirements short-term and tactical, are all the urgent requirements equally important?

Complete - ah, but what does that mean?

When is a Business Requirements Document complete?

Scope completeness - covering a well-defined area of the business in terms of capability, process, org structure, ...

Viewpoint completeness - showing requirements across all viewpoints - people, process, data, technology, ...

Constraint completeness - identifying requirements relating to privacy, security, compliance, risk, ...

Transition plan - not just data migration but also process change, big bang versus phased, ...

Quality and support plan - verification, validation and testing, monitoring and improvement. What is critical before going live versus what can be refined later.

RAID - risks, assumptions, issues and dependencies 

Important note - this is not to say that a business requirements document must always be complete in all respects, as it's usually okay for requirements to develop and evolve over time. But it is important to be reasonably clear about those aspects of the requirements that are assumed to be more or less complete and stable within a time horizon appropriate for the stated purpose, and to make these assumptions explicit. For example, if you are using this document to select a COTS product, and that product needs to be a good fit for requirements for at least n years, then you would want to have a set of requirements that is likely to be broadly valid for this length of time. (And avoiding including design decisions tied to today's technology.)

Conversely, where the requirements cannot be pinned down at this stage, it is important to avoid making premature solution decisions based on optimistic assumptions. There may be a general requirement for flexibility in some areas. 

On the CMO-CIO disconnect

A study from Accenture exposes a disconnect between the Chief Information Officer (CIO) and the Chief Marketing Officer (CMO), with only one in 10 of the executives surveyed being satisfied with the current level of collaboration between CMOs and CIOs.

Key findings of the study, based on a survey of 400 senior marketing and 250 information technology (IT) executives in 10 countries, include:

• CMOs believe IT doesn’t make the marketing function a priority.
• More than thirty percent of CMOs believe IT keeps marketing out of the loop and does not make time and technical resources available.
• Thirty six percent of CMOs say IT deliverables fall short of expectations.
• Forty six percent of CIOs say marketing does not provide an adequate level of business requirements.
• Despite CIOs appearing more open to engaging with CMOs, only 45 percent of CIOs say that supporting marketing is near or at the top of their list of priorities.

Accenture argues that this disconnect threatens the ability of companies to deliver effective customer experiences, and suggests some ways for CIOs and CMOs to work more effectively together.

The report contains some interesting hints of cognitive differences between the two functions. Take for example the concept of "requirement". On the one hand, marketing wants IT to respond faster and more flexibly to "market requirements". Whereas IT complains that marketing doesn't provide adequate and stable definition of "business requirements". This indicates a clash between two conflicting notions of what counts as a legitimate requirement, and the Accenture report doesn't explicitly suggest a resolution of this conflict.

There are also interesting differences in the implied value system. The marketing function tends to place higher value on hard-to-quantify business benefits such as "customer insight", whereas the IT function tends to place higher value on hygiene factors such as privacy and security. In both cases, these priorities may be influenced by the way budgets and targets are allocated to each function by the organization as a whole, and the ways in which different kinds of investment and operational expenditure can be legitimately cost-justified. Let us imagine that in a particular organization, the CIO can only justify investing in a new Customer Insight system if she can show that this system will produce measurable improvements in business outcomes. Whereas the CMO can only justify devoting any resources to customer privacy if she can show that security breaches would have a measurable effect on customer satisfaction or corporate reputation. (This may be relatively easy in some sectors, much harder in other sectors.)

There are some prevailing stereotypes of marketing and IT, which would suggest they are on different planets: one function being precise and highly numerate, the other being imprecise and unreliable.  In reality, they are much closer together, and should be able to collaborate closely, if only they can manage to speak the same language.


Source: The CMO-CIO Disconnect, Computer Weekly, August 2013.

Brownfield Requirements Engineering

Some partial and selective notes from the RESG meeting on Brownfield Requirements Engineering, held at BCS headquarters on October 12th 2010. By Richard Veryard.

---

Chairman's Introduction

Ian Alexander opened the meeting by pointing out the gulf between the demands of real engineering projects (95% brownfield) and the body of knowledge about requirements engineering offered by academics and authors (95% greenfield), and sketching some of the characteristic features of brownfield requirements. The importance of this topic is demonstrated by an excellent turnout for this meeting.

Brownfield Systems Engineering

Ian Gallagher of Altran Praxis presented some experience from several large systems projects, both military and civilian. The typical scenario is a major upgrade of existing systems to improve performance and deal with obsolescence; the upgrade may therefore include different types of requirement

• doing new things
• doing old things in new ways
• migrating away from ageing technologies, proprietary systems or restricted technologies (e.g. those subject to export licenses)

The upgrade may be regarded as a single large increment (from AS-IS to TO-BE) but is often managed as a series of smaller increments.

With greenfield engineering, the requirements of the engineering project are pretty much the same as the requirements of the TO-BE system, but brownfield engineering introduces a critical choice: whether to write total requirements for the TO-BE system or to write requirements for the project (or separate increments) based on the difference(s) between AS-IS and TO-BE. IanG expressed a strong preference for the former, but acknowledged that this isn't always acceptable to key stakeholders, especially if the effort would be perceived as excessive. But in any case it's obviously important to be clear what kind of requirements we are talking about.

Beyond the scope of the TO-BE system, there may be a larger system-of-systems context, in which other equally large systems are the subject of equally large brownfield engineering activity. This raises questions of the interoperability of the increments, and the coordination between autonomous brownfield engineering projects, which is another important issue for brownfield requirements engineering.

A key issue is the pressure to reuse and repurpose existing assets. There are three possible reasons for this.

1. Rational economics - genuine cost-saving based on lifetime exploitation of assets
2. Irrational inertia - desire to justify past investment decisions, resistance to change
3. Commercial - vested interests from key stakeholders (such as contractors) to maintain proprietary dependencies

Requirements such as migrating towards more "open" architectures are therefore not only relevant for an immediate brownfield project but also have important implications for the economics of future brownfield projects.

Managing Brownfield Financial Software

Phil Cantor of Smartstream then talked about his past and present experience managing and maintaining software products in the financial sector. (Most of his examples came from previous companies he had worked for.) The requirements of such products don't come exclusively (or even primarily) from the "users" but from a body of knowledge that the package vendor is expected to master. Phil gave the example of an obscure piece of financial calculation that is now only required in the Philippines: users elsewhere in the world may not know about this, but a package that doesn't cater for this requirement will be unacceptable to a global bank.

The package vendor then has to balance a wishlist of enhancement requests and bug reports from hundreds of user organizations, with the practical constraints of maintaining - and hopefully improving the quality of - an existing body of software.

For me, one of the most interesting issues raised by Phil was when he talked about the platform and its requirements. There is an internal platform supporting the user-facing components of the product suite, containing common services and suchlike, and the requirements for this platform cannot be derived purely from the requirements of the package as a whole. Furthermore, the package as a whole serves as a platform for the business processes of the user organizations (Phil's customers), so the same question arises at that level as well. However, this is not particularly a brownfield issue.

Phil also mentioned the challenges of training. From a sociotechnical perspective this is a major brownfield issue, since the performance of the TO-BE system will be affected by the user habits and expectations (e.g. knowing where to find things) carried forward from the AS-IS system. Training is a solution (and often not a very good one) to some set of sociotechnical requirements, and there may be better ways of conveying a new set of habits and expectations to the users and technical staff, but clearly we need to have an understanding of these requirements as well.

Discussion

Lots of further issues came up in the discussion, and I didn't manage to capture half of them. Someone raised the question of scale - what do you do with a brownfield situation with a complex network of interdependent pieces of hardware and software, with the need for upgrades constrained by small budgets and massive complexity. This led to the question of timing - how do you decide whether to upgrade something now or to defer the upgrade until next year. (Nobody actually mentioned the concept of technical debt, but that is clearly relevant here.)


Finally, my memory and notes can be supplemented by a few choice tweets.

@rmonhem much debate over how much to document requirements with significant re-use of existing applications

@rmonhem most difficult challenges have often been posed by cultural, commercial and legal factors.

@jiludvik Bit too much focus on techniques documenting and signing off detailed reqs. This is not specific for brownfield projects!

@jiludvik Phil Cantor's pres has been very entertaining and spot on for brownfield req's eng. I can certainly relate to many challenges he mentioned

Intelligence as Requirement

Let's suppose an organization suffers from a number of the symptoms of organizational stupidity and decides to do something about it. (Not an easy decision, but let's start from there.) How do we determine the detailed requirements for organizational and technical systems change?

Many brownfield requirements analysis projects start with negative requirements - we want to eliminate this bottleneck, reduce this wastage or variation, take out this cost and risk. This involves a combination of factual observation (some bottleneck exists in the target system, with such-and-such consequences) and a value judgement (the bottleneck and its consequences are undesirable relative to some value system).

These negative requirements then need to be converted into positive requirements - we want to improve certain whole-system properties (such as organizational intelligence), and we think we can achieve that by making certain changes to the whole enterprise (regarded as a sociotechnical system). These changes may include engineering new or improved mechanisms, as well as improved use of the existing mechanisms.

Typically, there are many different things we could consider doing to improve these whole-system properties, so we produce a wishlist of potential requirements. But such wishlists create all kinds of difficulties. It may not be feasible to do all of the items on the wishlist, firstly because of limited resources and secondly because some of the items may be mutually incompatible or redundant. So we try to create a meaningful subset of the wishlist, to produce a set of real requirements that is both feasible and affordable.

If the items on the wishlist were all completely independent of one another, then we could categorize each item into subjective categories (say "essential" and "desirable"). Or perhaps we could perform a cost-benefit analysis on each item, calculating a risk-weighted accounting value of the item in terms of its ROI or NPV. This would allow us to rank the wishlist with the most important, valuable or critical requirements at the top; the final set of "requirements" is then defined as that subset of the possible requirements above some arbitrary cut-off line.

Of course both the subjective sorting of requirements ("essential" versus "desirable") and the estimation of costs, benefits and risks are evaluated differently from different stakeholder positions, so the final list is typically fought over between stakeholders. Furthermore, things change over the duration of an engineering project, for various reasons, so the cut-off line comes under pressure.

But a more fundamental problem with this way of defining requirements is that the items on the wishlist are usually mutually dependent, at least to some extent. The costs, benefits and risks of each item may vary according to which other items are chosen, and the number of possible permutations is astronomically large. Therefore the challenge of defining a meaningful and feasible subset of the original wishlist becomes an architectural one - finding some underlying deep structure within the wishlist that permits stakeholders to make intelligent and informed choices.

Can requirements be "solved"?

Much writing on systems architecture, requirements engineering and related disciplines starts from the premise of a distinction between problem-space and solution-space. This is an extremely useful distinction under certain conditions, but we are increasingly facing challenges in which these conditions no longer hold valid. Here are some examples.

1. Where the goal is not to design some complex artefact but to deliver some set of outcomes. For example, to make an organization more efficient or intelligent, to make a tax system more equitable, or to make a community more self-sufficient.

2. Where the target system is complex and governed by conflicting value systems, so that the challenge is to formulate and negotiate a series of interventions, from which a useful and acceptable change to the target system might emerge.

3. Where the agency for change is embedded within the target system, rather than being external. Anyone who gets involved in the project becomes part of the target system, and it no longer makes sense to ask the popular question "are you part of the problem or part of the solution?"

---

One of the attractions of the problem/solution distinction is that it helps to build agreement around a statement of the problem, and focuses the debate on different ideas and preferences for solving the problem. But to see the limitations of this distinction, we only have to look at the political sphere. There is surprisingly little difference between the political parties in their statement of the fundamental problems facing the government; but there are wide differences in the policy instruments they advocate to solve these problems. Of course, these differences are not just alternative sociotechnical designs, but are produced by different belief systems (how the economy really works, what are the root causes of crime and terrorism, and so on) as well as different value systems (what kinds of prosperity and well-being matter most).

So is there a discontinuity between highly complex and messy sociopolitical problems on the one hand and relatively simple and self-contained engineering problems on the other? To answer this kind of question, some people appeal to a simplistic interpretation of the Cynefin framework, treating it as if it were a simple 2x2 matrix mandating different problem-solving or problem-tackling methodologies according to the degree of complexity in the problem domain - but of course this kind of answer begs lots of questions about the relationship between problem and solution which a more subtle and sophisticated application of Cynefin should expose.

Any human system (including organizations) will display some degree of complexity and messiness, and it is in this context that the concept of "requirements" needs to be understood. In a separate post, I shall explore how such a concept of requirements applies to organizational intelligence.

---

Requirements engineering is about the interaction between five kinds of system. In the problem-solving paradigm, these fit together as follows.

• A deliverable system is designed to be a "solution" to some "problem". (In a simple case, this could be a single complex artefact or a coordinated series of interventions.)
• A target system is where the "problem" is located, and where the "solution" must fit. (In a simple case, this could be an organization unit or operational business process.)
• A value system provides the criteria justifying why the problem is a problem, and why the solution would count as an improvement. (In a simple case, there is a single sponsor whose value system is paramount.)
• A belief system, an explanatory model or theory that explains the cause of the problem and the plausibility of the solution. (In a simple case, everyone broadly shares the same belief system.)
• An intervention system, a force of people and resources mobilized to do something relevant. (In a simple world, the intervention system is a project team under the exclusive governance of a project sponsor with a unified and dominant value system.)

Within the problem-solving paradigm, the relationships between these five are pretty straightforward, and it may not be worth articulating and analysing them separately. However, as we move beyond the problem-solving paradigm, these become plural rather than singular, and the relationships between them become far more interesting.

---

See also

• Niklas Björnerstedt, The obsolete idea of requirements (Feb 2010)
• Richard Veryard, Zero-Based Requirements (Sept 2010)
• Richard Veryard, Third-Order Requirements Engineering (Slideshare presentation based on a paper I wrote in 1995 with John Dobson).

Generalization as Business Strategy

#entarch @BBCRadio4 Just listening to a business story on the radio: the music retailer HMV is now going to be selling clothes from its flagship Oxford Street store. Of course this kind of thing is nothing new - retailers have always tried to diversify, and there are clothes shops in Oxford Street that are trying to sell CDs, while large retail chains that started as grocery stores now sell books, clothing, computers, mobile phones and furniture, as well as financial services.

For many enterprise architects, this kind of diversification strategy seems like a "no-brainer". If you have a robust retail process, together with the assets and infrastructure to support the process, then surely it makes sense to make as much use of the process as you can.

But successful diversification is by no means a "no-brainer" (how I despise that term and its implications), but requires careful consideration and planning. This kind of planning should be meat and drink for enterprise architects, but not if they imagine that the decision follows automatically from some abstract process model.

So what models and techniques would be relevant to support an intelligent diversification strategy? And how comfortable are enterprise architects with using these models and techniques to support business decision-making?

These are questions I've looked at before on this blog (see the generic v specific category) and will certainly look at again. Meanwhile, I'd welcome your ideas.

Making Conversations Visible

@sbskmi gave an excellent presentation at the RESG AGM yesterday evening, offering a survey of sensemaking tools.

Simon's research group at the Open University has a tool called Compendium, which belongs to a long and respectable line of issue and argument mapping tools, going back to IBIS (Horst Rittel, 1972). Simon showed a range of recent initiatives that demonstrate that issue and argument mapping has, as Simon puts it, "come of age".

Compendium itself has been designed to allow logical arguments to be supported by rich media evidence, such as video. So we can manipulate conflicting knowledge claims, together with the ethnographic material that might be relevant to resolving these claims.

The particular interest of tools like Compendium (and its Web 2.0 cousin Cohere) to the Requirements Engineering community is the desire to document design rationale, and these tools can certainly be used for this purpose. But they can equally be used to support real-time control systems, and Simon showed an emergency response coordination scenario, with web service links to various information feeds, as well as the ability to produce mashups of various kinds.

However the aspect of Simon's work that interested me most was the potential link to organizational intelligence. For example, he displayed a model of creative competences for complex challenges, based on the work of Palus and Horth, and showed how Compendium could use visual images to support the Palus and Horth methodology. (There are some parallels with the Repertory Grid technique, which I introduced into data modelling in the 1980s.) Simon also showed how Compendium could be used to compose machine intelligence with human intelligence. This helps to realise the original vision of Takehito Matsuda, who twenty years ago defined organizational intelligence as "the interactive-aggregative complex of human intelligence and artificial intelligence in an organization".

As Brenda Dervin argues, sense-making is triggered by anomalies and exceptions. The point about an exception is that it forces us to review and revise our pet models and theories, or at least it should do so. As Lakatos pointed out however, in his brilliant essay Proofs and Refutations, people typically deploy various tactics for dismissing exceptions in order to preserve their favourite models and theories, including Monster-Barring and Monster-Adjustment. (See my piece on Models and Monsters.) Thus the relationship between any given piece of evidence and a complex argument is an act of interpretation and is itself subject to argument.

To understand an argument or rationale, we need to pay attention not just to the domain (subject matter) of the argument but also the discourse or discursive practice. Within a large organization, there are several competing discourses, and the intelligence of the whole organization depends critically on a healthy interchange between different discourses. For example, arguments based on conventional accounting practice may bias the organization towards certain ways of solving problems, and make some kinds of innovation impossible; so sometimes management needs to be able to step away from the accounting paradigm and look at other kinds of rationale for organizational change. It would be interesting to see if models and software tools could support a conversation that straddled multiple discourses.

But in any case, these tools for collective sense-making and decision-making should fit nicely into an overall architecture for organizational intelligence.

---

Making Conversations Visible

View more presentations from Simon Buckingham Shum.

Zero-Based Requirements

Brown-Field Requirements

One view of requirements engineering is that its purpose is to produce a complete and coherent statement of what some system-of-systems is required to do - in other words, its behaviour in the broadest sense, including "functional", "non-functional" and "commercial" requirements.

In a "Green-Field" scenario, we might imagine that this statement of requirements would result in the procurement and installation of a system of systems meeting the stated requirement. Requirements engineering is focused on understanding exactly what is required, and specifying it in an unambiguous and testable form.

But in almost all engineering projects, some system of systems - technical or sociotechnical - already exists, and the practical purpose is to make some planned changes to it. So people in the RE community are starting to talk more about "Brown-Field" requirements.

(RESG Event: Managing Brownfield Project Requirements, London, October 12th 2010)

Gap Analysis

An obvious starting point for brownfield requirements analysis would seem to be the identification of a gap between desire and reality. People often produce two models - an AS-IS model that describes the existing system, and a TO-BE model that describes its future replacement. The engineering requirements are then derived from the differences between the AS-IS model and the TO-BE model. This will typically result in a solution, possibly involving rebuilding some of the subsystems, replacing or upgrading some of the component parts, adding some new stuff or stripping out old stuff, rewiring the network, retraining the people, resetting system policies and parameters, and so on. In addition to a solution blueprint, showing how all these elements are to be configured, there will also be a transition strategy, indicating how (and in what sequence) all these changes will be installed. There are usually operational constraints - for example, a requirement to keep critical business processes running at an acceptable level during the transition period.

Imagine you want to rebuild your kitchen. You have to think about fitting new units into the existing space, or possibly moving a wall to give yourself more space. You have to decide whether you are going to keep the existing fridge (which you only bought last year) or buy a new one. And you have to think about how long you can manage without being able to cook. Moving the wall, or deciding to keep the old fridge, belong to the solution domain. But if you are going to do requirements analysis properly, there needs to be something in the statement of requirements that helps you determine these aspects of the solution.

In all but the smallest and most simple projects, there will be many solution variants. The decision to retain or replace a particular component may be based on a technical calculation of its likely performance and capacity within the new configuration, or may be based on a political calculation as to the most convenient budget from which to fund the replacement (in other words, preferably someone else's budget, if we can get away with not replacing it now).

Ten years ago this month, I wrote a piece about this in relation to Component-Based Software Engineering. Supply and Fit (CBDI Journal, September 2000).

But there is a more fundamental reason why there are many possible solutions - because making sustainable changes to complex systems is a tough challenge. Large and complicated change programmes aren't always the most effective; a small intelligent fix is often far better (and less risky) than any amount of optimistic meddle.

So before we can get to a solution blueprint and a transition strategy, we need an intervention strategy. This takes us out of the comfort zone of requirements engineering into general systems thinking.

Leverage Points

Donella Meadows identified twelve leverage points for making changes in complex systems, and suggested that these could be ranked according to their power. See original paper by Donella Meadows. A version is included in her posthumously published book Thinking in Systems (2008).


If the intervention strategy can be expressed as a combination of leverage points, then this raises the question for requirements engineering - how do we work through the requirements of changing a complex adaptive system in a way that could produce this kind of intervention strategy?

Zero Based Procurement

Finally, I wanted to make a comment about one of the (many) dysfunctional aspects of prevailing procurement practices. In his blogpost Was this NHS IT tender a stitch-up? (Computer World, September 2010), Tony Collins talks about the difficulties of referencing a specific product or system in a tender document. "If a user organisation has a system it’s happy with, and wants to keep and enhance, why would it want to go through the needless expense of an EC tendering, rather than simply renew the contract?"

Procurement rules may have been designed to prevent cosy and uncompetitive relationships between public sector organizations and their suppliers. They appear to have the effect of forcing each procurement to be treated as a separate exercise, starting each time from a blank sheet of paper, so that there is at least the theoretical possibility of giving new suppliers a chance. (This is similar to the principle of Zero-Based Budgeting.) Many people doubt that these mechanisms actually have any real effect on competition or value-for-money; but meanwhile, these mechanisms appear to have a strongly negative impact on through-life capability management. How can brownfield requirements engineering be done properly under these constraints?

What are silos good for?

@markgould13 tells us why the silo doesn't work

Of course they do work - after a fashion. As Mark points out, they are resilient structures, from which we can infer that they serve some purpose for someone in the organization, or the organization as a whole. It is common (Mark calls it a cliché) to reject information or work silos in most organizational contexts. But what exactly is the alternative?

Another useful observation about silos is that they generally represent some attempt, whether planned or emergent, to decompose an organization according to some notion of specialization and clustering. When people complain about silos this could be because they reject any kind of decomposition, but what is more likely is that they dislike this particular decomposition pattern. However, anti-silo rhetoric is often pretty vague about the difference if any between silo (=bad decomposition) and autonomous loosely coupled functional cluster (=good decomposition), and architects who automatically dismiss all previous attempts to structure an organization as "silos" create much the same impression as plumbers and electricians who automatically criticize the existing plumbing and wiring.


The main issue with any decomposition (whether or not we choose to label the chunks as "silos") is the coordination between the chunks, and I think this is Mark's main point. He quotes someone calls Ed Smith as saying

"Insight is the gap and overlap between silos."

Thinking about the requisite coordination between the chunks is an excellent route for understanding whether the architects should pay attention to the shape of the chunks or to improving the links and feedback loops connecting them. Or both. An architecture that is exclusively focused on cutting the enterprise into perfect chunks (relative to a fixed and abstract model of "the requirements") is probably not going to be much use in the long run.

---

Update

Chris Bird expanded his Observations on Silos (see comment below) on his own blog.

Excellent defence of silos by Venkat, The Silo Reconsidered

Interesting discussion on Linked-In - What are the advantages of working in silos?

 

Related post: Requisite Coordination (November 2010)

Business rules are forking

Another good post from @tetradian on business rules in response to James Taylor’s recent piece reporting Gartner's view that Business rules are king.

Tom agrees with James Taylor about the importance of discipline around business rules, but objects to an interpretation of business rules that goes back to Taylor's namesake Frederick Winslow Taylor.

Tom distinguishes between decision-support (where the decision is made by a collaboration between an automated system and human judgement) and automated decision-making (where there is no space for human judgement). This distinction is not always clear-cut - if the human actants within a complex business process lack the information, intelligence, attention or confidence to overrule the computer's suggested answer, then a decision-support system becomes defacto a decision-making system.

We should also distinguish between simple rules (binary logic) and more complex rules (modal logic, probabilistic logic). As Tom points out, much of the rule industry appears to assume simple composition of simple binary rules, which are inadequate for most interesting business problems (three quarters of his context space mapping framework).

Tom makes three important points

• We should not assume that the business rules are sufficient, invariant, accurate and complete, especially if they are derived from the people who run the existing processes. Therefore the identification and codification of business-rules generally leaves something to be desired. (One way of putting this is that the Real resists symbolization.)

• There needs to be a very strong emphasis on rule-maintenance, otherwise placing all the business-rules into an automated system will lead to a fit and forget attitude. (One way of putting this is a demand for double-loop or deutero-learning.)

• The viability of using automation for decision-making is dependent on the context.

---

In his new book Obliquity, John Kay discusses the example of waiting for a bus. According to the timetable, a bus should come every ten minutes. There are two rules that should help you decide whether to wait for the bus or walk - except that these two rules contradict each other.

Rule One says the longer you wait for the bus, the more likely it is to arrive soon. So if you have waited for nine minutes, it is practically certain to arrive in the next minute.

Rule Two says that the longer you wait for the bus, the more likely it is that Rule One is incorrect. So if you have waited more than nine minutes for the bus, it is starting to look as if the bus will never come at all.

If you have waited more than half-an-hour for the bus, then common sense suggests that Rule Two is in force. But as Kay points out, many people (including those who drove banks into bankruptcy) appear incapable of shifting from Rule One to Rule Two.

In real business situations, there is always a balance between following a rule and questioning the rule. It is not just automated systems that may fail to strike this balance; many people with significant business responsibility lack common sense. This is an important aspect of the context for decision-making.

---

Update September 2020

Prompted by the current interest in RPA, I've been looking over my old blogposts on Business Rules.

I'm not sure I agree with Tom's point that humans are good at modal logic - there are whole schools of psychotherapy devoted to unpacking common misuses of modal operators, for example reframing "always" and "never" into "sometimes". But individuals may be better than machines (whether computer machines or bureaucratic machines). 

James Taylor's post quotes Jim Sinur recommending a focus on high volatility rules - which would seem to tell against the "fit and forget" attitude Tom warns us about. But jumping forward ten years, we find RPA vendors recommending a focus on using bots for the low volatility rules. Meanwhile "mutant algorithms" try to tackle probabilistic decision-making. The issues raised in his post are clearly relevant to this new technology landscape.

Business Rule Concepts

by Ronald G Ross. Third Edition: Getting to the Point of Knowledge. Business Rule Solutions 2009.
ISBN 0-941049-07-8

A copy of this book arrived in the post this week. I don't know who sent it but I assume it's a review copy from the publishers. (In which case, the publishers have chosen not to implement the standard Cover-Note rule. Perhaps they have implemented the If-Anyone-Reviews-It-We'll-Probably-See-Something-On-Twitter rule instead.)

As the title indicates, the book provides a detailed account of the concept of Business Rule, and appears to offer some elements of a methodology for doing something or other. My normal procedure for reviewing such books is to apply the Four Causes.

• Final Cause - what's the purpose, for whom
  
• Material Cause - what is the stuff that is being worked on
• Efficient Cause - who does it, and how
  
• Formal Cause - what's the conceptual framework

Most of the book is taken up with a conceptual framework for business rules. What does a business rule look like, and how can different kinds of business rule be classified. So that's the Formal Cause.

Ross argues that just as the human body is composed of bones (structure), muscles (power) and nervous system (control), so business is composed of factual knowledge (structure), processes (power) and business rules (control), expressed as nouns and verbs.

To me, this looks like a huge simplification. For a start, in order for the human body to do anything useful, the muscles need energy. The biological control system doesn't only control the muscles, it also controls the respiration system - delivering stored energy to the muscles. And for a business to be viable, we generally need to pay attention to the adverbs as well as the nouns and verbs. But in order to consider whether Ross's simplification is useful for some purpose, we need to look at the other three causes.

A business rule is supposed to describe the business. This suggests that there is some stuff (let's call it the implicit logic of the business) from which properly constituted rules can be elicited - the Material Cause. Rules are correct and complete if they accurately and fully capture the AS-IS or TO-BE logic of the business. (Ross tells us little about verification and validation, except to say that it's easier if the rules are held in some kind of repository.)

But which logic? Ross doesn't want to get sucked into expert systems or artificial intelligence (p 23). He argues that the point of establishing business rules is to give you more time for strategic and tactical thinking, problem solving and other stuff (pp 4-5). I infer from this that he is not trying to elicit the logic of these higher-level activities (where my interest in Organizational Intelligence is focused): his methodology concentrates on what he calls Operational Decisions.

The book also contains a number of rules about rules. For example

• A rule saying there shall be no rule restricting the age at which a customer can open an account (p 92). This kind of rule is important in any organization where different units can define some of their own rules.
  
• A rule saying that the cheapest rule should be executed first (p 128). This kind of rule is important for balancing the internal efficiency of a system with the external customer experience, and assumes we have some way of associating different kinds of cost with a rule. (The cheapest rule from the programmer's point of view may be very expensive in terms of customer satisfaction and lost business.)
  

These rules-about-rules are not formalized, and so I'm presuming they are not within the scope of the business rules methodology. However, some important rules-about-rules are stated in the Business Rules Manifesto (p 143-144). For example

6.3 - A business rule system must always be able to explain the reasoning by which it arrives at conclusions or takes action. 

6.4 - Rules are based on truth values. How a rule's truth value is determined or maintained is hidden from users.

I suspect that these two rules conflict with each another. Unfortunately, I can't verify this suspicion, because the vocabulary used to express these business rules is not coordinated (see p 108). Perhaps Mr Ross should eat his own fish-fingers.

Now let's look at the Efficient Cause. The book seems to be aimed at a business analyst trying to specify the requirements for a system - either a completely automated system or what Ross calls a Really Smart System, which may include people making smart operational decisions with the support of rules that are rendered as guidance messages. However, I didn't get a very clear picture of how Business Rules might fit into a normal systems development methodology. Indeed, the back cover blurb advertises "a common-sense approach to solving today's operational business problems", but I could find nothing in the book that looked like a real business problem (as opposed to fairly routine business requirements.)

(To be fair, I should say that Ross's separation between facts, processes and rules seems consistent with some of the business modelling ideas I have published through the CBDI Forum, so I'm not saying it's wrong, merely that it may not be the whole story.)

Last but not least, the Final Cause. What is the value of this methodology, to whom? Ross asserts that the explicit separation of factual knowledge, processes and business rules produces more effective solutions, and therefore more successful (effective, adaptive) business behaviour (p 5). His approach has the "potential for closing the requirements gap between business people and IT" (p 28). And he asserts that for some class of crucial everyday decisions "you can capture all the business rules and make precise or optimal decisions" (p 31).

Ross doesn't provide any evidence for these assertions, but he doesn't have to, because they are "proven". Humph.

So where does that four-cause analysis leave us? What Ross gives us is a lot of detailed discussion about business rules and how they can be analysed. If you are already sold on the concept of business rules, and possibly interested in using business rules to design a layer of capability services that are decoupled from the entity layer and the process layer, then you may find the book very helpful.

---

Why have I devoted a long blogpost to discussing this book? Because I think it's important for design methodologies to be presented in a balanced way - covering all four causes. Ross's book is typical of many such books - fairly sound on the Formal Cause, not so strong on the other three causes - and I think this imbalance helps to explain the patchy adoption of such methodologies.

Related Posts: Deconstructing the Grammar of Business (June 2009), Business Rules are Forking (March 2010)

Pay by Results - Service Efficiency

Redmonk's James Governor reports from a Logica briefing

"nice! UK Medical Research Council pays more money to its call center supplier Logica the fewer calls it takes! good service = fewer calls!"

In other words, service payment increases as service utilization goes down. This is a very interesting example - somewhere between output-based pricing and value-based pricing - and may well be deliberately based on John Seddon's concept of "Failure Demand".

Seddon's concept is based on the observation is that there are two reasons why a service is invoked.

• Sometimes because the consumer actually wants something of direct value - Seddon calls this Value Demand.

• But more often because something has gone wrong somewhere, and the consumer wants it fixed. Seddon calls the Failure Demand, and observes that much of the activity of service-based organizations (especially call centres) is coping with Failure Demand.
  

Clearly the answer is not to design services so that they can handle Failure Demand more efficiently and cheaply. The answer isn't even to automate the Failure Demand, so that the User can get a status report without talking to a real human being. The answer is to design joined-up systems that don't generate so much Failure Demand in the first place.

A number of management consultants/consultancies have latched onto this concept.

• Dave Droar (Arrod)
• JSK Solutions
• Sharp-End Training
• Richard Oliver (Purposive Drift)

And here is the concept From the Horse's Mouth (via Benjamin Mitchell)


What does this concept mean for the design of service-oriented systems? One thing I draw from this is that it provides yet another argument why you shouldn't start the design from a set of use cases. And I am also interested in the implications for service pricing and governance of the arrangement between MRC and Logica. If you know of any similar examples, please comment below or contact me.

Two Kinds of Business Model

Cross-posted to the new CBDI Forum group on Linked-In.

---

Am currently pulling together material on business improvement for an article in the CBDI Journal. Any ideas, contributions, questions, examples, challenges, pitfalls, or other input welcome.

One of the first problems I've been facing is the question of the Business Model. It has become clear to us that there are (at least) two entirely different notions of Business Model.

The business notion of Business Model is The-Way-We-Do-Business, What-Distinguishes-Us-From-Our-Competitors. For example, Pile-Em-High-And-Sell-Em-Cheap or Ship-Em-Quick-But-Collect-Cash-Quicker. I call this Business Model (B). That's what any normal person outside IT thinks a business model is.

The computer notion of Business Model is a collection of line-and-box diagrams, such as data diagrams or process flow diagrams, drawn in IDEF or ER or UML or BPMN or ArchiMate or something like that, together with rigorous semantics for each of the elements on the diagram, which describe the conceptual structure of the business. I call this Business Model (C). These diagrams typically abstract away from many of the things that differentiate an enterprise from its competitors.

Now it is perfectly clear that many business and IT improvements don't involve any change to Business Model (C) whatsoever. For example, a major data cleansing exercise may involve a lot of work for the computer department, and produce some significant benefits for the business, but unless the database schema is altered then there is no need to change Business Model (C).

Incidentally, lots of SOA case studies seem to fall into this category. For example, reducing cycle time or latency probably doesn't change any of the UML diagrams, it merely changes the service level associated with one of the elements depicted on one of the diagrams.

Some business improvements do involve change to both Business Model (B) and Business Model (C). For example, when airlines went over from unique flight numbers to a code-sharing system, in which the same flight could belong to several different airlines and have several different code numbers. This kind of thing clearly affects database schemas and programs, as well as operational relationships between an airline and its partners.

Another good example involving change to both Business Model (B) and Business Model (C) is when retailers started to introduce loyalty cards, thus allowing them to identify a customer as the same again.

So here's a question for the enterprise architects and business analysts out there. If the business model is going to drive business improvement, which business model is it? Are you using any formal methods and notations for Business Model (B), or do you jump straight to formal methods and notations for Business Model (C)?

And are there any other kinds of Business Model I should be looking at?

---

CBDI Journal articles are now available to non-members.

• In Search of a Business Improvement Methodology (December 2008)
• Exploring Business Improvement (January 2009)

On this blog, several follow-up posts on Broken Business Models

• 16 December 2008
• 19 December 2008
• 1 February 2009
• 12 February 2009

Business Improvement from Richard Veryard

So-Called Non-Functional Requirements

I agree completely with Nick Malik when he objects to Non-Functional Requirements: the "All-Other" classification.

It is certainly a conceptual error, as Nick explains, to define an engineering concept in negative terms, in other words in terms of what it isn't. But in addition to being a conceptual error, it is also a psychological error, because it makes us believe that the concept is somehow unimportant. Here's what I said in my 2001 book on the Component-Based Business (page 105).

In some cases, some technical requirements of a component can be explicitly derived from the overall solution requirements. In other cases, they are left implicit or delegated to "technical" personnel. These requirements may include such factors as availability, reliability and performance.

These requirements are often regarded as second-class ones, deserving less respect and attention. Much time and energy is typically devoted to getting them right, but this is often grudging and carries less status and reward. Businessmen often call these "hygiene factors", as if to say that they're about the same level as cleaning the office toilets. And software engineers convey a similar attitude when they refer to requirements as "non-functional" - they might as well call them dysfunctional and be done with it.

When I wrote that book, I objected to the term "non-functional requirements" because I believed that such requirements were as important as so-called "functional requirements", but I have since changed my mind. What I now believe is that such requirements are very much more important, at least from an architectural perspective, than the so-called "functional requirements".

This is because it is generally easier to plug in or mash in extra functionality, with fewer architectural implications, than to plug in extra performance. Plugging in a bit of extra functionality is like adding a satellite navigation device or an entertainment console to your automobile. But if you want to carry double the passengers at twice the speed, you have to totally reengineer the vehicle and the engine. Let me quote myself again, this time from last year [Blueprints]

In a traditional waterfall development, many people thought it was a good idea to address the functional requirements first (logical design), and then tweak the design (physical design) until you satisfied the non-functional requirements as well. But when you are composing solutions from prebuilt enterprise services, this approach just doesn't wash. Indeed, it may now be the other way around: if a service assembly fails some functional requirement, you may be able to plug/mash in some additional service to fill the gap; but if it fails the non-functional requirements you may have to throw the whole thing away and start again.

In my opinion, therefore, the architect needs to worry about those requirements that have major structural implications, and can mostly leave the functional requirements to the business analysts and developers.

In general, these structural requirements can't be defined or managed at the level of the individual service, but need attention to the overall geometry of the solution. So a better name for them might be "holistic".

Meanwhile, another extremely important class of requirements arises from the commercial constraints. Someone once told me that when an IKEA designer is given a design task, the starting point for the design is the intended retail price of the finished item. "Design a lamp we can sell for 20 Euros". An SOA equivalent would be to tell the service designer "build me an invoicing service that will operate for less than x dollars per thousand". This aspect of requirements engineering is completely ignored in most design methods, which assume that the price is tacked on at the end.

So I guess anyone who thinks the quality requirements are non-functional probably thinks the commercial requirements are non-non-functional.

---

Update

Words matter. But if we toss "requirements" in favor of *design*, we defenestrate "NFR" in the same motion!https://t.co/ndQeo3YS4e

— VisArch (@ruthmalan) October 20, 2016

---

References

Ruth Malan, Non-Functional Words - Tricky Beasties (January 2014)
Nick Malik, Non-Functional Requirements (October 2008)

Another Brick

The Harvard Business Review has just discovered SOA - and guess what? It's like Lego bricks!

The article is written by Ric Merrifield of Microsoft (creator of the Motion methodology) and a couple of his mates.



The Next Revolution in Productivity. Ric Merrifield, Jack Calhoun & Dennis Stevens. Harvard Business Review, June 2008

Business Analysis Skills

Andrew Johnston comes away from the 2008 Enterprise Architecture Conference in London worried about A Shortage of Analysts.

One of the major causes of project failure is that the requirements aren't properly understood and communicated, but this isn't exactly a new problem. What is new is that emerging development paradigms, from SOA to outsourcing, affect the opportunities for requirements analysts to learn their craft. In the past, analysts and developers typically worked together in the same organization - developers consumed and actively critiqued the work of the analysts. (When I was a programmer, I asked so many awkward questions of the analysts they decided to make me an analyst instead.)

However, although the experience of collaboration between analysts and developers is undoubtedly useful in the acquisition and refinement of analysis skills, and it is important that analysts have an appreciation of "making it work", I am not sure I agree with Andrew that analysts need to have a development or equivalent background. Someone might equally argue that analysts should have a systems testing background - "making it not work" - because that would give them a strong sense of what can possibly go wrong with an apparently simple idea for business improvement.

Twenty or thirty years ago, it may have made sense to recruit business analysts from the ranks of developers. After all, they were the only people in the organization who understood IT, they were often better educated than the "end-users" (in those days we were mostly dealing with automating administrative systems, so the end-users were mostly clerks), they could think logically, and they had a kind of intellectual curiosity that had lured them into IT in the first place.

But these assumptions no longer hold. Intellectual ability and IT literacy are certainly not unique to the IT department - in some organizations nowadays the IT department comes a long way behind the "user" departments in these qualities. And it is in the user departments that managers have experience of making a business process work - surely this is more relevant to business requirements analysis than making a computer system work?

Merging Capability Modeling with Process Modeling

Nick Malik asks about the relationship between capability model and process modelling. "If your company is a process oriented one (or becoming a process oriented one), how do you fit in capability modeling? Do these two methods conflict?"

Nick suggests two answers. In this post I want to offer a third answer. But first we have to understand the purpose of each type of modelling.

Nick mentions two purposes for capability modelling - project planning and service design.

• Project Portfolio Alignment - "With capability modeling, you create a hierarchy of the company's capabilities. You then identify the capabilities that best align to corporate strategy, and which one of them need the most work. Focus your work there, and you get a big payoff through focused investment."

• Service Design - "The activities are where you actually perform automation. You connect services to these activities. This is where work is done."

Nick's first answer is that a capability is a top-level chunk of business, while processes are at a lower level. In a later post, Nick says this is how FEA handles capability and process modelling. As Nick acknowledges, FEA actually calls them business areas (or Lines of Business) rather than capabilities. In my 2001 book on the Component-Based Business, I called these things Business Components, and this terminology is still used by IBM in its Component Business Model. I think this is a useful construct, but that's not exactly what I call capability.

Nick's second answer is that a capability corresponds to the objectives of a specific organizational unit. Capabilities are not directly linked to processes; both capabilities and processes use and share atomic-level activities. This is better, but I am uncomfortable with the organizational tie-up.

Meanwhile, the latest version of MODAF does explicitly include "capability" as a first class construct. MODAF includes a hierarchical decomposition of capabilities, and a many-to-many mapping between capabilities and processes (which it calls "operational activities"). This is a lot closer to my notion of capability.

Another way of getting to my notion of capability is to take Nick's second answer and remove all mention of organization units. While the organization structure may give us some important clues about what an organization does and how it does it, I am careful to avoid hard-coding the organization structure into my business models.

So what is the point of modelling capabilities as well as processes? To justify capability modelling, I want to articulate two additional purposes.

• Management of Variation - Capabilities can often be decomposed into highly generic elements and asset-specific elements. (See my example of Green Coffee Beans.) Decoupling these capabilities helps to drive higher levels of cross-process and cross-organizational sharing.

• Viable System Design - A complete system needs to include management capabilities such as planning and coordination as well as operational capabilities. With a capability model, I find it much easier to check for completeness than with a process model.

So I end up with a notion of business capability which maps easily onto business services, supported by information capabilities which may be implemented as software services. This notion allows for a much finer level of granularity than Nick's first answer, but I think it is much more flexible and powerful than Nick's second answer.

The capability defines WHAT the business does; the process defines HOW the business does it (and the organization structure defines WHERE the business does it). So we expect the capability to be more stable than the process; many capabilities should be shareable not only between different processes within a single enterprise, but between multiple enterprises.

This then raises the question of capability modelling techniques - how do we analyse capabilities? We certainly need to map capabilities onto outcomes (as in Nick's second answer) , but we also need to analyse commonalities and variations within and between capabilities, as well as other dependencies between capabilities. In fact the diagram I find most useful, both for planning and for service design, is the capability dependency diagram. MODAF V 1.1 includes a suggested notation for capability dependency diagrams, but I use a slightly different notation.

Sources

• MODAF Version 1.1 (April 2007)
• L. Cherbakov et al, Impact of service orientation at the business level (IBM Systems Journal, 44/4, 2005)
• M. Dodani, The Architecture of Business (JOT 17/4, 2008)
• R. Veryard, Component-Based Business: Plug and Play (Springer, 2001)
• R. Veryard, Capability Dependency (CBDI Journal, May 2007)

Use Cases for Business Requirements

Jan Kettenis has written a long and eloquent post defending Use Cases as a method of describing Business Requirements: About Business Processes, Use Cases and SOA (1). "What I sometimes hear (and I don't exaggerate) is that a service delivers some reusable piece of functionality that cannot be pinpointed to a specific use case, so use case are useless." Jan clearly thinks this is wrong, and constructs a moderately complex example to illustrate how use cases can be useful.

I do agree that use cases can be useful - but my own view is that they are mainly useful for describing a solution (however abstract) rather than a pure requirement. In Jan's example, use cases help to specify the behaviour of an entity called Parking Services. (According to Jan, this entity is an organization rather than a system, but I don't think it makes a difference to my argument.)

But how do you decide that there should be a single entity called Parking Services in the first place? It is a design decision (business design, not software design) to cluster this behaviour together rather than distribute it across multiple entities. Because I want my description of the requirements to leave such design decisions open, I prefer to describe the requirements purely in terms of some set of outcomes for some set of stakeholders. I think there are usually many different solutions to requirements like these, and each solution can be described in terms of some collaboration between a number of entities behaving in certain ways.

Thus I accept that use cases have a place in the requirements and design process, but I think it is often a good idea to defer the production of use cases until the solution space has been adequately explored.

I think Jan actually concedes this point. "Normally a use case will not be created in one blow at this level of detail. It is more likely to be created in a few iterations of which the first one might suffice to document only the stakeholders, goal and main success scenario together with some business rules."

There is another problem with a use-case-driven approach to business requirements, and that relates to reuse/repurposing. If I start my service design from a set of use cases for the "Parking Services" entity, then there is no reason to expect these services to be of any use in any other context. Yes, a service is supposed to deliver a "reusable piece of functionality" - and so we probably don't want to restrict the use of a given service to a single use case. (Is this what Jan means by "pinpointed"?)

When I analyse business requirements, I want to think about flexibility: that means thinking about supporting not a single interaction but a range of possible interactions. For example, what are all the ways that a doctor can interact with a drug company, what are all the channels (direct and indirect) that a doctor might use to get information about a drug company's products, and how can we build rich services that support as many of these interactions as possible?

Jan ends his post by highlighting one of the most valuable consequences of documenting use cases - exposing questions and issues about the business process, which arise from the rigour of use case analysis. Rgorous analysis is always praiseworthy, but sometimes it requires just as much discipline to avoid preempting design decisions.

Business Modelling for SOA

I have developed an abstract business metamodel, which underlies the CBDI's approach to business modelling for SOA.

I have described some of the pieces of this approach in my recent articles for the CBDI Journal. It is also covered more comprehensively in the Business Modelling for SOA workshop I'm teaching for CBDI in association with Ameritas and Everware.

The emphasis is on understanding (from a business perspective) the potential decoupling of WHAT from HOW, in order to generalize the WHAT (which may be implemented as shared services) and differentiate the HOW (which may be implemented as one or more separate architectural layers).

That probably sounds a bit theoretical. I plan to describe a few examples in this blog, to show how it works in practice.

Business Intelligence Requirements

One of the problems of Business Intelligence (BI) is that you cannot define static information needs for advanced BI requirements such as data mining. Instead, you typically need to produce an entire class of enquiries ranging over all possible variables that might affect business performance.

For example, in manufacturing there are large numbers of people doing statistical process control, trying to manage derived business objects such as Process Variation, trying to identify, measure and eliminate causes of process variation. For each possible variable, or combination of variables, we may want to generate specific monitoring/measuring processes, as well as specific corrective/preventative actions.

If you want to achieve economies of scale in data mining, one of the difficulties is that you have to find systematic ways of linking different levels of abstraction within a single enquiry. For example: "which context variables are statistically correlated with these customer behaviours?" - where the object CONTEXT VARIABLE appears to belong to the metamodel rather than the model. Obviously there are ways of collapsing meta-level into the model itself, but these are fraught with danger for the logically naive: resulting in extremely clumsy class diagrams, and in error-prone or grossly inefficient implementations.

I am currently investigating something like a software factory / DSL approach to deal with the dynamic nature of emerging information needs. I'm hoping to find a safe (or at least safer) way to express specific data mining enquiries (e.g. for manufacturing or statistical process control) - reducing the potential for logical error and inefficiency, and increasing the potential for churning them out quickly.

If you have relevant ideas or experience, please contact me.



Post previously called Service-Oriented Business Intelligence, which didn't reflect the content. Name of post changed 3 April 2016