Towards Data Model Harmony

Last week, someone asked me how I go about producing a data model. I found out afterwards that my answer was considered too brief. So here's a longer version of my answer.

---

The first thing to consider is the purpose of the modelling. Sometimes there is a purely technical agenda - for example, updating or reengineering some data platform - but usually there are some business requirements and opportunities - for example to make the organization more data-driven. I prefer to start by looking at the business model - what services does it provide to its customers, what capabilities or processes are critical to the organization, what decisions and policies need to be implemented, and what kind of evidence and feedback loops can improve things. From all this, we can produce a high-level set of data requirements - what concepts, how interconnected, at what level of granularity, etc. - and working top-down from conceptual data models to produce more detailed logical and physical models.


But there are usually many data models in existence already - which may be conceptual, logical or physical. Some of these may be formally documented as models, whether using a proper data modelling tool or just contained in various office tools (e.g. Excel, PowerPoint, Visio, Word). Some of them are implicit in other documents, such as written policies and procedures, or can be inferred ("reverse engineered") from existing systems and from the structure and content of data stores. Some concepts and the relationships between them are buried in people's heads and working practices, and may need to be elicited.

And that's just inside the organization. When we look outside, there may be industry models and standards, such as ACORD (insurance) and GS1 (groceries). There may also be models pushed by vendors and service/platform providers - IBM has been in this game longer than most. There may also be models maintained by external stakeholders - e.g., suppliers, customers, regulators.

There are several points to make about this collection of data models.

• There will almost certainly be conflicts between these models - not just differences in scope and level/ granularity, but direct contradictions.

• And some of these models will be internally inconsistent. Even the formal ones may not be perfectly consistent, and the inferred/ elicited ones may be very muddled. The actual content of a data store may not conform to the official schema (data quality issues).

• You probably don't have time to wade through all of them, although there are some tools that may be able to process some of these automatically for you. So you will have to be selective, and decide which ones are more important.

• In general, your job is not simply to reproduce these models (minus the inconsistencies) but to build models that will support the needs of the business and its stakeholders. So looking at the existing models is necessary but not sufficient.

So why do you need to look at the "legacy" models at all?  Here are the main reasons.

• Problems and issues that people may be experiencing with existing systems and processes can often be linked to problems with the underlying data models.

• Inflexibility in these data models may constrain future business strategies and tactics.

• New systems and processes typically need to transition from existing ones - not just data migration but also conceptual migration (people learning and adopting a revised set of business concepts and working practices) - and/or interoperate with them (data integration, joined-up business).

• Some of the complexity in the legacy models may be redundant, but some of it may provide clues about complexity in the real world. (The fallacy of eliminating things just because you don't understand why they're there is known as Chesterton's Fence. See my post on Low-Hanging Fruit.) The requirements elicitation process typically finds a lot of core requirements, but often misses many side details. So looking at the legacy models provides a useful completeness check.

If your goal is to produce a single, consistent, enterprise-wide data model, good luck with that. I'll check back with you in ten years to see how far you've got. Meanwhile, the pragmatic approach is to work at multiple tempos in parallel - supporting short term development sprints, refactoring and harmonizing in the medium term, while maintaining steady progress towards a longer-term vision. Accepting that all models are wrong, and prioritizing the things that matter most to the organization.

The important issues tend to be convergence and unbundling. Firstly, while you can't expect to harmonize everything in one go, you don't want things to diverge any further. And secondly, where two distinct concepts have been bundled together, trying to tease them apart - at least for future systems and data stores - for the sake of flexibility.

Finally, how do I know whether the model is any good? On the one hand, I need to be able to explain it to the business, so it had better not be too complicated or abstract. On the other hand, it needs to be able to reflect the real complexity of the business, which means testing it against a range of scenarios to make sure I haven't embedded any false or simplistic assumptions.

---

Longer answers are also available. Would you like me to run a workshop for you?

---

Wikipedia: All Models are Wrong, Chesterton's Fence

Related posts

How Many Products? (October 2004), Modelling Complex Classification (February 2009), Deconstructing the Grammar of Business (June 2009), Conceptual Modelling - Why Theory (November 2011)


Declaration of interest - in 2008(?) I wrote some white papers for IBM concerning the use of their industry models.

Knowledge and Memory

Once upon a time, people thought of an information model as defining the structure of the stuff you want to remember. Nowadays, this definition is too restrictive: it might possibly be adequate for a system/database designer, but is not adequate for an architect. 

Here are three examples where my knowing and using information is not dependent on my remembering it.

1. I recently helped an SME with its PCI DSS submission. One of the critical points is that they avoid a lot of trouble by NEVER storing their customers' credit card details, merely transmitting these details directly to Barclaycard using a secure device. Clearly the credit card details are stored in various places elsewhere, but our systems don't store them anywhere, not even in cache.

2. When I did my driving test, many years ago, the Highway Code had a table of stopping distances that you were supposed to remember. I decided it would be easier to remember the formula and calculate as required. Obviously that's a design decision.

3. My computer remembers frequently used email addresses. But if I want to get back in touch with someone, I will use Linked-In or Google to find his current email address rather than use the one on my computer which may be out-of-date.

The system/database developer and the architect may use the same patterns for defining an entity or entity type, but they have different agendas, and therefore different success criteria. The architect may be less rigorous about some aspects, and needs to be more rigorous about some other aspects.

The system/database developer may see an entity as "something you need to remember etc.". An architect sees an entity as "something you need to exchange information about". The information model establishes a consistent basis for information exchange between people, systems, services, processes and organizations. I can only talk with you about customers, share your customer data, use your customer-related services, and so on, if either (a) you and I have the same understanding of CUSTOMER or (b) there is a mapping between your notion of CUSTOMER and mine. We can call this semantic interoperability.

This idea of semantic interoperability underlies the Open Group vision of Boundaryless Information Flow.

If you have a private list of customers on your laptop, then you can identify them in any adhoc manner you choose. But if you want to share the list with other people, the identity criteria become important.

So which entity types is the architect is most interested in? Primarily the ones that are referenced in the interfaces between systems and services. There are lots of other things that you might wish to remember, monitor and/or direct, but if they can be encapsulated inside some service or component they have no architectural significance. For example, the business needs to know the prevailing VAT rate; so you build a common VAT routine with the VAT rate hidden inside.

Conceptual Modelling - Why Theory

@dougnewdick asks Why Bother With Concepts? In his answer, he quotes the Gang of Four song Why Theory. Each day seems like a natural fact/but what we think changes how we act.

I think there is a critical difference between two interpretations of the term Conceptual Architecture. I'd describe Doug's sample conceptual architecture as essentially a component architecture, which happens to be at the conceptual level of abstraction. In other words, it provides a conceptual understanding of how the components work together.

That's not at all the same as understanding how the concepts themselves work together, which is what I think one needs to satisfy the Gang of Four requirement for an architecture-of-concepts, and which would also be implied by Doug's reference to the Wikipedia article on Philosophical Analysis.

The Wikipedia article mentions critiques of the traditional approach to philosophical analysis by Quine and Wittgenstein. I have used some of their ideas - although with no claim to profundity - in my own approach to conceptual modelling. See for example my 1992 book on Information Modelling.

To take a concrete example, if we are going to build systems and management practices to anticipate competitor behaviour, it might help have to have a robust concept of COMPETITOR. I once worked with a company that thought it knew which its competitors were, and were quite indignant when a competitive threat appeared from an entirely different quarter. Traditional philosophical analysis implies you can fully characterize a concept like COMPETITOR in advance, but Wittgenstein and Quine teach us to be cautious of monothetic, appearently objective conceptual definitions.


One of the problems with conceptual models is that they are often too generic, and therefore insufficiently meaningful. Anyone who has ever been on a data modelling course can produce a model with CUSTOMER and ORDER and PRODUCT. But proper conceptual analysis entails taking concepts apart to understand how they work in a particular business context. Where do products come from, what does it mean to be a product, how do products evolve, what are the essential and non-essential differences between products? Why do we have this many products, and what are the forces that would result in increasing or decreasing the number of products? What does the business know about products, and what does the business want to know?

Doug's post was prompted by Peter Bakker's post Factual Architecture, which expressed a concern that concepts can be too abstract and unverifiable – failing to connect with audiences, failing to connect with reality. The way that some architects try to verify concepts is to demonstrate how the concepts could be represented in some technical system - but I prefer to encourage stakeholders to verify concepts in the business domain itself. This leads me to a slightly different notion of Factual Architecture, which would provide an empirical business context in which the business concepts and knowledge can be grounded.

Is this really the way it is? Or a contract in our mutual interest? (Gang of Four: Contract)

See also Big Data and Organizational Intelligence (November 2018)

Coherence Premium

Interesting article by Paul Leinwand and Cesare Mainardi, The Coherence Premium. HBR June 2010 http://www.booz.com/media/uploads/HBR_Coherence_Premium.pdf

The authors define the "coherent company" as one that "has aligned its differentiating internal capabilities with the right external market position". They claim that most companies lack coherence because they pay too much attention to the external and not enough to the internal. "We are suggesting that companies start from the opposite direction, figuring out what they're really good at and then developing those capabilities (three to six at most) until they're best in class and interlocking. From there, strategy becomes a matter of aligning that distinctive capabilities system with the right marketplace opportunities."


We may observe in passing that enterprise architects are often criticized for the opposite tendency - paying too much attention to internal structure without knowing how to align this to external demand. 



The authors offer two examples of enterprise models that could be expressed as an interconnected set of capabilities, although the detail of the interconnections are not described in the paper.

Retail (based on Wal-Mart)

• vendor management
• point-of-sale data analytics
• logistics
• working capital management

Consumer healthcare (based on Pfizer)

• Science-based innovation (I interpret this to be "technology" rather than "product")
• Influencing regulation and government policy
• New product development (including licensing and acquisition)
• Claims-based marketing (in other words, giving the consumer verifiable and relevant chunks of knowledge)
• Channel management
• Portfolio management

The authors claim to have a procedure for quantifying something they call "Coherence Premium" for a company.

1. Define the segments the company serves
2. Identify the capabilities that drive value for the company in each segment
3. Determine the number of common capabilities across all the segments the company serves.

They have calculated the coherence premium for a number of large companies including Campbell's, Clorox, Coca-Cola, ConAgra, General Mills, Heinz, Kimberly-Clark, Kraft, Nestlé, PepsiCo, Procter&Gamble, Sara Lee, Unilever and Wrigley's. These are then mapped against EBIT margin%, showing a very rough correlation.

Unfortunately, however, this graph doesn't show the two companies that they examined in greatest detail earlier in the paper, namely Wal-Mart and Pfizer. This may be because these companies face complex multi-sided markets, and this would cause difficulties for their simple procedure. It would seem that the kind of coherence they praised in these two companies would not be adequately represented by their simplistic "coherence premium" metric.


There have been many previous attempts to address the issues raised by Leinwand and Cesare Mainardi. In her blogpost Coherence - the new alignment, commenting on a Booz presentation of their concepts, Naomi Stanford recalls an earlier book on The Power of Alignment by George Labovitz and Victor Rosansky. Also see the discussion of cohesion costing in Nicholas Whittall and Philip Boxer, Agility and Value for Defence (pdf).

Value of models 2

#entarch What are enterprise models for? What is their purpose, status, perspective, scope, meaning?

Even the scope isn't as straightforward as you might think. Okay, the enterprise model is usually supposed to be "enterprise-wide" - but how wide is that exactly? Do customers and supply chain partners belong to the enterprise for modelling purposes, or not? (Don't tell me you can resolve such questions by looking at the organization chart, because that's just another enterprise model, so you're just shifting the problem.) And even if we know how wide, we still need to know how deep. For some purpose, from some perspective.

What about the status of an enterprise model - in other words, what kind of knowledge it claims to represent? Is it supposed to be a simplified description of the current state of the enterprise (AS-IS)? Or an idealized blueprint of some future state of the enterprise (TO-BE)?

Enterprise architects typically produce both AS-IS and TO-BE models, and then produce transition plans to get from AS-IS to TO-BE. This is therefore an exercise in requirements engineering at the enterprise level. The difference between AS-IS and TO-BE can be analysed from many perspectives including engineering (technical design and specification), accounting (cost estimation and financial justification) and sociotechnical (human systems management, sometimes disdainfully referred to as "implementation factors").

In the past I have talked a lot about this process, and I may have created the false impression that the sole purpose for modelling is to support this process.

But enterprise models have another very important purpose. There is a collective management need to make sense of the complexities of the business, and so an enterprise model is a sense-making tool as well as an engineering tool.

For example, an enterprise model might give us a picture of an organization as a set of conversations and collaborations, thus helping the participants in these conversations and collaborations to make them more systematic and focused,

Looking at this model from an engineering perspective, we might identify opportunities to install more sophisticated mechanisms if that's justified, but often it's about making better use of the existing mechanisms.

If the enterprise architect always looks at the enterprise and its models from an engineering perspective, then "better use of the existing mechanisms" may not seem very important.

But I see this as part of the job of enterprise architect – not just fixing the mechanisms but understanding and advising on their effective use. The enterprise architect must therefore be capable of looking at things from a sociotechnical or business perspective, not just an engineering perspective.

If our purpose here goes beyond conventional EA practice, then the requisite scope and focus of the enterprise models changes as well. Conventional enterprise models are dominated by the operational processes, with management information typically regarded as mere superstructure. If we want to improve the manageability and intelligence of the whole organization as a complex sociotechnical system of systems, we need to be interested in the management system - the information flows and coordination mechanisms and feedback loops - not just the operational system.

An enterprise model is therefore a map to support this kind of investigation, as well as a sense-making map to be used within the management system itself, not just a blueprint for reengineering the enterprise.

---

Thanks to Sally Bean and Jon Durrant for useful discussion.
See also Value of Models 1.

Model-Driven Architecture?

@greblhad suggests building Lego models as a technique for enterprise architecture: "If your #entarch can not be illustrated in 3D using LEGO then you have a problem".

The reason Lego cannot reliably express architecture is because just looking at the model doesn't tell us which elements are architecturally significant. For example, if a particular Lego model has a row of small bricks over the window instead of larger ones, and uses green bricks for the base, how can we know whether the architect actually specified this detail, or whether the architect left these details unspecified and the person building the model simply used the bricks that were available? In other words, the same physical model could have been produced by two different architectures: one including the instruction "Use small bricks over the window and green bricks for the base", and one including the instruction "Don't buy more bricks if you don't have to". (The latter being derived from one of the popular architectural principles that supposedly drive architectural practice.) Note that in this example, the two possible instructions are at two different logical levels, and the illustration may mislead us about the architect's real intentions. For a different example, see my post What if architects designed our communities?

At its best, LEGO (even with people figures, as suggested by @tetradian) shows a momentary instantiation of an architecture, and is therefore at the wrong level of abstraction.

I accept that we are not talking about using Lego models as a specification tool. But illustration only works if you know what is being illustrated, as Wittgenstein pointed out (Philosophical Investigations). Furthermore, the representational nature of the model may be problematic. If Lego only has a limited palette of colours, do we interpret green to mean any shade of green, or that particular shade? Does the use of green bricks in the model indicate the use of green bricks in the planned structure, or does it have some other meaning, for example indicating the need for specially treated bricks? And so on.

@greblhad was inspired by LEGO's new "product", called LEGO® Serious Play®. Perhaps enterprise architecture needs creative play as well as rigorous specification, but it would be surprising if the same tools and methods supported both.

Deconstructing The Grammar of Business

@JohnIMM (John Owens) trots out a familiar piece of advice about data modelling today.

"Want to know what data entities your business needs? Start with the nouns in the business function names."

Starting with the nouns is a very old procedure. I can remember sitting through courses where the first exercise was to underline the nouns in a textual description of some business process. So when I started teaching data modelling, I decided to make this procedure more interesting. I took an extract from George Orwell's essay on Hop-Picking, and got the students to underline the nouns. Then we worked out what these nouns actually signified. For example, some of them were numbers and units of measure, some of them were instances, and some of them were reifications. (I'll explain shortly what I mean by reification.) Only a minority of the nouns in this passage passed muster as data entities. Another feature of the extract was that it used a lot of relatively unfamiliar terms - few of us had experience measuring things in bushels, for example - and I was able to show how this analytical technique provided a way of getting into the unfamiliar terminology of a new business area. I included this example in my first book, Pragmatic Data Analysis, published in 1984 and long out of print.

One problem with using this procedure in a training class is that it gives a false impression of what modelling is all about. Modelling is not about translating a clear written description into a clear diagrammatic structure; in the real world you don't have George Orwell doing your observation and writing up your interview notes for you.

Now let me come on to the problem of reification. The Zachman camp has started to use this word (in my view incorrectly) as an synonym of realisation - in other words, the translation and transformation of Ideas into Reality. (They claim this notion can be traced back to the ancient Greeks, but they do not provide any references to support this claim. As far as I am aware, this is a mediaeval notion; it can for example be found in the work of the Arab philosopher ibn Arabi, who talks about entification in apparently this sense.) However, modern philosophers of language use the word "reification" to refer the elevation of abstract ideas (such as qualities) to Thingness. One of the earliest critics of reification was Ockham, who objected to the mediaeval habit of multiplying abstract ideas and reified universals; his principle of simplicity is now known as Ockham's Razor.

In our time, Quine showed how apparently innocent concepts often contained hidden reification, and my own approach to information modelling has been strongly influenced by Quine. For example, I am wary of taking "customer" as a simple concept, and prefer to deconstruct it into a bundle of bits of intentionality and behaviour and other stuff. (See my post on Customer Orientation.) As for business concepts like "competitor" or "prospect", I generally regard these these as reifications resulting from business intelligence.

Reification tends to obscure the construction processes - tempting us to fall into the fallacy of regarding the reifications as if they directly reflected some real world entities. (See my posts on Responding to Uncertainty 1 and 2.) So I like to talk about ratification as a counterbalance to reification - making the construction process explicit.

Of course, John Owens is right insofar as the grammar of the data model should match the grammar of the process model. And of course for service-oriented modelling, the grammar of the capabilities must match that of the core business services. But what is the grammar of the business itself? Merely going along with the existing nouns and verbs may leave us short of discovering the deep structural patterns. 

 

Update May 2024. The distinction I'm making here between reification and its opposite, which I've called ratification, can be compared to Simondon's distinction between ontology and ontogenesis, so I shall need to write more about that. Meanwhile, I now acknowledge the possibility that some notion of reification might be found among the Neoplatonists but that's several hundred years after Plato himself.

Related posts: Reification and Ratification November 2003), Business Concepts and Business Types (May 2009), Business Rule Concepts (December 2009), The Topography of Enterprise Architecture (September 2011), Conceptual Modelling - Why Theory (November 2011), From AS-IS to TO-BE (October 2012), BankSpeak (May 2015), Mapping out the entire world of objects (July 2020)

Business Concepts and Business Types

cross-posted from SOA Process blog

---

One thing I keep getting asked why the CBDI Service Architecture and Engineering method distinguishes between the Business Concept Model (describing things as they are in the real world) and the Business Type Model (describing things as they are represented within the enterprise systems).

Here's a simple example. As I mention in my post Will Libraries Survive? there is a conceptual difference between book title and book copy. Some libraries identify each physical copy individually, while other libraries merely count the physical copies of a given book title. The physical copies are distinct in the real world, but may be indistinguishable in the library systems. Information strategy includes making this kind of choice.

Let's suppose a library buys ten copies of a book. To start with these copies are physically indistinguishable. The library then attaches some identifying marks to the book, including an ID number. This marking depend on the information strategy - the library could choose to give all ten copies the same number or could choose to give each copy a different number.

If we assume that this ID number is used throughout the library systems, then it is clearly important to know whether the ID number identifies the book title or the book copy. So this is a critical element of the Business Type Model, which reflects a whole series of strategic decisions of this nature, and then feeds into the identification of core business services.

One important form of business improvement is to increase the differentiation in the Business Type Model. For example, a library that previously didn't distinguish between physical copies could decide to make this distinction in future. But this business improvement doesn't change the underlying reality of books, so the Business Concept Model stays the same.

Extract from Business Information Improvement (CBDI Journal, May 2009)

Will Libraries Survive?

Following my post on Library Collaboration, Anders Jangbrand commented

"Book available on net do not have same limitation. No phys copies. Will libraries survive?"

That's an excellent question for two reasons. Anders always asks good questions, but another reason I like this one is because I am already working on an answer. But first a little history. 

In the dim and distant past when I first learned data modelling, one of the standard exercises in books and training courses was to model a library. These were of course models of the library as an information processing system. Before computing, libraries were managed with cabinets full of hand-written cards. There would be one or more cabinets containing the library catalog, there would be cabinets for members' names and addresses, outstanding loans, reservations, books awaiting repair, and so on. 

So it was an apparently straightforward task to model this lot; but there are some hidden traps for the beginner. For example, BOOK sometimes means the book title (as when you reserve a book) and sometimes means the physical copy (which is what you borrow and return). When I teach data modelling, I generally give students the freedom to fall into these traps so I can show them how to get out again. 

(Some libraries identify each physical copy individually, while other libraries merely count the physical copies of a given book title. The physical copies are distinct in the real world, but may be indistinguishable in the library's card or computer systems. Information strategy includes making this kind of choice. See Business Concepts and Business Types to see how this is handled in the CBDI SAE method.) 

The great advantage of using a library as a teaching example was that most people had used libraries and had some idea how they operate. However, some people started to think that the example was a bit old-fashioned, so they developed a Video Rental example instead. Video rental has pretty much the same information processing structure as a library, so all they needed to do was take the library example and change some of the words on the diagrams. And nothing much changed when video tapes were replaced by DVDs, although there were some minor complications if you wanted to handle both tapes and discs during the transition period. 

 The traditional library is indeed threatened, but it will probably outlive video rental, which is threatened by much the same forces to an even greater extent. If we want to manage these forces, we need to move away from an information processing model onto a different kind of model. 

Libraries and video rental operate a very similar information processing model. But if we want to think about the survival of libraries and video rental, we need a model that shows how libraries and video rental are different from one another, and to what extent libraries or video rental can play a valuable role in the service economy of the future. 

The first point to note here is that the book plays a much more varied role in people's lives than the video, even today. The vast majority of DVDs are either feature films or collections of TV programmes, consumed as entertainment, and video rental essentially caters for this market. Books are also consumed for entertainment, but they are also used for reference, study, research and other purposes, and most libraries support a broad range of purposes. If we want to forge a sustainable role for the library of the future, we need to engage with these purposes, and possibly explore some newly emerging purposes as well. 

Some people may argue that the business model should be technologically neutral. Anything you can do with a book, you can do with a DVD as well. For example, if you want to learn Spanish, it shouldn't matter whether you borrow a book from the library or rent a DVD. But until video rental gears itself up to support this kind of market, there will continue to be a real difference in the business model between books and DVDs. 

The next point to note is that we can open up the idea of the consumer. A lot of people like to read books in groups. The whole group reads the same book (see ambiguity of BOOK above), and then the members meet once a month to discuss the books. This represents an interesting opportunity for the librarian - to provide services to the whole reading group rather than to individual readers. For example, if the library has a dozen copies of a recent novel, then this novel can be offered to the local reading group for next month's meeting. 

Now suppose we have a network of a couple hundred libraries around the region, supporting a thousand reading groups between them. Recent novels can circulate around the libraries in sets, to support the needs and interests of the reading groups. 

 See what we've done here. All sorts of interesting business opportunities emerge by these two conceptual shifts. Firstly changing the concept of READER from individual to reading group (and creating a new composite service for the whole group). Secondly changing the concept of LIBRARY from single library to a network of libraries (and creating new kinds of collaborative process). 

Once we've identified these conceptual shifts, we can go back into the information processing model to work out the practical logistics. But the point I'm making here is that conceptual shifts of this kind (and the business opportunities that are associated with them) don't appear unless you step outside the information processing perspective and look at the business through a different lens. 

Of course I haven't quite answered Anders' question yet. But I'm working towards an answer ...

Decision-Making Services

James Taylor (ebizQ) explains why we need a separation between process and decision.

• Interesting debate on business process and decisions (April 2009)
• Decision services (Jan 2007)

One of the reasons for this separation is that decisions are subject to business policies or rules, and such policies are often subject to change over time and/or variation between organizational units. If you code the decisions into process logic, perhaps using BPML to generate process services, then you will have to modify the code or BPLM script whenever the policy changes, and implement different versions for different parts of the organization.

But there are some more flexible ways of architecting decision services.

• General-purpose rule engine, driven by a standard policy language.
• Clusters of related business decisions (for example planning and scheduling) can be wrapped into a capability service.

The advantage of these approaches is that they help to separate the generic parts of the process from the specific parts, and thus increase reuse and sharing of the generic parts, while increasing accuracy and differentiation in the specific parts.

In his post on business processes and decision services, James discusses some of the technical issues that arise with this separation. It is certainly true that some technical optimization may be desirable in some situations, and this may mean implementing and deploying the process and decision services together as a fairly tightly coupled unit of software (which CBDI SAE calls an Automation Unit). But at the specification level, the logical services should still remain distinct.

For more examples of policy separation see my articles on Business Modelling for the CBDI Journal, which are archived on the Everware-CBDI website http://everware-cbdi.com/cbdi-journal-archive.

Enterprise as a Network of Events 2

Following my post on Enterprise as a Network of Events, Nigel Green drew my attention to something he wrote a couple of years ago on the case for a clear distinction between events and content, in which he points out that "aggregated events become content over time".

As it happens, I touched on something like this in my 1992 book on Information Modelling. At that time, the specific question I was addressing was how to represent the history of organizational change in a data model. There are two basic options.

1. History is represented as a series of snapshots of the organization at specific times. If you want to know what the change was, you just have to compare two snapshots. (For example, this is how page history is represented on Wikipedia.)
   
2. History is represented as a series of changes (differences). Then if you want to know what the state of the organization was at a given time, you just have to aggregate the changes forwards or backwards from a fixed point.

From a purely logical point of view, it doesn't matter which of these two representations you adopt, because you can always derive one from the other. In practical terms, however, you may find that one representation is a lot easier to work with than the other.

What both representations have in common is a definition of information in terms of difference: a difference that makes a difference, as Bateson puts it.

Information as Difference

Hold your hand perfectly still, palm upwards and resting comfortably on a table. With your other hand, drop a small coin into the palm. You will feel the impact, and if the coin is cold, you will feel the coldness of the metal. Soon however, you will feel nothing. The nerve cells don't bother repeating themselves. They will only report to the brain when something changes. Information is difference.

A lizard hunting insects operates on the same principle. The lizard's eye only reports movement to the lizard's brain. If the hunted insect settles on a leaf, the lizard literally cannot see it. But the moment the insect starts to move, whop, the lizard can see it again, and the tongue flickers out and catches it.

But there are differences and differences. Information is difference that makes a difference. You were probably aware, as you dropped the coin into your palm, your eyes told you automatically, without your brain even asking, what the value of the coin was; but you were probably not aware what date it was minted. This is because (unless you are a numismatist) the value of the coin makes a difference to you whereas its date doesn't.

What is it that makes a difference to a lizard, to a numismatist, to you? Surely not the same things. What is information for the lizard is not information for you, and what is information for you is not information for the lizard.

This is why the perspective of information is important. Perspective defines what counts as information at all, perspective defines to whom the information makes a difference.

Enterprise Strategy and Difference

Now what's this got to do with the enterprise? There are two important differentiation questions for any enterprise: how do They differentiate Us, and how do We differentiate Them.

Firstly how do They differentiate Us. In other words what differences in our products and services or organization are (a) going to be noticed by external stakeholders such as customers and (b) going to affect the behaviour and attitudes of these stakeholders.

Sophisticated marketing organizations pay a lot of attention to this kind of thing. Does it make a difference if we put this in a blue envelope? Does it make a difference if we have a female voice here? Sometimes you have to experiment with differences that don't matter, in order to have a refined view of what differences do matter.

Secondly, how do We differentiate Them. What are the strong signals in the environment that we must respond to, and what are the weak signals we might respond to? An organization that responded to every weak signal would be impossibly unstable, so most organizations have operational processes that respond to strong signals, and some form of business intelligence that scans the environment for weak signals and identifies a subset of these signals demanding some kind of response.

Sometimes the appropriate response to some detected difference or discrepancy is some kind of business improvement, such as business process improvement and/or system engineering. Which means that the history of the organization is inscribed into its current structure and processes.

"What's the purpose of this process step?"

"We've done that extra check ever since we got badly burned in the XYZ deal five years ago."

If we knew enough (which of course we don't), we might conceivable infer the current state of the organization from a careful analysis of all the events that have ever happened to it. That would join this post back to where we started. Neat but impractical.

But that's not actually what I'm trying to do. I think understanding an organization as an information-processing system helps us to do two practical and concrete things.

• Diagnose the current problems and opportunities facing a complex organization.

• Design an event-driven business improvement process, based on business intelligence.

My next question is: what kind of business model supports these challenges.

Enterprise as a Network of Events

From the early days of SOA, we've talked about understanding the enterprise as a network of services, but clearly this is not the only possible viewpoint. Can we understand the enterprise as a network of events?

In a recent post SOA and Events - Two Sides of BAM?, Ramesh Loganathan (Progress) explained how a process view of the enterprise could be flipped into an event-oriented view. Ramesh's example is in governance risk and compliance (GRC), but it looks as if the same idea could be applied to a much broader range of business requirements.

Meanwhile, in Decision Management and software development, James Taylor wants to model the enterprise as a series of decisions. But of course a decision can be understood as a special kind of event. See Model Driven Engineering and Event Processing by Opher Etzion (IBM). See also SOA is dead; long live Model-Driven SOA by Johan den Haan.

In his post Event, Rules, Processes, Decisions, Paul Vincent (TIBCO) identifies an important gap in the notations (BPMN, BMM) available for modelling these aspects of the enterprise. He talks about decisions in terms of rules, but I presume he is associating the rule with the decision process (this is how we authorize transactions) rather than with the specific instance of decision (I hereby authorize this transaction).

At the strategic level, we need to understand the relationship between an external set of possible events and an internal set of possible events. Each enterprise is capable of responding in certain ways to certain classes of external events. Making an enterprise more agile means enabling the enterprise to mobilize more appropriate responses to a greater range of external events, without proliferation of unnecessary complexity. In system thinking this is called Requisite Variety.

So if we think about the enterprise as a network of events, this gives us a direct way to reason about strategic business improvement. But what are the CEP vendors doing in this space? I can see a lot of talk about specific event-driven applications, but where is the architectural thinking about the event-driven enterprise?


Update: After I posted this, Nigel Green drew my attention to something he wrote a couple of years ago on the case for a clear distinction between events and content, which he describes in terms of the wave-particle metaphor.

Business Capability Modelling

Picked up some fragments of a discussion between Colin Jack and Udi Dahan (Twitter, March 3rd-4th).

Colin Jack: Somehow our attempts at business capability modelling ended up producing a list of entities (Customer/Order). Meh.

Udi Dahan: When doing BC modeling, focus more on use cases

Colin Jack: I think I see what you mea, but I still think complexity cost of BC's is rel high so sometimes modules are enough

Udi Dahan: Complexity of BCs isn't that high - simple, straightforward pub/sub

I may be missing some of the pieces and context that might make these fragments more intelligible, but I'm going to comment anyway, on what I imagine they might be talking about.


A key challenge of any kind of business modelling or system model is to decompose into things that make sense. For many analysts, the things that make most sense are ... things. Objects. Entities. Assets.

If these things are important to the business, they need to be looked after. So when Colin identifies Customer and Order as business capabilities, he presumably means customer-management and order-management. The customer-management capability is typically going to be responsible for discovering and managing all aspects of all instances of CUSTOMER; and the order-management similarly for all instances of ORDER.

(Gunnar Peterson adopts a similar approach for security requirements - understanding security as the protection of a range of assets. Thus security capabilities are all asset-related. See my discussion Business Model Drives Security.)

Is there anything wrong with this as a capability model? Well, it tells us what needs to be managed or protected, but doesn't exactly tell us what needs to be done.

Udi offers a radically different approach: look at the process model. Not the whole processes, but broken down into Use Cases. What we'd expect from this approach is a much larger number of much smaller and simpler capabilities, and this is confirmed by the exchange between Colin and Udi.

When I do business capability model, I want to identify capabilities with certain characteristics - meaningful and self-contained chunks, with high cohesion and low coupling. Generally this means looking at both the entities and the process. In a complex situation it is useful to draw a matrix of the interactions between entities and processes, in order to work out the clusters.

There is growing interest in capability modelling and capability management, including through-life capability management, and the capability model is a powerful tool for thinking strategically about the loosely-coupled enterprise, so I guess I shall need to write some more on this topic soon.

Modelling Identity and Context

Language is Strange says Don Ferguson. He's talking about the curious difference in Spanish between ser and estar.

• ser is used for permanent characteristics (identity). For example, Don is American, I am British.

• estar is used for transient characteristics (context). I don't know exactly where Don is at the moment, but I'm in London. Don is now working for CA; he was working for Microsoft, and IBM before that.
  

This is an important distinction to make when designing information systems and information services. It affects the way databases and database keys can be designed (if it can change you shouldn't use it as a permanent identifier) and it affects the set of operations that have to be designed (if it can change, you have to be able to change it).

The question is: how do you decide whether something is permanent or transient. As Don points out, some of the distinctions in Spanish may seem counter-intuitive to an English-speaker. Location is always assumed to be transient, even for fairly solid things like the American Embassy, while origin (as in "I'm from Massachusetts") is assumed to be permanent.

When you are modelling a large complex enterprise, you really don't want to be working this kind of thing out on a case-by-case basis. So it is generally good enough to adopt some broad patterns (e.g. LOCATION is always going to be transient) even if it occasionally looks wrong. (It's usually better to err on the side of allowing something that isn't always going to be needed, rather than banning something someone might want.)

The "good enough" strategy is known as "satisficing". It may not be as perfect as full optimization, but it's a lot quicker. Clearly Spanish is good enough.

Modelling Complex Classification

Andrea Westerinen (Microsoft) posts some modelling guidelines, and she was told that some people's heads exploded when reading them.

• What makes for a "good" model/ontology
• Continuing "What makes for a "good" model?"

She identifies three fundamental modelling concepts, which she draws from the work of Guarino and Welty.

• Essence - these are properties that are true for all instances of a class and that "define" the semantics of the class, such as the "property" of "being human"
• Identity - properties that determine the equality of instances
• Unity - properties that define where the boundary of an instance is, or distinguish the parts of a "whole"

In my work on information modelling (for example in my 1992 book) I have long emphasized the importance of understanding semantic identity (how does something count as being "the same again") and semantic unity (which I tend to call membership - how does something count as inside or outside). 

But I have been critical of the assumption that we always define a class in terms of essential properties. This is known as monothetic classification, and can be contrasted with polythetic classification, which defines a class in terms of characteristic properties. As I teach in my information modelling workshops, many important objects of business attention are not amenable to simple monothetic classification (for example how does a commercial firm decide who counts as a COMPETITOR, how does the police decide who counts as a SUSPECT) and require a more complex logic. 

If you are just building transaction-based systems and services, you may be able to fudge these semantic issues. But if you want information services to support business intelligence and decision-support as well as transaction systems, then you have to get the semantics right. 

Of course I can see why Guarino, Welty and Westerinen want to insist on monothetic classification (which they call rigidity). But then how can they model the more fluid and fuzzy (and I think more interesting) business requirements? 

 (Sometimes the practitioners of "ontology" like to think that they are dealing with supremely abstract and generalized stuff, but if they make too many simplifying assumptions of that kind then their ontologies aren't sufficiently generalized after all.)

 

---

Rodney Needham, Polythetic Classification: Convergence and Consequences (Man, 10:3, September 1975), pp. 349-369. 

Richard Veryard, Information Modelling - Practical Guidance (Prentice-Hall 1992) pp 99-100

Stanford Encyclopedia of Philosophy: Wittgenstein on Family Resemblance

Wikipedia: Family Resemblance

Hypothetical Business Models

Robin Meehan (Smart421) asks a question about modelling time sequence dependencies in EA modelling tools

"I often want to model ‘what if’ scenarios, i.e. what the various architecture domains would look like if we proceeded with Option A, or B etc ... [but] I don’t want my various different options to ‘bleed’ into each other in the model. ... The only way I’ve found round this in the past is to duplicate parts of my model to keep them separate (in different packages etc) - which is a bit ugly."

Of course, the problem Robin identifies is more than just time sequence dependencies. Sometimes we have to explore the differences between A-followed-by-B and B-followed-by-A, but there may be many other permutations to explore. If you have to duplicate parts of the model for each permutation, this severely limits the number of permutations you can compare.

Many architects will try to solve this kind of problem by abstraction - coming up with a model that abstracts away from the differences between the options, so that the model supports all the possible options and all the possible permutations between the options. However, the standard model isn't then much good for comparing the options. In some cases it may be worth building a flexible capability that can support both A and B, but how do I tell whether this is one of those cases?

(Michael Bell's SOMF modelling approach does allow AS-IS and TO-BE to be shown on a single diagram, but I don't know how practical it is to put multiple alternatives on a single diagram.)

What Robin highlights here is that the standard modelling tools and notations aren't always very helpful at supporting choice, especially when the choices get at all complex. Often it seems that the tools only really work for documenting the architectural and design choices that have already been made, rather than for making the choices in the first place.

The best answer I can give Robin is to use the available tools, but to adopt a different modelling style. But I need a good case study to work on. Contributions welcome ...

Two Kinds of Business Model

Cross-posted to the new CBDI Forum group on Linked-In.

---

Am currently pulling together material on business improvement for an article in the CBDI Journal. Any ideas, contributions, questions, examples, challenges, pitfalls, or other input welcome.

One of the first problems I've been facing is the question of the Business Model. It has become clear to us that there are (at least) two entirely different notions of Business Model.

The business notion of Business Model is The-Way-We-Do-Business, What-Distinguishes-Us-From-Our-Competitors. For example, Pile-Em-High-And-Sell-Em-Cheap or Ship-Em-Quick-But-Collect-Cash-Quicker. I call this Business Model (B). That's what any normal person outside IT thinks a business model is.

The computer notion of Business Model is a collection of line-and-box diagrams, such as data diagrams or process flow diagrams, drawn in IDEF or ER or UML or BPMN or ArchiMate or something like that, together with rigorous semantics for each of the elements on the diagram, which describe the conceptual structure of the business. I call this Business Model (C). These diagrams typically abstract away from many of the things that differentiate an enterprise from its competitors.

Now it is perfectly clear that many business and IT improvements don't involve any change to Business Model (C) whatsoever. For example, a major data cleansing exercise may involve a lot of work for the computer department, and produce some significant benefits for the business, but unless the database schema is altered then there is no need to change Business Model (C).

Incidentally, lots of SOA case studies seem to fall into this category. For example, reducing cycle time or latency probably doesn't change any of the UML diagrams, it merely changes the service level associated with one of the elements depicted on one of the diagrams.

Some business improvements do involve change to both Business Model (B) and Business Model (C). For example, when airlines went over from unique flight numbers to a code-sharing system, in which the same flight could belong to several different airlines and have several different code numbers. This kind of thing clearly affects database schemas and programs, as well as operational relationships between an airline and its partners.

Another good example involving change to both Business Model (B) and Business Model (C) is when retailers started to introduce loyalty cards, thus allowing them to identify a customer as the same again.

So here's a question for the enterprise architects and business analysts out there. If the business model is going to drive business improvement, which business model is it? Are you using any formal methods and notations for Business Model (B), or do you jump straight to formal methods and notations for Business Model (C)?

And are there any other kinds of Business Model I should be looking at?

---

CBDI Journal articles are now available to non-members.

• In Search of a Business Improvement Methodology (December 2008)
• Exploring Business Improvement (January 2009)

On this blog, several follow-up posts on Broken Business Models

• 16 December 2008
• 19 December 2008
• 1 February 2009
• 12 February 2009

Business Improvement from Richard Veryard

Merging Capability Modeling with Process Modeling

Nick Malik asks about the relationship between capability model and process modelling. "If your company is a process oriented one (or becoming a process oriented one), how do you fit in capability modeling? Do these two methods conflict?"

Nick suggests two answers. In this post I want to offer a third answer. But first we have to understand the purpose of each type of modelling.

Nick mentions two purposes for capability modelling - project planning and service design.

• Project Portfolio Alignment - "With capability modeling, you create a hierarchy of the company's capabilities. You then identify the capabilities that best align to corporate strategy, and which one of them need the most work. Focus your work there, and you get a big payoff through focused investment."

• Service Design - "The activities are where you actually perform automation. You connect services to these activities. This is where work is done."

Nick's first answer is that a capability is a top-level chunk of business, while processes are at a lower level. In a later post, Nick says this is how FEA handles capability and process modelling. As Nick acknowledges, FEA actually calls them business areas (or Lines of Business) rather than capabilities. In my 2001 book on the Component-Based Business, I called these things Business Components, and this terminology is still used by IBM in its Component Business Model. I think this is a useful construct, but that's not exactly what I call capability.

Nick's second answer is that a capability corresponds to the objectives of a specific organizational unit. Capabilities are not directly linked to processes; both capabilities and processes use and share atomic-level activities. This is better, but I am uncomfortable with the organizational tie-up.

Meanwhile, the latest version of MODAF does explicitly include "capability" as a first class construct. MODAF includes a hierarchical decomposition of capabilities, and a many-to-many mapping between capabilities and processes (which it calls "operational activities"). This is a lot closer to my notion of capability.

Another way of getting to my notion of capability is to take Nick's second answer and remove all mention of organization units. While the organization structure may give us some important clues about what an organization does and how it does it, I am careful to avoid hard-coding the organization structure into my business models.

So what is the point of modelling capabilities as well as processes? To justify capability modelling, I want to articulate two additional purposes.

• Management of Variation - Capabilities can often be decomposed into highly generic elements and asset-specific elements. (See my example of Green Coffee Beans.) Decoupling these capabilities helps to drive higher levels of cross-process and cross-organizational sharing.

• Viable System Design - A complete system needs to include management capabilities such as planning and coordination as well as operational capabilities. With a capability model, I find it much easier to check for completeness than with a process model.

So I end up with a notion of business capability which maps easily onto business services, supported by information capabilities which may be implemented as software services. This notion allows for a much finer level of granularity than Nick's first answer, but I think it is much more flexible and powerful than Nick's second answer.

The capability defines WHAT the business does; the process defines HOW the business does it (and the organization structure defines WHERE the business does it). So we expect the capability to be more stable than the process; many capabilities should be shareable not only between different processes within a single enterprise, but between multiple enterprises.

This then raises the question of capability modelling techniques - how do we analyse capabilities? We certainly need to map capabilities onto outcomes (as in Nick's second answer) , but we also need to analyse commonalities and variations within and between capabilities, as well as other dependencies between capabilities. In fact the diagram I find most useful, both for planning and for service design, is the capability dependency diagram. MODAF V 1.1 includes a suggested notation for capability dependency diagrams, but I use a slightly different notation.

Sources

• MODAF Version 1.1 (April 2007)
• L. Cherbakov et al, Impact of service orientation at the business level (IBM Systems Journal, 44/4, 2005)
• M. Dodani, The Architecture of Business (JOT 17/4, 2008)
• R. Veryard, Component-Based Business: Plug and Play (Springer, 2001)
• R. Veryard, Capability Dependency (CBDI Journal, May 2007)

Event Modelling

In a post entitled Right-Sizing the Event Message, Nick Malik (Microsoft) states two opposing design principles for event modelling, and suggests a technique for producing a reasonable trade-off. In this post, I'm going to suggest an alternative technique, ending with a discussion of the principles.

Principle of Sufficiency: event messages need to have sufficient information in order for the subscriber to decide if it should consume or discard the event without requesting further information from another system

Principle of Simplicity: event messages should be as small as possible (but no smaller)

Nick's Example: You have two CRM systems, separated by geography. A CRM event message therefore needs to contain some geographical information, to enable each CRM system to determine whether to act upon the event or ignore it. But what other information should we include, that might be relevant to any of the would-be consumers of the message?

For the moment, let's accept Nick's statement of the problem. I want to talk about the solution first, before raising some more general issues about these two principles and Nick's example.

Nick's Technique: Follow the structure of the associated data warehouse. The event message should be based on the "fact table" and the first tier "dimension tables" within a "star schema". (Nick's example looks more like a snowflake schema to me, but never mind.) If this structure has already been defined, then easy-peasy - otherwise, follow the same technique as you'd use for modelling a data warehouse. In other words, treat "events" as "facts".

Nick's technique is certainly worth considering, because there are some strong parallels between data mining and (complex) event (stream) processing. Many event stream processing tools allow you (roughly speaking) to build and execute SQL-like enquiries in near-real-time against a continuous stream of events.

Nick is assuming that effectively each consumer (consuming system) operates its own event filter. But in principle, it should be possible for a consuming system to publish its own event-interest-profile ("This system is interested in European-CRM-events"). There might then be some filter or attenuator that reduces the "CRM-events" stream to the "European-CRM-events" stream, or possibly splits the "CRM-events" stream into several geog-specific event streams. (This filter or attenuator might be a stand-alone utility or event processing engine, or it might be embedded in the messaging platform.) Thus each system "owns" a filter, but it delegates the execution of this filter.

But I can go one step further. If I decouple the event filter from the operational response to the event, I can then change one without changing the other. For example, let's suppose we have just opened an office in South Africa. Someone decides to direct the South African CRM events to the European CRM system. This is a coordination-level decision, which should (if we are lucky) be handled purely at the event processing / orchestration level, without making any change to the (operational) CRM system itself.

So we now have two architectural layers: a coordination/orchestration layer, for which the principle of sufficiency is paramount, and an operational layer, for which the principle of simplicity is paramount. We use some form of attenuation, possibly but not necessarily a filter, to convert rich (sufficient) events to simple events.

Furthermore, this solution allows us to handle much more complex selection criteria than simple filtering. For example, for a customer who travels a lot, it may be complicated to work out which CRM system to use. In which system or systems do we handle an event involving a Japanese employee of an American corporation visiting France?

Based on this rearchitected solution, we can reinterpret the two principles we started with. Sufficiency from a coordination perspective, simplicity from an operational perspective, with efficient and effective transduction between the two perspectives.

Finally, let's wonder why on earth there are two CRM systems at all. The situation may have resulted from previous stove-pipe management, or from a history of merger and acquisition, but we might reasonably expect to see an evolutionary development path in which the two systems become one, or at least share an increasing quantity of common services. But this provides another strong reason to remove as much of the logic (which is going to change as the two systems evolve into one) into the orchestration layer.

SOMF

An article has just appeared on Wikipedia advertising something called The Service-Oriented Modelling Framework, based on a new book by one Michael Bell. Advertisements are usually excised from Wikipedia fairly speedily, so the article may not last long.

According to the article, Bell has invented a modelling process and language for SOA that is both anthropomorphic and holistic. If this is true, it is a remarkable achievement. Most modelling languages are materialist and reductionist - they describe certain aspects of the system-of-interest by reducing them to objects. Whereas an anthropomorphic model would ascribe human characteristics (e.g. personalities and desires) to the system-of-interest.

I guess I need to read the book when it comes out. Perhaps the publishers would care to send me a review copy?