An Alternative to the DIKW Pyramid

My 2012 post on the Co-Production of Data and Knowledge offered a critique of the #DIKW pyramid. When challenged recently to propose an alternative schema, I drew something quickly on the wall, against a past-present-future timeline. Here is a cleaned-up version.

Data is always given from the past – even if only a fraction of a second into the past.

We use our (accumulated) knowledge (or memory) to convert data into information – telling us what is going on right now. Without prior knowledge, we would be unable to do this. As Dave Snowden puts it, knowledge is the means by which we create information out of data. 

We then use this information to make various kinds of judgement into the future. In his book The Art of Judgment, Vickers identifies three types. We predict what will happen if we do nothing, we work out how to achieve what we want to happen, and we put these into an ethical frame.
 
Intelligence is about the smooth flow towards judgement, as well as effective feedback and learning back into the creation of new knowledge, or the revision/reinforcement of old knowledge.

And finally, wisdom is about maintaining a good balance between all of these elements - respecting data and knowledge without being trapped by them.

What the schema above doesn't show are the feedback and learning loops. Dave Snowden invokes the OODA loop, but a more elaborate schema would include many nested loops - double-loop learning and so on - which would make the diagram a lot more complex.
 
And although the schema roughly indicates the relationship between the various concepts, what it doesn't show is the fuzzy boundary between the concepts. I'm really not interested in discussing the exact criteria by which the content of a document can be classified as data or information or knowledge or whatever.

Update (July 2020): I have posted an alternative schema that identifies Three Types of Data. And for an alternative view of Wisdom, see my post on Reframing (February 2009).

Note: As an alternative to the word data (données), Bernard Stiegler (2019, p7) suggests the Husserlian word retention, and associates its opposite (protention) with desire, expectation, volition, will, etc. Thus lumping data and memory together, as well as lumping together Vickers' three types of judgement.

---

Dave Snowden, Sense-making and Path-finding (March 2007)

Bernard Stiegler, The Age of Disruption: Technology and Madness in Computational Capitalism (English translation, Polity Press 2019)

Geoffrey Vickers, The Art of Judgment: A Study of Policy-Making (1965)

Wikipedia: Retention and Protention

Related posts: Wisdom of the Tomato (March 2011), Co-Production of Data and Knowledge (November 2012), Three Types of Data (July 2020)

Right to Repair

One of the interesting dynamics of the service economy lies in the dialectic opposition between open and proprietary. I have mentioned some useful conceptual models in previous posts: Amin and Cohendet have proposed a model that classifies capabilities/services according to the dimensions of knowledge intensity and trust; meanwhile, Max Boisot's iSpace model traces the dynamics of knowledge from proprietary to open.

In my post on the New Economics of Manufacturing (Nov 2015), I described some of the economic forces behind the shift away from manufacturing products (including spare parts) and towards services.

Instead of trying to sell you overpriced tyres, the car manufacturer must make sure that only its accredited partners have the software to balance the wheels properly. In other words, not just architecting the product or even the process, but architecting the whole ecosystem.

But consumers (and regulators) are fighting back. Car owners in the USA have already won the right to repair, and now the farmers of Nebraska are now fighting a similar battle against the tractor manufacturers. True openness would force the manufacturers to publish the repair manuals as well as the interfaces, and allow independent repair shops and knowledgeable consumers to repair their own equipment without relying upon some dodgy download or counterfeit component.

This matches the Boisot model of stuff flowing from the proprietary world into the open world. I'm sure there will be more examples of this to come ...

---

Richard Chirgwin, European MPs push for right to repair rules (The Register, 6 Jul 2017)

Jason Koebler, Five States Are Considering Bills to Legalize the 'Right to Repair' Electronics (Motherboard 23 Jan 2017)

Jason Koebler, Why American Farmers Are Hacking Their Tractors With Ukrainian Firmware (Motherboard, 21 March 2017)

Gabe Nelson, Automakers agree to 'right to repair' deal (Automotive News, 25 January 2014)

Olivia Solon, A right to repair: why Nebraska farmers are taking on John Deere and Apple (Guardian, 6 March 2017)


Related posts

Knowledge and Culture (April 2006)
Tesco outsources core eCommerce (March 2009)
Ecosystem SOA (October 2009)
The New Economics of Manufacturing (November 2015)


Link added 18 July 2017

Three Notions of Maturity

Many enterprise architecture frameworks contain some notion of maturity, usually with some kind of nod in the direction of the SEI CMMI maturity model. I'm puzzled about this, because these notions of maturity don't much resemble the SEI's notion and sometimes have a completely different set of levels.

The SEI has produced several different maturity models, but they are all presented in terms of the maturity of capability or process - doing the right things right according to a standardized schema of What The Right Things Are. Applying a process-oriented notion of maturity to EA can only really refer to maturity of the EA process. But I think it is much more relevant to think about EA maturity of the organization as a whole, which calls for an outcome-oriented notion of maturity (perhaps defined in terms of some kind of "alignment") that is closer to Richard Nolan's Stages of Growth model, where maturity is seen as a state of perfect alignment between the business and its information systems. (I know Nolan originally formulated his ideas in terms of IT, but then so did lots of people in the 1980s including Zachman.)

Maturity can also be defined in terms of an alignment between espoused theory (what you think you are supposed to be doing) and theory-in-use (what you are actually doing). Obviously this kind of alignment is not restricted to IT. Maturity doesn't just mean being better at achieving your goals, it also means having more realistic goals in the first place. Some people might think that innovation feeds upon the unbounded vision and ambition of the immature, and that too much maturity (in this sense) could just result in negativity and middle-aged stagnation. See my note on EA Archetypes (EA-as-visionary, EA-as-realist).

Meanwhile, some enterprise architecture frameworks contain notions of maturity that are defined not in terms of process but in terms of knowledge. At the Unicom EA Forum on 21st March 2013, Kevin Smith presented his new PETF framework for Enterprise Transformation, whose four levels of maturity are based on the Four Stages of Competence.

1. Unconsciously incompetent
2. Consciously incompetent
3. Consciously competent
4. Unconsciously competent

Thus the highest level of maturity is where the knowledge has been internalized. I think this notion of maturity looks much closer to Nonaka's SECI model or Boisot's I-Space, which both contain notions of internalization. I-Space also contains the insight that "intellectual property" doesn't last for ever, because all useful knowledge eventually becomes public knowledge. Competence relative to one's competitors is always based on proprietary knowledge, while knowledge that is in the public domain cannot form the basis of competitive advantage.

Meanwhile I'm uncomfortable with the notion of unconscious competence for another reason: in a dynamic world it can quickly develop into complacency and arrogance. Whatever happened to continuous learning and improvement?

In order to maintain competitive advantage, both SECI and I-Space are essentially cyclic models, so that one can cycle back from level 4 (unconscious competence) to level 1 (unconscious incompetence). In my summary last week, I invoked the Black-Belt-to-White-Belt metaphor: even the most experienced black belt needs to return to the basics, needs humility and the desire to learn, needs to always think like a beginner rather than strutting around arrogantly like an expert. Obviously there are circumstances in which an enterprise may need to cycle round from competence to incompetence, before attaining a higher level of competence. Sometimes transformation must take an enterprise out of its comfort zone.

Finally, I wonder whether "maturity" is the right word at all. It is sometimes argued that different levels of maturity are suitable for different organizations, which is basically a form of contingency theory. In other words, some organizations never need to be fully mature.

Instead of maturity, I prefer a notion of excellence that includes (1) selecting the appropriate approach for a particular situation/context, and then (2) carrying out this approach both effectively and cost-effectively. I think this notion of excellence is supported by business excellence frameworks such as Baldrige and the European Quality Award. These frameworks don't say HOW you should do anything, merely that you should know WHAT you are doing, and WHY, and WHETHER it is working, and that you should be constantly striving to improve those things that matter most. Hopefully this gets around the innovation/maturity paradox I mentioned earlier.

---

Scroll down for discussion with Len Fehskens.

Related post: From Enabling Prejudices to Sedimented Principles (March 2013)


Thanks to Len Fehskens (in comments below) for pointing out the original source of the Four Stages of Competence.

Post last updated 31 August 2018

Showrooming in the Knowledge Economy

In the knowledge economy, service providers often give away selected portions of their intellectual property, in the form of webinars, white papers, blogposts, free downloads or whatever. They then hope to generate revenue from other products and services. So the free IP acts as a kind of showroom.

Many knowledge consumers regard this as an open invitation to a kind of knowledge showrooming, surfing the web and mopping up enough pieces of knowledge to avoid ever having to pay for knowledge products and services. Meanwhile, most large employers have clamped down on all expenditure, so it is often easier for well-paid staff to spend several hours fruitlessly browsing the internet than to get approval for $100 to buy a report or a subscription.

(Meanwhile, many people have even given up reading real books, and some kids seem to expect to go through college without ever opening a book, hoping that they can make do with the garbled summaries they can find on Wikipedia and elsewhere. Unfortunately, this tendency is reinforced by the fact that an increasing number of real books seem to have been researched in the same way.)

I confess that I have avoided paying for things sometimes. For example, if I can find a working draft of a paper online, I am generally reluctant to pay to read the final version, especially as (a) it may be almost indistinguishable from the draft version and (b) the author is unlikely to see any of the money.

Here's an example of knowledge showrooming. Supposing you want a method and a tool and some templates and some training and some support. If you get them all from the same company, you should expect to pay for at least some of that. But if you can find one company that lets you download a free software tool, and a second company that publishes its method on its website, and a third company that is offering free training events (probably funded by companies that are trying to sell tools and methods), this seems to offer a tempting solution.

Of course, having spent several hours finding this stuff, it may take you a lot longer to get all these disparate pieces to work together; and even if you succeed, your productivity using this mashed-up solution may leave a lot to be desired. However, time is generally valued less than money, so many knowledge consumers and their bosses may be quite satisfied.

On the supply side, given that there is absolutely no prospect of any agreement between knowledge providers as to which elements should be free and which should be charged-for, knowledge providers are effectively undercutting each other.

(I have unhappy experience of this myself. In the late 1990s, I developed a detailed methodology and associated training for developing component-based business solutions. Although I did sell some training, the software vendors started giving their methodologies away for nothing, in order to sell greater volumes of software. Even though I thought my methodology was better, I just couldn't compete with free.)

Universities are now struggling with the implications of publishing lecture notes and other course materials online. Does this undercut the value of a real degree? Or can they construct some kind of remote/online learning experience and expect to charge serious money to distance learning students?  @RogerShank is sceptical, and so am I.

---

Roger Shank, Why are universities so afraid of on line education? (November 2012)

See also my post Showrooming and Multi-Sided Markets (December 2012)

Co-Production of Data and Knowledge

Following Russell Ackoff, systems thinkers like to equate wisdom with systems thinking. As Nikhil Sharma points out,

"the choice between Information and Knowledge is based on what the particular profession believes to be manageable".

When this is described as a hierarchy, this is essentially a status ranking. Wisdom (which is what I happen to have) is clearly superior to mere knowledge (which is what the rest of you might have, if you're lucky).

---

Here's an analogy for the so-called hierarchy of Data, Information, Knowledge and Wisdom (DIKW).

• Data = Flour
• Information = Bread
• Knowledge = A Recipe for Bread-and-Butter Pudding
• Wisdom = Only Eating A Small Portion

Note that Information isn't made solely from Data, Knowledge isn't made solely from Information, and Wisdom isn't made solely from Knowledge. See also my post on the Wisdom of the Tomato.

---

That's enough analogies. Let me now explain what I think is wrong with this so-called hierarchy.

Firstly, the term hierarchy seems to imply that there are three similar relationships.

• between Data and Information
• between Information and Knowledge
• and between Knowledge and Wisdom

 as well as implying some logical or chronological sequence

• Data before Information
• Information before Knowledge
• Knowledge before Wisdom

while the pyramid shape implies some quantitative relationships

• Much more data than information
• Much more information than knowledge
• Tiny amounts of wisdom

But my objection to DIKW is not just that it isn't a valid hierarchy or pyramid, but it isn't even a valid schema. It encourages people to regard Data-Information-Knowledge-Wisdom as a fairly rigid classification scheme, and to enter into debates as to whether something counts as information or knowledge. For example, people often argue that something only counts as knowledge if it is in someone's head. I regard these debates as unhelpful and unproductive.

A number of writers attack the hierarchical DIKW schema, and propose alternative ways of configuring the four elements. For example, Dave Snowden correctly notes that knowledge is the means by which we create information out of data. Meanwhile Tom Graves suggests we regard DIKW not as layers but as distinct dimensions in a concept-space.

But merely rearranging DIKW fails to address the most fundamental difficulty of DIKW, which is a naive epistemology that has been discredited since the Enlightenment. You don't simply build knowledge out of data. Knowledge develops through Judgement (Kant), Circular Epistemology and Dialectic (Hegel), Assimilation and Accommodation (Piaget), Conjecture and Refutation (Popper), Proof and Refutation (Lakatos), Languaging and Orientation (Maturana), and/or Mind (Bateson).

These thinkers share two things: firstly the rejection of the Aristotelian idea of one-way traffic from data to knowledge, and secondly an insistence that data must be framed by knowledge. Thus we may validate knowledge by appealing to empirical evidence (data), but we only pick up data in the first place in accordance with our preconceptions and observation practices (knowledge). Meanwhile John Durham Peters suggests that knowledge is not the gathering but the throwing away of information. Marvellous Clouds, p 318

Among other things, this explains why organizations struggle to accommodate (and respond effectively to) weak signals, and why they persistently fail to connect the dots.

And if architects and engineers persist in trying to build information systems and knowledge management systems according to the DIKW schema, they will continue to fall short of supporting organizational intelligence properly.

---

For a longer and more thorough critique, see Ivo Velitchkov, Do We Still Worship The Knowledge Pyramid (May 2017)

Many other critiques are available ...

Gene Bellinger, Durval Castro and Anthony Mills, Data, Information, Knowledge, and Wisdom (Systems Thinking Wiki, 2004)

Tom Graves, Rethinking the DIKW Hierarchy (Nov 2012)

Patrick Lambe, From Data With Love (Feb 2010)

Nikhil Sharma, The Origins of the DIKW Hierarchy (23 Feb 2005)

Kathy Sierras, Moving up the wisdom hierarchy (23 April 2006)

Dave Snowden, Sense-making and Path-finding (March 2007)

Gordon Vala-Webb, The DIKW Pyramid Must Die (KM World, Oct 2012) - as reported by V Mary Abraham

David Weinberger, The Problem with the Data-Information-Knowledge-Wisdom Hierarchy (HBR, 2 February 2020)

DIKW Model (KM4dev Wiki)


Related posts: Connecting the Dots (January 2010), Too Much Information (April 2010), Seeing is not observing (November 2012), Big Data and Organizational Intelligence (November 2018), An Alternative to the DIKW Pyramid (February 2020)

---

Updated 8 December 2012
More links added 01 March 2020
Also merging in some material originally written in May 2006.

Smart Content

There are several characteristic features of so-called smart content.

• Content enhanced to be fit-to-purpose ... content that is organized and structured for customer tasks and needs, not just for the production, packaging and distribution of physical documents. (Mirko Minnich, ex Elsevier)

• Self-organizing and transparent content, organizing itself automatically depending on your context, goals, and workflow, and allowing you to see why it's doing what it's doing. (Mark Stefik, Xerox PARC)

• Granular at the appropriate level, semantically rich, useful across applications, and meaningful for collaborative interaction. (Gilbane Group)

• Has good metadata (not lots), fit for purpose, uses classifications to provide context and aid discoverability (Madi Solomon, Pearson)

And there are several characteristic technologies that are supposed to facilitate smart content, among other things. Some of these technologies are linked to Sir Tim Berners-Lee. Come on Tim!

• Semantic technologies to cross-reference and cross-polinate with other kinds of content. (Madi Solomon, Pearson)

As Natasha Fogel pointed out, "smart content is in the eye of the beholder" - in other words, the perceived smartness of content is relative to its context of use.

But in this post, I don't want to talk about the technologies themselves but about the emerging value propositions that may be supported by smart content. Last year, when he was a SVP at scientific and technical publisher Elsevier, Mirko Minnich talked about two key enablers for smart content. Firstly a value-adding process - transmuting content into scientific data, and transmuting scientific data into solutions. And secondly what he calls a product bridge, not only linking content with data but also linking the content business with the data analytics business. The product bridge appears to be a kind of platform, and Mirko was using the term "Smart Content" to refer to the platform itself as well as the content delivered on the platform.

Mirko's strategy at Elsevier represented a strong drive towards asymmetric design - in other words, recognizing that in order to deliver indirect value into a complex ecosystem you have to move away from a traditional product-based business model (in Elsevier's case, selling scientific journals) towards regarding your business as a multi-sided platform.


Mark Stefik (Xerox PARC) puts smart content into an organizational intelligence frame - the intelligence is now located (reified) in the content as well as in the people producing and consuming the content. Instead of the user asking "what content do I need", Mark wants the content to ask "who needs me?" Madi Solomon (Pearson) seems to be suggesting the exact opposite when he mentions the Big Shift from Push to Pull in his recent presentation on Smart Content. We can resolve this apparent contradiction only by understanding the intelligence as the property of the whole system rather than trying to locate it in one place - see my material on organizational intelligence.

---

Sources

Seth Grimes, Six definitions of smart content (Information Week, Sept 2010)Several of the quotes above come from this article.


Technology in Publishing (Editors Update, Elsevier, Jan 2011)
Next Generation Clinical Decision Support (Elsevier Press Release, Feb 2011)

Madi Solomon, Making Information Pay (April 2011)

From Convenience to Consilience - “Technology Alone Is Not Enough"

Wikipedia defines Consilience as the unity of knowledge (literally a jumping together of knowledge), which has its roots in the ancient Greek concept of an intrinsic orderliness that governs our cosmos, inherently comprehensible by logical process.

The word appears in an appreciation of Steve Jobs by @jonahlehrer, published in the New Yorker on October 7th 2011.

What set all of Jobs’s companies apart, from Pixar to NeXT to Apple, was, indeed, an insistence that computer scientists must work together with artists and designers—that the best ideas emerge from the intersection of technology and the humanities. One of the greatest achievements at Pixar was that we brought these two cultures together and got them working side by side, Jobs said in 2003. ... Perhaps the clearest demonstration can be seen in the design of the Pixar campus. ...

Jobs realized, however, that it wasn’t enough to simply create a space: he needed to make people go there. As he saw it, the main challenge for Pixar was getting its different cultures to work together, forcing the computer geeks and cartoonists to collaborate. (John Lasseter, the chief creative officer at Pixar, describes the equation this way: Technology inspires art, and art challenges the technology.) In typical fashion, Jobs saw this as a design problem. ...

That emphasis on consilience, even if it came at the expense of convenience, has always been a defining trait of Steve Jobs. In an age of intellectual fragmentation, Jobs insisted that the best creations occurred when people from disparate fields were connected together, when our distinct ways of seeing the world were brought to bear on a singular problem. It’s what happens when a calligrapher designs a computer font and when an animator strikes up a conversation with a programmer at the bathroom sink. The Latin crest of Pixar University says it all: Alienus Non Diutius. Alone no longer.

Jonah Lehrer, Steve Jobs: “Technology Alone Is Not Enough” (New Yorker, 7 October 2011)

---

Steve Jobs was widely regarded as an outstanding product architect - someone with exceptional aesthetic sense and attention to detail. The creation of the iTunes ecosystem probably ranks as solution architecture rather than just product architecture. But Jonah's account emphasizes Job's exceptional ability as an enterprise architect - putting people and organizations together to achieve a classical unity of knowledge, which Jonah calls consilience. According to Jonah, it was his secret sauce.

Other commentators have said similar things. When Jobs finally resigned, John @Gruber stated that Jobs’s greatest creation isn’t any Apple product. It is Apple itself. (Daring Fireball, August 2011) Building on this statement, Horace Dediu (@asymco) invites other companies to copy how Apple harbors the creative process and the technology processes under the same roof. If that occurs, Jobs will have left a legacy not just on his own companies (Apple, Pixar) but on all companies. (Polymath Asymco August 2011, reposted as Steve Jobs's Ultimate Lesson for Companies HBR August 2011)

---

Related Posts

Steve Jobs was not a visionary (October 2011)
From Coincidensity to Consilience (January 2015)

Link updated 18 April 2015

The Power and the Glory

#entarch A debate on Twitter about the power of enterprise architects (enhancing? enabling? influencing?), and another debate about the relationship between architecture and knowledge (does architecture count as a form of knowledge?) led to a question from @StevenvtVeld "An architecture as influencer? Closer to view, knowledge: how can this be an influencer?"

There are several ways in which knowledge influences what goes on.

• People and organizations perceive, interpret and evaluate the world according to their prior knowledge. The implications of this have been extensively explored by philosophers (Kant onwards), psychologists (Piaget) and systems scientists (Maturana and Varela).

• People and organizations decide and act according to their assessment of the likely outcomes and risks of the options they are aware of, based on their prior knowledge and understanding of causes and effects. (This may include conscious rationalization as well as unconscious or unspoken patterns of thought.)

• People and organizations learn selectively from experience. Theories and beliefs tend to be "sticky", they are not necessarily abandoned as soon as contrary evidence appears. Because present experience only slowly and imperfectly feeds into future knowledge, current behaviour is largely influenced by past knowledge.

• Within organizations, people attempt to control flows of knowledge for political advantage. This is sometimes known as a gatekeeper function.

• Closed systems of knowledge (sometimes called a discourse or discursive practice) may be elevated to a dominant status within an organization culture. For example, the accounting discourse has a dominant status in many commercial organizations, while the target-setting discourse has become dominant in the public sector. Arguments based on the dominant discourse typically bias the organization towards certain ways of solving problems, and make some kinds of innovation impossible. (See my post on Making Conversations Visible.)

Enterprise architects need to work with knowledge at two different logical levels - not only participating in the development and good use of relevant forms of knowledge (on architecture and related subjects) but also understanding and facilitating the structural and cultural aspects of knowledge development and use across the enterprise.

---

By the way, Graham Greene's novel The Power and the Glory was published in the USA as The Labyrinthine Ways. It is said to be one of President Obama's favourite books. Perhaps there is a hidden message in it for enterprise architects: I'd better read it sometime.

Unstructured knowledge

@SFDreverman asks (how) does unstructured knowledge hinder the collective intelligence of a group?

To start with, I am slightly puzzled by the word "hinder". Surely unstructured knowledge is usually better than no knowledge at all, although sometimes too much knowledge produces confusion and procrastination. But is structured knowledge always better than unstructured knowledge?

There are different kinds of structure that may be relevant here. Some kinds of knowledge may be formally codified and classified, especially for the purposes of storage in a library (which may be paper-based or electronic or both). But although this kind of structure may be useful for storage and retrieval, what is important for decision-making and learning (in other words, the processes contributing to collective intelligence) is a completely different kind of structure, which we might call its logical structure - the links and dependencies and entailments and contradictions between different fragments of knowledge and would-be knowledge.

Where does this logical structure come from? From another of the processes contributing to collective intelligence, namely sense-making. We make sense of knowledge by discovering (or creating) a logical structure for it. If we fail to find a logical structure for some random collection of would-be knowledge then that clearly limits our ability to do anything intelligent with it.

The discovery of a logical structure by a group of people depends in part on the intelligence and motivation of the group, and is not just an inherent quality of the knowledge itself. So if the knowledge remains unstructured, it may be because the group cannot or cannot be bothered to make sense of it. In which case it is the (insufficient) collective intelligence of the group that hinders the collective intelligence of the group.

The relationship between the storage structure and the logical structure remains problematic however. Library systems and knowledge management systems are generally not designed to reflect the logical structure of the knowledge contained within them, and this may make it hard to navigate the logical structure, or to retrieve knowledge according to the logical demands of the situation. So what we are talking about here is not unstructured knowledge but poorly and inappropriately structured knowledge. So the architecture and usability and effective use of knowledge management systems clearly have an impact on organizational intelligence.

Multiple styles of EA

@tetradian has an interesting post on Big EA, Little EA and Personal EA., based loosely on Patti Ancram's classification of knowledge management.

• Big KM is about top-down, structured and organizationally distinct “knowledge management”
• Little KM is about safe-fail experiments embedded in the organizational structure
• Personal KM is about access to tools and methods to ensure that knowledge, context, bits, fragments, thoughts, ideas are harvestable

---

As I see it, this classification identifies different styles that may possibly coexist, or perhaps different kinds of knowledge claim that may interact in interesting ways. (I don't like the word "layers" for this kind of classification, because it implies a particular structural pattern, which isn't appropriate here.)

I've used a slightly different division in the trust sphere, which might make sense here as well.

• Authority EA - this is a kind of top-down command-and-control EA, representing the will-to-power of the enterprise as a whole, and ultimately answerable to the CEO. This is what Tom calls Big EA.

• Commodity EA - this is where the EA is based on some kind of external product source - such as when the enterprise models are imported wholesale from IBM or SAP. This often resembles Big EA, but has some important differences.

• Network EA - this is where EA is based on informal and emergent collaboration between people and organizations. Tom calls it Little EA, but the collaborations can be very extended indeed - just think about some of the mashup ecosystems around Google or Twitter.

• Authentic EA - this is a personally engaged practice - what Tom calls Personal EA.

Once we have agreed that there are different styles, the really interesting question is not identifying and naming the styles, nor even saying that one style is somehow "better" than another style", but talking about how the different styles interact, and what are the implications for governance.

Towards an Enterprise Architecture of Knowledge

A customary way of modelling an enterprise is as a collection of purposeful behaviours - functions or processes or capabilities or whatever. Each function or process or capability encapsulates some knowledge or know-how. Sometimes this knowledge will be fairly static, so the behaviour will be stable and unchanging. And sometimes the underlying knowledge will change, so the behaviour will need to be adjusted to accommodate the most up-to-date and relevant knowledge. In some areas, such as product development and marketing, there may be a strategic imperative to find and assimilate new knowledge and ideas (about market trends, technology trends, competitor trends, and so on), to build new collaborative partnerships, and to experiment with new behaviours. These areas will rely on business intelligence - not just BI tools but all the practices associated with collecting and making sense of complex information.

The overall architecture of the enterprise can then be understood in terms of how these functions or processes or capabilities interoperate to produce a viable system-of-systems. The enterprise architect must understand which of these functions are relatively stable, which of them are strategically volatile, and create appropriate mechanisms for coordinating between them. These mechanisms are essentially behavioural ones - information flows, event triggers, process flows and so on.

But if we look at the enterprise through a knowledge management lens, we will see a different kind of structure. There is a series of overlapping knowledge domains - marketing, accounting, regulatory compliance, health and safety, employment, technology - each with its own specialist terminology and ways of thinking. So there is an architectural question about how these knowledge domains (we might also call them agendas or discourses) interoperate.

In a traditional organization, these knowledge domains only come together at board level, with one or more directors speaking for each significant domain. Thus the CTO speaks for the technology domain, the HR director speaks for the employment domain, and so on. Of course all the directors will try to influence the agenda of their peers - so everyone else might be citing random snippets of evidence of technology trends, or challenging the CTO's decisions and interpretations, pushing the CTO to explain or change direction. In a well-functioning organization, this will be all done in the spirit of healthy and vigorous debate.

In a hierarchical organization, some pieces of this debate can of course be delegated - either to internal departments or to external consultants - but the directors remain responsible for putting the pieces together.  

Thus the coherence and intelligence of a traditional organization depends heavily on the collective intelligence and effective functioning of the board of directors. But there are several evident problems with this approach. Firstly, we can observe many organizations that lack coherence and intelligence, so this approach is manifestly not working well enough for these organizations. Secondly, we may assume that the board of directors has a finite reasoning capacity - however clever they are as individuals, and however well-bonded they are as a team - and that they will not always be able to keep up with the increasing complexity and volatility of the business environment. And thirdly, we can see a lack of joined-up intelligence at the lower levels of the organization.

Some people think there is a different way of integrating an enterprise. If we can identify a single universal knowledge domain, then we can use this knowledge domain to join up everything else. For example, in a financially driven organization, management accounting acts as a unifying language across all functions. Conversely, many enterprise architects regard their favourite EA framework as providing such a unifying language. Although there have certainly been very large organizations that have been managed according to a single unifying principle (such as GEC under Lord Weinstock), the fact that there are several specialisms each wanting to be top dog helps to explain why this doesn't work very often.

A third idea would be a bottom-up Enterprise 2.0 swarm intelligence, in which a satisfactory resolution to all difficult issues emerges from uncontrolled debate and "sharing" across the organization. To read some of the material on the internet, it's tempting to believe that this is how some of the really cool organizations in Silicon Valley are organized. But even if this were true, there is still a need for some kind of architecture and governance, for example, providing suitable platforms and pathways for constructive and reasonably efficient problem-solving. Kevin Kelly, whose book Out of Control introduced me and many other people to the huge potential of bottom-up intelligence, is careful to warn that the bottom is not enough.

Which brings me to the fourth idea - regarding the coherence and intelligence of the knowledge-bearing organization as a real architectural issue. Not the kind of structure that enterprise architects are accustomed to modelling or managing perhaps, but this structure may have a significant effect on the business outcomes of the organization. Isn't this something enterprise architects should be interested in?

Intelligent Knowledge Management

@snowded has started a series of blogposts about knowledge sharing across silos (part one, part two). He starts by identifying a number of different types of knowledge that (some people think) it might be valuable to share.

• Data - avoiding duplication between data stores - for example consolidating all health records in a single database
  

• Information - ensuing all relevant information is available to support critical decisions and actions - see for example my post on Joined-up Healthcare
  

• Transactions - a sense that the citizen or customer has to navigate tortuous pathways through the bureaucracy

• Practice - good practice developed in one area is not appreciated or understood elsewhere
  

• Ideas - innovative combination and variation of issues, ideas, solutions and problems from within different silos

Obviously a mixed bunch of stuff here, and it seems very confusing to lump all of this under the heading of "knowledge management". As Dave points out, some of these look more like information flow. However, suppliers of knowledge management products and services (consultants and tool vendors) typically want to find as many different applications for their stuff as they can. Knowledge is therefore presented as a way of achieving a bunch of different things.

• more efficient information systems and working procedures
• improved decisions
• more effective and convenient customer service
• faster, more efficient and more consistent innovation
• greater cooperation and collaboration
  

Underlying the knowledge management agenda are a number of (usually) unexamined assumptions

• most of the knowledge we need already exists in people's heads - we just have to get it out
• knowledge is additive - the more knowledge we can "capture" or "extract" the better
• explicit knowledge is better than implicit or tacit knowledge - codification is a "good thing"
  

Thus knowledge management becomes a combination of documentation/codification and librarianship, plus whatever persuasion or coercion may be necessary to encourage people to participate in this game. Added to this is a kind of open-ended information retrieval, as exemplified by IBM's Lotus Knows campaign. (See my comments here.)

Undoubtedly there are some situations where this kind of thing can deliver direct value to the organization - for example in pharmaceuticals, where any delays in the assembly of regulatory information for a new drug can cost millions of dollars in lost revenue. But is this really knowledge management, or just a sophisticated form of information management?

An alternative approach to knowledge management is to leave it in people's heads, but then to provide knowledge maps that tell everyone whom to ask. The trouble with this approach is that the genuine experts often keep their heads down, for fear of being swamped with enquiries from around the globe, while attention-seekers use this as an opportunity to promote themselves. Thus questions of motivation and excellence must be addressed, and knowledge management becomes a branch of Human Resources.

---

Meanwhile, I've been reading a couple of older critiques of the 'knowledge management' industry.

• Erik Bergman, When bad things happen to good ideas (CIO, May 2001) via @rotkapchen
  
• T.D. Wilson, The nonsense of 'knowledge management' (Information Research, October 2002) via @oscarberg
  

Although these papers predate the recent explosion of web-based knowledge management tools (including Enterprise 2.0, however that is defined), the issues raised in these two papers largely remain unaddressed. Bergman points out that knowledge management is often regarded as a technological initiative. He mentions a number of very expensive white elephants, which failed largely because they failed to address the real business objectives or the real cultural and working practices of the organization. Wilson raises some more fundamental issues, suggesting that the knowledge management agenda is muddled and self-serving, and that Polyani's original concept of tacit knowledge has been grossly misinterpreted in order to promote an utopian fantasy of the manageability of the human mind.

So what (if anything) can we rescue from this mess?

Well we might start by shifting from "knowledge as a resource" to "knowledge as a process", as Gil Ariely argues in his paper Knowledge Management as a Methodology towards Intellectual Capital (Sept 2003, pdf). See also Interview with George Siemens (Dec 2009). Unfortunately, this phrase was long ago coopted by solution providers - for example in their paper Knowledge Asset Management, (June 2001), Ron Young and Gregoris N. Mentzas describe what they claim to be "one of the world’s first truly holistic and integrated knowledge management solutions". Holistic, huh?

Perhaps knowledge management shouldn't be about grabbing and hoarding stuff but about deploying it intelligently. This points us towards concepts like Evidence-Based Management, where knowledge is collected, analysed and deployed within a management loop. Consultancies promoting this and similar concepts include Pfeffer's and Sutton's Evidence-Based Management, as well as Management Epistemology. and the Knowledge Management Consortium - see for example the KMCI paper Corporate Epistemology (November 2003, pdf).

This kind of approach shifts the emphasis from knowledge sharing to knowledge embedding - grounding the work in the best available and critically evaluated knowledge, as well as actively seeking well-grounded knowledge to support organizational learning. Obviously collaboration is important here, but there is a distinction between collaborating-in-the-work (for example shared responsibility for decisions) and collaborating-in-the-knowledge (for example, shared responsibility for collecting and interpreting intelligence, connecting the dots). This distinction builds upon the RAEW technique we have long used for analysing organizations. A key question here is the relationship between decision and intelligence - how closely the expertise and authority should be coupled/aligned to the work itself.

What characterizes the intelligent organization is not just having more knowledge than its competitors, or even doing better at sharing the knowledge it has, but how effectively it invests its entire knowledge capital to increase its own viability and survival. This is so not a technology question, or even a best-practice question, but a strategic question. But you already knew that, didn't you?

---

see also When does Communication count as Knowledge Sharing?

Knowledge and Architecture

@EnterprisingA (Jon Ayre) suggests an #EAMantra "If you know it, add it to your architecture. If you think you know it, add that too."

My immediate objection to this rule was that it seems to turn the architecture into a kind of brain dump. A good architect knows many things, and if all these items of knowledge go indiscriminately into the architecture, the architecture gets rather cluttered.

@EnterprisingA 's first defence was to say that "the quality of a brain dump depends on the quality of the brain from which the dump comes". And of course only the architect gets to deposit knowledge into the architecture (otherwise there would be a free-for-all).

Unfortunately, the brilliance of the architect is no guarantee that the architect's thoughts all satisfy some meaningful criterion of quality. Intelligence doesn't necessarily mean always having better-quality knowledge, but it should mean having a better capability for processing and developing and filtering and revising knowledge. And as Dumbledore says, "Being - forgive me - rather cleverer than most men, my mistakes tend to be correspondingly huger".

So the architect's knowledge may well get into the architecture eventually, but it doesn't need to go straight in. But @EnterprisingA is (quite reasonably) worried about not writing stuff down. "Keeping it in your head until you are absolutely certain is a good way of never being certain (or being challenged)." "I'd like you to review the arch but it's not certain til it's reviewed so I haven't drawn it so you can't review it..." Of course that's true, but I don't agree that there are only two places to keep knowledge - either in your head or in the architecture. @rgechristie suggests a third possibility - an interim area for in progress work before it hits "the architecture" (that still needs to be accessible by all developers and consumers of the architecture).

***

My second objection was that the architect has a lot of knowledge that doesn't really belong in the architecture at all. There is a strong temptation to put too much into an architecture, and the architectural documents can easily get bloated with miscellaneous material that the architects imagine might be useful to developers and others. (Methodology documents are subject to the same tendency.) For example, the architect may review a design document, may spot a particularly egregious design flaw, and decide to add a rule into the architecture, so that the architecture evolves into a general-purpose design handbook. And if every pearl of wisdom spoken by the architect has to be preserved in "The Architecture", you end up with "The Baroque" - a highly decorated and convoluted style.

It is of course possible to slim down an architecture with too much detail, but in practice this doesn't happen as often as it should. It is much easier to add material to a document than to subtract it, so over time the documents get longer and harder to read.

***

Following our debate on Twitter, @EnterprisingA reformulates his mantra to specify architectural decisions only. "If you know it is the right architectural choice, add it to your architecture. If you think you know it is, add that too." Surely I can't disagree with the proposition that "architectural decisions should be in the architecture", can I?

Actually I do. Sometimes the best architectural decision is to leave something out of the architecture, to leave something deliberately open and unspecified, underdetermined rather than overdetermined. This is a principle of just-enough architecture, or Zen architecture. Sometimes less is more.

Knowledge Coupling

According to reports in the Financial Times, the Inland Revenue wishes to find a new IT contractor. Government IT outsourcing has been dominated by a small number of firms, particularly EDS and Accenture.

Apparently, some major service companies (including, it is rumoured, IBM Global Services) do not consider it worth submitting bids against the incumbant. Bidding costs can run into tens of millions, with very low chances of success.

Principle One: If you want to maintain the right to choose, you have to exercise your choice from time to time – otherwise, the choice becomes void.

Principle Two: If contractors have exclusive (or privileged) access to complex knowledge required both to maintain the service, and to participate meaningfully in the bidding process, then the choice may become economically non-viable.