Organizational Memory

Are social networking tools the solution to improving organizational memory (asks @EffectiveCIO Chuck Musciano)? Chuck's blog Legs and Memory suggests a link between memory and efficiency: "Forgotten items create more work, both at home and on the job."

Now that would be easy enough if nothing ever changed, and if last year's best practices still worked. But as Chuck points out, formal guides and detailed documentation fail because of continuous change. There is a tension between efficient adaptation and effective adaptability.

So Chuck argues that organizational memory is best retained in the heads of the people in the organization. And because captured organizational memories fade rapidly over time, you must reinforce your organizational memories, by constantly revisiting and updating them.

What's the role of social networking tools here then? Chuck doesn't believe that these tools are yet up to the job of real-time knowledge capture, and falls back on the old favourite - finding the person who knows what we need. Better than nothing perhaps, but way short of what is needed.

---

What is needed for organizational intelligence is the ability to use organizational memory without being controlled by the past. In his post (some) rules are meant to be broken, Brett Miller makes this point very well.

You can see this in the way many organizations apply the idea of “best practices”: capture past practices that worked and apply those practices, as is, to future situations that are similar. While this works fine for what I call “information” processes - and is a critical step in helping any organization improve - it is not appropriate for “knowledge” processes. ...

... This is not to say that past experiences should not be exploited in creating/acquiring new knowledge. Except for the rarest of occasions, most new knowledge created today is derivative of something past. It is important to know what has come before and learn from the successes and failures of others. The rules that come from those past lessons then become the framework for the future.

But of course organizational memory does not only consist of rules (best practices), but all sorts of other knowledge (including evidence, interpretation and stories) that may be relevant to developing next practice. A lot of this knowledge will surely be maintained in electronic form, not just in people's heads, some structured or semi-structured, using a broad range of social software.

Furthermore, evolving practices (organizational learning) will typically require collaboration between different people across the enterprise, and we may reasonably hope for software tools to support this learning.

Once upon a time, information systems maintained a clear distinction between highly structured data (in old-fashioned databases) and loosely structured information (in a wide range of formats, including Office). But we now need to find new ways of integrating this information to support the intelligent organization.

Loose Coupling: Innovation and the Web

"Cosy social networks are stifling innovation", according to an article in the New Scientist (5 August 2009).

This is a familiar argument in a new context. It has often been argued that corporate culture can inhibit innovation, and that groups responsible for experimental products and processes tend to perform better if they are put into a separate organization unit at some distance from the main offices. In recent years, many companies have set up R and D units in special science parks, co-located with similar units from other companies, usually with links to a nearby university. This is essentially applying architectural thinking to the geographical location and distribution of certain classes of capability, and reflects a common belief in the importance of these factors.

But interaction and clustering is nowadays much less dependent on physical geography, and much more dependent on virtual online communities and networks. The New Scientist article quotes Viktor Mayer-Schönberger of the National University of Singapore, who argues that today's software developers work in social networks in which everyone is closely linked to everyone else. "The over-abundance of connections through which information travels reduces diversity and keeps radical ideas from taking hold."

What Mayer-Schönberger sees as an over-abundance of connections is actually another form of tight coupling. If we want to build the capability for radical innovation, we need to create a decoupled space to support a loosely coupled knowledge cycle. Which means careful attention to the effects of social networking and organizational intelligence.

TOGAF 9 - Enterprise Continuum

The concept of Enterprise Continuum was present in TOGAF 8, but it is a little clearer in TOGAF 9.

What is an Enterprise Continuum? In TOGAF 8, it was defined as "a virtual repository", but this wasn't very clear. In TOGAF 9, it is now defined more explicitly as "a view of the Architecture Repository that provides methods for classifying architecture and solution artifacts as they evolve from generic Foundation Architectures to Organization-Specific Architectures". In other words, it is not the repository itself, but a way of classifying the content of the repository.

The relationship between the generic and the specific has always been a critical dimension of architecture, as I've argued on this blog many times (see label: generic v specific). Enterprise Continuum provides a framework for architects to reason about the right level of generality or specificity for a given artefact.

Here are some implications of this framework.

• Progressive Refinement - Many methods assume progression from the generic to the specific, sometimes known as "refinement". But this is not the only possible approach, and is not always the most appropriate. (See my posts on Progressive Design Constraints and Analyzing the Rusty Lawnmower.)
  

• Management of Variation - Capabilities can often be decomposed into highly generic elements and asset-specific elements. (See my example of Green Coffee Beans.) Decoupling these capabilities helps to drive higher levels of cross-process and cross-organizational sharing. With SOA, this can be supported by separate architectural layers. (See my posts on Homogeneous Business Vocabulary and Merging Capability Modelling with Process Modelling.)

• Adaptation and Adaptability - To achieve agility or flexibility, an enterprise or system or solution may need to be under-determined. This is sometimes confused with abstraction. (See my posts on Adaptation and Adaptability.)
  

• Strategic Differentiation - Enterprise architects need to understand the ways in which a given company is similar to any other company in the same industy sector, and what particular things differentiate this company from other companies. (See my posts The General and The Particular and Service Planning.)

But although this is a critical question for architects, it is poorly supported by the prevailing tools and methods, which often reduce the question to simplistic abstraction hierarchies. Let us hope that TOGAF 9 stimulates further development in this area.

Learning by Doing

Harry Pierson posts on software (especially social software) Getting Better With Use, and refers to the keynote discussion between Bill Gates and Tim O'Reilly at Mix06 (transcript).

As I blogged at the time, I thought Bill didn't quite answer the question Tim had in mind. Bill's notion of software getting better was largely about Microsoft's ability to improve products (including security features), which was amplified by a larger install volume. This is reasonably close to the classic economic notion of Learning by Doing (see note).

Microsoft also benefits from the network externalities of products such as MS-Office. Each user gets value from interoperability with other MS-Office users - so the more users, the more potential value. But this can hardly be characterized as MS-Office itself getting better. (Tim dismisses this as "Network Effect 1.0".)

Another way software can get better is through a learning process embedded in the software, whereby the software adapts to the preferences and history of the user. As Harry explains, such personal software has to be "designed to learn". And in any case, this could be regarded merely as adaptation rather than real learning. (The software itself doesn't get any better, but a single instance gets more tightly aligned to a particular user.)

But, as Harry points out, Tim's notion of Web 2.0 includes an entirely different notion of software getting better with use.

Every ... consumer brings his own resources to the party. There's an implicit "architecture of participation", a built-in ethic of cooperation, in which the service acts primarily as an intelligent broker, connecting the edges to each other and harnessing the power of the users themselves.

The community is at the centre. Tim is certainly not just talking (as Bill does) about a community on the side, contributing the occasional add-on or comment.

Tim and Harry both use the word "automatic", but I think this word needs to be treated with some caution. What they are talking about is a social mechanism that (like most social mechanisms) works when it works, that may be amenable to some social engineering, but isn't "automatic" in the sense of a fixed mechanism with predictable results.

---

This is not a blog about Web 2.0 but about SOA and the service-based business, so I wanted finally to link the discussion of learning-by-doing back to SOA.

The benefits of SOA are often stated in terms of adaptability, agility, flexibility, responsiveness. But this implies processes of adaptation and learning. So we need to ask: How do business systems (both organizational and technical) improve? Where is the learning located? What is the nature of the feedback loop?

1. The learning loop goes through the software developers. (The software development acts as a gate/brake on the learning process.)
   
2. A learning process is contained in the software or service. (Learning can take place in real-time, but only for things that have been explicitly anticipated in software development.)
   
3. The learning process is distributed across the community usage of the software or service.

In planning for SOA, we need to think about these learning processes, and how they may be accommodated in any system architecture. SOA gives us new and more flexible ways of implementing such learning processes, but only if we identify the learning requirements properly. We are going to need a business model that includes the learning capabilities as well as the operational capabilities, and an architecture that mobilizes these capabilities in a loosely coupled manner.

---

Note: In economics, Learning by Doing refers to the theory that increased production volume can result in improvements in supply-side efficiency and effectiveness. This is the use of the term currently preferred by Wikipedia. Not to be confused with the popular use of the term in education, where it refers to theories about hands-on experience ("monkey see, monkey do").

Technorati Tags: learning SOA service-oriented social software Web 2.0

Capability and Business Strategy

In their book The Only Sustainable Edge, John Hagel and John Seely Brown describe two main schools of business strategy: the Core Competence school and the Leverage school. The approach advocated by Hagel and Seely-Brown represents a synthesis of these two schools. A closely related approach is advocated by David Alberts ("Power to the Edge").

	Core Competence	Leverage	Sustainable Edge
Focus	Resource-Based	Network-Based	Edge-Based
Source of Strategic Advantage	Identifying and strengthening core competences within the firm	Mobilizing resources outside the firm	Taking power to the edge of the organization.
Key advocates	Gary Hamel and C.K. Prahalad	Adam Brandenburger and Barry Nalebuff, James Moore	John Hagel and John Seely Brown, David Alberts.

Whereas the Core Competence school operates with a fixed notion of the required capability of the firm, the Sustainable Edge school involves dynamic specialization, connectivity and leveraged capability building. In other words, a successful business organization requires meta-capability – the capability to build capability, and the capability to build capability-building partnerships.

In our view, this kind of meta-capability is essential for the adaptive enterprise – after all, the defining capability of the adaptive enterprise is the capacity to adapt. However, in order to implement the adaptive enterprise, it is necessary to be specific about the nature of adaptation and adaptive capability required in a given context.

The capability model allows us to reason about which capabilities (or meta-capabilities) should be regarded as core competences.

 

Extract from Business Modeling for SOA - Part 1 What the Business Does (January 2006). See also Business-Driven SOA (May 2004), Business-Driven SOA 2 (June 2004), Business Systems Planning for SOA (May 2006). All available from CBDI Journal Archive.

Inscription and Loose Coupling

While searching for something else, I happened upon a dissertation on Business Processes and IT in the Pharmaceutical Industry, by Kai A. Simon. The dissertation contains a lot of useful material, including a comparison between the change management practices of several large consulting houses. (PDF is available via the Waellisch website.)

But what I want to talk about here is a theoretical aspect of loose coupling. Simon derives a theory of loose coupling from Madeleine Akrich's concept of inscription.

The relation between global and local aspects of an infrastructure can be analyzed through the concept of inscription (Akrich 1992). ... Taking this point of departure, we can describe how inscription occurs at technology and organizational level and what impact it has on the relation between IT and organization.

• Technology inscription can be defined as the rigidity of the technology in constraining the users in the way they are related to the technical object. In other words, it refers to the way technological systems can be used within or outside their design and which forms of work-arounds the system allows or prevents.
• Organizational inscription, on the other hand, reflects the level of freedom or rigidity in organizational procedures or, in other words, the extent to which organizational agents are allowed to reshape the ways in which the technical object are used with respect to organizational rules.

To the extent that technology inscription and organizational inscription are orthogonal, we get a 2x2 matrix.

Here is Simon's account of the four quadrants, including some comments on the implications for BPR and process improvement.

 

Strict alignment 

In this case, the design of organizational procedures leaves no room for local adaptation. At the same time, technology is rigid: There is no option for use outside the defined context. Standardization of technology and organizational procedures and strict alignment between these elements typically characterize the infrastructure. 

In most process improvement initiatives, the aim is to develop and implement a strictly aligned organizational and technical infrastructure, following a pre-defined process design and using information systems that are supporting this design efficiently. 

 

Rigid Technology

Organizational procedures are open for local adaptation, while technology does not permit changes in use. Infrastructure is characterized by tensions between global and local organization procedures aiming at satisfying the same objectives, but differing in the means for their achievement. 

[Simon describes a change management project that fell into this category, despite the original intention to develop a strictly aligned infrastructure.] The reason can be found in the lack of control that was exercised with regard to process compliance. It was assumed that all monitors would comply with the globally designed process and senior management was not aware of the local adaptations that took place.

 

Loose coupling

Organizational procedures and technology use can be redefined and adapted locally. The infrastructure allows adaptation to internal and environmental dynamics and is typical of knowledge intensive organizations. 

[Simon describes a change management project with this intention, which was discontinued after the organization underwent a merger.] 

 

Rigid organization

In this context, organizational procedures are strictly defined at global level, while technology is open for modifications. The infrastructure is characterized by tensions between different technologies adopted at local level, or local variations in technology use. This context is typical for a post-merger situation, where the merging firms are aiming at developing a common and standardized set of organizational procedures, but maintain their individual technical infrastructures.

As described here, loose coupling gives you THREE different types of flexibility - technological flexibility, organizational flexibility AND flexibility in the relationship between technology and organization.

Reference: AKRICH M., 1992, «The De-scription of Technical Objects» , in BIJKER W., LAW J., (ed.), Shaping Technology/Building Society. Studies in Sociotechnical Change, Cambridge Mass., MIT Press, p.205-224.

---

UPDATE: I have just received an email from Kai Simon, who writes:

The version of my dissertation you have found on the Waellisch site is not the final one, but a draft that was modified in several aspects before it was published. E.g., the final version does not contain a comparison of four consulting approaches, but only of those that were used in the case study company. 

However, the section you analyse has not been changed. You can find the final version in the publication section of my web site (http://www.informatik.gu.se/~kai 

The section on inscription and loose coupling is based on the work that I have conducted with Antonio Cordella, one of my colleagues from my time at the Viktoria Institute in Sweden (http://www.viktoria.se). So it was a joined idea and I would like to give Antonio the credits he deserves.

Our concept was the result of our research, but we also received valuable input from the late Claudio Ciborra. He was in the lead for the book "From Control to Drift - The Dynamics of Corporate Information Infrastructures" (Oxford University Press) that summarizes the results of multiple case studies on the manageability of IT infrastructures.
 

Thanks Kai

Sense and Respond

A couple of blog postings I've read in the past week or so (ERP4IT and Patrick Kerpan) have led me back to the work of Paul Strassmann (website, blog, photo). (I have met Strassmann a long time ago, he visited us at JMA in the 1980s, there was talk of some collaboration, but I don't think anything ever came of it.)

Browsing Strassmann's website, I found a matrix, which he had loaded onto Flickr. [Update: matrix no longer available.] For a brief explanation of the matrix, see his commentary on The Next Wave of IT Investments.

The benefits of IT are often expressed predominantly in terms of time (making things faster). Strassmann's matrix has a column for time, but also a column for scope. As I've argued before, we need to think about the abstract geometries of IT-enabled business as a whole and not just the linear properties of specific business processes.

Strassmann puts "sense and respond" in the final row of the matrix. I think this is consistent with a vision of on-demand autonomic computing that goes way beyond the levels of adaptability and responsiveness that IT vendors are currently offering.

There are several overlapping versions of sense-and-respond in circulation.

Short term	IT Vendors	Fujitsu has a registered trademark on the term Sense and Respond for its consultancy-based approach to problem-solving.
Medium term	Business School	The term Sense and Respond is used in relation to the Adaptive Enterprise. See for example the work of Stephen Haeckel (book, website). See also this article on the Sense-and-Respond Enterprise.
Longer term	Technical Research	Academic research is underway on the Event Web. See the Infospheres Project at Caltech. See also these articles on Event-Driven Architecture: 1 2 3 4.

Reposted because of Blogger error plus minor corrections.
Fujitsu link changed.

Value of Emptiness

One of the perceived benefits of SOA is business agility and system adaptability. In this post, I am going to discuss some aspects of this.

---

There is a lot of vague and woolly talk about business agility - especially from the software world, where a wide variety of software products, platforms and paradigms are optimistically supposed to have some magical effect on flexibility. But try selling flexibility to the CFO of a large company. ("Excuse me sir, would you like to buy some flexibility? It will only cost you ten million dollars." "Exactly how much flexibility do I get for ten million dollars?" "Ooo, loads and loads, honest. Look, here's a graph we've just made up. And a 2x2 matrix.")

---

Among other things, business agility means keeping your options open.

• Options are worth more when conditions are uncertain.
• In financial circles, the value of options is calculated using the Brook-Scholes formula. Stock options increase in value with the volatility of the underlying stock. This encourages risk-taking by executives.
• Real-Option theory applies the same financial logic to management options. See book Real Options by Amram and Kulatilaka.

In the "plug-and-play" service economy, the business may gain option-value in several ways:

1. the ability to replace specific partners
2. the ability to flex the boundary - moving specific services inwards or outwards across the organization boundary
3. the ability to access heterogeneous domains
   
4. the ability to change the configuration (geometry)
   

To go beyond bland optimism, we need to think rigorously about the business value of openness and extensibility, and how this value can be achieved by suitable configurations of organizational and technical systems - including SOA.

---

Technical artefacts (such as computers) often have empty expansion slots. Ben Hyde discusses the option value of empty slots - who benefits from this unused real-estate?

When the vendor creates a slot he’s relinquishing control over some amount of value-creating energy implicit in his offering.

If there is anything consistent about how they get filled in - for example they all start sporting graphics cards - that’s not stable. The industry will absorb any consistent slot usage into the core.

The space of empty slots [... appears to be ...] a long tail [... which ...] actually has negative value. Since the majority of buyers don’t use the slots they are getting no value from them. The hardware makers are including them not because of buyer demand but because the dominant players in the market want them. The dominant players in the market want them because they tap into the value created by the generative process that all that empty real estate creates.

In other words, the expansion slots provide some temporary free space for innovation. But the big players monitor how this free space is used, and will seek to recapture any rich pickings.

Martin Geddes discusses a number of other technologies (past and present) that have added value by providing options. Many of these were initially dismissed as inefficient. Martin's list includes relational databases and XML - we might add web services.

---

Compare this with the adaptability of buildings.

Lots of old houses were built with outside toilets and no bathroom. Most of those that survive today have had inside toilets and bathrooms added. Many people convert loftspace to make extra rooms, or add an extension into the back garden. Old houses are often relatively easy to alter or extend, subject to planning regulations.

Construction companies build new houses on tight estates with shallow roofs - so there is insufficient space for expansion. Bathrooms are attached to so-called master bedrooms, thus constraining alternative ways of using the space. It is as if the construction companies deliberately set out to build houses that are pre-adapted to a specific domestic norm, thus forcing people to move house every time their domestic requirements changed.

See Stewart Brand: How Buildings Learn.

---

The human being is a highly inefficient machine, with few natural advantages over other species and practically no innate skills. It takes many years before it can find its own food, or run half as fast as an animal. It has a large brain, consuming vast amounts of energy and other nutrients. The exceptionally long period of dependence, together with its feeble skills in relation to other animals, represents a considerable biological burden.

How on earth does this gross inefficiency survive? In complex and dynamic environments, there is a compensating evolutionary advantage - the human has lots of expansion slots - can learn new skills, including collective skills. It can develop new languages - both natural languages and problem-specific vocabularies. (DSL anybody?)

---

So what are the expansion slots of a business? Obviously new products, new suppliers, new business partners, new channels. But more importantly, new responses to changing demand. With a stratified model of the demand-side, and with appropriate supply-side platforms, we should be able to (re)design a business to deliver requisite levels of agility. And this will naturally include space for innovation.

---

Update: Stewart Brand has now introduced the term pace layering for the principle that stratification should be based on the differential rate of change. The term shearing layers refers to what happens when this principle is not followed.

See also Andrew K Johnston, Business Flexibility - An Analogy (2005)

Adaptation and Adaptability

Adam Brown from John McAslan and Partners gave a talk recently about his restoration/renovation work with Swiss Cottage Library. Dan Hill has some notes on his blog.

Interesting aside on how difficult it is to work with modernist buildings given their focus on functionality - if the function changes over time, the building can resist change. With the library, Brown kept alluding to how difficult it was to work with certain layers, given the amount of change required (not just in contemporary services etc, but in building in the modern notion of what a library is (internet access, coffee shops, DVD lending - as well as books).

This leads to a discussion about the adaptability of a modernist architecture.

1. Basil Spence is undoubtedly a great architect in many ways, and the Swiss Cottage Library remains beautiful, even if its functionality is now somewhat dated.

2. Functionalism involves a high degree of adaptation (to a given conception of function/functionality).

3. Adaptation conflicts with adaptability. The more you optimize to the present, the more you close off alternative futures.

4. Great buildings should be able to accommodate change, age gracefully. Perhaps one of the errors of modernism was to imagine that change would no longer be necessary.

---

The modernist attitude is widespread in software engineering. Model-driven architecture represents a type of modernism: form should follow function.

Stewart Brand's book on How Buildings Learn contains the notion of Shearing Layers - functional layers that have a different natural rate of change. A flexible structure allows each layer to be changed independently. But where the layers are too tightly coupled, the differential rates of change tear the structure apart.



The Shearing Layers concept is highly relevant to SOA.

---

Update: Stewart Brand has now introduced the term pace layering for the principle that stratification should be based on the differential rate of change. The term "shearing layers" refers to what happens when this principle is not followed.

Half Brain Half Biscuit

IBM's acquisition of AlphaBlox/HalfBrain has prompted a fight between Barry Briggs and Phil Wainewright about fat/thin clients, with additional comments from Eric Newcomer (Iona).

• Applications with Half a Brain
• Rich clients, network wealth
• Thick or Poor?
• It's the mobile device, not the server

I can remember similar fights ever since the good ol' days of client/server. And in the past some IT bosses have taken public stands one way or the other.

But the point about SOA is adaptability. Architectural designs must be structured to leave open those things we are uncertain about. Which means that since we are uncertain about the tactical advantages of client/server in the future, we simply define the required services without binding them to the specific location. Then in principle the fatness of the client can be changed at whim, perhaps even shifted dynamically halfway through an operation according to some network load policies.

In a previous post, I wrote about the implications of this for mobile computing.

You'll notice that Bill and Scott don't fight about this anymore. They obviously understand SOA and its implications.