Data Strategy - More on Agility

Continuing my exploration of the four dimensions of Data Strategy. In this post, I bring together some earlier themes, including Pace Layering and Trimodal.

The first point to emphasize is that there are many elements to your overall data strategy, and these don't all work at the same tempo. Data-driven design methodologies such as Information Engineering (especially the James Martin version) were based on the premise that the data model was more permanent than the process model, but it turns out that this is only true for certain categories of data.

So one of the critical requirements for your data strategy is to manage both the slow-moving stable elements and the fast-moving agile elements. This calls for a layered approach, where each layer has a different rate of change, known as pace-layering.

The concept of pace-layering was introduced by Stewart Brand. In 1994, he wrote a brilliant and controversial book about architecture, How Buildings Change, which among other things contained a theory about evolutionary change in complex systems based on earlier work by the architect Frank Duffy. Although Brand originally referred to the theory as Shearing Layers, by the time of his 1999 book he had switched to calling it Pace Layering. If there is a difference between the two, Shearing Layers is primarily a descriptive theory about how change happens in complex systems, while Pace Layering is primarily an architectural principle for the design of resilient systems-of-systems.

In 2006, I was working as a software industry analyst, specializing in Service-Oriented Architecture (SOA). Microsoft invited me to Las Vegas to participate in a workshop with other industry analysts, where (among other things) I drew the following layered picture.

SPARK Workshop Day 2

Here's how I now draw the same picture for data strategy. It also includes a rough mapping to the Trimodal approach.

---

Giles Slinger and Rupert Morrison, Will Organization Design Be Affected By Big Data? (J Org Design Vol 3 No 3, 2014)

Wikipedia: Information Engineering, Shearing Layers 

Related Posts: Layering Principles (March 2005), SPARK 2 - Innovation or Trust (March 2006), Enterprise Tempo (October 2010), Beyond Bimodal (May 2016), Data Strategy - Agility (December 2019)

Data Strategy - Agility

This is one of a series of posts looking at the four key dimensions of data and information that must be addressed in a data strategy - reach, richness, agility and assurance.

---

In previous posts, I looked at Reach, which is about the range of data sources and destinations, and Richness, which is about the complexity of data. Now let me turn to Agility - the speed and flexibility of response to new opportunities and changing requirements.

Not surprisingly, lots of people are talking about data agility, including some who want to persuade you that their products and technologies will help you to achieve it. Here are a few of them.

Data agility is when your data can move at the speed of your business. For companies to achieve true data agility, they need to be able to access the data they need, when and where they need it. Pinckney

Collecting first-party data across the customer lifecycle at speed and scale. Jones

Keep up with an explosion of data. ... For many enterprises, their ability to collect data has surpassed their ability to organize it quickly enough for analysis and action. Scott

How quickly and efficiently you can turn data into accurate insights. Tuchen

But before we look at technological solutions for data agility, we need to understand the requirements. The first thing is to empower, enable and encourage people and teams to operate at a good tempo when working with data and intelligence, with fast feedback and learning loops.

Under a trimodal approach, for example, pioneers are expected to operate at a faster tempo, setting up quick experiments, so they should not be put under the same kind of governance as settlers and town planners. Data scientists often operate in pioneer mode, experimenting with algorithms that might turn out to help the business, but often don't. Obviously that doesn't mean zero governance, but appropriate governance. People need to understand what kinds of risk-taking are accepted or even encouraged, and what should be avoided. In some organizations, this will mean a shift in culture.

Beyond trimodal, there is a push towards self-service ("citizen") data and intelligence. This means encouraging and enabling active participation from people who are not doing this on a full-time basis, and may have lower levels of specialist knowledge and skill.

Besides knowledge and skills, there are other important enablers that people need to work with data. They need to be able to navigate and interpret, and this calls for meaningful metadata, such as data dictionaries and catalogues. They also need proper tools and platforms. Above all, they need an awareness of what is possible, and how it might be useful.

Meanwhile, enabling people to work quickly and effectively with data is not just about giving them relevant information, along with decent tools and training. It's also about removing the obstacles.

Obstacles? What obstacles?

In most large organizations, there is some degree of duplication and fragmentation of data across enterprise systems. There are many reasons why this happens, and the effects may be felt in various areas of the business, degrading the performance and efficiency of various business functions, as well as compromising the quality and consistency of management information. System interoperability may be inadequate, resulting in complicated workflows and error-prone operations.

But perhaps the most important effect is on inhibiting innovation. Any new IT initiative will need either to plug into the available data stores or create new ones. If this is to be done without adding further to technical debt, then the data engineering (including integration and migration) can often be more laborious than building the new functionality the business wants.

Depending on whom you talk to, this challenge can be framed in various ways - data engineering, data integration and integrity, data quality, master data management. The MDM vendors will suggest one approach, the iPaaS vendors will suggest another approach, and so on. Before you get lured along a particular path, it might be as well to understand what your requirements actually are, and how these fit into your overall data strategy.

And of course your data strategy needs to allow for future growth and discovery. It's no good implementing a single source of truth or a universal API to meet your current view of CUSTOMER or PRODUCT, unless this solution is capable of evolving as your data requirements evolve, with ever-increasing reach and richness. As I've often discussed on this blog before, one approach to building in flexibility is to use appropriate architectural patterns, such as loose coupling and layering, which should give you some level of protection against future variation and changing requirements, and such patterns should probably feature somewhere in your data strategy.

Next post - Assurance

---

Richard Jones, Agility and Data: The Heart of a Digital Experience Strategy (WayIn, 22 November 2018)

Tom Pinckney, What's Data Agility Anyway (Braze Magazine, 25 March 2019)

Jim Scott, Why Data Agility is a Key Driver of Big Data Technology Development (24 March 2015)

Mike Tuchen, Do You Have the Data Agility Your Business Needs? (Talend, 14 June 2017)

Related posts: Enterprise OODA (April 2012), Beyond Trimodal: Citizens and Tourists (November 2019)

Complexity is not a problem

There is a common view in the enterprise architecture world that complexity is a big problem, perhaps the biggest problem, and that the primary task of enterprise architecture is to deal with this complexity.

• "The most pressing problem of the organization is complexity. Complexity is the primary responsibility of the enterprise architect." (Roger Sessions, Controlling Complexity in Enterprise Architecture pdf June 2007)

• "Enterprises are instances of complex adaptive systems having many interacting subcomponents whose interactions yield complex behaviors.  Enterprise Architecture is a way of understanding and managing such complexity." (Beryl Bellman Managing Organizational Complexity pdf FEAC Oct 2009)

Indeed, I'm sure I've said things like this myself. But if complexity is a problem, whose problem is it? I am not seeing a huge rush of businessmen hiring enterprise architects just to deal with The Complexity Problem. Usually they have much more practical problems that they want addressing.

Complexity is a problem amplifier

So here's the thing. Apart from architects, people generally don't see complexity as their problem. What they do often acknowledge, however, is that complexity makes their problems worse. Furthermore, complexity may be one of the reasons they can't solve their problems for themselves. So complexity is a relevant factor, it's just not the problem itself.

Complexity as a lifestyle choice

And where does the complexity come from? Ironically, complexity is often created by failed attempts to reduce or eliminate complexity. In a post about the AntiFragile Enterprise (Jan 2013), Alan Hakami warns against "a tendency to build over-governed solutions that try to 'manage' complexity or uncertainty".

And where is the motivation to eliminate complexity? Suppose you go to the doctor with a headache, and the doctor says the reason you keep getting these headaches is you don't get enough fresh air and exercise. The pills just give you a temporary relief from the symptoms. You know she's right, but somehow you can't manage to get up early enough to go for a run before work. So you continue to get the headaches and you continue to need the pills. Indeed, if the pills work, you may start to exercise even less than you did before.

According to this analogy, an organization chooses the level of complexity it is comfortable with, and may well resist attempts to shift to a higher or lower level.

Complexity as a smokescreen

In my post on Devious Management and Investment Risk (January 2004), I suggested that complexity can sometimes be a deliberate tactic to conceal something, and therefore serves as a clue that something is being concealed. For example, a tangle of complicated transactions. Obviously those responsible for the concealment will resist any attempt to strip away this kind of complexity. So there is no point in attacking the complexity directly, you need to identify what is behind it.

Complexity is an opportunity amplifier

If one organization is better at handling complexity than its competitors, as a consequence of superior agility and/or intelligence, then it can use complexity as a weapon. And many organizations use this weapon against customers or regulators. As @JackGavigan says, complexity is only a problem for those who lack the brain power to deal with and exploit it.

So that gives the enterprise architect a rather different perspective on complexity, doesn't it?

---

Related posts: Complexity-Based Pricing (June 2008), On the Causes of Business Complexity (October 2012)

Updated 27 June 2020

Enterprise OODA

In response to @Griff0Jones, I promised to beef up the coverage of OODA in my book on Organizational Intelligence (draft now available via LeanPub). I should welcome any comments and suggestions on the following, as well as pointers to any practical examples.

---

The choices we make at the personal level are influenced by our experiences and our environment. We are not always fully aware of these influences, and may need someone else to point them out to us. The same is true of the strategic and operational choices made by organizations.

In a rapidly changing environment, we need a feedback loop that continuously aligns our behaviour to these changes. This is an important aspect of agility. And in a competitive situation, competitive success depends on our being more agile than our competitors - in other words, having a faster and more accurate loop.

A good model for this is the OODA (Observe, Orient, Decide, Act) loop created by John Boyd. Some people confuse this with the PDCA (Plan, Do, Check, Act) loop popularized by Shewhart and Deming. However, the key difference between PDCA and OODA is the explicit inclusion of sense-making, which Boyd calls Orientation. Boyd himself produced a second model, IOHAI, which is largely an expansion on the sense-making area.



In order for the OODA loop to produce real agility, there needs to be agility in each of its parts.

Agile Observation. One criticism that has been levelled at the OODA loop (for example Benson and Rotkoff) is the tendency for what you observe to be narrowed to just those things that seem to help with decision making. (David Murphy describes this tendency as inevitable.)

Simon Thornington raises a related issue in a comment to David Murphy's blog. So much of what is observed is via instrumentation (or alternatively the output of models). Without the observer having prior knowledge of the construction and assumptions of the models, he or she cannot orient effectively based on the observations. Simon suggests that this was a factor in the recent Airbus crash, various space program mishaps and throughout finance.


Agile Orientation. David Murphy points out that really big risks are often not acted upon because we are oriented so that we cannot decide, and reminds us what happened to those people who tried to act against the firm’s orientation at MF Global, or Enron or HBOS.

Discussing strategic planning through the OODA lens, Henrik Mårtensson points out the importance of orientation. If we only know one strategic paradigm, and choose a strategic method from within the range of options provided by the paradigm, we loose (sic) the ability to improve beyond what the paradigm allows. ... Boyd believed it is absolutely necessary to be able to switch paradigms at will.


Agile Decision, Agile Action. Henrik argues that operating with a crippled OODA loop and a strategic model that separates strategy and action may not kill you, but the faster the environment changes, the more hampered your organization will be by its own strategic model. Henrik recommends William Dettmer's Strategic Navigation, which in his opinion combines the principles of Maneuver Warfare with the analysis and planning tools of The Logical Thinking Process. The result is fast high quality strategic planning, and seamless integration between planning and execution.

---

Sources and further reading

Kevin Benson and Steven Rotoff, Goodbye OODA Loop (Armed Forces Journal, October 2011)

Blogs by Henrik Mårtensson, David Murphy, Chet Richards, and Spartan Cops.

A Twin-Track Approach to Government IT

#ukgovit @instituteforgov has just published a report called System Error: Fixing the Flaws in Government IT.

The report recommends a twin-track approach to government IT, based on the two concepts of Agile and Platform.

"The platform must standardise and simplify core elements of government IT. For any elements of IT outside the platform, new opportunities should be explored using agile principles. These twin approaches should be mutually reinforcing: the platform frees up resource to focus on new opportunities while successful agile innovations are rapidly scaled up when incorporated into the platform."

The report acknowledges the tension between these two concepts ...

"Treating items as commodities reduces cost but can limit flexibility; coordinating elements of IT across departments frees up resources but may move them further from frontline users; common standards support interoperability but also restrict the freedoms to innovate."

... and offers some general ideas for managing this tension.

• To act fully in the interests of government, an agile approach requires a light touch form of coordination at a system level. 

• To minimise duplication of effort in solving the same problems, there needs to be system-wide transparency of agile initiatives. 

• Existing elements of the platform also need periodic challenge. ... Transparency, publishing feedback and the results of experiments openly, will help to keep the pressure on the platform for continual improvement as well as short-term cost savings.

Trouble is, some of this stuff is really hard. The report talks glibly about "a less than intelligent customer", referring first to business users having an inadequate conception of the possible, and then to the public sector as a whole lacking the collective knowledge and skills to negotiate effectively with suppliers. This lack of intelligence is apparently blamed on the V-model development process, which creates the impression that the adoption of Agile methods would solve this problem. But the idea of Agile as a silver bullet is a dangerous one, as many people have already pointed out on the Linked-In discussion group.

One way of understanding the twin track approach is to think of the different kinds of economics involved.

• 'Platform' means delivering economies of scale and economics of scope.
• 'Agile' means delivering economies of alignment.

Combining the two introduces some complex architectural challenges, as I've written about here and elsewhere before. We call this Asymmetric Design. For an example of this approach applied to public sector IT, see an analysis of the CSA Case by Philip Boxer and myself. See also The Impact of Governance Approaches on SoS Environments (pdf) by Philip Boxer and others.


In the current economic situation, the public sector as a whole is charged with making massive cost savings, and it is crazy to imagine that cost savings of this scale would not be associated with significant structural change, including IT systems. This kind of disruptive innovation goes way beyond the economies of scale and scope, and introduces some serious questions about the economics of alignment.

The word "architecture" is mentioned a few times in the Institute for Government report, but only in passing as something that the Government CIO will look after. (Mostly technology or solution architecture, I only found one single reference to business architecture.) So there is an implicit idea of central thinking and hierarchical governance. But there are some architectural challenges here that are some way beyond the current practices of enterprise architecture.

Governance is also a significant problem. The report comments on the pendulum swings between centralized and decentralized provision, which is something we noted in the CSA case, and was also present in the case of ContactPoint (which we were in the middle of writing up when it was cancelled). Such pendulum swings are often a characteristic symptom of weak or unsustained governance.

Not only is this stuff structurally complicated, but there are some commercial stakeholders that have every incentive to maintain the complicated status quo, thanks to a grossly dysfunctional procurement process.

And there is an even bigger problem with the report, which is that it looks at government IT exclusively from within government - in other words, from the perspective of civil servants. For example, the report adopts a supply-side notion of "joined-up government", understood largely in terms of internal linkages and efficiencies between systems, and fails to mention the demand-side notion of "joined-up government" that involves a coherent experience for the citizen. (See my post on Joined-Up Government from December 2005.)

Meanwhile the notion of "user" appears to refer mainly to civil servants and other public sector workers. Surely the purpose of government IT is not to provide direct value to civil servants but to provide various forms of indirect value to individual citizens and socioeconomic communities.

The report regrets that "government IT [is] falling further and further behind the fast-paced and exciting technological environment that citizens interact with daily" and indicates "the potential for IT ... fundamentally changing the relationship between citizen and state". "Around the world governments are using technology to help them deliver better services, be more transparent and accountable, and connect more directly with their citizens." (Examples are cited from Canada, USA and Malaysia.)

And yet the report fails to explain how "agile" can adequately represent the demand side requirements of citizens, interacting with a broad range of government services while going about their public business. There is a completely different notion of "platform" required here - government as a platform, which Tim O'Reilly and others have been talking about for a couple of years. And a different notion of agility, which goes a lot further than agile software development.

---

Other commentary

See Linked-In discussion group

Harry Metcalfe (2 March 2011) observed that many of the recommendations in the report were really hard, and was one of the first to complain about the insufficient attention to procurement in the report.

Ecosystem SOA 2

What are the problems of large complex sociotechnical systems? How far do SOA and enterprise architecture help to address this problem space, and what else might we need?


When I started writing about SOA and the service-based business over ten years ago, I defined two "cuts" across the service ecosystem. One cut separates inside from outside, while the other cut separates supply from demand.

(This diagram was included in my 2001 book on the Component-Based Business, and frequently referenced in my work for the CBDI Forum. For a brief extract from the book, see my Slideshare presentation on the Service Ecosystem.)

The inside/outside cut is sometimes called encapsulation. It decouples the external behaviour of a service from its internal implementation, and can be described in terms of knowledge - the outside has limited knowledge of the inside, and vice versa. (The cut is also sometimes called transparency - for example location transparency, which means that external viewers can't see where something is located.)

The supply/demand cut is about delegation, and can be described in terms of responsibility. Getting these two cuts right may yield economics of scale and scope; and the business case for SOA as a development paradigm is often formulated in terms of reusing and repurposing shared services.

For relatively small and simple SOA projects, it may be feasible to collapse the difference between these two cuts, and treat them as equivalent. (The inside/outside relationship and the supply/demand relationship are sometimes both described as "contracts", although they are clearly not the same kind of contract.) However, enterprise-scale SOA requires a proper articulation of both cuts: confusing them can result in suboptimal if not seriously dysfunctional governance and procurement. Many people in the SOA world still fail to understand the conceptual importance of these cuts, and this may help to explain why some organizations have had limited success with enterprise-scale SOA.

Going beyond enterprise SOA as it is generally understood, there is a third cut separating two views of a system: the system-as-designed (whose structure and behaviour and rules can perhaps be expressed in some formal syntax such as UML, BPMN or ArchiMate) and the system-in-use (whose actual performance is embedded/situated in a particular social or business context). This cut is critical for technology change management, because of the extent to which the designed system underdetermines the pragmatics of use. I have been talking about this cut for over twenty years, but only more recently working out how to articulate this cut in composition with the other two cuts.

One important reason for looking at the pragmatics of use is to understand the dimensions of agility. In many settings, we can see a complex array of systems and services forming a business platform, supporting a range of business activities. If no agility is required in the business, then it may not matter if the platform is inflexible, forcing the business activities to be carried out in a single standardized manner. But if we assume that agility is a critical requirement, then we need to understand how the flexibility of the platform supports the requisite variety of the business.

More generally, understanding the pragmatics of use leads to the recognition of a third kind of economic value alongside the economics of scale and the economics of scope: the economics of alignment. The value of a given system-of-systems depends on how it is used to deliver real (joined-up) business outcomes, across the full range of business demands. (I'm afraid I get impatient with people talking glibly and simplistically about business/IT alignment, without paying attention to the underlying complexity of this relationship.)

Understanding these three cuts (and analysing their implications) is critical to understanding and managing a whole range of complex systems problems - not just SOA and related technologies, not even just software architecture, but any large and complex sociotechnical systems (or systems-of-systems). If the three cuts are not understood, the people in charge of these systems tend not to ask the right questions. Questions of pragmatics are reduced to questions of platform design; while questions of the cost-justification and adoption of the platform are reduced to a simple top-down model of business value. Meanwhile the underlying business complexity (requisite variety) will be either misplaced (e.g. buried in the platform) or suppressed (e.g. constrained by the platform).

So there are three challenges I face as a consultant, attempting to tackle this kind of complex problem. The first challenge is to open up a new way of formulating the presenting problem, based on the three cuts. The second challenge is to introduce systematic techniques for analysing the problem and visualizing the key points. And the third challenge is to identify and support any organizational change that may be needed.

---

With thanks to Philip Boxer and Bernie Cohen. For a different formulation of the three cuts, together with a detailed example, see their new paper "Why Critical Systems Need Help to Evolve" Computer, vol. 43, no. 5, pp. 56-63, May 2010, doi:10.1109/MC.2010.150. See also Philip Boxer, When is a stratification not a universal hierarchy? (January 30th, 2007)


Related post Ecosystem SOA (October 2009)

Economics of agility 2

In my previous post on the Economics of Agility, I noted how little material has been published on this topic.

As Nicholas Whittall and Philip Boxer point out in their contribution to the recent debate on The Meaning of Value-for-Money in Defence Acquisition (RUSI, February 2009), there is an important link between agility and alignment. See also their earlier piece on Agility and Innovation in Acquisition (RUSI, February 2008).

The first observation is that defence acquisition - just like systems acquisition most anywhere - operates on a much slower tempo than the requirements of the business. The "business" of a military organization is running military campaigns; thus when writing for the defence community, Whittall and Boxer refer to the Campaign Tempo and the Acquisition Tempo.

The second observation is that there is a complex set of activities (such as orchestration, customization, and improvisation) involved in bridging between Demand (the demands of the campaign or business) and Supply (the procurement of specific systems and devices). These activities operate on an intermediate tempo, which Whittall and Boxer call the Alignment Tempo.

"Meeting the campaign tempo then depends on the alignment tempo possible, which in turn depends on the acquisition tempo at which gaps can be filled. Any slowness in acquisition tempo leads to increased bricolage and process short cuts to enable the alignment tempo to keep up with the campaign tempo. Thus, ‘agility’ finds its richest expression in the ability of the alignment tempo to meet the required campaign tempo at the lowest cost – i.e. to maximise the value-for-defence."

The challenge is then to produce just enough variety within the acquisition to optimize the economics of alignment. Boxer has developed a technique of Cohesion-Based Costing (not yet published), which "offers a means to attach a value to the cost of introducing flexibility". This kind of technique will clearly be of enormous benefit within the SOA world.

 

Related post Enterprise Tempo (October 2010)

Economics of agility

@andygreenawalt says he hasn't seen much on the economics of agility. "Open source breaks $ vs security battle. Investment impregnation kills agility."

Andy's comment arose from a discussion of agile security. See my article Agile Security for SOA (CBDI Journal, June 2005).

More generally, the economics of agility is critical to the business case for SOA, and a few of us have sometimes called it the SOA Sweet Spot (March 2007). See my article on the Agile Business Process (CBDI Journal, July-August 2007)

Looking around, I also found some work by Marc Rix called Bottom-Line SOA: The Economics of Agility (April 2007). See comments by Alastair Bathgate, Bill Barr and Gary Smith, with Marc's response.

And back in 2004, Stuart Robbins wrote something on the economics of agility for grid management (pdf).

But compared to the huge amount about the economics of scale (sharing, reuse), that's not much is it?

Agility and Variation 2

In a post called Kitchen Sink Variability, Harry "Devhawk" Pierson (Microsoft) lays into a research note by Ronald Smeltzer (Zapthink): Why Service Consumers and Service Providers Should Never Directly Communicate.

As Harry explains, there is a contradiction between Variability and the principles of Agile Development. (I discussed some aspects of this contradiction in my post on Agility and Variation.) Harry thinks Zapthink is advocating what he calls Kitchen Sink Variability - let everything vary except the kitchen sink.

Without architecture, this would indeed be a crazy idea.

Harry appeals to the ExtremeProgramming principle of YouArentGonnaNeedIt, which says "don't build in extra functionality". But does this principle apply to flexibility - does flexibility count as a kind of functionality? Should we really avoid building in flexibility?

If building in flexibility means constructing specific additional stuff (mechanisms, abstractions) to anticipate non-specific future variation, then this would seem to entail additional cost (development overhead, maintenance overhead, runtime overhead) with no clear benefits. We might reject this kind of thing as "over-engineering".

But if building in flexibility means using appropriate architectural patterns and existing technologies that allow you to ignore future variation, that seems to be a different thing altogether. Zapthink is talking about a specific pattern - decoupling the service consumer from the service provider - that is supported by current SOA platforms. That seems to make a lot of sense to me. But Zapthink justifies this pattern by appealing to a general principle of variability, and this is what has aroused Harry's scorn.

For my part, I don't think SOA means uncritically embracing all aspects of variation and variability. I have always said Loose Coupling is a choice, not a mandate. (See my earlier post Loose Coupling 2). But let's understand where the variation comes from. It is not because enterprise architects are going around promoting supply-side variation. More likely because many enterprise architects are struggling with increasing levels of demand-side variation. The traditional choice has always been to suppress as much variation as possible, but for many organizations (and their customers) this choice is no longer acceptable.

The relationship between variation and complexity is not linear. A truly agile architecture should be able to accommodate some increase in variation without increasing complexity by the same amount. But of course there is still some cost. SOA improves the trade-off, but there is still a trade-off.

The bottom line is that architects need to be able to reason intelligently about variation and variability. Not abstraction for the sake of abstraction, not decoupling for the sake of decoupling, but abstraction and decoupling driven by the needs of the enterprise. Loose Coupling is not a mandate but a choice.

Business Case for SOA 2

Andrew McAfee recently posted The Case Against the Business Case., which indicates why it's wrong (or at least dangerous) to make a business case in technological terms alone.

"I’ve probably seen hundreds of business cases that identify the benefits of adopting one piece of IT or another, assign a dollar value to those benefits, then ascribe that entire amount to the technology alone when calculating its ROI. The first two steps of this process are at best estimates, and at worst pure speculation. The final step gives no credit and assigns no value to contemporaneous individual- and organization-level changes."

I agree with this absolutely. I remember listening to a presentation on a successful computing project (I forget the details now) which ensured the commercial survival of some dockyard in the Far East. The speaker mentioned, almost as an aside, that there was some minor organization change required as well. And I remember thinking that there could easily be another presentation going on right now in a conference room somewhere else, where the consultants responsible for the organization change were boasting of their success in ensuring the commercial survival of the dockyard, and mentioning the computing project (if at all) as a minor side-condition.

So we have an excellent return on investment, if we just compare the cost of the computing project with the profitability of the dockyard. Presumably we also have an excellent return of investment if we just compare the cost of the organizational change with the profitability of the dockyard. But does the business case still work if you include both? The challenge for business case is scoping - where do you draw the line of what costs and benefits to include. Whose costs and whose benefits? (IT is notorious for neglecting user costs.) And what types of costs and benefits do you even recognise?

This has always been a problem for business case. Here's an example from a paper I wrote on "Reasoning about Systems and their Properties" (September 2000).

"During the 1980 UK mining strike, both sides constructed a plausible argument relating to the productivity of the system: one side claimed that it was more cost-effective to close the pits, the other side claimed that it was more cost-effective to keep them open. Apparently small differences in the way the system was scoped and in the choice of time horizon can have a large effect ..."

Andrew McAfee argues that a business case should be formulated not in terms of absolute business value, but in terms of the business capability you are getting in return for a given IT expenditure. It then becomes a business decision whether (and how much) to invest in a given capability.

This makes a lot of sense in terms of agility and flexibility as well. I have long argued that we should regard adaptability as a special kind of capability: adapt-ability, the ability to adapt. We need to show these capabilities on our business capability models, not just the operational business-as-usual capabilities. And we need to become much more precise about the adaption capability that a given technology (such as SOA) should be able to deliver.

Agility and Variation

There are several strands of agile development and quality management that are concerned with reducing (some measure of) variation in order to maximize (some measure of) performance and (some measure of) quality. Much of this thinking is derived from the quality guru W. Edwards Deming.

David J. Anderson, a follower of Goldratt's Theory of Constraints (TOC), has been discussing this recently in his Agile Management Weblog.

• Driving Variation Out
• Deming and Goldratt
• Reducing Variation in FDD

So I was interested to read Anderson's account of a Sushi Lunch, where he complained about the inflexible and unresponsive service provided in a top Japanese restaurant, and the dissatisfaction experienced especially by a junior member of the Anderson family. Anderson senior interprets this in terms of the Theory of Constraints:

This restaurant had a broken organizational structure and poor separation of responsibilities. The "Anderson lunch project" should rightly have been the responsibility of the waiter who should have been playing the project manager role (and maybe the program manager role). The waiter should have analyzed our requirements and understood our priorities. This should have been communicated to the chefs and the order of production of our sushi should have been negotiated against the competing orders at the time. The sushi chefs should have been purely responsible for the production of sushi. They should not have had any project management, program management or scheduling responsibility.

But I don't think this tells the whole story. It seems to me that the restaurant was unable and/or unwilling to accommodate the demand variation caused by the Anderson toddler. A supply-side optimization led directly to a poor demand-side experience, and indirectly to a supply-side disruption.

Traditional operational managers often believe their primary task is to achieve supply-side productivity by reducing variation. This view of the primary task encourages them to suppress demand-side variation, and to ignore demand-side signals that would complicate or contradict this view of the primary task.

What are the implications of this for service-oriented architectures? We now have technologies that support greater flexibility in managing demand-side variation in relation to supply-side variation. From a business point of view, this is one of the main benefits of SOA technologies.

 

See also The Universe at the End of the Restaurant (April 2009)


Links updated 18 October 2016, 18 October 2020.

Value of Emptiness

One of the perceived benefits of SOA is business agility and system adaptability. In this post, I am going to discuss some aspects of this.

---

There is a lot of vague and woolly talk about business agility - especially from the software world, where a wide variety of software products, platforms and paradigms are optimistically supposed to have some magical effect on flexibility. But try selling flexibility to the CFO of a large company. ("Excuse me sir, would you like to buy some flexibility? It will only cost you ten million dollars." "Exactly how much flexibility do I get for ten million dollars?" "Ooo, loads and loads, honest. Look, here's a graph we've just made up. And a 2x2 matrix.")

---

Among other things, business agility means keeping your options open.

• Options are worth more when conditions are uncertain.
• In financial circles, the value of options is calculated using the Brook-Scholes formula. Stock options increase in value with the volatility of the underlying stock. This encourages risk-taking by executives.
• Real-Option theory applies the same financial logic to management options. See book Real Options by Amram and Kulatilaka.

In the "plug-and-play" service economy, the business may gain option-value in several ways:

1. the ability to replace specific partners
2. the ability to flex the boundary - moving specific services inwards or outwards across the organization boundary
3. the ability to access heterogeneous domains
   
4. the ability to change the configuration (geometry)
   

To go beyond bland optimism, we need to think rigorously about the business value of openness and extensibility, and how this value can be achieved by suitable configurations of organizational and technical systems - including SOA.

---

Technical artefacts (such as computers) often have empty expansion slots. Ben Hyde discusses the option value of empty slots - who benefits from this unused real-estate?

When the vendor creates a slot he’s relinquishing control over some amount of value-creating energy implicit in his offering.

If there is anything consistent about how they get filled in - for example they all start sporting graphics cards - that’s not stable. The industry will absorb any consistent slot usage into the core.

The space of empty slots [... appears to be ...] a long tail [... which ...] actually has negative value. Since the majority of buyers don’t use the slots they are getting no value from them. The hardware makers are including them not because of buyer demand but because the dominant players in the market want them. The dominant players in the market want them because they tap into the value created by the generative process that all that empty real estate creates.

In other words, the expansion slots provide some temporary free space for innovation. But the big players monitor how this free space is used, and will seek to recapture any rich pickings.

Martin Geddes discusses a number of other technologies (past and present) that have added value by providing options. Many of these were initially dismissed as inefficient. Martin's list includes relational databases and XML - we might add web services.

---

Compare this with the adaptability of buildings.

Lots of old houses were built with outside toilets and no bathroom. Most of those that survive today have had inside toilets and bathrooms added. Many people convert loftspace to make extra rooms, or add an extension into the back garden. Old houses are often relatively easy to alter or extend, subject to planning regulations.

Construction companies build new houses on tight estates with shallow roofs - so there is insufficient space for expansion. Bathrooms are attached to so-called master bedrooms, thus constraining alternative ways of using the space. It is as if the construction companies deliberately set out to build houses that are pre-adapted to a specific domestic norm, thus forcing people to move house every time their domestic requirements changed.

See Stewart Brand: How Buildings Learn.

---

The human being is a highly inefficient machine, with few natural advantages over other species and practically no innate skills. It takes many years before it can find its own food, or run half as fast as an animal. It has a large brain, consuming vast amounts of energy and other nutrients. The exceptionally long period of dependence, together with its feeble skills in relation to other animals, represents a considerable biological burden.

How on earth does this gross inefficiency survive? In complex and dynamic environments, there is a compensating evolutionary advantage - the human has lots of expansion slots - can learn new skills, including collective skills. It can develop new languages - both natural languages and problem-specific vocabularies. (DSL anybody?)

---

So what are the expansion slots of a business? Obviously new products, new suppliers, new business partners, new channels. But more importantly, new responses to changing demand. With a stratified model of the demand-side, and with appropriate supply-side platforms, we should be able to (re)design a business to deliver requisite levels of agility. And this will naturally include space for innovation.

---

Update: Stewart Brand has now introduced the term pace layering for the principle that stratification should be based on the differential rate of change. The term shearing layers refers to what happens when this principle is not followed.

See also Andrew K Johnston, Business Flexibility - An Analogy (2005)

dotBiz

I have just come across an interesting talk by Shai Agassi of SAP, entitled Achieving Enterprise Agility.

Slides (pdf) from Accelerating Change 2004. Voicetrack (various formats, 18mB audio, 38 minutes) courtesy of IT Conversations.

The first 10 minutes or so provides some business background, including airline examples. Then he starts talking about composition, which I think is the most significant part of his talk. He has a vision of what he calls dotBiz (roughly equivalent to the Service-Based Business), for which he makes massive estimates of market size from around 2008 onwards. (After about 25 minutes he starts to rush through the rest of his material. It's probably not worth listening beyond this point.)

From the slides, you might get the impression of a ho-hum nothing-special software product architecture. But his talk implies something much more radical: a stratification of business.

At the top of the stack are niche companies with small solution-oriented microprocesses. This are assembled from "composites" - he refers to a "conveyor belt" of partly-assembled (orchestrated?) subprocesses. Below this are tens of thousands of business objects exposed as web services. These then sit on top of a layered enterprise platform, presumably supporting supply-side composition.

Agassi then characterizes the historical position of the big four, roughly as follows:

• IBM outsourcing non-core process - "on-demand" understood primarily in terms of variable volume of standardized process
• Microsoft preferred for local non-scaleable solutions, but not taken seriously for enterprise solutions - "start here but don't know whether it can scale"
• Oracle specializing in supporting big one-off differentiated process - "stuff you want to build on your own"
• SAP specializing in non-differentiated non-core process - "best practices that always run"

In other words, each represents a different mix of differentiation, integration and scale. Agassi's characterization provides a possible explanation of the current strategy of the big four: all trying to converge onto the same central ground, using SOA to get the best of all three worlds: differentiation, integration and scale. However, in this presentation, he doesn't get to the most interesting question: how business stratification helps this agenda.