Data Strategy - More on Agility

Continuing my exploration of the four dimensions of Data Strategy. In this post, I bring together some earlier themes, including Pace Layering and Trimodal.

The first point to emphasize is that there are many elements to your overall data strategy, and these don't all work at the same tempo. Data-driven design methodologies such as Information Engineering (especially the James Martin version) were based on the premise that the data model was more permanent than the process model, but it turns out that this is only true for certain categories of data.

So one of the critical requirements for your data strategy is to manage both the slow-moving stable elements and the fast-moving agile elements. This calls for a layered approach, where each layer has a different rate of change, known as pace-layering.

The concept of pace-layering was introduced by Stewart Brand. In 1994, he wrote a brilliant and controversial book about architecture, How Buildings Change, which among other things contained a theory about evolutionary change in complex systems based on earlier work by the architect Frank Duffy. Although Brand originally referred to the theory as Shearing Layers, by the time of his 1999 book he had switched to calling it Pace Layering. If there is a difference between the two, Shearing Layers is primarily a descriptive theory about how change happens in complex systems, while Pace Layering is primarily an architectural principle for the design of resilient systems-of-systems.

In 2006, I was working as a software industry analyst, specializing in Service-Oriented Architecture (SOA). Microsoft invited me to Las Vegas to participate in a workshop with other industry analysts, where (among other things) I drew the following layered picture.

SPARK Workshop Day 2

Here's how I now draw the same picture for data strategy. It also includes a rough mapping to the Trimodal approach.

---

Giles Slinger and Rupert Morrison, Will Organization Design Be Affected By Big Data? (J Org Design Vol 3 No 3, 2014)

Wikipedia: Information Engineering, Shearing Layers 

Related Posts: Layering Principles (March 2005), SPARK 2 - Innovation or Trust (March 2006), Enterprise Tempo (October 2010), Beyond Bimodal (May 2016), Data Strategy - Agility (December 2019)

Data Strategy - Agility

This is one of a series of posts looking at the four key dimensions of data and information that must be addressed in a data strategy - reach, richness, agility and assurance.

---

In previous posts, I looked at Reach, which is about the range of data sources and destinations, and Richness, which is about the complexity of data. Now let me turn to Agility - the speed and flexibility of response to new opportunities and changing requirements.

Not surprisingly, lots of people are talking about data agility, including some who want to persuade you that their products and technologies will help you to achieve it. Here are a few of them.

Data agility is when your data can move at the speed of your business. For companies to achieve true data agility, they need to be able to access the data they need, when and where they need it. Pinckney

Collecting first-party data across the customer lifecycle at speed and scale. Jones

Keep up with an explosion of data. ... For many enterprises, their ability to collect data has surpassed their ability to organize it quickly enough for analysis and action. Scott

How quickly and efficiently you can turn data into accurate insights. Tuchen

But before we look at technological solutions for data agility, we need to understand the requirements. The first thing is to empower, enable and encourage people and teams to operate at a good tempo when working with data and intelligence, with fast feedback and learning loops.

Under a trimodal approach, for example, pioneers are expected to operate at a faster tempo, setting up quick experiments, so they should not be put under the same kind of governance as settlers and town planners. Data scientists often operate in pioneer mode, experimenting with algorithms that might turn out to help the business, but often don't. Obviously that doesn't mean zero governance, but appropriate governance. People need to understand what kinds of risk-taking are accepted or even encouraged, and what should be avoided. In some organizations, this will mean a shift in culture.

Beyond trimodal, there is a push towards self-service ("citizen") data and intelligence. This means encouraging and enabling active participation from people who are not doing this on a full-time basis, and may have lower levels of specialist knowledge and skill.

Besides knowledge and skills, there are other important enablers that people need to work with data. They need to be able to navigate and interpret, and this calls for meaningful metadata, such as data dictionaries and catalogues. They also need proper tools and platforms. Above all, they need an awareness of what is possible, and how it might be useful.

Meanwhile, enabling people to work quickly and effectively with data is not just about giving them relevant information, along with decent tools and training. It's also about removing the obstacles.

Obstacles? What obstacles?

In most large organizations, there is some degree of duplication and fragmentation of data across enterprise systems. There are many reasons why this happens, and the effects may be felt in various areas of the business, degrading the performance and efficiency of various business functions, as well as compromising the quality and consistency of management information. System interoperability may be inadequate, resulting in complicated workflows and error-prone operations.

But perhaps the most important effect is on inhibiting innovation. Any new IT initiative will need either to plug into the available data stores or create new ones. If this is to be done without adding further to technical debt, then the data engineering (including integration and migration) can often be more laborious than building the new functionality the business wants.

Depending on whom you talk to, this challenge can be framed in various ways - data engineering, data integration and integrity, data quality, master data management. The MDM vendors will suggest one approach, the iPaaS vendors will suggest another approach, and so on. Before you get lured along a particular path, it might be as well to understand what your requirements actually are, and how these fit into your overall data strategy.

And of course your data strategy needs to allow for future growth and discovery. It's no good implementing a single source of truth or a universal API to meet your current view of CUSTOMER or PRODUCT, unless this solution is capable of evolving as your data requirements evolve, with ever-increasing reach and richness. As I've often discussed on this blog before, one approach to building in flexibility is to use appropriate architectural patterns, such as loose coupling and layering, which should give you some level of protection against future variation and changing requirements, and such patterns should probably feature somewhere in your data strategy.

Next post - Assurance

---

Richard Jones, Agility and Data: The Heart of a Digital Experience Strategy (WayIn, 22 November 2018)

Tom Pinckney, What's Data Agility Anyway (Braze Magazine, 25 March 2019)

Jim Scott, Why Data Agility is a Key Driver of Big Data Technology Development (24 March 2015)

Mike Tuchen, Do You Have the Data Agility Your Business Needs? (Talend, 14 June 2017)

Related posts: Enterprise OODA (April 2012), Beyond Trimodal: Citizens and Tourists (November 2019)

Bus Safety Announcement

Transport for London (TfL) reckons around 3000 people are injured every year by slips, trips and falls on London buses. So it is running trials of an automated system that announces the departure of the bus from the stop.

"Please hold on, the bus is about to move"

or as Bon Jovi might say

"We've gotta hold on ready or not."

The problem is that these alerts often come after the bus is already halfway down the road.

"Whoa, we're half-way there."

As the BBC News explains, the timing of the alert is based on the average amount of time a bus would spend at a bus stop, and is often hopelessly inaccurate. Passengers have taken to social media in droves to complain or mock. Many have wondered whether it was such a problem in the first place, and whether an alert would help to alleviate the problem. Others have pointed out the potential value of such an alert for certain categories of passenger - such as the elderly or visually impaired - but of course this only works if the alert comes at the right time.

---

I haven't spoken to anyone at TfL about this, but I can imagine what happened. In order to get a trial up and running quickly, they didn't have time (or permission) to link the alert with any of the systems on board the bus that could have sent a more accurate event signal. So we have a stand-alone system, knocked up quickly, as an experimental solution to a problem that most people hadn't previously recognized. In the trimodal scheme, this is a classic Pioneer project.

"For love we'll give it a shot."

So if the trial isn't laughed into touch, then maybe the Settlers can take over and do the alert properly.

"Take my hand, we'll make it. I swear."

And the Town Planners can come up with a joined-up long-term vision for passenger comfort and safety. Altogether now ...

"Whoa, livin' on a prayer."

---

Update

The wording of the announcement has changed, but the timing hasn't. It now says.

"Please hold on while the bus is moving"

What can I say to that?

"Standing on the ledge, I show the wind how to fly. When the world gets in my face, I say, have a nice day."

---

Londoners hit out at 'mistimed' bus safety alerts (BBC News, 14 January 2018)

Nadia Khomami, Please hold on: TfL urged to get a grip over annoying bus warnings (Guardian, 15 January 2018)

Eleanor Rose, TfL anger London commuters again with replacement bus announcement that is 'still annoying as hell' (Standard, 26 January 2018)

Londoners baffled by 'bonkers' bus safety announcements warning them 'the bus is about to move' (Evening Standard,15 January 2018).


For more on Trimodal IT, see my post Beyond Bimodal (May 2016)

Beyond Bimodal

Ten years ago (March 2006) I attended the SPARK workshop in Las Vegas, hosted by Microsoft and inspired by Christopher Alexander. (Every participant received a copy of Alexander's Timeless Way of Building beforehand.) One of the issues we debated extensively was the apparent dichotomy between highly innovative, agile IT on the one hand, and robust industrial-strength IT on the other hand. This dichotomy is often referred to as bimodal IT.

In those days, much of the debate was focused on technologies that supposedly supported one or other mode. For example SOA and SOAP (associated with the industrial-strength end) versus Web 2.0 and REST (associated with the agile end).

But the interesting question was how to bring the two modes back together. Here's one of the diagrams I drew at the workshop.



As the diagram shows, the dichotomy involves a number of different dimensions which sometimes (but not always) coincide.

• Scale
• Innovation versus Core Process
• Different rates of change (shearing layers or pace layering)
• Top-down ontology versus bottom up ontology ("folksonomy")
• Systems of engagement versus systems of record
• Demand-side (customer-facing) versus supply side
• Different levels of trust and security

Even in 2006, the idea that only industrial-strength IT can handle high volumes at high performance was already being seriously challenged. There were some guys from MySpace at the workshop, handling volumes which were pretty impressive at that time. As @Carnage4Life put it, My website is bigger than your enterprise.


Bimodal IT is now back in fashion, thanks to heavy promotion from Gartner. But as many people are pointing out, the flaws in bimodalism have been known for a long time.

One possible solution to the dichotomy of bimodalism is an intermediate mode, resulting in trimodal IT. Simon Wardley has characterized the three modes using the metaphor of Pioneers, Settlers, and Town Planners. A similar metaphor (Commandos, Infantry and Police) surfaced in the work of Robert X Cringely sometime in the 1990s. Simon reckons it was 1993.

Asked "Isn't bimodal new?" ... god no. It's a bad rehash of ideas from a decade or more ago. Even "tri" modal dates back to 1993.

— swardley (@swardley) April 27, 2016

Trimodal doesn't necessarily mean three-speed. Some people might interpret the town planners as representing ‘slow,’ traditional IT. But as Jason Bloomberg argues, Simon's model should be interpreted in a different way, with town planners associated with commodity, utility services. In other words, the town planners create a robust and agile platform on which the pioneers and settlers can build even more quickly. This is consistent with my 2013 piece on hacking and platforms. Simon argues that all three (Pioneers, Settlers, and Town Planners) must be brilliant.

absolutely @mjbrender @richardveryard @TheEbizWizard : all three must be brilliant ... https://t.co/DHMtkmrgzs

— swardley (@swardley) May 4, 2016

Characterizing a mode as "slow" or "fast" may be misleading, because (despite Rob England's contrarian arguments) people usually assume that "fast" is good and "slow" is bad. However, it is worth recognizing that each mode has a different characteristic tempo, and differences in tempo raise some important structural and economic issues. See my post on Enterprise Tempo (Oct 2010).



Updated - corrected and expanded the description of Simon's model.  Apologies for Simon for any misunderstanding on my part in the original version of this post.

---

Jason Bloomberg, Bimodal IT: Gartner's Recipe For Disaster (Forbes, 26 Sept 2015)

Jason Bloomberg, Trimodal IT Doesn’t Fix Bimodal IT – Instead, Let’s Fix Slow (Cortex Newsletter, 19 Jan 2016)

Jason Bloomberg, Bimodal Backlash Brewing (Forbes, 26 June 2016)

Rob England, Slow IT (28 February 2013)

Bernard Golden, What Gartner’s Bimodal IT Model Means to Enterprise CIOs (CIO Magazine, 27 January 2015)

John Hagel, SOA Versus Web 2.0? (Edge Perspectives, 25 April 2006)

Dion Hinchcliffe, How IT leaders are grappling with tech change: Bi-modal and beyond (ZDNet, 14 January 2015)

Dion Hinchcliffe, IT leaders inundated with bimodal IT meme (ZDNet, 1 May 2016)

Dare Obasanjo, My website is bigger than your enterprise (March 2006)

Richard Veryard, Notes from the SPARK workshop (March 2006), Enterprise Tempo (October 2010), A Twin-Track Approach to Government IT (March 2011),

Richard Veryard, Why hacking and platforms are the future of NHS IT (The Register, 16 April 2013)

Richard Veryard and Philip Boxer, Metropolis and SOA Governance (Microsoft Architecture Journal, July 2005)

Simon Wardley, Bimodal IT - the new old hotness (13 November 2014)

Simon Wardley, On Pioneers, Settlers, Town Planners and Theft (13 March 2015)

Lawrence Wilkes and Richard Veryard, Extending SOA with Web 2.0 (CBDI Forum for IBM, 2007)


updated 27 June 2016

Enterprise Architecture Tempo

#entarch Like many complex capabilities, enterprise architecture operates at several different tempi. (See my post on Enterprise Tempo.) A workshop at the SEI last year identified three tempi, which it called "Operating Modes".

1. At the lowest level, enterprise architects operate in an urgent response mode, reacting to crises as they arise.
2. Next, enterprise architects may operate in a continuous improvement mode, making incremental changes and generally avoiding crises.
3. Finally, enterprise architects may operate in a transformative change mode, collaborating with business leaders to enable new business capabilities and new business models.

The SEI observes that, in practice, enterprise architects are nearly always operating in all three modes, but suggests that the most effective organizations will spend less effort in the urgent response and more in transformative change.

Most of the process frameworks for enterprise architecture concentrate on the longer-term operating modes - sometimes called tactical and strategic planning. Some frameworks distinguish between solution architecture (with a typical cycle time of 6-18 months) and enterprise architecture (with a typical cycle time of 3-7 years).

But surely it is the less effective organizations, where more effort is devoted to urgent response, that are in greater need of process guidance? As I said in my post on Enterprise Architecture as Viable System, the dilemma for EA is how to appear useful in a crisis without (a) throwing all the EA professional discipline out of the window or (b) demanding at least four months to carry out a proper study. So what good is a process framework that simply tells such organizations that they really ought to put the urgent stuff aside and concentrate on long-term transformation? Or a process framework that is incapable of Learning from Stories?

---

John Klein and Michael Gagliardi, A Workshop on Analysis and Evaluation of Enterprise Architectures (pdf) SEI, November 2010.

Elsewhere the term "operating modes" has been used for the EA archetypes identified by John Gøtze and his colleagues, but this is completely different. See my post on the Dimensions of EA maturity.

Fast data - speed over precision?

@bmichelson posts a piece on an HP website called Fast Data: Speed over precision for better decision-making (21 June 2011), summarizing a recent interview in MIT Sloan with Ali Riaz and Sid Probstein of software company Attivio https://sloanreview.mit.edu/article/why-companies-have-to-trade-perfect-data-for-fast-info/.


Riaz and Probstein are not claiming that better decision-making is just about speed: it also requires a continuous refinement loop, including rethinking one's premises. In other words, we need multiple feedback loops at different speeds, to handle different levels of complexity. As I pointed out in my piece on TIBCO's slogan Two Second Advantage, although simple decisions can be taken quickly, complex decisions need what I call "time for understanding".

One of the architectural challenges of organizational intelligence is to coordinate a complex array of sense-making and decision-making processes operating at different characteristic tempi, and to maintain a proper balance between the very fast (reactive) and the comparatively slow (reflective). Probstein identifies Amazon.com as a successful exemplar.

It is common for technologists to make a fetish of speed, but business effectiveness and organizational improvement need "time for understanding" as well. Riaz and Probstein appear to understand this, and I look forward to seeing how this understanding is supported by their software.

An Architectural History of Social Networking

@ruskin147 is in California to meet the networking pioneers, including Stewart Brand.

In 1985, Stewart Brand was one of the founders of an online community called the Well. As Rory reports, the Well provided "inspiring stories of the power of online communication, as its members used its forums to share their lives, their thoughts, and their passions - whether it be for obscure technology questions or discussions about the meaning of life".

In 1994, Brand wrote a brilliant and controversial book about architecture, How Buildings Change, which among other things contained a theory about evolutionary change in complex systems based on earlier work by the architect Frank Duffy and the ecologist Robert O'Neill. The theory was then known as Shearing Layers; Brand now prefers to call it Pace Layering. If there is a difference between the two, Shearing Layers is primarily a descriptive theory about how change happens in complex systems, while Pace Layering is primarily an architectural principle for the design of resilient systems of systems.

In the original Shearing Layers theory, the slowest-moving layer is known as Site. In the context of physical buildings, this is the geographical setting, the urban location, and the legally defined lot, whose boundaries and context outlast generations of ephemeral buildings [via Wikipedia].

An interesting question for Brand then is whether social networking continues to occupy the same "Site" as early initiatives such as the Well. In other words, the boundaries and context outlasting generations of ephemeral technology.

Brand told Rory that he saw some of the same principles in Facebook that had governed The Well: "I'm really impressed at a lot of the instincts that Zuckerberg has had. Taking non-anonymity as an absolutely fundamental value of his company and thereby beating off the competition. A Facebook identity is one of the most valuable things his company offers. The lack of anonymity is what gives it value." [Friendster, Facebook and the Well]

But that's not quite the same as asserting a historical path from the Well to the present day. The critical question here is not how aware Zuckerberg and his associates were with the history of social networking, but to what extent this history directly or indirectly influenced their actions and choices. For example, our collective understanding of "anonymity" is coloured by the past couple of decades of internet and pre-internet activity. Zadie Smith (a near contemporary of Zuckerberg at Harvard) talks about Zuckerberg's idea about what a person is, or should be, and worries that her own idea of personhood is nostalgic, irrational, inaccurate [New York Review, 25 Nov 2010]. And of course ironic.

Where exactly did Zuckerberg's idea of personhood come from? There may be a sense in which echoes of the Well may persist in Facebook through the way such concepts are understood and managed. And although we wouldn't necessarily take Brand's opinion of this at face value (or for that matter Zuckerberg's) there can be few people whose opinion would be so interesting and well-informed.

---

See also

Roger Hudson, The Evolving Web - A Pace Layering view of the development of the Web and the W3C (March 2008)

Howard Silverman, Panarchy and Pace in the Big Back Loop (originally published on People and Place, March 2009, republished on Solving for Pattern, Oct 2012)

Matt Webb et al, Twitter thread discussing the provenance of the Shearing Layers concept (April 2021)

I have written several other posts on this blog discussing various aspects and applications of pacelayering.

Links updated 11 January 2018, 23 April 2021. Inserted reference to O'Neill.

Enterprise Structure

#entarch There are some critical structural issues for business organizations, which have critical implications for IT architecture, including generating new requirements for management information systems and collaboration platforms. However, these structural issues arise not from technology alone but from the structural complexity of markets and the business environment, and from the need to manage a complex organization as a viable and dynamic sociotechnical system.

It is perhaps curious that enterprise architects, even those who insist that EA is not just about IT, should be so quick to frame all structural issues in terms of IT architecture. The industry analyst firm Forrester got some stick recently for a list of "Top Ten Issues" that were all technologically oriented.

To understand the complex structural issues of business organizations, we need to look beyond the methods and models derived from computer science and database theory, and draw from such disciplines as systems thinking and management cybernetics. Here are some of the (overlapping) structural themes I have been looking at.

• Asymmetrical demand 
• Business as a platform
• Coupling (the enterprise as a loosely coupled network of sociotechnical components and services - as in my book on the Component-Based Business)
• Differentiation and integration
• Edge organization
• Enterprise tempo (different activities with different characteristic tempi)
• Ethical dilemmas (conflicts of interest, moral hazard)
• Multi-sided markets (different stakeholder classes with complementary demands)
• Organizational intelligence
• Viability (command and control, VSM)

 
 It's not easy to get one's head around some of these structural themes in the abstract, so I'm going to try and produce a series of concrete vignettes showing how these themes play out in specific industry sectors.

Enterprise Tempo

An enterprise operates at several different tempi. For example

• A retail chain has one tempo aligned to the customer visiting the store, a longer tempo for purchasing and logistics, and a longer one still for planning and establishing new stores

• A military organization operates a campaign tempo (perhaps measured in months) and an acquisition tempo (possibly measured in decades). See my post on the Economics of Agility.

• An operational control room can produce instant responses (akin to reflex action), considered responses (requiring human intelligence) and long-term improvement. See my post Organizational Intelligence in the Control Room.

• A consultancy firm has a job tempo (solving specific problems for its clients) and a capability tempo (developing the knowledge, skills and practices to maintain its competitive edge).

Differences in tempo also exist between an enterprise and key external stakeholders. For example, the customers of a retail grocery chain may wish to eat three meals a day, but usually visit the store on a more infrequent tempo. Meanwhile, some of the suppliers may base their production on an annual harvest.

Such differences in tempo raise some important structural and economic issues.

• Alignment - how are operations coordinated across different tempi, and what are the costs of alignment? (In their paper Agility and Value for Defence, Nicholas Whittall and Philip Boxer consider this question in relation to the military scenario outline above.)

• Resource bargaining - how do we divide scarce assets (time, money, management attention, capacity for risk-taking and innovation) between different tempi?

• Intelligence - how are knowledge flows and learning loops affected by the different tempi?

In ecology, it is usually thought that the slow-moving processes dominate the faster-moving processes. This is one of the axioms of the architectural theory of pace layering.

 

See also Beyond Bimodal (May 2016)

Economics of agility 2

In my previous post on the Economics of Agility, I noted how little material has been published on this topic.

As Nicholas Whittall and Philip Boxer point out in their contribution to the recent debate on The Meaning of Value-for-Money in Defence Acquisition (RUSI, February 2009), there is an important link between agility and alignment. See also their earlier piece on Agility and Innovation in Acquisition (RUSI, February 2008).

The first observation is that defence acquisition - just like systems acquisition most anywhere - operates on a much slower tempo than the requirements of the business. The "business" of a military organization is running military campaigns; thus when writing for the defence community, Whittall and Boxer refer to the Campaign Tempo and the Acquisition Tempo.

The second observation is that there is a complex set of activities (such as orchestration, customization, and improvisation) involved in bridging between Demand (the demands of the campaign or business) and Supply (the procurement of specific systems and devices). These activities operate on an intermediate tempo, which Whittall and Boxer call the Alignment Tempo.

"Meeting the campaign tempo then depends on the alignment tempo possible, which in turn depends on the acquisition tempo at which gaps can be filled. Any slowness in acquisition tempo leads to increased bricolage and process short cuts to enable the alignment tempo to keep up with the campaign tempo. Thus, ‘agility’ finds its richest expression in the ability of the alignment tempo to meet the required campaign tempo at the lowest cost – i.e. to maximise the value-for-defence."

The challenge is then to produce just enough variety within the acquisition to optimize the economics of alignment. Boxer has developed a technique of Cohesion-Based Costing (not yet published), which "offers a means to attach a value to the cost of introducing flexibility". This kind of technique will clearly be of enormous benefit within the SOA world.

 

Related post Enterprise Tempo (October 2010)