Data Strategy - More on Agility

Continuing my exploration of the four dimensions of Data Strategy. In this post, I bring together some earlier themes, including Pace Layering and Trimodal.

The first point to emphasize is that there are many elements to your overall data strategy, and these don't all work at the same tempo. Data-driven design methodologies such as Information Engineering (especially the James Martin version) were based on the premise that the data model was more permanent than the process model, but it turns out that this is only true for certain categories of data.

So one of the critical requirements for your data strategy is to manage both the slow-moving stable elements and the fast-moving agile elements. This calls for a layered approach, where each layer has a different rate of change, known as pace-layering.

The concept of pace-layering was introduced by Stewart Brand. In 1994, he wrote a brilliant and controversial book about architecture, How Buildings Change, which among other things contained a theory about evolutionary change in complex systems based on earlier work by the architect Frank Duffy. Although Brand originally referred to the theory as Shearing Layers, by the time of his 1999 book he had switched to calling it Pace Layering. If there is a difference between the two, Shearing Layers is primarily a descriptive theory about how change happens in complex systems, while Pace Layering is primarily an architectural principle for the design of resilient systems-of-systems.

In 2006, I was working as a software industry analyst, specializing in Service-Oriented Architecture (SOA). Microsoft invited me to Las Vegas to participate in a workshop with other industry analysts, where (among other things) I drew the following layered picture.

SPARK Workshop Day 2

Here's how I now draw the same picture for data strategy. It also includes a rough mapping to the Trimodal approach.

---

Giles Slinger and Rupert Morrison, Will Organization Design Be Affected By Big Data? (J Org Design Vol 3 No 3, 2014)

Wikipedia: Information Engineering, Shearing Layers 

Related Posts: Layering Principles (March 2005), SPARK 2 - Innovation or Trust (March 2006), Enterprise Tempo (October 2010), Beyond Bimodal (May 2016), Data Strategy - Agility (December 2019)

Beyond Trimodal - Citizens and Tourists

I've been hearing a lot about citizens recently. The iPaaS vendors are all talking about citizen integration, and at the Big Data London event this week I heard several people talking about citizen data science.

Gartner's view is that development can be divided into two modes - Mode 1 (conventional) and Mode 2 (self-service), with the citizen directed towards Mode 2. For a discussion of how this applies to Business Intelligence, see my post From Networked BI to Collaborative BI (April 2016).

There is a widespread view that Gartner's bimodal approach is outdated, and that at least three modes are required - a trimodal approach. Simon Wardley's version of the trimodal approach characterizes the roles as pioneers, settlers and town planners. My initial idea on citizen integration was that the citizen-expert spectrum could be roughly fitted into this approach as follows. If the town planners had set things up properly, this would enable easy integration by pioneers and settlers. See my recent posts on DataOps - Organizing the Data Value Chain and Strategy and Requirements for the API ecosystem.

But even if this works for citizen integration, it doesn't seem to work for data science. In her keynote talk this week, Cassie Kozyrkov of Google discussed how TensorFlow had developed from version 1.x (difficult to use, and only suitable for data science pioneers) to version 2.x (much easier to use and suitable for the citizen). And in his talk on the other side of the hall, Chris Williams of IBM Watson also talked about advance in tools that made data science much easier.


That doesn't mean that everyone can do data science, at least not yet, nor that the existing data science skills will become obsolete. Citizen data scientists are those who take data science seriously, who are able and willing to acquire some relevant knowledge and skill, but are performing data science in support of their main job role rather than as an occupation in its own right. 

We may therefore draw a distinction between two types of user - the citizen and the tourist. The tourist may have a casual interest but no serious commitment or responsibility. An analytics or AI platform may well provide some self-service support for tourists as well as citizens, but these will need to be highly constrained in their scope and power.

Now if we add the citizen and the tourist to the pioneer, settler and town planner, we get a pentamodal approach. The tourists may visit the pioneers and the towns, but probably isn't very interested in the settlements. Whereas the citizens mainly occupy the settlements - in other words, the places built by the settlers.

I wonder what Simon will make of this idea?

---

Andy Callow, Exploring Pioneers, Settlers and Town Planners (3 January 2017)

Jen Underwood, Responsible Citizen Data Science. Yes, it is Possible (9 July 2019)


For further discussion and references on the trimodal approach, see my post Beyond Bimodal (May 2016)

Strategy and Requirements for the API ecosystem

Is there a framework or methodology for establishing the business / ecosystem requirements to drive API strategy and development?

At an industry event I attended recently, hosted by a company that sells tools and technologies for the API ecosystem, some of the speakers advised that when presenting to non-technical stakeholders, you need to talk about service value/benefit rather than APIs. But this raises an important question, how to identify and quantify service benefit, and how to negotiate share of value between different players in the ecosystem?

One of the ideas of the API economy is that you don't have to maintain all the capabilities yourself, but you find other enterprises that can provide complementary capabilities. So you need to identify and understand what capabilities are available, and map combinations of these capabilities against the demands and unfulfilled needs of potential customers. Then having identified in broad terms what capabilities you wish to combine with your own, and worked out where the service boundaries should be, you may select organizations to partner with and agree business and commercial terms, or create a platform to which many third parties can connect. The technical design of the API should then reflect the service boundaries and commercial arrangements.

In the early days of service-oriented software engineering, people always wanted us to tell them how large their services should be. Not just macro versus micro, but broad (generic) versus narrow (specific). To what extent should a service be completely purpose-agnostic - in other words, with no restrictions on how or where it may be used - or does this conflict with other design goals such as reliability or data protection?

The answer is that it depends not only on what you are trying to do, but how you want to manage and govern your service architecture. A broadly scoped, purpose-agnostic service (or service platform) can achieve wide usage and economies of scale, but may be more complex to configure, test and use, whereas a more narrowly scoped context-specific service might be easier to use but with lower reuse potential. Among other things, this affects how much of the service composition and orchestration can be done by the service provider (supply side), and how much is left to the service consumer (demand-side). And even on the supply side, it affects how much work needs to be done by the integration experts ("town planners"), and how much can be left to citizen integration ("pioneers" and "settlers").

One version of this challenge can be found in large global organizations, working out exactly what functionality should be provided centrally as shared services, and what functionality should be left to local operations. Ideally, the service architecture should be aligned with the business and organizational architecture.

The word "economy" also implies attention to accounting issues - sharing costs and benefits between different players. Although we may regard cloud as almost infinitely extensible, this doesn't come without cost: if the number of service calls goes through the roof, someone has to pay the cloud provider's bill. This is already an issue within large organizations, where we commonly find arguments about whose budget will pay for something. And I have seen some great ideas come to nothing, because the benefits were spread too thinly and nobody was able to fund them.

So although vague appeals to innovation and imagination might be good enough for a marketing pitch, serious strategic thinking is about discovering where there is untapped value in your business and its environment, and working out exactly how an API strategy is going to help you unlock this value.

---

At the CBDI Forum, we were talking about these issues many years ago: our Service Architecture and Engineering® methodology is still available from Everware-CBDI. Here are some of the articles I wrote for the CBDI Journal.

• Identifying Web Services (February 2002) 
• Service Economics (September 2005)

More at  http://everware-cbdi.com/cbdi-journal-archive

DataOps - Organizing the Data Value Chain

At #TalendConnect today frequent mention of #DataOps, although according to a post I found on the Talend blog from earlier this year, Talend prefers the term collaborative data management.

Data Preparation ... should be envisioned as a game-changing technology for information management due to its ability to enable potentially anyone to participate. Armed with innovative technologies, enterprises can organize their data value chain in a new collaborative way. Talend

I've always insisted that the data value chain should end not with delivering insight (so-called actionable intelligence) but with delivering business outcomes (actioned intelligence), and I was pleased to hear some of today's speakers making the same point. However, there are still voices within the industry that have a narrower view of DataOps, and I note with concern that the DataOps Manifesto identifies the goal of DataOps in terms of the early and continuous delivery of valuable analytic insights.

Although there will always be a place for analytic reports and dashboards, I always expected that these would gradually make way for analytic insights being rendered as services and integrated into operational business systems and processes, to create closed-loop business intelligence. There are many good examples of this today, especially in the manufacturing world. There are also systems that deliver insights directly to customers or end-users, perhaps in the form of recommendations. But a lot of the discussion of the data-driven enterprise still seems to be based on a dashboard mindset.

And who actually does the DataOps? A presentation from Virtusa showed a three-step DataOps process - pipeline, innovation and value - which suggests a trimodal approach. So the Town Planners would do the pipeline (building generic and highly customizable data preparation frameworks), Pioneers would do the innovation (experimental proof of concept), and the Settlers would roll out the value. I shall be interested to see some practical implementations of this approach.

Meanwhile, simplistic notions of democratization (or citizen integration) often divides people into two camps - experts and citizens - and this polarization is encouraged by Gartner's promotion of Bimodal IT. But this leads people to believe that you can have either trust or speed/agility but not both. And as Jonathan Gill of Talend emphasized in his keynote today, digital leaders don't recognize this dichotomy.

---

Jean-Michel Franco, 3 Key Takeaways from the 2019 Gartner Market Guide for Data Preparation (Talend, 26 April 2019)

Wikipedia: DataOps

Related posts: Service-Oriented Business Intelligence (September 2005), SPARK 2 Innovation or Trust (March 2006), Analytics for Adults (January 2013), From Networked BI to Collaborative BI (April 2016), Beyond Bimodal (May 2016), Towards the Data-Driven Business (August 2019), Beyond Trimodal - Citizens and Tourists (November 2019)

Bus Safety Announcement

Transport for London (TfL) reckons around 3000 people are injured every year by slips, trips and falls on London buses. So it is running trials of an automated system that announces the departure of the bus from the stop.

"Please hold on, the bus is about to move"

or as Bon Jovi might say

"We've gotta hold on ready or not."

The problem is that these alerts often come after the bus is already halfway down the road.

"Whoa, we're half-way there."

As the BBC News explains, the timing of the alert is based on the average amount of time a bus would spend at a bus stop, and is often hopelessly inaccurate. Passengers have taken to social media in droves to complain or mock. Many have wondered whether it was such a problem in the first place, and whether an alert would help to alleviate the problem. Others have pointed out the potential value of such an alert for certain categories of passenger - such as the elderly or visually impaired - but of course this only works if the alert comes at the right time.

---

I haven't spoken to anyone at TfL about this, but I can imagine what happened. In order to get a trial up and running quickly, they didn't have time (or permission) to link the alert with any of the systems on board the bus that could have sent a more accurate event signal. So we have a stand-alone system, knocked up quickly, as an experimental solution to a problem that most people hadn't previously recognized. In the trimodal scheme, this is a classic Pioneer project.

"For love we'll give it a shot."

So if the trial isn't laughed into touch, then maybe the Settlers can take over and do the alert properly.

"Take my hand, we'll make it. I swear."

And the Town Planners can come up with a joined-up long-term vision for passenger comfort and safety. Altogether now ...

"Whoa, livin' on a prayer."

---

Update

The wording of the announcement has changed, but the timing hasn't. It now says.

"Please hold on while the bus is moving"

What can I say to that?

"Standing on the ledge, I show the wind how to fly. When the world gets in my face, I say, have a nice day."

---

Londoners hit out at 'mistimed' bus safety alerts (BBC News, 14 January 2018)

Nadia Khomami, Please hold on: TfL urged to get a grip over annoying bus warnings (Guardian, 15 January 2018)

Eleanor Rose, TfL anger London commuters again with replacement bus announcement that is 'still annoying as hell' (Standard, 26 January 2018)

Londoners baffled by 'bonkers' bus safety announcements warning them 'the bus is about to move' (Evening Standard,15 January 2018).


For more on Trimodal IT, see my post Beyond Bimodal (May 2016)

Beyond Bimodal

Ten years ago (March 2006) I attended the SPARK workshop in Las Vegas, hosted by Microsoft and inspired by Christopher Alexander. (Every participant received a copy of Alexander's Timeless Way of Building beforehand.) One of the issues we debated extensively was the apparent dichotomy between highly innovative, agile IT on the one hand, and robust industrial-strength IT on the other hand. This dichotomy is often referred to as bimodal IT.

In those days, much of the debate was focused on technologies that supposedly supported one or other mode. For example SOA and SOAP (associated with the industrial-strength end) versus Web 2.0 and REST (associated with the agile end).

But the interesting question was how to bring the two modes back together. Here's one of the diagrams I drew at the workshop.



As the diagram shows, the dichotomy involves a number of different dimensions which sometimes (but not always) coincide.

• Scale
• Innovation versus Core Process
• Different rates of change (shearing layers or pace layering)
• Top-down ontology versus bottom up ontology ("folksonomy")
• Systems of engagement versus systems of record
• Demand-side (customer-facing) versus supply side
• Different levels of trust and security

Even in 2006, the idea that only industrial-strength IT can handle high volumes at high performance was already being seriously challenged. There were some guys from MySpace at the workshop, handling volumes which were pretty impressive at that time. As @Carnage4Life put it, My website is bigger than your enterprise.


Bimodal IT is now back in fashion, thanks to heavy promotion from Gartner. But as many people are pointing out, the flaws in bimodalism have been known for a long time.

One possible solution to the dichotomy of bimodalism is an intermediate mode, resulting in trimodal IT. Simon Wardley has characterized the three modes using the metaphor of Pioneers, Settlers, and Town Planners. A similar metaphor (Commandos, Infantry and Police) surfaced in the work of Robert X Cringely sometime in the 1990s. Simon reckons it was 1993.

Asked "Isn't bimodal new?" ... god no. It's a bad rehash of ideas from a decade or more ago. Even "tri" modal dates back to 1993.

— swardley (@swardley) April 27, 2016

Trimodal doesn't necessarily mean three-speed. Some people might interpret the town planners as representing ‘slow,’ traditional IT. But as Jason Bloomberg argues, Simon's model should be interpreted in a different way, with town planners associated with commodity, utility services. In other words, the town planners create a robust and agile platform on which the pioneers and settlers can build even more quickly. This is consistent with my 2013 piece on hacking and platforms. Simon argues that all three (Pioneers, Settlers, and Town Planners) must be brilliant.

absolutely @mjbrender @richardveryard @TheEbizWizard : all three must be brilliant ... https://t.co/DHMtkmrgzs

— swardley (@swardley) May 4, 2016

Characterizing a mode as "slow" or "fast" may be misleading, because (despite Rob England's contrarian arguments) people usually assume that "fast" is good and "slow" is bad. However, it is worth recognizing that each mode has a different characteristic tempo, and differences in tempo raise some important structural and economic issues. See my post on Enterprise Tempo (Oct 2010).



Updated - corrected and expanded the description of Simon's model.  Apologies for Simon for any misunderstanding on my part in the original version of this post.

---

Jason Bloomberg, Bimodal IT: Gartner's Recipe For Disaster (Forbes, 26 Sept 2015)

Jason Bloomberg, Trimodal IT Doesn’t Fix Bimodal IT – Instead, Let’s Fix Slow (Cortex Newsletter, 19 Jan 2016)

Jason Bloomberg, Bimodal Backlash Brewing (Forbes, 26 June 2016)

Rob England, Slow IT (28 February 2013)

Bernard Golden, What Gartner’s Bimodal IT Model Means to Enterprise CIOs (CIO Magazine, 27 January 2015)

John Hagel, SOA Versus Web 2.0? (Edge Perspectives, 25 April 2006)

Dion Hinchcliffe, How IT leaders are grappling with tech change: Bi-modal and beyond (ZDNet, 14 January 2015)

Dion Hinchcliffe, IT leaders inundated with bimodal IT meme (ZDNet, 1 May 2016)

Dare Obasanjo, My website is bigger than your enterprise (March 2006)

Richard Veryard, Notes from the SPARK workshop (March 2006), Enterprise Tempo (October 2010), A Twin-Track Approach to Government IT (March 2011),

Richard Veryard, Why hacking and platforms are the future of NHS IT (The Register, 16 April 2013)

Richard Veryard and Philip Boxer, Metropolis and SOA Governance (Microsoft Architecture Journal, July 2005)

Simon Wardley, Bimodal IT - the new old hotness (13 November 2014)

Simon Wardley, On Pioneers, Settlers, Town Planners and Theft (13 March 2015)

Lawrence Wilkes and Richard Veryard, Extending SOA with Web 2.0 (CBDI Forum for IBM, 2007)


updated 27 June 2016