Data Governance Overview

In many organizations there is a growing awareness of the need for data governance. This is often driven by a perception that something is lacking - perhaps related to data quality or accountability. So this leads to a solution based on data ownership, and the monitoring and control of data quality. But is that all there is to data governance?

Data management operates at many levels, and there may be governance issues at each of these levels.

 

 

In some organizations, we may be constrained to work bottom-up, starting from level 1 and 2, because this is where the perceived issues are focused, and where we can engage people to establish the correct activities and control mechanisms. Some level 3 issues may be perceived, but it may initially be difficult to find people willing to commit any effort to resolving them. Level 4 and 5 issues are "above the ceiling", at least within these forums. 

There may be some clashes between the five levels, especially the higher ones. For example, at Level 5 the organization may assert a vision of a data-driven strategy, delivering strategic advantage to the business, while at Level 4 the data-related policies of an organization might be predominantly defensive, for example concerned with privacy, security and other compliance issues. If these two levels are not properly aligned, this is a matter for data governance, but one which bottom-up data governance is unlikely to reach. 

So what would top-down data governance look like? 

Purpose (final cause) - for example, to drive the reach, richness, agility and assurance of enterprise data - see my posts on DataStrategy

Approach (efficient cause) - to establish policies and processes that align the enterprise to these purposes 

Structure (formal cause) - an understanding of the contradictions and conflicts that might enable or inhibit change - what kinds of collaboration and coordination might be possible - and how might these structures themselves be altered 

Problem space (material cause) - finding issues that people are willing to invest time and energy to address 

Bottom-up data governance is driven by the everyday demands of case workers and decision-makers within the organization, the data fabric not being fit for purpose for fairly mundane tasks. Top-down data governance would be driven by senior management trying to push a transformation of data management and culture - always provided that they can give this topic much attention alongside all the other transformations they are trying to push. (I have some experiences from different organizations, which I plan to mash together into a generic example. Watch this space.)

But what if that's not where the true demand lies? 

Philip Boxer and I have been looking at alternatives to these top-down/bottom-up hierarchical forms, which we call edge-driven governance. If the traditional top-down / bottom-up can be thought of as North-South, then this is East-West. We recently got together at the Requisite Agility conference to discuss how far our thinking had developed (separately but largely in parallel) since we wrote our original papers for the Microsoft Architecture Journal. During this time, Phil spent a few years at the Software Engineering Institute (SEI) while I was mostly working at the architectural coal-face. There's a lot of really interesting work that has emerged recently, including further work by David Alberts, and a book on Bifurcation edited by Bernard Stiegler.

So how can we make practical interventions into these data governance issues from an East-West perspective? There are some critical concepts that are in play in this world, including agility, alignment, centricity, flexibility, simplicity, variety. Everyone pays lip service to these concepts in their own way, but they turn out to be highly unstable, capable of being interpreted in quite different ways from different stakeholder positions. 

So to pin these concepts down requires what John Law and Annemarie Mol call Ontological Politics. This include asking the Who-For-Whom question - agility for whom, flexibility for whom, etc. Philip has developed an abstract framework he calls Ontic Scaffolding, to support practical efforts to move “Across and Up”.

Note - videos from the Requisite Agility conference are currently in post-production and should be available soon. I'll post a link here when it is.

---

Philip Boxer, East-West Dominance (Asymmetric Leadership, April 2006)

Philip Boxer, Pathways across the 3rd epoch domain (Slideshare, November 2019)

Philip Boxer and Richard Veryard, Taking Governance to the Edge (Microsoft Architecture Journal 6, August 2006) 

Annemarie Mol, Ontological Politics (Sociological Review 1999)

Bernard Stiegler and the Internation Collective (eds), Bifurcate: There Is No Alternative (trans Daniel Ross, Open Humanities Press, 2021) 

Richard Veryard and Philip Boxer, Metropolis and SOA Governance: Towards the Agile Metropolis (Microsoft Architecture Journal 5, July 2005) 

 

Related topics on Philip's blog: Agility 

Related topics on Richard's blog: EdgeStrategy, Top-Down, RequisiteVariety

NEW eBook How to do things with data (Leanpub 2022)

Enterprise as Noun

@tetradian spots a small, subtle yet very welcome sentence in the latest version of TOGAF. 

"An enterprise may include partners, suppliers, and customers as well as internal business units."

Although similar statements can be found in earlier versions of TOGAF, and the Open Group (which produces TOGAF) has been talking about the "extended enterprise" and "boundarylessness" for over twenty years, these ideas have often seemed to take a back seat. So let's hope that the launch of TOGAF 10 will trigger a new emphasis on the "extended enterprise". 

But why is this so difficult? It seems people are much more comfortable thinking about "the enterprise" or "the organization" as a fixed thing with a clearly defined perimeter. An enterprise is assumed to be a legal entity, with a dedicated workforce, one or more bank accounts and perhaps even a stock exchange listing. 

Among other things, this encourages a perimeter-based approach to risk and security, where the primary focus is to control events crossing the perimeter in order to protect and manage what is inside. Companies often seek to export risk and uncertainty onto their customers or suppliers.

Alternatives to the fortress model of security are known as de-perimeterization. The aptly named Jericho Forum was set up by the Open Group in 2004 to promote this idea, now merged into the Open Group's Security Forum.

Instead of enterprise as noun, what about enterprise as verb, thinking of enterprise simply as a way of being enterprising? This would allow us to consider transient collaborations as well as permanent legal entities. Businesses are sometimes described as "going concerns", and this phrase shifts our attention from the identity of the business to questions of activity, viability and intentionality.

• Activity - What is going on?
• Viability - Is it going? Is it going to go on going?
• Intentionality - Where is it supposed to be going? Whose concern is it?
  

With this way of talking about enterprise, does it still make sense to talk about inside and outside, what belongs to the organization and what doesn't? I think it probably does, as long as we are careful not to take these terms too literally. Enterprises are assemblages of knowledge, power, responsibility, etc, and these elements are never going to be distributed uniformly. So even if we don't have a strict line dividing the inside from the outside, we can may be able to detect a gradient from inwards to outwards, and there may be opportunities to alter the gradient and make some strategic improvements, which we might frame in terms of agility or requisite variety. Hence Power to the Edge.

 

---

Related posts: Dividing Risk (April 2005), Jericho (May 2005), Power to the Edge (December 2005), Boundary, Perimeter, Edge (March 2007), Outside-In Architecture (August 2007), Ecosystem SOA (June 2010)

Developing Data Strategy

The concepts of net-centricity, information superiority and power to the edge emerged out of the US defence community about twenty years ago, thanks to some thought leadership from the Command and Control Research Program (CCRP). One of the routes of these ideas into the civilian world was through a company called Groove Networks, which was acquired by Microsoft in 2005 along with its founder, Ray Ozzie. The Software Engineering Institute (SEI) provided another route. And from the mid 2000s onwards, a few people were researching and writing on edge strategies, including Philip Boxer, John Hagel and myself.

Information superiority is based on the idea that the ability to collect, process, and disseminate an uninterrupted flow of information will give you operational and strategic advantage. The advantage comes not only from the quantity and quality of information at your disposal, but also from processing this information faster than your competitors and/or fast enough for your customers. TIBCO used to call this the Two-Second Advantage.

And by processing, I'm not just talking about moving terabytes around or running up large bills from your cloud provider. I'm talking about enterprise-wide human-in-the-loop organizational intelligence: sense-making (situation awareness, model-building), decision-making (evidence-based policy), rapid feedback (adaptive response and anticipation), organizational learning (knowledge and culture). For example, the OODA loop. That's my vision of a truly data-driven organization.

There are four dimensions of information superiority which need to be addressed in a data strategy: reach, richness, agility and assurance. I have discussed each of these dimensions in a separate post:

• Data Strategy - Reach
• Data Strategy - Richness
• Data Strategy - Agility
• Data Strategy - Assurance

---

Philip Boxer, Asymmetric Leadership: Power to the Edge

Leandro DalleMule and Thomas H. Davenport, What’s Your Data Strategy? (HBR, May–June 2017) 

John Hagel III and John Seely Brown, The Agile Dance of Architectures – Reframing IT Enabled Business Opportunities (Working Paper 2003)

Vivek Ranadivé and Kevin Maney, The Two-Second Advantage: How We Succeed by Anticipating the Future--Just Enough (Crown Books 2011). Ranadivé was the founder and former CEO of TIBCO.

Richard Veryard, Building Organizational Intelligence (LeanPub 2012)

Richard Veryard, Information Superiority and Customer Centricity (Cutter Business Technology Journal, 9 March 2017) (registration required)

Wikipedia: CCRP, OODA Loop, Power to the Edge

Related posts: Microsoft and Groove (March 2005), Power to the Edge (December 2005), Two-Second Advantage (May 2010), Enterprise OODA (April 2012), Reach Richness Agility and Assurance (August 2017)

Data Strategy - Reach

This is one of a series of posts looking at the four key dimensions of Data and Information that must be addressed in a data strategy - reach, richness, agility and assurance.

---

Data strategy nowadays is dominated by the concept of big data, whatever that means. Every year our notions of bigness are being stretched further. So instead of trying to define big, let me talk about reach.

Firstly, this means reaching into more sources of data. Instead of just collecting data about the immediate transactions, enterprises now expect to have visibility up and down the supply chain, as well as visibility into the world of the customers and end-consumers. Data and information can be obtained from other organizations in your ecosystem, as well as picked up from external sources such as social media. And the technologies for monitoring (telemetrics, internet of things) and surveillance (face recognition, tracking, etc) are getting cheaper, and may be accurate enough for some purposes.

Obviously there are some ethical as well as commercial issues here. I'll come back to these.

Reach also means reaching more destinations. In a data-driven business, data and information need to get to where they can be useful, both inside the organization and across the ecosystem, to drive capabilities and processes, to support sense-making (also known as situation awareness), policy and decision-making, and intelligent action, as well as organizational learning. These are the elements of what I call organizational intelligence. Self-service (citizen) data and intelligence tools, available to casual as well as dedicated users, improve reach; and the tool vendors have their own reasons for encouraging this trend.

In many organizations, there is a cultural divide between the specialists in Head Office and the people at the edge of the organization. If an organization is serious about being customer-centric, it needs to make sure that relevant and up-to-date information and insight reaches those dealing with awkward customers and other immediate business challenges. This is the power-to-the-edge strategy.

Information and insight may also have value outside your organization - for example to your customers and suppliers, or other parties. Organizations may charge for access to this kind of information and insight (direct monetization), may bundle it with other products and services (indirect monetization), or may distribute it freely for the sake of wider ecosystem benefits.

And obviously there will be some data and intelligence that must not be shared, for security or other reasons. Many organizations will adopt a defensive data strategy, protecting all information unless there is a strong reason for sharing; others may adopt a more offensive data strategy, seeking competitive advantage from sharing and monetization except for those items that have been specifically classified as private or confidential.

How are your suppliers and partners thinking about these issues? To what extent are they motivated or obliged to share data with you, or to protect the data that you share with them? I've seen examples where organizations lack visibility of their own assets, because they have outsourced the maintenance of these assets to an external company, and the external company fails to provide sufficiently detailed or accurate information. (When implementing your data strategy, make sure your contractual agreements cover your information sharing requirements.)

Data protection introduces further requirements. Under GDPR, data controllers are supposed to inform data subjects how far their personal data will reach, although many of the privacy notices I've seen have been so vague and generic that they don't significantly constrain the data controller's ability to share personal data. Meanwhile, GDPR Article 28 specifies some of the aspects of data sharing that should be covered in contractual agreements between data controllers and data processors. But compliance with GDPR or other regulations doesn't fully address ethical concerns about the collection, sharing and use of personal data. So an ethical data strategy should be based on what the organization thinks is fair to data subjects, not merely what it can get away with.

There are various specific issues that may motivate an organization to improve the reach of data as part of its data strategy. For example:

• Critical data belongs to third parties
• Critical business decisions lacking robust data
• I know the data is in there, but I can't get it out.
• Lack of transparency – I can see the result, but I don’t know how it has been calculated.
• Analytic insight narrowly controlled by a small group of experts – not easily available to general management
• Data and/or insight would be worth a lot to our customers, if only we had a way of getting it to them.

In summary, your data strategy needs to explain how you are going to get data and intelligence

• From a wide range of sources
• Into a full range of business processes at all touchpoints
• Delivered to the edge – where your organization engages with your customers

---

Next post Richness

Related posts

Power to the Edge (December 2005)
Reach, Richness, Agility and Assurance (August 2017)
Setting off towards the data-driven business (August 2019)
Beyond Trimodal - Citizens and Tourists (November 2019)

Capability and Business Strategy

In their book The Only Sustainable Edge, John Hagel and John Seely Brown describe two main schools of business strategy: the Core Competence school and the Leverage school. The approach advocated by Hagel and Seely-Brown represents a synthesis of these two schools. A closely related approach is advocated by David Alberts ("Power to the Edge").

	Core Competence	Leverage	Sustainable Edge
Focus	Resource-Based	Network-Based	Edge-Based
Source of Strategic Advantage	Identifying and strengthening core competences within the firm	Mobilizing resources outside the firm	Taking power to the edge of the organization.
Key advocates	Gary Hamel and C.K. Prahalad	Adam Brandenburger and Barry Nalebuff, James Moore	John Hagel and John Seely Brown, David Alberts.

Whereas the Core Competence school operates with a fixed notion of the required capability of the firm, the Sustainable Edge school involves dynamic specialization, connectivity and leveraged capability building. In other words, a successful business organization requires meta-capability – the capability to build capability, and the capability to build capability-building partnerships.

In our view, this kind of meta-capability is essential for the adaptive enterprise – after all, the defining capability of the adaptive enterprise is the capacity to adapt. However, in order to implement the adaptive enterprise, it is necessary to be specific about the nature of adaptation and adaptive capability required in a given context.

The capability model allows us to reason about which capabilities (or meta-capabilities) should be regarded as core competences.

 

Extract from Business Modeling for SOA - Part 1 What the Business Does (January 2006). See also Business-Driven SOA (May 2004), Business-Driven SOA 2 (June 2004), Business Systems Planning for SOA (May 2006). All available from CBDI Journal Archive.

Power to the Edge

Power to the edge is about changing the way individuals, organizations, and systems relate to one another and work.

• empowerment of individuals at the edge of an organization
• adoption of an edge organization, with greatly enhanced peer-to-peer interactions.
• moving senior personnel into roles that place them at the edge

Power to the edge is being presented in the military domain as the correct response to increased uncertainty, volatility, and complexity. Clearly these factors also apply to civilian enterprises, both commercial and public sector.

Military use of the term comes from a book by David S. Alberts and Richard E. Hayes, of the US Department of Defense Command and Control Research Program (DoD CCRP). See also presentation material by Dr Margaret Myers.

Groove (acquired by Microsoft in March 2005) always liked this concept – see blogs by Ray Ozzie and Michael Helfrich. See also blogs by Doug Simpson and Nathan Wallace.

Philip Boxer and I wrote a couple of articles for the Microsoft Architecture Journal on the implications for Service Oriented Architecture. Philip and Carole Eigen also applied the concept to the psychoanalytic study of organizations.

I found a weblog rant here to the effect that Power to the Edge is all about speeding up information flow, just another name for Reengineering. In my view, this is a fundamental misunderstanding. Obviously Power to the Edge may call for improved flow of information: quality and complexity as well as quantity and speed. But Power to the Edge is not the improved flow itself but what it enables – which is a fundamental transformation in the geometry of the organization away from a hierarchical command-and-control structure. And such structures are still as common in civilian/commercial organizations as in the military, if not more so.

---

David S. Alberts and Richard E. Hayes, Power to the Edge: Command … Control … in the Information Age (CCRP June 2003) PDF version available online (1.7 MB).

Philip Boxer, Taking power to the edge by empowering the edge role (Asymmetric Leadership, 24 January 2006)

Philip Boxer and Carole Eigen, Taking power to the edge of the organisation: re-forming role as praxis (ISPSO Symposium, Baltimore, June 2005) (abstract) (presentation) (paper)

Philip Boxer and Richard Veryard, Taking Governance to the Edge (Microsoft Architecture Journal 6, August 2006)

Margaret Myers, Power to the Edge Through Net Centricity – Transformation of the Global Information Grid (CHIPS July-September 2002) Slides (pdf).

John Stenbit, Moving Power to the Edge (CHIPS July-September 2003)

Richard Veryard and Philip Boxer, Metropolis and SOA Governance: Towards the Agile Metropolis (Microsoft Architecture Journal 5, July 2005)

Wikipedia: Power to the Edge

Cross-posted to AsymmetricLeadership blog

See also Demise of the Super Star (August 2004), Governance at the Edge (August 2004) Microsoft and Groove (March 2005), Developing Data Strategy (December 2019)

Updated 7 December 2019

Governance at the Edge

Lots of edge thinking in the web services world at the moment.

Peter Cousins (Iona) has been talking about Integration at the Edge. Good summary by Steve Vinoski (also Iona).

The core argument here is that change and variety and business value occur at the edge of the organization, so that's where technology needs to be focused – including integration technology.

Some supporting evidence comes from the Yankee Group, which finds a surprisingly large number of web service projects addressing supply chain integration. Alice LaPlante (Web Services Pipeline) suggests this means that web services are most valuable at the seams where organizations are trying to meet and share data and functionality. Web Services Thrive On Enterprise 'Edge'

Radovan Janecek adds: OK, so if we agree we should do the integration on the edge, then I add: do the management on the edge too. Otherwise, you will fall in the ESB trap again. (RJ expands this here.)

Meanwhile, leading management thinkers are arguing for organizations to take power to the edge - and this is also motivated by matters of change and variety and business value. Thus the technology agenda coincides with and supports the business agenda.

---

Update: Following this post, Philip Boxer and I wrote two articles for the Microsoft Architecture Journal.

Metropolis and SOA Governance: Towards the Agile Metropolis (Architecture Journal 5, July 2005)

Taking Governance to the Edge (Architecture Journal 6, August 2006)