Data Estimation - Stockpile Reports

My attention has been drawn to a company called Stockpile Reports, which provides a data estimation service for piles of material. As I understand it, the calculations are based on visual images of a pile of material, possibly obtained using either a drone or a handmobile phone app, from which a detailed 3D model of the pile is produced, allowing the volume, weight or value of the material to be estimated.

I don't have any information about how good these estimates are, but they must be a lot more accurate than simply gauging the height of the pile, and I guess they are probably good enough for the organizations that use the service. In this blogpost, I want to make some observations about this service and its potential value to user organizations, as well as some general thoughts about aggregation, variety and governance

The top benefit promised by Stockpile Reports is to eliminate painful write-offs. Because inventory is shown as an asset in a company's accounts, and companies don't like having to make large adjustments when these asset values turn out to be grossly inaccurate.

But this makes it seem as if the primary motivation for accurate inventory data is financial accounting. While I understand that many execs may be concerned about this, especially CFOs, financial accounting is a largely retrospective process. Whereas operational excellence and other forms of strategic advantage have a more immediate need for accurate inventory data.

And this is not just about the relative importance of financial reporting versus operational excellence etc. It is also about top-down versus bottom-up. Because a write-off essentially means that the total figure is incorrect. This plays into a top-down view of data quality and trustworthy data. Stockpile provide a service whose primary purpose appears to be to reassure investors that the financial accounts are correct.

This leads me to talk about aggregation. Top-down aggregation is about calculating a number, from a given body of data, for a given purpose. For example, the purpose might be financial accounting, management accounting, or billing. (Where billing essentially means aggregating everything that has happened during the past month in order to determine how much the customer owes us.) With some exceptions (such as Activity-Based Costing), accounting is largely premised on reducing everything to a standard set of numbers.

So this implies three capabilities. 

1. The capability of producing the correct number 
2. The capability of inspecting the source data and calculation in order to explain or verify the number. (For example, the reason for itemized billing is to justify the number at the bottom of the invoice.) 
3. The capability of preserving the source data and calculation, so that inspection and audit will be possible into the future. (Also mechanisms to allow external parties to trust the integrity of the numbers.) 

All of these capabilities are simplified if we can eliminate as much variation as possible. If Stockpile’s drones are solely assessing the size of different heaps of stuff in order to aggregate them, then the fact that the heaps are all different is simply an annoying difficulty that the technology is designed to solve. Thus the variation between the piles is smoothed over.

Some degree of standardization may also be needed to support and enable Statistical Process Control. 

But what if the differences between the heaps are interesting and strategically significant? Now we start to look at the data in a different way - no longer top-down aggregation but something else. The Stockpile Reports claim is that estimating the size of the piles more frequently produces more accurate data. The immediate benefit is more consistent data and fewer nasty surprises. Presumably there is also a feedback loop, so that when a pile of gravel is loaded onto trucks, we can see if the quantity on the trucks matches the quantity we thought we had in the pile. So this is what might generate greater accuracy (over time), assuming other conditions remains unaltered. 

Where is the variety in this business problem?

• Different shapes and sizes of pile
• Vegetation or other extraneous material on top of the pile
• Different kinds of material - e.g. gravel versus sand
• Lack of consistency in the material - e.g. the quality of the gravel may depend on the quality of the original rock, plus the operational characteristics of the extraction machinery. Even within a single pile, the gravel may not be completely homogeneous.
• Weather. Presumably wet gravel is heavier than dry gravel.
• Control. What difference does it make if the pile of gravel is at your customer’s or supplier’s site rather than your own? How transparent are inventory levels across organizational boundaries? Do companies really want to share this information, or might they have reasons for hiding or distorting the true state of their inventory? 

These are all factors that may need to be considered when estimating the pile. But there is a more fundamental question, which is about the meaning and use of the number that is produced by this estimation process, given the variety of industry operating contexts - so I wonder whether there is a universal understanding of what counts as a pile. When estimating the size of a pile of gravel, does it matter what the gravel is going to be used for? What about the end-to-end journey of the gravel: if a pile of gravel is moved to a new location, is it still the same pile?

Depending on the frequency of viewing the pile, it may be possible to track changes in the shape of the pile, and this might provide information about stock movements (including unauthorized ones). For example, it may be possible to infer something about the movement vehicle (truck, wheelbarrow) from the difference made to the shape of the pile. For some materials, there may also be natural processes that change the volume of the pile over time - for example organic material may dry out or decay.

But this discussion of variety brings us to an important question - variety for whom. If some degree of requisite agility is needed to provide requisite variety, where does this agility need to be situated? How much does Stockpile Reports know (need to know) about its customer’s context of use? 

There may be variation within a single customer, to provide economies of scale and scope (to those customers), and to allow these organizations to operate with requisite agility in relation to a dynamic demand. Stockpile Reports might also wish to manage variation across different customers. For example, there may be some value in aggregating or comparing data from different customers - for example, calibrating the estimates of similar materials, allowing a new customer to get up to speed more quickly. 

There may be some value to both Stockpile Reports and its customers from closer coordination and mutual agility. But this raises some important questions. Firstly, how this value is shared between them. Secondly what kind of mutual trust does this require. And therefore how this coordination is to be governed.

See also my post on Testing Multiples (July 2022). Philip Boxer's analysis of the Stockpile example can be found here Accelerating demand tempos and the double-V cycle and here The dialectics implied by the Q-sectors (April 2023).

Data Governance Overview

In many organizations there is a growing awareness of the need for data governance. This is often driven by a perception that something is lacking - perhaps related to data quality or accountability. So this leads to a solution based on data ownership, and the monitoring and control of data quality. But is that all there is to data governance?

Data management operates at many levels, and there may be governance issues at each of these levels.

 

 

In some organizations, we may be constrained to work bottom-up, starting from level 1 and 2, because this is where the perceived issues are focused, and where we can engage people to establish the correct activities and control mechanisms. Some level 3 issues may be perceived, but it may initially be difficult to find people willing to commit any effort to resolving them. Level 4 and 5 issues are "above the ceiling", at least within these forums. 

There may be some clashes between the five levels, especially the higher ones. For example, at Level 5 the organization may assert a vision of a data-driven strategy, delivering strategic advantage to the business, while at Level 4 the data-related policies of an organization might be predominantly defensive, for example concerned with privacy, security and other compliance issues. If these two levels are not properly aligned, this is a matter for data governance, but one which bottom-up data governance is unlikely to reach. 

So what would top-down data governance look like? 

Purpose (final cause) - for example, to drive the reach, richness, agility and assurance of enterprise data - see my posts on DataStrategy

Approach (efficient cause) - to establish policies and processes that align the enterprise to these purposes 

Structure (formal cause) - an understanding of the contradictions and conflicts that might enable or inhibit change - what kinds of collaboration and coordination might be possible - and how might these structures themselves be altered 

Problem space (material cause) - finding issues that people are willing to invest time and energy to address 

Bottom-up data governance is driven by the everyday demands of case workers and decision-makers within the organization, the data fabric not being fit for purpose for fairly mundane tasks. Top-down data governance would be driven by senior management trying to push a transformation of data management and culture - always provided that they can give this topic much attention alongside all the other transformations they are trying to push. (I have some experiences from different organizations, which I plan to mash together into a generic example. Watch this space.)

But what if that's not where the true demand lies? 

Philip Boxer and I have been looking at alternatives to these top-down/bottom-up hierarchical forms, which we call edge-driven governance. If the traditional top-down / bottom-up can be thought of as North-South, then this is East-West. We recently got together at the Requisite Agility conference to discuss how far our thinking had developed (separately but largely in parallel) since we wrote our original papers for the Microsoft Architecture Journal. During this time, Phil spent a few years at the Software Engineering Institute (SEI) while I was mostly working at the architectural coal-face. There's a lot of really interesting work that has emerged recently, including further work by David Alberts, and a book on Bifurcation edited by Bernard Stiegler.

So how can we make practical interventions into these data governance issues from an East-West perspective? There are some critical concepts that are in play in this world, including agility, alignment, centricity, flexibility, simplicity, variety. Everyone pays lip service to these concepts in their own way, but they turn out to be highly unstable, capable of being interpreted in quite different ways from different stakeholder positions. 

So to pin these concepts down requires what John Law and Annemarie Mol call Ontological Politics. This include asking the Who-For-Whom question - agility for whom, flexibility for whom, etc. Philip has developed an abstract framework he calls Ontic Scaffolding, to support practical efforts to move “Across and Up”.

Note - videos from the Requisite Agility conference are currently in post-production and should be available soon. I'll post a link here when it is.

---

Philip Boxer, East-West Dominance (Asymmetric Leadership, April 2006)

Philip Boxer, Pathways across the 3rd epoch domain (Slideshare, November 2019)

Philip Boxer and Richard Veryard, Taking Governance to the Edge (Microsoft Architecture Journal 6, August 2006) 

Annemarie Mol, Ontological Politics (Sociological Review 1999)

Bernard Stiegler and the Internation Collective (eds), Bifurcate: There Is No Alternative (trans Daniel Ross, Open Humanities Press, 2021) 

Richard Veryard and Philip Boxer, Metropolis and SOA Governance: Towards the Agile Metropolis (Microsoft Architecture Journal 5, July 2005) 

 

Related topics on Philip's blog: Agility 

Related topics on Richard's blog: EdgeStrategy, Top-Down, RequisiteVariety

NEW eBook How to do things with data (Leanpub 2022)

Uber and the Economics of Disruption

In a previous post on Uber Mathematics, I asked how the magic of digital would allow a centralized company with international overheads (Uber or Lyft) to provide a service more cheaply and cost-effectively than local cab companies.

Uber once promised it would achieve economies of scale, thanks to network effects. But then, as Ali Griswald comments,

What is scale if not a company that operates in 72 countries and more than 10,500 cities, which last year had 118 million active users every month and completed 6.3 billion rides/trips/deliveries? Uber is the definition of scale, yet it is still nowhere near consistent and reliable profitability.

Uber appears to retain an advantage over local cab companies in terms of consumer convenience, which we can frame in terms of the economics of alignment. But as Henry Graber points out, it remains to be seen how much consumers are willing to pay for this convenience when they are no longer being heavily subsidized by over-optimistic investors, and Uber is no longer the cheapest option.

Another point Graber makes about Uber is that it has disrupted alternative modes of transport, by a combination of predatory pricing and political influence, and the regulatory environment in many cities has been altered in Uber's favour.

There are processes here that operate at different tempos. There is a demand tempo - getting sufficient numbers of drivers to a specific location at a specific time to satisfy peaks of demand. And an acquisition tempo - how long does it take to increase the number of available drivers, given the necessary investment in vehicles and equipment, driver training, and a whole range of safety issues.

Because there's a big difference between these two tempos, the challenge for a healthy ecosystem is in the intermediate tempo, which we call the integration tempo. What are the options for cities to restore requisite variety to local transportation?

---

Aaron Gordon, Uber And Lyft Don't Have A Right To Exist (Jalopnik, 30 August 2019), Uber and Lyft Are Out of Ideas, Jacking Up Prices in Desperation for Profit (Vice, 27 May 2022)

Henry Grabar, The Decade of Cheap Rides Is Over (Slate, 18 May 2022)

Ali Griswold, Uber wants to show you the money (Oversharing, 10 May 2022)

Enterprise as Noun

@tetradian spots a small, subtle yet very welcome sentence in the latest version of TOGAF. 

"An enterprise may include partners, suppliers, and customers as well as internal business units."

Although similar statements can be found in earlier versions of TOGAF, and the Open Group (which produces TOGAF) has been talking about the "extended enterprise" and "boundarylessness" for over twenty years, these ideas have often seemed to take a back seat. So let's hope that the launch of TOGAF 10 will trigger a new emphasis on the "extended enterprise". 

But why is this so difficult? It seems people are much more comfortable thinking about "the enterprise" or "the organization" as a fixed thing with a clearly defined perimeter. An enterprise is assumed to be a legal entity, with a dedicated workforce, one or more bank accounts and perhaps even a stock exchange listing. 

Among other things, this encourages a perimeter-based approach to risk and security, where the primary focus is to control events crossing the perimeter in order to protect and manage what is inside. Companies often seek to export risk and uncertainty onto their customers or suppliers.

Alternatives to the fortress model of security are known as de-perimeterization. The aptly named Jericho Forum was set up by the Open Group in 2004 to promote this idea, now merged into the Open Group's Security Forum.

Instead of enterprise as noun, what about enterprise as verb, thinking of enterprise simply as a way of being enterprising? This would allow us to consider transient collaborations as well as permanent legal entities. Businesses are sometimes described as "going concerns", and this phrase shifts our attention from the identity of the business to questions of activity, viability and intentionality.

• Activity - What is going on?
• Viability - Is it going? Is it going to go on going?
• Intentionality - Where is it supposed to be going? Whose concern is it?
  

With this way of talking about enterprise, does it still make sense to talk about inside and outside, what belongs to the organization and what doesn't? I think it probably does, as long as we are careful not to take these terms too literally. Enterprises are assemblages of knowledge, power, responsibility, etc, and these elements are never going to be distributed uniformly. So even if we don't have a strict line dividing the inside from the outside, we can may be able to detect a gradient from inwards to outwards, and there may be opportunities to alter the gradient and make some strategic improvements, which we might frame in terms of agility or requisite variety. Hence Power to the Edge.

 

---

Related posts: Dividing Risk (April 2005), Jericho (May 2005), Power to the Edge (December 2005), Boundary, Perimeter, Edge (March 2007), Outside-In Architecture (August 2007), Ecosystem SOA (June 2010)

Regulation and Complexity

@timharford's article So Many Numbers, So Little Time yesterday prompted me to think about the causal relationship between regulation and complexity

There are at least three contrasting ideas about this relationship.

1. Red tape directly causes complexity. The solution to complexity is therefore to cut red tape. See for example the OECD document Overcoming Barriers to Administrative Simplification Strategies: Guidance for Policy Makers (pdf, 2009).

2. Conflicts of interest and mutual mistrust create complexity. The response to this complexity manifests itself in complicated forms of contractual negotiation, governance and regulation, which is the outward symptom of the intrinsic complexity of the situation. See for example my discussion of the Oil Industry example in my post Beyond Multiview.

3. Regulation triggers complex behaviour. In his article, Tim Harford suggests that it isn't always economic complexity causing regulatory complexity: in the financial industry, the causation often runs the other way. (Chester Spatt made a similar point in a keynote address earlier this year. Complexity of Regulation, HBLR June 2012.) Every attempt by the regulators to introduce greater complexity into market controls, in the hope of regulating the market more effectively, prompts an increase in the complexity of market behaviour.

Complex rules invite complex rule-bending, says Tim Harford. If at least some of the market players are more agile than the regulators, then they can always escape real regulation by shifting market operations to a higher complexity than the regulators can control. In which case it is not really the regulators who are controlling the market but the market players themselves. So what price Requisite Variety?

There is another twist to the story. Bad behaviour by large companies (Enron, WorldCom) has prompted legislation such as Sarbanes-Oxley, which turns out to be more burdensome for small companies than for larger companies. There seem to be significant economies of scale in such areas as compliance and tax avoidance: large companies can afford armies of accountants and lawyers and lobbyists and PR companies, some of the largest and most profitable companies pay practically no corporation tax. So who really benefits from all this red tape?

(In consequence, many start-up companies are designed to be acquired by a large company rather than undergo the massive administrative costs of a public flotation. Some commentators suggest that this tendency reduces the contribution of start-up companies to the macroeconomic goals of growth and full employment.)

See also Compliance and Control (May 2005), Requisite Variety and Wall Street Regulation (May 2012)

(updated October 15th 2012)

On the Causes of Business Complexity

#bizarch #complexity  @lawrencewilkes asks if business is "inherently complex", or have businesses just allowed it to become so? Are there vested interests at work who are striving for complexity?

(In my blogpost on The Future of Enterprise Architecture, I had asserted that real business is inherently complex. Lawrence's question can be found in the comments to this blogpost.)

What is complexity?

There are several competing notions of complexity in the literature, and circulating on the Internet, and I don't want to get bogged down in complexity theory. So I'm going to see how far I can get with this discussion without a formal definition of complexity. Instead of a definition, I am going to list some of the characteristic features of complex systems, taken from a book by Flood and Carson, derived in part from a paper by F.E. Yates.

1. Significant interactions
2. High number of parts, degrees of freedom or interactions
3. Non-linearity
4. Broken symmetry / asymmetry
5. Non-holonomic constraints - for example the Falling Cat Problem
6. Hierarchy and emergence
7. Aesthetics

R.L Flood and E.R. Carson, Dealing with Complexity (1988)

Causes of complexity

There are several possible causes of complexity and/or complication in systems. Roughly speaking, we can identify at least four five or six causes.

• Emergent complexity - complexity that is the consequence of many small and apparently unrelated decision and actions, which interact in unanticipated ways. Complicated IT systems can often be explained in this way, and some IT experts talk as if all complexity can be explained thus.

• Transitional complexity - complexity that is caused by the need to maintain multiple operational models during transformation, especially when there are many different transformations underway at the same time. And when large organizations such as the NHS are undergoing constant and continual change, there may be no end to this form of complexity. (I am indebted to Richard Gilyead for adding this one - see comments below.)

• Perverse complexity - complexity that is caused by clumsy efforts to reduce complexity. Often the complexity is merely displaced or exported into a system that is even less able to cope with it, thus weakening the overall ecosystem.

• Contrived complexity - complexity that is deliberately created to benefit some stakeholders (vested interests) at the expense of others. This is sometimes called Complexity-as-a-Weapon. See my post on Complexity-Based Pricing.

• Regulatory complexity - see my post on Regulation and Complexity.

• Irreducible complexity - complexity that reflects the real complexity of the demand environment.

In some circumstances, these factors may combine to produce yet more complexity.

Examples of irreducible business complexity

As I see it, there are many types of irreducible complexity in business, including various forms of asymmetry, indirect value, multi-sidedness and non-linearity.

For example, a supermarket chain can optimize/innovate its relationships with its suppliers, or it can optimize/innovate its relationships with its customers, or it can optimize/innovate its store layout. Complexity arises when it tries to optimize and innovate all three simultaneously - which is of course exactly what it does need to do in order to keep one step ahead of its competitors.

For example, complexity arises in the health service when it tries to orchestrate thousands of different clinical pathways, using any number of expensive resources, in any number of permutations, in a way that delivers both a joined-up experience for the patient and a cost-effective system for the taxpayer.

These kinds of complexity are not caused by support services such as IT or HR but by the structure of demand in these sectors. One challenge for IT is to build appropriate information systems and platforms that support the effective management of real business complexity. (One of the problems with the NHS National Programme for IT (NPfIT) was that despite having cost a vast amount of money, the system simply didn't have the requisite variety to tackle the real problems, and was seen as an irrelevant white elephant by a critical mass of influential stakeholders.) And one challenge for HR or OD is to build appropriate organization structures and incentive schemes that align with the complexities of the business.

Final remarks

I have sometimes talked about irreducible complexity in terms of requisite variety - the idea that an enterprise needs to be aligned with the complexity of the environment. However, this term may be misapplied, as it focuses on variety as being the most important aspect of complexity - but in other situations, we may wish to look at other aspect of complexity, such as Asymmetric Demand.

In any case, the alignment of the complexity of the enterprise with the complexity of the environment is a strategic choice. An enterprise may choose to ignore or attenuate external complexity, and offer a simplistic one-size-fits-all solution to its customers. Many large organizations do exactly this, possibly destroying long-term value for themselves and their customers, and yet they roll on from crisis to crisis, until some crisis finally finishes them off.

The capability of an organization to manage a given level of complexity is linked to what I call Organizational Intelligence. I believe that increasing organizational intelligence has an indirect strategic value, because it increases the ability of the organization to align itself with its environment. We may frame the benefits of this in terms of sustainability and business excellence.

Afterword: Tackling complexity

Some comments here and on Twitter remind me about various approaches that may be used for tackling particular forms of complexity, including Continuous Modernization, Simple Iterative Partitions (SIP) and the Theory of Constraints (TOC). Typically these are focused on a specific problem, such as better ROI from IT investments (@RSessions). These approaches may be successful in some situations, but the notion of complexity outlined here is broader than the range of any popular complexity-busting tool. 


(updated March 20th 2013)

Six Views of Business Architecture

#bizarch Does it make sense to produce a business architecture from a single viewpoint, such as a Process Viewpoint or a Capability Viewpoint? Or do we need a proliferation of viewpoints, perhaps arranged in a 2x2 matrix such as the Zachman Framework?

In its Business Architecture Overview, the Object Management Group Business Architecture Working Group has identified five key views of a business, which it calls 1) the Business Strategy view, 2) the Business Capabilities view, 3) the Value Stream view, 4) the Business Knowledge view, and 5) the Organizational view. See my previous post on the Five Views of Business Architecture (OMG), in which I also identify a sixth one, which I've been calling the Management or Cybernetic view. (Strictly speaking these are viewpoints rather than views, but I'm trying to stick to the OMG terminology as far as possible.)

I have used all of these views for business modelling, and I am convinced that they are all useful - although they may not all be required on every project. (I don't expect you to believe this without further evidence, and I am currently assembling a set of worked examples.)  I have presented earlier versions of this schema in the past, with slightly different names, but I believe the current names and explanations are clearer.

Business Motivation View (aka Purpose)

This viewpoint describes what the business achieves for itself and its stakeholders (direct and indirect value). It specifies desired, potential and actual performance in terms of goals and objectives as well as positive and negative outcomes. (For example, a risk is a potential negative outcome.)

Outcomes may be described qualitatively or quantitatively, and there are different analysis techniques and notations associated with the transition from quantity to quality.

A more sophisticated view would consider the uncertainty associated with these outcomes, as well as the potential conflict between the goals and values of different stakeholders. It would also recognize that there is often a subtle difference between the official or espoused goals and objectives of an enterprise and its defacto goals and objectives. See my post on Enterprise POSIWID.

The OMG calls this the Business Strategy view, but I think this is a misleading name. The strategy doesn't reside in motivation, but in how the business mobilizes itself to satisfy some set of motivations.

The name Business Motivation view reflects the Business Motivation Model (BMM), which provides some limited coverage of this view. Alternatively, I could call this the Business Performance view.

Business Capability View (aka Component)

This viewpoint describes how the business delivers direct and indirect value in response to the challenges of the environment. Many business architects merely decompose the business into something that looks like an old-fashioned functional hierarchy, but there are several important architectural issues that need to be addressed within the business capability view.

• Coordination of internal and external capability. When an organization chooses to delegate or outsource a particular capability, the coordination of the outsourced capability remains.

• Capability dependencies - how strength or weakness in one capability affects the performance of other capabilities.

• Requisite variety. An organization chooses what range and variety of events it is capable of responding to, and this variety is reflected in specific capabilities.

• Through-life capability management. Establishing a capability requires some level of investment, in terms of equipment and recruitment and training and so on. The costs of maintaining a capability may escalate, especially if the demands keep changing.

• Performance, quality and value-for-money. It is always possible to devote more energy and resources to improving any given capability, but this is only justified if it is reflected in business performance (aka "the bottom line"). Business excellence models (such as Baldrige and EFQM) address this kind of alignment.

Capabilities may be encapsulated in what I have called Business Components, as in my 2001 book on the Component-Based Business. Thus we could also call this the Component View.

Business Activity View (aka Process)

This viewpoint describes the day-to-day behaviour of the business, or what the OMG calls "business in motion". Traditionally processes are seen as a chain of value-adding process steps, thus the OMG calls this the Value Stream view. However, the value network paradigm looks significantly more powerful than the value chain paradigm, and I prefer to use the neutral term Business Activity.

Business Knowledge View (aka Concept)

This is a well-established architectural viewpoint, previously known under various names such as Conceptual Information Model or Business Data Architecture. It basically captures what the business knows, and the semantics of how the business talks to itself and its partners. This viewpoint is critical for business interoperability. Knowledge also includes contextual knowledge, which is essential for providing differentiated services.

At this level, I'm not distinguishing between data, information and knowledge. Classifying information as "structured" and "unstructured" is not as important to the business architecture as it is to the IT architecture. What is important, however, is the question of business epistemology - what it is possible for the business to know, and with what level of confidence. There are various sociotechnical mechanisms that may make it possible for a 21st century business to know all sorts of things that were simply not available before. But of course, the business architect is not directly interested in the mechanisms themselves - these belong to some other architecture.

Responsibility View (aka Organization or Relationship)

The OMG defines an organizational view in terms of the relationships between organizational units, so I think the term Responsibility View is a better name for this view. I have done a lot of work on business relationship modelling, looking in particular at questions of the organizational responsibility and accountability for activities and outcomes, and this work can be traced back to the ORDIT methodology of the early 1990s. These relationships and organizational interfaces may be represented as services, with various levels of formality in defining and measuring service levels. There are also important architectural issues in dealing with the idea of Business-As-A-Platform, in particular in relation to multi-sided markets.

Cybernetic View (aka Management or Governance)

Finally, the cybernetic view describes how the organization and its management thinks. This covers all the aspects of organizational intelligence, from information gathering, through sense-making and decision-making, to organizational memory and learning.

See separate post on the Cybernetic View.

Hybrid Views

These views are not sacrosanct, and there are some useful business models that may not fit neatly into these six viewpoints. For example, Martin Ould's Activity Role Diagrams seem to involve a combination of Activity and Responsibility, while Stafford Beer's Viable Systems Model combines the Capability View and the Cybernetic view.

I am also convinced that business strategy is not contained within a single view (as the OMG would have it) but is about the relationships between the views.

Levels of Abstraction

Each of these viewpoints can be applied at different levels of abstraction. For example, in business relationship modelling, we may identify a partner-independent model (which merely defines an abstract relationship with a notional business partner) and a partner-specific model (which defines a concrete and contractual relationship with a real business partner).  Meanwhile, in the capability view we may need to distinguish between abstract capabilities (which some frameworks regard as the only true capabilities) and sociotechnical lumps that implement these capabilities (which may be called components or nodes).

This relationship between abstract and concrete is what Zachman calls reification, and everyone else calls realization. Note that both the partner-independent model and the partner-specific model belong to the business architecture and not to any other architectural domain.

---

My booklet on Business Architecture Viewpoints is now available in draft on the LeanPub platform. http://leanpub.com/businessarchitecture-viewpoints/

Complexity and Power 2

#entarch #orgintelligence In an earlier post, I pointed out the architectural trade-off between Complexity and Power. There may be a choice between a simple-but-inefficient solution and a more sophisticated solution, and this choice has important structural implications. The architectural design of a large enterprise involves any number of these choices.

Furthermore, a given organization has a finite capacity for managing sophistication and complexity, and a limited willingness to invest in the management costs and overheads that may be required to extract the full benefit from more powerful systems.

There are two agendas that each appear to address opposite sides of this situation. The first is the simplification agenda, advocated most persuasively by my friend Roger Sessions, which addresses the problem that most organizations and their support systems (including IT) are riddled with unnecessary complexity.

The second is the maturity agenda, which suggests that an organization can extract more benefit from certain systems and practices and technologies by becoming more sophisticated. Maturity models are typically arranged as a series of maturity levels, and should be derived (although I suspect many of them aren't) from a robust theory of organizational change and technology adoption.

There are some subtle and usually ignored issues in the interaction between these two agendas, but I don't want to go into these issues here. Instead, I want to point to a couple of critical questions

• how much complexity and power does a given enterprise need (requisite variety)
• how should this complexity and power be distributed and connected

The first question looks like a simple engineering question, which should be bread-and-butter for requirements engineers. The need for complexity and power is driven by some set of demands, together with an economic argument that links cost to value (possibly but not necessarily expressed in financial terms).

The second question is clearly an architectural question. Two things are intuitively obvious about the design of complex machines: not every component needs the same amount of power and complexity; however, some components need proportionality of power and complexity. (It doesn't make sense to put a massive engine in a tiny car, and a fast car needs more powerful brakes than a slow car.)

Power and complexity are higher-order examples of so-called non-functional requirements. Architects need to be able to reason about the composition and decomposition of non-functional requirements.

• Let P be some property of systems and components - say performance or quickness or reliability or security or throughput or whatever. If the components of a system have properties P(1) ... (Pn), what can we say about the composite property P of the whole system?

• Conversely, if we have a requirement for the whole system to have property P, what can we say about the properties P(1) ... (Pn) of each component?

The enterprise is a sociotechnical system, so I'm not just talking about technical components here. A racing car needs a driver whose reaction speeds are matched to the speed of the car, and a maintenance team capable of rapidly diagnosing and fixing all the minor technical problems that could occur during the race.

Similarly, an enterprise needs a set of management capabilities whose intelligence is matched (proportional) to the power and complexity of the operations and operational systems - thus we have an architectural view of the enterprise as an intelligent sociotechnical system, with an appropriate balance of power and complexity.

I don't want to hear that the complexity has been completely removed from the technical systems, because I then fear that the necessary complexity has been merely displaced onto the people inside the organization - or worse, onto the customers.

Conversely, I don't want to hear that the complexity has been completely embedded inside the technical systems, because I then don't trust the people inside the organization to fully manage this complexity.

What I want to hear is that there is just enough complexity and power in the technical systems AND at least enough intelligence in the human systems to manage this complexity and power properly. And I expect the enterprise architect to be responsible for maintaining this proper balance.

Organizations as Brains

This post is based on Chapter 3 of Gareth Morgan's classic book Images of Organization (Sage 1986), which opens with the following question: "Is it possible to design organizations so that they have the capacity to be as flexible, resilient, and inventive as the functioning of a brain?"

To start with, Morgan makes two important distinctions. The first distinction is between two different notions of rationality, and the second involves two contrasting uses of the "brain" metaphor.

Mechanistic or bureaucratic organizations rely on what Morgan (following Karl Mannheim) calls "instrumental rationality", where people are valued for their ability to fit in and contribute to the efficient operation of a predetermined structure. Morgan contrasts this with "substantial rationality", where elements of organization are able to question the appropriateness of what they are doing and to modify their action to take account of new situations. Morgan states that the human brain possesses higher degrees of substantial rationality than any man-made system. (pp78-79)

Morgan also observes a common trend to use the term "brain" metaphorically to refer to a centralized planning or management function within an organization, the brain "of" the firm. Instead, Morgan wants to talk about brain-like capabilities distributed throughout the organization, the brain "as" the firm. Using the brain metaphor in this way leads to two important ideas. Firstly, that organizations are information processing systems, potentially capable of learning to learn. And secondly, that organizations may be holographic systems, in the sense that any part represents and can stand in for the whole. (p 80)

The first of these two ideas, organizations as information processing systems, goes back to the work of James March and Herbert Simon in the 1940s and 1950s. Simon's theory of decision-making leads us to understand organizations as kinds of institutionalized brains that fragment, routinize and bound the decision-making process in order to make it manageable. (p 81) According to this theory, the  organization chart does not merely define a structure of work activity, it also creates a structure of attention, interpretation and decision-making. (p 81) Later organization design theorists such as Jay Galbraith showed how this kind of decision-making structure coped with uncertainty and information overload, either by reducing the need for information or by increasing the capacity to process information. (pp 82-83)

Nowadays, of course, much of this information processing capacity is provided by man-made systems. Writing in the mid 1980s, Morgan could already see the emergence of the virtual organization, embedded not in human activity but in computer networks. If it wasn't already, the organization-as-brain is now indisputably a sociotechnical system. The really big question, Morgan asks, is whether such organizations will also become more intelligent. (p84)

The problem here is that man-made systems (bureaucratic as well as automatic) tend towards instrumental rationality rather than substantial rationality. Such systems can produce goal-directed behaviour under four conditions. (p87)

1. The capacity to sense, monitor and scan significant aspects of their environment
2. The ability to relate this information to the operating norms that guide system behaviour
3. Ability to detect significant deviations from these norms
4. Ability to initiate corrective action when discrepancies are detected.

But this is merely single-loop learning, whereas true learning-to-learn calls for double-loop learning.  Morgan identifies three factors that inhibit double-loop learning. (pp89-90)

1. Division of responsibilities cause a fragmentation of knowledge and attention.
2. Bureaucratic accountability and asymmetric information produce ethical problems such as deception. (This is a form of the principal-agent problem.) 
3. Organizations also suffer from various forms of collective self-deception, resulting in a gap between "espoused theory" and "theory-in-use".

and he goes on to identify four design principles that may facilitate double-loop learning. (pp 91-95)

1. Encourage and value openness and reflectivity. Accept error and uncertainty.
2. Recognize the importance of exploring different viewpoints. 
3. Avoid imposing structures of action. Allow intelligence and direction to emerge.
4. Create organizational structures and principles that help implement these principles.

The flexible, self-organizing capacities of a brain depend on four further design principles, which help to instantiate the notion of the "holographic" organization. (pp 98-103)

1. Redundancy of function - each individual or team has a broader range of knowledge and skills than is required for the immediate task-at-hand, thus building flexibility into the organization.
2. Requisite variety - the internal diversity must match the challenges posed by the environment. All elements of an organization should embody critical dimensions of the environment.
3. Minimal critical specification - allow each system to find its own form.
4. Learning to learn - use autonomous intelligence and emergent connectivity to find novel and progressive solutions to complex problems.

In conclusion, innovative organizations must be designed as learning systems that place primary emphasis on being open to enquiry and self-criticism. The innovative attitudes and abilities of the whole must be enfolded in the parts. (p 105) Morgan identifies two major obstacles to implementing this ideal.

1. The realities of power and control. (p 108)
2. The inertia stemming from existing assumptions and beliefs. (p 109)

Morgan says he favours the brain metaphor because of the fundamental challenge it presents to the bureaucratic mode of organization. (pp 382-3) Writing in the mid 1980s, Morgan noted that computing facilities were often used to increase centralization, and to reinforce bureaucratic principles and top-down hierarchical control, and expressed a hope that this was a consequence of the limited vision of system designers rather than a necessary consequence of the new technologies. "The principles of cybernetics, organizational learning, and holographic self-organization provide valuable guidelines regarding the direction [technology] change might take." (p 108) A quarter of a century later, let's hope we're finally starting to move in the right direction.

Ecosystem SOA 2

What are the problems of large complex sociotechnical systems? How far do SOA and enterprise architecture help to address this problem space, and what else might we need?


When I started writing about SOA and the service-based business over ten years ago, I defined two "cuts" across the service ecosystem. One cut separates inside from outside, while the other cut separates supply from demand.

(This diagram was included in my 2001 book on the Component-Based Business, and frequently referenced in my work for the CBDI Forum. For a brief extract from the book, see my Slideshare presentation on the Service Ecosystem.)

The inside/outside cut is sometimes called encapsulation. It decouples the external behaviour of a service from its internal implementation, and can be described in terms of knowledge - the outside has limited knowledge of the inside, and vice versa. (The cut is also sometimes called transparency - for example location transparency, which means that external viewers can't see where something is located.)

The supply/demand cut is about delegation, and can be described in terms of responsibility. Getting these two cuts right may yield economics of scale and scope; and the business case for SOA as a development paradigm is often formulated in terms of reusing and repurposing shared services.

For relatively small and simple SOA projects, it may be feasible to collapse the difference between these two cuts, and treat them as equivalent. (The inside/outside relationship and the supply/demand relationship are sometimes both described as "contracts", although they are clearly not the same kind of contract.) However, enterprise-scale SOA requires a proper articulation of both cuts: confusing them can result in suboptimal if not seriously dysfunctional governance and procurement. Many people in the SOA world still fail to understand the conceptual importance of these cuts, and this may help to explain why some organizations have had limited success with enterprise-scale SOA.

Going beyond enterprise SOA as it is generally understood, there is a third cut separating two views of a system: the system-as-designed (whose structure and behaviour and rules can perhaps be expressed in some formal syntax such as UML, BPMN or ArchiMate) and the system-in-use (whose actual performance is embedded/situated in a particular social or business context). This cut is critical for technology change management, because of the extent to which the designed system underdetermines the pragmatics of use. I have been talking about this cut for over twenty years, but only more recently working out how to articulate this cut in composition with the other two cuts.

One important reason for looking at the pragmatics of use is to understand the dimensions of agility. In many settings, we can see a complex array of systems and services forming a business platform, supporting a range of business activities. If no agility is required in the business, then it may not matter if the platform is inflexible, forcing the business activities to be carried out in a single standardized manner. But if we assume that agility is a critical requirement, then we need to understand how the flexibility of the platform supports the requisite variety of the business.

More generally, understanding the pragmatics of use leads to the recognition of a third kind of economic value alongside the economics of scale and the economics of scope: the economics of alignment. The value of a given system-of-systems depends on how it is used to deliver real (joined-up) business outcomes, across the full range of business demands. (I'm afraid I get impatient with people talking glibly and simplistically about business/IT alignment, without paying attention to the underlying complexity of this relationship.)

Understanding these three cuts (and analysing their implications) is critical to understanding and managing a whole range of complex systems problems - not just SOA and related technologies, not even just software architecture, but any large and complex sociotechnical systems (or systems-of-systems). If the three cuts are not understood, the people in charge of these systems tend not to ask the right questions. Questions of pragmatics are reduced to questions of platform design; while questions of the cost-justification and adoption of the platform are reduced to a simple top-down model of business value. Meanwhile the underlying business complexity (requisite variety) will be either misplaced (e.g. buried in the platform) or suppressed (e.g. constrained by the platform).

So there are three challenges I face as a consultant, attempting to tackle this kind of complex problem. The first challenge is to open up a new way of formulating the presenting problem, based on the three cuts. The second challenge is to introduce systematic techniques for analysing the problem and visualizing the key points. And the third challenge is to identify and support any organizational change that may be needed.

---

With thanks to Philip Boxer and Bernie Cohen. For a different formulation of the three cuts, together with a detailed example, see their new paper "Why Critical Systems Need Help to Evolve" Computer, vol. 43, no. 5, pp. 56-63, May 2010, doi:10.1109/MC.2010.150. See also Philip Boxer, When is a stratification not a universal hierarchy? (January 30th, 2007)


Related post Ecosystem SOA (October 2009)

Differentiation and Integration

In my post on Business Design Choices, I suggested that the real challenge for business architecture was to appreciate the economic and social impact of structure. I identified some advanced business design choices where business architecture has a useful role to play, and where some IT people might have some relevant aptitude. In this post, I am going to expand on one of these.

---

Where does standardization make sense, and where should requisite variety be deployed? This question goes back to the work of Lawrence and Lorsch (L2), and has recently been rediscovered (in slightly different terms) by Ross, Weill and Robertson (RWR).

In their book Enterprise Architecture as Strategy, RWR define something they call an Operating Model, with two independent dimensions, business process standardization and integration. "Although we often think of standardization and integration as two sides of the same coin, they impose different demands. Executives need to recognize standardization and integration as two separate decisions." (p27)

Many people in the IT world take for granted that standardization (reduction in variety) is a good thing.  RWR acknowledge the benefits of standardization not only in terms of throughput and efficiency, but also predictability. However, they point out the potential downside of standardization both as a state (standardized processes limit local innovation) and as a state-change (politically difficult and expensive to rip out and replace perfectly good and occasionally superior systems and processes).

Integration, which RWR define primarily in terms of data sharing, is also assumed to be a good thing. RWR identify the benefits of integration in terms of efficiency, coordination, transparency and agility, and acknowledge the challenge of integration as a state-change in terms of "difficult, time-consuming decisions".

The two dimensions of standardization and integration produce a two-by-two matrix as follows.

Operating Model Quadrants (Adapted by Clive Finkelstein from Figure 2.3 of “Enterprise Architecture as Strategy”) - Source The Enterprise Newsletter #38.


Although RWR are careful to present a contingency theory of enterprise strategy, in which any of these four operating models may be strategically valid, the conventional rhetoric of the two-by-two matrix places the preferred strategy into the top right quadrant - thus Unification. Many enterprise architects from a traditional IT background may feel most comfortable with the Unification quadrant. (There may be a process of idealization here - see Schwartz.) Indeed, RWR go on to present an Enterprise Architecture Maturity Model, which starts with the easy bits of enterprise architecture (where things are already Unified) and ends with the challenging bits (where things are or should be Diversified).

---

L2 also identify two dimensions, which they call differentiation and integration. They tend to see increasing differentiation as a healthy response to increasing opportunity and complexity - a growing organization in a growing market - "faster change and greater heterogeneity" (p 235). Differentiation is not merely a difference in working practices, but includes at its core "the difference in cognitive and emotional orientation among managers in different functional departments" (p11).

Integration is then the administrative response to this increasing differentiation - how to maintain the overall coherence and viability of the enterprise as a whole. Integration is defined as "the quality of the state of collaboration that exists among departments that are required to achieve unity of effort by the demands of the environment" (p11). The topic of integration covers both the organizational state and the mechanisms to produce and support this state. For L2, the challenges of integration are not primarily IT related (data sharing) but personal and political (conflict resolution).

L2 don't offer a two-by-two matrix of Differentiation and Integration in their 1967 book (I guess the two-by-two matrix hadn't then established itself as an essential consultancy tool), but if they did then presumably the top-right quadrant would be High Differentiation, High Integration. This very roughly corresponds to what RWR call Coordination.

But in any case, a two-by-two matrix would be misleading. The point isn't to choose whether you have differentiation and integration or not; the point is to determine how much and what kinds of differentiation  and integration you need. L2 are explicit in their support of contingency theory (different strategies being appropriate for different organizations depending on environmental factors). The more complex and dynamic the environment, the greater the need for differentiation and integration.

If we are to take business architecture seriously as a discipline, then this kind of question is clearly central. In his post on Contingency Theory and Enterprise Architecture, Andy Blumenthal argues that contingency theory entails keeping your options open, but sometimes it just means making appropriate strategic choices. If the slogan "enterprise architecture as strategy" is to mean anything at all, then surely this is what it means.

---

Lawrence, P., and Lorsch, J., "Differentiation and Integration in Complex Organizations" Administrative Science Quarterly 12, (1967), 1-30. (see summary here)

Paul R. Lawrence and Jay W. Lorsch, Organization and Environment, Managing Differentiation and Integration. Harvard University 1967.

Jeanne W. Ross, Peter Weill and David C. Robertson, Enterprise Architecture as Strategy. Harvard Business School Press 2006.

Howard S. Schwartz, Narcissistic Process and Corporate Decay – The Theory of the Organizational Ideal. 1990. See his paper On the Psychodynamics of Organizational Disaster (Columbia Journal of World Business, Spring 1987) See also Joe Bormel's blogpost on Narcissism, Oxygen and HCIT Vision (June 2009).


Related posts

Business Design Choices (January 2010)
EA Effectiveness and Process Standardization (August 2012)

Enterprise as a Network of Events

From the early days of SOA, we've talked about understanding the enterprise as a network of services, but clearly this is not the only possible viewpoint. Can we understand the enterprise as a network of events?

In a recent post SOA and Events - Two Sides of BAM?, Ramesh Loganathan (Progress) explained how a process view of the enterprise could be flipped into an event-oriented view. Ramesh's example is in governance risk and compliance (GRC), but it looks as if the same idea could be applied to a much broader range of business requirements.

Meanwhile, in Decision Management and software development, James Taylor wants to model the enterprise as a series of decisions. But of course a decision can be understood as a special kind of event. See Model Driven Engineering and Event Processing by Opher Etzion (IBM). See also SOA is dead; long live Model-Driven SOA by Johan den Haan.

In his post Event, Rules, Processes, Decisions, Paul Vincent (TIBCO) identifies an important gap in the notations (BPMN, BMM) available for modelling these aspects of the enterprise. He talks about decisions in terms of rules, but I presume he is associating the rule with the decision process (this is how we authorize transactions) rather than with the specific instance of decision (I hereby authorize this transaction).

At the strategic level, we need to understand the relationship between an external set of possible events and an internal set of possible events. Each enterprise is capable of responding in certain ways to certain classes of external events. Making an enterprise more agile means enabling the enterprise to mobilize more appropriate responses to a greater range of external events, without proliferation of unnecessary complexity. In system thinking this is called Requisite Variety.

So if we think about the enterprise as a network of events, this gives us a direct way to reason about strategic business improvement. But what are the CEP vendors doing in this space? I can see a lot of talk about specific event-driven applications, but where is the architectural thinking about the event-driven enterprise?


Update: After I posted this, Nigel Green drew my attention to something he wrote a couple of years ago on the case for a clear distinction between events and content, which he describes in terms of the wave-particle metaphor.

Event Reduction

Opher Etzion on Event Reduction

“in some cases, the business benefit of event processing system is actually reducing the number of events, and focus the decision makers on the important events”

An event processing system generally has some degree of event reduction and aggregation. There may be real-world events that are not represented as system events (for example, a continuous real-world process is reduced to a series of periodic instrument readings, and the intermediate points are not captured). And some events may be front-end captured but subject to some filtering so that not all of them are passed onto the business applications.

In both cases, the total behaviour of the enterprise system may be affected by the quantity and variety of events that get through to the business applications. Too many events, and the decision-makers may be unable to focus on the important ones. Too few events, and the enterprise may lack requisite variety. A good cybernetic systems design would include feedback loops, so that the quantity and variety of events can be adjusted until the enterprise works to the maximum efficiency and effectiveness.

So an event processing architecture needs some kind of variety regulator (“regulator” in the engineering sense, not the legal compliance sense) – some kind of monitoring and control. For example, changing the frequency of some measurement, or changing the filtering or aggregation rules.

In very simple event processing, the systems designers might possibly be clever enough to calculate the right measurement frequencies and event processing rules in advance, although I don’t know of any robust methodology for doing this. For complex event processing the mathematics start to get too difficult, and we need to rely on proper engineering and not just simple arithmetic.

Clouds and Clocks 4

An interesting debate on SaaS and Control

• Cloudy Future for the Enterprise and most likely for ISVs too (Gianpaolo Carraro, Microsoft)
• Tradeoffs in SaaS (Charlie Bass, EDS)

Gianpaolo starts out by suggesting a simple trade-off between user control and the economics of scale. With SaaS, the user forgoes some control, in order to benefit from (some share of) the economics of scale achieved by the service provider. Gianpaolo divides control into two aspects: Control of Features and Control of SLA.

Charlie thinks it comes down to a perspective of what is "control". He suggests a different notion of control, whereby a contract with a service provider would give you more legal control than doing the work in-house.

But as far as I can see, both of them are talking about Command (setting the target conditions) rather than Control (regulating the system to satisfy the target conditions). See Alberts and Hayes, Understanding Command and Control (pdf).

Gianpaolo associates the Control of Features with who builds the service (Implementation) and the Control of SLA with who runs the service (Deployment). (This is a bit of a simplification; I shall ignore that for the moment.) But in Service-Oriented Architecture, there is a third and perhaps the most important role: who specifies the service. It is possible for the user (or a community of users, or an independent regulatory body) to provide the specification, and for the provider merely to deliver an acceptable implementation and deployment. Returning to the distinction between Command and Control, we may associate the Command of both Features and SLA with who specifies the service.

But what kind of service are we talking about here? The economics of scale are most obviously associated with simple one-size-fits-all services that are symmetric and replicable - what Philip Boxer calls r-type services. In most cases such services are unilaterally specified by the service provider, offer little if any contextual variety to the service user, and control tends to focus on quality control and cost control.

With more complex types of service, the service user typically demands a much greater fit to a specific use-context, and the question of control expands to include requisite variety and the economics of governance. But does this mean that the economics of scale go out of the window? The challenge for very large and complex service-based systems is to combine the economics of scale AND scope AND governance.

BPM and SOA

At the BCS Business Information Systems SIG last night to hear a talk by Howard Smith of CSC. Meanwhile, I've been listening to an ACM Queue podcast with Mike Vizard and Edwin Khodabakchian (formerly Collaxa, now Oracle). 

I have spoken to Edwin and his (present and former) colleagues in the past. I hadn't met Howard before, but I was already familiar with his work and I am in contact with some of his associates in the BPMI world. 

The following post is something I've been thinking about for a while. I am happy to acknowledge the influence of Howard and Edwin, who both know a lot more about the BPM side than I do. But there are some things I'm saying here that I haven't heard either of them say, so blame me (not them) if you don't understand or agree.

---

What is the relationship between Business Process Management and SOA?

A good place to start is with the idea that SOA gives you the decomposition of functionality into (standardized) services, while BPM gives you the assembly of these services into (flexible) business solutions.

1. We can decouple the specification of the process from the specification of the units of work making up the process. (In my Business Modelling for SOA material, I describe this as separating WHAT from HOW.)
2. We can put the specification of the process into a standard process language. (The preferred choice here is BPEL, for various reasons, although there are some known issues.)
3. We can automate and distribute the assembly of the process. (With appropriate tools, the assembly of the process or the selection of an appropriate process variant can be done either in real-time or dynamically at the point of need.)
4. We can then take power to the edge - delegating process management to the people working at the edge of the organization.
5. Ultimately, not just the process but the process management becomes event-driven. This gives us requisite variety at the process level.
6. Process architects now need to work at a higher level of abstraction. Instead of specifying a standard one-size-fits-all process, they need to produce what we might call a metaprocess - frameworks and patterns that support process management. They need to pay attention to architectural process issues, such as coordination and interoperability risk.

These ideas are probably some way from mainstream adoption; but there seem to be enough early adopters experimenting with some of these ideas to keep the BPM market moving forward.

---

Business Process Innovation

The second part of Howard's talk was about Business Process Innovation. If we imagine that BPM plus SOA gives us a reasonable way of automating the business process, then the next challenge is that of automating business process change. 

Howard's working hypothesis is that business process change is driven by problems. CSC has adopted a Russian problem-solving methodology called TRIZ, and has been working on a process-oriented version of TRIZ called P-TRIZ. 

Most of the audience (including myself) were not familiar with TRIZ, and there was a lively discussion about its strengths and weaknesses. My first impression is that TRIZ would not be suitable for all types of business problem, as it seems to lack adequate constructs for modelling complex dynamic systems. However, it does have some interesting characteristics that may make it particularly amenable to automation. Firstly, there is a systematic approach to problem decomposition. Secondly, there is a complete enumeration of solution strategies, and a useful collection of abstract solution patterns. 

Howard showed us a stand-alone prototype tool that supported a simple version of TRIZ. What would be really interesting (and this is presumably CSC's plan) would be to have this functionality integrated into the BPM tools. Then we would be able to have local process-oriented problem-solving, with process architects able to concentrate on the emergent properties of the whole. 

This looks like a significant contribution to the overall BPM/SOA vision outlined above.

Compliance and Control

The US Sarbanes-Oxley Act of 2002 (SOX) mandates what is effectively a systems engineering solution to the problem of bezzle. Reliability of a company's published accounts is achieved not by human oversight alone, but by a set of information and control systems that ensures information quality and management accountability. Executive officers are required to sign the accounts and are criminally liable for any inaccuracy. The act also mandates near-real-time disclosure of any material events.

One of the basic tenets of control theory is that a control system must have as much flexibility and variety as the system it is trying to control. This is known as Requisite Variety. Previous attempts at internal audit and control have either themselves lacked flexibility and responsiveness, thus compromising their ability to deliver effective control, or have imposed inflexibility and unresponsiveness on the underlying system. Neither of these outcomes is acceptable.

For the management controls to be effective there must be some way for the management system to intervene in the transaction system. There is an architectural choice between direct and indirect intervention.

Direct	Indirect
When the management subsystem detects an anomaly, it immediately intervenes in the transaction subsystem - possibly changing the status of some transaction record, or moving a transaction to a different account within the general ledger. Changes to the transaction system may also involve adjustments to elements of policy or context governing the transaction subsystem.	When the management subsystem detects an anomaly, it triggers a piece of workflow, or generates some additional transactions, which are fed back into the transaction system. A human supervisor is notified, who may have the authority to make changes in the transaction system and/or undertake further investigation.
This approach tightly couples the management subsystem and the transaction subsystem into a single unified system. The management subsystem is intrusive into the transaction subsystem.	This approach maintains loose coupling between the management subsystem and the transaction subsystem. The software component of the management subsystem provides non-intrusive monitoring. This can be implemented as an instance of the Observer pattern.

The SOA principle of loose coupling clearly favors the indirect, non-intrusive approach. However, we must be careful to ensure that the indirect control remains effective. There needs to be some coordination between the collaboration or process management layer in the management subsystem, and the collaboration or process management layer in the transaction subsystem.
In a dynamic SOA environment, we actually need a double instance of the Observer pattern. The management system needs to monitor dynamic changes to the transaction workflow, as well as the transactions themselves.
The management subsystem must be as adaptable as the transaction subsystem - and this calls for the same SOA design principles to be applied to both.

Information Management / Business Intelligence	Information architects may wish to think of SOX requirements in terms of data integrity - providing a guarantee of consistency and completeness across multiple diverse applications and data stores. For information management, Sarbanes-Oxley provides a push towards real-time closed-loop information management and business intelligence. Many BI products are now Web Service enabled, and this makes it easier to quickly plug BI capabilities such as monitoring and data mining into the business process.
Model-Driven Compliance	It is impossible for an organization of any size or complexity to achieve SOX compliance without some serious modeling effort - involving both data and process, and showing how data are transformed across multiple and diverse applications and data stores.
Web Service Based EAI	SOX compliance typically requires a major rewiring job to corporate financial information systems. In some cases, some additional applications and packages will need to be quickly plugged in to complete the solution. New or reengineered interfaces should be rendered as Web Services by default. Organizations that are already using a Web Service or EAI platform will be able to leverage this to achieve SOX compliance more quickly.

---

See also


CBDI Report April 2004: Sarbanes-Oxley Drives Web Services Adoption
Notions: Bezzle

SOA Principles

Several pundits have attempted to define the principles of Service-Oriented Architecture (SOA). Here is a summary of the SOA principles I have been propounding.

Service Ecosystem	Business viability depends on delivering services into a broader service ecosystem. Thus the service economy drives the business (which drives further services, which ultimately drive technology.)
Service Economy	Each service represents a unit of value. Services are regarded as intrinsically tradeable, and have both an exchange value and a use value. (Of course we may often choose not to exercise the option to trade services, for various reasons.)
Service Integrity	Each service represents a meaningful 'whole' from the user-side as well as from the supply-side. Service coherence, reliability and 'wholeness' promotes broad and robust use/reuse.
Loose Coupling / Rich Coupling	Open (typically asynchronous) connections between organizations, components and services. Interoperability between human activity and software services.
Differentiated Service	Functionality, quality or cost vary with circumstances, including identity and context. This helps to generate requisite variety in the service ecosystem.
Multiple Provision	Availability of alternative services or service implementations, biodiversity. This produces agile and robust systems, and also helps to generate requisite variety in the service ecosystem.
Complexity / Stratification	Complex networks of services must be understood as systems of systems. Such systems are generally organized in layers: one layer acts as a platform of services for the layer above.
Distributed Intelligence	Not only is functionality distributed across a network of services, but the intelligence governing this functionality is also distributed. Systems with distributed intelligence may be amenable to much more radical change than centralized ones.
Model-Based Management	Using business models to drive all aspects of system/service management seamlessly through development, testing/simulation and operations to provide a common understanding and visibility of systems and services to monitor and control all aspects of system design and operational performance.
Component-Based Business	Loosely coupled networks of independent business components.
Emergent Order	The service economy evolves into a continous network of value-adding services, through a series of structure-preserving transformations.

I shall try to expand and illustrate these in future posts.