Sunday, December 01, 2019
Data Strategy - Richness
In my previous post, I looked at Reach, which is about the range of data sources and destinations. Richness of data addresses the complexity of data - in particular the detailed interconnections that can be determined or inferred across data from different sources.
For example, if a supermarket is tracking your movements around the store, it doesn't only know that you bought lemons and fish and gin, it knows whether you picked up the lemons from the basket next to the fish counter, or from the display of cocktail ingredients. And can therefore guess how you are planning to use the lemons, leading to various forms of personalized insight and engagement.
Richness often means finer-grained data collection, possibly continuous streaming. It also means being able to synchronize data from different sources, possibly in real-time. For example, being able to correlate visits to your website with the screening of TV advertisements, which not only gives you insight and feedback on the effectiveness of your marketing, but also allows you to guess which TV programmes this customer is watching.
Artificial intelligence and machine learning algorithms should help you manage this complexity, picking weak signals from a noisy data environment, as well as extracting meaningful data from unstructured content. From quantity to quality.
In the past, when data storage and processing was more expensive than today, it was a common practice to remove much of the data richness when passing data from the operational systems (which might contain detailed transactions from the past 24 hours) to the analytic systems (which might only contain aggregated information over a much longer period). Not long ago, I talked to a retail organization where only the basket and inventory totals reached the data warehouse. (Hopefully they've now fixed this.) So some organizations are still faced with the challenge of reinstating and preserving detailed operational data, and making it available for analysis and decision support.
Richness also means providing more subtle intelligence, instead of expecting simple answers or trying to apply one-size-fits all insight. So instead of a binary yes/no answer to an important business question, we might get a sense of confidence or uncertainty, and an ability to take provisional action while actively seeking confirming or disconfirming data. (If you can take corrective action quickly, then the overall risk should be reduced.)
Next post: Agility
Saturday, February 09, 2013
Complexity is not a problem
- "The most pressing problem of the organization is complexity. Complexity is the primary responsibility of the enterprise architect." (Roger Sessions, Controlling Complexity in Enterprise Architecture pdf June 2007)
- "Enterprises are instances of complex adaptive systems having many interacting subcomponents whose interactions yield complex behaviors. Enterprise Architecture is a way of understanding and managing such complexity." (Beryl Bellman Managing Organizational Complexity pdf FEAC Oct 2009)
Indeed, I'm sure I've said things like this myself. But if complexity is a problem, whose problem is it? I am not seeing a huge rush of businessmen hiring enterprise architects just to deal with The Complexity Problem. Usually they have much more practical problems that they want addressing.
Complexity is a problem amplifier
So here's the thing. Apart from architects, people generally don't see complexity as their problem. What they do often acknowledge, however, is that complexity makes their problems worse. Furthermore, complexity may be one of the reasons they can't solve their problems for themselves. So complexity is a relevant factor, it's just not the problem itself.
Complexity as a lifestyle choice
And where does the complexity come from? Ironically, complexity is often created by failed attempts to reduce or eliminate complexity. In a post about the AntiFragile Enterprise (Jan 2013), Alan Hakami warns against "a tendency to build over-governed solutions that try to 'manage' complexity or uncertainty".
And where is the motivation to eliminate complexity? Suppose you go to the doctor with a headache, and the doctor says the reason you keep getting these headaches is you don't get enough fresh air and exercise. The pills just give you a temporary relief from the symptoms. You know she's right, but somehow you can't manage to get up early enough to go for a run before work. So you continue to get the headaches and you continue to need the pills. Indeed, if the pills work, you may start to exercise even less than you did before.
According to this analogy, an organization chooses the level of complexity it is comfortable with, and may well resist attempts to shift to a higher or lower level.
Complexity as a smokescreen
Complexity is an opportunity amplifier
If one organization is better at handling complexity than its competitors, as a consequence of superior agility and/or intelligence, then it can use complexity as a weapon. And many organizations use this weapon against customers or regulators. As @JackGavigan says, complexity is only a problem for those who lack the brain power to deal with and exploit it.
So that gives the enterprise architect a rather different perspective on complexity, doesn't it?
Sunday, October 14, 2012
Regulation and Complexity
There are at least three contrasting ideas about this relationship.
1. Red tape directly causes complexity. The solution to complexity is therefore to cut red tape. See for example the OECD document Overcoming Barriers to Administrative Simplification Strategies: Guidance for Policy Makers (pdf, 2009).
2. Conflicts of interest and mutual mistrust create complexity. The response to this complexity manifests itself in complicated forms of contractual negotiation, governance and regulation, which is the outward symptom of the intrinsic complexity of the situation. See for example my discussion of the Oil Industry example in my post Beyond Multiview.
3. Regulation triggers complex behaviour. In his article, Tim Harford suggests that it isn't always economic complexity causing regulatory complexity: in the financial industry, the causation often runs the other way. (Chester Spatt made a similar point in a keynote address earlier this year. Complexity of Regulation, HBLR June 2012.) Every attempt by the regulators to introduce greater complexity into market controls, in the hope of regulating the market more effectively, prompts an increase in the complexity of market behaviour.
Complex rules invite complex rule-bending, says Tim Harford. If at least some of the market players are more agile than the regulators, then they can always escape real regulation by shifting market operations to a higher complexity than the regulators can control. In which case it is not really the regulators who are controlling the market but the market players themselves. So what price Requisite Variety?
There is another twist to the story. Bad behaviour by large companies (Enron, WorldCom) has prompted legislation such as Sarbanes-Oxley, which turns out to be more burdensome for small companies than for larger companies. There seem to be significant economies of scale in such areas as compliance and tax avoidance: large companies can afford armies of accountants and lawyers and lobbyists and PR companies, some of the largest and most profitable companies pay practically no corporation tax. So who really benefits from all this red tape?
(In consequence, many start-up companies are designed to be acquired by a large company rather than undergo the massive administrative costs of a public flotation. Some commentators suggest that this tendency reduces the contribution of start-up companies to the macroeconomic goals of growth and full employment.)
See also Compliance and Control (May 2005), Requisite Variety and Wall Street Regulation (May 2012)
(updated October 15th 2012)
Thursday, October 04, 2012
On the Causes of Business Complexity
(In my blogpost on The Future of Enterprise Architecture, I had asserted that real business is inherently complex. Lawrence's question can be found in the comments to this blogpost.)
What is complexity?
There are several competing notions of complexity in the literature, and circulating on the Internet, and I don't want to get bogged down in complexity theory. So I'm going to see how far I can get with this discussion without a formal definition of complexity. Instead of a definition, I am going to list some of the characteristic features of complex systems, taken from a book by Flood and Carson, derived in part from a paper by F.E. Yates.
1. Significant interactions
2. High number of parts, degrees of freedom or interactions
3. Non-linearity
4. Broken symmetry / asymmetry
5. Non-holonomic constraints - for example the Falling Cat Problem
6. Hierarchy and emergence
7. Aesthetics
Causes of complexity
There are several possible causes of complexity and/or complication in systems. Roughly speaking, we can identify at least
- Emergent complexity - complexity that is the consequence of many small and apparently unrelated decision and actions, which interact in unanticipated ways. Complicated IT systems can often be explained in this way, and some IT experts talk as if all complexity can be explained thus.
- Transitional complexity - complexity that is caused by the need to maintain multiple operational models during transformation, especially when there are many different transformations underway at the same time. And when large organizations such as the NHS are undergoing constant and continual change, there may be no end to this form of complexity. (I am indebted to Richard Gilyead for adding this one - see comments below.)
- Perverse complexity - complexity that is caused by clumsy efforts to reduce complexity. Often the complexity is merely displaced or exported into a system that is even less able to cope with it, thus weakening the overall ecosystem.
- Contrived complexity - complexity that is deliberately created to benefit some stakeholders (vested interests) at the expense of others. This is sometimes called Complexity-as-a-Weapon. See my post on Complexity-Based Pricing.
- Regulatory complexity - see my post on Regulation and Complexity.
- Irreducible complexity - complexity that reflects the real complexity of the demand environment.
In some circumstances, these factors may combine to produce yet more complexity.
Examples of irreducible business complexity
As I see it, there are many types of irreducible complexity in business, including various forms of asymmetry, indirect value, multi-sidedness and non-linearity.
For example, a supermarket chain can optimize/innovate its relationships with its suppliers, or it can optimize/innovate its relationships with its customers, or it can optimize/innovate its store layout. Complexity arises when it tries to optimize and innovate all three simultaneously - which is of course exactly what it does need to do in order to keep one step ahead of its competitors.
For example, complexity arises in the health service when it tries to orchestrate thousands of different clinical pathways, using any number of expensive resources, in any number of permutations, in a way that delivers both a joined-up experience for the patient and a cost-effective system for the taxpayer.
These kinds of complexity are not caused by support services such as IT or HR but by the structure of demand in these sectors. One challenge for IT is to build appropriate information systems and platforms that support the effective management of real business complexity. (One of the problems with the NHS National Programme for IT (NPfIT) was that despite having cost a vast amount of money, the system simply didn't have the requisite variety to tackle the real problems, and was seen as an irrelevant white elephant by a critical mass of influential stakeholders.) And one challenge for HR or OD is to build appropriate organization structures and incentive schemes that align with the complexities of the business.
Final remarks
I have sometimes talked about irreducible complexity in terms of requisite variety - the idea that an enterprise needs to be aligned with the complexity of the environment. However, this term may be misapplied, as it focuses on variety as being the most important aspect of complexity - but in other situations, we may wish to look at other aspect of complexity, such as Asymmetric Demand.
In any case, the alignment of the complexity of the enterprise with the complexity of the environment is a strategic choice. An enterprise may choose to ignore or attenuate external complexity, and offer a simplistic one-size-fits-all solution to its customers. Many large organizations do exactly this, possibly destroying long-term value for themselves and their customers, and yet they roll on from crisis to crisis, until some crisis finally finishes them off.
The capability of an organization to manage a given level of complexity is linked to what I call Organizational Intelligence. I believe that increasing organizational intelligence has an indirect strategic value, because it increases the ability of the organization to align itself with its environment. We may frame the benefits of this in terms of sustainability and business excellence.
Afterword: Tackling complexity
Some comments here and on Twitter remind me about various approaches that may be used for tackling particular forms of complexity, including Continuous Modernization, Simple Iterative Partitions (SIP) and the Theory of Constraints (TOC). Typically these are focused on a specific problem, such as better ROI from IT investments (@RSessions). These approaches may be successful in some situations, but the notion of complexity outlined here is broader than the range of any popular complexity-busting tool.
(updated March 20th 2013)
Sunday, March 27, 2011
Complexity and Value - Is Amazon bothered?
This raises an important question about Amazon's capability and willingness to manage complexity - specifically differentiation. Amazon already uses multiple alternative delivery channels, but introducing an automatic selection according to the value and size of the item might well further complicate its delivery processes. If Amazon wished to differentiate customers according to the convenience / cost of different delivery channels, it would presumably need to have some geographical reasoning capability. And if Amazon wished to differentiate those products that would fit through a customer's letterbox, it would presumably need to know the size of each customer's letterbox.
We can easily imagine that Amazon could build this kind of capability, perhaps using Google Maps and Google Streetview to check the exact location of Sally's house and estimate the size of her letterbox. But has Sally a right to complain if Amazon doesn't bother?
To what extent do customers have a reasonable expectation of sophisticated differentiation of service - from organizations in general or from Amazon in particular? We may note that most large commercial organizations are way behind Amazon in their ability to enhance customer value through differentiation, while most small commercial organizations are way behind Amazon in their ability to integrate across a complex ecosystem. Because Amazon has already done some amazing things, we may expect it to go even further. But even Amazon has a practical limit of how much complexity it can manage.
We are surrounded by organizations that really can't be bothered, and the value deficit in some sectors is quite disgraceful. So it may be unfair to pick on Amazon, an organization that has done more than most to stretch the limits of the possible. But Sally is right - the future belongs to those organizations able and willing to pay attention to this kind of detail, if it helps to produce direct or indirect value.
Thursday, February 17, 2011
EA Talk on Managing Complexity
Outline
As business becomes more complex, enterprises are faced with structural complexities that critically affect business performance. Enterprise architecture provides a way of planning and coordinating both human systems and technical systems (including IT), and thereby overcoming the structural inhibitors to business performance. I shall talk about some emerging practical methods for business architecture, and show how these can be used in tandem with the standard popular approaches to manage the true complexity of the business.Slides
Following my talk, Professor Bernie Cohen talked about some related work by Philip Boxer and himself. Slides no longer available. https://www.slideshare.net/PhilipBoxer/mgt-complex-ea-feb-2011-healthcare
Thursday, December 16, 2010
Complexity and Power 2
Furthermore, a given organization has a finite capacity for managing sophistication and complexity, and a limited willingness to invest in the management costs and overheads that may be required to extract the full benefit from more powerful systems.
There are two agendas that each appear to address opposite sides of this situation. The first is the simplification agenda, advocated most persuasively by my friend Roger Sessions, which addresses the problem that most organizations and their support systems (including IT) are riddled with unnecessary complexity.
The second is the maturity agenda, which suggests that an organization can extract more benefit from certain systems and practices and technologies by becoming more sophisticated. Maturity models are typically arranged as a series of maturity levels, and should be derived (although I suspect many of them aren't) from a robust theory of organizational change and technology adoption.
There are some subtle and usually ignored issues in the interaction between these two agendas, but I don't want to go into these issues here. Instead, I want to point to a couple of critical questions
- how much complexity and power does a given enterprise need (requisite variety)
- how should this complexity and power be distributed and connected
The first question looks like a simple engineering question, which should be bread-and-butter for requirements engineers. The need for complexity and power is driven by some set of demands, together with an economic argument that links cost to value (possibly but not necessarily expressed in financial terms).
The second question is clearly an architectural question. Two things are intuitively obvious about the design of complex machines: not every component needs the same amount of power and complexity; however, some components need proportionality of power and complexity. (It doesn't make sense to put a massive engine in a tiny car, and a fast car needs more powerful brakes than a slow car.)
Power and complexity are higher-order examples of so-called non-functional requirements. Architects need to be able to reason about the composition and decomposition of non-functional requirements.
- Let P be some property of systems and components - say performance or quickness or reliability or security or throughput or whatever. If the components of a system have properties P(1) ... (Pn), what can we say about the composite property P of the whole system?
- Conversely, if we have a requirement for the whole system to have property P, what can we say about the properties P(1) ... (Pn) of each component?
The enterprise is a sociotechnical system, so I'm not just talking about technical components here. A racing car needs a driver whose reaction speeds are matched to the speed of the car, and a maintenance team capable of rapidly diagnosing and fixing all the minor technical problems that could occur during the race.
Similarly, an enterprise needs a set of management capabilities whose intelligence is matched (proportional) to the power and complexity of the operations and operational systems - thus we have an architectural view of the enterprise as an intelligent sociotechnical system, with an appropriate balance of power and complexity.
I don't want to hear that the complexity has been completely removed from the technical systems, because I then fear that the necessary complexity has been merely displaced onto the people inside the organization - or worse, onto the customers.
Conversely, I don't want to hear that the complexity has been completely embedded inside the technical systems, because I then don't trust the people inside the organization to fully manage this complexity.
What I want to hear is that there is just enough complexity and power in the technical systems AND at least enough intelligence in the human systems to manage this complexity and power properly. And I expect the enterprise architect to be responsible for maintaining this proper balance.
Monday, December 13, 2010
Can Single Source of Truth work?
My first observation is that even if we take this success story at face value, it doesn't tell us much about the possibilities of SSOT in an environment such as the UK NHS that is several orders of magnitude more complicated/complex. I'm guessing the Children’s Cancer Hospital Egypt 57357 (CCHE) doesn't have as many different types of "truth" to manage as the NHS.
- one type of patient (children)
- one type of condition (cancer)
- a single building
My third observation is that single-source of truth may be a bureaucratic fantasy, but responsible doctors will always strive to get best-truth rather than sole-truth. People in bureaucratic organizations don't always stick to the formal channels, and often have alternative ways of finding out what they need to know. So perhaps the Egyptian doctors at CCHE have managed to preserve alternative sources of information, and the "single source of truth" is merely a bureaucratic illusion.
See my previous post What's Wrong with the Single Source of Truth?
Wednesday, September 15, 2010
The Future of Enterprise Architecture
Enterprise architects are mostly pretty skilled at identifying duplication, inconsistency and lack of interoperability across complex systems of systems, which they regard as fragmentation and waste. As a consequence, they
In pursuing this mission, they find themselves opposed to narrow or short-term thinking from various stakeholders - line managers wanting quick and cheap solutions to local problems, project managers or external contractors trying to meet tight project goals, infrastructure managers wanting to protect established assets and arrangements. A committed enterprise architect will therefore get into a series of battles inside the organization, defending the principles of simplification and unification. (An uncommitted enterprise architect will withdraw from these battles and devote energy to producing abstract plans that will be ignored by everyone else - but it's difficult to see any value whatsoever in this kind of disengagement.)
The primary weapon in these battles is the Plan - an abstract blueprint that represents an idealized TO-BE architecture. This Plan needs to embody the principles of simplification and unification, not only because that's the mission, but also because that's the only way it's going to make sense to senior management, without whose support the battles cannot be won. The Plan is therefore not just a weapon but a story, perhaps even a myth.
But here's the problem: real business is inherently complex. Therefore battles against complexity (not always, but often enough to cause concern) are ultimately battles against the business. Or worse - against the customers of the business.
The mission to simplify and unify has undoubtedly had some successes, but it has also produced some massive failures - projects that have gone way over budget without coming anywhere close to delivering the promised business benefits. Some of the best-known examples can be found in the public sector because these are open to public scrutiny, but the private sector is not immune to this kind of failure. In the UK this week, the National Programme for IT in the health service has been finally put out of its misery, following cancellation of a number of other projects that were supposed to deliver both improved outcomes and massive cost savings.
To the extent that projects like this appear to be following an old-style enterprise architecture agenda, these failures serve to undermine the credibility of enterprise architecture in the eyes of many stakeholders, and should pose a series of hard questions for enterprise architecture practice.
Of course enterprise architects could try to attribute all such failures to errors of execution rather than errors of intention or planning (in other words, the vision was okay, the overall plan was okay, but the project managers screwed up). Alternatively, they could try to find fault with each of the failed plans, identifying ways in which these plans disobeyed some of the core principles of enterprise architecture.
But I believe these examples force us to do something more fundamental - to rethink the mission of enterprise architecture. If the mission to simplify and unify conflicts with the mission to "align" (whatever that means) with the business, then perhaps enterprise architects need to find a new mission.
Simplification and unification is merely a strategy in the management of complexity. As I see it the enterprise architect's job is not to manage-away the complexity of business or IT, but to find ways to make the complexity more manageable.
More manageable for whom? Just about anybody. Business line managers need to be able to manage the complexity of their business operation, and the top management team needs to be able to manage the complexity of the whole organization. Meanwhile, business support managers (such as IT and HR) need to be able to manage the complexity of their own domain as well as providing specialist support for the rest of the organization.
Thus the enterprise architect should be helping the CIO answer the following two questions.
- How can I manage the complexity of IT operations, systems and services?
- How can IT contribute to managing the complexity of the rest of the organization (including other support functions)?
- How can I manage the complexity of people management?
- How can I help develop the capability of the whole organization to collectively manage complexity?
The IT department can support the management of complexity across the organization by providing a set of useful tools, including business intelligence tools, decision support tools, social networking tools and so on. (The provision of management information has always been a major responsibility of the IT department, and this cannot be done properly without understanding how and why management information is used.) The HR department has a different set of tools, such as training programmes, incentive packages, career paths, and so on. The enterprise architect's role here is not to manage the detailed implementation of all these different tools, but to understand the overall structural requirements and implications.
Architecture is basically about structure. Business is facing some increasing complex structural problems, and the primary task for enterprise architects is to tackle these structural problems. Tackle, not solve. The collective ability of the whole organization to manage the essential complexity of demand, from customers and the rest of the environment, depends strongly (although not solely) on the emergent structure of the organization and its systems, and this is where the enterprise architect's attention should be focused.
Related posts: Who architects the organization? (September 2010), Complexity is not a problem (February 2013)
Updated 16 January 2013. Links added 12 September 2021
Thursday, June 10, 2010
Ecosystem SOA 2
When I started writing about SOA and the service-based business over ten years ago, I defined two "cuts" across the service ecosystem. One cut separates inside from outside, while the other cut separates supply from demand.
(This diagram was included in my 2001 book on the Component-Based Business, and frequently referenced in my work for the CBDI Forum. For a brief extract from the book, see my Slideshare presentation on the Service Ecosystem.)
The inside/outside cut is sometimes called encapsulation. It decouples the external behaviour of a service from its internal implementation, and can be described in terms of knowledge - the outside has limited knowledge of the inside, and vice versa. (The cut is also sometimes called transparency - for example location transparency, which means that external viewers can't see where something is located.)
The supply/demand cut is about delegation, and can be described in terms of responsibility. Getting these two cuts right may yield economics of scale and scope; and the business case for SOA as a development paradigm is often formulated in terms of reusing and repurposing shared services.
For relatively small and simple SOA projects, it may be feasible to collapse the difference between these two cuts, and treat them as equivalent. (The inside/outside relationship and the supply/demand relationship are sometimes both described as "contracts", although they are clearly not the same kind of contract.) However, enterprise-scale SOA requires a proper articulation of both cuts: confusing them can result in suboptimal if not seriously dysfunctional governance and procurement. Many people in the SOA world still fail to understand the conceptual importance of these cuts, and this may help to explain why some organizations have had limited success with enterprise-scale SOA.
Going beyond enterprise SOA as it is generally understood, there is a third cut separating two views of a system: the system-as-designed (whose structure and behaviour and rules can perhaps be expressed in some formal syntax such as UML, BPMN or ArchiMate) and the system-in-use (whose actual performance is embedded/situated in a particular social or business context). This cut is critical for technology change management, because of the extent to which the designed system underdetermines the pragmatics of use. I have been talking about this cut for over twenty years, but only more recently working out how to articulate this cut in composition with the other two cuts.
One important reason for looking at the pragmatics of use is to understand the dimensions of agility. In many settings, we can see a complex array of systems and services forming a business platform, supporting a range of business activities. If no agility is required in the business, then it may not matter if the platform is inflexible, forcing the business activities to be carried out in a single standardized manner. But if we assume that agility is a critical requirement, then we need to understand how the flexibility of the platform supports the requisite variety of the business.
More generally, understanding the pragmatics of use leads to the recognition of a third kind of economic value alongside the economics of scale and the economics of scope: the economics of alignment. The value of a given system-of-systems depends on how it is used to deliver real (joined-up) business outcomes, across the full range of business demands. (I'm afraid I get impatient with people talking glibly and simplistically about business/IT alignment, without paying attention to the underlying complexity of this relationship.)
Understanding these three cuts (and analysing their implications) is critical to understanding and managing a whole range of complex systems problems - not just SOA and related technologies, not even just software architecture, but any large and complex sociotechnical systems (or systems-of-systems). If the three cuts are not understood, the people in charge of these systems tend not to ask the right questions. Questions of pragmatics are reduced to questions of platform design; while questions of the cost-justification and adoption of the platform are reduced to a simple top-down model of business value. Meanwhile the underlying business complexity (requisite variety) will be either misplaced (e.g. buried in the platform) or suppressed (e.g. constrained by the platform).
So there are three challenges I face as a consultant, attempting to tackle this kind of complex problem. The first challenge is to open up a new way of formulating the presenting problem, based on the three cuts. The second challenge is to introduce systematic techniques for analysing the problem and visualizing the key points. And the third challenge is to identify and support any organizational change that may be needed.
With thanks to Philip Boxer and Bernie Cohen. For a different formulation of the three cuts, together with a detailed example, see their new paper "Why Critical Systems Need Help to Evolve" Computer, vol. 43, no. 5, pp. 56-63, May 2010, doi:10.1109/MC.2010.150. See also Philip Boxer, When is a stratification not a universal hierarchy? (January 30th, 2007)
Related post Ecosystem SOA (October 2009)
Saturday, November 28, 2009
Complexity and Power
Obviously we have to be clear as to what counts as an element (for example functions), and what counts as a connection. Using the SIP lens, it is possible to see how certain architectural styles (including those popular in the SOA world, such as hub-and-spoke or layered) only deliver simplicity (and the benefits of simplicity) if we can assume that only certain kinds of connection are significant. Roger's view is that this assumption is unwarranted and invalid.
In general, the so-called functional requirements are associated with the elements and the logical connections between them. In my view, architects also need to pay attention to the nature of the connections (coupling) because these will have important consequences on the structure and behaviour of the system as a whole. For example, synchronous versus asynchronous. At present, Roger's complexity calculations don't differentiate between different kinds of connection, so it would be interesting to investigate the costs and risks associated with different kinds of connections, to see how much difference it could make.
Roger's primary interest is in IT systems, but the same principles would appear to apply to processes and organizations. If you are running a factory, you have an architectural choice about the connection between say the moulding shop and the paint shop. With an asynchronous flow you have two loosely coupled operations separated by a buffer of work-in-progress; with a synchronous flow you have two tightly-coupled operations connected on a just-in-time basis. The former is a lot easier to manage, but it has an overhead in terms of inventory cost, storage cost, increased elapsed time, slower response to changes in demand, and so on. The latter may be more efficient under certain conditions, but it can be more volatile and the impact is much greater when something goes wrong anywhere in the process.
Intuitively, there seems to be a difference in complexity between these two solutions. The first is simpler, because the connection between the two systems is weaker; the second is more complex. With greater complexity comes greater power but also greater risk. Surely this is exactly the kind of architectural trade-off that enterprise architects should be qualified to consider. Roger's SIP methodology does give the architect a very simple lens to try and understand system-of-system complexity. Not everyone agrees with Roger's definition of complexity, and we can find some radically different notions of complexity for example in the Cynefin world, but at least Roger is raising some important issues. The EA world certainly needs to pay a lot more attention to questions like these.
Tuesday, December 16, 2008
Broken Business Models
In many different industry sectors, commentators are calling the business models broken.
- Automobiles: Automotive News
- Banking and Finance: Will Hutton, Seeking Alpha, MarketWatch
- Computer Games: Kotaku, Bruce on Games, GigaOm, Broken Toys
- Electronics and EDA (Electronic Design Automation): Sramana Mitra, ZDNet
- Governments: USA Today
- Mail Order: F. Curtis Barry
- Newspapers: Elias Bizannes, Ryan Sholin via Virtual Economics, Louis Gray
- Open Source Software: Business Week
- Printing: EE Times
- Retail: Marketing Week, Lean UK (pdf)
- Venture Capital: VentureBeat
- All Of The Above: Robin Bloor
Firstly, we don't have to take all of this at face value. I don't know of any sectors that are entirely unaffected by the economic downturn, but struggling isn't the same as broken. Broken doesn't mean having a bad few months, laying some people off, being extra careful (or worried) until economic conditions improve. Broken means fundamentally and irretrievably non-viable. It means that the old way you were making money has gone away and may never come back.
On the radio the other morning there was a discussion of the troubles at the Chicago Tribune. Someone was talking about "the business model" of the great traditional newspapers. Clearly this is Business Model (B) in my terms.
One of the things this kind of business model does is describe where and how value is created - for example whether you are using newspaper sales or advertising revenue to fund high-quality journalism. This is not a value chain in the traditional sense. Capability X generates the funds to support capability Y, but there is strict decoupling between X and Y, so that for example we don't allow advertisers or retail newsagents to influence the news-gathering agenda. There are lots of business models in which one activity subsidizes another one. (Doesn't that mean that enterprise architects need diagrams to show the generation and consumption of funds?)
This is not the only kind of broken business model. In some cases, a business model has long been cracked and it was only a matter of time before things fell apart. For example, people in the telecoms world have known for twenty years that the cost of a telephone call was on a diminishing curve, and that telecoms companies needed to find some other source of revenue.
In some industries, people are putting the blame on complexity. Sometimes even sophisticated investors were unable to understand how their money was being invested. In today's news we discover that a major hedge fund, run by Bernard Madoff, was being run as some kind of Ponzi scheme [BBC News, 16 December 2008, see also Bill Burnham on Madoff Madness]
So what kinds of business model are not broken? Chastened, the surviving hedge fund operators are talking about a return to simplicity and transparency. Complex enterprises based on fragile cross-subsidy may need to be unbundled. Perhaps we shall see a return to business models that we can explain to our children.
After a forest fire, it is the simple pine tree that reappears first, taking advantage of a clear space for quick growth before the more complex trees can get reestablished. Perhaps we may look forward to several years of straight and upright business models?
This is one of a series of posts on Broken Business Models
See also Two Kinds of Business Model (December 2008)
Wednesday, November 19, 2008
Lego Bricks and Outsourcing
Seems the Danes were losing money, so back in 2006 they decided to outsource production to Flextronics, an electronics company in the Czech Republic [Lego Press Release, June 2006].
Now here's the clever bit. Flextronics would do the plastic moulding, but Lego would retain the manufacture of the electronics.
Kevin Meyer is sarcastic. "Sure, why not? Have an electronics manufacturer make the plastics, and a plastics manufacturer keep on making the electronics. That's bound to succeed, right?"
The Fast Company wrote a case study about Mass Customization called Brick by Brick - Lego's New Building Blocks. And Booz Allen Hamilton wrote a case study called "Rebuilding Lego Brick By Brick", (strategy+business August 2007 via Slow Movement).
The Booz Allen report had sub-headings like "Approaching the Value Chain Holistically" At that time it looked like the strategy was working.
But now Lego and Flextronics have parted company. “It has become increasingly obvious to the two parties that it would be more optimal for the LEGO Group to manage its global manufacturing set up."
Kevin Meyer provides an excellent history, and asks Did Lego Learn a Lesson? Did You?
I read the case as a story of complexity. The product had got more and more complex, and so had the production process. It isn't obvious how an apparently simple outsourcing deal does anything to address this complexity.
By the way, if you search for Lego and complexity, you will also find some wireless network algorithm called Locally Enabled Global Optimization, and a discrete event simulation technique called Listener Event Graph Objects. Presumably these have nothing to do with the Danish toy company. But sometimes people want to reuse a familiar set of initials.
Monday, October 06, 2008
Defense Procurement
"When it comes to procurement, for the better part of five decades, the trend has gone towards lower numbers as technology gains made each system more capable. In recent years these platforms have grown ever more baroque, ever more costly, are taking longer to build, and are being fielded in ever dwindling quantities."The US Department of Defense appears to have a very clear understanding of the opportunities for Service-Oriented Architecture in enhancing the procurement process and improving the economics of governance. In July 2008, the SOA Acquisition Working Group of the AFEI produced a useful set of Industry Recommendations for DoD Acquisition of Information Services and SOA Systems.
"The issue then becomes how we build this kind of innovative thinking and flexibility into our rigid procurement processes here at home. The key is to make sure that the strategy and risk assessment drives the procurement, rather than the other way around."
One point that comes across very clearly from this paper is the opportunity to separate the provision of different layers within a layered architecture. So we may have one set of suppliers providing the underlying layers, and a different set of suppliers providing the upper layers.
The AFEI paper identifies three roles - Platform Providers and Commodity Infrastructure, Component Developers and Enterprise Managers - with what it calls Firewalls between them. (See Figure 3 in the AFEI paper.) These firewalls roughly correspond to the first two of what Philip Boxer calls the three asymmetries of demand. The third asymmetry (not explicitly shown in the AFEI paper) is between the Enterprise Managers and the Customers. (Including customers in the schema is essential if you want to move towards a risk-reward model of procurement, as the DoD evidently does, because it is the customers who ultimately generate the effects.)
Managing all this complexity almost certainly results in higher management cost than if you simply outsource the whole lot to a large system integrator, but it also shifts the management control (architectural governance) back to the acquiring organization.
So which kind of organizations would want to do this then? It's a question of complexity - the greater the level of complexity and volatility in your requirements, the greater the potential benefits from having finer-grained and stratified control over your procurement.
And what kind of organizations are capable of this fine-grained stratified control? Perhaps not too many at the moment, which is why we need a capability maturity model for SOA that provides a roadmap for organizations to develop an appropriate level of SOA capability.
Sources
AFEI SOA Acquisition Working Group, "Industry Recommendations for DoD Acquisition of Information Services and SOA Systems", July 2008 (register on the AFEI website for free download)
For explanation of the three asymmetries, see Richard Veryard & Philip Boxer, Metropolis and SOA Governance, Microsoft Architecture Journal, July 2005.
Thursday, October 02, 2008
More Flattery for the Finance Sector?
The judges have decided to award the overall prize to a company in the finance sector. According to the rubric, this is for a project producing cost-savings of around a million dollars.
Meanwhile, one of the runners-up, a company in the healthcare sector, produced cost-savings of around 6 billion dollars a year.
So why didn't the healthcare company win the overall contest? Were the judges sceptical about the benefits, or did the project did not conform to some technical purist notion of SOA? (If so, it should not have been awarded a runner-up prize either.) I wrote to one of the judges asking for the criteria to be published, and here they are: Winner snapshots and judging criteria (Oct 2008). The judging was based on a weighted scoring scheme, in which the financial case study ended up with the most points.
I am sure that the judges followed the criteria and the judging process in good faith, but I still can't see why the finance company won. There is a tendency within the SOA industry of having a rather distorted (rose-tinted) view of the finance sector. (See my earlier post IBM Flatters the Finance Sector.) Perhaps one outcome of the current Financial Catastrophe may be that the SOA industry will have to pay a bit more attention to sectors where service-oriented principles can be more usefully applied.
Which are those sectors? In my post Theory and Practice (August 2005) I suggested that the most interesting applications of SOA are going to be found towards the right of the following chart.
Simple Interoperability | Moderate Interoperability | Complex Interoperability |
Financial Services Retail SupplyChain Logistics (Basic) Travel | Automotive eGovernment SupplyChain Logistics (Advanced) Telecoms | Aerospace and Defence Consumer Electronics Healthcare and Pharmaceuticals |
I haven't seen anything in the last three years to alter this chart significantly.
Updated 26 March 2013
Wednesday, June 11, 2008
Complexity-Based Pricing
The term "Complexity as a Weapon" only occurs a few times on the Internet. Ten years ago, Scott Bradner wrote a column for ComputerWorld called Complexity as a Weapon. He was referring mainly to large software organizations resisting simple standards, as a tactic for achieving an advantage over smaller organizations with fewer resources. In a discussion with Russell Ackoff on Why few organizations adopt systems thinking, Jim Leemann talks about complexity as a tactic for resisting any kind of positive change. And an organization called Focus on the Global South accused oil companies of using complex accounting solely in order to reduce tax liability (Destroy and Profit, pdf).
But Gordon's Notes discusses Complexity-as-a-Weapon in the context of service pricing, where complexity is designed to prevent service consumers optimizing their service costs (so-called ninja shopping), as I indicated in my review of the Support Economy.
"Providers retaliate with an increasingly impenetrable array of extras, excesses and surcharges, which aim to recoup profit margins while making accurate price comparison near-impossible."
In Manufactured Uncertainty, the Farnam Street blog tracks how the practice of unbundling (for the purpose of making it harder for consumers to compare prices) has spread from mobile phone companies to airlines.
Excessive complexity of this kind is ultimately self-defeating: it adds to everyone's transaction costs, and erodes the value and viability of the service itself.
See also Billing Blunders
Monday, April 30, 2007
Economics of Style
"Time and again we recognize that a scope based approach to maturity is the most useful determinant of maturity that decomposes well to each of the stream views because services are increasingly useful as they are used more widely, and conversely expanding the SOA scope beyond traditional organizational boundaries is without question the hardest thing to achieve."In a fascinating recent paper, Chunglin Kwa identifies two contrasting perspectives on complexity, which he calls "Romantic" and "Baroque". From the Romantic perspective, complexity increases with scope (emerging from the increasingly stormy and ambitious interaction between diverse elements); from the Baroque perspective, complexity increases with detail and specificity (emerging from the increasingly precise delineation of fractal elements).
David is quite right to indicate the difficulty of expanding the SOA scope beyond traditional organizational boundaries. I have spent a lot of time talking about this myself. But I don't think this is the only kind of difficulty/complexity we need to address, and I should not like just to equate maturity with ambition.
References
Chunglin Kwa, ‘Romantic and Baroque Conceptions of Complex Wholes in the Sciences’, in John Law and Annemarie Mol (eds), Complexities: Social Studies of Knowledge Practices (Durham, NC and London: Duke University Press, 2002), pp. 23–54
John Law, And if the Global Were Small and Non-Coherent? Method, Complexity and the Baroque (pdf), Centre for Science Studies, Lancaster University, 2003.
Thursday, March 16, 2006
Grey Matter
So I was interested to hear a report on the radio today about recent studies of the human brain. As I understand it, the brain is divided into grey matter (which does the basic thinking) and white matter (which wires everything together). So perhaps we can very crudely equate the grey matter with services, and the white matter with the orchestration of the services.
Now here's the interesting bit - the ratio of white matter to grey matter in different people. Pathological liars have significantly more white matter than other people. (It seems that skilled falsehood requires greater orchestration. It is not clear whether people lie because their brains enable them to lie, or whether some brains develop in particular ways because their owners are exercising the capacity to lie.)
Meanwhile, autistic people tend to have a much lower amount of white matter than other people. There are two characteristics of autism that may be relevant here: difficulty in interoperability, and difficulty with anything other than the literal truth.
Has all this got anything to do with SOA and BPEL? You decide.
Source: BBC Radio 4, Leading Edge, March 16th, 2006
(full programme available online for a week after broadcast).
Technorati Tags: SOA service-oriented
Monday, January 23, 2006
Context-Aware Services 3
My view of context-aware services is that any aspect of a service may be differentiated according to context. The service I get from a supermarket is fairly simple, so perhaps there isn't much scope for variation. But each customer may get a different set of special offers, and this can be generated dynamically, according to the contents of the shopping basket or the path through the store. A customer with a known taste for raw eggs, or a history of returning stale products, may get a warning that a selected product is close to expiry. But is that all? If all we're talking about is commodity advertising, then there is possibly no difference between selling soap powder and lending books.
But I'm not particularly interested in libraries selling me nappies, because I don't think that's their job. I'm interested in ways that the library can serve me better (not just target me better) through an awareness of my context. This is intrinsic differentiation - in other words, differentiation that is relevant to the service in hand.
For example, my son has just done a school project on a mathematician of his choice. He chose Florence Nightingale (the inventor of the pie chart). If he had gone into a library for help, would he have been offered a scholarly history of the Crimean War or an academic thesis about mortality statistics and their graphical representation?
Can a computerized system offer anything approaching the sensitivity and common sense that we still expect from a human librarian? At one extreme, there are standardized search systems, which will give you exactly the same answer whether you are a schoolchild or a BBC researcher or an LSE postgraduate. At the other extreme, there may be inflexible classification systems that assume that a child is only interested in reading books that are designated suitable for that age group.
One of the most important aspects of context is how the service fits into what the consumer (the customer, the library reader) is trying to do. Do I have an essay or article or thesis to write, and when is it due? Am I reading Nietzsche because I am learning German, or because I am learning existentialism (or both)? Am I reading Bede because I am studying history or historians (or both)? Does it make sense to read Locke without also reading Hume? What stage of my learning have I reached? Do I need an edition with glossary, with scholarly notes, with English translation facing? What is my preferred style of learning, my preferred style of researching a topic? Surely questions like these are relevant to improving the service offered by a library to a particular reader?
Ultimately, context-awareness takes us down a path of embracing user diversity. Not just user semantics, but user pragmatics. How much of the reader's context can the library possibly deal with, and what other service providers might the library collaborate with? There are some seriously complex models here.
I see context-awareness as a very significant challenge - introducing modes of complexity that most organizations have never dealt with before - but with the potential reward of offering massive improvements to the experienced quality of service.
Thursday, September 29, 2005
Supply Chain Complexity
There are many sources of supply chain complexity implied in this piece, besides the complexity and continuing evolution of technical standards.
- a significant focus on ad hoc activity particularly with customers, where one off sales are common
- large numbers of partners with differing characteristics and capabilities
- high complexity in the standards
- rate of change of partners’ technical solutions
- multiple versions of standards in the value chain
- geopolitical issues e.g. local legal, native language and logistical issues
- myriad of technologies and standards
What is needed here is to analyse the dimensions of complexity in detail (obviously this analysis won't be identical for different companies) and produce a service geometry with a clean but flexible division between core and non-core.
Technorati Tags: B2B complexity outsourcing supply chain technology adoption