Ecosystem SOA 2

What are the problems of large complex sociotechnical systems? How far do SOA and enterprise architecture help to address this problem space, and what else might we need?


When I started writing about SOA and the service-based business over ten years ago, I defined two "cuts" across the service ecosystem. One cut separates inside from outside, while the other cut separates supply from demand.

(This diagram was included in my 2001 book on the Component-Based Business, and frequently referenced in my work for the CBDI Forum. For a brief extract from the book, see my Slideshare presentation on the Service Ecosystem.)

The inside/outside cut is sometimes called encapsulation. It decouples the external behaviour of a service from its internal implementation, and can be described in terms of knowledge - the outside has limited knowledge of the inside, and vice versa. (The cut is also sometimes called transparency - for example location transparency, which means that external viewers can't see where something is located.)

The supply/demand cut is about delegation, and can be described in terms of responsibility. Getting these two cuts right may yield economics of scale and scope; and the business case for SOA as a development paradigm is often formulated in terms of reusing and repurposing shared services.

For relatively small and simple SOA projects, it may be feasible to collapse the difference between these two cuts, and treat them as equivalent. (The inside/outside relationship and the supply/demand relationship are sometimes both described as "contracts", although they are clearly not the same kind of contract.) However, enterprise-scale SOA requires a proper articulation of both cuts: confusing them can result in suboptimal if not seriously dysfunctional governance and procurement. Many people in the SOA world still fail to understand the conceptual importance of these cuts, and this may help to explain why some organizations have had limited success with enterprise-scale SOA.

Going beyond enterprise SOA as it is generally understood, there is a third cut separating two views of a system: the system-as-designed (whose structure and behaviour and rules can perhaps be expressed in some formal syntax such as UML, BPMN or ArchiMate) and the system-in-use (whose actual performance is embedded/situated in a particular social or business context). This cut is critical for technology change management, because of the extent to which the designed system underdetermines the pragmatics of use. I have been talking about this cut for over twenty years, but only more recently working out how to articulate this cut in composition with the other two cuts.

One important reason for looking at the pragmatics of use is to understand the dimensions of agility. In many settings, we can see a complex array of systems and services forming a business platform, supporting a range of business activities. If no agility is required in the business, then it may not matter if the platform is inflexible, forcing the business activities to be carried out in a single standardized manner. But if we assume that agility is a critical requirement, then we need to understand how the flexibility of the platform supports the requisite variety of the business.

More generally, understanding the pragmatics of use leads to the recognition of a third kind of economic value alongside the economics of scale and the economics of scope: the economics of alignment. The value of a given system-of-systems depends on how it is used to deliver real (joined-up) business outcomes, across the full range of business demands. (I'm afraid I get impatient with people talking glibly and simplistically about business/IT alignment, without paying attention to the underlying complexity of this relationship.)

Understanding these three cuts (and analysing their implications) is critical to understanding and managing a whole range of complex systems problems - not just SOA and related technologies, not even just software architecture, but any large and complex sociotechnical systems (or systems-of-systems). If the three cuts are not understood, the people in charge of these systems tend not to ask the right questions. Questions of pragmatics are reduced to questions of platform design; while questions of the cost-justification and adoption of the platform are reduced to a simple top-down model of business value. Meanwhile the underlying business complexity (requisite variety) will be either misplaced (e.g. buried in the platform) or suppressed (e.g. constrained by the platform).

So there are three challenges I face as a consultant, attempting to tackle this kind of complex problem. The first challenge is to open up a new way of formulating the presenting problem, based on the three cuts. The second challenge is to introduce systematic techniques for analysing the problem and visualizing the key points. And the third challenge is to identify and support any organizational change that may be needed.

---

With thanks to Philip Boxer and Bernie Cohen. For a different formulation of the three cuts, together with a detailed example, see their new paper "Why Critical Systems Need Help to Evolve" Computer, vol. 43, no. 5, pp. 56-63, May 2010, doi:10.1109/MC.2010.150. See also Philip Boxer, When is a stratification not a universal hierarchy? (January 30th, 2007)


Related post Ecosystem SOA (October 2009)

A business case for collaborative action

#SOA #insurance #govit @mcgoverntheory asked if anyone was interested in learning how insurance carriers partnered with state GOV to stand up SOA for online coverage verification?

Where is the business case for this initiative? There is certainly a demand, going back to this 2001 article on the Growing DMV Reporting Headache, and James points to a March 2004 document on the IICMVA website called Online Insurance Verification Using Web services to verify auto insurance coverage (pdf).

The IICMVA document outlines the problem - costs incurred by uninsured motorists, complicated and varied procedures for verifying insurance between different states - and so it makes clear that a web service solution might be a good idea. But a business case has to go further than this.

Firstly it must show that the benefits outweigh the costs (and risks). In a collaborative scheme, it isn't always obvious who is going to pay for the scheme, so there may be alternative options for funding the development followed by some kind of charge-back during operations. Each party that is investing in the scheme will presumably need a business case to justify its own investment. But if there aren't sufficient benefits to cover everyone's costs, then the scheme just isn't viable at all.

Searching the internet, I found a presentation by Loren McGlade and Jon Neiditz from the ACORD/LOMA Insurance Systems Forum (May 2009) mcglade_neiditz.pdf; mcglade_neiditz.mov; mcglade_neiditz.mp3. This presentation contains a lot more than the March 2004 document in terms of quantifying the potential benefits and cost-savings of the scheme.

And secondly, in a collaborative scheme like this, the viability depends on the number of people who join. In this case, how many insurance companies and how many states have to participate, in order for the scheme to deliver the benefits. Given that the benefits are largely dealing with the complication caused by the differences in regulation between states, this only makes sense if multiple states are involved. But how many?

I've done a number of business cases of this kind, where the viability of a multi-agent scheme depends critically on the rate of adoption, so I'd be very interested to see how this aspect of the business case for the IICMVA scheme is worked out.

More generally, there is a widespread belief in the potential benefits and cost savings of shared service provision, especially in the public sector. However, there is often a lack of realism in how these shared services are to be implemented. If the shared services are too inflexible, then people won't use them without some degree of coercion; thus the costs and risks are higher than expected, and the benefits lower. I'm looking at a number of situations in the UK where public sector agencies are being encouraged to use nationally provided services, and it's an extremely complex task to work out how these services can be fitted asymmetrically into a joined-up solution that delivers the best outcomes. (Please contact me if you are facing this kind of task.)

Economics of agility

@andygreenawalt says he hasn't seen much on the economics of agility. "Open source breaks $ vs security battle. Investment impregnation kills agility."

Andy's comment arose from a discussion of agile security. See my article Agile Security for SOA (CBDI Journal, June 2005).

More generally, the economics of agility is critical to the business case for SOA, and a few of us have sometimes called it the SOA Sweet Spot (March 2007). See my article on the Agile Business Process (CBDI Journal, July-August 2007)

Looking around, I also found some work by Marc Rix called Bottom-Line SOA: The Economics of Agility (April 2007). See comments by Alastair Bathgate, Bill Barr and Gary Smith, with Marc's response.

And back in 2004, Stuart Robbins wrote something on the economics of agility for grid management (pdf).

But compared to the huge amount about the economics of scale (sharing, reuse), that's not much is it?

Service-Oriented CashFlow

In difficult economic times, the most important business measure for many firms is not profit or revenue growth or cost-saving but cashflow. Which means that the business case for SOA shifts away from relatively boring arguments about cost-saving (economics of scale), and strategic but highly imprecise arguments about flexibility (economics of scope), towards careful comparison of the dynamic economic viability of different business models (economics of governance alignment).

There aren't yet many SOA case studies that talk about cashflow. When I search for "service-oriented" and "cashflow", I keep finding references to a healthcare distribution company Owens & Minor, based in Mechanicsville VA, which started an 4-year SOA project sometime in 2005. (Speed Wins at Owens & Minor, Feb 2005) So nearly finished yet, chaps?

As I've explained in previous posts, the Pay-As-You-Go model has two advantages for the service consumer - better cashflow (because there is less financial latency between expenditure and return) and reduced risk (because you only spend as much as you need). As economic conditions become more volatile, and credit becomes more expensive, both of these advantages become more valuable.

Valuable for the consumer that is. The SaaS vendors (cheerleader Phil Wainewright) quite rightly see this as an opportunity for SaaS. But of course the challenge for the SaaS vendors is how to provide these benefits without themselves incurring excess cost and risk.

Billing and credit control are obviously key elements of the SaaS infrastructure, and some of Phil's clients offer useful services in this space. So are we moving towards a stratified SaaS world: (service-as-a-service)-as-a-service? And what are the economics of that?

---

Phil Wainewright on CashFlow: A Few Financial Home Truths. For mechanisms for handling billing see Phil Wainewright on Easing the SaaS-to-cash cycle and Marco Seiriö (RuleCore) on More CEP SaaS Pain.

Note: for various reasons, we now prefer the term Economics of Alignment. 

Updated 25 October 2013

Case Studies

There is a significant demand for technology case studies, from would-be adopters and practitioners of specific technologies. There is also a considerable supply of technology case studies, mostly from vendors.

But I don't see the supply meeting the demand. There seems to be a gap between what people want to know and what people are willing to publish.

Most so-called case studies take this form:

Gringotts Bank (NYSE:GOBL) needed a fast yet secure customer response across multiple legacy vaults so they installed WebHogz Enterprise Edition Version 6.66. Gringott's Chief Architect Bill Weasley said Our productivity has whizzed up by 31.4%. WebHogz CTO Harry Potter said This application clearly demonstrates the magical superiority of our product over muggle alternatives.

I regard these as press releases rather than genuine case studies. What purpose can they possibly serve apart from name-dropping? We are told that a respectable large organization is happy with the product, but we are given few if any details on how the product was used. Impressive numbers may be quoted, but we have no idea how they are measured or what to compare them with.

One thing we really want to know is about practical lessons - difficulties and pitfalls. An alternative perspective perhaps?

However, Dr Aco Malfoy, a security consultant with Arthur T. Riddle, voiced some concerns about the WebHogz solution. A number of our clients have had problems with the stability and performance of the product. Because of these concerns we have advised Gringotts Bank to install a separate firewall system and invest in additional dragonware, which will add considerably to the overall cost of the WebHogz project.

That's not much better. It may identify some areas of concern, but still doesn't give us a rounded evaluation of a project, successful or otherwise.

And academic studies often aren't much use either.

Professor Hermione Granger, a fellow of Halloween College at Hogsfjord University, has completed a three-year comparative study of vaulting technologies in collaboration with a number of industrial partners including WebHogz Labs. Our preliminary findings, based on an action research paradigm, do seem really quite promising, but we obviously need much more research funding before we can produce more reliable and detailed figures.

What we need is a lot more detailed, warts-and-all case studies, in plain English. But there are never enough organizations willing to expose themselves to independent public scrutiny. Any organizations willing to volunteer?



Update: this blogpost originally referred to SOA case studies. I have removed the specific references to SOA, as I believe my argument applies to any technology or product.

Selling SOA by the pound

Steve Jones suggests that SOA needs marketing. Unfortunately, he only mentions one-way communication - from the SOA programme to the business. True marketing involves two-way communication [see post on Marketing] - this entails listening and responding to what the business wants. And that means what a particular business specifically wants, not just vague generalizations like "adaptability" and "customer satisfaction" [see Seth Godin on Business Cliches].

There is some question about the extent to which it is even possible to sell SOA to the business - whether it makes sense to measure the ROI of SOA. Those challenging this include Randy Heffner (via James McGuire) and John Devadoss.

But there is a more fundamental question - does the business need to hear about SOA at all? As Nick Malik asks: When we talk to the business, should we sell SOA? His argument is that good SOA is essentially free, so you shouldn't have to sell it separately. This is what he says in an earlier post, sarcastically titled I want one of those SOA thingamajiggies:

'I’ve decided that the best way to “Sell SOA” to the business is not to sell SOA to the business. Let’s talk about the asthetics, the features, the speed and reliability, the automation of their business processes to allow them to focus and innovate where it counts. Let’s not talk about standard parts.'

Lorraine Lawson disagrees. Summarizing a recent discussion on LinkedIn, she concludes:

'Of course, you shouldn’t start out the conversation by talking about the technical aspects of SOA. Business value is definitely the place to begin. But be prepared to discuss services and explain why SOA is better that EAI or standard development practices — my guess is, there will be questions.'

The claim that SOA is free (under certain initial conditions) echoes the claims of some quality management gurus - notably Philip Crosby - who claimed that quality was free (again under certain initial conditions).

But let's suppose, for the sake of argument, that good quality management, or good SOA, really is free, in the sense that it doesn't add anything to the financial costs associated with certain activities (manufacturing, IT development, whatever). Even if this is true, it doesn't mean we don't have to sell SOA to the business. SOA makes demands on the business - perhaps not financial or technological ones, but demands associated with architecture and governance. For SOA to be successful (let alone "free"), the business may need to consider certain structural opportunities, and to engage with IT requirements in new ways. Even if we don't sell "SOA" to the business, we should still be able to sell "agile enterprise" and the "economics of governance" - not just as vague generalizations but specifically grounded in the needs of a particular business. Because without that connection to the business, SOA will be unable to deliver anything of real value.

SOA Sweet Spot

In a post entitled Your SOA is JABOWS, Microsoft's Nick Malik identifies what he calls the SOA Sweet Spot, which represents the intersection of two areas.

1. Typically, the greatest benefits of IT come from automating processes that execute often. It may be hard to cost-justify an n-person-year project to automate some process, if that process only occurs once a year, or on an exceptional basis.
2. Typically the greatest benefits of SOA come from supporting automation in areas that change frequently. As I said in my fourth post on BPM and SOA, the business case for SOA typically becomes stronger as the volatility increases.

Nick expresses this as a two-by-two matrix with Frequency of Occurrence along one axis and Frequency of Change along the other axis. The sweet spot is then in the top right quadrant.

Someone called Malcolm Anderson posted a comment to Nick's blog, challenging the difference between Frequency of Occurrence and Frequency of Change. Nick replied with a simple retail illustration.

Of course, if you choose to regard everything as undifferentiated process, then the two dimensions of Nick's matrix possibly collapse into a single dimension. This may be the point behind Malcolm's comment. Nick's matrix depends on an articulation of two different types of variation at two different logical levels - equivalent to the item and the batch (manufacturing) or the phenotype and the genotype (biological evolution). SOA allows us to implement a stratified solution in which these two types of variation are decoupled - yielding both economics of scale (based on frequency of occurrence) and economics of scope (based on frequency of change). This is of course an architectural solution, one which demands true SOA rather than JBOWS.

JBOWS or JaBoWS?

By the way, the term JBOWS appeared in an article by Joe McKendrick: The Rise of the JBOWS Architecture (or Just a Bunch of Web Services) (September 2005). Bobby Wolf of IBM (who picked up the term via James Governor) prefers to call it JaBoWS. I happen to prefer to stick with the term JBOWS, if only because an Internet search for Jabows yields all sorts of other stuff I'm not interested in.

Technorati Tags: SOA service-oriented

BPM and SOA 4

An interesting discussion this week with some of the top BPM people at IBM, on the nature of the relationship between BPM and SOA.

One of the critical elements of this relationship is the way the business case for BPM is connected with the business case for SOA. One of the ways to get the business interested in SOA is to promise BPM-related benefits.

But why do we need SOA to deliver BPM benefits? It may be technically possible to deliver many of these BPM benefits without using full SOA. Indeed, there may be people who claim to have produced similar benefits in the past, before SOA technology was available. So the business case for SOA in this context reduces to a technical argument about the greater efficiency or flexibility of SOA in delivering BPM benefits.

If an organization has a broken business process, which requires a single one-off effort to fix, then the case for using SOA is based on a simple comparison of two alternative development projects. But if an organization has a volatile business process, which requires constant modification, then the case for using SOA is based on an expectation of frequent modification. We would normally expect the business case for SOA to become stronger as the volatility increases.

We can use a simple spreadsheet model to compare costs and timescales and risks between two technical alternatives. We can produce graphs that show how the cost-benefit curve shifts as we vary the assumptions about volatility.

But the spreadsheet models, useful though they are, only tell half the story. The critical element of the business case is the architectural dependency between BPM and SOA. How much does BPM need SOA? In order to get the spreadsheet to calculate the synergy between BPM and SOA, we need to be able to quantify the architectural dependency - pick a number that is greater than zero and less than one.

But where does the number come from? There are three possibilities.

1. Guesswork. If you don't know any better, one number is as good as another.
   
2. Statistics. If you have a large quantity of historical data, you may be able to demonstrate some degree of correlation between SOA expenditure and BPM expenditure.
   
3. Algebraic reasoning. Infer the level of synergy from the structural characteristics of the architecture.

The business case as a whole is extremely sensitive to this number, and we need to develop better ways of determining it.

Technorati Tags: BPM business process SOA service-oriented

Business Case for SOA 3

ZDnet blogger Joe McKendrick has uncovered an interesting difference of opinion between stakeholders in the US Government and Defense space about the viability of SOA, as described in the following posts:

• Defense doesn't rest with SOA (Nov 10)
• Study: the public sector ‘gets’ SOA, or at least ’shared services’ (Nov 15)

The US Defense Information Systems Agency (DISA) seems pretty committed to SOA. The problem apparently lies with some large US contractors, who seem to think that this threatens a profitable line of business selling software to the Department of Defense. David Bryan, a retired general who is now a vice-president at Northrop Grumman Defense Group, says that "Industry is struggling to determine a sound business case for the SOA approach". [source: FCW.com, Nov 8]

There are four things I want to say in response to this, which relate to the business case for SOA.

1. The Department of Defense evidently appreciates the potential of strategic-level or Enterprise SOA to reduce their IT costs and to improve the efficiencies and flexibility of their business. And if SOA enables the Department of Defense to get better value-for-money from its contractors, then this strengthens the business case for SOA within the Department of Defense. (DISA is clearly doing something right.)

2. If the Department of Defense insists on the SOA approach, then this will create a very strong business case for SOA within the ecosystem of defense contractors. (If you don't want the work, there's plenty others that do.)

3. SOA is not just a technological initiative, but carries significant changes to organization and governance, including procurement arrangements. Enterprise SOA is, in part, about commodity services - buying pre-existent capability - and minimizing the custom development.

4. There is a learning curve for defense contractors. To be successful in the enterprise SOA market, traditional integrators will have to change their business models from being dependent on big custom software development projects. But some are still loading their bids with software costs, according to DISA Procurement Director Evelyn Palma. IBM has already won a major supply contract under the new regime, and we might expect future contracts to go to the most SOA-literate contractors.

Business Case for SaaS

Gianpaolo Carraro (Microsoft) has produced an interesting diagram laying out the benefits of Software-as-a-Service (SaaS) to various actors.

SaaS stakeholder concerns (Gianpaolo Carraro)

This makes clear that different actors have different motivations to get involved in SaaS - and for that matter SOA. Each actor may have several parallel motivations, with different perceived urgency levels. In his blog, Gianpaolo discusses the typical short-term and longer-term interests for each of the types of enterprise he identifies. This is useful material, and helps us to see different adoption paths for SOA and SaaS within different organizations.

But there is a common pattern here as well. The potential benefits of SaaS indicated here are mostly architectural ones (Gianpaolo is an architect himself, after all) - changing the configuration of the stack, changing your own position within the stack - delivering not just economies of scale but also economies of scope and economies of governance.

So I'm not just interested in the content of Gianpaolo's picture, but the underlying process. This is the kind of business case that depends more on PowerPoint than on Excel. The trouble with this, of course, is that PowerPoint lacks rigour. Ideally, we need to develop better ways of producing architectural models that support reasoning about business benefits.

Business Case for SOA 2

Andrew McAfee recently posted The Case Against the Business Case., which indicates why it's wrong (or at least dangerous) to make a business case in technological terms alone.

"I’ve probably seen hundreds of business cases that identify the benefits of adopting one piece of IT or another, assign a dollar value to those benefits, then ascribe that entire amount to the technology alone when calculating its ROI. The first two steps of this process are at best estimates, and at worst pure speculation. The final step gives no credit and assigns no value to contemporaneous individual- and organization-level changes."

I agree with this absolutely. I remember listening to a presentation on a successful computing project (I forget the details now) which ensured the commercial survival of some dockyard in the Far East. The speaker mentioned, almost as an aside, that there was some minor organization change required as well. And I remember thinking that there could easily be another presentation going on right now in a conference room somewhere else, where the consultants responsible for the organization change were boasting of their success in ensuring the commercial survival of the dockyard, and mentioning the computing project (if at all) as a minor side-condition.

So we have an excellent return on investment, if we just compare the cost of the computing project with the profitability of the dockyard. Presumably we also have an excellent return of investment if we just compare the cost of the organizational change with the profitability of the dockyard. But does the business case still work if you include both? The challenge for business case is scoping - where do you draw the line of what costs and benefits to include. Whose costs and whose benefits? (IT is notorious for neglecting user costs.) And what types of costs and benefits do you even recognise?

This has always been a problem for business case. Here's an example from a paper I wrote on "Reasoning about Systems and their Properties" (September 2000).

"During the 1980 UK mining strike, both sides constructed a plausible argument relating to the productivity of the system: one side claimed that it was more cost-effective to close the pits, the other side claimed that it was more cost-effective to keep them open. Apparently small differences in the way the system was scoped and in the choice of time horizon can have a large effect ..."

Andrew McAfee argues that a business case should be formulated not in terms of absolute business value, but in terms of the business capability you are getting in return for a given IT expenditure. It then becomes a business decision whether (and how much) to invest in a given capability.

This makes a lot of sense in terms of agility and flexibility as well. I have long argued that we should regard adaptability as a special kind of capability: adapt-ability, the ability to adapt. We need to show these capabilities on our business capability models, not just the operational business-as-usual capabilities. And we need to become much more precise about the adaption capability that a given technology (such as SOA) should be able to deliver.

Business Case for SOA

With pre-SOA technologies, the business case is typically weakened by uncertainty. We have to factor in the possibility that the benefits will be less than expected, and the costs and timescales greater.

With SOA, the business case may be strengthened by uncertainty. SOA helps to protect the business and its systems against risk, and keeps more options open. The greater the uncertainty, the greater the value of options.

1. The classic way of constructing a business case is to estimate the costs and the benefits, estimate the timescale in which these costs and benefits will occur, and use this to calculate a return on investment (ROI). Timescale is important in investment decisions - costs and benefits are usually spread over an extended period. Earlier benefits are worth more than later benefits. A standard accounting technique for calculating ROI by aggregating costs and benefits over different times is known as discounted cash flow (DCF). Future cash flows are discounted even if they are 100% certain.
   
2. But in most situations there are various dimensions of uncertainty - about whether, how much, and when benefits will be achieved; and whether additional costs and delays may be incurred. So the business case needs to factor in an adjustment for risk, which can reduce the economic feasibility of a proposition. Higher uncertainty is usually reflected in a higher discount rate, and this reduces the value of future benefits, especially longer-term ones.
   
3. If we now add SOA into the equation, we may be able to identify differences to cost, timescale and benefit that are produced by SOA. SOA is then justified if the net effect is positive (in other words, if there is any additional cost from SOA, it is more than offset by faster or increased benefits, or reduced cost elsewhere). In some cases, SOA may convert a non-feasible proposition into a feasible one.
4. Finally, we may adjust the SOA increment for its effect on risk. In many cases, one of the most important elements of a business case for SOA is its positive contribution to risk management, and therefore feasibility. A well-design loosely-coupled architecture will be protected against events that might otherwise increase total lifetime costs or reduce benefits.

Technorati Tags: business case ROI SOA service-oriented