Semantic Coupling

In a recent post, Semantic Coupling, the Elephant in the SOA Room, Rocky Lhotka identified semantic coupling as one of the challenges of SOA. Udi Dahan agrees that semantic coupling is harder, but adds that in his view SOA is all about addressing this issue. Meanwhile Fergal Somers, chief architect at Cape Clear, doesn't think it is so hard in practice, although he acknowledges that the relevant standards are not yet mature.

Any systems that are linked together as part of a broader workflow involves semantic-coupling as defined above, but so what? We have been building these systems for some time.

Although I wouldn't go as far as saying SOA is all about any one thing in particular (see my earlier post on Ambiguity), I also agree that semantic coupling (and semantic interoperability) are important.

Rocky's argument is based on a manufacturing analogy.

• In simple manufacturing, the economics of scale involves long production runs, so that you can spread the setup costs across a large volume.

• In agile manufacturing, the economics of scope involves minimizing the setup costs, so that you can have shorter production runs without affecting the economics of scale.

• I interpret Rocky's argument as saying that a major element of the setup costs for services involves matching the semantics.

Part of the economic argument for SOA is that it can deliver economics of scope (adaptability, repurposing) as well as economics of scale (productivity).

But there's more. If we combine SOA with some other management innovations, we may also be able to improve the economics of alignment. I don't think this is illustrated by Rocky's manufacturing analogy.

However, Kenneth LeFebvre reads more into Rocky's post than I did.

There is meaning to the interaction between a consumer and a service. What does this mean? SOA is all about making the connections between applications using “services” but it does not bridge the gap between the real world of business and the “virtual” world that runs within our software. This is precisely the problem object-oriented design was intended to solve, and was just beginning to do so, until too much of the development population abandoned it in search of the next holy grail: SOA.

At my request, Kenneth has elaborated on this statement in a subsequent post SOA OOA and Bridging the Gap. I agree with him that the rhetoric of OO was as he describes. But I still don't see much evidence that "it was just beginning to do so", and I remain unconvinced by his argument that some things are better represented by objects than by services. (More concrete examples please Kenneth.)

For a definition of the economics of scale, scope and alignment, see Philip Boxer's post Creating Economies of Alignment (October 2006).

---

Note: earlier material used the term Economics of Governance. For various reasons, we now prefer the term Economics of Alignment.

Updated 25 October 2013

Clouds and Clocks 4

An interesting debate on SaaS and Control

• Cloudy Future for the Enterprise and most likely for ISVs too (Gianpaolo Carraro, Microsoft)
• Tradeoffs in SaaS (Charlie Bass, EDS)

Gianpaolo starts out by suggesting a simple trade-off between user control and the economics of scale. With SaaS, the user forgoes some control, in order to benefit from (some share of) the economics of scale achieved by the service provider. Gianpaolo divides control into two aspects: Control of Features and Control of SLA.

Charlie thinks it comes down to a perspective of what is "control". He suggests a different notion of control, whereby a contract with a service provider would give you more legal control than doing the work in-house.

But as far as I can see, both of them are talking about Command (setting the target conditions) rather than Control (regulating the system to satisfy the target conditions). See Alberts and Hayes, Understanding Command and Control (pdf).

Gianpaolo associates the Control of Features with who builds the service (Implementation) and the Control of SLA with who runs the service (Deployment). (This is a bit of a simplification; I shall ignore that for the moment.) But in Service-Oriented Architecture, there is a third and perhaps the most important role: who specifies the service. It is possible for the user (or a community of users, or an independent regulatory body) to provide the specification, and for the provider merely to deliver an acceptable implementation and deployment. Returning to the distinction between Command and Control, we may associate the Command of both Features and SLA with who specifies the service.

But what kind of service are we talking about here? The economics of scale are most obviously associated with simple one-size-fits-all services that are symmetric and replicable - what Philip Boxer calls r-type services. In most cases such services are unilaterally specified by the service provider, offer little if any contextual variety to the service user, and control tends to focus on quality control and cost control.

With more complex types of service, the service user typically demands a much greater fit to a specific use-context, and the question of control expands to include requisite variety and the economics of governance. But does this mean that the economics of scale go out of the window? The challenge for very large and complex service-based systems is to combine the economics of scale AND scope AND governance.

A Twin-Track Approach to Government IT

#ukgovit @instituteforgov has just published a report called System Error: Fixing the Flaws in Government IT.

The report recommends a twin-track approach to government IT, based on the two concepts of Agile and Platform.

"The platform must standardise and simplify core elements of government IT. For any elements of IT outside the platform, new opportunities should be explored using agile principles. These twin approaches should be mutually reinforcing: the platform frees up resource to focus on new opportunities while successful agile innovations are rapidly scaled up when incorporated into the platform."

The report acknowledges the tension between these two concepts ...

"Treating items as commodities reduces cost but can limit flexibility; coordinating elements of IT across departments frees up resources but may move them further from frontline users; common standards support interoperability but also restrict the freedoms to innovate."

... and offers some general ideas for managing this tension.

• To act fully in the interests of government, an agile approach requires a light touch form of coordination at a system level. 

• To minimise duplication of effort in solving the same problems, there needs to be system-wide transparency of agile initiatives. 

• Existing elements of the platform also need periodic challenge. ... Transparency, publishing feedback and the results of experiments openly, will help to keep the pressure on the platform for continual improvement as well as short-term cost savings.

Trouble is, some of this stuff is really hard. The report talks glibly about "a less than intelligent customer", referring first to business users having an inadequate conception of the possible, and then to the public sector as a whole lacking the collective knowledge and skills to negotiate effectively with suppliers. This lack of intelligence is apparently blamed on the V-model development process, which creates the impression that the adoption of Agile methods would solve this problem. But the idea of Agile as a silver bullet is a dangerous one, as many people have already pointed out on the Linked-In discussion group.

One way of understanding the twin track approach is to think of the different kinds of economics involved.

• 'Platform' means delivering economies of scale and economics of scope.
• 'Agile' means delivering economies of alignment.

Combining the two introduces some complex architectural challenges, as I've written about here and elsewhere before. We call this Asymmetric Design. For an example of this approach applied to public sector IT, see an analysis of the CSA Case by Philip Boxer and myself. See also The Impact of Governance Approaches on SoS Environments (pdf) by Philip Boxer and others.


In the current economic situation, the public sector as a whole is charged with making massive cost savings, and it is crazy to imagine that cost savings of this scale would not be associated with significant structural change, including IT systems. This kind of disruptive innovation goes way beyond the economies of scale and scope, and introduces some serious questions about the economics of alignment.

The word "architecture" is mentioned a few times in the Institute for Government report, but only in passing as something that the Government CIO will look after. (Mostly technology or solution architecture, I only found one single reference to business architecture.) So there is an implicit idea of central thinking and hierarchical governance. But there are some architectural challenges here that are some way beyond the current practices of enterprise architecture.

Governance is also a significant problem. The report comments on the pendulum swings between centralized and decentralized provision, which is something we noted in the CSA case, and was also present in the case of ContactPoint (which we were in the middle of writing up when it was cancelled). Such pendulum swings are often a characteristic symptom of weak or unsustained governance.

Not only is this stuff structurally complicated, but there are some commercial stakeholders that have every incentive to maintain the complicated status quo, thanks to a grossly dysfunctional procurement process.

And there is an even bigger problem with the report, which is that it looks at government IT exclusively from within government - in other words, from the perspective of civil servants. For example, the report adopts a supply-side notion of "joined-up government", understood largely in terms of internal linkages and efficiencies between systems, and fails to mention the demand-side notion of "joined-up government" that involves a coherent experience for the citizen. (See my post on Joined-Up Government from December 2005.)

Meanwhile the notion of "user" appears to refer mainly to civil servants and other public sector workers. Surely the purpose of government IT is not to provide direct value to civil servants but to provide various forms of indirect value to individual citizens and socioeconomic communities.

The report regrets that "government IT [is] falling further and further behind the fast-paced and exciting technological environment that citizens interact with daily" and indicates "the potential for IT ... fundamentally changing the relationship between citizen and state". "Around the world governments are using technology to help them deliver better services, be more transparent and accountable, and connect more directly with their citizens." (Examples are cited from Canada, USA and Malaysia.)

And yet the report fails to explain how "agile" can adequately represent the demand side requirements of citizens, interacting with a broad range of government services while going about their public business. There is a completely different notion of "platform" required here - government as a platform, which Tim O'Reilly and others have been talking about for a couple of years. And a different notion of agility, which goes a lot further than agile software development.

---

Other commentary

See Linked-In discussion group

Harry Metcalfe (2 March 2011) observed that many of the recommendations in the report were really hard, and was one of the first to complain about the insufficient attention to procurement in the report.

Ecosystem SOA 2

What are the problems of large complex sociotechnical systems? How far do SOA and enterprise architecture help to address this problem space, and what else might we need?


When I started writing about SOA and the service-based business over ten years ago, I defined two "cuts" across the service ecosystem. One cut separates inside from outside, while the other cut separates supply from demand.

(This diagram was included in my 2001 book on the Component-Based Business, and frequently referenced in my work for the CBDI Forum. For a brief extract from the book, see my Slideshare presentation on the Service Ecosystem.)

The inside/outside cut is sometimes called encapsulation. It decouples the external behaviour of a service from its internal implementation, and can be described in terms of knowledge - the outside has limited knowledge of the inside, and vice versa. (The cut is also sometimes called transparency - for example location transparency, which means that external viewers can't see where something is located.)

The supply/demand cut is about delegation, and can be described in terms of responsibility. Getting these two cuts right may yield economics of scale and scope; and the business case for SOA as a development paradigm is often formulated in terms of reusing and repurposing shared services.

For relatively small and simple SOA projects, it may be feasible to collapse the difference between these two cuts, and treat them as equivalent. (The inside/outside relationship and the supply/demand relationship are sometimes both described as "contracts", although they are clearly not the same kind of contract.) However, enterprise-scale SOA requires a proper articulation of both cuts: confusing them can result in suboptimal if not seriously dysfunctional governance and procurement. Many people in the SOA world still fail to understand the conceptual importance of these cuts, and this may help to explain why some organizations have had limited success with enterprise-scale SOA.

Going beyond enterprise SOA as it is generally understood, there is a third cut separating two views of a system: the system-as-designed (whose structure and behaviour and rules can perhaps be expressed in some formal syntax such as UML, BPMN or ArchiMate) and the system-in-use (whose actual performance is embedded/situated in a particular social or business context). This cut is critical for technology change management, because of the extent to which the designed system underdetermines the pragmatics of use. I have been talking about this cut for over twenty years, but only more recently working out how to articulate this cut in composition with the other two cuts.

One important reason for looking at the pragmatics of use is to understand the dimensions of agility. In many settings, we can see a complex array of systems and services forming a business platform, supporting a range of business activities. If no agility is required in the business, then it may not matter if the platform is inflexible, forcing the business activities to be carried out in a single standardized manner. But if we assume that agility is a critical requirement, then we need to understand how the flexibility of the platform supports the requisite variety of the business.

More generally, understanding the pragmatics of use leads to the recognition of a third kind of economic value alongside the economics of scale and the economics of scope: the economics of alignment. The value of a given system-of-systems depends on how it is used to deliver real (joined-up) business outcomes, across the full range of business demands. (I'm afraid I get impatient with people talking glibly and simplistically about business/IT alignment, without paying attention to the underlying complexity of this relationship.)

Understanding these three cuts (and analysing their implications) is critical to understanding and managing a whole range of complex systems problems - not just SOA and related technologies, not even just software architecture, but any large and complex sociotechnical systems (or systems-of-systems). If the three cuts are not understood, the people in charge of these systems tend not to ask the right questions. Questions of pragmatics are reduced to questions of platform design; while questions of the cost-justification and adoption of the platform are reduced to a simple top-down model of business value. Meanwhile the underlying business complexity (requisite variety) will be either misplaced (e.g. buried in the platform) or suppressed (e.g. constrained by the platform).

So there are three challenges I face as a consultant, attempting to tackle this kind of complex problem. The first challenge is to open up a new way of formulating the presenting problem, based on the three cuts. The second challenge is to introduce systematic techniques for analysing the problem and visualizing the key points. And the third challenge is to identify and support any organizational change that may be needed.

---

With thanks to Philip Boxer and Bernie Cohen. For a different formulation of the three cuts, together with a detailed example, see their new paper "Why Critical Systems Need Help to Evolve" Computer, vol. 43, no. 5, pp. 56-63, May 2010, doi:10.1109/MC.2010.150. See also Philip Boxer, When is a stratification not a universal hierarchy? (January 30th, 2007)


Related post Ecosystem SOA (October 2009)

Service Ecosystem and Market Forces

One of the problems with a network of services is that the responsibilities and costs and risks are often in the wrong place.

In this post I'm going to explain what I mean by this statement, outline some of the difficulties, and then make some modest proposals.

The statement is based on a notion of the efficiency of an ecosystem. If there is one service provider and a thousand service consumers, it may be more efficient for the ecosystem as a whole if the service provider includes some particular capability or responsibility within the service, instead of each service consumer having to do this. In addition to the economics of scale, there may be economics of governance - for example, increased costs of managing the service relationship, especially if the service provider doesn't provide a complete service (in some sense).

One important application of this idea is in security, risk and liability. There is a very good discussion of this in the recent British House of Lords Science and Technology Committee Report into “Personal Internet Security", who specifically address the question whether ISPs and banks should take greater responsibility for the online security of their customers.

"A lot of people, notably the ISPs and the Government, dumped a lot of the responsibility onto individuals, which neatly avoided them having to shoulder very much themselves. But individuals are just not well-informed enough to understand the security implications of their actions, and although it’s desirable that they aren’t encouraged to do dumb things, most of the time they’re not in a position to know if an action is dumb or not." [via LightBlueTouchpaper]

In other words, the responsibility should be placed with the player who has (or may reasonably be expected to have) the greatest knowledge and power to do something about it. In many cases, this is the service provider. Some of us have been arguing this point for a long time - see for example my post on the Finance Industry View of Security (June 2004).

Similar arguments may apply to self-service. When self-service is done well, it provides huge benefits of flexibility and availability. When self-service is done poorly, it merely imposes additional effort and complexity. (Typical example via Telepocalypse). Some service providers seem to regard self-service primarily as a way of reducing their own costs, and do not seem much concerned about the amount of frustration experienced by users. (And this kind of thing doesn't just apply to end-consumers - similar considerations often apply between business partners.)

But it's all very well saying that the service provider ought to do X and the service consumer ought to do Y. What if there is no immediate incentive for the service provider to adopt this analysis? There are two likely responses.

1. "We don't agree with your analysis. Our analysis shows that the service consumer ought to do X."
2. "We agree it might be better if service providers always did X. But our competitors aren't doing X, and we don't want to put ourselves at a disadvantage."

More fundamentally, there may be a challenge to the possibility of making any prescriptive judgements about what ought to happen in a complex service ecoystem. This challenge is based on the assertion that such judgements are always relative to some scope and perspective, and can easily be disputed by anyone who scopes the problem differently, or takes a different stakeholder position.

Another fundamental challenge is based on the assertion that in an open competitive market, the market is always right. So if some arrangement is economically inefficient, it will sooner or later be replaced by some other arrangement that is economically superior. On this view, regulation can only really achieve two things: speed this process up, or slow it down.

But does this mean we have to give up architecture in despair - simply let market forces take their course? One of the essential characteristics of an open distributed world is that there is no central design architectural authority. Each organization within the ecosystem may have people trying to exercise some architectural judgement, but the overall outcome is the result of complex interplay between them.

How this interplay works, whether it is primarily driven by economics or by politics, is a question of governance. We need to spell out a (federated?) process for resolving architectural questions in an efficient, agile and equitable manner. This is where IT governance looks more than ever like town planning.

Notes

The House of Lords Science and Technology Committee Report into “Personal Internet Security" was published on August 10th 2007 (html, pdf). Richard Clayton , who was a specialist adviser to the committee, provides a good summary on his blog. Further comments by Bruce Schneier and Chris Walsh.

BYOD - Bring Your Own Device

By popular demand, many companies are shifting ownership of elements of corporate infrastructure onto their employees. This is known as BYOC (bring your own computer) or BYOD (bring your own device).

There are many aspects to this trend.

1. Culture. Talented recruits may see this kind of choice as a desirable feature of a future employer. Some of them may have a strong personal commitment to a particular device; others may ask about BYOD policy as a quick way of getting a general impression of company culture and its attitude towards employees.

(Even if BYOD is a common request at interview, this doesn't mean it is a genuine requirement. In some cases, the BYOD request could be similar to the apparently crazy riders that performers may add to contracts as a way of testing the diligence and attention to detail of the organizers. The best-known example of such a contract rider is Van Halen's insistence on a bowl of MnMs with the brown ones removed. See "Brown out" at snopes.com.)

2. Interoperability. There is a need for interoperability within the enterprise (endo-interoperability) as well as interoperability with external platforms (exo-interoperability). Within the enterprise, people expect to be able to use common services (email, communications, content management, and so on) regardless of device. When I'm in the office, I want to be able to connect my device to office devices such as printers and projectors, as well as using the office network and servers. When I'm working at home, I want to be able to connect my device into the office systems, and use my device for web conferences and other events. But I also want to be able to connect my device into public platforms such as Facebook.


3. Innovation. Early adopters like to carry the latest and most fashionable device, even if this doesn't yet support all the required corporate services in a robust manner.

4. Business continuity and risk. A person's productivity can be seriously impaired if the device is lost or develops a fault. Conversely, a company's security can be seriously impaired if an employee uses an unverified emergency device such as her teenage son's phone. Does BYOD imply the rapid availability of backup devices of every conceivable brand, or does the company provide a limited range of standard devices for emergency use?

5. Support. Does the device deliver all the required corporate services correctly, efficiently and securely? Whose responsibility is it to verify and test these services on the given device, and to sort out the (inevitable) configuration problems? What knowledge and expertise is needed to provide adequate support across the full range of devices?

6. Economics. Device provision within large organizations was traditionally based on the economics of scale. We purchase thousands of identical devices, install the same software and services on each one, and issue these to our employees. We can obtain good discounts from the hardware and software suppliers, and we can train our support staff to provide efficient support across a narrow range of products. But this approach fails to deal with the complexities of the modern business organization where each employee has different needs, often calling for additional non-standard software and services, or even newer devices. So most modern organizations shift to provision of devices based on the economics of scope - giving everyone a flexible device platform to which additional software and services can be easily added. Then the move to BYOD takes us into the economics of alignment - optimizing the lifetime cost of device provision against the lifetime benefits to the organization and the individual within the context of use.

7. BYOD represents a shift in the balance between two kinds of device vendor - the ones who sell thousands of devices at a time by schmoozing the CIO and the ones who sell devices to individuals via consumer channels. (As a result, some stakeholders may be cynical and unsympathetic to any objection to BYOD from the CIO quarter.)

8. More fundamentally, BYOD represents a shift in the balance of power between two kinds of knowledge. The corporate IT folk supposedly know more about the corporate services and about quality attributes such as reliability and security. However, the individual employee knows more about the context of use. The architectural question here is aligning the device selection, configuration and use with the emerging requirements of the individual in the job. This is ultimately a question of governance, which needs to be guided by appropriate BYOD policies.


A lot of architectural issues then.

---

Fiona Graham, BYOC: Should employees buy their own computers? (BBC News 14 January 2011)

Fiona Graham, BYOD: Bring your own device could spell end for work PC (BBC News 14 February 2012)

Eric Vanderburg, Four Keys to Successful BYOD (CIO 14 February 2012)


Related posts:

Bring your own expectations (May 2014)

Service-Oriented CashFlow

In difficult economic times, the most important business measure for many firms is not profit or revenue growth or cost-saving but cashflow. Which means that the business case for SOA shifts away from relatively boring arguments about cost-saving (economics of scale), and strategic but highly imprecise arguments about flexibility (economics of scope), towards careful comparison of the dynamic economic viability of different business models (economics of governance alignment).

There aren't yet many SOA case studies that talk about cashflow. When I search for "service-oriented" and "cashflow", I keep finding references to a healthcare distribution company Owens & Minor, based in Mechanicsville VA, which started an 4-year SOA project sometime in 2005. (Speed Wins at Owens & Minor, Feb 2005) So nearly finished yet, chaps?

As I've explained in previous posts, the Pay-As-You-Go model has two advantages for the service consumer - better cashflow (because there is less financial latency between expenditure and return) and reduced risk (because you only spend as much as you need). As economic conditions become more volatile, and credit becomes more expensive, both of these advantages become more valuable.

Valuable for the consumer that is. The SaaS vendors (cheerleader Phil Wainewright) quite rightly see this as an opportunity for SaaS. But of course the challenge for the SaaS vendors is how to provide these benefits without themselves incurring excess cost and risk.

Billing and credit control are obviously key elements of the SaaS infrastructure, and some of Phil's clients offer useful services in this space. So are we moving towards a stratified SaaS world: (service-as-a-service)-as-a-service? And what are the economics of that?

---

Phil Wainewright on CashFlow: A Few Financial Home Truths. For mechanisms for handling billing see Phil Wainewright on Easing the SaaS-to-cash cycle and Marco Seiriö (RuleCore) on More CEP SaaS Pain.

Note: for various reasons, we now prefer the term Economics of Alignment. 

Updated 25 October 2013

Defense Procurement

Let me start with a couple of quotes from a speech by US Secretary of Defense Robert M. Gates, Washington, D.C., Monday, September 29, 2008.

"When it comes to procurement, for the better part of five decades, the trend has gone towards lower numbers as technology gains made each system more capable. In recent years these platforms have grown ever more baroque, ever more costly, are taking longer to build, and are being fielded in ever dwindling quantities."

"The issue then becomes how we build this kind of innovative thinking and flexibility into our rigid procurement processes here at home. The key is to make sure that the strategy and risk assessment drives the procurement, rather than the other way around."

The US Department of Defense appears to have a very clear understanding of the opportunities for Service-Oriented Architecture in enhancing the procurement process and improving the economics of governance. In July 2008, the SOA Acquisition Working Group of the AFEI produced a useful set of Industry Recommendations for DoD Acquisition of Information Services and SOA Systems.

One point that comes across very clearly from this paper is the opportunity to separate the provision of different layers within a layered architecture. So we may have one set of suppliers providing the underlying layers, and a different set of suppliers providing the upper layers.

The AFEI paper identifies three roles - Platform Providers and Commodity Infrastructure, Component Developers and Enterprise Managers - with what it calls Firewalls between them. (See Figure 3 in the AFEI paper.) These firewalls roughly correspond to the first two of what Philip Boxer calls the three asymmetries of demand. The third asymmetry (not explicitly shown in the AFEI paper) is between the Enterprise Managers and the Customers. (Including customers in the schema is essential if you want to move towards a risk-reward model of procurement, as the DoD evidently does, because it is the customers who ultimately generate the effects.)

Managing all this complexity almost certainly results in higher management cost than if you simply outsource the whole lot to a large system integrator, but it also shifts the management control (architectural governance) back to the acquiring organization.

So which kind of organizations would want to do this then? It's a question of complexity - the greater the level of complexity and volatility in your requirements, the greater the potential benefits from having finer-grained and stratified control over your procurement.

And what kind of organizations are capable of this fine-grained stratified control? Perhaps not too many at the moment, which is why we need a capability maturity model for SOA that provides a roadmap for organizations to develop an appropriate level of SOA capability.

Sources

AFEI SOA Acquisition Working Group, "Industry Recommendations for DoD Acquisition of Information Services and SOA Systems", July 2008 (register on the AFEI website for free download)

For explanation of the three asymmetries, see Richard Veryard & Philip Boxer, Metropolis and SOA Governance, Microsoft Architecture Journal, July 2005.

Loose Coupling 2

Is loose coupling a defining characteristic of the service-based business and/or a core principle of SOA? ZDnet's analysts have been producing a set of IT commandments (13 at the last count), and Joe McKendrick's latest contribution is Thou Shalt Loosely Couple.

Joe quotes John Hagel's definition of loose coupling, which refers to reduced interdependencies between modules or components, and consequently reduced interoperability risk. Hagel clearly intends this definition to apply to dependencies between business units, not just technical artefacts. I think this is fine as far as it goes, but is not precise enough to my taste.

In his post The Developer's View of SOA: Just adding complexity?, Ron Ten-Hove (Sun Microsystems) defines loose coupling in terms of knowledge - "the minimization of the "knowledge" a service consumer has of a service provider it is is using, and vice versa".

(It is surely possible to link these defining notions - interoperability and interdependency, risk and knowledge - at a deep level, but I'm not going to attempt it right now.)

I want to have a notion of loose coupling that applies to sociotechnical systems of systems - and therefore needs to cover organizational interdependencies and interoperability as well as technical. I have previously proposed a definition of Loose Coupling based on Karl Weick's classic paper on loosely coupled organizations.

The trouble with proclaiming the wonders of loose coupling is that it sounds as if tight coupling was just a consequence of stupid design and/or stupid technology. It fails to acknowledge that there are sometimes legitimate reasons for tight coupling.

Ron Ten-Hove puts forward a more sophisticated argument for loose coupling. He acknowledges the advantages of what he calls a mixed service model, namely that "it allows for creation of a component model that combines close-coupling and loose-coupling in a uniform fashion". But he also talks about the disadvantages of this model, in terms of reduced SOA benefits and increased developer complexity, at least with the current technology.

Loose coupling is great, but it is not a free lunch. It is not simply a bottom-up consequence of the right design on the right platform. Sometimes loose coupling requires a top-down forcing-apart. I think the correct word for this top-down forcing-apart is deconfliction, although when I use this word it causes some of my colleagues to shudder in mock horror.

Deconfliction is a word used in military circles, to refer to the active principle of making one unit independent of another, and this will often include the provision of redundant supplies and resources, or a tolerance of reduced utilization of some central resources. Deconfliction is a top-down design choice.

Deconfliction is an explicit acceptance of the costs of loose coupling, as well as the benefits. Sometimes the deconflicted solution is not the most efficient in terms of the economics of scale, but it is the most effective in terms of flexibility and interoperability. This is the kind of trade-off that military planners are constantly addressing.

[See earlier post on Deconfliction and Interoperability]

Sometimes coupling is itself a consequence of scale. At low volumes, a system may be able to operate effectively in asynchronous mode. At high volumes, the same system may have to switch to a more synchronous mode. If an airport gets two incoming flights per hour, then the utilization of the runway is extremely low and planes hardly ever need to wait. But if the airport gets two incoming flights per minute, then the runway becomes a scarce resource demanding tight scheduling, and planes are regularly forced to wait for a take-off or landing slot. Systems can become more complex simply as a consequence of a change in scale.

(See my earlier comments on the relationship between scale and enterprise architecture: Lightweight Enterprise.)

In technical systems, loose coupling carries an overhead - not just an operational overhead, but a design and governance overhead. Small-grained services may give you greater decoupling, but only if you have the management capability to coordinate them effectively. In sociotechnical systems, fragmentation may impair the effectiveness of the whole, unless there is appropriate collaboration.

In summary, I don't see loose coupling as a principle of SOA. I prefer to think of it as a design choice. I think it's great that SOA technology give us better choices, but I want these choices to be taken intelligently rather than according to some fixed rules. SOA entails just-enough loose coupling with just-enough coordination. What is important is getting the balance right.

Two-Sided Markets

There has been a lot of buzz around two-sided and multi-sided markets lately.

• Wikipedia: Two-Sided Markets

• Harvard Business Review: New Research Explores Multi-Sided Markets (March 2006), Strategies for Two-Sided Markets (Oct 2006)

• Emergic: Overview, Pricing, Examples, Business Application

In his HBS March interview, Andrei Hagiu identifies Wal-Mart as an example of an organization that is transforming from a traditional merchant into a two-sided platform. Let’s look at the (asymmetric) structure of this transformation.

The traditional retailer acts as a hub in the food supply chain, aggregating food supply from fields and factories, and distributing food to workshops and private kitchens. This is essentially a positional strategy: the retailer seeks to establish and maintain a strategic position within a value chain, as the bottleneck/hinge point between upstream and downstream. Within the positional strategy, the business drivers are understood in terms of the economics of scale and the economics of scope.



But if we shift from a value-chain perspective to a service-oriented perspective (effects-ladder), we can see that the retailer is providing a service (=delivering value) downwards as well as upwards - it is a food distribution platform for farmers and manufacturers as well as a food supply platform for consumers and catering companies.

So instead of drawing the merchant in the middle, we can draw the merchant as a new kind of platform providing various kinds of market interaction.




This takes us from a positional strategy to a relational strategy. No longer just focused on the economies of scale and scope, the relational strategy emphasizes how economies of governance are generated in relation to two kinds of demand context. The big question for a company such as Wal-Mart is how to balance the exploitation of each of these forms of asymmetric advantage.

 

See also

Philip Boxer, Asymmetric Demand is Multi-Sided Demand (October 2011)

Richard Veryard, SOA as Multi-Sided Platform (September 2009)

Service Escrow - Iron Mountain

Last month I riffed with Gianpaolo Carraro about Service Escrow. A few days later, a company called Iron Mountain launched an SaaS escrow service, covering the source code, system documentation and data.

• Iron Mountain press release
• Link to ThinkStrategies white paper (written by Jeff Kaplan, commissioned by Iron Mountain)

Gianpaolo (who is one of Microsoft's SaaS experts) refers to companies providing this service as SaaS undertakers. But it might be better to call them SaaS support. By acting as a safety net for SaaS providers and their customers, they may reduce the risk associated with SaaS and contribute indirectly to SaaS market growth.

I phoned Iron Mountain to find out more about their SaaS Escrow service, and the extent to which it differed from the traditional software escrow service. Here are some of the main points of our conversation.

Storage Model

Iron Mountain stores both the software (source code, object code and documentation - all of which belongs to the SaaS provider) and the data (which belongs to the SaaS user). Whereas traditional software escrow can often operate on a fairly slow cycle, with new software versions deposited in a fairly leisurely manner, SaaS escrow generally calls for live data backup.

If the escrow conditions are triggered, Iron Mountain will release the software and data to the SaaS user. To avoid any perceived conflict of interest, Iron Mountain does not operate the software - even on an emergency basis. But this means that the SaaS developers (or some appointed third party) must install and test the software on a new server, load and test the data, and then restore the service.

This is probably not something that can or should be done overnight - so it is not going to provide much protection against a sudden and unexpected failure of a business critical service. A more likely scenario is a gradual worsening of the relationship between the SaaS provider and the SaaS user, and a growing dissatisfaction with service levels and support arrangements, approaching the point where the SaaS provider is in breach of contract. SaaS escrow means that the SaaS user has a reasonable exit from an unsatisfactory relationship, and cannot be held to ransom because the SaaS provider controls a key business asset.

The economic benefits of escrow therefore fall under the economics of governance - making sure the SaaS user has proper control of the relationship.

Charging Model

Although the SaaS provider may benefit from the existence of SaaS escrow, the prime beneficiary is the SaaS user. Historically, it has always been the user who has negotiated and paid for escrow. However, Iron Mountain is increasingly seeing more complex arrangements whereby the software provider pays for the escrow and passes these charges onto the sofware user.

In the case of SaaS escrow, a typical arrangement would be that the SaaS provider pays for the deposit of the software, while the SaaS user pays for the deposit of the data (depending on the data volumes).

It is in the interests of the SaaS user that Iron Mountain always holds the latest version of the software. Iron Mountain therefore encourages the SaaS provider to deposit software upgrades as frequently as necessary - preferably online - and does not charge on the basis of frequency. (Remember that SaaS software may undergo a faster improvement cycle than traditional software packages.)

Management Process

SaaS escrow only works if the SaaS provider has adequate software configuration management and data management, so that it becomes a matter of routine to send controlled copies to Iron Mountain. There is a certain amount of ongoing verification and audit that needs to be carried out, involving all the parties to the escrow arrangement, and Iron Mountain sees this as an important aspect of its own role.

Remember that the SaaS user does not see the software unless and until the escrow conditions are triggered - so it isn't possible to test the escrow arrangements in a trial run exercise. Instead, it is necessary to test the process at a higher level of abstraction, to provide some reassurance to the SaaS user that the escrow would work adequately.

Future

This is early days for the SaaS escrow market, and I shall be interested to see how the market develops ...

Selling SOA by the pound

Steve Jones suggests that SOA needs marketing. Unfortunately, he only mentions one-way communication - from the SOA programme to the business. True marketing involves two-way communication [see post on Marketing] - this entails listening and responding to what the business wants. And that means what a particular business specifically wants, not just vague generalizations like "adaptability" and "customer satisfaction" [see Seth Godin on Business Cliches].

There is some question about the extent to which it is even possible to sell SOA to the business - whether it makes sense to measure the ROI of SOA. Those challenging this include Randy Heffner (via James McGuire) and John Devadoss.

But there is a more fundamental question - does the business need to hear about SOA at all? As Nick Malik asks: When we talk to the business, should we sell SOA? His argument is that good SOA is essentially free, so you shouldn't have to sell it separately. This is what he says in an earlier post, sarcastically titled I want one of those SOA thingamajiggies:

'I’ve decided that the best way to “Sell SOA” to the business is not to sell SOA to the business. Let’s talk about the asthetics, the features, the speed and reliability, the automation of their business processes to allow them to focus and innovate where it counts. Let’s not talk about standard parts.'

Lorraine Lawson disagrees. Summarizing a recent discussion on LinkedIn, she concludes:

'Of course, you shouldn’t start out the conversation by talking about the technical aspects of SOA. Business value is definitely the place to begin. But be prepared to discuss services and explain why SOA is better that EAI or standard development practices — my guess is, there will be questions.'

The claim that SOA is free (under certain initial conditions) echoes the claims of some quality management gurus - notably Philip Crosby - who claimed that quality was free (again under certain initial conditions).

But let's suppose, for the sake of argument, that good quality management, or good SOA, really is free, in the sense that it doesn't add anything to the financial costs associated with certain activities (manufacturing, IT development, whatever). Even if this is true, it doesn't mean we don't have to sell SOA to the business. SOA makes demands on the business - perhaps not financial or technological ones, but demands associated with architecture and governance. For SOA to be successful (let alone "free"), the business may need to consider certain structural opportunities, and to engage with IT requirements in new ways. Even if we don't sell "SOA" to the business, we should still be able to sell "agile enterprise" and the "economics of governance" - not just as vague generalizations but specifically grounded in the needs of a particular business. Because without that connection to the business, SOA will be unable to deliver anything of real value.

Ford and Fordism

In his piece Flashlights not Bailouts, Phil Gilbert describes the classic CBD/SOA argument: standardization/rationalization/reuse.

"One of the three (Ford) is in demonstrably better shape than the other two, and it's no mystery why. Two years ago, when he took the reins of Ford, Alan Mulally identified two things that needed to change: parts costs have to go down, and engineering productivity must go up."

"Get it? The white collar workers who design the cars have to move from artisan to engineer, and they need to work together across all the company's platforms to use common parts."

I asked Phil if this was Fordism? Phil responded thus:

"Ford used standardization of task to drive down costs. I'm suggesting 'standardization' of transparency to drive down risk."

Thus Fordism progresses from the economics of scale to the economics of governance. This fits well with the story about SOA I've been telling in this blog. http://tinyurl.com/d3ronn

SOA and Web 2.0

Economic principles such as network effect and the long tail play a role in how Web 2.0 may impact your business.

Network Effect

• The value of a service in one place depends on the amount of usage elsewhere – the more the better.

• This is especially true of services that offer some form of communication – the value of the service depends partly on the number of other people you can communicate with.

• There is a positive feedback loop – as more people use the service, it may increase in value, thus persuading more people to use the service. This feedback loop may result in a rapid growth curve.

• This leads to the concept of critical mass – the level at which the growth curve becomes self-sustaining.

Long Tail

• The traditional Pareto principle (80% of the value comes from 20% of the cases) assumes a significant proportion of fixed costs. For example, a bricks-and-mortar bookshop must devote shelf space to any book in stock, and there is a cost associated with excess inventory.

• But in a service-based economy, it may be possible to alter the supply-side economics – for example shifting from fixed costs to variable costs – and this means that the rare items (both individually and in aggregate) could be as profitable as the popular items.

• The economic viability of the long tail depends critically on bringing down the fixed costs of servicing the long tail, and shifting to a more flexible variable-cost regime.

Web 2.0 and SOA together can help provide a cost-effective way of accessing these network effects and long-tail benefits.

• Cost reduction – leveraging existing systems through services, helping to cut cost of customization

• Cost mitigation – providing services not solutions, shifting responsibilities to collaborating partners. For example, customer self-service, small communities providing their own solutions. Thus collaboration helps spread the cost.

• Shift from fixed cost to variable cost – for example through “pay-as-you-go” on-demand pricing – thus helping to reduce the risk and the barriers to adoption.

At the same time, IT organizations are also often looking for ways to improve the delivery of solutions to their business users. Web 2.0 technologies and approaches may help reduce developer effort and time to solution, as well as providing more agile solutions that can be rapidly remixed and re-assembled to meet changing demands.

Challenges/concerns

In order to achieve these Web 2.0 benefits, there are some questions that need to be answered:

• What are the implications for IT management? How can I guarantee the continued security of my core systems? How can I provide the necessary quality of service to my existing users as well as the new ones?

• What are the implications for business management? How can I guarantee the continued integrity of my products and services? If third parties are able to mash my services with external services, how can I retain control of my brand and corporate image?

• What are the implications for corporate governance and compliance? Do these external arrangements introduce new risks?

Clearly many organizations will not be prepared to go ahead with Web 2.0 without good answers to questions like these. This is why it will be important to have a well-structured approach to Web 2.0 and SOA.

---

This is an extract from an article Lawrence Wilkes and I wrote in February Extending SOA with Web 2.0. Commissioned by IBM and available free from the CBDI Forum website.

Service Planning 2

Nick Malik asks Should SOA be Top Down or Bottom Up. He suggests that architects need to pay attention to the big pieces and can ignore the small pieces.

"While the architects do not need to know about every moving part, they DO need to be aware of the largest of those parts, and make sure that they are managed well. This is similar to city planning, where the city needs to work with a large employer or a large retailer (like Walmart) to make sure that roads and parking and congestion issues are managed, without having to worry about cafe and card shop that are also employers, but have a minimal impact on the infrastructure."

Some people argue for a "fractal" notion of the service economy. While the word "fractal" isn't always used in its precise mathematical sense, its use seems to imply that the service portfolio should have a good mixture of different sizes / granularities (although not necessarily in the same architectural layer). Such a mix of different sizes is also advocated by Christopher Alexander.

In city planning, small retail outlets such as cafes and card shops may individually have less economic or environmental impact than a major retailer such as Wal-Mart. But collectively, they may have at least as much impact. (This is a "long tail" argument.) One of the challenges for city planning is to achieve a good balance between the large few and the many small. (Of course this raises questions of governance as well as architecture, politics as well as economics.)

If the total weight of the large pieces is greater than the total weight of the small pieces (whatever measure we choose for "weight"), then this is itself an architectural choice, with important implications for project agility and enterprise agility.

Most people in this game think they know what the terms "top-down" and "bottom-up" mean, but these terms are commonly used in different (contrary, confusing) ways. If an architect only worries about fitting the big pieces together, and assumes that the small pieces will somehow look after themselves in the remaining ("negative") space, this sounds like one version of what some people would call "top-down".

What if the architect concentrates on providing "positive space" in which the small pieces can thrive, and prevents the big pieces from encroaching on this space. What if the architect concentrates on the interfaces between the pieces, rather than the pieces themselves. Is this "top-down" or "bottom-up"?

I don't really care what we call this - although it would be good to have a more precise way of talking about it - but I think this is an equally valid strategy.



See also: What does Top-Down mean? (September 2011)