Strategy and Requirements for the API ecosystem

Is there a framework or methodology for establishing the business / ecosystem requirements to drive API strategy and development?

At an industry event I attended recently, hosted by a company that sells tools and technologies for the API ecosystem, some of the speakers advised that when presenting to non-technical stakeholders, you need to talk about service value/benefit rather than APIs. But this raises an important question, how to identify and quantify service benefit, and how to negotiate share of value between different players in the ecosystem?

One of the ideas of the API economy is that you don't have to maintain all the capabilities yourself, but you find other enterprises that can provide complementary capabilities. So you need to identify and understand what capabilities are available, and map combinations of these capabilities against the demands and unfulfilled needs of potential customers. Then having identified in broad terms what capabilities you wish to combine with your own, and worked out where the service boundaries should be, you may select organizations to partner with and agree business and commercial terms, or create a platform to which many third parties can connect. The technical design of the API should then reflect the service boundaries and commercial arrangements.

In the early days of service-oriented software engineering, people always wanted us to tell them how large their services should be. Not just macro versus micro, but broad (generic) versus narrow (specific). To what extent should a service be completely purpose-agnostic - in other words, with no restrictions on how or where it may be used - or does this conflict with other design goals such as reliability or data protection?

The answer is that it depends not only on what you are trying to do, but how you want to manage and govern your service architecture. A broadly scoped, purpose-agnostic service (or service platform) can achieve wide usage and economies of scale, but may be more complex to configure, test and use, whereas a more narrowly scoped context-specific service might be easier to use but with lower reuse potential. Among other things, this affects how much of the service composition and orchestration can be done by the service provider (supply side), and how much is left to the service consumer (demand-side). And even on the supply side, it affects how much work needs to be done by the integration experts ("town planners"), and how much can be left to citizen integration ("pioneers" and "settlers").

One version of this challenge can be found in large global organizations, working out exactly what functionality should be provided centrally as shared services, and what functionality should be left to local operations. Ideally, the service architecture should be aligned with the business and organizational architecture.

The word "economy" also implies attention to accounting issues - sharing costs and benefits between different players. Although we may regard cloud as almost infinitely extensible, this doesn't come without cost: if the number of service calls goes through the roof, someone has to pay the cloud provider's bill. This is already an issue within large organizations, where we commonly find arguments about whose budget will pay for something. And I have seen some great ideas come to nothing, because the benefits were spread too thinly and nobody was able to fund them.

So although vague appeals to innovation and imagination might be good enough for a marketing pitch, serious strategic thinking is about discovering where there is untapped value in your business and its environment, and working out exactly how an API strategy is going to help you unlock this value.

---

At the CBDI Forum, we were talking about these issues many years ago: our Service Architecture and Engineering® methodology is still available from Everware-CBDI. Here are some of the articles I wrote for the CBDI Journal.

• Identifying Web Services (February 2002) 
• Service Economics (September 2005)

More at  http://everware-cbdi.com/cbdi-journal-archive

Business as a Platform - Amazon

#bizarch Excellent article by @haydn1701 on Why Amazon Succeeds (Forbes, 29 April 2012) via @davegray and @ruthmalan.

Haydn Shaunessy points out the following features of Amazon's strategy.


Understanding ecosystem

Shaunessy starts by defining the notion of ecosystem fairly narrowly, and links it to the information market. "Bezos is not as great as Jobs at playing the information market but he is good." This may well be true, but it misses the point. Jeff Bezos's understanding of ecosystem has always been much more profound than that of any of his peers, and Amazon's ecosystem is a lot broader than zen marketing.

One way of seeing it is that Jobs coerced people to do things that were in his (Apple's) interests, whereas Bezos gives them opportunities to do things in their own interest, from which he benefits in more subtle but perhaps more sustainable ways. But of course there's always another story.

Radical adjacency

The ability to go beyond normal business practice and to seize opportunity in widely adjacent markets.

Business-as-a-platform

Shaunessy identifies a number of characteristics

• Tearing business away from "the straight jacket of old management theory" - "the outdated idea of core competency" 

• Providing a new way to scale business

• Bringing (shared) value to all parties

Technical enablers

Shaunessy identifies two in particular

• Universal connectors - not merely technical protocols but contractual protocols that radically reduce the transaction costs of using the platform 

• Cloud computing

Organizational enablers

Shaunessy talks about ability to attract and retain very talented, imaginative resources. I would also add the ability to build these into an effective and innovative company - what I call Organizational Intelligence.


Strategic Outcome

This combination of strategies has allowed Amazon and Apple to develop what Shaunessy calls complex options portfolios. He notes that "the ecosystem is full of businesses that can jump quickly into new opportunity" and argues that this yields the true meaning of shared value.

Google as a Platform (not)

Google vs Amazon (again). @davidsprott reckons Steve Yegge's rant is spot on. Steve Yegge is a software engineer who used to work for Amazon and now works for Google, despite the supposedly accidental publication of a long and opinionated rant (his words) complaining that

'[what] Google doesn't do well is Platforms. We don't understand platforms. We don't "get" platforms.'

(For more context, see Steve Yegge's second thoughts on Google+).

Waddya mean, Google doesn't get platforms? Surely Google is a major player in platform, especially in the so-called Cloud? Dion Hinchcliffe sounded fairly convinced when he was Comparing Amazon's and Google's Platform-as-a-Service (PaaS) Offerings back in April 2008.

"Amazon and Google have strategically built up an extensive set of services over the last few years and have made some very interesting assumptions that will determine who their customers are (consumers, startups, enterprises) and what type of business models can sit on top of them (advertising, subscriptions, cheapest source of outsourced computing resources). ... Google and Amazon have emerged to be the leaders in this space while Microsoft, IBM, and especially Oracle and SAP are either well behind or have unclear plans to enter the PaaS space. Both of these companies formed their DNA around the world of the Web and deeply understand how to leverage the enormous strengths of the Web platform."

So what went wrong? This guy (Yegge) sure knows one thing, says @pardhas, building platforms is not just collecting your products on one plate. For Yegge, platforms is about eating your own dogfood. Not just Amazon, but also Facebook - hey, even Microsoft understands platforms better than Google.

But there is something missing from Steve Yegge's account. He sees the (lack of) platform from an engineering perspective, but what he doesn't talk about is the enterprise/ecosystem perspective.

@davidsprott's tweet ends with the formula "SOA + platforms = competitive advantage".  David and I have written a great deal about ecosystem SOA and ecosystem architecture, and we have long credited Jeff Bezos of Amazon as someone who "gets" ecosystem.

Among other things, "getting ecosystem" means understanding the variety of ways in which the social complexity of collaborations create value for the customer, and therefore how, from the perspective of the supplier, platform architectures can capture indirect value. See Philip Boxer's presentation on supporting social complexity in collaborative enterprises, as well as my presentation on Next Generation Enterprise Architecture, both from the recent Unicom EA Forum in London.

As I pointed out in my recent VPEC-T analysis of Google, Google is adopting a positional strategy - capturing some territory and defending it against its competitors. Amazon and Apple have shifted towards open source competition on their platforms (relational strategy) while Google is still closed-source.

In his post on Tomorrow's Networked Economy, @JDeragon praises Google+ for becoming an integrated portal. Er, wasn't that AOL's strategy? And Yahoo's for that matter? Maybe Google has greater ability to execute this strategy than they did, but it still looks more like yesterday's networked economy. Have you read Kevin Kelly's book?

---

For Apple's shift from product to platform, see my post on Disney, Pixar, Apple and Jobs from February 2006.

For the shift from positional stategy to relational strategy, see Philip Boxer and Bernie Cohen, Triply Articulated Modelling of the Anticipatory Enterprise and Philip Boxer, Architectures that integrate  differentiated behaviours.

For more on Ecosystem SOA and Ecosystem Architecture, please browse the Ecosystem category on this blog. Here are some further links.

Jeff Bezos Letter to Shareholders (via Geekwire) (April 2011)
Bob Ellinger, Enterprise SOA vs Ecosystem SOA (April 2011)
Vaughan Merlyn, From Enterprise Architecture to Ecosystem Architecture (July 2008)
David Sprott, Introducing Smart Ecosystem Architecture (October 2009)

---

Ecosystem SOA 2

What are the problems of large complex sociotechnical systems? How far do SOA and enterprise architecture help to address this problem space, and what else might we need?


When I started writing about SOA and the service-based business over ten years ago, I defined two "cuts" across the service ecosystem. One cut separates inside from outside, while the other cut separates supply from demand.

(This diagram was included in my 2001 book on the Component-Based Business, and frequently referenced in my work for the CBDI Forum. For a brief extract from the book, see my Slideshare presentation on the Service Ecosystem.)

The inside/outside cut is sometimes called encapsulation. It decouples the external behaviour of a service from its internal implementation, and can be described in terms of knowledge - the outside has limited knowledge of the inside, and vice versa. (The cut is also sometimes called transparency - for example location transparency, which means that external viewers can't see where something is located.)

The supply/demand cut is about delegation, and can be described in terms of responsibility. Getting these two cuts right may yield economics of scale and scope; and the business case for SOA as a development paradigm is often formulated in terms of reusing and repurposing shared services.

For relatively small and simple SOA projects, it may be feasible to collapse the difference between these two cuts, and treat them as equivalent. (The inside/outside relationship and the supply/demand relationship are sometimes both described as "contracts", although they are clearly not the same kind of contract.) However, enterprise-scale SOA requires a proper articulation of both cuts: confusing them can result in suboptimal if not seriously dysfunctional governance and procurement. Many people in the SOA world still fail to understand the conceptual importance of these cuts, and this may help to explain why some organizations have had limited success with enterprise-scale SOA.

Going beyond enterprise SOA as it is generally understood, there is a third cut separating two views of a system: the system-as-designed (whose structure and behaviour and rules can perhaps be expressed in some formal syntax such as UML, BPMN or ArchiMate) and the system-in-use (whose actual performance is embedded/situated in a particular social or business context). This cut is critical for technology change management, because of the extent to which the designed system underdetermines the pragmatics of use. I have been talking about this cut for over twenty years, but only more recently working out how to articulate this cut in composition with the other two cuts.

One important reason for looking at the pragmatics of use is to understand the dimensions of agility. In many settings, we can see a complex array of systems and services forming a business platform, supporting a range of business activities. If no agility is required in the business, then it may not matter if the platform is inflexible, forcing the business activities to be carried out in a single standardized manner. But if we assume that agility is a critical requirement, then we need to understand how the flexibility of the platform supports the requisite variety of the business.

More generally, understanding the pragmatics of use leads to the recognition of a third kind of economic value alongside the economics of scale and the economics of scope: the economics of alignment. The value of a given system-of-systems depends on how it is used to deliver real (joined-up) business outcomes, across the full range of business demands. (I'm afraid I get impatient with people talking glibly and simplistically about business/IT alignment, without paying attention to the underlying complexity of this relationship.)

Understanding these three cuts (and analysing their implications) is critical to understanding and managing a whole range of complex systems problems - not just SOA and related technologies, not even just software architecture, but any large and complex sociotechnical systems (or systems-of-systems). If the three cuts are not understood, the people in charge of these systems tend not to ask the right questions. Questions of pragmatics are reduced to questions of platform design; while questions of the cost-justification and adoption of the platform are reduced to a simple top-down model of business value. Meanwhile the underlying business complexity (requisite variety) will be either misplaced (e.g. buried in the platform) or suppressed (e.g. constrained by the platform).

So there are three challenges I face as a consultant, attempting to tackle this kind of complex problem. The first challenge is to open up a new way of formulating the presenting problem, based on the three cuts. The second challenge is to introduce systematic techniques for analysing the problem and visualizing the key points. And the third challenge is to identify and support any organizational change that may be needed.

---

With thanks to Philip Boxer and Bernie Cohen. For a different formulation of the three cuts, together with a detailed example, see their new paper "Why Critical Systems Need Help to Evolve" Computer, vol. 43, no. 5, pp. 56-63, May 2010, doi:10.1109/MC.2010.150. See also Philip Boxer, When is a stratification not a universal hierarchy? (January 30th, 2007)


Related post Ecosystem SOA (October 2009)

Ecosystem SOA

The SOA world is finally catching up with some of the ecosystem ideas that I published in my 2001 book on the Component-Based Business (see my Slideshare presentation) and developed further in several articles and presentations for the CBDI Forum over a number of years.

• Supply and Fit (September 2000)
  
• Identifying Components 2 (October 2001)
• Opportunities in the Insurance Ecosystem (December 2004)
• Business Strategy Planning for the Service Economy (May 2006)
  

The biological approach to creating business and software services is radically different to the solution-driven approach, and is based on biological and ecological metaphors.

• First we identify an ecosystem, which may contain both human users and existing artefacts.
• Then we identify services that would be meaningful and viable in this ecosystem.
• Then we procure devices that enable the release and delivery of these services into the ecosystem.

I previously defined Three Types of Requirements Engineering, and we can map these onto different styles of SOA.

Solution-Driven (Specific)	Solution-Driven (General)	Evolution-Driven
Identify Business Problem Identify "Users" Negotiate Requirements Define Solution	Identify Domain Identify Domain Experts Define Requirements Design Solution Kit	Identify Ecosystem Identify Services Procure and Release Devices
Experimental SOA	Enterprise SOA	Ecosystem SOA

(Some people use the term Web Oriented Architecture (WOA) for what I'm calling Ecosystem SOA.)


If we regard these as phases of maturity, then we can have a straightforward roadmap from left to right, as in for example the CBDI SOA Roadmap. However, some organizations may need to tackle these styles of SOA in parallel rather than in sequence.

---

A service portfolio plan for Enterprise SOA can be based on an enterprise model that identifies the capabilities of the enterprise and clusters these into domains. In the CBDI Forum's SAE methodology, the domains are classified as Core and Contextual according to a matrix derived from Geoffrey Moore. (Note how the domains migrate around the matrix over time.) See for example my blogpost Tesco outsources core eCommerce.



A similar model, derived from Amin and Cohendet's book Architecture of Knowledge, explicitly describes Core and Periphery in terms of knowledge intensity. In other words, the reason something is classified as Core is because it encapsulates some important (strategic) knowledge.

For example, an insurance company knows more about insurance than about cars, so when providing car insurance it may decide to partner with other organizations that know more about cars than about insurance. This results in a composition of insurance-related services and car-related services (for example, determining the insurable value of a used car). Such compositions can be either directed (in other words, composed by a single dominant player) or collaborative (in other words, emerging from the interaction between multiple players within the ecosystem).

For example, an online retailer knows about her products and services, but doesn't wish to become an expert in credit card handling, or to be responsible for data protection and security, so she delegates these concerns to a specialist provider that knows more about these matters.

Similar considerations can apply to industry consortia, such as ACORD (for the insurance market). ACORD can define generic service-based assets (such as models, schemas, interfaces and so on) for insurance. However, insurance companies will also wish to use generic service-based assets to cover requirements that are not insurance-specific, such as customer management or complaints handling, where ACORD may not be able to add any knowledge-value, and it would be appropriate for ACORD to regard these as peripheral to its own activities rather than core.

---

So one approach to Ecosystem SOA is to push out from the enterprise into the ecosystem. John Hagel calls this Inside-Out Architecture, which he contrasts with Outside-In Architecture. (See my post on Outside-In Architecture.)


An Outside-In Architecture starts with a model of (the flows of) knowledge and value in the ecosystem as a whole. The strategic question for an enterprise is how to find way of both contributing value to the ecosystem, and drawing value from the ecosystem, through the provision of ecologically viable services.



For example, a telecoms company might reasonably consider that its core competence is something to do with communications. So a positional strategy would drive it to a dominant position in the middle of a large communication ecosystem, providing a platform of services that add value to a diverse range of communication activities by other people. The dominant position would allow it to negotiate a strong share of the value generated.

However, respect for the ecosystem would lead it to leave sufficient value to third parties to maintain the economic health of the remainder of the ecosystem. Instead of a simple positional strategy, a relational strategy (based on mutual trust with ecosystem partners) should produce a more sustainable and ecologically sound ecosystem.

---

Service Ecosystem from Richard Veryard


Related post: Ecosystem SOA 2 (June 2010)

Straight-Through Processing 3

Where have all the agents gone? asks Seth Godin. He is talking about travel agents, stock brokers, real estate, and so on.

"The problem with being a helpful, efficient but largely anonymous middleman is pretty obvious. Someone can come along who is cheaper, faster and more efficient. And that someone might be the customer aided by a computer."

Straight-through processing - sometimes seen as one of the potential sources of value from service-oriented architecture - is a mixed blessing, as I've pointed out before. There is always a danger of disintermediation, especially if the added value provided by the middleman appears trivial or easy to replicate. As Seth Godin points out, if you are just providing an anonymous human approximation of Google, don't bother - Google does it better.

In some industries, the middleman had become lazy - taking a cut for not doing very much. My local travel agent never gave me any useful information or advice, merely handed out brochures from the travel companies and struggled with the complexities of the booking system; so I stopped using them. And in the financial markets there are still companies that think they deserve a percentage of your pension fund in return for pressing a few buttons now and again.

In the past companies like these have often been able to take advantage of a strategic position in some larger process, extracting "rent" but without creating value. And there are doubtless still many niches in the system ecosystem that can be exploited in this way.

However, service ecosystem niches that allow companies to draw rent without creating value are going to be short-lived, and rightly so. In hard times, the percentage should only go to companies that are providing genuine value. (Actually the same principle applies all the time, but people are more motivated to follow this principle now than they have ever been before.) And ecosystem SOA should help us to make sure of this.

Then business strategy in a dynamic service ecosystem should not be based on finding and maintaining positional advantage, but on creating and maintaining genuinely productive relationships.

---

Previous posts

• Straight-through processing (Feb 2005)
• Straight-through processing 2 (April 2005)

Event Processing Example - Turbulent Markets

Can Real-Time Profit and Loss tame the turbulent markets? ask Bob Giffords (independent analyst) and Mark Palmer (Streambase).

The simple answer is No. Turbulence is a complex systems phenomenon. (Complex event processing is not primarily about complex systems, although some of the advocates of complex event processing might benefit from knowing a bit more about complex systems.)

If we want to know how turbulence can be tamed, we need to understand the root causes of turbulence, in terms of the non-linear effects of feedback loops. This is properly a job for market regulators. For example, central banks may try to reduce volatility in the money supply, and have sophisticated economic models to support their analysis. But the recent history of market regulation is a sorry one. Some analysts have argued that regulations such as Basel2 actually amplify volatility and turbulence in the system, because they force individual banks to execute transactions in response to market movements in order to maintain key ratios.

So it would be interesting to see an application of complex event processing in regulating a complex system. But this is not what the white paper is about. Perhaps wisely, it doesn't actually talk about taming the markets, merely about riding (=profiting from) the markets.

If some players have better tools, including CEP systems, this may give them an advantage in a competitive turbulent market. But this raises three important questions at the ecosystem level,

1. How does the use of these tools affect the market itself? Does the level of turbulence increase or decrease?

2. If the players with the best tools are those that profit the most from turbulence, then they possibly have an interest in promoting increased turbulence, even if this is damaging to everyone else.

3. What would happen to the ecosystem if these tools become commonplace? Would the advantages of these tools be reduced if everyone else had them?

Ecosystem Advantage

Mark O'Neill (Vordel) discusses a couple of SOA projects with an interesting goal - causing your customers to use less of your products. This is not competitive advantage, at least not directly, it is an advantage for the ecosystem as a whole, which becomes more efficient, less wasteful.

Mark's examples are in the utility sector

• Gridwise - a US consortium of energy companies and technology companies
  
  • Pacific Northwest National Laboratory aims to empower utility customers (IBM April 2007)
  • How SOA could change the way you buy electricity (Network World, Oct 2007)

• EPAL - the Portuguese water board (one of Mark's own clients)

Similar thinking applies to the "pay-as-you-drive" concept in insurance, encouraging drivers to use (and pay for) less risk.

These schemes are potentially very attractive. There are three main challenges here.

• Identification - analyzing an ecosystem systematically to discover opportunities for creating or releasing additional value

• Mobilization - aligning the ecosystem to the new improved distribution of value - easier where there is a single powerful player or industry regulator that can drive and enforce (and perhaps fund) the initiative until it becomes self-sustaining, otherwise more difficult

• Ecosystem side-effects - working out satisfactory multilateral solutions to complications such as privacy and security

Related Posts: Pay as you drive (Oct 2006) (June 2008) (April 2009)

Outside-In Architecture

Among many other interesting topics in his latest post (Unanswered Questions ...), John Hagel distinguishes between "inside-out" and "outside-in" architectures.

For Hagel, inside-out architectures are those that have evolved in many large organizations over the past couple of decades: starting from centralized IT architectures, gradually embracing distributed computing (client-server) and SOA, and very cautiously moving towards B2B. Such architectures are generally transactional, and controlled ("held captive" says Hagel) by IT architects.

A key challenge for inside-out architectures is the ability to connect and coordinate activities across large numbers of independent business partners, and to scale up to large and highly complex service-based ecosystems. Hagel believes that outside-in architectures will provide better support for complex collaborative ecosystems. He calls this relational - not using this term in the traditional IT sense ("relational databases"), but focused on supporting business relationships (and long-term business transactions) rather than short-term transactions.

The term "outside-in" seems to have a range of overlapping meanings, including:

• Looking at your business from the customer's perspective. There are several consultancies that claim to offer this as a service. For example, here's a curious page on the Outside-In company - with painfully bad graphic design and a revealing spelling mistake ("ourside in"). See also High-Yield Methods.

• Business-driven systems. For example, Zapthink uses the term to refer to "flexible business processes that respond to the way humans work, rather than processes that constrain humans to work the way the systems want them to work".

• Strategic Outsourcing. See for example What India and China can teach us about Innovation by Hagel and Seely-Brown.
  

• Outside-in SOA. This term seems to have been coined by Dave Linthicum, and refers mainly to interaction with external services (SaaS). See also Chuck Allen.
  

These are all very interesting trends to be sure, but is there a deeper pattern underlying some or all of them?

• Some of them are largely about extending the scope of the solution - finding new (more cost-efficient, more flexible) ways of satisfying the same old requirements. I think strategic outsourcing and SaaS come under this heading.

• Some of them are about extending the scope and perspective of the requirements - trying to add value to the customer's process, not just your own internal processes.
  

• And some of them hint at the need to reframe architecture itself - looking at the positive space between systems and organizations, not just the systems themselves.
  

Hagel avers that outside-in architectures must be "designed from the outset to support sustained collaboration". I agree that these opportunities call for a new kind of architectural thinking, and I agree that it may not always be easy to make the transition from existing inside-out architectures; but I think the evolutionary path is possible, given sufficient business motivation.

Service Ecosystem and Market Forces

One of the problems with a network of services is that the responsibilities and costs and risks are often in the wrong place.

In this post I'm going to explain what I mean by this statement, outline some of the difficulties, and then make some modest proposals.

The statement is based on a notion of the efficiency of an ecosystem. If there is one service provider and a thousand service consumers, it may be more efficient for the ecosystem as a whole if the service provider includes some particular capability or responsibility within the service, instead of each service consumer having to do this. In addition to the economics of scale, there may be economics of governance - for example, increased costs of managing the service relationship, especially if the service provider doesn't provide a complete service (in some sense).

One important application of this idea is in security, risk and liability. There is a very good discussion of this in the recent British House of Lords Science and Technology Committee Report into “Personal Internet Security", who specifically address the question whether ISPs and banks should take greater responsibility for the online security of their customers.

"A lot of people, notably the ISPs and the Government, dumped a lot of the responsibility onto individuals, which neatly avoided them having to shoulder very much themselves. But individuals are just not well-informed enough to understand the security implications of their actions, and although it’s desirable that they aren’t encouraged to do dumb things, most of the time they’re not in a position to know if an action is dumb or not." [via LightBlueTouchpaper]

In other words, the responsibility should be placed with the player who has (or may reasonably be expected to have) the greatest knowledge and power to do something about it. In many cases, this is the service provider. Some of us have been arguing this point for a long time - see for example my post on the Finance Industry View of Security (June 2004).

Similar arguments may apply to self-service. When self-service is done well, it provides huge benefits of flexibility and availability. When self-service is done poorly, it merely imposes additional effort and complexity. (Typical example via Telepocalypse). Some service providers seem to regard self-service primarily as a way of reducing their own costs, and do not seem much concerned about the amount of frustration experienced by users. (And this kind of thing doesn't just apply to end-consumers - similar considerations often apply between business partners.)

But it's all very well saying that the service provider ought to do X and the service consumer ought to do Y. What if there is no immediate incentive for the service provider to adopt this analysis? There are two likely responses.

1. "We don't agree with your analysis. Our analysis shows that the service consumer ought to do X."
2. "We agree it might be better if service providers always did X. But our competitors aren't doing X, and we don't want to put ourselves at a disadvantage."

More fundamentally, there may be a challenge to the possibility of making any prescriptive judgements about what ought to happen in a complex service ecoystem. This challenge is based on the assertion that such judgements are always relative to some scope and perspective, and can easily be disputed by anyone who scopes the problem differently, or takes a different stakeholder position.

Another fundamental challenge is based on the assertion that in an open competitive market, the market is always right. So if some arrangement is economically inefficient, it will sooner or later be replaced by some other arrangement that is economically superior. On this view, regulation can only really achieve two things: speed this process up, or slow it down.

But does this mean we have to give up architecture in despair - simply let market forces take their course? One of the essential characteristics of an open distributed world is that there is no central design architectural authority. Each organization within the ecosystem may have people trying to exercise some architectural judgement, but the overall outcome is the result of complex interplay between them.

How this interplay works, whether it is primarily driven by economics or by politics, is a question of governance. We need to spell out a (federated?) process for resolving architectural questions in an efficient, agile and equitable manner. This is where IT governance looks more than ever like town planning.

Notes

The House of Lords Science and Technology Committee Report into “Personal Internet Security" was published on August 10th 2007 (html, pdf). Richard Clayton , who was a specialist adviser to the committee, provides a good summary on his blog. Further comments by Bruce Schneier and Chris Walsh.

Three Types of Requirements Engineering

What is the starting point for an SOA modeling exercise? Traditionally, we have often understood the purpose of modeling in terms of the design of a specific or generic solution to some business problem or opportunity. But with the service-based business, we may have to start further back – modeling the business ecosystem in order to identify problems and opportunities in the first place.

In the service economy, the business focus is to provide value-adding services to its customers. What you as a business have to decide is: what is a reasonable and economically viable set of services to offer. And then you build a platform of services. This is not a complete solution to one problem – it is a partial solution to many different problems, in many different contexts. That's how you get reuse, economics of scale and scope, and ultimately business viability.

So we have three types of requirements engineering, as shown in the following table.

Solution-Driven (Specific)	Solution-Driven (General)	Evolution-Driven
Identify Business Problem Identify "Users" Negotiate Requirements Define Solution	Identify Domain Identify Domain Experts Define Requirements Design Solution Kit	Identify Ecosystem Identify Services Procure and Release Devices

Extract from Business Modelling for SOA (CBDI Journal, March 2005)

Business Geometry

A business or value chain is composed in a geometric structure. (In the past we have often called this structure an architecture, but this word has lots of different and tangential associations for different people.)

In SOA, we design a business or value chain as a network of services. This is a powerful geometrical pattern. But there may be many possible network geometries satisfying a given business requirement, all of which count as satisfying the principles of SOA. For example: hub/spoke or peer/peer.

Business Stack

Another common geometric pattern for SOA is stratification. A business process is composed of services from a set of lower-level services, presented as a platform. A good example of a business platform is the set of retail services offered by Amazon and eBay. Other service providers have built further retail/logistical services on top of the Amazon/eBay platforms.

Each platform is in turn built upon even lower services. At the lower levels, there may be collections of IT-based services, known as ESB. There may also be sociotechnical service platforms, such as call centres.

Thus we have a stratified geometry, in which a person tackling a problem at a given level is presented with a collection of available services, formed into a virtual platform. This can be thought of as a business stack, with one plaform stacked on top of another. Some of the lower layers of the stack may appear to be purely technical services; however a more complete picture should reveal the existence of an IT organization maintaining the platform, in other words a human platform of administrators and programmers supporting the technical platform.

Variable Geometry, Variable Granularity

While the SOA principles may provide some geometrical guidance, and mandate certain geometrical patterns, there is still a design job to determine the geometry. This design job may be easy when the requirement is trivial, but gets harder as the complexity increases.

In many situations, the demand side has more variation than a human designer (or design team) can accommodate. (We characterize this as an asymmetry of demand, which calls for a process of asymmetric design.) We need to start thinking about variable geometry solutions, where the geometry itself can be adapted on-demand.

For example, in the past we have assumed that granularity has to be fixed at design time. But we can conceive of a web service platform that detects patterns of demand-side composition, defines new composite services automatically, describes and publishes these new services in real time, and notifies likely users of the new service, complete with an incentive to switch. We can conceive of a web service platform that analyses the message content of a certain service, and produces a substitute service with a smaller footprint that would satisfy most of the uses in a more elegant way.

Value Landscape

I use the term value landscape to refer to the distribution of cost, benefit and risk across a complex market ecosystem, such as the insurance industry. Technology (including SOA) influences business geometry, not least because it affects transaction costs. The shape of the value landscape changes (has already started to change) as the result of B2B and B2C and BPO. Companies that once occupied safe market positions (the high ground) may find their commercial advantage slipping into the sea, or they may find themselves cut off from their natural markets or supply chains.

See Microsoft Blog Insurance Value Chain

Let's suppose an insurance company has the following strategic aims:

1. Profitability, short-term viability. To deliver the maximum service value as cost-effectively as possible, using available input services and technologies as efficiently as possible, with minimum costs/risks of change.
2. Adaptability, medium-term viability. To understand and respond to changing demand for insurance services, and to trends in cost and risk, both internally and across the industry. To develop and deploy new services to exploit new business opportunities and avoid emerging business threats.
3. Survival, long-term viability. Making sure the core business proposition remains valid, and doesn't get eroded by more agile players. Taking strategic action in relation to structural changes in the insurance industry.

If we are doing business geometry for an insurance company, we need to think about the insurance industry as an ecosystem. We need an AS-IS model of the present ecosystem (largely based on pre-SOA technologies) and a TO-BE model of an idealized ecosystem (based on SOA). We can expect the pre-SOA ecosystem to evolve into some form of SOA ecosystem, although we may not have much idea which of the possible changes is going to happen first. To satisfy all three strategic aims, an insurance company needs to exploit the pre-SOA ecosystem, and prepare for the SOA ecosystem.

Note that this situation may force the insurance company to implement a variable geometry, both in the organizational platform and in the IT platform. Otherwise, it will either have to operate suboptimally for an extended period, or incur significant organization costs and IT costs every time the industry takes another step towards SOA.

Amazon and Ebay

Amazon, eBay and PayPal are creating platforms for eCommerce.

• Amazon Web Services
• eBay Developers Programme
• PayPal Merchant Tools

• Reprogramming Amazon (Business Week, December 2003)
• Amazon CTO: 'We've Just Scratched The Surface' (Information Week, July 26, 2004)

At CBDI Forum, we have been tracking these developments from time to time. See news commentaries: July 2004, February 2004, February 2004, July 2003. See also Jeff Bezos and Ecosystem Thinking.

---

Jeffrey McManus of eBay claimed recently that 'we make inefficient markets efficient'. (Source: Aaron Johnson blog.)

In his blog, Jonathan Schwartz (Sun Microsystems) talks about selling computers on eBay, as a web services experiment. Schwartz expects this experiment to provide "unvarnished" pricing information, as well as providing practical experience with a service-oriented supply chain. This seems to support the McManus claim.

However, efficiency often conflicts with adaptability. How wise is it for Sun and other firms to link to the eBay platform? What is the likely ecosystem effect of allowing eCommerce services to be defined and controlled by a small number of like-minded firms?

I've just had a look at the eBay technical documentation. Developers must follow detailed procedures for design, testing and certification, before they are permitted to plug into the live eBay environment. Rightly so no doubt, but this raises questions about service life cycle management and technical change.

In response to the possible difficulties of raw eBay, a number of service providers are now offering DropOff services that provide a simple eBay front-end. These include AuctionDrop, AuctionWagon, QuikDrop and Picture It Sold! and 1StopAuctions. The last of these makes strong and explicit claims about the difficulty of using eBay. (Success here means not just completing the transaction but getting a good price.)

"To successfully sell an item on eBay requires marketing and selling strategies unique to eBay – and learning those strategies can take years to master. 1Stopauctions.com has that unique knowledge to successfully sell your items on eBay. We have been buying and selling on eBay since 1998. Over the years we have found the secrets of launching successful listings to obtain the highest possible price ... and so on ..."

For me, the appearance of these secondary service providers simply confirms that the big issue is business hegemony (power) - the value ladder now belongs to Amazon and eBay, they control the terms on which everyone else is trading.


Lawrence Wilkes, Amazon and eBay Web Services (CBDI Journal, October 2004)

Trusting Services

If there is only one implementation of a service then trusting the service entails trusting the implementation entails trusting the supplier. Furthermore, if there is only one channel to access this implementation, then this also entails trusting all intermediaries.

Typically this is done by having a large trusted supplier with a reputable brand. IBM or EDS. Large suppliers have a valuable brand image to maintain, and therefore may be supposed to have too much at stake to allow any service to fail.

We can break into this loop in three places. Firstly, we may note the ability of some firms to decouple appearance from reality. Reputation is affected not by real failure but by the perception of failure.

Secondly, we may be able to decouple the implementation from the supplier, through some kind of guaranteed escrow. If the supplier cannot deliver the service with the specified level of quality, then we or someone else will immediately take over the implementation (together with all resources required, such as persistent data). We should expect the web service platforms to support this.

More importantly, we should be able to decouple the service from a single implementation. It might be stupid to rely on a business critical service if there is only one small supplier. But if there are lots of interchangeable suppliers offering equivalent services, the disappearance of
one supplier doesn't matter very much.

Thus web services allows us to shift from trusting a single source to trusting multiple sources (market or ecosystem). Without this step, the service-based business contains many dependencies, and the business process may seem critically vulnerable.

The process implications of this are twofold. Firstly, the SOA process should include some consideration of the appropriate number/diversity of implementations of a given process - and should certainly not assume that the design task is always to select a single best-possible implementation. Secondly, the enterprise model of an SOA solution should allow some visibility and management of the true dependencies and consequent risk involved, since multiple independent implementations of a given service will generally be expected to increase the overall availability and reduce the overall risk.


Updated 11 December 2013

Jeff Bezos and Ecosystem Thinking

Lots of commentators and blogs have picked up on a recent Business Week story about Amazon (December 22, 2003).

But Amazon has been working towards this for a long time indeed. When I searched the web for Bezos and ecosystem, the first page of hits included a story from June 2000, in which a senior manager at HP acknowledged that he had gotten ecosystem thinking from talking to Jeff Bezos.

Recent developments by Amazon and eBay (as described by David Sprott and Lawrence Wilkes at the CBDi Forum) establish a business service stack for the eCommerce sector.

But this isn't just a clever tactical move. It follows logically from a strategic direction that Amazon has been pursuing for a considerable time. During the dot-Com boom, there was lots of superficial admiration of Amazon, and lots of companies trying to emulate it. But few of them mastered ecosystem thinking, and few of them survived the downturn.

---

Alan Deutschman, Inside the mind of Jeff Bezos (Fast Company, 1 August 2004)

Steve Hardy, Guidelines from the Book of Bezos (Creative Generalist, 5 August 2004)

Robert D Hof, Reprogramming Amazon (Business Week, 22 December 2003)

David Jastrow, HP Keynote embraces ecosystem thinking (CRN, 15 June 2000)

Lawrence Wilkes, Amazon and eBay Web Services (CBDI Journal, October 2004)


Related posts: Binary Advice (July 2004), Amazon and Ebay (August 2004), Business as a Platform - Amazon (May 2012), The Idea of Showrooming (July 2017)

For more on ecosystem thinking, see my 2001 book on Component-Based Business. Available from Amazon (of course).


links updated 11 March 2020