Strategy and Requirements for the API ecosystem

Is there a framework or methodology for establishing the business / ecosystem requirements to drive API strategy and development?

At an industry event I attended recently, hosted by a company that sells tools and technologies for the API ecosystem, some of the speakers advised that when presenting to non-technical stakeholders, you need to talk about service value/benefit rather than APIs. But this raises an important question, how to identify and quantify service benefit, and how to negotiate share of value between different players in the ecosystem?

One of the ideas of the API economy is that you don't have to maintain all the capabilities yourself, but you find other enterprises that can provide complementary capabilities. So you need to identify and understand what capabilities are available, and map combinations of these capabilities against the demands and unfulfilled needs of potential customers. Then having identified in broad terms what capabilities you wish to combine with your own, and worked out where the service boundaries should be, you may select organizations to partner with and agree business and commercial terms, or create a platform to which many third parties can connect. The technical design of the API should then reflect the service boundaries and commercial arrangements.

In the early days of service-oriented software engineering, people always wanted us to tell them how large their services should be. Not just macro versus micro, but broad (generic) versus narrow (specific). To what extent should a service be completely purpose-agnostic - in other words, with no restrictions on how or where it may be used - or does this conflict with other design goals such as reliability or data protection?

The answer is that it depends not only on what you are trying to do, but how you want to manage and govern your service architecture. A broadly scoped, purpose-agnostic service (or service platform) can achieve wide usage and economies of scale, but may be more complex to configure, test and use, whereas a more narrowly scoped context-specific service might be easier to use but with lower reuse potential. Among other things, this affects how much of the service composition and orchestration can be done by the service provider (supply side), and how much is left to the service consumer (demand-side). And even on the supply side, it affects how much work needs to be done by the integration experts ("town planners"), and how much can be left to citizen integration ("pioneers" and "settlers").

One version of this challenge can be found in large global organizations, working out exactly what functionality should be provided centrally as shared services, and what functionality should be left to local operations. Ideally, the service architecture should be aligned with the business and organizational architecture.

The word "economy" also implies attention to accounting issues - sharing costs and benefits between different players. Although we may regard cloud as almost infinitely extensible, this doesn't come without cost: if the number of service calls goes through the roof, someone has to pay the cloud provider's bill. This is already an issue within large organizations, where we commonly find arguments about whose budget will pay for something. And I have seen some great ideas come to nothing, because the benefits were spread too thinly and nobody was able to fund them.

So although vague appeals to innovation and imagination might be good enough for a marketing pitch, serious strategic thinking is about discovering where there is untapped value in your business and its environment, and working out exactly how an API strategy is going to help you unlock this value.

---

At the CBDI Forum, we were talking about these issues many years ago: our Service Architecture and Engineering® methodology is still available from Everware-CBDI. Here are some of the articles I wrote for the CBDI Journal.

• Identifying Web Services (February 2002) 
• Service Economics (September 2005)

More at  http://everware-cbdi.com/cbdi-journal-archive

The Hungry Tapeworm

This week, three American companies announced a joint venture to sort out healthcare for their own employees. Ambitious, huh?

This is not the first time large American companies have tried to challenge tho market power of healthcare providers. According to Warren Buffett, "the ballooning costs of healthcare act as a hungry tapeworm on the American economy". Intel and Walmart are among those that have previously ventured into this area. In 2016, 20 companies including Coca Cola, American Express, IBM and Macy’s joined the Health Transformation Alliance (HTA). So why should anyone take this latest attempt seriously? Only because the three companies are Amazon, Berkshire Hathaway and JPMorgan Chase. And Amazon (need I remind you?) eats everyone's lunch.

• 2010: Amazon Is Still Eating eBay's Lunch

• 2012: How Amazon ate Sears' lunch

• 2017: Hey Walmart, Amazon is eating your lunch

• 2017: Amazon Eats Google’s Lunch

• 2017: Why Amazon is eating the world

John Naughton sees this as a typical play for a data hungry tech giant, based on two hypotheses.

• Transactional data will lead to transactional efficiencies. The joint venture starts with the three companies experimenting on their own employees, who will "tell Amazon and its algorithms what works and doesn’t work". 

• "Mastery of big data might yield clinical benefit".

As Pressman and Lashinsky note, the experiment is based on a pretty good sample of Americans: "a diverse workforce spanning low-wage normal folk to the most elite of our society".


Amazon is obviously a major player in the data and analytics world, but so is IBM, which is playing an important role in the HTA. Not only is IBM a corporate member, but IBM Watson Health will do the data and analytics. According to Pharmaceutical Commerce, it will "aggregate participating HTA member companies' data, enabling insights both into outcomes of medical interventions, as well as wellness initiatives to improve employees’ health".

And what about Google? Google Health was discontinued in 2011, following a lack of widespread adoption. Perhaps data isn't the whole story.


But Amazon is not just about data. In an article published before this announcement, Zack Kanter attributes Amazon's strategic dominance to SOA. "Each piece of Amazon is being built with a service-oriented architecture, and Amazon is using that architecture to successively turn every single piece of the company into a separate platform — and thus opening each piece to outside competition."

Moazed and Johnson discuss the platform implications of the healthcare announcement. They argue that "platforms thrive with fragmentation, not consolidation", and that "the new platform needs to offer enough potential scale to outweigh those risks, otherwise manufacturers may be too afraid to join". Sarah Buhr sees this as an opportunity for smaller players, such as Collective Health.

Three employers, even large ones, probably won’t have enough muscle to negotiate fair prices for healthcare and pharma. But if Bezos can create the right expectations, and provide a flexible platform for smaller players ...

---

Health Transformation Alliance sets its 2017 agenda (Pharmaceutical Commerce, 9 March 2017)

Amazon alliance takes on ‘hungry tapeworm’ of healthcare costs (Pharmaceutical Technology, 1 February 2018)

Sarah Buhr, Collective Health Wants To Replace The Health Insurance Industry With A Software Program (TechCrunch, 11 Aug 2014)

Sarah Buhr, Amazon’s new healthcare company could give smaller healthtech players a boost (TechCrunch, 30 Jan 2018)

Paul Demko, Amazon's new health care business could shake up industry after others have failed (Politico, 30 January 2018)

Zack Kanter, Why Amazon is eating the world (TechCrunch, 14 May 2017)

Paul Martyn, Healthcare Consumerism: Taming The Hungry Tapeworm (Forbes, 30 January 2018)

Alex Moazed and Nicholas L Johnson, Amazon's Long-Awaited Health Care Platform (Inc, 30 January 2018)

John Naughton, Healthcare is a huge industry – no wonder Amazon is muscling in (Observer, 4 February 2018)

Aaron Pressman and Adam Lashinsky, Data Sheet—Why Jeff Bezos Just Might Crack the Health Care Challenge (Fortune, 31 January 2018)

Jordan Weissmann, Can Amazon, Berkshire Hathaway, and JPMorgan Revolutionize Health Care? (Slate, 30 Jan 2018)

Wikipedia: Google Health


Updated 5 February 2018

Lawrence Wilkes

My friend and former colleague Lawrence Wilkes died on Friday, after a short illness. Lawrence and I joined James Martin Associates (JMA) on the same day in 1986, so we had known each other for half a lifetime.

JMA was a small consultancy advising organizations on the use of the Information Engineering Methodology and assisting Texas Instruments in developing and supporting the IEF toolset. The Information Engineering part of the company was acquired by Texas Instruments Software in 1991. In 1997, it was sold to Sterling Software and many of us left the company. David Sprott and Lawrence set up the CBD Forum (later CBDI Forum), as a think tank for component-based development, evolving into component-based development and integration, and then evolving into service-oriented architecture (SOA).

As David has written in his fulsome tribute, he and Lawrence spent several years explaining SOA to the large technology companies, including IBM, Intel, Microsoft and Sun. (I can add that they had an article on SOA in the very first issue of the Microsoft Architecture Journal, and I co-wrote something with him for the second issue.)

For my part, I collaborated frequently with them, and became a regular contributor to the monthly CBDI Journal. When the CBDI Forum merged with the US-based consulting firm Everware, I joined Everware-CBDI as a full-time consultant for a few years, working with Lawrence and others to develop a substantial knowledgebase for service architecture and engineering. Although many of us contributed content, it was Lawrence who provided the overall structure and turned our contributions into a coherent whole.

Lawrence was a tireless innovator and perceptive industry analyst, generous with his energy and insight to colleagues and friends. It was a shock when I learned of his illness and forced retirement, and a further shock to learn of his quick demise. I will miss him.

---

Links

Lawrence Wilkes Blog, Slideshare

David Sprott and Lawrence Wilkes, Understanding Service-Oriented Architecture (Microsoft Architecture Journal 1, January 2004)

Lawrence Wilkes and Richard Veryard, Service-Oriented Architecture: Considerations for Agile Systems (Microsoft Architecture Journal 2, April 2004)

David Sprott, Remembering Lawrence Wilkes – SOA Pioneer (30 April 2017)

CBDI Journal Archive

Twin-Track Architecture

#entarch This post follows discussions with Graham Berrisford of Avancier about the relationship between Enterprise Architecture (EA) and Solution Architecture (SA).

---

What seems to make sense is to describe EA/SA as a twin-track process (similar to the twin-track process that aligns service development with solution development in SOA).

Twin-track is essentially an abstract process pattern, used to align two activity streams at different levels of granularity. As far I recall, I first encountered it in the Select Perspective methodology, described in two books by my friend and former colleague Paul Allen. The CBDI SAE methodology also uses the twin-track approach.

In SOA, there is a stream of activity to produce and maintain services, and another stream of activity to use these services to build solutions. These two streams need to be connected but not synchronized. The two tracks may be separated organizationally - split into different teams or org units, or even separate companies - provided that there are suitable governance mechanisms for allocating resources and responsibilities, and resolving issues. People may specialize in one or other stream.

A twin-track processes doesn't assume either top-down or bottom-up. In some cases, you could start with the solution requirements and then specify the desired services (top-down). In other cases, you could start with a set of available services, and then assemble these into solutions (bottom-up). In practice, the twin-track approach accommodates a mixture of top-down and bottom-up. The important point however is that the two have to be connected at a series of touch-points.

We can now describe the relationship between enterprise architecture and solution architecture in similar terms. If solution architecture is disconnected from enterprise architecture, then the solution architects are likely to produce point solutions instead of enterprise solutions. Conversely if enterprise architecture is disconnected from solution architecture, then it is likely to produce grand, simplistic and ultimately unimplementable schemes. Again, we don't have to assume that either EA or SA is dominant, but that they work together towards common goals.

---

For other posts on twin-track: browse, subscribe. 

Service Boundaries in SOA

A service has explicit boundaries

This is one of the four tenets of SOA, promulgated by Microsoft and others around 2004. In its (now discontinued) Connected Services Framework (3.0 SP1), it is identified as one of the four principles of service-oriented design.

• Boundaries are explicit
• Services are autonomous
• Services share schema and contract, not class
• Service compatibility is determined based on policy

I found a presentation on the MSDN website by Udi Dahan called SOA in the Real World (ppt), in which this tenet is expanded as follows.

Services run in a separate process from their clients
A boundary must be crossed to get from the client to the service – network, security, …

Also on the MSDN website, I found some design guidance from ramkoth called Service Boundaries, Business Services and Data, in which the same tenet is expanded as follows.

Services explicitly choose to expose certain (business) functions outside the boundary. Another tenet of SOA is that a service - within its boundary - owns, encapsulates and protect its private data.

In SOA, dollar signs and trust boundaries, Rocky Lhotka describes SOA as a mechanism for bridging trust boundaries. He argues that SOA is expensive (in terms of complexity and performance), and can only decrease the overall cost and complexity when used for inter-application communication across trust boundaries. Trust is not just a security issue, but also a future-proofing issue, because we can't always be sure what future applications will do. A trust boundary therefore helps protect us from an uncertain future, and builds some degree of flexibility into the system of systems.

Where did all the boundaries go?

Now here's the funny thing. All of the pieces I've quoted above were published in 2004, as are the vast majority of hits in the first several pages of Internet search. Apart from one Bill Poole, who published a few blogposts in early 2008 on Service Boundaries, the internet seems to have more or less dropped the concept of service boundary. So am I looking in the wrong place, or has the internet been infected by the misleading rhetoric of boundarylessness?

I'm also struck by the apparently rapid disappearance of something that had been promulgated as an SOA principle. If you think that principles are timeless truths, think again.

Update

In his latest post Service Boundaries Aren’t Process Boundaries Udi Dahan replies with a correction of his 2004 presentation. He points out that his later posts (from 2007 onwards) no longer assert that services must run in a separate "system" or "process". (One reason for this is that the concepts of "system" or "process" belong in a different architectural view.) He still appears to believe in explicit service boundaries, but he is now thinks it's more appropriate to base these on business capabilities. Meanwhile in a new post on Service Boundaries, Lawrence Wilkes outlines the CBDI position, explains that the service boundaries are orthogonal to various other boundaries, including resource, technology and organizational boundaries, and shows how services can be used to cross these boundaries.

Ecosystem SOA 2

What are the problems of large complex sociotechnical systems? How far do SOA and enterprise architecture help to address this problem space, and what else might we need?


When I started writing about SOA and the service-based business over ten years ago, I defined two "cuts" across the service ecosystem. One cut separates inside from outside, while the other cut separates supply from demand.

(This diagram was included in my 2001 book on the Component-Based Business, and frequently referenced in my work for the CBDI Forum. For a brief extract from the book, see my Slideshare presentation on the Service Ecosystem.)

The inside/outside cut is sometimes called encapsulation. It decouples the external behaviour of a service from its internal implementation, and can be described in terms of knowledge - the outside has limited knowledge of the inside, and vice versa. (The cut is also sometimes called transparency - for example location transparency, which means that external viewers can't see where something is located.)

The supply/demand cut is about delegation, and can be described in terms of responsibility. Getting these two cuts right may yield economics of scale and scope; and the business case for SOA as a development paradigm is often formulated in terms of reusing and repurposing shared services.

For relatively small and simple SOA projects, it may be feasible to collapse the difference between these two cuts, and treat them as equivalent. (The inside/outside relationship and the supply/demand relationship are sometimes both described as "contracts", although they are clearly not the same kind of contract.) However, enterprise-scale SOA requires a proper articulation of both cuts: confusing them can result in suboptimal if not seriously dysfunctional governance and procurement. Many people in the SOA world still fail to understand the conceptual importance of these cuts, and this may help to explain why some organizations have had limited success with enterprise-scale SOA.

Going beyond enterprise SOA as it is generally understood, there is a third cut separating two views of a system: the system-as-designed (whose structure and behaviour and rules can perhaps be expressed in some formal syntax such as UML, BPMN or ArchiMate) and the system-in-use (whose actual performance is embedded/situated in a particular social or business context). This cut is critical for technology change management, because of the extent to which the designed system underdetermines the pragmatics of use. I have been talking about this cut for over twenty years, but only more recently working out how to articulate this cut in composition with the other two cuts.

One important reason for looking at the pragmatics of use is to understand the dimensions of agility. In many settings, we can see a complex array of systems and services forming a business platform, supporting a range of business activities. If no agility is required in the business, then it may not matter if the platform is inflexible, forcing the business activities to be carried out in a single standardized manner. But if we assume that agility is a critical requirement, then we need to understand how the flexibility of the platform supports the requisite variety of the business.

More generally, understanding the pragmatics of use leads to the recognition of a third kind of economic value alongside the economics of scale and the economics of scope: the economics of alignment. The value of a given system-of-systems depends on how it is used to deliver real (joined-up) business outcomes, across the full range of business demands. (I'm afraid I get impatient with people talking glibly and simplistically about business/IT alignment, without paying attention to the underlying complexity of this relationship.)

Understanding these three cuts (and analysing their implications) is critical to understanding and managing a whole range of complex systems problems - not just SOA and related technologies, not even just software architecture, but any large and complex sociotechnical systems (or systems-of-systems). If the three cuts are not understood, the people in charge of these systems tend not to ask the right questions. Questions of pragmatics are reduced to questions of platform design; while questions of the cost-justification and adoption of the platform are reduced to a simple top-down model of business value. Meanwhile the underlying business complexity (requisite variety) will be either misplaced (e.g. buried in the platform) or suppressed (e.g. constrained by the platform).

So there are three challenges I face as a consultant, attempting to tackle this kind of complex problem. The first challenge is to open up a new way of formulating the presenting problem, based on the three cuts. The second challenge is to introduce systematic techniques for analysing the problem and visualizing the key points. And the third challenge is to identify and support any organizational change that may be needed.

---

With thanks to Philip Boxer and Bernie Cohen. For a different formulation of the three cuts, together with a detailed example, see their new paper "Why Critical Systems Need Help to Evolve" Computer, vol. 43, no. 5, pp. 56-63, May 2010, doi:10.1109/MC.2010.150. See also Philip Boxer, When is a stratification not a universal hierarchy? (January 30th, 2007)


Related post Ecosystem SOA (October 2009)

SOA and Risk Management

#soa #risk In this post, I identify some contrasting views on the relationship between SOA and risk.

SOA involves innovation, and innovation always introduces new risk

• SOA Benefits, Challenges and Risk Mitigation (Eric Roch, 7 March 2006)
• How SOA increases your security risk (Gartner via ComputerWorld, March 2007)
• SOA Benefits outweigh risks (IBM via SearchSOA, April 2008)
• SOA adds risk to an activity that is already “success challenged” (CIO Update, August 2008)
  

SOA helps reduce risk

• SOA Reduces Risk and Cost in Shared Services Projects (Merrill Douglas, Public CIO Magazine, 7 May 2008)
• SOA Helps Cadbury Schweppes Reduce Risk and Increase Flexibility (CapGemini case study)
• SOA may be taking the risk out of cloud computing (EDS via ZDNet, April 2009)

SOA providing visibility and control of aggregate risk and unexpected behaviour

• CEP and real-time risk (Chris Martins, Progress Apama, March 2008)
• Exclusive Interview With Nick Leeson: An Inside Look at Rogue Trading (Advanced Trading, March 2008) (via Aleri)

SOA complicating visibility and control of aggregate risk

• Can a Service-oriented Architecture Hinder Sarbanes-Oxley Compliance Efforts? (Internal Auditor, undated)

Therefore ... risk management as one area likely to see spending increasing

• Aleri October 2008
  

Oh yeah? Any evidence of this?

What does Application Modernization really mean?

@cbdi #SOA My friends at the CBDI Forum have been applying their expertise in Service-Oriented Architecture to the question of Application Modernization. See David Sprott's blogpost The meaning of life and application modernization (Jan 2010) See also Application Modernization Special Issue (CBDI Journal December 2009).

Of course, people have been talking about Application Modernization for many years (see note), but David's approach is strongly based on his favourite architectural principles - not just SOA principles (loose coupling, abstraction, and so on) but also model-driven development.

In general terms, modernization contains two opposing ideas - separation (Latour calls this "purification") and mixture ("hybridization"). In application modernization, separation of concerns is a well-established architectural notion, related to modularization and encapsulation. Hybridization refers to things that are neither one thing nor the other - for example, the creation of processes and systems that are neither wholly automated nor wholly manual, neither wholly inhouse nor wholly outsourced, but some complex and potentially volatile mixture. This means creation of highly reusable components and services, that can be redeployed and repurposed in unforeseen ways.

---

Bruno Latour, We have never been modern (trans Catherine Porter, Harvard University Press, 1993)


Peter J. Leithart, Semi-Modernism (November 2006) We Have Never Been Modern (October 2013) "We Have Never Been Modern" (December 2014)

Wikipedia: Separation of Concerns

---

Note: In October 2006, HP, Intel and Oracle jointly launched an Application Modernization Initiative (HP Press Release. See also Dana Gardner, Not just a nip and tuck, application modernization extends the lifecycle of legacy code assets, October 2006). During 2008, a couple of niche application modernization companies were acquired: Jacada by Software AG (ebizQ, January 2008) and Relativity Technologies by Microfocus (Press Release, December 2008). IBM's webpage on Business application modernization services is dated October 2008.

About this blog - not just SOA

Prompted by a rude comment about me that my pal Lawrence found on a Google group, I spent the weekend thinking about the title of this blog.

It's no longer just about Service-Oriented Architecture (SOA), if indeed it ever was. I have always spent a lot of space talking about business architecture and enterprise architecture, believing that these topics are important in their own right, as well as essential for getting SOA right. On the more technical side, I have talked about process architecture and event architecture, as well as service architecture and engineering - again with the belief that these topics must be linked somehow. There are architectures of trust and security, architectures of knowledge, learning and intelligence, and architectural issues going beyond the boundaries of the enterprise into the ecosystem.

So I'm going to try calling it Richard Veryard on Architecture. SOA fans will continue to find useful and thought-provoking ideas about service architecture and the service-based business, and I hope to reach a few more SOA sceptics as well. Meanwhile, you may also wish to subscribe to my other blogs: Richard Veryard on Computing for general stuff on the software industry, and Demanding Change for systems thinking and organizational transformation. And I am richardveryard on Twitter.

SOA Planning and Subsidiarity

The principle of subsidiarity states that central planning only applies to those things that require central planning.

Applying this principle during the planning process means finding an appropriate level of consistency and sharing of policy and services and infrastructure. The concept of subsidiarity refers to the scoping level at which a given set of actions and outcomes are coordinated. Which aspects can be (or must be) determined locally, and which aspects can be determined centrally? Which aspects are managed at department level, or at sector level, or at national level? This calls for architectural thinking about management.

In the early phases of SOA adoption, the primary challenge may be to constrain departmental autonomy over some key issues. However, in the later phases of SOA adoption, SOA-based thinking can be used progressively to decouple enterprise activity. Business transformation may sometimes reduce the need for tight synchronization between different processes, while technology and common standards help reduce the interoperability risks. Such changes may have the effect of empowering a degree of greater local autonomy and differentiation (diversity) against a common platform of shared services. This outcome becomes progressively available as the SOA adoption becomes more mature.

Ecosystem SOA

The SOA world is finally catching up with some of the ecosystem ideas that I published in my 2001 book on the Component-Based Business (see my Slideshare presentation) and developed further in several articles and presentations for the CBDI Forum over a number of years.

• Supply and Fit (September 2000)
  
• Identifying Components 2 (October 2001)
• Opportunities in the Insurance Ecosystem (December 2004)
• Business Strategy Planning for the Service Economy (May 2006)
  

The biological approach to creating business and software services is radically different to the solution-driven approach, and is based on biological and ecological metaphors.

• First we identify an ecosystem, which may contain both human users and existing artefacts.
• Then we identify services that would be meaningful and viable in this ecosystem.
• Then we procure devices that enable the release and delivery of these services into the ecosystem.

I previously defined Three Types of Requirements Engineering, and we can map these onto different styles of SOA.

Solution-Driven (Specific)	Solution-Driven (General)	Evolution-Driven
Identify Business Problem Identify "Users" Negotiate Requirements Define Solution	Identify Domain Identify Domain Experts Define Requirements Design Solution Kit	Identify Ecosystem Identify Services Procure and Release Devices
Experimental SOA	Enterprise SOA	Ecosystem SOA

(Some people use the term Web Oriented Architecture (WOA) for what I'm calling Ecosystem SOA.)


If we regard these as phases of maturity, then we can have a straightforward roadmap from left to right, as in for example the CBDI SOA Roadmap. However, some organizations may need to tackle these styles of SOA in parallel rather than in sequence.

---

A service portfolio plan for Enterprise SOA can be based on an enterprise model that identifies the capabilities of the enterprise and clusters these into domains. In the CBDI Forum's SAE methodology, the domains are classified as Core and Contextual according to a matrix derived from Geoffrey Moore. (Note how the domains migrate around the matrix over time.) See for example my blogpost Tesco outsources core eCommerce.



A similar model, derived from Amin and Cohendet's book Architecture of Knowledge, explicitly describes Core and Periphery in terms of knowledge intensity. In other words, the reason something is classified as Core is because it encapsulates some important (strategic) knowledge.

For example, an insurance company knows more about insurance than about cars, so when providing car insurance it may decide to partner with other organizations that know more about cars than about insurance. This results in a composition of insurance-related services and car-related services (for example, determining the insurable value of a used car). Such compositions can be either directed (in other words, composed by a single dominant player) or collaborative (in other words, emerging from the interaction between multiple players within the ecosystem).

For example, an online retailer knows about her products and services, but doesn't wish to become an expert in credit card handling, or to be responsible for data protection and security, so she delegates these concerns to a specialist provider that knows more about these matters.

Similar considerations can apply to industry consortia, such as ACORD (for the insurance market). ACORD can define generic service-based assets (such as models, schemas, interfaces and so on) for insurance. However, insurance companies will also wish to use generic service-based assets to cover requirements that are not insurance-specific, such as customer management or complaints handling, where ACORD may not be able to add any knowledge-value, and it would be appropriate for ACORD to regard these as peripheral to its own activities rather than core.

---

So one approach to Ecosystem SOA is to push out from the enterprise into the ecosystem. John Hagel calls this Inside-Out Architecture, which he contrasts with Outside-In Architecture. (See my post on Outside-In Architecture.)


An Outside-In Architecture starts with a model of (the flows of) knowledge and value in the ecosystem as a whole. The strategic question for an enterprise is how to find way of both contributing value to the ecosystem, and drawing value from the ecosystem, through the provision of ecologically viable services.



For example, a telecoms company might reasonably consider that its core competence is something to do with communications. So a positional strategy would drive it to a dominant position in the middle of a large communication ecosystem, providing a platform of services that add value to a diverse range of communication activities by other people. The dominant position would allow it to negotiate a strong share of the value generated.

However, respect for the ecosystem would lead it to leave sufficient value to third parties to maintain the economic health of the remainder of the ecosystem. Instead of a simple positional strategy, a relational strategy (based on mutual trust with ecosystem partners) should produce a more sustainable and ecologically sound ecosystem.

---

Service Ecosystem from Richard Veryard


Related post: Ecosystem SOA 2 (June 2010)

SOA as Multi-Sided Platform

One of the ways to think about the complexity of SOA is as a multi-sided platform.

Some of the pioneering work on multi-sided platforms has been published by Andrei Hagiu at the Harvard Business School.  (Follow links from his homepage to some of his recent papers.)

In November 2006, I posted a brief commentary on Two-Sided Markets, referencing his work, and indicating its relevance to a service-oriented business strategy.

As yet, relatively few people have made the connection between multi-sided markets and SOA. In June 2007, Richard Friedman posted something useful on Multi-sided Platforms for Business or Software, and I found a paper entitled Optimizing the Supplier Selection and Service Portfolio of a SOA Service Integrator presented to a 2008 conference by a group of German researchers.

One of the attractions of SOA is that it should allow people and organizations to collaborate more effectively and cost-effectively. From the supply-side perspective, this means collaboration between the users of your platform. Your platform may or may not support a rich and open variety of collaborations, including mashups and other third-party initiatives, and this richness and openness significantly affects the demand-side value of your SOA platform.

The critical economic question here is not the economics of scale on the supply-side but the economics of scope and agility on the demand-side. Thus the critical question for platform design is how open/closed these platforms are, and the extent to which they constrain (overdetermine) or enable (underdetermine) demand-side activity.

SOA Retrospective

As this is my last day as a full-time employee of the CBDI Forum, I thought I'd permit myself a little retrospective discussion on Service Oriented Architecture.

My own understanding of the "service" concept can be traced back to a number of things I was doing in the 1990s, including enterprise modelling for open distributed processing (ODP), as well as early structured methods for component-based development (CBDI or CBSE). From this perspective, it wasn't hard for me to see that the service concept represented a significant departure from the software paradigms I had learned at university (everything from SIMULA to PROLOG).

By the late 1990s, the software vendors were pushing component-based development. It was already obvious to some of us that the most important thing about a component was not the internal mechanism (its implementation) but the service it delivered. I started attending meetings of the CBDI Forum, and writing for the CBDI Journal. My main focus however was not software but business - developing a business modelling approach that would link component-based software with component-based business transformation. I developed a component methodology called SCIPIO, and in 2001 I published a book on the Component-Based Business, which tried to explain business architecture in terms of business components - coherent chunks of business capability, interacting through service relationships.

But although my friends at CBDI Forum understood this, we found that a lot of people in the software industry still saw components merely as reusable lumps of software - a rehash of modular programming or object-oriented programming. What was often missing from this story was any sense of architecture. However, even at that time, a few methodologies did take architecture seriously: there were some good ideas (including explicit recognition of the service concept) in Select Perspective, for example.

By the early 2000s, people had started to talk about service-orientation and service-oriented architecture. Gradually the large software vendors got their hands on these concepts, and started to sell products and platforms into a growing market called "SOA". Lots of people started to equate SOA with these products and platforms. Meanwhile, some SOA evangelists started to make sweeping (and usually ungrounded) promises about business agility and transformation. A gap started to emerge between expectation and reality.

Some of us had experienced something similar before. In the 1980s, we had worked with Information Engineering and CASE tools, which were sold on the promise of transforming IT productivity. However, achieving these outcomes generally depended very little on the features of the technology and much more on how the technology was implemented, used and managed. I developed a technology change management roadmap, which JMA (and later Texas Instruments) used for planning the adoption of its methodology and technology into some of its larger customers.

So many years later, with my help, CBDI developed an SOA Roadmap, along similar lines, in order to help large customers adopt and exploit SOA. And in 2006, I joined CBDI as a full-time employee. One of my tasks was to help turn a considerable body of knowledge (as published over many years in the CBDI Journal) into a well-structured methodology (known as SAE - Service Architecture and Engineering).

Although my instinctive sympathies were always with the evangelists who talked about what kind of future business transformations might be possible by extrapolating the available technologies, I started to get impatient with their inability to talk concretely about business change. At the other extreme, I was underwhelmed by some of the rather narrow and uninteresting uses to which SOA technologies were sometimes being put. Regular readers of this blog will have seen me writing frequently on these subjects.

One source of growing complexity, however, is that SOA can no longer be regarded (if it ever was) as an isolated discipline. On the one hand, it tends to be combined with all sorts of other technologies and practices, including BPM, Business Intelligence, Event Processing (Complex or Otherwise), Grid, Cloud, and anything 2.o. On the other hand, it tends to be allied (sometimes awkwardly) with other architectural disciplines, notably Enterprise Architecture. And there are usually major business changes going on, such as Merger and Acquisition. Whether you are doing a technology change roadmap or a business case, you probably need to juggle several of these factors. In many situations, SOA is no longer the rising star of technology adoption but merely one of the supporting acts. David Sprott, the founder of the CBDI Forum, now sees service architecture as business as usual (BAU), saying that "the key questions to be answered revolve around the level of standardization, componentization and rationalization in business and IT" (SOA in the Recession, Feb 2009). Some SOA pundits have even suggested that SOA (or at least the label "SOA") is dead. See my post Has SOA gone for a Burton?

I have no doubt that the CBDI Forum will quietly continue applying the concepts and principles of SOA to practical business problems, unbothered by all this hype. I remain more interested in business transformation than in technology for its own sake, but I shall continue to blog here from time to time on SOA and related topics. For other topics relating to systems thinking for innovation and business survival, you may want to subscribe to my Demanding Change blog.

---

Update April 2019

For many years, the Journal was available to subscribers only. Everware-CBDI has now released most of the Journal archives, with over seventy of my articles written between 2002 and 2011, as well as hundreds of further articles by David, Lawrence and others.

CBDI Journal Archive: http://everware-cbdi.com/cbdi-journal-archive

See also Lawrence Wilkes tribute (May 2017), Richard@CBDI (April 2019)

What's Wrong with Layered Service Models?

@JohanDenHaan pointed me at a couple of articles by Bill Poole

• Layered Service Models are Bad
• Layered Service Models are Bad (continued)

Bill takes a particular layered service model, a plausible combination of layers such as you might find in any number of popular SOA books, and shows how he can use this model to produce an inefficient and inflexible design. 

Of course, all this shows is that there are problems with a particular layered service model, as interpreted by Bill. (The authors of the books from which Bill has taken this model might claim that Bill had misunderstood their thinking, but whose fault is that?) It doesn't show that all layered service models are bad. 

As Bill points out, one reason commonly cited in support of the layered service model approach is the hope that services will be highly reusable. But there is a much more important reason for layering - based on the expectation that each layer has a different characteristic rate of change. (This principle is known as pace layering.) 

When done properly, layering should make things more flexible. When done badly (as in Bill's example) then layering can have the opposite effect. 

One of the problems is that the people inventing these layered service models often confuse classification with layering. We can identify lots of different types of service, therefore each type must go into a separate layer. A deeper problem with these models is that their creation is based purely on clever thinking rather than systematic and empirical comparison of alternatives. Too much architectural opinion is based on "here's a structural pattern that seems to make sense" rather than "here's a structural pattern that has been demonstrated to produce these effects". 

See my post on Layering Principles. You can't just take an SOA principle ("loose coupling is good", "layering is good"), apply it indiscriminately and expect SOA magic to occur.

SOA Sourcebook (Open Group)

I have just got my hands on a copy of the SOA Sourcebook, produced by the SOA Work Group of The Open Group and launched at the Open Group Architecture Practitioner Conference. (See my review of the Open Group Boston Grid.) I also spoke with Chris Harding, Forum Director for SOA and Semantic Interoperability. 

Most of the ideas in the SOA Sourcebook have been circulating around the SOA world for some time, and there is (not surprisingly) a lot of overlap (with more or less variation) with material the CBDI Forum has published from 2003 onwards. 

Some of this material has also found its way into TOGAF 9, but the SOA working group is not tightly coupled with the Architecture Forum (responsible for TOGAF). 

The SOA Sourcebook defines SOA as an architectural style, although it also defines it as a construction technique. As an architectural style, service orientation can apply to the business architecture as well as to the software (application) architecture and the technology (infrastructure) architecture, and you can have any of these without the other two if you want; but as the SOA Sourcebook acknowledges "many people believe that the greatest benefits of SOA are obtained when it is applied to the business architecture". 

However, the SOA Sourcebook focuses on the IT component of enterprise architecture - in other words, service orientation as applied to the software (application) architecture - and this is where the view of SOA as a construction technique would also make sense. 

The motto of the Open Group is "Boundaryless Information Flow". The SOA Sourcebook shows how SOA can achieve permeable boundaries, but of course that's not the same thing as lack of boundaries. For a deeper analysis of boundaries, you are going to have to look at Chapter 4 of my book on the Component-Based Business. But there are clearly construction issues as well as strategic requirements when joining loosely coupled lumps of enterprise as well as loosely coupled lumps of software. 

Inclusion in a long term IT strategy, created by Enterprise Architecture, is the only good justification for large-scale SOA. From an SOA perspective, this might be interpreted as saying that the de facto purpose of EA is to justify SOA. Clearly there is a common interest between EA practitioners and SOA practitioners, which The Open Group will doubtless continue to develop.

 

More posts on TOGAF including one on Boundaryless Information Flow (June 2011)

Where has all the SOA gone?

What on earth did the banks do with all the money?

No, I'm not talking about the money they blew on dodgy financial instruments. I'm not talking about Sir Fred Goodwin's pension. I mean all the money they spent on SOA, CEP, and other leading edge software.

When I first read Joe McKendrick's headline (Analyst: SOA may have reduced severity of mortgage crisis), I thought he meant that the crisis would have been even worse if the banks hadn't been doing SOA.

But when I read the piece, I found that he meant the exact opposite - that the crisis might not have been so bad if the banks had done SOA properly. This is based on a recent statement by James Governor, at a presentation hosted by IBM (Virtual Global SOA Forum),

"if we'd invested more in integration over the last few years, and better data governance, we would have significantly reduced the risk in terms of our banking infrastructure, our credit infrastructure, the kinds of mortgages that went out there"

In other words, despite countless SOA projects across the finance sector, the banks have failed to deliver adequate integration and data governance.

People with incredibly long memories may remember IBM's claim that "the banking and insurance industries lead in the maturity of their SOA deployments" [IBM Flatters Finance Sector (June 2008)]. Or that it was a bank that won the SOA Consortium Case Study Contest [More Flattery for the Finance Sector (September 2008)].

I don't think anyone is saying that these SOA deployments could have saved the world economy from crisis. But couldn't smarter SOA have made a difference to the outcome? With hindsight, how clever were all those SOA deployments, really? How can a bank have SOA maturity if it isn't dealing with integration and data governance?

I agree with James that SOA should make a significant contribution to restoring the operational viability of the banks. But I don't agree that the right answer is just more investment in SOA. I think what is needed is more intelligent investment, sensibly planned to deliver lasting business benefit from value-for-money SOA. That's what real SOA maturity means.

Architecture or Firefighting?

For many organizations, the main worry today is Business Survival.

ZDNet blogger Ian Finley reckons "companies will need architects when the crisis is over, but today, they need fire fighters". (When the building’s on fire, who calls an architect?)

In response, fellow ZDNet blogger Joe McKendrick said that "the problem is we don’t have enough architecture as it is, and even when times are good, we’re always firefighting". (SOA 2009: Do we need architects or firefighters?) Mike Kavis agrees: "The reason there are so many fires is that there is so little architecture!"

Comparing the IT crisis with the financial crisis, my colleague David Sprott talks about a toxic mess of IT assets that can’t be easily cleared up. As he says, "although we may be cancelling projects and reducing activity, unless we get smart on architecture we are increasing the toxic mess". Also known as technical debt or technology debt (HT @jpmorgenthal).

As far as I can tell from Joe's later post, Ian and Joe have found a middle point of agreement: focus-focus-focus on the essentials. (2009: a year of extreme focus)

These are not the only voices arguing that Little SOA is more practical/pragmatic than Big SOA. See for example Jordan Braunstein.

---

At the bottom of these debates is an important cultural clash.

In many organizations, there are strong cultural divisions within IT - developers versus architects versus project managers. (In previous posts I have used the metaphor of planets: Mars, Venus, Saturn, or Uranus, while of course Enterprise Architects are from Pluto.) The current climate maybe seems to favour the project managers and the developers over the architects. But that doesn't mean that the architects should abandon their principles and expertise, and pretend to be project managers or developers. As if that's what anyone needs.

What architects do need to do is find a new way of engaging with the business and the rest of IT, in a way that achieves a better balance between the different cultures.

Some organizations have always been strongly focused on delivery - following the JFDI management style. In other organizations, the emphasis has often been on careful planning and inclusive consensus-building.

There are problems with both extremes. As a consultant, I once had the experience of working with two organizations at the opposite ends of this spectrum at the same time - trying to get the first organization to slow down and the second organization to speed up. As a result of this experience, I came up with a novel approach to working with organizational culture, based on the five elements of ancient Chinese thought. See my Slideshare presentation: Elements of Change.

---

Further reading

Martin Fowler, Technical Debt (Oct 2003)
Ruth Malan, Technical Debt (March 2013)
Dharmesh Shah, Understanding Technology Debt (August 2005)


Updated 14 March 2017

SOA and Holism 2

Holism is another one of those much-abused words. Does it mean anything at all for Service-Oriented Architecture?

In a keynote address to ENASE 2008, Tony Shan introduces "a holistic 9-point list of SOA wisdom". What magic can convert a 9-point list into holistic wisdom?

Volker Hoyer (SAP) has developed what he describes as a Holistic analysis framework for Collaborative e-Business Process Modelling.

Rex Lee (Research in Motion) has written a series of posts on Holism and Enterprise 2.0

• Part 1: Introduction to Holism & E2.0
• Part 2: Organizational Complexity
• Part 3: Process Complexity
• Part 4: The Real Opportunity

For Rex, holism gives you "a complete unified end-to-end understanding of your company, products/services, processes and customers". Rex sees complexity as the enemy of holism.

"Complexity ends up destroying our ability to achieve a complete holistic (end-to-end) understanding of the company, the products/services, and its customers. ... The bigger the organization and the larger the hierarchy, the more difficult it is to truly understand the entire business. This lack of holism, eventually results in decisions and ideas that may be good for a small group but is actually not the best option for the larger group."

But this seems to be based on a thoroughly confused notion of holism. What I think Rex is talking about here isn't holism at all, but panorama. A panoramic view is a complete or wide-angled view of a situation, sometimes called "The Big Picture".

Holism is something quite different. Holism addresses the inherent complexity and irreducibility of systems. An holistic view may perceive fractal patterns, with the same structures recurring at many different scales and levels: this certainly seems to be relevant to SOA. An analyst who adopts an holistic approach may perceive a problem in one system as a reflection of a problem in a much larger system. An SOA architect needs to address problems at different levels of abstraction, and to understand how these are all interconnected and integrated.

If the people talking about holism understand this kind of thing, why aren't they talking about it? And why do they use the word "holism" as if it were just a fancy synonym for "whole"?

TOGAF 9 - Business-Led SOA

Shall we see SOA as a software engineering challenge or a business opportunity? TOGAF 9 makes a clear and unequivocal statement in favour of the latter, and identifies the following characteristic features of Business-Led SOA.

• Rich domain knowledge of both horizontal, cross-cutting concerns, such as human resources (HR), finance, and procurement alongside vertical, industry-specific concerns

• A structured, quantitative understanding of business value, costs, differentiators, and quality measures

• Broad understanding of current capability, showing both business capability and how it is supported by IT

• Broad understanding of the feasibility and viability of particular SOA technology-driven solutions

Source: TOGAF 9 22.3

This is contrasted with "a technology-centric, developer-led SOA community that maintains a core focus on the similarities across industries, organizations, and products to achieve benefit". 

(In other words, an emphasis on standardization, and the one-size-fits-all economics of scale characteristic of the technology-centric approach, are complemented with an emphasis on business capability within a specific business context.) 

Enterprise architecture is then positioned as "a set of tools and techniques to link top-down business-led SOA to bottom-up developer-led SOA". Enterprise architecture becomes "a foundation for service-orienting an organization". 

(In other words, enterprise architecture is what puts the A into SOA.)

Technology-centric SOA already makes a clear distinction between the specification of a software service (sometimes known as the service contract), and the implementation and deployment of this service. Business-led SOA applies the same principle at the business level: "Defining business services allows an organization to differentiate between business capability, the fulfilment of business capability, and the consumption of business capability".

(In other words, enterprise architecture supports service-orienting the business, not just service-orienting the applications. Service-orienting the business doesn't just involve IT reengineering, and "justifying the cost of IT reengineering against business value", but also business reengineering.)

Is this enough? Is the glass half-full or half-empty? Some commentators are complaining that the business end of TOGAF is still disappointingly thin. But I prefer to see it as a move in the right direction. And I can see some promising openings here for the work we've been doing on alternative business models ...

---

Footnote

Business-led but technology-centric? See my post on Jargon Orienteering.

For further posts on TOGAF, browse the TOGAF label.

SOA and Holism

Wikipedia's definition of holism traces back to Jan Smuts

Holism ... is the idea that all the properties of a given system (biological, chemical, social, economic, mental, linguistic, etc.) cannot be determined or explained by its component parts alone. Instead, the system as a whole determines in an important way how the parts behave.

To what extent does this concept apply to SOA? My own view is that SOA needs to be understood from the Systems-of-Systems Engineering paradigm rather than from the Systems Engineering or Software Engineering paradigm. This helps us to deal with a range of system level phenomena including Feature Interaction.

In my writings I've drawn on the recent work of Christopher Alexander, from A New Theory of Urban Design to The Nature of Order, where Alexander talks about something he calls structure-preserving transformations.

According to the New Theory (or at least my interpretation of it, which I call Organic Planning) each act of transformation should be a step within a larger and open-ended evolutionary development, and should have three aspects.

1. Produce something at some level
2. Complete something (larger) that was already part-developed - typically by linking smaller (lower-level) things and peer (same-level) things that already existed, or were created in previous steps.
3. Create new opportunities - Alexander calls this hinting-at.

For example, an information service called Product Catalogue might (1) establish a single point for accessing product information, (2) linking together data from multiple legacy data stores and third-party feeds, while (3) hinting towards a new business process (yet to be fully specified) in which the product catalogue becomes a dynamic object, using some kind of real-time business intelligence.

I published a very simplified version of this in the CBDI Journal in 2004, suggesting that the service designer needed to look in four directions (upwards, downwards, sideways, inwards). I understand that this was picked up and referenced by the ArchiMate people, for example in a 2005 book called Enterprise Architecture at Work, and it has been implemented in the Telelogic tool. My colleague Tony Bidgood has recently published an article on ArchiMate, with some further examples.

• Richard Veryard, Business Driven SOA (CBDI Journal, May 2004), Business Driven SOA Governance (CBDI Journal, June 2004)
• Tony Bidgood, Archimate Standard for Enterprise Service Modelling (CBDI Journal, August 2008)

Here's how I intended the four directions to map to the New Theory of Organic Planning

1. Inwards: Functional correctness
2a. Downwards: Integrating and composing smaller stuff
2b. Sideways: Interoperability
3. Upwards: Larger whole

Organic Planning is described in my 2001 book on the Component-Based Business, and there is a short version on my website, but I didn't make this explicit in my 2004 article.

My expectation has always been that a series of transformations (whether structure-preserving or otherwise) would be expressed as a series of model pairs, in which the nth TO-BE becomes the (n+1)th AS-IS. But some alternative modelling notations (such as Michael Bell's SOMF) allow both AS-IS and TO-BE to be expressed in a single model, so it would be very interesting to see how a series of transformations could be expressed and analyzed.

Organic Planning from Richard Veryard

 

 See also SOA and Holism 2 (February 2009)