SOA Sourcebook (Open Group)

I have just got my hands on a copy of the SOA Sourcebook, produced by the SOA Work Group of The Open Group and launched at the Open Group Architecture Practitioner Conference. (See my review of the Open Group Boston Grid.) I also spoke with Chris Harding, Forum Director for SOA and Semantic Interoperability.

Most of the ideas in the SOA Sourcebook have been circulating around the SOA world for some time, and there is (not surprisingly) a lot of overlap (with more or less variation) with material the CBDI Forum has published from 2003 onwards.

Some of this material has also found its way into TOGAF 9, but the SOA working group is not tightly coupled with the Architecture Forum (responsible for TOGAF).

The SOA Sourcebook defines SOA as an architectural style, although it also defines it as a construction technique. As an architectural style, service orientation can apply to the business architecture as well as to the software (application) architecture and the technology (infrastructure) architecture, and you can have any of these without the other two if you want; but as the SOA Sourcebook acknowledges "many people believe that the greatest benefits of SOA are obtained when it is applied to the business architecture".

However, the SOA Sourcebook focuses on the IT component of enterprise architecture - in other words, service orientation as applied to the software (application) architecture - and this is where the view of SOA as a construction technique would also make sense.

The motto of the Open Group is "Boundaryless Information Flow". The SOA Sourcebook shows how SOA can achieve permeable boundaries, but of course that's not the same thing as lack of boundaries. For a deeper analysis of boundaries, you are going to have to look at Chapter 4 of my book on the Component-Based Business. But there are clearly construction issues as well as strategic requirements when joining loosely coupled lumps of enterprise as well as loosely coupled lumps of software.

Inclusion in a long term IT strategy, created by Enterprise Architecture, is the only good justification for large-scale SOA. From an SOA perspective, this might be interpreted as saying that the de facto purpose of EA is to justify SOA. Clearly there is a common interest between EA practitioners and SOA practitioners, which The Open Group will doubtless continue to develop.

Semi-structured processes

A lot of the focus in the BPM world is on highly repeatable, controlled processes. These processes follow a strict management lifecycle.

1. analysed and designed by process experts on behalf of process owners,
2. implemented by process engineers using sophisticated BPM tools and workflow engines
3. performed by armies of obedient employees or compliant customers
4. monitored and calibrated by statisticians using sophisticated analytical tools

This is essentially the BPM lifecycle presented by Ann Rosenberg of SAP at the London TOGAF conference today.

In this lifecycle, we can achieve process agility and innovation through rapid redesign and recalibration. If the design and control phases can be make more efficient and effective (for example by being supported by tools), then the set-up costs are reduced, and this improves the economics of scope as well as the economics of scale.

At the other extreme, there may be some processes in an enterprise that are completely adhoc, with no repeatable structure. For example, a decision to merge with a competitor may involve a number of highly complex steps (investigation, evaluation, due diligence and so on) but without any systematic process structure.

Is there something in between these two extremes? Well there certainly should be. In my previous post on Activity-Based Computing, I mentioned four important examples.

• Sales. The sales team produces a detailed proposal in response to a complex request from a customer or prospect.

• Customer Complaints. Each complaint follows a different path, which depends on the nature and content of the complaint.

• Problem solving.

• Medical intervention.

All of these processes are semi-structured. The people doing these jobs are given some degrees of freedom to customize the process to their own understanding of the requirements of a particular customer or patient or situation. However, they are still accountable and their performance is measured; and therefore there must be some underlying process structure.

Who is doing the process management? All the participants in the process are participating in the management of the process, and should have access to all the management tools (not just BPM but also BAM and Business Intelligence). Then the job of the process architect is not to lay down the process but to create a collaborative process space in which everyone can be productive and appropriately innovative, with the right balance of variation and standardization, freedom and control.

So the big question for the process architect is not optimizing one process, or even optimizing lots of different processes, but managing the process continuum. In this way, the process architect becomes the enterprise architect.

Update: see Semi-structured processes 2

Decision-Making Services

James Taylor (ebizQ) explains why we need a separation between process and decision.

• Interesting debate on business process and decisions (April 2009)
• Decision services (Jan 2007)
  

One of the reasons for this separation is that decisions are subject to business policies or rules, and such policies are often subject to change over time and/or variation between organizational units. If you code the decisions into process logic, perhaps using BPML to generate process services, then you will have to modify the code or BPLM script whenever the policy changes, and implement different versions for different parts of the organization.

But there are some more flexible ways of architecting decision services.

• General-purpose rule engine, driven by a standard policy language.
• Clusters of related business decisions (for example planning and scheduling) can be wrapped into a capability service.
  

The advantage of these approaches is that they help to separate the generic parts of the process from the specific parts, and thus increase reuse and sharing of the generic parts, while increasing accuracy and differentiation in the specific parts.

In his post on business processes and decision services, James discusses some of the technical issues that arise with this separation. It is certainly true that some technical optimization may be desirable in some situations, and this may mean implementing and deploying the process and decision services together as a fairly tightly coupled unit of software (which CBDI SAE calls an Automation Unit). But at the specification level, the logical services should still remain distinct.

For more examples of policy separation, see my articles on Business Modelling for the CBDI Journal. (Some of these require paid subscription.)

Motoring as a Service

Listening to Power Drive, a BBC radio programme on electric cars. (I heard the live broadcast yesterday evening, and I have now downloaded the podcast from the BBC website to listen again).

I fully expected to hear the voice of Shai Agassi, and I was not disappointed. Until a couple of years ago, Shai was the rising star at SAP: the founder of Netweaver, the champion of service-oriented architecture (SOA) within SAP, frequently talked of as a future CEO. Then he suddenly quit the software industry to work on electric cars.

Shai was one of the first software executives to get the concept of business as a platform of services. (See my post dotBiz from January 2005). He is now talking about motoring as a service, with what he calls a platform-based approach to create a new business model. Shai's company Better Place is building a network infrastructure for rapid charging and battery replacement. (Coverage from about 17 minutes into the BBC programme.)

Shai makes five important points about a service-based approach to motoring.

• Establish a separation between car and battery - when you reach a charging spot, you swap your empty battery for a fully charged battery.

• Don't solve mileage problem by bigger batteries but by better infrastructure - put charging spots closer together.

• We're getting the infrastructure right before we expect people to buy the cars.
  

• Motorists pay for real service (mileage not car)

• Shift the industry, not one car at a time. Car-makers should be able to make more profit with electric cars than gasoline-based cars.
  

The business model is copied from the mobile telephone business. The consumer has a choice between a pay-as-you-go model and a contract model. You can buy a low-mileage contract or an unlimited mileage contract. If you are willing to sign a 24 month contract, you get a better deal for your miles, and you may find a supplier willing to give you a free car.

Of course, this is only economically viable if you can get a large-scale adoption of the new technology. Shai talked about computer simulation models they are using to calculate the business case for a whole country (in terms of reduced oil imports) and to plan the distribution of capacity.

"You're still a software person at heart." says the interviewer and Shai agrees. "At the core of this is a huge software system", he says.

See also Shai Agassi: A bold plan for mass adoption of electric cars (TED Talks)

Towards the Intelligent Business

What do organizations need to be successful in good times and to survive in bad times?

It's not just about the people (a few talented individuals on large bonuses). And it's not just about the software systems (sophisticated real-time complex event processing or whatever). What matters is how the people and the systems collaborate effectively to address emerging business opportunities and threats, in an agile and innovative manner. I call this Organizational Intelligence.

Organizational intelligence is like human intelligence - it involves the ability to solve problems, to make good judgements, to deal with complex situations in an effective and uncomplicated manner, and to learn from experience.

Many of the technologies I've been talking about on this blog can be used to support Organizational Intelligence, including Business Intelligence (BI), Business Process Management (BPM), Complex Event Processing and Enterprise 2.0.

Conversely, the concept of Organizational Intelligence helps us to work out two things.

1. To identify the potential contribution of these technologies to business survival (business value).
   
2. To determine how these technologies can be combined to support the requirements of a specific organization (architecture).

Technologies for Organizational Intelligence

View more presentations from Richard Veryard.

IT Innovation at Small Bakery

Can your IT department innovate like this bakery? asks Mark Raskino (Gartner) via Sally Bean. Mark describes an East London bakery that uses Twitter to notify its customers whenever freshly baked bread comes out of the oven.

The system involves a specially designed wall-mounted device that the bakers can operate with one hand, while holding a tray of bread with the other.

Mark clearly doesn't think most IT departments would be comfortable with this kind of system. So I asked Mark and Sally who would lead that kind of innovation in a typical IT department. Would it be the business analysts? The enterprise architects?

The trouble is that many people in those kind of IT roles have been trained to value abstraction and adaptability above everything else, and to avoid thinking about mundane technology when building pure business models. But how often does that kind of abstract thinking produce concrete innovations like this bakery? In many organizations it is quite the opposite - technology-driven opportunistic change to the business model is discouraged because it runs contrary to prevailing thinking. And as Mark points out, corporate IT departments don't like purpose-designed hardware. This system is not designed for adaptability, it is designed to improve the business.

So this project was driven by a marketing agency. Business innovation may depend on the latest technology; but wouldn't it be a disgrace if it turned out that the most innovative companies are those too small to have an IT department?

Scale and Governance

Following my post Does Britain Need Smaller Banks? Fred Fickling objected to my point that large banks are more difficult to regulate.

"Not sure that the simple answer here is really the right one. Size & corp responsibility aren't necessarily inversely related."

Of course, some large companies can have a highly refined sense of ethics and social responsibility, although this doesn't always protect them from strong disagreements with external lobbyists, as Shell discovered when it wished to decommission the Brent Spar.

In any case, not all large companies can be trusted to regulate their own behaviour in the public interest. Companies pay tax where it suits them, move (or threaten to move) operations from one country to another, shrug off massive fines, and can sometimes behave as if they were above the law, as Fred concedes. In the long term, these legacy corporations may be doomed, but in the short term they can still cause a lot of disruption in the financial ecosystem.

Of course there are no simple answers. I certainly think some companies are too large, and I hope there will always be good opportunities for smaller companies to compete. What I'm calling for is a way of reasoning intelligently about the size of companies, which balances the interests of all stakeholders in a fair and governable manner. I think this is a proper topic for business architecture. And perhaps some of what we've learned from SOA about granularity and governance can be applied to this greater problem domain.

Ford and Fordism

In his piece Flashlights not Bailouts, Phil Gilbert describes the classic CBD/SOA argument: standardization/rationalization/reuse.

"One of the three (Ford) is in demonstrably better shape than the other two, and it's no mystery why. Two years ago, when he took the reins of Ford, Alan Mulally identified two things that needed to change: parts costs have to go down, and engineering productivity must go up."

"Get it? The white collar workers who design the cars have to move from artisan to engineer, and they need to work together across all the company's platforms to use common parts."

I asked Phil if this was Fordism? Phil responded thus:

"Ford used standardization of task to drive down costs. I'm suggesting 'standardization' of transparency to drive down risk."

Thus Fordism progresses from the economics of scale to the economics of governance. This fits well with the story about SOA I've been telling in this blog. http://tinyurl.com/d3ronn

Does Britain Need Smaller Banks?

George Osborne, possibly the next Chancellor of the United Kingdom Exchequer, has asked some good questions about the right size of banks. (Robert Peston, Tories to break up banks?, BBC 8th April 2009)

"We cannot allow one part of our economy to behave in a way that puts the rest of the economy at risk when it fails. We need to think deeply about whether we can sustain banks that are not only too big to fail, but potentially too big to bail."

"We should look at whether Britain in fact needs smaller banks. For it would be a bitter irony if we came out of this crisis with a banking system that was even more concentrated and even riskier than the one we had before it."

It is no longer obvious that large banks are any more financially viable than small banks. What seems pretty clear is that large banks are more difficult to regulate. What was the economic logic behind all those mergers and acquisitions in the banking sector anyway? Whatever may have been gained in terms of the economies of scale were totally blown away by the economies of governance.

Umair Haque asks (rhetorically) why bankers never talk about reforming banks. However, as Osborne points out, now that the British government owns a large chunk of the banking sector, there is an opportunity to make radical changes to the sector as a whole, rather than merely controlling the behaviour of each company within the sector. I hope this will allow a serious debate on the scale and granularity of business. See my earlier post on Business Model and Financial Viability.

Update: Here's what Henry Mintzberg has to say about General Motors, in a piece called America's monumental failure of management (Globe and Mail, 16 March 2009).

We hear now about "too big to fail." "Too big to succeed" is more like it. General Motors has been going slowly and painfully bankrupt for decades, managerially as well as financially. The new money will only put off its demise. Americans will have to face this reality sooner or later.

See also Towards Partition (Economic Principles, April 2010).
and Do banks need more intelligence? (Demanding Change, May 2010)

Pay as you drive 3

Couple of posts from the Smart 421 people on usage-based car insurance.

• Telematics and retail insurance (aka “PAYD”)
• Usage based car insurance… why bother?

Their conclusion: it isn't going to save anyone enough money to justify the cost and inconvenience and to overcome the concern for privacy. Adoption only becomes viable when the devices become a commodity (multiple providers, standard interface, routinely fitted to new cars), with a strong carrot/stick from the government (e..g legislation or road charging).

This is pretty close to what I said when the Norwich Union experiment was abandoned last year (Pay as you drive 2).

The Smart 421 bloggers also have some comparisons with Pay-As-You-Go models for other utilities, including domestic energy and water, and telecoms. Telecoms is clearly more complex than energy and water consumption, and has much more complicated pricing schemes. What are the lessons for software pricing - e.g. Anything-as-a-Service?

Enterprise as a Network of Events 2

Following my post on Enterprise as a Network of Events, Nigel Green drew my attention to something he wrote a couple of years ago on the case for a clear distinction between events and content, in which he points out that "aggregated events become content over time".

As it happens, I touched on something like this in my 1992 book on Information Modelling. At that time, the specific question I was addressing was how to represent the history of organizational change in a data model. There are two basic options.

1. History is represented as a series of snapshots of the organization at specific times. If you want to know what the change was, you just have to compare two snapshots. (For example, this is how page history is represented on Wikipedia.)
   
2. History is represented as a series of changes (differences). Then if you want to know what the state of the organization was at a given time, you just have to aggregate the changes forwards or backwards from a fixed point.

From a purely logical point of view, it doesn't matter which of these two representations you adopt, because you can always derive one from the other. In practical terms, however, you may find that one representation is a lot easier to work with than the other.

What both representations have in common is a definition of information in terms of difference: a difference that makes a difference, as Bateson puts it.

Information as Difference

Hold your hand perfectly still, palm upwards and resting comfortably on a table. With your other hand, drop a small coin into the palm. You will feel the impact, and if the coin is cold, you will feel the coldness of the metal. Soon however, you will feel nothing. The nerve cells don't bother repeating themselves. They will only report to the brain when something changes. Information is difference.

A lizard hunting insects operates on the same principle. The lizard's eye only reports movement to the lizard's brain. If the hunted insect settles on a leaf, the lizard literally cannot see it. But the moment the insect starts to move, whop, the lizard can see it again, and the tongue flickers out and catches it.

But there are differences and differences. Information is difference that makes a difference. You were probably aware, as you dropped the coin into your palm, your eyes told you automatically, without your brain even asking, what the value of the coin was; but you were probably not aware what date it was minted. This is because (unless you are a numismatist) the value of the coin makes a difference to you whereas its date doesn't.

What is it that makes a difference to a lizard, to a numismatist, to you? Surely not the same things. What is information for the lizard is not information for you, and what is information for you is not information for the lizard.

This is why the perspective of information is important. Perspective defines what counts as information at all, perspective defines to whom the information makes a difference.

Enterprise Strategy and Difference

Now what's this got to do with the enterprise? There are two important differentiation questions for any enterprise: how do They differentiate Us, and how do We differentiate Them.

Firstly how do They differentiate Us. In other words what differences in our products and services or organization are (a) going to be noticed by external stakeholders such as customers and (b) going to affect the behaviour and attitudes of these stakeholders.

Sophisticated marketing organizations pay a lot of attention to this kind of thing. Does it make a difference if we put this in a blue envelope? Does it make a difference if we have a female voice here? Sometimes you have to experiment with differences that don't matter, in order to have a refined view of what differences do matter.

Secondly, how do We differentiate Them. What are the strong signals in the environment that we must respond to, and what are the weak signals we might respond to? An organization that responded to every weak signal would be impossibly unstable, so most organizations have operational processes that respond to strong signals, and some form of business intelligence that scans the environment for weak signals and identifies a subset of these signals demanding some kind of response.

Sometimes the appropriate response to some detected difference or discrepancy is some kind of business improvement, such as business process improvement and/or system engineering. Which means that the history of the organization is inscribed into its current structure and processes.

"What's the purpose of this process step?"

"We've done that extra check ever since we got badly burned in the XYZ deal five years ago."

If we knew enough (which of course we don't), we might conceivable infer the current state of the organization from a careful analysis of all the events that have ever happened to it. That would join this post back to where we started. Neat but impractical.

But that's not actually what I'm trying to do. I think understanding an organization as an information-processing system helps us to do two practical and concrete things.

• Diagnose the current problems and opportunities facing a complex organization.

• Design an event-driven business improvement process, based on business intelligence.

My next question is: what kind of business model supports these challenges.