Special Powers of the Architect - Abstraction

Does the Enterprise Architect have special powers?

---

One idea is that the intrinsic essence of an enterprise architect is to think about abstraction. (By the way, abstraction seems to be a category that can only be defined polythetically; it appears to have many characteristic features, but we seem to be unable to work out any defining features.)

Suppose that the use of an enterprise architect (if any) is to solve practical business problems. For this purpose, enterprise architects are interchangeable with systems thinkers, just as Jaffa Cakes are interchangeable with Fig Rolls. For a large tea party or enterprise transformation programme, we might have a plate containing Jaffa Cakes and Fig Rolls and various other cakes and biscuits.

But presumably that's not the only thing that an enterprise architect is good for.

So one theory-practice question is this. What are the essential things that an enterprise architect brings to solving practical problems? If it is abstraction, what special powers (if any) does this give to the practising enterprise architect? And what are the essential things that a systems thinker brings to solving practical problems?

Abstraction is associated with a number of myths and pitfalls

• The myth that radical structural and behavioural change can and should be driven by a few simple and powerful ideas. On the misuse of general principles (Jan 2012)

• Valuing abstraction and adaptability above everything else. Avoiding thinking about mundane technology when building pure business models. IT Innovation at Small Bakery (April 2009)

• The relationship between the general and the specific is poorly supported by the prevailing tools and methods, which often reduce the question to simplistic abstraction hierarchies. TOGAF 9 - Enterprise Continuum (Feb 2009).  Instead of generalizing everything as much as possible, architects need to reason explicitly about system structure and dynamics - cohesion and coupling, composition and decomposition, change and emergence. They need to understand granularity and stratification as active choices, rather than text-book patterns. Architecture and Abstraction (May 2004)

• Mainstream EA frameworks typically appear as premature codifications of ungrounded abstractions. The attempted diffusion of these codified abstractions is then ineffectual, because these ungrounded categories lack any relevance to real business challenges. ... So instead of discussing concrete business problems, we find ourselves discussing generic categories of business problem. Towards Next Practice EA (May 2013) 

See also

• Special Powers of the Architect - Getting The Big Picture (Dec 2010)
• Special Powers of the Architect - Judgment (January 2012)

Updated 26 October 2016

Shuffle

In his blog Not Bad for a Cubicle, Chandler Howell analyses How does the music industry not get it?, arguing that piracy is a natural consumer response to the service-oriented strategy of the music industry.

• The recording industry [... has adopted] a self-destructive strategy of eschewing the long-term development of artists, preferring instead the quick buck of one-hit wonders ... If an artist is only going to have one CD worth buying and that one CD only has one or two songs worth listening to, then “schoolyard copying” is going to have a material impact on sales of that CD.

• There’s no point in paying for a CD because the consumer has been trained by the music industry itself to not value that CD. Recorded music has become a disposable commodity.

There are several complex issues here.

The one I want to focus on here is not the morality of piracy and DRM but on the structural changes that emerge from the conflicts of interest between different parties on the supply side - composers, performers, recording industry, music distribution channels/technologies (e.g. radio broadcast, iTunes/iPod). Among other things, these conflicts of interest have had an effect on the different granularity with which recorded music is supposed to be consumed.

Whole CD versus single song? 3-minute song versus extended medley? Concept album versus collection of greatest hits?

One of the consumer issues about moving from vinyl to CD was the apparent need to repurchase all your favourite music. But there was another important change, which altered the balance of power against the musician - the shuffle function. You no longer needed to listen to your favourite Beatles album in the fixed sequence provided by John, Paul, George H, Ringo and George M; you could break a coherent sequence of songs (often with carefully considered changes of key and mood) into a random series of fragments - every time a new experience.

(Why do we do this to music? I've never seen a shuffle function on a DVD player.)

With MP3 players, consumers can now hang a tiny juke-box around their necks, and listen to a constant shuffle of their favourite tunes. But why bother, when there are so many specialist digital radio stations that provide almost exactly the same service without the hassle of downloading and selecting the songs?

But some people are arguing that the internet restores the power of the musician, who can record and release a single song very quickly, without needing big record companies or fancy CD production. It is also technically possible to subscribe to vast quantities of live music from your favourite band. (Imagine having a virtual seat in every single concert in next year's tour.)

What are the general lessons of this for the service-based business?

• Technology can alter the preferred granularity of the basic service - how it's paid for, how it's delivered, how it's consumed, how it's combined into composite services.

• New business opportunities can be opened up by thinking about alternative levels of granularity.

• Flexible systems (both demand-side and supply-side) must be able to handle many different levels of granularity - preferably at the same time.

[Update] After I posted this, Chris Anderson made a similar point in the Long Tail blog, with a post called Shorter, Faster, Smaller, referring in turn to Seth Godin's Small is the New Big. Chris writes:

Note that this increased range of distribution options can allow for longer content, too, with the rise of TV shows on DVD (where you can watch much of a season at a single sitting) as a prime example. But the overall trend is toward shorter, faster, smaller everything. We're increasingly fine-slicing both the time we give media and the media itself.

Quite.

---

Another update: The Anti-Shuffle Brigade [BBC News 18 January 2011]

See also Adorno's concept of Atomized Listening, part of his deeper critique of the commodification of music.

Apple and Intel

My CBDI colleague Lawrence Wilkes has just posted a commentary (Proprietary or What?) on the Apple/Intel controversy, pointing out some of the SOA implications.

This example also illustrates something I've discussed before - that bundling/unbundling and questions of architectural flexibility look different from different stakeholder positions. Flexibility FOR WHOM. As Lawrence points out, Apple chooses its technical architecture not purely on abstract technical arguments, but according to its view of commercial forces.

Another colleague once had an expensive German car, where he needed to replace the entire engine to fix a wonky sparkplug (or something like that). I used this example in my 2001 book and elsewhere, because it nicely illustrated some important issues about the granularity of components and the potential architectural conflict of interest between supplier and customer. The supplier deliberately constrains the ability of the customer to reuse and reconfigure, and chooses an architecture that retains control over what the customer is able to do, in support of its own commercial objectives.

This entails a very strong link from the business modelling to the architecture. Where business situations are at all complex, architectural questions of granularity and coupling/decoupling may have real business significance, and cannot be determined purely on technical grounds.

Note also that for flexibility to be maintained, it has to be exercised. Upgrades are always a bit problematic for platform vendors with a large community of independent application developers. Apple developers who have used pre-OSX services are now apparently being forced to upgrade to Cocoa. Although there might be some short-term pain for some of them, the likely medium-term benefit would be an increase in the overall flexibility and robustness of the Apple ecosystem. Furthermore, Apple is sending an important message about change and flexibility to its developer community. Perhaps it should make a point of changing chip vendor more often.

Technorati Tags: Apple Intel SOA service-oriented

Straight-Through Processing

In an SOA context, straight-through processing means passing some input services onto your customers, with zero intermediation. This only works if the input and output services are in the same domain.

• Insurance. You buy insurance through a broker. You may pay either the broker or the insurer. (If you pay the insurer direct, then there is presumably some commission paid back to the broker.) You may claim through a service that is actually provided by a separate claims process, under the insurers’ brand.
• Travel. You buy through a travel agent. The services are delivered by the airline and hotel. (You are given a reference number to pick up your tickets and check into the hotel.)
• Logistics. You buy something on eBay, pay via PayPal, and the item is delivered by courier. (You are given a reference number so that you can liaise directly with the courier company.)

These supply chains are more or less transparent, until something goes wrong. If there are problems with the service, you may start by complaining direct to the ultimate service provider, but if you cannot sort things out you have to go back to the person/organization you have the contractual relationship with. There is a contractual chain which may not coincide with the operational information supply chain. However, there has to be a management control chain (higher level information supply) that supports the contractual chain.

In some of these sectors there is a serious risk of disintermediation. For example, many people have stopped using travel agents, and book their airline tickets directly with the airline over the internet. This is possible because the airline domain is just as accessible to the travelling public as to travel agents - in some cases more so. (Sometimes the agents don't have access to all the microservices. And sometimes even the airline employees don't have access to all the microservices. See this example.)

What about the next step? Can you disintermediate the airline by chartering your own plane? This is possible but it's a lot more difficult for two reasons. Firstly because it takes you to a different level of granularity - you have to charter the whole plane, not just one seat. Secondly, the aeroplane-chartering domain introduces (makes visible) a lot of new complications. One way of putting this is that if you want to charter a plane, you have to learn a different Domain Specific Language (DSL).

In many (perhaps most) business sectors, the input DSL is different to the output DSL. So the information supply chain involves considerable processing (transformation).

Let's consider the pharmaceutical industry as a much more typical example of a complex supply chain. A drug company produces a drug, together with lots of information relating to the drug. There are several different categories of information user: the patient who takes the drug, the medical practitioner who prescribes and/or dispenses the drug, the health service or insurer that pays for the drug, and the regulator who monitors the safety of the drug. The translations and transformations between these information domains calls for careful negotiation, not just mechanical mapping.

Many of the SOA champions are over-simplifying the picture by using examples where there is a single domain. The trouble with this is that the examples aren't very interesting, and moreover invite disintermediation. To find sustainable business benefits from SOA, we need to look at more complex ecosystems. Like pharma.

(Anyone with good examples, or interesting problem domains, please contact me.)


Related posts: Straight-Through Processing 2 (April 2005), Straight-Through Processing 3 (March 2009)

How many products?

How many products are there? This is a recent discussion on the Now Economy blog.

Mark Baker is sarcastic. "A fascinating question. Almost as fascinating as why anyone would want to know!" I find this sarcasm strange because it was actually Mark's recommendation that led me to the Now Economy blog in the first place.

So let me say why I am interested in the "how many" question?

Firstly, the absolute number may not be as important as the trend. If the number is doubling (or halving) every year, this is an important phenomenon. (In biology, are new species evolving faster than the old species are becoming extinct? In commerce, are new high-tech startups being founded faster than they can be killed or acquired by the big boys? Is consumer choice increasing or decreasing?)

Secondly, the process of asking the question tells us something about the semantics of the situation. When I'm doing requirements engineering for computer systems, I always ask "how many" questions. Some people think I'm doing this because I want to look ahead to hardware and network capacity. Perhaps, but the main reason is that I find out more about the underlying concepts are used here.

This aspect of data modelling is crucial for web services. If I have a service that references PRODUCT, I need to know what granularity of product it assumes. Is every tiny variation counted as a separate product? And if I'm desiging web services, I need to find a level of granularity that is meaningful for the users.

Note that the Now Economy discussion quickly shifts from How Many Products to How Many Product Codes. As if there was a simple correspondence between product and product code. But this begs the question. I have worked in major oil companies where the correspondence between products and product codes is very difficult to manage. It turns out that this apparent simplification introduces more complexity: we may now have to pay attention to the correspondence between the granularity of (multiple alternative) Product Codes and the granularity of (multiple alternative notions of) Product.

These aspects of semantics are ignored by many web service practitioners, who optimistically hope that everyone has the same understanding of the same implicit data model. After all, we're all using the same XML schema, so how could we possibly misunderstand one another?

Asking how many is a great way of discovering we're not on the same wavelength.


Related post: How Many Events? (December 2008)

Business Geometry

A business or value chain is composed in a geometric structure. (In the past we have often called this structure an architecture, but this word has lots of different and tangential associations for different people.)

In SOA, we design a business or value chain as a network of services. This is a powerful geometrical pattern. But there may be many possible network geometries satisfying a given business requirement, all of which count as satisfying the principles of SOA. For example: hub/spoke or peer/peer.

Business Stack

Another common geometric pattern for SOA is stratification. A business process is composed of services from a set of lower-level services, presented as a platform. A good example of a business platform is the set of retail services offered by Amazon and eBay. Other service providers have built further retail/logistical services on top of the Amazon/eBay platforms.

Each platform is in turn built upon even lower services. At the lower levels, there may be collections of IT-based services, known as ESB. There may also be sociotechnical service platforms, such as call centres.

Thus we have a stratified geometry, in which a person tackling a problem at a given level is presented with a collection of available services, formed into a virtual platform. This can be thought of as a business stack, with one plaform stacked on top of another. Some of the lower layers of the stack may appear to be purely technical services; however a more complete picture should reveal the existence of an IT organization maintaining the platform, in other words a human platform of administrators and programmers supporting the technical platform.

Variable Geometry, Variable Granularity

While the SOA principles may provide some geometrical guidance, and mandate certain geometrical patterns, there is still a design job to determine the geometry. This design job may be easy when the requirement is trivial, but gets harder as the complexity increases.

In many situations, the demand side has more variation than a human designer (or design team) can accommodate. (We characterize this as an asymmetry of demand, which calls for a process of asymmetric design.) We need to start thinking about variable geometry solutions, where the geometry itself can be adapted on-demand.

For example, in the past we have assumed that granularity has to be fixed at design time. But we can conceive of a web service platform that detects patterns of demand-side composition, defines new composite services automatically, describes and publishes these new services in real time, and notifies likely users of the new service, complete with an incentive to switch. We can conceive of a web service platform that analyses the message content of a certain service, and produces a substitute service with a smaller footprint that would satisfy most of the uses in a more elegant way.

Value Landscape

I use the term value landscape to refer to the distribution of cost, benefit and risk across a complex market ecosystem, such as the insurance industry. Technology (including SOA) influences business geometry, not least because it affects transaction costs. The shape of the value landscape changes (has already started to change) as the result of B2B and B2C and BPO. Companies that once occupied safe market positions (the high ground) may find their commercial advantage slipping into the sea, or they may find themselves cut off from their natural markets or supply chains.

See Microsoft Blog Insurance Value Chain

Let's suppose an insurance company has the following strategic aims:

1. Profitability, short-term viability. To deliver the maximum service value as cost-effectively as possible, using available input services and technologies as efficiently as possible, with minimum costs/risks of change.
2. Adaptability, medium-term viability. To understand and respond to changing demand for insurance services, and to trends in cost and risk, both internally and across the industry. To develop and deploy new services to exploit new business opportunities and avoid emerging business threats.
3. Survival, long-term viability. Making sure the core business proposition remains valid, and doesn't get eroded by more agile players. Taking strategic action in relation to structural changes in the insurance industry.

If we are doing business geometry for an insurance company, we need to think about the insurance industry as an ecosystem. We need an AS-IS model of the present ecosystem (largely based on pre-SOA technologies) and a TO-BE model of an idealized ecosystem (based on SOA). We can expect the pre-SOA ecosystem to evolve into some form of SOA ecosystem, although we may not have much idea which of the possible changes is going to happen first. To satisfy all three strategic aims, an insurance company needs to exploit the pre-SOA ecosystem, and prepare for the SOA ecosystem.

Note that this situation may force the insurance company to implement a variable geometry, both in the organizational platform and in the IT platform. Otherwise, it will either have to operate suboptimally for an extended period, or incur significant organization costs and IT costs every time the industry takes another step towards SOA.