Under the pavement

@pbmobi asserts that "Infrastructure Architecture is the foundation for #entarch, not the other way round". This suggestion draws on a book review of "The Works: Anatomy of a City", in which the reviewer suggests that the tubes, wires and pipes under the pavements of New York are collectively more important than the city of towers above them.

So that's one sense of the word "foundation".

@pbmobi also quotes Frank Gehry via @nate_berg "There’s a lot of layers of bureaucracy that make it impossible to do creative work in cities."

So that's a completely different sense of the word "layer".


Bruno Latour gave a brilliant lecture at Brunel University in April 1998. Among other things, he talked about the (usually invisible) stuff under the pavements of Paris. As I recall, he showed some slides of various control rooms, each providing a different slice (are these layers or perspectives?) of the Parisian infrastructure. The point here is that there isn't one homogenous infrastructure, but a complex system of infrastructure systems that barely talk to each other except in an emergency. Sadly, the lecture is no longer available on the Brunel website, but I found a transcript on a Hungarian website.

---

Nate Berg, Frank Gehry on City Building, Atlantic Cities, 9th Jan 2012.

Bruno Latour, "Thought Experiments in Social Science: from the Social Contract to Virtual Society"
1st Virtual Society? Annual Public Lecture Brunel University 1st April 1998. [transcript] See also [Invisible Paris, pdf].

Alex Marshall, The Works Reveals City's Essential Systems. Spotlight Vol. 5, No. 2. January 26, 2006. Review of Kate Ascher, The Works: Anatomy of a City. Penguin Press 2005

See also my post OrgIntelligence in the Control Room (October 2010)

What's Wrong with Layered Service Models?

@JohanDenHaan pointed me at a couple of articles by Bill Poole

• Layered Service Models are Bad
• Layered Service Models are Bad (continued)

Bill takes a particular layered service model, a plausible combination of layers such as you might find in any number of popular SOA books, and shows how he can use this model to produce an inefficient and inflexible design. 

Of course, all this shows is that there are problems with a particular layered service model, as interpreted by Bill. (The authors of the books from which Bill has taken this model might claim that Bill had misunderstood their thinking, but whose fault is that?) It doesn't show that all layered service models are bad. 

As Bill points out, one reason commonly cited in support of the layered service model approach is the hope that services will be highly reusable. But there is a much more important reason for layering - based on the expectation that each layer has a different characteristic rate of change. (This principle is known as pace layering.) 

When done properly, layering should make things more flexible. When done badly (as in Bill's example) then layering can have the opposite effect. 

One of the problems is that the people inventing these layered service models often confuse classification with layering. We can identify lots of different types of service, therefore each type must go into a separate layer. A deeper problem with these models is that their creation is based purely on clever thinking rather than systematic and empirical comparison of alternatives. Too much architectural opinion is based on "here's a structural pattern that seems to make sense" rather than "here's a structural pattern that has been demonstrated to produce these effects". 

See my post on Layering Principles. You can't just take an SOA principle ("loose coupling is good", "layering is good"), apply it indiscriminately and expect SOA magic to occur.

Business Value from SOA - Sharecropping

This post starts by talking about the value proposition of Web 2.0, and then makes some more general points about the distribution of value in a layered system-of-systems.

---

A couple of years ago, Nick Carr suggested that a lot of Web 2.0 business was akin to sharecropping. The platform (MyFace or FlickTube or whatever) owns the space, and the users grow the content.

• Nick Carr, Sharecropping: The Long Tail, The Sharecroppers' Tools
• D'Arcy Norman, On Social Network Sharecropping, Sharecropping Clarification
• Jesse Vincent, Web 2.0 is Sharecropping
  

I think the most interesting aspect of this metaphor is the idea of a stratification of value, with different value proposition in each stratum. As Nick puts it "sharecroppers operate happily in an attention economy while their overseers operate happily in a cash economy".


Some people interpreted this as implying some kind of exploitation, and there have been vigorously defensive rebuttals of Carr's suggestion.

• Ed Felton, Sharecropping 2.0? Not Likely
• Mike Masnick, Are Sites Like Digg, YouTube And MySpace Still Exploitation If People Get What They Want?

Ed and Mike both insist that there is a connection between the attention economy and the cash economy. And this connection is evident from the success of Second Life as well, where there is a reasonable stable rate of exchange between virtual money and real money. For that matter, there is even a rate of exchange between virtual crime and real crime [Woman in Jail Over Virtual Murder].

But Nick isn't denying a connection between the platform provider's value proposition and the platform user's value proposition: indeed he too insists they are connected, but also points out that they are different.


Tim Bray had previously used the sharecropping metaphor to refer to independent developers building applications on major platforms. Here too we have a complex layered ecosystem, in which different stakeholders build value in a positive-sum game, but with some significant trust issues arising from the asymmetric relationships.

• The Web's The Place
  

Let me end my making a more general point, which I think is applicable to all kinds of layered systems-of-systems and not just Web 2.0. In a stratified ecosystem, there is at least the possibility that value is created by a platform provider, and that another kind of value is provided by the platform user, and that these two kinds of value interact in a complex way. Clearly we would like to know what kinds of stratification are most conducive to value creation, and this is a fascinating architectural question with no easy answers.

Homogeneous Business Vocabulary?

Nick Malik asks whether a common vocabulary is a blessing or curse.

Who benefits from a common vocabulary - whose agenda is it? IT architects tend to like a common vocabulary because it means they can more easily use the same systems and data stores; bureaucrats tend to like a common vocabulary because it means they can impose the same kinds of procedures and performance targets.

Let's look at the bureaucratic agenda first. In the old days, the bus company had passengers, hospitals had patients, prisons had prisoners, and universities had students. Now they are all supposed to be regarded as "customers", and driven by the same things: "choice" and "customer satisfaction". This reframing has had mixed results - perhaps a few beneficial effects in places, but also some damaging or absurd consequences.

• BBC News: Students: Customers or learners?
• Trisha Torrey: Patient as Customer
  

Meanwhile, IT organizations want to deploy similar solutions across a range of business domains. Here are a few examples picked at random.

• Unisys: Customer Loyalty Solution
• Anari Intelligence: Passenger and Customer Profiling (hedging their bets there!)

Meanwhile, IT architects are often lumpers rather than splitters, and so they like to produce information models with a relatively small number of highly generalized objects like PARTY, which mean absolutely nothing to a real business person.

So in some enterprises, and especially in the public sector, the IT architects may be aligned with the central bureaucrats against the line-of-business. Maybe sometimes there really is a good reason for the diversity of business vocabulary, not just idiot managers being obstinate.

With a stratified Service-Oriented Architecture, it becomes possible to get the best of both worlds - building some highly generic services in one layer, which support a range of different specialized and context-specific ontologies in the layer above. So it becomes possible to accommodate a broader range of requirements without imposing a common vocabulary. Of course this raises some complexity issues, which many IT architects would prefer not to have to deal with. For more on these complexity issues, see the Asymmetric Design blog.

ERP Platforms

In my previous post Does ERP matter? I discussed the business requirements for enterprise resource planning - a management style involving tight command-and-control coupling between different enterprise functions (sales & marketing, distribution, manufacturing, purchasing).

However, this is not the question addressed in Shai Agassi's original post Does ERP matter? His primary concern (as it has been for much of his time at SAP) is the impact of SOA on ERP. He avers (I think rightly) that enterprise software cannot be assembled from JBOWS (just a bunch of web services), largely because of concerns about semantic consistency and compliance.

In the past, major vendors persuaded their customers that the way to achieve semantic consistency and compliance was to purchase the entire application suite from a single supplier. While at SAP, Agassi has championed the creation of a radical SOA-based alternative to this strategy. If you build the rules into the ERP platform (e.g. Netweaver), then your chosen vendor (e.g. SAP) can maintain hegemony and control over a heterogeneous but properly architected collection of enterprise services.

Including of course the multiple inconsistent implementations of SAP software that global mergers sometimes throw up.

So what matters now, according to Agassi, is not the ERP but the ERP platform. If you are going to implement enterprise software, then use a platform that is specifically designed to support enterprise software, rather than a general-purpose software platform.

There are various other terms for this platform, including the bland (Business Process Platform or BPP) and the downright ugly (Applistructure). For recent commentary, see Sam Lowe, Michael "Mitch" Hatscher (via Matej) and Phil Windley (via John Gøtze).

Moving the platform upwards is a powerful architectural strategy, as I've argued before. But these strategies, while enabled and encouraged by SOA, remain challenging both technically and conceptually. It will be interesting to see whether this strategy goes on the back burner once Agassi is out of SAP.

SOA Sweet Spot

In a post entitled Your SOA is JABOWS, Microsoft's Nick Malik identifies what he calls the SOA Sweet Spot, which represents the intersection of two areas.

1. Typically, the greatest benefits of IT come from automating processes that execute often. It may be hard to cost-justify an n-person-year project to automate some process, if that process only occurs once a year, or on an exceptional basis.
2. Typically the greatest benefits of SOA come from supporting automation in areas that change frequently. As I said in my fourth post on BPM and SOA, the business case for SOA typically becomes stronger as the volatility increases.

Nick expresses this as a two-by-two matrix with Frequency of Occurrence along one axis and Frequency of Change along the other axis. The sweet spot is then in the top right quadrant.

Someone called Malcolm Anderson posted a comment to Nick's blog, challenging the difference between Frequency of Occurrence and Frequency of Change. Nick replied with a simple retail illustration.

Of course, if you choose to regard everything as undifferentiated process, then the two dimensions of Nick's matrix possibly collapse into a single dimension. This may be the point behind Malcolm's comment. Nick's matrix depends on an articulation of two different types of variation at two different logical levels - equivalent to the item and the batch (manufacturing) or the phenotype and the genotype (biological evolution). SOA allows us to implement a stratified solution in which these two types of variation are decoupled - yielding both economics of scale (based on frequency of occurrence) and economics of scope (based on frequency of change). This is of course an architectural solution, one which demands true SOA rather than JBOWS.

JBOWS or JaBoWS?

By the way, the term JBOWS appeared in an article by Joe McKendrick: The Rise of the JBOWS Architecture (or Just a Bunch of Web Services) (September 2005). Bobby Wolf of IBM (who picked up the term via James Governor) prefers to call it JaBoWS. I happen to prefer to stick with the term JBOWS, if only because an Internet search for Jabows yields all sorts of other stuff I'm not interested in.

Technorati Tags: SOA service-oriented

Business Service Architecture - Railway Edition

Martin Geddes draws some interesting parallels between the telecom system/infrastructure and the UK railway system/infrastructure, and discusses some of the effects of economic and political incentives in both systems. He says there are no easy answers, not even easy questions.

I agree about the (lack of) easy answers, but I think there are some important questions. My first question is an architectural one - what is the geometry of the platform stack?



Getting the stack right isn’t easy. The UK rail system got it disastrously wrong. The rail company (formerly Railtrack, now Network Rail) bought rail maintenance services from engineering companies, and sold rail availability to train operating companies.

But the service level agreements didn’t add up. What service levels do you need from the engineering companies in order to guarantee a given level of rail availability to the train operating companies? And how do you verify that you are getting the required service levels? (These are what I call algebraic problems.)

This is an extremely complex business, for which the rail company lacked the necessary coordination capability. One of the consequences of this incompetence was a serious rail crash, caused by grossly inadequate rail maintenance. But this isn’t just a question of local incompetence. There is a fundamental architectural flaw in the design of the stack as a whole, which fails to tackle some serious and complex questions.

The architectural question here is not just the proper distribution of capability between the layers of the stack, but the bridging between platforms with radically different ontologies - the ontology of rail maintenance is quite different to the ontology of train operation.

So although I agree with Martin about the economic and political problems, I think there are some deeper structural (geometrical, architectural) problems which make it impossible to fix the system merely by tweaking the economics and politics.


Related posts: Business Geometry (September 2004), Railway Edition 2 (August 2006), SOA Algebra (Jan 2007), Services Not All Like Laundry (July 2008)


updated 18 July 2017

SPARK 2 - Innovation or Trust?

Day 2 of the SPARK workshop was an attempt to develop frameworks that encapsulated the architectural response to the challenges identified in Day 1. The photoset on Flickr includes some dreadful pictures of me (here with Michael Platt of Microsoft and here with Nick Gall of Gartner), but it also includes copies of most of the flip charts.

The group I was in (which included Glen Harper, Dion Hinchcliffe and Michael Putz) tried to summarize some of the issues around the business stack.



Note the differential rate of change, as well as the gradients of innovation and trust. Note also the questions of horizontal and vertical coupling, which the group discussed but did not resolve.

This is a framework not a fixed solution. For example, in some cases the trust/compliance regime may be stricter (or at least different) at the top of the stack (think healthcare and HIPPA), but in most cases the greatest perceived risks will be associated with the major business assets (legacy) at the bottom of the stack. And (as Anne Thomas Manes reminded us) enterprise innovation isn't always going to be focused exclusively on customer-facing stuff, but may be focused on supply chain or product development or elsewhere.

Different technologies will be appropriate for different levels of the stack - for example we might expect to see SOAP and WS-* at the bottom of the stack (high trust, high engineering) and REST at the top of the stack (low trust, agile).

Of course, stratified architectures and stack diagrams are not new, but they have traditionally been produced from a purely technological perspective (client/service, 3-tier, n-tier computing, and so on). To my mind, the new architectural challenge here is to manage the stratification of layers in a way that responds in an agile and effective way to (the complexity of) the business/user challenges. (Hence Asymmetric Design.)


See also Beyond Bimodal (May 2016)

Layering Principles

Martin Fowler reports on a workshop in which people (a good group of people, [but] it was hardly a definitive source of enterprise development expertise) vote on principles for software layering.

Some people take this exercise seriously. JohnLim describes the result of the vote as excellent guidelines, while Paul Gielens adds his own vote. Others are more critical, including David Anderson and Rob Diana.

To my mind, even if you could collect up the most experienced people in the software world, I distrust the idea that you could get a coherent and meaningful set of architectural principles from a vote.

Architectural principles must come from reflective practice and/or grounded theory. For example, I can derive layering from a differential theory of change, as follows.


Purpose - What is layering supposed to achieve?

A well-layered artefact or system is more adaptable, because some types of variation can be accommodated within one layer without significantly affecting adjacent layers.


Form - What is the underlying structure of layering?

Boundaries between layers represent step changes with respect to some form of variation, from some perspective:

• Differential change over time
• A split between relatively homogeneous and relatively heterogeneous 

Process - How do layers get established? Layers emerge from an evolutionary process, in which a series of small alterations affect the architectural properties of a system or (often unplanned and unremarked by the so-called architects).

• Redundant layers (where there is insufficient difference in variation between two adjacent layers) tend to gradually fuse together. 
• Flexibility that is not used or exercised will attenuate. 
• Engineers under time pressure will take shortcuts that compromise the official separation between layers. 
• Where there is excessive differentiation within a single layer, this will tend to split apart, initially in an incoherent way.

Material - What is the source of a particular layering?

Layering comes from the experience of variation.



Related post: What's wrong with layered service models? (May 2009), Data Strategy - More on Agility (March 2020)

dotBiz

I have just come across an interesting talk by Shai Agassi of SAP, entitled Achieving Enterprise Agility.

Slides (pdf) from Accelerating Change 2004. Voicetrack (various formats, 18mB audio, 38 minutes) courtesy of IT Conversations.

The first 10 minutes or so provides some business background, including airline examples. Then he starts talking about composition, which I think is the most significant part of his talk. He has a vision of what he calls dotBiz (roughly equivalent to the Service-Based Business), for which he makes massive estimates of market size from around 2008 onwards. (After about 25 minutes he starts to rush through the rest of his material. It's probably not worth listening beyond this point.)

From the slides, you might get the impression of a ho-hum nothing-special software product architecture. But his talk implies something much more radical: a stratification of business.

At the top of the stack are niche companies with small solution-oriented microprocesses. This are assembled from "composites" - he refers to a "conveyor belt" of partly-assembled (orchestrated?) subprocesses. Below this are tens of thousands of business objects exposed as web services. These then sit on top of a layered enterprise platform, presumably supporting supply-side composition.

Agassi then characterizes the historical position of the big four, roughly as follows:

• IBM outsourcing non-core process - "on-demand" understood primarily in terms of variable volume of standardized process
• Microsoft preferred for local non-scaleable solutions, but not taken seriously for enterprise solutions - "start here but don't know whether it can scale"
• Oracle specializing in supporting big one-off differentiated process - "stuff you want to build on your own"
• SAP specializing in non-differentiated non-core process - "best practices that always run"

In other words, each represents a different mix of differentiation, integration and scale. Agassi's characterization provides a possible explanation of the current strategy of the big four: all trying to converge onto the same central ground, using SOA to get the best of all three worlds: differentiation, integration and scale. However, in this presentation, he doesn't get to the most interesting question: how business stratification helps this agenda.

Adaptation and Adaptability

Adam Brown from John McAslan and Partners gave a talk recently about his restoration/renovation work with Swiss Cottage Library. Dan Hill has some notes on his blog.

Interesting aside on how difficult it is to work with modernist buildings given their focus on functionality - if the function changes over time, the building can resist change. With the library, Brown kept alluding to how difficult it was to work with certain layers, given the amount of change required (not just in contemporary services etc, but in building in the modern notion of what a library is (internet access, coffee shops, DVD lending - as well as books).

This leads to a discussion about the adaptability of a modernist architecture.

1. Basil Spence is undoubtedly a great architect in many ways, and the Swiss Cottage Library remains beautiful, even if its functionality is now somewhat dated.

2. Functionalism involves a high degree of adaptation (to a given conception of function/functionality).

3. Adaptation conflicts with adaptability. The more you optimize to the present, the more you close off alternative futures.

4. Great buildings should be able to accommodate change, age gracefully. Perhaps one of the errors of modernism was to imagine that change would no longer be necessary.

---

The modernist attitude is widespread in software engineering. Model-driven architecture represents a type of modernism: form should follow function.

Stewart Brand's book on How Buildings Learn contains the notion of Shearing Layers - functional layers that have a different natural rate of change. A flexible structure allows each layer to be changed independently. But where the layers are too tightly coupled, the differential rates of change tear the structure apart.



The Shearing Layers concept is highly relevant to SOA.

---

Update: Stewart Brand has now introduced the term pace layering for the principle that stratification should be based on the differential rate of change. The term "shearing layers" refers to what happens when this principle is not followed.

Business Geometry

A business or value chain is composed in a geometric structure. (In the past we have often called this structure an architecture, but this word has lots of different and tangential associations for different people.)

In SOA, we design a business or value chain as a network of services. This is a powerful geometrical pattern. But there may be many possible network geometries satisfying a given business requirement, all of which count as satisfying the principles of SOA. For example: hub/spoke or peer/peer.

Business Stack

Another common geometric pattern for SOA is stratification. A business process is composed of services from a set of lower-level services, presented as a platform. A good example of a business platform is the set of retail services offered by Amazon and eBay. Other service providers have built further retail/logistical services on top of the Amazon/eBay platforms.

Each platform is in turn built upon even lower services. At the lower levels, there may be collections of IT-based services, known as ESB. There may also be sociotechnical service platforms, such as call centres.

Thus we have a stratified geometry, in which a person tackling a problem at a given level is presented with a collection of available services, formed into a virtual platform. This can be thought of as a business stack, with one plaform stacked on top of another. Some of the lower layers of the stack may appear to be purely technical services; however a more complete picture should reveal the existence of an IT organization maintaining the platform, in other words a human platform of administrators and programmers supporting the technical platform.

Variable Geometry, Variable Granularity

While the SOA principles may provide some geometrical guidance, and mandate certain geometrical patterns, there is still a design job to determine the geometry. This design job may be easy when the requirement is trivial, but gets harder as the complexity increases.

In many situations, the demand side has more variation than a human designer (or design team) can accommodate. (We characterize this as an asymmetry of demand, which calls for a process of asymmetric design.) We need to start thinking about variable geometry solutions, where the geometry itself can be adapted on-demand.

For example, in the past we have assumed that granularity has to be fixed at design time. But we can conceive of a web service platform that detects patterns of demand-side composition, defines new composite services automatically, describes and publishes these new services in real time, and notifies likely users of the new service, complete with an incentive to switch. We can conceive of a web service platform that analyses the message content of a certain service, and produces a substitute service with a smaller footprint that would satisfy most of the uses in a more elegant way.

Value Landscape

I use the term value landscape to refer to the distribution of cost, benefit and risk across a complex market ecosystem, such as the insurance industry. Technology (including SOA) influences business geometry, not least because it affects transaction costs. The shape of the value landscape changes (has already started to change) as the result of B2B and B2C and BPO. Companies that once occupied safe market positions (the high ground) may find their commercial advantage slipping into the sea, or they may find themselves cut off from their natural markets or supply chains.

See Microsoft Blog Insurance Value Chain

Let's suppose an insurance company has the following strategic aims:

1. Profitability, short-term viability. To deliver the maximum service value as cost-effectively as possible, using available input services and technologies as efficiently as possible, with minimum costs/risks of change.
2. Adaptability, medium-term viability. To understand and respond to changing demand for insurance services, and to trends in cost and risk, both internally and across the industry. To develop and deploy new services to exploit new business opportunities and avoid emerging business threats.
3. Survival, long-term viability. Making sure the core business proposition remains valid, and doesn't get eroded by more agile players. Taking strategic action in relation to structural changes in the insurance industry.

If we are doing business geometry for an insurance company, we need to think about the insurance industry as an ecosystem. We need an AS-IS model of the present ecosystem (largely based on pre-SOA technologies) and a TO-BE model of an idealized ecosystem (based on SOA). We can expect the pre-SOA ecosystem to evolve into some form of SOA ecosystem, although we may not have much idea which of the possible changes is going to happen first. To satisfy all three strategic aims, an insurance company needs to exploit the pre-SOA ecosystem, and prepare for the SOA ecosystem.

Note that this situation may force the insurance company to implement a variable geometry, both in the organizational platform and in the IT platform. Otherwise, it will either have to operate suboptimally for an extended period, or incur significant organization costs and IT costs every time the industry takes another step towards SOA.

SOA Change Management

With a layered picture of SOA, it is clear that each layer undergoes a lifecycle – but at different cycle times. Some of the layers have a real-time lifecycle measured in seconds, while others have a lifecycle measured in weeks, months or years. The real-time lifecycles are managed by the SOA platform/tools, while the slower lifecycles should be managed by change management tools and disciplines. Should be. Our observation is that some layers are well-managed, while others aren't.

SOA change management calls for each layer to be properly managed, and the links between layers to be properly managed. We need to know when a change in one layer impacts the next layer. (For example, if the functionality or performance or cost of a service changes, this might prompt retesting of a service composition or closer monitoring of system performance, and may even require redesigning the use of this service.)

And because each layer may be managed independently, change management across multiple layers becomes an exercise in collaboration. We need a publish/subscribe model not just for the services themselves, but for changes to the services.

Impact analysis may be both downwards and upwards. Sometimes the service provider needs some assurance that the service user is using the service properly. (For example, eBay demands certification.)

With proper encapsulation, some changes are private to the service provider and should not need to be notified to the service users. But it is not always clear exactly where to draw the dividing line between public changes and private changes. The service user may be forced to trust the service provider (and the whole service supply chain) to manage this encapsulation correctly, and to publish all changes that may impact the service use. But there are always going to be some service providers who get the encapsulation wrong, and there are always going to be some service uses that are too critical (business critical, safety critical) to rely on network trust alone.

The solution to this is partly organizational and partly technical. 360 degree intelligence tools can monitor the service network, identify patterns of behaviour that indicate significant changes, and publish independent change notifications. Thus you are not solely dependent on the service provider to tell you that the service has changed – you may get an alert because some other user of the same service has started to hit problems. Implementing this kind of intelligence is hard, because it not only requires different bits of technology to be integrated, but also good collaboration between multiple organizations to make the process work effectively.

CBDI Newswire (public access)
CBDI Report SOA LifeCycle (restricted access)

---

Update

Steven Cohn describes some SOA lifecycle requirements that he says go beyond the present capabilities of the Microsoft platform.