Agility and Variation 2

In a post called Kitchen Sink Variability, Harry "Devhawk" Pierson (Microsoft) lays into a research note by Ronald Smeltzer (Zapthink): Why Service Consumers and Service Providers Should Never Directly Communicate.

As Harry explains, there is a contradiction between Variability and the principles of Agile Development. (I discussed some aspects of this contradiction in my post on Agility and Variation.) Harry thinks Zapthink is advocating what he calls Kitchen Sink Variability - let everything vary except the kitchen sink.

Without architecture, this would indeed be a crazy idea.

Harry appeals to the ExtremeProgramming principle of YouArentGonnaNeedIt, which says "don't build in extra functionality". But does this principle apply to flexibility - does flexibility count as a kind of functionality? Should we really avoid building in flexibility?

If building in flexibility means constructing specific additional stuff (mechanisms, abstractions) to anticipate non-specific future variation, then this would seem to entail additional cost (development overhead, maintenance overhead, runtime overhead) with no clear benefits. We might reject this kind of thing as "over-engineering".

But if building in flexibility means using appropriate architectural patterns and existing technologies that allow you to ignore future variation, that seems to be a different thing altogether. Zapthink is talking about a specific pattern - decoupling the service consumer from the service provider - that is supported by current SOA platforms. That seems to make a lot of sense to me. But Zapthink justifies this pattern by appealing to a general principle of variability, and this is what has aroused Harry's scorn.

For my part, I don't think SOA means uncritically embracing all aspects of variation and variability. I have always said Loose Coupling is a choice, not a mandate. (See my earlier post Loose Coupling 2). But let's understand where the variation comes from. It is not because enterprise architects are going around promoting supply-side variation. More likely because many enterprise architects are struggling with increasing levels of demand-side variation. The traditional choice has always been to suppress as much variation as possible, but for many organizations (and their customers) this choice is no longer acceptable.

The relationship between variation and complexity is not linear. A truly agile architecture should be able to accommodate some increase in variation without increasing complexity by the same amount. But of course there is still some cost. SOA improves the trade-off, but there is still a trade-off.

The bottom line is that architects need to be able to reason intelligently about variation and variability. Not abstraction for the sake of abstraction, not decoupling for the sake of decoupling, but abstraction and decoupling driven by the needs of the enterprise. Loose Coupling is not a mandate but a choice.