Trusting the Schema

A long time ago, I did some work for a client that had an out-of-date and inflexible billing system. The software would send invoices and monthly statements to the customers, who were then expected to remit payment to clear the balance on their account.

The business had recently introduced a new direct debit system. Customers who had signed a direct debit mandate no longer needed to send payments.

But faced with the challenge of introducing this change into an old and inflexible software system, the accounts department came up with an ingenious and elaborate workaround. The address on the customer record was changed to the address of the internal accounts department. The computer system would print and mail the statement, but instead of going straight to the customer it arrived back at the accounts department. The accounts clerk used a rubber stamp PAID BY DIRECT DEBIT, and would then mail the statement to the real customer address, which was stored in the Notes field on the customer record.

Although this may be an extreme example, there are several important lessons that follow from this story.

Firstly, business can't always wait for software systems to be redeveloped, and can often show high levels of ingenuity in bypassing the constraints imposed by an unimaginative design.

Secondly, the users were able to take advantage of a Notes field that had been deliberately underdetermined to allow for future expansion.

Furthermore, users may find clever ways of using and extending a system that were not considered by the original designers of the system. So there is a divergence between technology-as-designed and technology-in-use.

Now let's think what happens when the IT people finally get around to replacing the old billing system. They will want to migrate customer data into the new system. But if they simply follow the official documentation of the legacy system (schema etc), there will lots of data quality problems.

And by documentation, I don't just mean human-generated material but also schemas automatically extracted from program code and data stores. Just because a field is called CUSTADDR doesn't mean we can guess what it actually contains.

Here's another example of an underdetermined data element, which I presented at a DAMA conference in 2008. SOA Brings New Opportunities to Data Management.

In this example, we have a sales system containing a Business Type called SALES PROSPECT. But the content of the sales system depends on the way it is used - the way SALES PROSPECT is interpreted by different sales teams.

• Sales Executive 1 records only the primary decision-maker in the prospective organization. The decision-maker’s assistant is recorded as extra information in the NOTES field. 
• Sales Executive 2 records the assistant as a separate instance of SALES PROSPECT. There is a cross-reference between the assistant and the boss

Now both Sales Executives can use the system perfectly well - in isolation. But we get interoperability problems under various conditions.

• When we want to compare data between executives
• When we want to reuse the data for other purposes
• When we want to migrate to new sales system 

(And problems like these can occur with packaged software and software as a service just as easily as with bespoke software.)

 

So how did this mess happen? Obviously the original designer / implementer never thought about assistants, or never had the time to implement or document them properly. Is that so unusual? 

And this again shows the persistent ingenuity of users - finding ways to enrich the data - to get the system to do more than the original designers had anticipated. 

 

And there are various other complications. Sometimes not all the data in a system was created there, some of it was brought in from an even earlier system with a significantly different schema. And sometimes there are major data quality issues, perhaps linked to a post before processing paradigm.

 

Both data migration and data integration are plagued by such issues. Since the data content diverges from the designed schemas, it means you can't rely on the schemas of the source data but you have to inspect the actual data content. Or undertake a massive data reconstruction exercise, often misleadingly labelled "data cleansing".

There are several tools nowadays that can automatically populate your data dictionary or data catalogue from the physical schemas in your data store. This can be really useful, provided you understand the limitations of what this is telling you. So there a few important questions to ask before you should trust the physical schema as providing a complete and accurate picture of the actual contents of your legacy data store.

• Was all the data created here, or was some of it mapped or translated from elsewhere? 
• Is the business using the system in ways that were not anticipated by the original designers of the system? 
• What does the business do when something is more complex than the system was designed for, or when it needs to capture additional parties or other details?
• Are classification types and categories used consistently across the business? For example, if some records are marked as "external partner" does this always mean the same thing? 
• Do all stakeholders have the same view on data quality - what "good data" looks like?
  
• And more generally, is there (and has there been through the history of the system) a consistent understanding across the business as to what the data elements mean and how to use them?
  

Related posts: Post Before Processing (November 2008), Ecosystem SOA 2 (June 2010), Technology in Use (March 2023)

Customer Multiple

A friend of mine shares an email thread from his organization discussing the definition of CUSTOMER, disagreeing as to which categories of stakeholder should be included and which should be excluded.

Why is this important? Why does it matter how the CUSTOMER label is used? Well, if you are going to call yourself a customer-centric organization, improve customer experience and increase customer satisfaction, it would help to know whose experience, whose satisfaction matters. And how many customers are there actually?

My friend's organization provides services to A, which are experienced by B and paid for by C, based on a contractual agreement with D. This is a complex network of actors with overlapping roles, and the debate is about which of these count as customers and which don't. I have often seen similar confusion elsewhere.

My friend asks: Am I supposed to have a different customer definition for different teams (splitter), or one customer definition across the whole business (lumper)? As an architect, my standard response to this kind of question is: it depends.

One possible solution is to prefix everything - CONTRACT CUSTOMER, SERVICE CUSTOMER, and so on. But although that may help sort things out, the real challenge is to achieve a joined-up strategy across the various capabilities, processes, data, systems and teams that are focused on the As, the Bs, the Cs and the Ds, rather than arguing as to which of these overlapping groups best deserves the CUSTOMER label.

Sometimes there is no correct answer, but a best fit across the board. That's architecture for you!

 

Many business concepts are not amenable to simple definition but have fuzzy boundaries. In my 1992 book, I explain the difference between monothetic classification (here is a single defining characteristic that all instances possess) and polythetic classification (here is a set of characteristics that instances mostly possess). See also my post Modelling Complex Classification (February 2009).

But my friend's problem is a slightly different one: how to deal with multiple conflicting monothetic definitions. One possibility is to lump all the As, Bs, Cs and Ds into a single overarching CUSTOMER class, and then provide different views (or frames) for different teams. But this still leaves some important questions open, such as which of these types of customer should be included in the Customer Satisfaction Survey, whether they all carry equal weight in the overall scores, and whose responsibility is it to improve these scores.

 

In her book on medical ontology, Annemarie Mol develops Marilyn Strathern's notion of partial connections as a way of overcoming an apparent fragmentation of identity - in our example, between the Contract Customer and the Service Customer - when these are sometimes the same person.

Being one shapes and informs the other while they are also different identities. ... Not two different persons or one person divided into two. But they are partially connected, more than one, and less than two. Mol pp 80-82

Mol argues that frictions are vital elements of wholes,

... a tension that comes about inevitably from the fact that, somehow, we have to share the world. There need not be a single victor as soon as we do not manage to smooth all our differences away into consensus. Mol p 114

Mol's book is about medical practice rather than commercial business, but much of what she says about patients and their conditions applies also to customers. For example, there are some elements that generally belong to "the patient", and although in some cases there may be a different person (for example a parent or next-of-kin) who stands proxy for the patient and speaks on their behalf, it is usually not considered necessary to mention this complication except when it is specifically relevant.

Similar complexities can be found in commercial organizations. Let's suppose most customers pay their own bills but some customers have more complicated arrangements. It should be possible to hide this kind of complexity most of the time.

Human beings can generally cope with these elisions, ambiguities and tensions in practice, but machines (by which I mean bureaucracies as well as algorithms) not so well.  Organizations tend to impose standard performance targets, monitored and controlled through standard reports and dashboards, which fail to allow for these complexities. My friend's problem is then ultimately a political one, how is responsibility for "customers" distributed and governed, who needs to see what, and what consequences may follow.

(As it happens, I was talking to another friend yesterday, a doctor, about the way performance targets are defined, measured and improved in the National Health Service. Some related issues, which I may try to cover in a future post.)

---

Annemarie Mol, The Body Multiple: Ontology in Medical Practice (Duke University Press 2002)

Richard Veryard, Information Modelling - Practical Guidance (Prentice-Hall 1992) 

• Polythetic Classification Section 3.4.1 pp 99-100
• Lumpers and Splitters Section 6.3.1, pp 169-171
  

Boundary Objects and Artful Integration

I once did some data architecture and modelling for NHS Blood and Transplant. This is a UK-wide agency responsible for blood, organs and other body parts, managing transfers from donors to recipients.

One of the interesting challenges for this kind of organization is the need for collaboration between different specialist disciplines. Some teams are responsible for engaging with potential and regular donors, encouraging and arranging donation sessions for blood and plasma. Meanwhile there are other teams who need an extremely precise biomedical profile of each donor, to ensure safety as well as identifying people with rare blood types. While there is a conceptual boundary between these two sets of concerns, the teams need to collaborate effectively and reliably across this boundary.

So in terms of data and interoperability, we have an entity (in this case the donor) that is viewed in significantly different ways, but with a common identity. In the past, I've talked about two-faced entities or hinge entities, but the term that is generally used nowadays is Boundary Object.

We define boundary object as those objects that both inhabit several communities of practice and satisfy the informational requirements of each of them. Bowker and Star p 16

Boundary objects are the canonical forms of all objects in our built and natural environments. Bowker and Star p 307

In general, such boundary objects tend to be weakly structured, while being linked to much stronger structures in each separate domain. While boundary objects are considerably more than just data objects, they raise important questions for the data architect, who needs a critical eye for possible multiplicity or misfit that might compromise interoperability.

Simple multiplicity occurs when there are different levels of granularity each side of the boundary - one side lumping things together, the other side splitting them apart. In a library, for example, the people responsible for the catalogue may understand BOOK to refer to the title, while the people responsible for managing loans may want each physical copy to be represented as a separate instance of BOOK. And while people outside a warehouse may be happy with a notion of STOCK LOCATION that simply points to the warehouse as a single location, people inside the warehouse will want a more fine-grained notion, telling them more precisely where the stock can be found - for example, which shelf in which aisle.

Misfits can occur when there are competing notions of inclusion or classification - for example, does PRODUCT include the products and services of our partners as well as our own, does it include legacy products we no longer sell, does it include products that haven't been launched yet?

And with people, it may not be clear whether we are interested in the person or the role.

One way of detecting these issues is simply to ask how many there are. If you get widely different answers, this is a pretty good indicator that people aren't talking about the same thing. But sometimes the discrepancies can be more subtle and harder to detect.

And resolving (brokering) these issues can often be a political challenge, as Kimble et al argue, not merely a technical one. Specialists may be reluctant to share even a highly simplified version of their view of an object, for fear that this information might be misunderstood and misapplied. And yet there may be some value in sharing some of this information. Furthermore, there may be considerable resistance to relaxing any of the strong constraints on either side of the boundary. So the data architect needs to negotiate exactly what the boundary object will be and how much it should contain.

In his earlier writings, Étienne Wenger described this as a broker role.

The job of brokering is complex. It involves processes of translation, coordination and alignment between perspectives. It requires enough legitimacy to influence the development of a practice ... it also requires the ability to link practices by facilitating transactions between them and to cause learning by introducing into a practice, elements of another. Wenger 1998, p 109

He now talks more generally about system convening, which combines and reinterprets several different roles, including that of broker.

If a group needs some scaffolding and enabling, call a facilitator. A broker is ideal for helping to translate ideas from one practice to another. A weaver will join the dots, strategically connecting people into new networks. An inspiring visionary with charisma is not necessarily a systems convener. Nor is a person who convenes an event or manages systems change or multi-stakeholder processes. None of these roles in themselves are systems convening, although systems conveners often play some of them and it is quite possible that a person reinterpreting one of these roles ends up adopting a systems-convening approach. Wenger-Trayner 2021, p 28

The creation of boundary objects seems to require a combination of these roles. Lucy Suchman talks about artful integration, attempting to shift the frame of design practice and its objects from the figure of the heroic designer and associated next new thing, to ongoing, collective practices of sociomaterial configuration, and reconfiguration in use. 

 

---

 

Geoffrey Bowker and Sarah Leigh Star, Sorting Things Out (MIT Press, 1999) - online extract

Mary L Darking, Integrating on line learning technologies into higher education (LSE 2004) 

Chris Kimble, Corinne Grenier and Karine Goglio-Primard, Innovation and Knowledge Sharing Across Professional Boundaries: Political Interplay between Boundary Objects and Brokers (International Journal of Information Management, October 2010)

Lucy Suchman, Located Accountabilities in Technology Production, (Centre for Science Studies, Lancaster University, 2000-2003)

Etienne Wenger, Communities of Practice: Learning, Meaning, and Identity (New York: Cambridge University Press, 1998)

Etienne and Beverly Wenger-Trayner, System Convening: A crucial form of leadership for the 21st century (Social Learning Lab, 2021)

Wikipedia: Boundary Object

Metadata as a Process

Data sharing and collaboration between different specialist areas requires agreement and transparency about the structure and meaning of the data. This is one of the functions of metadata.

I've been reading a paper (by Professor Paul Edwards and others) about the challenges this poses in interdisciplinary scientific research. They identify four characteristic features of scientific metadata, noting that these features can be found within a single specialist discipline as well as cross-discipline.

• Fragmentation - many people contributing, no overall control
  
• Divergent - multiple conflicting versions (often in Excel spreadsheets)
• Iterative - rarely right first time, lots of effort to repair misunderstandings and mistakes
• Localized - each participant is primarily focused on their own requirements rather than the global picture

They make two important distinctions, which will be relevant to enterprise data management as well.

Firstly between product and process. Instead of trying to create a static, definitive set of data definitions and properties, which will completely eliminate the need for any human interaction between the data creator and data consumer, assume that an ongoing channel of communication will be required to resolve emerging issues dynamically. (Some of the more advanced data management tools can support this.)

Secondly between precision and lubrication. Tight coupling between two systems requires exact metadata, but interoperability might also be achievable with inexact metadata plus something else to reduce any friction. (Metadata as the new oil, perhaps?)

Finally, they observe that metadata typically falls into the category of almost standards.

Everyone agrees they are a good idea, most have some such standards, yet few deploy them completely or effectively.

Does that sound familiar? 

---

J Bates, The politics of data friction (Journal of Documentation, 2017)

Paul Edwards, A Vast Machine (MIT Press 2010). I haven't read this book yet, but I found a review by Danny Yee (2011)

Paul Edwards, Matthew Mayernik, Archer Batcheller, Geoffrey Bowker and Christine Borgman, Science Friction: Data, Metadata and Collaboration (Social Studies of Science 41/5, October 2011), pp. 667-690. 

Martin Thomas Horsch, Silvia Chiacchiera, Welchy Leite Cavalcanti and Björn Schembera, Research Data Infrastructures and Engineering Metadata. In Data Technology in Materials Modelling (Springer 2021) pp 13-30

Jillian Wallis, Data Producers Courting Data Reusers: Two Cases from Modeling Communities (International Journal of Digital Curation, 2014, 9/1, 2014) pp 98–109

The Ethics of Interoperability

In many contexts (such as healthcare) interoperability is considered to be a Good Thing. Johns and Stead argue that "we have an ethical obligation to develop and implement plug-and-play clinical devices and information technology systems", while Olaronke and Olusola point out some of the ethical challenges produced by such interoperability, including "data privacy, confidentiality, control of access to patients’ information, the commercialization of de-identified patients’ information and ownership of patients’ information". All these authors agree that interoperability should be regarded as an ethical issue.

Citing interoperability as an example of a technical standard, Alan Winfield argues that "all standards can ... be thought of as implicit ethical standards". He also includes standards that promote "shared ways of doing things", "expressing the values of cooperation and harmonization".

But should cooperation and harmonization be local or global? American standards differ from European standards in so many ways - voltages and plugs, paper sizes, writing the date the wrong way. Is the local/global question also an ethical one?

One problem with interoperability is that it is often easy to find places where additional interoperability would deliver some benefits to some stakeholders. However, if we keep adding more interoperability, we may end up with a hyperconnected system of systems that is vulnerable to unpredictable global shocks, and where fundamental structural change becomes almost impossible. The global financial systems may be a good example of this.

So the ethics of interoperability is linked with the ethics of other whole-system properties, including complexity and stability. See my posts on Efficiency and Robustness. Each of these whole-system properties may be the subject of an architectural principle. And, following Alan's argument, an (implicitly) ethical standard.

With interoperability, there are questions of degree as well as of kind. We often distinguish between tight coupling and loose coupling, and there are important whole-system properties that may depend on the degree of coupling. (This is a subject that I have covered extensively elsewhere.)

What if we apply the ethics of interoperability to complex systems of systems involving multiple robots? Clearly, cordination between robots might be necessary in some situations to avoid harm to humans. So the ethics of interoperability should include the potential communication or interference between heterogeneous robots, and this raises the topic of deconfliction - not just for airborne robots (drones) but for any autonomous vehicles. Clearly deconfliction is another (implicitly) ethical issue. 

---

Note: for a more detailed account of the relationship between interoperability and deconfliction, see my paper with Philip Boxer on Taking Governance to the Edge (Microsoft Architecture Journal, August 2006). For deconfliction in relation to Self-Driving Cars, see my post Whom does the technology serve? (May 2019).



Mauricio Castillo-Effen and Nikita Visnevski, Analysis of autonomous deconfliction in Unmanned Aircraft Systems for Testing and Evaluation (IEEE Aerospace conference 2009)

Michael M. E. Johns and William Stead, Interoperability is an ethical issue (Becker's Hospital Review, 15 July 2015)

Milecia Matthews, Girish Chowdhary and Emily Kieson, Intent Communication between Autonomous Vehicles and Pedestrians (2017)

Iroju Olaronke and Olaleke Janet Olusola, Ethical Issues in Interoperability of Electronic Healthcare Systems (Communications on Applied Electronics, Vol 1 No 8, May 2015)

Richard Veryard, Component-Based Business (Springer 2001)

Richard Veryard, Business Adaptability and Adaptation in SOA (CBDI Journal, February 2004)

Alan Winfield, Ethical standards in robotics and AI (Nature Electronics Vol 2, February 2019) pp 46-48



Related posts: Deconfliction and Interoperability (April 2005), Loose Coupling (July 2005), Efficiency and Robustness (September 2005), Making the world more open and connected (March 2018) Whom does the technology serve? (May 2019)

Making the World More Open and Connected

Last year, Facebook changed its mission statement, from "Making The World More Open And Connected" to "Bringing The World Closer Together".

As I said in September 2005, interoperability is not just a technical question but a sociotechnical question (involving people, processes and organizations). (Some of us were writing about "open and connected" before Facebook existed.) But geeks often start with the technical interface, or what is sometimes called an API.

For many years, Facebook had an API that allowed developers to snoop on friends' data: this was shut down in April 2015. As Constine reported at the time, this was not just because the API was "kind of shady" but also to "deny developers the ability to build apps ... that could compete with Facebook’s own products". Sandy Paralikas (himself a former Facebook executive) made a similar point (as reported by Paul Lewis): Facebook executives were nervous about the commercial value of data being passed to other companies, and worried that the large app developers could be building their own social graphs.

In other words, the decision was not motivated by concern for user privacy but by the preservation of Facebook's hegemony.

When Tim Berners-Lee first talked about the Giant Global Graph in 2007, it seemed such a good idea. When Facebook launched the Open Graph in 2010, this was billed as "a taste of the future where everything can be more personalized". Like!

---

Philip Boxer and Richard Veryard, Taking Governance to the Edge (Microsoft Architecture Journal, August 2006)

Josh Constine, Facebook Is Shutting Down Its API For Giving Your Friends’ Data To Apps (TechCrunch, 28 April 2015)

Josh Constine and Frederic Lardinois, Everything Facebook Launched At f8 And Why (TechCrunch, 2 May 2014)

John Lanchester, You Are the Product (London Review of Books, 17 August 2017)

Paul Lewis, 'Utterly horrifying': ex-Facebook insider says covert data harvesting was routine (Guardian, 20 March 2018)

Caroline McCarthy, Facebook F8: One graph to rule them all (CNet, 21 April 2010)

Sandy Parakilas, We Can’t Trust Facebook to Regulate Itself (New York Times, 19 November 2017)

Wikipedia: Giant Global Graph, Open API,


Related Posts SOA Stupidity (September 2005), Social Networking as Reuse (November 2007), Security is Downstream from Strategy (March 2018), Connectivity Hunger (June 2018)

Knowledge and Memory

Once upon a time, people thought of an information model as defining the structure of the stuff you want to remember. Nowadays, this definition is too restrictive: it might possibly be adequate for a system/database designer, but is not adequate for an architect. 

Here are three examples where my knowing and using information is not dependent on my remembering it.

1. I recently helped an SME with its PCI DSS submission. One of the critical points is that they avoid a lot of trouble by NEVER storing their customers' credit card details, merely transmitting these details directly to Barclaycard using a secure device. Clearly the credit card details are stored in various places elsewhere, but our systems don't store them anywhere, not even in cache.

2. When I did my driving test, many years ago, the Highway Code had a table of stopping distances that you were supposed to remember. I decided it would be easier to remember the formula and calculate as required. Obviously that's a design decision.

3. My computer remembers frequently used email addresses. But if I want to get back in touch with someone, I will use Linked-In or Google to find his current email address rather than use the one on my computer which may be out-of-date.

The system/database developer and the architect may use the same patterns for defining an entity or entity type, but they have different agendas, and therefore different success criteria. The architect may be less rigorous about some aspects, and needs to be more rigorous about some other aspects.

The system/database developer may see an entity as "something you need to remember etc.". An architect sees an entity as "something you need to exchange information about". The information model establishes a consistent basis for information exchange between people, systems, services, processes and organizations. I can only talk with you about customers, share your customer data, use your customer-related services, and so on, if either (a) you and I have the same understanding of CUSTOMER or (b) there is a mapping between your notion of CUSTOMER and mine. We can call this semantic interoperability.

This idea of semantic interoperability underlies the Open Group vision of Boundaryless Information Flow.

If you have a private list of customers on your laptop, then you can identify them in any adhoc manner you choose. But if you want to share the list with other people, the identity criteria become important.

So which entity types is the architect is most interested in? Primarily the ones that are referenced in the interfaces between systems and services. There are lots of other things that you might wish to remember, monitor and/or direct, but if they can be encapsulated inside some service or component they have no architectural significance. For example, the business needs to know the prevailing VAT rate; so you build a common VAT routine with the VAT rate hidden inside.

BYOD - Bring Your Own Device

By popular demand, many companies are shifting ownership of elements of corporate infrastructure onto their employees. This is known as BYOC (bring your own computer) or BYOD (bring your own device).

There are many aspects to this trend.

1. Culture. Talented recruits may see this kind of choice as a desirable feature of a future employer. Some of them may have a strong personal commitment to a particular device; others may ask about BYOD policy as a quick way of getting a general impression of company culture and its attitude towards employees.

(Even if BYOD is a common request at interview, this doesn't mean it is a genuine requirement. In some cases, the BYOD request could be similar to the apparently crazy riders that performers may add to contracts as a way of testing the diligence and attention to detail of the organizers. The best-known example of such a contract rider is Van Halen's insistence on a bowl of MnMs with the brown ones removed. See "Brown out" at snopes.com.)

2. Interoperability. There is a need for interoperability within the enterprise (endo-interoperability) as well as interoperability with external platforms (exo-interoperability). Within the enterprise, people expect to be able to use common services (email, communications, content management, and so on) regardless of device. When I'm in the office, I want to be able to connect my device to office devices such as printers and projectors, as well as using the office network and servers. When I'm working at home, I want to be able to connect my device into the office systems, and use my device for web conferences and other events. But I also want to be able to connect my device into public platforms such as Facebook.


3. Innovation. Early adopters like to carry the latest and most fashionable device, even if this doesn't yet support all the required corporate services in a robust manner.

4. Business continuity and risk. A person's productivity can be seriously impaired if the device is lost or develops a fault. Conversely, a company's security can be seriously impaired if an employee uses an unverified emergency device such as her teenage son's phone. Does BYOD imply the rapid availability of backup devices of every conceivable brand, or does the company provide a limited range of standard devices for emergency use?

5. Support. Does the device deliver all the required corporate services correctly, efficiently and securely? Whose responsibility is it to verify and test these services on the given device, and to sort out the (inevitable) configuration problems? What knowledge and expertise is needed to provide adequate support across the full range of devices?

6. Economics. Device provision within large organizations was traditionally based on the economics of scale. We purchase thousands of identical devices, install the same software and services on each one, and issue these to our employees. We can obtain good discounts from the hardware and software suppliers, and we can train our support staff to provide efficient support across a narrow range of products. But this approach fails to deal with the complexities of the modern business organization where each employee has different needs, often calling for additional non-standard software and services, or even newer devices. So most modern organizations shift to provision of devices based on the economics of scope - giving everyone a flexible device platform to which additional software and services can be easily added. Then the move to BYOD takes us into the economics of alignment - optimizing the lifetime cost of device provision against the lifetime benefits to the organization and the individual within the context of use.

7. BYOD represents a shift in the balance between two kinds of device vendor - the ones who sell thousands of devices at a time by schmoozing the CIO and the ones who sell devices to individuals via consumer channels. (As a result, some stakeholders may be cynical and unsympathetic to any objection to BYOD from the CIO quarter.)

8. More fundamentally, BYOD represents a shift in the balance of power between two kinds of knowledge. The corporate IT folk supposedly know more about the corporate services and about quality attributes such as reliability and security. However, the individual employee knows more about the context of use. The architectural question here is aligning the device selection, configuration and use with the emerging requirements of the individual in the job. This is ultimately a question of governance, which needs to be guided by appropriate BYOD policies.


A lot of architectural issues then.

---

Fiona Graham, BYOC: Should employees buy their own computers? (BBC News 14 January 2011)

Fiona Graham, BYOD: Bring your own device could spell end for work PC (BBC News 14 February 2012)

Eric Vanderburg, Four Keys to Successful BYOD (CIO 14 February 2012)


Related posts:

Bring your own expectations (May 2014)

Notes on Interface

At an architecture workshop in Vienna last week, I put together a few slides to address some questions about interfaces. I promised to post an expanded and cleaned-up version, and here it is. Some of this material was developed with my former colleague Lawrence Wilkes. If you have more questions please ask.

Notes on Interface from Richard Veryard

Towards an Enterprise Architecture of Knowledge

A customary way of modelling an enterprise is as a collection of purposeful behaviours - functions or processes or capabilities or whatever. Each function or process or capability encapsulates some knowledge or know-how. Sometimes this knowledge will be fairly static, so the behaviour will be stable and unchanging. And sometimes the underlying knowledge will change, so the behaviour will need to be adjusted to accommodate the most up-to-date and relevant knowledge. In some areas, such as product development and marketing, there may be a strategic imperative to find and assimilate new knowledge and ideas (about market trends, technology trends, competitor trends, and so on), to build new collaborative partnerships, and to experiment with new behaviours. These areas will rely on business intelligence - not just BI tools but all the practices associated with collecting and making sense of complex information.

The overall architecture of the enterprise can then be understood in terms of how these functions or processes or capabilities interoperate to produce a viable system-of-systems. The enterprise architect must understand which of these functions are relatively stable, which of them are strategically volatile, and create appropriate mechanisms for coordinating between them. These mechanisms are essentially behavioural ones - information flows, event triggers, process flows and so on.

But if we look at the enterprise through a knowledge management lens, we will see a different kind of structure. There is a series of overlapping knowledge domains - marketing, accounting, regulatory compliance, health and safety, employment, technology - each with its own specialist terminology and ways of thinking. So there is an architectural question about how these knowledge domains (we might also call them agendas or discourses) interoperate.

In a traditional organization, these knowledge domains only come together at board level, with one or more directors speaking for each significant domain. Thus the CTO speaks for the technology domain, the HR director speaks for the employment domain, and so on. Of course all the directors will try to influence the agenda of their peers - so everyone else might be citing random snippets of evidence of technology trends, or challenging the CTO's decisions and interpretations, pushing the CTO to explain or change direction. In a well-functioning organization, this will be all done in the spirit of healthy and vigorous debate.

In a hierarchical organization, some pieces of this debate can of course be delegated - either to internal departments or to external consultants - but the directors remain responsible for putting the pieces together.  

Thus the coherence and intelligence of a traditional organization depends heavily on the collective intelligence and effective functioning of the board of directors. But there are several evident problems with this approach. Firstly, we can observe many organizations that lack coherence and intelligence, so this approach is manifestly not working well enough for these organizations. Secondly, we may assume that the board of directors has a finite reasoning capacity - however clever they are as individuals, and however well-bonded they are as a team - and that they will not always be able to keep up with the increasing complexity and volatility of the business environment. And thirdly, we can see a lack of joined-up intelligence at the lower levels of the organization.

Some people think there is a different way of integrating an enterprise. If we can identify a single universal knowledge domain, then we can use this knowledge domain to join up everything else. For example, in a financially driven organization, management accounting acts as a unifying language across all functions. Conversely, many enterprise architects regard their favourite EA framework as providing such a unifying language. Although there have certainly been very large organizations that have been managed according to a single unifying principle (such as GEC under Lord Weinstock), the fact that there are several specialisms each wanting to be top dog helps to explain why this doesn't work very often.

A third idea would be a bottom-up Enterprise 2.0 swarm intelligence, in which a satisfactory resolution to all difficult issues emerges from uncontrolled debate and "sharing" across the organization. To read some of the material on the internet, it's tempting to believe that this is how some of the really cool organizations in Silicon Valley are organized. But even if this were true, there is still a need for some kind of architecture and governance, for example, providing suitable platforms and pathways for constructive and reasonably efficient problem-solving. Kevin Kelly, whose book Out of Control introduced me and many other people to the huge potential of bottom-up intelligence, is careful to warn that the bottom is not enough.

Which brings me to the fourth idea - regarding the coherence and intelligence of the knowledge-bearing organization as a real architectural issue. Not the kind of structure that enterprise architects are accustomed to modelling or managing perhaps, but this structure may have a significant effect on the business outcomes of the organization. Isn't this something enterprise architects should be interested in?

From Espresso to Instant

Starbucks is changing its business model. Or as CEO Howard Schultz tells the Huffington Post, Staying Real in an Instant. Starbucks will be selling shots of instant coffee, for under a dollar a cup. The UK price is said to be around 60p.

Some people may have thought that "espresso" was the Italian word word for speed. (It isn't - it means "pressed".) So what could be faster than express coffee? Instant coffee!

Of course the word "instant" isn't about getting the coffee more quickly either, it is about doing away with all that fancy machinery, in whose use Starbucks makes such a charade of training its baristas. (A year ago, Schultz ordered all US stores to close for a three-hour training session "as part of an effort to improve coffee quality and revive the chain's flagging fortunes" [Guardian, 26 Feb 2008].)

Shultz now claims to be responding to the increasing mobility of consumers. "Imagine a cup of Starbucks VIA Ready Brew on a mountaintop" he says, as if willing us to imagine millions of Starbucks customers on some remote and implausible trek.

But clearly his real interest is selling mass market coffee. He hopes that the Starbucks instant coffee will be not only better-tasting but also "paradigm-changing" (whatever that means), and hopes "to turn on a whole new set of coffee drinkers to the Starbucks brand". But the obvious risk is that the old set will be turned off. He acknowledges that this move is a gamble (he calls it "a considered bet"), and expects "to learn a lot ... over the coming weeks". You bet.

In what sense does this count as a new business model? Starbucks already sells ground coffee and coffee beans in supermarkets across the USA. Many rival coffee purveyors have already shifted to the Gillette model, in which the coffee machines are sold cheap or practically given away, and you make your money selling overpriced pods of coffee.

The challenge faced by Starbucks is not choosing one business model, but attempting to combine two or three different (and possibly incompatible) business models at the same time. Such composition faces questions of cross-subsidy, brand dilution or erosion. Are there any reliable rules or patterns governing the interoperability (compatibility and composition) of business models?

See also

• Starbucks to Combat Brand Erosion with Loyalty Program (Marketing Vox, March 2008)
• Starbucks: How Growth Destroyed Brand Value (Business Week, July 2008)

Update

• Why Starbucks should become a retail bank (John Gapper, FT, May 2009)
  

Market Predictions - CEP

What lies in store?

Complex Event Processing (CEP) guru David Luckham asks What are your predictions for 2009?

What challenges?

Prediction is hard, especially for experts. What is easier is to identify some of the challenges that have to be addressed. For example:

• Detection (Tim Bass)
  
• Interoperability (Opher Etzion, Tim Bass)
• Scope and Terminology (Opher Etzion, Paul Vincent)
  

CEP Products or Services?

David's starting point was a Forrester estimate of the market for CEP software and services. But what kind of services are these? Is Forrester just talking about conventional systems integration services - e.g. paying consultants to build and install your CEP systems? Or are we starting to see a genuine service economy based around the trading and collaborative processing of events?

A certain amount of this kind of thing goes on in the security world, with specialist firms performing what is essentially collective event processing for a number of customers (Monitoring-as-a-Service). But apart from that, I haven't seen much evidence of a distributed economy of complex events.

But why might this kind of thing be particularly interesting in 2009? Because the prevailing economic environment may make it harder to justify people going it alone, building large and complex event-processing applications for their own use.

Shortening the lead

Most people are expecting 2009 to be a tough year. In such circumstances, there is a widespread reluctance to invest scarce resources on remote gains; any vendors trying to sell solutions to business will need to find innovative ways of shortening and tightening the lead between investment and return.

For example, instead of vendors offering an elaborate set of CEP products, together with consultants skilled in wiring them together, there may be demand for out-of-the-box hosted solutions.

Shared Event Semantics

But in order to share events and event processing, firms will need to have a common event model. Which brings us back to some of the challenges identified by Opher, Paul and Tim ...

And Not Just Complex Event Processing ...

These arguments don't just apply to complex event processing, but to many areas of new technology. I'll look at some other areas in subsequent posts ...

Progressive Design Constraints

When I wrote my piece on Post-Before-Processing, I wasn't thinking about how it might apply to the design process. So when I read Saul Caganoff's reply, on Progressive Data Constraints, my first reaction was, well that's interesting but completely different. But when I read Saul's piece again more slowly, I started to see a common pattern between designing an engineering system (using CAD) and designing a response to a complex unstructured set of events (using what first Vickers and then Checkland call an "appreciative system"). 

In both cases, there is a pitfall known as "jumping to conclusions" - seeking premature closure as a result of an inability to tolerate incompleteness, inconsistency or uncertainty. There is a related pitfall of psychological attachment - being unable to abandon or revise one's earlier decisions. One of the most important skills for the business analyst or systems designer is the ability to throw away his/her first attempt and start again, or to make radical revisions to an existing artefact. (I once built an advanced modelling class with a series of exercises designed to give students opportunities to practise this skill. I think classes like this are still pretty rare.)

There are two apparently opposite approaches to design. According to some design methodologies, we are supposed to start with a vague, generic and all-inclusive concept, and gradually add specific detail and refinement until we have something that we can implement. Work-in-progress should always be consistent and horizontally complete - it just lacks vertical completeness. Alternatively, we start with a bunch of conflicting and sometimes incoherent requirements, and try to find a way of satisfying as much of them as possible. Saul's description fits the second of these.

Bringing the topic back to SOA at the end of his post, Saul indicates the relevance of this kind of thinking to SOA design-time governance. Perhaps because of my background in Open Distributed Processing (ODP), I always think of SOA in terms of open distributed systems, whose design is never complete, always open-ended. So design-time governance always spills over into run-time governance, or is it the other way around?

If we take the SOA project seriously, we should be building systems and services that will interoperate with third-party systems and services, using unstructured data and complex events from a broad range of heterogeneous sources, to be reused and repurposed in user contexts we haven't yet thought about. So we have to be prepared for the emergence of "undocumented features" or "anomalies" (or whatever we want to call them) - and these may emerge at any point in the lifecycle.

Of course in mission-critical or safety-critical systems, there are certain categories of system failure that cannot be permitted. And even for non-critical systems, there is a limit to the amount of buggy behaviour that users will tolerate. But we don't achieve an acceptable level of quality in any kind of complex service-oriented systems engineering by pretending we can design and test our way to perfection in a single step.

Homogeneous Business Vocabulary?

Nick Malik asks whether a common vocabulary is a blessing or curse.

Who benefits from a common vocabulary - whose agenda is it? IT architects tend to like a common vocabulary because it means they can more easily use the same systems and data stores; bureaucrats tend to like a common vocabulary because it means they can impose the same kinds of procedures and performance targets.

Let's look at the bureaucratic agenda first. In the old days, the bus company had passengers, hospitals had patients, prisons had prisoners, and universities had students. Now they are all supposed to be regarded as "customers", and driven by the same things: "choice" and "customer satisfaction". This reframing has had mixed results - perhaps a few beneficial effects in places, but also some damaging or absurd consequences.

• BBC News: Students: Customers or learners?
• Trisha Torrey: Patient as Customer
  

Meanwhile, IT organizations want to deploy similar solutions across a range of business domains. Here are a few examples picked at random.

• Unisys: Customer Loyalty Solution
• Anari Intelligence: Passenger and Customer Profiling (hedging their bets there!)

Meanwhile, IT architects are often lumpers rather than splitters, and so they like to produce information models with a relatively small number of highly generalized objects like PARTY, which mean absolutely nothing to a real business person.

So in some enterprises, and especially in the public sector, the IT architects may be aligned with the central bureaucrats against the line-of-business. Maybe sometimes there really is a good reason for the diversity of business vocabulary, not just idiot managers being obstinate.

With a stratified Service-Oriented Architecture, it becomes possible to get the best of both worlds - building some highly generic services in one layer, which support a range of different specialized and context-specific ontologies in the layer above. So it becomes possible to accommodate a broader range of requirements without imposing a common vocabulary. Of course this raises some complexity issues, which many IT architects would prefer not to have to deal with. For more on these complexity issues, see the Asymmetric Design blog.

Animation

There really is something on the Internet for everyone. While my son was laughing his head off at the singing kittens on RatherGood.com, I was watching a simple and elegant animation explaining how SOA benefits the education and research world, courtesy of JISC/e-Framework (via Jack van Hoof).




Jack reckons the animation "perfectly shows the principles and benefits of a Canonical Data Model". Well, up to a point. It shows how SOA supports information sharing and interoperability, but an innocent viewer might interpret this as merely a question of syntax (represented by different templates in the animation) and might not appreciate the importance of semantics. Jack himself has written well on semantics in his blog, for example in his post How to mediate semantics in an EDA.

But for a 5-minute animated film - if we don't expect too much from it - excellent!

Social Networking as Reuse

Just been reading an excellent blog post by Tim Berners-Lee about the Giant Global Graph, telling a familiar story in a powerful and elegant fashion, and describing the development of the internet and semantic web in terms of increasing abstraction, interoperability and reuse.

• The Net - full name International Information Infrastructure (III). Abstracting away from the wiring to allow interoperability and reuse of computers.

• The Web - full name World Wide Web (WWW). Abstracting away from computers to allow interoperability and reuse of documents.

• The Graph - proposed name Giant Global Graph (GGG). Abstracting away from documents to allow interoperability and reuse of resources and relationships.

Berners-Lee focuses on the interoperability of relationship descriptions - friend-of-a-friend (FOAF) descriptions and the like - but of course these can be regarded merely as a special kind of document - a document that happens to be expressed in the form of a graph.

But that set me wondering about the next logical step. For some people at least, social networking is not just about finding new acquaintances (so-called "friends") but also finding new ways of connecting with existing acquaintances. I accepted a Linked-In invitation from someone I'd corresponded with on a Complexity-and-Management mailing list, and found myself deluged with messages asking me (among other things) to vote for him in some local election in the USA and to buy real estate in Florida. While this kind of reuse might be regarded as at least a breach of etiquette, if not downright spam, it is not unusual for people to try and forge business relationships with people they have met socially, and vice versa.

Why do wealthy people send their children to expensive schools? Not just because it gets them a better education, and a better chance of getting into a better university but because it buys them into the "old boy network". It is not necessary to know exactly how the child will benefit from membership in this network to be convinced of its value (and the disadvantages for those excluded, for whatever reason).

Here's a fictional example of the old boy network at work. An executive within the prawn sandwich industry talks to a journalist who is the Chief Sandwich Correspondent of the Daily Prawn, and is told about a plan by the National Prawn Authority to reduce the permitted level of iodine. When she gets back to her office she picks up the phone ... The following week, she and the Deputy Prawn Minister both happen to be present at an informal lunch, at which the conversation happens to touch upon the iodine question ...

I think it might take a while before Linked-In and Facebook can replicate this kind of affordance. This is not just a question of shared semantics (meaning) but of shared purpose (pragmatics).

However, this example illustrates the kind of instrumentality (use-purpose) implicit in social networking. We may wish to use other people's knowledge and know-how, we may wish other people to use our own knowledge and know-how, and we may wish to help our friends as well as ourselves. What we don't want is to feel we are being "used".

But just as the net has changed the way we think about computers ("the network IS the computer"), and the web has changed the way we think about (hyper-) documents and (web) services, so the graph (if that is what we must call it) is going to change the way we think about friendship and about social protocols (etiquette in the broadest sense).

If there are many people who have the required know-how, it is the graph that tells me which one to contact. Do I pick the one that is closest, or with the strongest connections? Do I pick one who is as distant as possible from my competitors? Am I looking to pay back past favours, or to put someone in my debt? Or do I pick someone who is outside the usual bunch, who would help me extend my graph into new areas?

It is already a known phenomenon that people sometimes seem more concerned with the quantity of their "friends" than with the quality of their friendships. As we get better tools for visualizing the Graph (across multiple platforms), some people may start to worry about the shape of their graph.

In the old days, people who were obsessed with the social position of their social acquaintances (and would adjust their relationships with people whose social position altered) were known as snobs or social climbers. Now there are people who wish to know how many venture capitalists are contained in their FOAF graph, in how many different countries, and people who would upgrade an acquaintance when he moves from a university post to an executive post. Plus ca change.

So that seems to be the direction we are heading. The graph is not merely a machine-readable description of a social network, it is the social network itself. And the value afforded by social networking comes from abstraction and reuse. In which case, there are some challenging implications to explore ...

Police Radio

As technological interoperability improves, so we are faced with interoperability problems at a higher level.

The latest illustration of this comes in a story about the new UK police Airwave system, which provides a common communication platform for police forces across the country. It turns out that these communications are hindered by the use of regional dialects and accents, as well as local slang. Having spent £2.9 billion on new radios, the government will now spend an extra £25,000 training policemen to talk proper.

And more succinctly. It seems the system can't support long rambling conversations. NIPA, the government agency responsible for "improving" the police, worries that "regional phrases might take much longer to say than a clipped national term". Presumably the slow drawl of a rural policeman will have to be replaced by a rapid urban patter.

Technological interoperability should be taken for granted (NOT "a thing of the past", as I originally posted) - although the price tag for the Airwave system reminds us that significant investment is still required to achieve this. But technological interoperability is never enough - it merely creates the opportunity to start work on new modes of collaboration and interoperability at the semantic level.

Police Cautioned Over Loose Talk. Sunday Telegraph, June 17th 2007.

Cross-Purposes

Seth Godin reports an interesting juxtaposition.

• Pain reliever recalled after metal found in pills.
  
• Google finds a matching image to illustrate the story.
  

Google is trying to be helpful here - repurposing an image (from elsewhere) to add value to a service.



Unfortunately, it's the right drug but the wrong brand. For many purposes, Google's semantics would be adequate. For recall purposes, however, the difference between DRUG and BRAND is very important.

This kind of semantic mismatch is what causes some of the trickiest interoperability risks.

Technorati Tags: Google interoperability SOA semantics service-oriented

Semantic Coupling

In a recent post, Semantic Coupling, the Elephant in the SOA Room, Rocky Lhotka identified semantic coupling as one of the challenges of SOA. Udi Dahan agrees that semantic coupling is harder, but adds that in his view SOA is all about addressing this issue. Meanwhile Fergal Somers, chief architect at Cape Clear, doesn't think it is so hard in practice, although he acknowledges that the relevant standards are not yet mature.

Any systems that are linked together as part of a broader workflow involves semantic-coupling as defined above, but so what? We have been building these systems for some time.

Although I wouldn't go as far as saying SOA is all about any one thing in particular (see my earlier post on Ambiguity), I also agree that semantic coupling (and semantic interoperability) are important.

Rocky's argument is based on a manufacturing analogy.

• In simple manufacturing, the economics of scale involves long production runs, so that you can spread the setup costs across a large volume.

• In agile manufacturing, the economics of scope involves minimizing the setup costs, so that you can have shorter production runs without affecting the economics of scale.

• I interpret Rocky's argument as saying that a major element of the setup costs for services involves matching the semantics.

Part of the economic argument for SOA is that it can deliver economics of scope (adaptability, repurposing) as well as economics of scale (productivity).

But there's more. If we combine SOA with some other management innovations, we may also be able to improve the economics of alignment. I don't think this is illustrated by Rocky's manufacturing analogy.

However, Kenneth LeFebvre reads more into Rocky's post than I did.

There is meaning to the interaction between a consumer and a service. What does this mean? SOA is all about making the connections between applications using “services” but it does not bridge the gap between the real world of business and the “virtual” world that runs within our software. This is precisely the problem object-oriented design was intended to solve, and was just beginning to do so, until too much of the development population abandoned it in search of the next holy grail: SOA.

At my request, Kenneth has elaborated on this statement in a subsequent post SOA OOA and Bridging the Gap. I agree with him that the rhetoric of OO was as he describes. But I still don't see much evidence that "it was just beginning to do so", and I remain unconvinced by his argument that some things are better represented by objects than by services. (More concrete examples please Kenneth.)

For a definition of the economics of scale, scope and alignment, see Philip Boxer's post Creating Economies of Alignment (October 2006).

---

Note: earlier material used the term Economics of Governance. For various reasons, we now prefer the term Economics of Alignment.

Updated 25 October 2013

Bathroom Interoperability

There is an urban myth that Americans in Europe always complain about the showers and the plumbing. My fellow Brit Phil Wainewright, currently on tour around the US West Coast, has just posted a complaint about Hotel Sink Stoppers on his SaaS blog.

Phil says this is Off-Topic. But I thought his travails with the sink stoppers were actually (there's a British word for you) actually rather relevant to SOA and SaaS.

Firstly, Phil's complaint involves an escalation of service failure. The sink stoppers fail. And as a result, the sink fails to provide the service that Phil requires - a sinkful of hot water. Which means that his preferred method of shaving fails.

What kind of failure is this? There are three possibilities.

Firstly, it could be an design failure (error of execution). The sink stopper doesn't do what it was supposed to do, period.

Secondly, it could be an architectural failure (error of planning). The sink stopper itself is fine, it just doesn't work very well in combination with this particular sink.

Thirdly, it could be a strategic failure (error of intention). The sink stopper wasn't ever intended to hold water, merely to stop your wedding ring going down the pipes.

Or maybe it isn't a failure at all - merely a clever security device to stop people leaving the taps on and flooding the room downstairs.

From the supply-side perspective (namely the hotel), Phil's inability to shave the way he likes probably ranks lower in the hotel's scheme of things than Phil's inability to cause a flood. So even if it were aware of the problem, the hotel would probably choose to degrade the quality of service experienced by Phil, rather than incur the theoretical risk of a complete service outage elsewhere.

Ultimately, this is an example of asymmetric demand. The hotel simply doesn't recognize Phil's service requirement - there is a value deficit.

Pass the SOAP.

[Update] See further comments by Vinnie M with reply from Phil W.

Technorati Tags: SaaS SOA service-oriented