Information Sharing

A fascinating statement (pdf) to the House of Lords today by MI5 chief Dame Manningham-Buller (profile), which is already being interpreted as a convoluted excuse for torture (news report).

Here is a brief summary of her argument.

	Asymmetric threat (Al Qaeda) calls for an orchestrated response ("enhanced international co-operation"). This in turn calls for information exchange to "maximize the flow of intelligence" to and from foreign agencies.
	This cooperation and exchange depends not just on mutual awareness (of specific requirements) but also motivation. Manningham-Buller is clearly unwilling to take any action that might reduce this motivation.
	Suppliers of information generally obscure the provenance of information, as a result of various local policies (such as the protection of sources) and practices (such as the use of liaison officers with tightly defined briefs as information channels).
	Users of information generally seek to discover as much context as possible, because this helps both with assessing its reliability, and with interpreting it properly.
	However the availability of context is limited by the policies and practices of the information supplier, as well as an unwillingness on the part of the user to put pressure on the supplier that might damage the supplier's motivation to co-operate in future.
	Information that is obtained from particular contexts (individuals in detention) has to be interpreted with great care. "Where the Agencies are not aware of the circumstances in which the intelligence was obtained, it is likely to be more difficult to assess its reliability." (In other words, we know that torture doesn't always produce the truth.)
	"The Agencies will often not know the location or details of detention." (In other words, foreign agencies don't tell us whether detainees have been tortured, and we are too polite to ask.)
	On at least two occasions we have obtained accurate and useful information without inquiring too closely about its provenance. (Therefore this justifies our policy of not demanding provenance.)

Manningham-Buller's statement raises some important political and ethical questions about torture and our cooperation with countries where torture is tolerated. There are also some important questions about the extent to which evidence with dubious provenance can be used in legal proceedings.

However, what I want to do here is to draw out some more general points about information sharing in a loosely coupled world. The intelligence situation described by Manningham-Buller provides an example of an extremely common pattern of conflict of interest, whereby suppliers of information are seeking to attenuate the information for various reasons, and the users of information are seeking to amplify the information in various ways. I have spoken to senior Army officers in the past about the difficulties of sharing sensitive information with other forces that are supposedly on the same side. But similar questions arise in commercial collaborations as well - how much information can be shared with partners, suppliers or customers, and how can we systematically analyse "need-to-know" questions?

Organizations are struggling to collaborate effectively in situations of imperfect trust. In an open distributed world, data provenance is a key issue. We need better tools for understanding and negotiating these complex information ecosystems. Watch this space.

Technorati Tags: information sharing SOA service-oriented trust