Regulation and Complexity

@timharford's article So Many Numbers, So Little Time yesterday prompted me to think about the causal relationship between regulation and complexity

There are at least three contrasting ideas about this relationship.

1. Red tape directly causes complexity. The solution to complexity is therefore to cut red tape. See for example the OECD document Overcoming Barriers to Administrative Simplification Strategies: Guidance for Policy Makers (pdf, 2009).

2. Conflicts of interest and mutual mistrust create complexity. The response to this complexity manifests itself in complicated forms of contractual negotiation, governance and regulation, which is the outward symptom of the intrinsic complexity of the situation. See for example my discussion of the Oil Industry example in my post Beyond Multiview.

3. Regulation triggers complex behaviour. In his article, Tim Harford suggests that it isn't always economic complexity causing regulatory complexity: in the financial industry, the causation often runs the other way. (Chester Spatt made a similar point in a keynote address earlier this year. Complexity of Regulation, HBLR June 2012.) Every attempt by the regulators to introduce greater complexity into market controls, in the hope of regulating the market more effectively, prompts an increase in the complexity of market behaviour.

Complex rules invite complex rule-bending, says Tim Harford. If at least some of the market players are more agile than the regulators, then they can always escape real regulation by shifting market operations to a higher complexity than the regulators can control. In which case it is not really the regulators who are controlling the market but the market players themselves. So what price Requisite Variety?

There is another twist to the story. Bad behaviour by large companies (Enron, WorldCom) has prompted legislation such as Sarbanes-Oxley, which turns out to be more burdensome for small companies than for larger companies. There seem to be significant economies of scale in such areas as compliance and tax avoidance: large companies can afford armies of accountants and lawyers and lobbyists and PR companies, some of the largest and most profitable companies pay practically no corporation tax. So who really benefits from all this red tape?

(In consequence, many start-up companies are designed to be acquired by a large company rather than undergo the massive administrative costs of a public flotation. Some commentators suggest that this tendency reduces the contribution of start-up companies to the macroeconomic goals of growth and full employment.)

See also Compliance and Control (May 2005), Requisite Variety and Wall Street Regulation (May 2012)

(updated October 15th 2012)

Broken Business Models 2

I heard someone on the radio the other day talking about bankruptcy. [BBC Radio 4, InBusiness, 18 December 2008] One of the features of the current economic crisis is that there seems to be a much shorter leadtime between a company's getting into difficulty, its creditors and suppliers getting edgy, and the company directors calling in the administrators.

Once upon a time I worked for a company that sometimes had threadbare cashflow, limping from month to month, the accounts department constantly trying to press customers to pay early and suppliers to accept late payment. (I should say I was not a director of this company, and didn't know much about this until later.) This company survived and prospered; I am sure many now-successful companies have had similar periods of uncertainty in their history.

Directors know they will be held personally liable if the company collapses, especially if they continue to trade when the company is non-viable. This is not just a question of being risk-averse, it is also that there has been a systemic change over the past few years. New kinds of regulation (such as Sarbanes-Oxley) combine with new technologies (such as SOA) to promote and enable maximum transparency and minimum latency. This greatly reduces the ability and willingness of company directors to wing their way through a crisis, thus making the crisis worse.

SOA is in no way responsible for the current crisis, but the presence of SOA and related technologies may have had some effect on the way that the crisis has developed and is continuing to develop.

The answer for risk-averse company directors is not to turn their backs on transparency and rapid-response, but to embrace these things, not just for the sake of their companies but to preserve their own wealth. "SOA or debtor's prison" - now that makes a compelling argument doesn't it?

---

This is one of a series of posts on Broken Business Models

• 16 December 2008
• 19 December 2008
• 1 February 2009
• 12 February 2009 

See also Two Kinds of Business Model (December 2008)

Data Provenance

In my previous post on Information Sharing, I discussed some of the problems of information sharing in a loosely-coupled world, with special reference to the security services. There is further discussion of the social and political aspects of this at IntoTheMachine and Demanding Change. In this blog, I am continuing to focus on the information management aspects.

On Friday, I had a briefing about an EU research project on data provenance, led by IBM. This project is currently focused on the creation and storage of provenance data (in other words data that describe the provenance of other data - what we used to call an audit trail). The initial problem they are addressing is the fact that provenance data (audit trails and logs) are all over the place - incomplete and application-specific - and this makes it extremely hard to trace provenance across complex enterprise systems. The proposed solution is the integration of provenance data from heterogeneous sources to create one or more provenance stores. These provenance stores may then be made available for interrogation and analysis in a federated network or grid, subject to important questions of trust and security.

In art history, provenance means being able to trace a continuous history of a painting back to the original artist - for example proving the version of the Mona Lisa currently in the Louvre is the authentic work of Leonardo da Vinci. As it happens, we don't have a completely watertight provenance for the Mona Lisa, as it was stolen by an Italian artist in 1911, and remained absent from the Louvre until 1913. Most art lovers assume that the painting that was returned to the Louvre is genuine, but there is a gap in the audit trail in which an excellent forgery might possibly have been committed. [See The Day The Mona Lisa was Stolen. I learned about this story from Darien Leader's book Stealing the Mona Lisa.] Provenance may also involve an audit trail of other events in the painting's history, such as details of any restoration or repair.

In information systems, provenance can be understood as a form of instrumentation of the business process - instrumentation and context that allows the source and reliability of information to be validated and verified. (Quality wonks will know that there is a subtle distinction between validation and verification: both are potentially important for data provenance; and I may come back to this point at a later date.) Context data are used for many purposes besides provenance, and so provenance may involve a repurposing of instrumentation (data collection) that is already carried out for some other purpose, such as business activity monitoring (BAM). Interrogation of provenance is at the level of the business process, and IBM is talking about possible future standards for provenance-aware business processes.

Provenance-awareness is an important enabler for compliance to various regulations and practices, including Basel, Sarbanes-Oxley, and HIPPA. If a person or organization is to be held accountable for something, then this typically includes being accountable for the source and reliability of relevant information. Thus provenance must be seen as an aspect of governance.

Provenance is also an important issue in complex B2B environments, where organizations are collaborating under imperfect trust. From a service-oriented point of view, I think what is most interesting about data provenance is not the collection and storage of provenance data, but the interrogation and use. This means we don't just want provenance-aware business processes (supported by provenance-aware application systems) but also provenance-aware objects and services. Some objects (especially documents, but possibly also physical objects with suitable RFID encoding) may contain and deliver their own provenance data, in some untamperable form. Web services may carry some security coding that allows the consumer to trust the declared provenance of the service and its data content. There are some important questions about composition and decomposition - how do we reason architecturally about the relationship between provenance at the process/application level and provenance at the service/object level?

We also need an analysis and design methodology for provenance. How do you determine how much provenance data will be adequate to satisfy a given compliance requirement in a given context (surely standards cannot be expected to provide a complete answer to this) and how do you compose a sufficiently provenance-aware business process either from provenance-aware services, or from normal services plus some additional provenancing services. Conversely, in the design of services to be consumed outside the enterprise, there are important analysis and design questions about the amount of provenance data you are prepared to expose to your service consumers. The EU project includes methodology as one of its deliverables, due sometime next year. In the meantime, IBM hopes that people will start to implement the provenance architecture, as described on the project website, and provide practical input.


Updated 30 April 2013

Compliance and Control

The US Sarbanes-Oxley Act of 2002 (SOX) mandates what is effectively a systems engineering solution to the problem of bezzle. Reliability of a company's published accounts is achieved not by human oversight alone, but by a set of information and control systems that ensures information quality and management accountability. Executive officers are required to sign the accounts and are criminally liable for any inaccuracy. The act also mandates near-real-time disclosure of any material events.

One of the basic tenets of control theory is that a control system must have as much flexibility and variety as the system it is trying to control. This is known as Requisite Variety. Previous attempts at internal audit and control have either themselves lacked flexibility and responsiveness, thus compromising their ability to deliver effective control, or have imposed inflexibility and unresponsiveness on the underlying system. Neither of these outcomes is acceptable.

For the management controls to be effective there must be some way for the management system to intervene in the transaction system. There is an architectural choice between direct and indirect intervention.

Direct	Indirect
When the management subsystem detects an anomaly, it immediately intervenes in the transaction subsystem - possibly changing the status of some transaction record, or moving a transaction to a different account within the general ledger. Changes to the transaction system may also involve adjustments to elements of policy or context governing the transaction subsystem.	When the management subsystem detects an anomaly, it triggers a piece of workflow, or generates some additional transactions, which are fed back into the transaction system. A human supervisor is notified, who may have the authority to make changes in the transaction system and/or undertake further investigation.
This approach tightly couples the management subsystem and the transaction subsystem into a single unified system. The management subsystem is intrusive into the transaction subsystem.	This approach maintains loose coupling between the management subsystem and the transaction subsystem. The software component of the management subsystem provides non-intrusive monitoring. This can be implemented as an instance of the Observer pattern.

The SOA principle of loose coupling clearly favors the indirect, non-intrusive approach. However, we must be careful to ensure that the indirect control remains effective. There needs to be some coordination between the collaboration or process management layer in the management subsystem, and the collaboration or process management layer in the transaction subsystem.
In a dynamic SOA environment, we actually need a double instance of the Observer pattern. The management system needs to monitor dynamic changes to the transaction workflow, as well as the transactions themselves.
The management subsystem must be as adaptable as the transaction subsystem - and this calls for the same SOA design principles to be applied to both.

Information Management / Business Intelligence	Information architects may wish to think of SOX requirements in terms of data integrity - providing a guarantee of consistency and completeness across multiple diverse applications and data stores. For information management, Sarbanes-Oxley provides a push towards real-time closed-loop information management and business intelligence. Many BI products are now Web Service enabled, and this makes it easier to quickly plug BI capabilities such as monitoring and data mining into the business process.
Model-Driven Compliance	It is impossible for an organization of any size or complexity to achieve SOX compliance without some serious modeling effort - involving both data and process, and showing how data are transformed across multiple and diverse applications and data stores.
Web Service Based EAI	SOX compliance typically requires a major rewiring job to corporate financial information systems. In some cases, some additional applications and packages will need to be quickly plugged in to complete the solution. New or reengineered interfaces should be rendered as Web Services by default. Organizations that are already using a Web Service or EAI platform will be able to leverage this to achieve SOX compliance more quickly.

---

See also


CBDI Report April 2004: Sarbanes-Oxley Drives Web Services Adoption
Notions: Bezzle