Testing Multiples

From engineering to medicine, professionals are often forced to rely on tests that are not always completely accurate. In this post, I shall look at the tests that millions of people were obliged to use during the pandemic, to check whether they had COVID-19. The two most common tests were Lateral Flow and PCR. Lateral Flow was quicker and more convenient, while PCR took longer (because the sample had to be sent to a lab) and was supposedly more accurate.

There was also a difference in the data collected from these tests. Whereas all the results from the PCR tests should have been available in the labs, the results from the lateral flow tests were only reported under certain circumstances. There was no obligation to report a negative test unless you needed access to something, and people sometimes chose not to report positive tests because of the restrictions that might follow. And of course people only took the tests when they had to, or wanted to. When people had to pay for the tests, this obviously made a big difference.

To compensate for these limitations, some random screening was carried out, which was designed to produce more reliable and representative datasets. However, these datasets were much smaller.

 

So what can we do with this kind of data? Firstly, it tells us something about the disease - whether it is distributed evenly across the country or concentrated in certain places, how quickly it is spreading. If we can combine the test results with other information about the test subjects, we may be able to get some demographic information - for example, how is the disease affecting people of different age, gender or race, how is it affecting different job categories. And if we have information from the health service, we can estimate how many of those testing positive end up in hospital.

This kind of information allows us to make predictions - for example, future demand for hospital beds, possible shortages of key workers. It also allows us to assess the effects of various protective measures - for example, to what extent does mask-wearing, social distancing and working from home reduce the rate of transmission.

Besides telling us about the disease, the data should also be able to tell us something about the tests. And the accuracy of the predictions provides a feedback loop, which may enable us to reassess either the test data or the predictive models.

 

In her book The Body Multiple, Annemarie Mol discusses the differences between two alternative tests for atherosclerosis, and describes how clinicians deal with cases where the two tests appear to provide conflicting results, as well as cases where there may be other reasons to question the test results. Instead of having a single view of the disease, she talks about its multiplicity or manyfoldedness.

But questioning the test results in a particular case, or highlighting particular issues with a given test, does not mean denying the overall value of the test. Most of the time we can continue to regard a test as useful, even as we are considering ways of improving it.

If and when we introduce a new or improved test, we may then wish to translate data between tests. In other words, if test A produced result X, then we would have expected test B to produce result Y. While this kind of translation may be useful for statistical purposes, we need to be careful about its use in individual cases.

For many people, the second discourse appears to undermine the first discourse. If we can't always trust the data, can we ever trust the data? During the COVID pandemic, many rival interpretations of the data emerged; some people chose interpretations that confirmed their preconceptions, while others turned away from any kind of data-driven reasoning.

 

The COVID pandemic became a politically contentious field, so what if we look at other kinds of testing? In safety engineering, components and whole products are subjected to a range of tests, which assess the risk of certain kinds of failure. Obviously there are manufacturers and service providers with a commercial interest in how (and by whom) these tests are carried out, and there may be regulators and researchers looking at how these tests can be improved, or to detect various forms of cheating, but ordinary consumers don't generally spend hours on YouTube complaining about their accuracy and validity.

Meanwhile even basic corporate reporting may be subject to this kind of multiplicity, as illustrated in my recent post on Data Estimation (July 2022).

So there is a level of complexity here, which not all data users may feel comfortable with, but which data professionals may not feel comfortable about hiding. In a traditional report, these details are often pushed into footnotes, and in an online dashboard there may be symbols inviting the user to drill down for further detail. But is that good enough?

---

Annemarie Mol, The Body Multiple: Ontology in Medical Practice (Duke University Press 2002)

Wikipedia: COVID-19 testing

Related posts: Data-Driven Reasoning - COVID (April 2022), Data Estimation (July 2022)

Is there a Single Version of Truth about Stents?

Clinical trials are supposed to generate reliable data to support healthcare decisions and policies at several levels. Regulators use the data to control the marketing and use of medicines and healthcare products. Clinical practice guidelines are produced by healthcare organizations (from the WHO downwards) as well as professional bodies. Clinicians apply and interpret these guidelines for individual patients, as well as prescribing medicines, products and procedures, both on-label and off-label.

Given the importance of these decisions and policies for patients, there are some critical issues concerning the quality of clinical trial data, and the ability of clinicians, researchers, regulators and others to make sense of these data. Obviously there are significant commercial interests involved, and some players may be motivated to be selective about the publication of trial data. Hence the AllTrials campaign for clinical trial transparency.

But there is a more subtle issue, to do with the way the data are collected, coded and reported. The BBC has recently uncovered an example that is both fascinating and troubling. It concerns a clinical trial comparing the use of stents with heart bypass surgery. The trial was carried out in 2016, funded by a major manufacturer of stents, and published in a prestigious medical journal. According to the article, the two alternatives were equally effective in protecting against future heart attacks.

But this is where the controversy begins. Researchers disagree about the best way of measuring heart attacks, and the authors of the article used a particular definition. Other researchers prefer the so-called Universal Definition, or more precisely the Fourth Universal Definition (there having been three previous attempts). Some experts believe that if you use the Universal Definition instead of the definition used in the article, the results are much more one-sided: stents may be the right solution for many patients, but are not always as good as surgery.

Different professional bodies interpret matters differently. The European Association for Cardio-thoracic Surgery (EACTS) told the BBC that this raised serious concerns about the current guidelines based on the 2016 trial, while the European Society of Cardiology stands by these guidelines. The BBC also notes the potential conflicts of interests of researchers, many of whom had declared financial relationships with stent manufacturers.

I want to draw a more general lesson from this story, which is about the much-vaunted Single Version of Truth (SVOT). By limiting the clinical trial data to a single definition of heart attack, some of the richness and complexity of the data are lost or obscured. For some purposes at least, it would seem appropriate to make multiple versions of the truth available, so that they can be properly analysed and interpreted. SVOT not always a good thing, then.

See my previous blogposts on Single Source of Truth.

---

Deborah Cohen and Ed Brown, Surgeons withdraw support for heart disease advice (BBC Newsnight, 9 December 2019) See also https://www.youtube.com/watch?v=_vGfJKMbpp8

Debabrata Mukherjee, Fourth Universal Definition of Myocardial Infarction (American College of Cardiology, 25 Aug 2018)

See also Off-Label (March 2005), Is there a Single Version of Truth about Statins? (April 2019), Ethics of Transparency and Concealment (October 2019)

Is there a Single Version of Truth about Statins?

@bengoldacre provides some useful commentary on a BBC news item about statins. In particular, he notes a detail from the original research paper that didn't make it into the BBC news item - namely the remarkable lack of agreement between GPs and hospitals as to whether a given patient had experienced a cardiovascular event.

This is not a new observation: it was analysed in a 2013 paper by Emily Herrett and others. Dr Goldacre advised a previous Health Minister that "different data sources within the NHS were wildly discrepant wrt to the question of something as simple as whether a patient had had a heart attack". The minister asked which source was right - in other words, asking for a single source of truth. But the point is that there isn't one.

Data quality issues can be traced to a number of causes. While some of the issues may be caused by administrative or technical errors and omissions, others are caused by the way the data are recorded in the first place. This is why the comparison of health data between different countries is often misleading - because despite international efforts to standardize classification, different healthcare regimes still code things differently. And despite the huge amounts of NHS money thrown at IT projects to standardize medical records (as documented by @tonyrcollins), the fact remains that primary and secondary healthcare view the patient completely differently.

See my previous blogposts on Single Source of Truth

---

Tony Collins, Another NPfIT IT scandal in the making? (Campaign4Change, 9 February 2016)

Emily Herrett et al, Completeness and diagnostic validity of recording acute myocardial infarction events in primary care, hospital care, disease registry, and national mortality records: cohort study (BMJ 21 May 2013)

Michelle Roberts, Statins 'don't work well for one in two people' (BBC News, 15 April 2019)

Benoît Salanave et al, Classification differences and maternal mortality: a European study (International Journal of Epidemiology 28, 1999) pp 64–69

The Hungry Tapeworm

This week, three American companies announced a joint venture to sort out healthcare for their own employees. Ambitious, huh?

This is not the first time large American companies have tried to challenge tho market power of healthcare providers. According to Warren Buffett, "the ballooning costs of healthcare act as a hungry tapeworm on the American economy". Intel and Walmart are among those that have previously ventured into this area. In 2016, 20 companies including Coca Cola, American Express, IBM and Macy’s joined the Health Transformation Alliance (HTA). So why should anyone take this latest attempt seriously? Only because the three companies are Amazon, Berkshire Hathaway and JPMorgan Chase. And Amazon (need I remind you?) eats everyone's lunch.

• 2010: Amazon Is Still Eating eBay's Lunch

• 2012: How Amazon ate Sears' lunch

• 2017: Hey Walmart, Amazon is eating your lunch

• 2017: Amazon Eats Google’s Lunch

• 2017: Why Amazon is eating the world

John Naughton sees this as a typical play for a data hungry tech giant, based on two hypotheses.

• Transactional data will lead to transactional efficiencies. The joint venture starts with the three companies experimenting on their own employees, who will "tell Amazon and its algorithms what works and doesn’t work". 

• "Mastery of big data might yield clinical benefit".

As Pressman and Lashinsky note, the experiment is based on a pretty good sample of Americans: "a diverse workforce spanning low-wage normal folk to the most elite of our society".


Amazon is obviously a major player in the data and analytics world, but so is IBM, which is playing an important role in the HTA. Not only is IBM a corporate member, but IBM Watson Health will do the data and analytics. According to Pharmaceutical Commerce, it will "aggregate participating HTA member companies' data, enabling insights both into outcomes of medical interventions, as well as wellness initiatives to improve employees’ health".

And what about Google? Google Health was discontinued in 2011, following a lack of widespread adoption. Perhaps data isn't the whole story.


But Amazon is not just about data. In an article published before this announcement, Zack Kanter attributes Amazon's strategic dominance to SOA. "Each piece of Amazon is being built with a service-oriented architecture, and Amazon is using that architecture to successively turn every single piece of the company into a separate platform — and thus opening each piece to outside competition."

Moazed and Johnson discuss the platform implications of the healthcare announcement. They argue that "platforms thrive with fragmentation, not consolidation", and that "the new platform needs to offer enough potential scale to outweigh those risks, otherwise manufacturers may be too afraid to join". Sarah Buhr sees this as an opportunity for smaller players, such as Collective Health.

Three employers, even large ones, probably won’t have enough muscle to negotiate fair prices for healthcare and pharma. But if Bezos can create the right expectations, and provide a flexible platform for smaller players ...

---

Health Transformation Alliance sets its 2017 agenda (Pharmaceutical Commerce, 9 March 2017)

Amazon alliance takes on ‘hungry tapeworm’ of healthcare costs (Pharmaceutical Technology, 1 February 2018)

Sarah Buhr, Collective Health Wants To Replace The Health Insurance Industry With A Software Program (TechCrunch, 11 Aug 2014)

Sarah Buhr, Amazon’s new healthcare company could give smaller healthtech players a boost (TechCrunch, 30 Jan 2018)

Paul Demko, Amazon's new health care business could shake up industry after others have failed (Politico, 30 January 2018)

Zack Kanter, Why Amazon is eating the world (TechCrunch, 14 May 2017)

Paul Martyn, Healthcare Consumerism: Taming The Hungry Tapeworm (Forbes, 30 January 2018)

Alex Moazed and Nicholas L Johnson, Amazon's Long-Awaited Health Care Platform (Inc, 30 January 2018)

John Naughton, Healthcare is a huge industry – no wonder Amazon is muscling in (Observer, 4 February 2018)

Aaron Pressman and Adam Lashinsky, Data Sheet—Why Jeff Bezos Just Might Crack the Health Care Challenge (Fortune, 31 January 2018)

Jordan Weissmann, Can Amazon, Berkshire Hathaway, and JPMorgan Revolutionize Health Care? (Slate, 30 Jan 2018)

Wikipedia: Google Health


Updated 5 February 2018

Is Organizational Integration a Good Thing?

Some members of the UK Government are keen on integrating health and social services. In his first speech as Minister of State for Care Services, @NormanLamb said

"The consensus behind integrated care is pretty universal. In government, in think tanks, in patient groups everyone sees it as A Good Thing." (Transcript of speech at @TheKingsFund, 11 September 2012)

And Junior Health Minister Dan Poulter is just as passionate. Integration of NHS and social care "is like the holy grail", he told the Guardian recently (30 Oct 2012).


But not everyone agrees. Jane Young is a disability consultant and campaigner. She asks Would the integration of health and social care promote independent living? (Guardian 8 Nov 2012), and argues that it would not.

"Rather than medical treatment, disabled people need assistance to perform such varied everyday tasks as driving, bathing, dressing, typing, cooking, parenting activities etc. None of these functions is normally carried out by medically trained professionals, so on this basis it is illogical for the Department of Health to be wedded to the integration of health and social care services."

Meanwhile Jeremy Hunt, the Secretary of State for Health, sounds an ambivalent note.

"But structures are only a means to an end.  What really matters is better health and care outcomes." (25 October 2012)

 What are the problems that integration might tackle. There are many symptoms of poorly joined-up services. Jonathon Tomlinson documents some from his practice as an East London GP.

• Adverse social factors, such as poverty and social exclusion, have a critical impact on the efficiency and productivity of healthcare.

• It is impossible to discuss diabetic control or smoking cessation with someone whose housing depends on her benefits which have just been cut.  

• Patients cannot follow routine healthcare advice when their lives are disorganized as a result of financial stress, or when they cannot afford to pay for prescriptions.

• Hospitals, clinics and surgeries are full of people who don’t know where else to go. Hospitals beds are blocked by patients who lack sufficient social support for them to be cared for elsewhere.  Hospital staff report readmitting the same patients week after week because they cannot cope at home.

Based on: A perfect storm: welfare meets healthcare (June 2012)
(slightly reworded)

I agree with Jeremy Hunt that outcomes matter more than structures. Obviously this covers the individual needs of patients and their carers, but also includes broader economic and social outcomes, such as higher quality and value-for-money, to be achieved through innovation and leadership.

Hunt describes integration in terms of "a culture of cooperation", "meaningful contact" (e.g. between GPs, consultants, local authorities and social care providers) and "bringing people together". But how are these things to be achieved? By better processes? By heroic leadership? Hunt merely appeals to new structural mechanisms - specifically the Health and Wellbeing Boards, and Healthwatch - which will somehow bring about a sufficient level of "meaningful contact".

I presume that Jane Young has no objection to some level of "meaningful contact". Her main objection to "integration" seems to be that she doesn't want to see the Department of Health managing services that do not require medical training, thereby implying that organizational boundaries should be primarily aligned to skills rather than outcomes.

But it seems to me that "meaningful contact" alone cannot bridge the structural barriers to joined-up care. If patients are getting the wrong (expensive and inconvenient) care package because there isn't funding for the right care package, this needs to be addressed during the budgeting and commissioning phase, not by better coordination in the delivery phase. Surely we need to start by understanding what overall capabilities and processes are required for effective management and delivery and governance of care, before we start allocating responsibility for these capabilities and processes to various agencies.

---

Let me just take a step back for a moment. Something called "integration" is being put forward as a structural solution to some set of problems. But there is a great deal of confusion about what "integration" actually means, what this "integration" might achieve, and whether there are any unpleasant side-effects. Some people may think that "integration" between A and B merely means establishing effective channels of communication between A and B, while others may think "integration" means shared planning and commissioning, integrated governance, or even full merger.

In my opinion, structural solutions to complex problems is (or should be) the job of the business architect, and I believe that business architecture can play a vital role in clarifying the requirements for "integration" and working out the practical details. So we need to apply some of the business architecture viewpoints to thinking about the integration of care.

We might imagine that the ultimate in integration would be to put all healthcare and social care into a single monolithic organization, but there needs to be some differentiation of structure even within an apparently monolithic organization, so that would merely reframe the problem rather than solving it. There is still a challenging architectural question - what structuring and organizational design principles to use for carving up responsibilities and negotiating exchanges between different units.

There is nothing logically wrong with the idea that responsibility goes with expertise, as Jane Young favours, except for the fact that it doesn't deal with the observed symptoms. Evidence-based healthcare is taken very seriously, and there would be strong objection to applying some quack nostrum without proper study, but the evidence base for organizational change in the NHS seems to be very much weaker.

See my earlier posts Resistance to Architecture and Illusion of Architecture.

---

(Update) Of course integration is not just a concern for the public sector. Compare the latest changes in leadership at Microsoft "aimed at ensuring the firm continues to be a dominant player in the sector". Microsoft CEO Steve Ballmer said "The products and services we have delivered to the market in the past few months mark the launch of a new era at Microsoft. To continue this success it is imperative that we continue to drive alignment across all Microsoft teams, and have more integrated and rapid development cycles for our offerings." (BBC News 13 November 2012) See my post Functional Organization at Microsoft (Nov 2012)

---

This is one of a series of posts on The Purpose of Business Architecture.

By the way, places are still available on my Business Architecture Workshops (January 29th-31st)

The Meaning of Iteration

There is a curious ambiguity about the word "iteration". Sometimes it means doing exactly the same thing, over and over again; sometimes it means doing something slightly different each time.

When a process is drawn as a loop, we need to understand what exactly this means. Which aspects of the process are the same for each execution of the process, and which aspects are different? What is the nature of the feedback, allowing each iteration to improve on previous iterations, and what is the scope for learning and development?

For example, a business process architecture may include a product development cycle. We usually understand this to mean not that the products themselves are recycled, but that there is some accumulation of knowledge and experience (trial and error, learning by doing) that allows each product and each product-related activity to learn from and improve upon previous iterations.

Therefore a process model that is drawn as a loop or cycle cannot be interpreted in the same way as a process model that is drawn as a simple production line or value chain, because there is something else important going on. This is one of the reasons I distinguish the Cybernetic View (which specifically looks at feedback and its effects) from the Activity View (which looks at the work itself). The Cybernetic View allows the architect to pay attention to the effectiveness and efficiency of the feedback, which is not the same thing as the effectiveness and efficiency of the underlying activities but is at a different logical level.

One process that is attracting a lot of interest at the moment is the commissioning process. Commissioning is an important component of a healthcare system, and I also heard Jennifer Saunders recently talking about commissioning at the BBC. I wanted to see how people were depicting the commissioning process, so I did an internet search for images associated with the term "commissioning". The top six diagrams (ignoring one diagram that wasn't really about commissioning but about paying commission) all showed commissioning as some kind of cycle.





This selection of diagrams reflects a common agreement that commissioning can be thought of as a cycle. However, although the wide variety of diagram styles might be regarded as merely a fashion statement, it could be a sign of a more fundamental methodological issue. It is not clear that all these sources share the same understanding of what it means for commissioning to be regarded as a cycle. What do these diagrams tell us about how the commissioning process develops and evolves and hopefully improves over time? Are there different categories of improvement, with different cycle times?


There is also the question of responsibility. Are the same people responsible for both executing and improving the commissioning process, or does improvement call for a different kind of distributed intelligence?

So there are lots of important and interesting questions here, which a traditional process model doesn't help much with.

The Illusion of Architecture

#bizarch #entarch @hspplease Following my post on the Resistance to Architecture within the National Health Service, I came across this cynical remark by Tony Riley of the Health Solutions Partnership.

"We really don't know why the term 'healthcare commissioning architecture' has become so closely associated with this reform. Architecture would give us the belief that detailed blueprints were being followed as is in the design of a building, a methodology that has scientific merit. When it comes to the Secretary of State’s healthcare reform there has been no blueprint detailed enough to prevent the chaos having been played out over the last year across the NHS commissioning and provider side with more to follow we suspect. So perhaps the illusion of architecture has been used to give us a false sense of reassurance." Tony Riley, Healthcare Commissioning Architecture. The UK Health Bill translated and implemented (Health Solutions Partnership, 2011)

Architecture is known for creating illusions (also known as visions), but (as Baudrillard pointed out) the ability to create illusions can also generate self-delusion.

Another well-known paradox of architecture is the conflict between architecting for the real world (complete with conflict and compromise) and architecting for a fantasy "theme park" world in which there is no place for conflict and compromise. In a 1997 book on Disney, this is called the Architecture of Reassurance. (This was also the title of a short film released in 2000.)

An implication of Riley's remark is that senior people within the NHS are paying lip-service to architecture as a substitute for actually doing any proper architecture. If this is true, it would explain the apparent reluctance to expose organizational designs for public and professional scrutiny. But I for one don't find this very reassuring.

Jean Baudrillard, Truth or Radicality: The Future of Architecture (1999)

Karal Ann Marling (ed), Designing Disney's Theme Parks: The Architecture of Reassurance (Abbeville Press, 1997). Review by Frankie Roberto (undated) See also Carnegie Magazine 1999

Mimi Yiu, Virtually Transparent Structures (2003).

Resistance to Architecture

A few weeks ago, I was at a meeting to discuss the UK National Health Service reforms. One of the speakers, a senior NHS administrator, used the word "architecture" in her presentation. Twice. Clearly referring to business/organizational architecture rather than physical architecture.

Encouraged by this, a couple of us approached her afterwards and asked her what kind of architectural work was going on, but we were treated with diplomatic hostility. (She didn't want to talk to us, and clearly wanted to escape as quickly as possible.) As far as we could make out, what she had meant by architecture was that she had some complicated organizational design in her head, but she definitely wasn't going to take the political risk of making this design clear to anyone else.

There may well be some proper business architecture work going on around the NHS reforms, but we have been looking for a while and haven't found much yet. (Which is a bit worrying, given the scale of the structural change that is underway.) If you know of anything I'd be delighted to hear from you.

There have perhaps always been individuals within large organizations who see some personal political advantage in maintaining obscure and complicated organizations that only they can understand and manipulate, and these individuals are probably always going to see good business architecture as a threat. But is there any reason why an organization as a whole should resist business architecture? Are there some organizations where that kind of small-p-political approach to management is so deeply ingrained in the culture that good business architecture is incompatible?

In his book, The Systems Approach and Its Enemies, C West Churchman identified four enemies of the systems approach: Politics, Ethics and Morality (the dominance of the "Big Idea"), Religion (irrational attachment to traditional forms), and Aesthetics (intuition, irrational optimism). Although as Churchman himself acknowledges, the notion of "enemy" is itself problematic from a systems perspective.

If politics may be one source of resistance to business architecture within the NHS, another source may be the "Big Idea", also known as "Principles". One of the reasons I am less enthusiastic about principles than many of my fellow architects is that I often see principles being used not as a starting point for hard work but as a substitute for it. In the public sector, we often see broad principles such as Choice or Competition being bandied about, with no serious attempt to work out how these so-called principles might work in practice. And if the structure and behaviour of the NHS is completely determined by these abstract principles, as some would have us believe, then there is obviously no point wasting time getting business architects involved. See my post On the misuse of general principles (Jan 2012).

A third reason for steering clear of business architecture might be because everyone believes that the fundamental structural problems are so deeply embedded in existing institutions that there is no point hiring business architects who will merely tell us what we already know. So we go on tinkering with the details, shifting responsibility for commissioning from one bunch of bureaucrats to another bunch of bureaucrats, but not making any real inroads into the institutional separations between primary and secondary care, or between healthcare and social care.

The final reason for thinking that business architecture is unnecessary is because of the Faustian pact with senior staff. The woman I heard talking recently presents an attractive and optimistic vision of transformation, plausible to nearly everyone except a few politically motivated snipers. And of course business architects. No wonder they regard us as the enemy.

See also The Illusion of Architecture (September 2012)

Intelligent Business Operations

@opheretzion reposts a healthcare example from @jimsinur concerning resource allocation in surgeries.

The loop is as follows.

Decision / Planning	Scheduling and resource allocation for the following day, using simulation and optimization tools.
Information Gathering	Real-time tracking of selected "things" (physicians, nurses, equipment; monitor of procedure duration and status) using a range of devices (sensors and cameras).
Sensemaking	Detecting deviations from plan - "things going wrong"
Decision / Planning	Revising the plan in "near real time"

Obviously this is an impressive use of the relevant technologies, and it may well deliver substantial benefits in terms of supply-side cost-effectiveness as well as a safer and better experience for the patient. This is essentially a goal-directed feedback loop.

However, we may note the following limitations of this loop.

1. Decision / Planning is apparently based on a fixed pre-existing normative model of operations - in other words a standard "solution" that should fit most patients' needs. This may be a reasonable assumption for some forms of routine surgery, but doesn't seem to allow for the always-present possibility of surprise when you cut the patient open.

2. Information Gathering is based on a fixed set of things-to-be-monitored. Opher calls this "real-time tracking of everything" - but of course this is a huge exaggeration. Perhaps the most important piece of information that cannot be included in this rapid feedback loop is the patient outcome. We might think that this cannot be determined conclusively until much later, but there may be some predictive metrics (perhaps the size of the incision?) that may be strongly correlated with patient outcomes.

3. Sensemaking is extremely limited - there is no time to understand what is going wrong, or to carry out deeper root-cause analysis and learning. All we can do is to react according to previously established "best practice".

4. Replanning is limited to detecting and quick-fixing deviations from the plan. See my post on Real-Time Events.

Full organizational intelligence needs to integrate this kind of rapid goal-directed feedback loop with a series of deeper analytic sensemaking and learning loops. For example, we might want to monitor how a given surgical procedure fits into a broader care pathway for the patient. Real-time monitoring is then useful not only for near-real-time operational intelligence but also for longer-term innovation.

---

Jim Sinur Success Snippet (January 2012)

Opher Etzion Medical Use Case (January 2012)

And please see my draft Organizational Intelligence Primer, now available on LeanPub.

On the misuse of general principles

#entarch There is a common fallacy among enterprise architects that radical structural and behavioural change can and should be driven by a few simple and powerful ideas. Alas, the public sector is strewn with the disastrous consequences of this fallacy.

We can find countless examples from the National Health Service (NHS) in the UK. For Steve Harrison, Honorary Professor of Social Policy at the University of Manchester, the idea that NHS reorganisations can be triggered by a few general ideas is one of the Seven Fallacies of English Health Policy. He points out that high levels of abstraction (beloved by academics and architects alike) do not allow proper assessment of the plausibility of claims about benefits of reorganisation and how the system will work. (HT @mellojonny)

Where do health reorganization principles come from? I asked a popular search engine, and was led to a paper called Basic Principles of Information Technology Organization in Health Care Institutions (JAMIA 1997); (I suppose from the high search ranking of this paper that it is a widely used source for such principles.) The paper concludes that all organizations MUST have certain characteristics, based on a single case study where these characteristics seemed to be beneficial; in other words, arguing from the particular to the general. (I'm sure there must be some more rigorous studies, but they don't seem to get as good search rankings for some reason.)

But many of the principles that govern sweeping architectural reforms of the public sector aren't even derived by thinly based generalization from such observed vignettes, but are derived from purely abstract concepts such as "choice" and "competition" and "justice", to which each may attach his or her own politically motivated interpretation.

This leads to several levels of failure - not only failure of execution and planning (because the generalized principles are not sufficiently refined to provide realistic and coherent solutions to complex practical problems) but also failure of intention (because a vague but upbeat set of principles helps to conceal the fact that the underlying vision remains woolly).

What's Wrong with the Single Version of Truth

As @tonyrcollins reports, a confidential report currently in preparation on the NHS Summary Care Records (SCR) database will reveal serious flaws in a massively expensive database (Computer Weekly, March 2010). Well knock me down with a superbug, whoever would have guessed this might happen?

"The final report may conclude that the success of SCRs will depend on whether the NHS, Connecting for Health and the Department of Health can bridge the deep cultural and institutional divides that have so far characterised the NPfIT. It may also ask whether the government founded the SCR on an unrealistic assumption: that the centralised database could ever be a single source of truth."

There are several reasons to be ambivalent about the twin principles Single Version of Truth (SVOT) and Single Source of Truth (SSOT), and this kind of massive failure must worry even the most fervent advocates of these principles.

Don't get me wrong, I have served my time in countless projects trying to reduce the proliferation and fragmentation of data and information in large organizations, and I am well aware of the technical costs and business risks associated with data duplication. However, I have some serious concerns about the dogmatic way these principles are often interpreted and implemented, especially when this dogmatism results (as seems to be the case here) in a costly and embarrassing failure.

The first problem is that Single-Truth only works if you have absolute confidence in the quality of the data. In the SCR example, there is evidence that doctors simply don't trust the new system - and with good reason. There are errors and omissions in the summary records, and doctors prefer to double-check details of medications and allergies, rather than take the risk of relying on a single source.

The technical answer to this data quality problem is to implement rigorous data validation and cleansing routines, to make sure that the records are complete and accurate. But this would create more work for the GP practices uploading the data. Officials at the Department of Health fear that setting the standards of data quality too high would kill the scheme altogether. (And even the most rigorous quality standards would only reduce the number of errors, could never eliminate them altogether.)

There is a fundamental conflict of interest here between the providers of data and the consumers - even though these may be the same people - and between quality and quantity. If you measure the success of the scheme in terms of the number of records uploaded, then you are obviously going to get quantity at the expense of quality.

So the pusillanimous way out is to build a database with imperfect data, and defer the quality problem until later. That's what people have always done, and will continue to do, and the poor quality data will never ever get fixed.

The second problem is that even if perfectly complete and accurate data are possible, the validation and data cleansing step generally introduces some latency into the process, especially if you are operating a post-before-processing system (particularly relevant to environments such as military and healthcare where, for some strange reason, matters of life-and-death seem to take precedence over getting the paperwork right). So there is a design trade-off between two dimensions of quality - timeliness and accuracy. See my post on Joined-Up Healthcare.

The third problem is complexity. Data cleansing generally works by comparing each record with a fixed schema, which defines the expected structure and rules (metadata) to which each record must conform, so that any information that doesn't fit into this fixed schema will be barred or adjusted. Thus the richness of information will be attenuated, and useful and meaningful information may be filtered out. (See Jon Udell's piece on Object Data and the Procrustean Bed from March 2000. See also my presentation on SOA for Data Management.)

The final problem is that a single source of information represents a single source of failure. If something is really important, it is better to have two independent sources of information or intelligence, as I pointed out in my piece on Information Algebra. This follows Bateson's slogan that "two descriptions are better than one". Doctors using the SCR database appear to understand this aspect of real-world information better than the database designers.

It may be a very good idea to build an information service that provides improved access to patient information, for those who need this information. But if this information service is designed and implemented according to some simplistic dogma, then it isn't going to work properly.

---

Update. The Health Secretary has announced that NHS regulation will be based on a single version of the truth.

"in the future the chief inspector will ensure that there is a single version of the truth about how their hospitals are performing, not just on finance and targets, but on a single assessment that fully reflects what matters to patients"

Roger Taylor, Jeremy Hunt's dangerous belief in a single 'truth' about hospitals (Guardian 26 March 2013)



Updated 28 March 2013

Enhanced Services

One of the governance mechanisms described in recent paper on the Acquisition of Information Services and SOA Systems by the SOA Acquisition Working Group (described in my earlier post on Defense Procurement) is a so-called Enhanced Services Model, which aims to improve responsiveness.

'In the current model contractor’s incentives are to provide the minimal solution that meets the requirements. As the requirements grow and change, Engineering Change Proposals (ECPs) are required, resulting in real or perceived “scope creep”, and always in cost growth. This results in delays in deployment and fielded capability.

Under the enhanced services model, the contractor will be able to exceed the requirements by providing expanded or additional services. These could be “sold” to other users as a common reusable service that others do not have to develop. The additional services could also provide the basis for further development as the system matures.'

The paper signals an intention of the DoD to apply the concept of Enhanced Services in the defence domain. However, at present much of the discussion about enhanced services around the internet is not in the defence domain but in the healthcare domain. (Even if you search for DoD Enhanced Services you just seem to get stuff about healthcare for soldiers and veterans.)

In healthcare, the enhanced services model is used to allow and encourage service providers (e.g. healthcare system suppliers) to offer additional functionality, while allowing service consumers (e.g. clinics) to use these enhanced services in enhanced or specialized processes. In a complex ecosystem, this model allows scope for innovation and improvement.

The challenge with this approach is that it is not always possible to anticipate the value of enhancements. It also potentially increases the complexity of governance. However, there seems to be a strong opportunity to use the enhanced service model to provide added-value in complex situations where the service environment is not centrally controlled.

Joined-Up Healthcare

25 May 2007

A report has just been published on the sad death of Penny Campbell, who died of blood poisoning two years ago, after calling the health service eight times in four days [BBC News, May 25th 2007].

Apparently each of the calls was treated as a separate event, with no linkage made between them.

Joined-up services doesn't just mean joining up disparate events and processes. Sometimes it's hard enough to join up multiple instances of the same event/process.

In an earlier post on Healthcare Reform, I referred to the possibility of triage. If we can separate simple cases from complex ones, then nurses and other professionals can take some responsibility for the simple cases, call in the doctors for medium cases, and send the most complex cases into hospital.

But it appears that Penny Campbell's fate was the exact reverse of this. The system (if you can call it a system) of out-of-hours healthcare seems to result in highly trained doctors perfoming at a level of capability barely any better than what could be achieved by nurses and other professionals.

A number of issues for event-driven SOA there.

10 March 2010

 

This case was mentioned on BBC radio this morning. Dr Simon Eccles of NHS Connecting for Health appeared to suggest that the NPfIT Summary Care Records might have saved Penny Campbell's life. As Tony Collins points out in his blog (BMA and CfH argue it out over NPfIT Summary Care Records), this claim assumes that the relevant doctors would be able and willing to add sufficiently detailed notes to the Summary Care Record in such a case. As it happens, the current scheme (which is something like ten years overdue already) doesn't allow for such notes to be added to the Summary Care Record. (Perhaps the word "summary" provides a little clue there.)

The key point here is that "joined-up" doesn't just mean putting all the data into some vast and horribly expensive data store, so that thousands and thousands of people around the country can look at your summary care record, it's about having a joined-up process that allows the doctors and other health professionals actually involved in your case to use their medical expertise and intelligence to produce the best possible outcomes. Lots of people remain sceptical that the current scheme would ever achieve this. See my post What's Wrong with the Single Version of Truth.

Service Design and Culture

At the Tavistock Institute yesterday evening, to discuss some recent research on Service Design, presented by Paula Hyde based on her recent paper.

Paula Hyde & Huw Davies. Service Design, Culture and Performance: Collusion and Co-Production in Healthcare. Human Relations 57 (11). Available online until June 30

The immediate conclusions of the research (carried out in a mental health setting) was that attempts to improve service provision had largely failed, because although there were some surface changes in the organization, the fundamental relationships on which the service provision was based remained unaltered. There were some profound forces (explicable in terms of organizational psychology) that inhibited real change. [See also POSIWID post on Locking In].

During the discussion, several areas of healthcare and social care were mentioned where there appeared to be some conflict between the following service design objectives:

• to accord primacy to the service user in the design of the service
• to reduce the service user's dependency on (or abuse of) a given service
• to minimize various forms of risk (or uncertainty) to the service user, to the service provider, and to others
  
• to establish measurable improvements in service delivery (the dreaded targets)
  

Similar design conflicts can be found in some commercial settings, especially for services that are "unwanted", in the sense that companies would prefer not to have to provide these services, and the consumers would prefer not to need them. Many call centre operations deliver services that are unwanted in this sense, and these operations are often highly dysfunctional. [See my earlier post on Emergent Dysfunction.] Not so very different to the mental health systems described in the paper.

Service-Oriented Business Strategy

A service-oriented modelling approach helps us to identify alternative business strategies, involving the creation and deployment of added-value services.

• Identify value-added business services that can be seen (by customers) as more relevant to the context of use. 
• Identify value-added business services that are flexible / reusable (by customers) in multiple use-contexts. 
• Compose value-added business services in an efficient and reliable manner from internal and external capabilities. 
• Provide a service platform to support customers in composing our business services to solve their problems. 

I am in the process of defining an extended example from the pharmaceutical sector. The first part will now be published by CBDI in May 2005.

Why Pharma?

We selected the pharmaceutical industry for a worked example because it provides a good example of a complex information supply chain. A drug company (in collaboration with a distributed network of research and test) produces a drug, together with lots of information relating to the drug. There are several different categories of information user: the patient who takes the drug, the medical practitioner who prescribes and/or dispenses the drug, the health service or insurer that pays for the drug, and the regulator who monitors the safety of the drug.

Pharma appears to illustrate some general characteristics of complex service networks, so this example should be of relevance to other industry sectors.

Above all, for SOA illustration purposes, pharma has two advantages. Firstly, it isn't the same old boring examples everyone else is using (finance, travel, retail). And secondly, it isn't military.

In the past, drug companies have been able to make substantial profits from an essentially drug-centric process, getting high sales volumes for its blockbuster drugs from a largely undifferentiated mass of patients with a given condition. This business model treated the physician or clinic as pretty much equivalent to a retail outlet, and did not involve any relationship with the end-customer (the drug consumer). But this business model is subject to major challenge from several directions.

Approach

We model a business as an open system, whose viability depends on robust and appropriate interactions with a dynamic environment. (This contrasts with the closed system approach adopted by many traditional business modeling methods, whose focus is on producing a complete and coherent account of some internal configuration of processes and services, against a fixed view of the environment.)

We model a service-oriented business as a system of systems. Services here may include tasks automated in software (typically but not necessarily rendered as web services) as well as human tasks.

SOA is not just about decomposition – producing fine-grained services with maximum decoupling. Equally important is to think about composition – how these services can be integrated in many different ways to support a wide variety of demand.

In general terms, there are a number of distinct categories of stakeholder, each performing fairly complex functions in relation to the pharma value chain. A key SOA challenge for a drug company is to provide services to all these different stakeholders in a consistent and coordinated yet flexible way. In order to meet this challenge, we need to produce a series of models, from different stakeholder perspectives, showing how the services can be composed in various contexts of use.

In our view, the essential shift for service-oriented modeling is to view the services, not from the provider's perspective, but from the customer's perspective, and in the customer's context of use.
 

Strategic Reframe

From the perspective of business strategy, what we are looking at here is a strategic reframe of the pharma business. What is the drug company actually selling, and to whom? How does information and services become an integral component of the overall product offering?

The ultimate source of value for the patient is defined in terms of health. The provision of health to patients is based on the deployment of complex medical knowledge by a physician, and this in turn relies on information about particular drugs and combinations of drugs from the drug company and elsewhere. This essentially defines a value ladder, in which the value of the drugs contributes to the value of the heathcare.

While this kind of strategic reframing is widely discussed, what service-oriented modeling provides is a systematic way of determining and analyzing the strategic options, in terms of the value ladders that can be supported.

Can the drug company solve all the problems of healthcare? Can the physician solve all the problems of healthcare? The answer in both cases is of course NO. There is huge complexity involved, and the design goal for SOA is to define a reasonable separation of concerns between the physician and the drug company, that allows each of them to manage an appropriate part of the complexity.

---

Previous Post: SOA Pharma

SOA Pharma

While researching SOA developments and opportunities within the pharmaceutical industry, I have discovered some interesting concepts that seem to have broader implications.

Off-Label

In a pharmaceutical context, Off-Label refers to uses of drugs that are not approved by the regulators and cannot therefore be printed on the product label or officially promoted by the drug company. More generally, it refers to any unauthorized or emergent use of a product or service.

Telephone companies were originally oriented towards providing voice services. Computer users started to use modems, which were designed to extract added value from a voice connection by using it to transmit data. With Voice Over IP, we have now come full-circle: VOIP is designed to extract added value from a data connection by using it to transmit voice. These services are now officially sold by the telephone companies - off-label has become on-label.

Within a service economy or SOA, there is often an explicit intention that a service should be used as much as possible, in as many different contexts as possible. This helps to extract economies of scale/scope from the service. (In very simple terms, the more the service is used, the greater its use-value.) It also helps to achieve more flexible structures, since the overall behaviour of a service-based solution can be altered by changing the use of a given service.

A service designer can build a certain degree of generality and flexibility into a service. For example, by defining the preconditions to be as weak as possible, and by designing differentiation into the service itself.

But a service consumer can sometimes go beyond this, using the service in ways that are not intended by the service designer, may not be properly supported by the service provider, and may even be a formal breach in the terms and conditions. For example, an internet site may offer a weight tracking service for slimmers. A farmer may discover that he can use the same service to track the weight of his livestock, perhaps by falsifying some of the data input fields and by appropriately interpreting the data outputs.

If the service provider detects this, there are several possible responses. Firstly, to tighten the service preconditions to prevent this use, and possibly blacklist the offending consumer. Secondly, to tolerate this use, but monitor its impact on performance or possible liability. Thirdly, to enhance the service to fully support this use (perhaps under a different brand identity).

Why would a service consumer knowingly use a service in a way that is not officially supported by the service provider? Obviously there is some risk involved for the service consumer. But if the service can be twisted to deliver a desired result, is this necessarily a problem?

Some service providers will see off-label use as a form of abuse. Clearly there are some forms of abuse that threaten the security of the service provider and/or of legitimate users, and this risk must be properly managed. However, it is not necessary to regard all off-label uses as security threats. In many cases, it is better to regard off-label uses as business opportunities.

More importantly, we need to see off-label service usage as a governance issue. What controls (if any) are appropriate for off-label service usage, and how do we accommodate the asymmetry between on-label and off-label in the geometry of services.

Shared Care Protocol

With long-term healthcare, there are complex referral pathways involving several practitioners.

A shared care protocol is designed to ensure effective liaison and joint working between practitioners. It defines who is responsible for what actions and outcomes. For an example, see this response to this management conundrum from the Pharmacy Management journal.

An internet search for "shared care protocol" will find hundreds of specific protocols for specific drugs and other healthcare treatments.

The shared care protocol is essentially a piece of orchestration. There are two key questions here: firstly whether the protocol is standard or customized for each patient; secondly whether it is defined by a specialist team or through negotiation between multiple stakeholders (collaborative composition).

Helen Snowden and Sarah Marriott argue for local protocols. (Developing a local shared care protocol for managing people with psychotic illness in primary care, Psychiatric Bulletin (2003) 27: 261-266)
The notion of shared care protocol has obvious relevance for customer relationship management, where a range of customer-related actions need to be appropriately coordinated and orchestrated, in a way that is appropriately aligned to the unfolding responses of the customer. More generally, any business challenge that calls for a complex and coordinated response to some unfolding situation, which is found in some of the most interesting applications of SOA.

Surrogate EndPoints

The specification, orchestration and testing of healthcare (including drug treatment) is defined in terms of achieving certain endpoints. This enables a clinician to make connections between the clinical trial data and the desired patient outcomes.

But sometimes the real endpoints are difficult or impossible to measure quickly. So clinical trials and care protocols are defined in terms of surrogate endpoints. These are alternative metrics that are thought to provide a reasonable indication or prediction of the real endpoints. For example, measuring blood pressure or cholesterol levels as an indication of the likelihood of heart problems.

The problem for the clinician is that these surrogates introduce another level of complexity into the reasoning process. The statistical correlation between the surrogate endpoint and the real endpoint works for the average patient, and becomes less reliable as the patient diverges from the norm.

Business processes commonly use surrogate endpoints and metrics, which often relate to the intermediate steps of a business process. For example, an online marketing operation might measure the number of enquiries, based on an expectation that a certain proportion of these enquiries can be converted into sales. This leads to a functional decomposition: one subsystem is responsible for delivering enquiries, while another subsystem is responsible for converting enquiries into sales.

But the more you differentiate your operations to address different market niches, the more difficult it becomes to integrate the subsystems back together in a consistent and effective way. SOA allows more complex differentiation and integration, and potentially supports much greater levels of customer-specific behaviour at a given cost level. But in order to design this properly, we need to include both the real and the surrogate endpoints in our business models for SOA.

Links 

Center for Health Evidence: Users' Guides to Evidence-Based Practice


Richard Veryard, Business Modeling for SOA Worked Example: Pharmaceuticals (CBDI Journal May 2005)

See also Off-Label (March 2005)