Testing Multiples

From engineering to medicine, professionals are often forced to rely on tests that are not always completely accurate. In this post, I shall look at the tests that millions of people were obliged to use during the pandemic, to check whether they had COVID-19. The two most common tests were Lateral Flow and PCR. Lateral Flow was quicker and more convenient, while PCR took longer (because the sample had to be sent to a lab) and was supposedly more accurate.

There was also a difference in the data collected from these tests. Whereas all the results from the PCR tests should have been available in the labs, the results from the lateral flow tests were only reported under certain circumstances. There was no obligation to report a negative test unless you needed access to something, and people sometimes chose not to report positive tests because of the restrictions that might follow. And of course people only took the tests when they had to, or wanted to. When people had to pay for the tests, this obviously made a big difference.

To compensate for these limitations, some random screening was carried out, which was designed to produce more reliable and representative datasets. However, these datasets were much smaller.

 

So what can we do with this kind of data? Firstly, it tells us something about the disease - whether it is distributed evenly across the country or concentrated in certain places, how quickly it is spreading. If we can combine the test results with other information about the test subjects, we may be able to get some demographic information - for example, how is the disease affecting people of different age, gender or race, how is it affecting different job categories. And if we have information from the health service, we can estimate how many of those testing positive end up in hospital.

This kind of information allows us to make predictions - for example, future demand for hospital beds, possible shortages of key workers. It also allows us to assess the effects of various protective measures - for example, to what extent does mask-wearing, social distancing and working from home reduce the rate of transmission.

Besides telling us about the disease, the data should also be able to tell us something about the tests. And the accuracy of the predictions provides a feedback loop, which may enable us to reassess either the test data or the predictive models.

 

In her book The Body Multiple, Annemarie Mol discusses the differences between two alternative tests for atherosclerosis, and describes how clinicians deal with cases where the two tests appear to provide conflicting results, as well as cases where there may be other reasons to question the test results. Instead of having a single view of the disease, she talks about its multiplicity or manyfoldedness.

But questioning the test results in a particular case, or highlighting particular issues with a given test, does not mean denying the overall value of the test. Most of the time we can continue to regard a test as useful, even as we are considering ways of improving it.

If and when we introduce a new or improved test, we may then wish to translate data between tests. In other words, if test A produced result X, then we would have expected test B to produce result Y. While this kind of translation may be useful for statistical purposes, we need to be careful about its use in individual cases.

For many people, the second discourse appears to undermine the first discourse. If we can't always trust the data, can we ever trust the data? During the COVID pandemic, many rival interpretations of the data emerged; some people chose interpretations that confirmed their preconceptions, while others turned away from any kind of data-driven reasoning.

 

The COVID pandemic became a politically contentious field, so what if we look at other kinds of testing? In safety engineering, components and whole products are subjected to a range of tests, which assess the risk of certain kinds of failure. Obviously there are manufacturers and service providers with a commercial interest in how (and by whom) these tests are carried out, and there may be regulators and researchers looking at how these tests can be improved, or to detect various forms of cheating, but ordinary consumers don't generally spend hours on YouTube complaining about their accuracy and validity.

Meanwhile even basic corporate reporting may be subject to this kind of multiplicity, as illustrated in my recent post on Data Estimation (July 2022).

So there is a level of complexity here, which not all data users may feel comfortable with, but which data professionals may not feel comfortable about hiding. In a traditional report, these details are often pushed into footnotes, and in an online dashboard there may be symbols inviting the user to drill down for further detail. But is that good enough?

---

Annemarie Mol, The Body Multiple: Ontology in Medical Practice (Duke University Press 2002)

Wikipedia: COVID-19 testing

Related posts: Data-Driven Reasoning - COVID (April 2022), Data Estimation (July 2022)

Responsible Beta Testing

Professor Luciano Floridi has recently made clear his opposition to irresponsible beta testing, by which he means "trying something without exactly knowing what one is doing, to see what happens, on people who may not have volunteered to be the subjects of the test at all".

In a private communication, Professor Floridi indicates that he would regard some of the recent experiments in facial recognition as examples of irresponsible beta testing. Obviously if the police are going to arrest people who decline to participate, these are not exactly willing volunteers.

Some of the tech giants have got into the habit of releasing unreliable software to willing volunteers, and calling this a "beta programme". There are also voices in favour of something called permanent beta, which some writers regard as a recipe, not just for technology but also for living in a volatile world. So the semantics of "beta" has become thoroughly unclear.

However, I think this kind of activity does not represent the original purpose of beta testing, which was the testing of a product without the development team being present. Let me call this responsible beta testing. While it is understood that beta testing cannot be guaranteed to find all the problems in a new product, it typically uncovers problems that other forms of verification and validation have missed, so it is generally regarded as a useful approach for many classes of system, and probably essential for safety-critical systems.

This is how this might work in a robotic context. Let's suppose you are building a care robot for the elderly. Before you put the robot into production, you are going to want to test it thoroughly - perhaps first with able-bodied volunteers, then with some elderly volunteers. During the testing, the robot will be surrounded with engineers and other observers, who will be able to intervene to protect the volunteer in the event of any unexpected or inappropriate behaviour on the part of the robot. Initially, this testing may take place in the lab with the active participation of the development team, but at least some of this testing would need to take place in a real care home without the development team being present. This may be called beta-testing or field testing. It certainly cannot be regarded as merely "trial and error".

For medical devices, this kind of testing is called a clinical trial, and there are strict regulations about how this should be done, including consent from those taking part in the trial based on adequate information and explanation, proper reporting of the results and any unexpected or unwanted effects, and with the ability to halt the trial early if necessary. It might be possible to establish similar codes of practice or even regulations for testing other classes of technology, including robotics.


(In the same communication, Professor Floridi fully agrees with the distinction made here, and affirms the crucial importance of responsible beta testing.)

---

Links

Lizzie Dearden, Police stop people for covering their faces from facial recognition camera then fine man £90 after he protested (Independent, 31 January 2019)

This blogpost forms part of an ongoing project to articulate Responsibility by Design. See Reponsibility by Design - Activity View (May 2019)

SOA Testing 2

Nearly finished my article on SOA testing.

Some people seem to think that SOA testing is all about testing web services – as if SOA testing is the same as JBOWS testing. But I am keen to emphasize other aspects of SOA testing as well.

One of the things I’ve been talking to a few people about is the testing of the service architecture itself. Some people have expressed doubts about this. Surely if there are any flaws in the architecture, these will become apparent when we test the other levels. Or perhaps we can just wait until the system goes live, and fix any architectural problems then.

I don’t agree. If we don’t test architecture, how do we know whether the architects have messed up? If we care about the quality of architecture, we need tests that are specifically designed to determine the correctness and robustness of the architecture, as well as tests to determine whether the intentions of the architects have been correctly realised. For example, we may decide to monitor calls between the layers, to verify that there aren’t any hidden dependencies, or to measure the performance overhead associated with each layer. Or when the architects have specifically designed multiple provision of some resource to avoid a single point of failure, we might monitor the utilization of this resource over a range of conditions to ensure that this aspect of the architecture is successful.

I suspect most people don’t bother – they assume that any flaws that don’t surface through other tests don’t matter. But I think this is an optimistic assumption, and system testing should never be based on optimism. Architectural testing may sometimes be difficult and expensive, and would therefore need to be justified against some assessment of business risk, but it should always be considered as an option.

So I’d welcome some comments from readers. Do you carry out tests that are specifically designed to verify and validate the architecture as a whole? If you don’t carry out such tests, is this (a) because you don’t have the time, (b) you don’t know how, or (c) you or your sponsors cannot see any value from such tests?

SOA Testing

I am pulling together some materials on SOA testing. (CBDI members can expect a journal article in due course, the rest of you will get a few snippets of work-in-progress here. Vendors who want to brief me on some fantastic new product in this area, or anyone else who wants to bend my ear on the subject, please get in touch soonest, or leave a comment below.)

There are four reasons why we need to talk about SOA testing.

• Complexity – networks of services may sometimes be more complex than traditional applications, and therefore may be more difficult to test thoroughly. (SOA doesn't necessarily make things more complex; but it allows people to do more complex things, and when people can they often will.)
  

• Scale – Internet-scale systems raise new challenges for volume testing. Last year, Skype crashed because the unusually high restart volumes associated with Microsoft's Patch Tuesday exposed a bug that hadn't been found in normal testing [Skype Skuppered]. These kinds of situation are practically impossible to simulate.

• Productivity – as development productivity increases, the ratio of testing effort to development effort increases unless there is equivalent innovation in testing. (This has always been a problem for software productivity tools, because the front-end innovation is more glamorous than the back-end testing.)

• Redundancy – the benefits of SOA stand or fall on the amount of regression testing that is required. This is perhaps the most important reason: if we get it wrong, the IT cost advantages of SOA might be completely blown away; but if we get it right, it can be one of the major sources of benefit.

Clouds and Clocks 3

Some CEP vendors (Aleri, Coral8, RuleCore) are boasting that their CEP software is "deterministic" or "consistent".

In other words, "clockwork" rather than "cloudlike". Mark Tsimelzon, President & CTO of Coral8, insists that we shouldn't think of an event cloud at all - he prefers the concept of event stream.

But what do "determinism" and "consistency" actually mean here, and why are they important?

Both Aleri and Coral8 define “determinacy” to mean that a set of input data produces the same output data.

Aleri adds that the order in which the data arrives doesn’t matter. This only follows from the definition of determinism if we assume that "set" means a pure set, without sequence. Perhaps a better word for this property is "commutativity". There is also the related property of idempotence, in which it doesn't matter if the same event or message appears more than once. Aleri also defines “consistency” to mean that the output is predictable from the input - which sounds equivalent to their definition of "determinacy" - but they go on to say that consistency means that "the results are the ones we’d expect" - which sounds more like correctness to me.

Meanwhile, Mark of Coral8 introduces three degrees of determinism: non-deterministic, single-stream and multiple-stream determinism. He hopes his post has demystified the notion of determinism a little. Er, thanks Mark.

But why does determinism matter? Mark explains that this has to do with testing, especially regression testing. If we cannot compare results from two runs of the same stream of events, then our traditional approach to software testing breaks down. At least, that's the only reason he mentions in his blog.

This is rather like the value of the controlled experiment in science. A laboratory provides a controlled environment, in which most of the variables can be fixed, so that the cause-and-effect can be isolated. Software testing also requires a controlled environment, so that any unexpected variation in results can be reliably traced to a specific cause - such as a software bug.

But complex systems are not particularly amenable to laboratory experiments, because the requisite complexity cannot be properly contained and controlled. And complex internet systems raise a similar challenge.

Determinism may be useful from a software engineering perspective, but it seems to deny some of the technological potential of complex event processing, including machine learning. Surely the whole point of machine learning is that the system doesn't always produce the same results on the same input, but is capable of producing improved results on the same input.

So I'd like to see a wider debate about the limits of determinism. I've been talking about this for a while in an SOA context - see my earlier post on Determinism.

---

Consistency and Determinacy (Aleri March 2008)

Marco Seiriö, Context Management (RuleCore June 2009)

Mark Tsimelzon, Determinism in CEP (Coral8 October 2007)

Mark Tsimelzon, Unclouding and streamlining your thinking about CEP use cases (Coral8 October 2007)


Links via WayBack Machine. Updated 2 December 2014.

Adaptive Path

Adaptive Path’s Peter Merholz recently talked to the founder of User Interface Engineering Jared Spool about user research. Here is an interesting example of website design from their conversation.

We were working with a client, testing a website that sold furniture. And the client had already done some testing. They had created some simple tasks, and one of the tasks was basically asking users to come to the site and buy a bookcase they might want to buy.

Every user in that study went to the search box, and typed in the word “bookcase,” immediately. So they were off trying to improve the hits on the word “bookcase,” and get the results better. We did a subsequent test, and instead of asking people to look for a bookcase, we gave them a slightly different task.

We said imagine you have 300 different paperback and hardcover books in cardboard boxes, all strewn through your living room. You need a way to organize this stuff such that you can find the stuff you like, and people who come over can be impressed with your collection. What do you do? People would go to the site, and then click on links. They’d click on furniture links; they’d type into the search engine storage systems, and then type in shelves. Not a single user in our phase of the study typed in the word bookcase. So by changing the description of the task, we got a completely different result.

In this example, the supply side has composed several instances of SHELF into a composite solution called BOOKCASE. But the demand side doesn't recognize this solution as relevant to the context of use, unless forced/led. So there is a fundamental asymmetry between supply and demand.

(The problem I experience with ready-made bookcases is that they are almost always the wrong size - with room for several inches of dust in front of the books. Do furniture-makers themselves only read large-format books? I have generally found it better to get bookcases made to my own specification.)

Notice that the supplier designs and tests the website in a way that reinforces the supply-side view of the world (ontology). Notice that the method for deconstructing this supply-side ontology involves reference to a user experience (living room strewn with books). Notice also that decomposing BOOKCASE back into SHELF doesn't completely eliminate the asymmetry - the demand-side notion of shelf (a surface on which I can store and display books) is not identical to the supply-side notion of shelf (a flat piece of rigid material that can be affixed in a horizontal position, with/without brackets and screws).

Software Outsourcing

Companies outsource their software development for several reasons:

1. It is cheaper because the outsource contractors use cheap labour (usually with development shops based in countries with lower wage rates).
2. It is cheaper because the outsource contractors can achieve economies of scale - for example through some form of software reuse. (Anyone really believe this?)
   
3. It allows companies to manage variations in the volume of software development work, without the need to hire extra staff.
4. It reduces the dependence on an inhouse IT department - especially where there is a history of poor performance.
   

But there are some perceived disadvantages of software outsourcing.

1. Increased "distance" and weak communication between the developers and the users. Issues of business/IT alignment.
   
2. Reduced client control over the software product and process. Issues of trust and security.
   

In this post, I want to discuss how a service-oriented approach affects this equation.

Architecture

The customer organization typically retains responsibility for requirements and architecture. This means that the software specifications are produced by business analysts and software architects in the customer organization. The outsource contractor then produces software artefacts according to these specifications and delivers these artefacts back to the customer organization, where they are tested and deployed.

The first problem here is with the depth of the testing. Most IT departments have got people who know how to do functional testing, usability testing, performance testing - but that's not enough. Presumably the architects have devoted some attention to producing an overall solution with certain architectural characteristics, such as flexibility/maintainability. The requirements will have been stratified into layers and decomposed into discrete units (modules, components, services); the architects will have sought to achieve high cohesion and low coupling, both horizontally and vertically.

However, developers may have an incentive to produce software that is inflexible and more expensive to maintain, because this generates future income for themselves. They can achieve this by introducing hidden coupling between layers, tight coupling where the architects intended loose coupling, inappropriate bindings of varying kinds. Even if these are detected, it is hard to determine whether they were inserted deliberately or merely slipped in as a result of poor development practices. And in most cases, they will not be detected until much later.

So it is crucial for architects in the customer organization to verify that the developers have conformed to the original architectural intentions. This entails an ability to discover the actual architectural structure of the delivered software artefacts - and this may involve a combination of code inspection (static analysis) and runtime monitoring (dynamic analysis). We are starting to see web service modelling and management tools that provide some support for architecture discovery; the next step will be tool support for architecture verification.

Automation

Service-oriented computing reinforces the importance of tools. In the past, development tools for modelling, design and code-generation were regarded as productivity aids. This meant they were sometimes used in high-wage economies (USA, Western Europe) to cut labour costs, but could not be cost-justified in lower-wage economies (Eastern Europe, India).

One modelling tool vendor tells me that this is now changing, at least in India (but not yet in Eastern Europe/Russia), where software companies acquire these tools to improve the relationship with the client and to improve software quality. In other words, the relationship between the outsourcing customer and the outsourcing provider is being automated - reducing the communication distance between developers and users.

Agility/Granularity

The growing importance of components and services opens up the possibility of software development at a much finer level of granularity, with small software artefacts developed independently and composed using web service protocols and orchestration languages (such as BPEL).

Although this could result in a degree of fragmentation, we may expect collaborative software development platforms to appear, which would have to deal with the administration, transaction costs and requirements/quality issues - probably with a strong flavour of agile development

Anticlockwise

In terms of the Boxer Model of Collaborative Composition, software outsourcing typically follows an anticlockwise path - focused on achieving economies of scale. There are significant challenges involved in moving to a clockwise path. I shall return to this subject in another post.