Responsible Beta Testing

Professor Luciano Floridi has recently made clear his opposition to irresponsible beta testing, by which he means "trying something without exactly knowing what one is doing, to see what happens, on people who may not have volunteered to be the subjects of the test at all".

In a private communication, Professor Floridi indicates that he would regard some of the recent experiments in facial recognition as examples of irresponsible beta testing. Obviously if the police are going to arrest people who decline to participate, these are not exactly willing volunteers.

Some of the tech giants have got into the habit of releasing unreliable software to willing volunteers, and calling this a "beta programme". There are also voices in favour of something called permanent beta, which some writers regard as a recipe, not just for technology but also for living in a volatile world. So the semantics of "beta" has become thoroughly unclear.

However, I think this kind of activity does not represent the original purpose of beta testing, which was the testing of a product without the development team being present. Let me call this responsible beta testing. While it is understood that beta testing cannot be guaranteed to find all the problems in a new product, it typically uncovers problems that other forms of verification and validation have missed, so it is generally regarded as a useful approach for many classes of system, and probably essential for safety-critical systems.

This is how this might work in a robotic context. Let's suppose you are building a care robot for the elderly. Before you put the robot into production, you are going to want to test it thoroughly - perhaps first with able-bodied volunteers, then with some elderly volunteers. During the testing, the robot will be surrounded with engineers and other observers, who will be able to intervene to protect the volunteer in the event of any unexpected or inappropriate behaviour on the part of the robot. Initially, this testing may take place in the lab with the active participation of the development team, but at least some of this testing would need to take place in a real care home without the development team being present. This may be called beta-testing or field testing. It certainly cannot be regarded as merely "trial and error".

For medical devices, this kind of testing is called a clinical trial, and there are strict regulations about how this should be done, including consent from those taking part in the trial based on adequate information and explanation, proper reporting of the results and any unexpected or unwanted effects, and with the ability to halt the trial early if necessary. It might be possible to establish similar codes of practice or even regulations for testing other classes of technology, including robotics.


(In the same communication, Professor Floridi fully agrees with the distinction made here, and affirms the crucial importance of responsible beta testing.)

---

Links

Lizzie Dearden, Police stop people for covering their faces from facial recognition camera then fine man £90 after he protested (Independent, 31 January 2019)

This blogpost forms part of an ongoing project to articulate Responsibility by Design. See Reponsibility by Design - Activity View (May 2019)