Netflix and Algorithms

Following my previous posts on Netflix, I have been reading a detailed analysis in Ed Finn's book, What Algorithms Want (2017).

Finn's answer to my question Does Big Data Drive Netflix Content? is no, at least not directly. Although Netflix had used data to commission new content as well as recommend existing content (Finn's example was House of Cards) it had apparently left the content itself to the producers, and then used data and algorithmic data to promote it. 

After making the initial decision to invest in House of Cards, Netflix was using algorithms to micromanage distribution, not production. Finn p99

Obviously something written in 2017 doesn't say anything about what Netflix has been doing more recently, but Finn seems to have been looking at the same examples as the other pundits I referenced in my previous post.

Finn also makes some interesting points about the transition from the original Cinematch algorithm to what he calls Algorithm 2.0.

The 1.0 model gave way to a more nuanced, ambiguity-laden analytical environment, a more reflexive attempt to algorithmically comprehend Netflix as a culture machine. ... Netflix is no longer constructing a model of abstract relationships between movies based on ratings, but a model of live user behavior in their various apps Finn p90-91

The coding system relies on a large but hidden human workforce, hidden to reinforce the illusion of pure algorithmic recommendations (p96) and perfect personalization (p107). As Finn sees it, algorithm 1.0 had a lot of data but no meaning, and was not able to go from data to desire (p93). Algorithm 2.0 has vastly more data, thanks to this coding system - but even the model of user behaviour still relies on abstraction. So exactly where is the data decoded and meaning reinserted (p96)?

As Netflix executives acknowledge, so-called ghosts can emerge (p95), revealing a fundamental incompleteness (lack) in symbolic agency (p96).

---

Ed Finn, What Algorithms Want: Imagination in the Age of Computing (MIT Press, 2017)

Alexis C. Madrigal, How Netflix Reverse-Engineered Hollywood (Atlantic, 2 January 2014)

Previous posts: Rhyme or Reason - The Logic of Netflix (June 2017), Does Big Data Drive Netflix Content? (January 2021)

Does the Algorithm have the Last Word?

In my post on the performativity of data (August 2021), I looked at some of the ways in which data and information can make something true. In this post, I want to go further. What if an an algorithm can make something final.

I've just read a very interesting paper by the Canadian sociologist Arthur Frank, which traces the curious history of a character called Devushkin - from a story by Gogol via another story by Dostoevsky into some literary analysis by Bakhtin.

In Dostoevsky's version, Devushkin complained that Gogol's account of him was complete and final, leaving him no room for change or development, hopelessly determined and finished off, as if he were already quite dead.

For Bakhtin, all that is unethical begins and ends when one human being claims to determine all that another is and can be; when one person claims that the other has not, cannot, and will not change, that she or he will die just as she or he always has been. Frank

But that's pretty much what many algorithms do. Machine learning algorithms extrapolate from historical data, captured and coded in ways that reinforce the past, while more traditionally programmed algorithms simply betray the opinions and assumptions of their developers. For example, we see recruitment algorithms that select men with a certain profile, while rejecting women with equal or superior qualifications. Because that's what's happened in the past, and the algorithm has no way of escaping from this.

 

I'm going to keep saying this, I think: just as we have a right to download the data any corp holds on us, we should also have the right to reset any view an algorithm has of us.

We used to be able to redefine and evolve ourselves. Algorithmic views of us hold us back.

— Russell Garner (@rgarner) March 4, 2022

The inbuilt bias of algorithms has been widely studied. See for example Safiya Noble and Cathy O'Neil.

David Beer makes two points in relation to the performativity of algorithms. Firstly through their material interventions.

Algorithms might be understood to create truths around things like riskiness, taste, choice, lifestyle, health and so on. The search for truth becomes then conflated with the perfect algorithmic design – which is to say the search for an algorithm that is seen to make the perfect material intervention.

And secondly through what he calls discursive interventions.

The notion of the algorithm is part of a wider vocabulary, a vocabulary that we might see deployed to promote a certain rationality, a rationality based upon the virtues of calculation, competition, efficiency, objectivity and the need to be strategic. As such, the notion of the algorithm can be powerful in shaping decisions, influencing behaviour and ushering in certain approaches and ideals.

As Massimo Airoldi argues, both of these fall under what Bourdieu calls Habitus - which means an inbuilt bias towards the status quo. And once the algorithm has decided your fate, what chance do you have of breaking free?

---

Massimo Airoldi, Machine Habitus: Towards a sociology of algorithms (Polity Press, 2022)

David Beer, The social power of algorithms (Information, Communication & Society, 20:1, 2017) 1-13, DOI: 10.1080/1369118X.2016.1216147

Safiya Noble, Algorithms of Oppression (New York University Press, 2018)

Cathy O'Neil, Weapons of Math Destruction (2016)

Arthur W. Frank, What Is Dialogical Research, and Why Should We Do It? (Qual Health Res 2005; 15; 964) DOI: 10.1177/1049732305279078

Carissa Véliz, If AI Is Predicting Your Future, Are You Still Free? (Wired, 27 December 2021)

Related posts: Could we switch the algorithms off? (July 2017), Algorithms and Governmentality (July 2019), Algorithmic Bias (March 2021), On the performativity of data (August 2021)

Walking Wounded

Let us suppose we can divide the world into those who trust service companies to treat their customers fairly, and those who assume that service companies will be looking to exploit any customer weakness or lapse of attention.

For example, some loyal customers renew without question, even though the cost creeps up from one year to the next. (This is known as price walking.) While other customers switch service providers frequently to chase the best deal. (This is known as churn. B2C businesses generally regard this as a Bad Thing when their own customers do it, not so bad when they can steal their competitors' customers.)

Price walking is a particular concern for the insurance business. The UK Financial Conduct Authority (FCA) has recently issued new measures to protect customers from price walking.

Duncan Minty, an insurance industry insider who blogs on Ethics and Insurance, believes that claims optimization (which he calls Settlement Walking) raises similar ethical issues. This is where the insurance company tries to get away with a lower claim settlement, especially with those customers who are most likely to accept and least likely to complain. He cites a Bank of England report on machine learning, which refers among other things to propensity modelling. In other words, adjusting how you treat a customer according to how you calculate they will respond.

My work on data-driven personalization includes ethics as well as practical considerations. However, there is always the potential for asymmetry between service providers and consumers. And as Tim Harford points out, this kind of exploitation long predates the emergence of algorithms and machine learning.

 

Update

In the few days since I posted this, I've seen a couple of news items about autorenewals. There seems to be a trend of increasing vigilance by various regulators in different countries to protect consumers.

Firstly, the UK's Competition and Markets Authority (CMA) has unveiled compliance principles to curb locally some of the sharper auto-renewal practices of antivirus software firms. (via The Register).

Secondly, new banking rules in India for repeating payments. Among other things, this creates challenges for free trials and introductory offers. (via Tech Crunch)

---

Machine Learning in UK financial services (Bank of England / FCA, October 2019)

FCA confirms measures to protect customers from the loyalty penalty in home and motor insurance markets (FCA, 28 May 2021)

Tim Harford, Exploitative algorithms are using tricks as old as haggling at the bazaar (2 November 2018)

Joi Ito, Supposedly ‘Fair’ Algorithms Can Perpetuate Discrimination (Wired Magazine, 5 February 2019)

Duncan Minty, Is settlement walking now part of UK insurance? (18 March 2021), Why personalisation will erode the competitiveness of premiums (7 September 2021)

Manish Singh, Tech giants brace for impact in India as new payments rule goes into effect (TechCrunch, 1 October 2021)

Richard Speed, UK competition watchdog unveils principles to make a kinder antivirus business (The Register, 19 October 2021)

Related posts: The Support Economy (January 2005), The Price of Everything (May 2017), Insurance and the Veil of Ignorance (February 2019)

Related presentations: Boundaryless Customer Engagement (October 2015), Real-Time Personalization (December 2015)

The Road Less Travelled

Are algorithms trustworthy, asks @NizanGP.

"Many of us routinely - and even blindly - rely on the advice of algorithms in all aspects of our lives, from choosing the fastest route to the airport to deciding how to invest our retirement savings. But should we trust them as much as we do?"

Dr Packin's main point is about the fallibility of algorithms, and the excessive confidence people place in them. @AnnCavoukian reinforces this point.

Always look under the hood, especially with AI: we need to, not only examine the algorithms used, but the training datasets that shape the algorithms. We have to find ways to avoid the tyranny of algorithms!

— Ann Cavoukian, Ph.D. (@AnnCavoukian) June 15, 2019

But there is another reason to be wary of the advice of the algorithm, summed up by the question: Whom does the algorithm serve?

Because the algorithm is not working for you alone. There are many people trying to get to the airport, and if they all use the same route they may all miss their flights. If the algorithm is any good, it will be advising different people to use different routes. (Most well-planned cities have more than one route to the airport, to avoid a single point of failure.) So how can you trust the algorithm to give you the fastest route? However much you may be paying for the navigation service (either directly, or bundled into the cost of the car/device), someone else may be paying a lot more. For the road less travelled.

The algorithm-makers may also try to monetize the destinations. If a particular road is used for getting to a sports venue as well as the airport, then the two destinations can be invited to bid to get the "best" routes for their customers - or perhaps for themselves. ("Best" may not mean fastest - it could mean the most predictable. And the venue may be ambivalent about this - the more unpredictable the journey, the more people will arrive early to be on the safe side, spreading the load on the services as well as spending more on parking and refreshments.)

In general, the algorithm is juggling the interests of many different stakeholders, and we may assume that this is designed to optimize the commercial returns to the algorithm-makers.

The same is obviously true of investment advice. The best time to buy a stock is just before everyone else buys, and the best time to sell a stock is just after everyone else buys. Which means that there are massive opportunities for unethical behaviour when advising people where / when to invest their retirement savings, and it would be optimistic to assume that the people programming the algorithms are immune from this temptation, or that regulators are able to protect investors properly.

And that's before we start worrying about the algorithms being manipulated by hostile agents ...

So remember the Weasley Doctrine: "Never trust anything that can think for itself if you can't see where it keeps its brain."

---

Nizan Geslevich Packin, Why Investors Should Be Wary of Automated Advice (Wall Street Journal, 14 June 2019)

Dozens of drivers get stuck in mud after Google Maps reroutes them into empty field (ABC7 New York, 26 June 2019) HT @jonerp

Related posts: Towards Chatbot Ethics (May 2019), Whom does the technology serve? (May 2019), Robust Against Manipulation (July 2019)


Updated 27 July 2019

Automated Tetris

Following complaints that Amazon sometimes uses excessively large boxes for packing small items, the following claim appeared on Reddit.

"Amazon uses a complicated software system to determine the box size that should be used based on what else is going in the same truck and the exact size of the cargo bay. It is playing automated Tetris with the packages. Sometimes it will select a larger box because there is nothing else that needs to go out on that specific truck, and by making it bigger, it is using up the remaining space so items don't slide around and break. This actually minimizes waste and is on the whole a greener system. Even if for some individual item it looks weird. It's optimizing for the whole, not the individual." [source: Reddit via @alexsavinme]

Attached to the claim is a link to @willknight's 2015 article about Amazon's robotic warehouses. The article mentions the packing problem but doesn't mention the variation of box sizes.

The claim quickly led to vigorous debate, both on Reddit and on Twitter. Here are a selection of the argument and counter-arguments.

• Suggesting that the Reddit claim was based on a misreading of the MIT article.

• Asserting that people working in warehouses (Amazon and other) were unaware of such an algorithm. (As if this were relevant evidence.)

• Evidence that equally sophisticated algorithms are in use at other retailers and logistics companies. (Together with an assumption that if others have them, Amazon must definitely have them.)

• Evidence that some operational inefficiencies exist at Amazon and elsewhere. (What, isn't Amazon perfectly optimized yet?)

• Providing evidence that computer systems would not always recommend the smallest possible box. For example, this comment: "At Target the systems would suggest a size but we could literally use whatever we wanted to. I constantly put stuff in smaller boxes because it just made so much more sense." (Furthermore, the humans being able to frustrate the intentions of the software.)

• Suggesting that errors in box sizes are sometimes caused by mix-up of units - one item going in a box large enough for a dozen.

• Pointing out that the solution described above would only work for transport between warehouses (where the vehicle is full for the whole trip) but wouldn't work for "last mile" delivery runs (where the vehicle becomes progressively more empty during the trip).

• Pointing out that the "last mile" is the most inefficient part of the journey. (But this doesn't stop retailers looking for efficiency savings earlier in the journey.)

• Pointing out that there were more efficient solutions for preventing packages shifting in transit - for example, inflatable bags.

• Pointing out that an overlarge box merely displaces the problem - the item can be damaged by sliding around inside the box.

• Complaining about the ethics, employment policies and environmental awareness of Amazon.

• Denigrating the intelligence and diligence of the workers in the Amazon warehouse. (Lazy? Really?)

Some people have complained that as the claim is evidently false, it counts as fake news and should be deleted. But it is certainly true that retailers and logistics companies are constantly thinking about ways of reducing packaging and waste, and there are several interesting contributions to the debate, even if some of the details may not quite work.

It's also worth noting that the claim is written in a highly plausible style - that's just how people in that world would talk. So maybe someone has come across a proposal or pilot or patent application along these lines, even if this exact solution was never fully implemented.

Some may doubt that such a solution would be "greener on the whole". But any solution architect should get the principle of "optimizing for the whole, not the individual". (Not always so easy in practice, though.) 

---

Will Knight, Inside Amazon’s Warehouse, Human-Robot Symbiosis (MIT Technology Review, 7 July 2015)

Wikipedia: Packing Problems

Expert Systems

Is there a fundamental flaw in AI implementation, as @jrossCISR suggests in her latest article for Sloan Management Review? She and her colleagues have been studying how companies insert value-adding AI algorithms into their processes. A critical success factor for the effective use of AI algorithms (or what we used to call expert systems) is the ability to partner smart machines with smart people, and this calls for changes in working practices and human skills.

As an example of helping people to use probabilistic output to guide business actions, Ross uses the example of smart recruitment.

But what’s the next step when a recruiter learns from an AI application that a job candidate has a 50% likelihood of being a good fit for a particular opening?

Let's unpack this. The AI application indicates that at this point in the process, given the information we currently have about the candidate, we have a low confidence in predicting the performance of this candidate on the job. Unless we just toss a coin and hope for the best, obviously the next step is to try and obtain more information and insight about the candidate.

But which information is most relevant? An AI application (guided by expert recruiters) should be able to identify the most efficient path to reaching the desired level of confidence. What are the main reasons for our uncertainty about this candidate, and what extra information would make the most difference?

Simplistic decision support assumes you only have one shot at making a decision. The expert system makes a prognostication, and then the human accepts or overrules its advice.

But in the real world, decision-making is often a more extended process. So the recruiter should be able to ask the AI application some follow-up questions. What if we bring the candidate in for another interview? What if we run some aptitude tests? How much difference would each of these options make to our confidence level?

When recruiting people for a given job, it is not just that the recruiters don't know enough about the candidate, they also may not have much detail about the requirements of the job. Exactly what challenges will the successful candidate face, and how will they interact with the rest of the team? So instead of shortlisting the candidates that score most highly on a given set of measures, it may be more helpful to shortlist candidates with a range of different strengths and weaknesses, as this will allow interviewers to creatively imagine how each will perform. So there are a lot more probabilistic calculations we could get the algorithms to perform, if we can feed enough historical data into the machine learning hopper.

Ross sees the true value of machine learning applications to be augmenting intelligence - helping people accomplish something. This means an effective collaboration between one or more people and one or more algorithms. Or what I call organizational intelligence.

---

Postscript (18 December 2017)

In his comment on Twitter, @AidanWard3 extends the analysis to multiple stakeholders.

Framed as risk reduction there are several different types of risk for different stakeholders in your scenario. An extended analysis still works. https://t.co/OvU03vH2eY

— Aidan Ward (@AidanWard3) December 14, 2017

This broader view brings some of the ethical issues into focus, including asymmetric information and algorithmic transparency. 

---

Jeanne Ross, The Fundamental Flaw in AI Implementation (Sloan Management Review, 14 July 2017)

Could we switch the algorithms off?

In his review of Nick Bostrom's book Superintelligence, Tim Adams suggests that Bostrom has been reading too much of the science fiction he professes to dislike. When people nowadays want to discuss the social and ethical implications of machine intelligence and intelligent machines, they naturally frame their questions after the popular ideas of science fiction: Frankenstein (Mary Shelley 1818), Rossum’s Universal Robots (Karel Čapek 1921), Three Laws of Robotics (Isaac Asimov 1942 onwards), Multivac (Asimov 1955 onwards), Hitchhiker's Guide to the Galaxy (Douglas Adams 1978 onwards).

• What happens if our creations hate us? Or get depressed.
• What happens if the robots rebel? How can they outwit the constraints we place upon them?*
• Can humans (Susan Calvin, Arthur Dent, Ronald Bakst) outwit the machines?

@DianeCoyle1859 echoes these questions when she asks whether humans could fight back against the superintelligence described by Nick Bostrom

• by unplugging them if they turn on us?
• by removing sensors and RFID tags and so on, to deny them the data they feed upon?

But the analogy that springs to my mind is that disentangling humanity from machine intelligence is likely to be at least as complicated as disentangling the UK economy from the EU. The global economy is dependent on complex cybernetic systems - from algorithmic trading to just-in-time supply chains, automated warehouses, air traffic control, all the troubles of the world. Good luck trying to phase that lot out.


*By the way, it's not that difficult to outwit humans. In a recent study, a raven outsmarted the scientists by inventing her own way of accessing a reward inside a box and was therefore excluded from further tests. And don't get me started on the intelligence of bees.

---

Tim Adams, Artificial intelligence: ‘We’re like children playing with a bomb’ (Guardian, 12 June 2016)

Marc Ambasna-Jones, Are Asimov's laws enough to stop AI stomping humanity? (The Register, 15 Aug 2017)

Isaac Asimov, The Life and Times of Multivac (1975 via Atari Archives) (this is the story featuring Ronald Bakst)

Diane Coyle, Do AIs drive autonomous vehicles? (15 July 2017)

Ian Johnston, Ravens can be better at planning ahead than four-year-old children, study finds (Independent, 13 July 2017)

Wikipedia: All the Troubles of the World, Multivac, Three Laws of Robotics


Update: added link to article by @mambjo 19 August 2017