(Image: David Lemm)
Software is deciding who gets a loan, who counts as a citizen and what prices you pay online. Who will step in when the machines get out of hand?
“AMAZON is all kinds of broken.” If you caught on 12 December last year, and were quick, you might have grabbed some exceptional bargains. For an hour only, Amazon was selling an odd mix of items – cellphones, video games, fancy-dress costumes, mattresses – for one penny.
Advertisement
. Goods usually marked £100 went for a 99.99 per cent discount. Hundreds of customers leapt at the chance, often buying in bulk. Even though Amazon reacted quickly and cancelled many orders, they were unable to recall those that their automated system had already dispatched from warehouses. Once set in motion, the process was hard to stop. Thanks to a software glitch, a handful of independent traders using Amazon’s Marketplace lost stock worth tens of thousands of dollars. Some faced bankruptcy.
. Most of the time they get on with business out of sight and out of mind. And business is booming. Automated processes are no longer simply tools at our disposal: they often make the decisions themselves. Much of the news we read, the music we listen to and the products we buy are served up automatically, based on statistical guesswork about what we want. Invisible chaperones shape our online experiences. Systems we can’t examine and don’t understand determine the route we take to work, the rates we get for mortgages, and the price we see for airfares.
“Algorithms are no longer simply tools: they often make the decisions”
Many are proprietary and all are complex, pushing them beyond public scrutiny. How can we be sure they’re playing fair? A new wave of algorithm auditors are on the case, intent on pulling back the curtain on the hidden workings and hunting for undue bias or discrimination. But is this the fix? Do algorithms need better policing, or must we accept their nature as a price we pay for our automated world?
There’s nothing inherently mysterious about them: an algorithm is simply a set of instructions for getting something done. The trouble is that algorithms get nested inside or bolted on to others, interacting in ever more complex ways. It can also be hard to predict how algorithms will behave with real-world data once released into the wild.
The scope of their influence is often unclear. Some people swear blind that they’ve seen , for example. Others think that’s bunk, an urban myth for our times. Such debates highlight the shadowy nature of today’s systems.
Not only are most algorithms secret recipes, sometimes even the developers who wrote them are in the dark. When at Northeastern University in Boston, Massachusetts, looked closely at how many of us have our search results skewed by factors like location and browsing history, she noted things even Google didn’t know: for example, that around 12 per cent of searches get personalised. Google engineers thanked her. They’d never made such measurements and hadn’t known the exact impact of their personalisation algorithms.
Exposing hidden algorithms can cause outrage. That’s what and his colleagues at the University of Michigan, Ann Arbor, found when they lifted the lid on Facebook’s newsfeed algorithms, which decide which posts from friends and family we actually see. The team compared filtered and unfiltered feeds and found that Facebook’s algorithms hid posts deemed uninteresting, according to unspecified criteria.
Around two-thirds of the participants in Sandvig’s study had no idea that algorithms were deciding what they saw. Many were shocked and upset when posts from close friends or family were excluded. Some had been blaming themselves or their friends for the algorithms’ work. “If you post something and it doesn’t get any comments or likes, people assume that either their friends don’t like the topic, or their friends don’t like them,” says Sandvig.
Even for news, it’s a popularity contest. During last year’s Ferguson riots in Missouri, for example, because these had hundreds of thousands of likes.
What Sandvig’s team did for Facebook, Hannak and her colleagues are doing for other online activity. Hannak is interested in how algorithms can tailor prices to different shoppers. In one , the researchers looked at how online retailers such as Walmart, Office Depot and Expedia varied prices according to factors including a user’s choice of browser, operating system and purchase history.
They found many instances of what they consider price discrimination though they are not sure of the rationale. Often the difference was small. Android users, for example, saw higher prices on about 6 per cent of items, though only by a few cents. In other cases, price quotes varied by up to $100. The greatest differences were typically seen between users who were logged in to a site and those who were not.
Crash damage
Hannak’s group now wants to understand exactly how location influences search results. They are simulating hundreds of Android phones and spreading them across Ohio using faked GPS coordinates. They’ll also be looking to see whether people from rich and poor neighbourhoods get different search results when hunting for financial services.
Evidence of that may already have come to light. Some think hidden algorithms played a part in the 2008 sub-prime mortgage crash. Between 2000 and 2007, US lenders like Countrywide Home Loans and DeepGreen doled out home loans at an unprecedented rate via automated online applications. “Everyone was saying what a great innovation it was,” says of Northern Iowa in Cedar Falls. “Everyone was very high on these fast web-based loans. No one anticipated the problem.”
“Everyone was high on fast web-based loans. no one anticipated the Problem”
The problem was granting so many high-risk loans without human oversight. Americans from minority groups suffered most in the resulting crash. Automated processes crunched through vast amounts of data to identify high-risk borrowers – who are charged higher interest rates – and targeted them to sell mortgages. “Those borrowers turned out to be disproportionately African American and Latino,” says of the , a public policy think tank based in Washington DC. “Algorithms played a role in that process.”
The exact degree to which algorithms were to blame remains unclear. But settled with several cities, including Baltimore, Chicago, Los Angeles and Philadelphia, for hundreds of millions of dollars over claims that their sub-prime lending had disproportionately affected minorities. Although the decision-making process big banks used to target and sell sub-prime loans may not have been new in itself, the reach and speed of those decisions when algorithms were the driving force was new. “It’s the scale factor,” says Gangadharan. “This was a problem that affected many people in the US and we have seen the effects fall along race and class lines in devastating ways.”
Automated systems are replacing human discretion in ever more important decisions. In 2012, the US State Department started using an algorithm to randomly select the winners of the green card lottery. The system was buggy, however: it awarded visas only to people who applied on the first day, says , a Princeton University computer scientist who is investigating the event. Those visas were rescinded, but it’s a good example of how hidden algorithms can have a life-changing effect.
In a similar example, the documents leaked by Edward Snowden revealed that the National Security Agency uses algorithms to decide whether a person is a US citizen. According to US law, only non-citizens can have their communications monitored without a warrant. In the absence of information about an individual’s birthplace or parents’ citizenship, . Is this person in contact with foreigners? Do they appear to have accessed the internet from a foreign country? Depending on what you do online, your citizenship might change overnight. “One day you might be a citizen, another you might be a foreigner,” says , at the University of Michigan in Ann Arbor. “It’s a categorical assessment based on an interpretation of your data, not your passport or your birth certificate.”
Algorithms are also used to police voter fraud. Several US states use software called Crosscheck to . But people have been deleted simply for having the same name. As with the sub-prime algorithms, minorities are again hit hardest. The names it scrubs are disproportionately those of black, Asian and Hispanic voters, who are more likely to share names – such as Jackson, Kim or Garcia.
The next scandal may be prison sentencing. Judges and lawyers in Missouri can use a website to make an . The system calculates incarceration costs for defendants, and weighs that against the likelihood the defendant will reoffend, based on prior criminal history and behavioural and demographic factors. Some think this will lead to minorities being given harsher sentences. Proxies like address, income and education level make it almost impossible to avoid racial bias. Similar systems are appearing across the US. “I think it’s terrifying,” says , a computer scientist at Haverford College in Pennsylvania.
The scales are falling from our eyes as the impact of algorithms is felt in almost every area of our lives. What should we do about it? In many of these examples, the problem is not the algorithms themselves, but the fact that they over-amplify an existing bias in the data.
Higher standards
“People who work with algorithms are comfortable with the idea that they might produce these unintended results,” says Sandvig. But for a growing number of people, that’s not good enough. , who works with Hannak at Northeastern University, thinks that large technology companies like Google and Facebook ought to be considered as public services that huge numbers of people rely on. “Given that they have a billion eyeballs, I think they have a responsibility to hold themselves to a higher standard,” he says.
Wilson thinks that automated systems might be made more trustworthy if users can control exactly how their results are personalised – such as leaving gender out of the equation or ignoring income bracket and address. It would also help us learn how these systems work, he says.
Others are calling for a new regulatory framework governing algorithms, much like we have for the financial industry, for example. A recent recommends that policy-makers pay more attention to what the algorithms do with the data they collect and analyse. To ensure accountability, however, there would need to be independent auditors who inspect algorithms and monitor their impact. We cannot leave it to governments or industry alone to respond to the problems, says Gangadharan.
“The big question now for me is who are the watchdogs,” says Sandvig. For now, he suggests it should be the researchers who are beginning to reveal algorithms’ broader effects. Wilson, for example, is looking into setting up dummy credit profiles to better understand price-fixing systems. But independent auditors face tough obstacles. For a start, digging around inside many automated services violates their terms of use agreement, which prohibits attempts to analyse how they work. Under the US , such snooping may even be illegal. And while public scrutiny is important, the details of proprietary algorithms need to be kept safe from competitors or hackers, for example.
What’s more, most automated systems are too complex for humans to inspect by hand. So some researchers have developed algorithms that check other algorithms. Kroll is working on a system that would let an auditor verify that an algorithm did what it was supposed to with what it was given. In other words, it would provide a foolproof way of checking that the outcome of the green card lottery, for example, was in fact random. Or that a driverless car’s algorithm for avoiding pedestrians treats both people walking and people in wheelchairs with the same caution.
Friedler has a different approach. By understanding the biases inherent in the underlying data, she hopes to eliminate bias in the algorithm. Her system looks for correlations between arbitrary properties – like height or address – and demographic groupings like race or gender. If the correlation is expected to lead to unwanted bias, then it would make sense to normalise the data. It is essentially affirmative action for algorithms, she says.
That’s fine for cases where discrimination is clear, where a system is found to be unfair or illegal. But what if there is disagreement about how an algorithm ought to behave? Many would say Facebook’s filtering of its newsfeed keeps it readable. Some would argue that highly personalised price adjustment can benefit both customers and retailers. What’s acceptable to some won’t be for others.
As Sandvig notes, unlike for financial systems, there are no standards of practice governing algorithms. But how we want them to behave may turn out to be a harder question for society to answer than we think. Maybe we’ll need an algorithm for that.
“How we want algorithms to behave may be a harder question to answer than we think”
Leader: “Living with the algorithms that run our lives“
This article appeared in print under the headline “No one in control”
