Weapons of Math Destruction

In this nonfiction book, data scientist and mathematician Catherine O’Neil’s Weapons of Math Destruction (2016) explores how math-driven models encoded in technology shape many people’s lives and opportunities in the United States. She calls these models weapons of math destruction (WMDs) for their ability to wreak mass havoc on the poor and marginalized peoples of America. This book deals with difficult subject matter, such as socioeconomic oppression, racial discrimination, gender inequality, and discrimination against individuals with mental and physical illnesses. The page numbers in this guide refer to the 2016 paperback version published by Broadway Books.

Summary

O’Neil identifies three components of the classic biased algorithm, or weapon of math destruction: it has scale, opacity, and propensity to cause damage. Algorithms with large scale affect many people and, therefore, have a higher than usual probability of negatively affecting at least some individuals. Opacity refers to an algorithm’s hiddenness, whether because it is proprietary or because the people implementing the algorithm have not learned how to understand the processing of the data they input. Damage refers to an algorithm’s negative impact on humans, whether because it perpetuates entrenched racism or because it is prone to sudden, catastrophic failure that might affect something as critical as a hospital system.

O’Neil rejects the assumption that algorithms are created and perform in a vacuum and are immune to bias. Just because machines are not emotional does not inoculate them against the biases held by their human engineers or by the data sets they select to train the algorithms. As a prime example of a WMD in action, she points to the car insurance policies in Florida. Statistics show that adults with no marks on their driving record who possess bad credit scores pay, on average, thousands of dollars more than individuals convicted of drunk driving who possess great credit scores. She sympathizes with teachers whose communities consider them excellent, who are nonetheless fired because their assessment metrics rate them as subprime educators. The worst feature of these questionable algorithms is that they are not amenable to analysis; that is, the people who lose out, and even the algorithms’ creators, often have no way of figuring out why they were rated so poorly. O’Neil calls such algorithms “black boxes,” borrowing a term from the scientific method that represents a part of a physical process that scientists cannot observe.

O’Neil asserts that WMDs accumulate much of their power steadily, by creating feedback loops. They do this by generating data that falsely confirm their faulty initial assumptions. For example, predictive policing algorithms guide officers to parts of cities where crimes have previously been committed. Since this very crime data is first generated by a criminal justice system that has racist leanings, it disproportionately leads cops to areas where they previously made biased judgments. Over time, the algorithm actually increases rates of criminality in the areas to US it draws police resources simply by increasing police presence there, resulting in arrests for minor offenses that might go unnoticed in other neighborhoods.

Throughout the book, O’Neil examines how WMDs function in nearly every domain of life: education, criminal justice, finances, healthcare, advertising, employment, politics, insurance, and social media. Many of these WMDs start with the same story: They were created for some relatively innocuous purpose, such as trying to manage large subsets of data in an efficient manner. However, the models and their data yielded massive profits. In turn, the objectives of the WMDs usually shift to profits over people. As a result, marginalized communities and individuals who are outliers, whether in terms of exceptional circumstances or deviations from what is considered the mainstream of society, are punished.

While looking at these systems can inspire immense frustration and even helplessness since many of these WMDs lack transparency and accessibility to the people they most exploit, O’Neil urges readers to demand accountability and spark outrage in the media when unfair practices are uncovered. Additionally, she urges individuals who want to do good in the world to be the ones to create new models and challenge the existing ones that don’t work to build a better society for all.