Week10(3/27)
Weapons of Math Destruction. (2016・Cathy O’Neil)
[Introduction]
"What’s more, thanks to the extraordinary powers that I loved so much, math was able to combine with technology to multiply the chaos and misfortune, adding efficiency and scale to systems that I now recognized as flawed."(p.10)"Nevertheless, many of these models encoded human prejudice, misunderstanding, and bias into the software systems that increasingly managed our lives."(p.10)
A case from Washington School district.
IMPACT: an assessment tool to evaluate teacher's work
"So Washington, like many other school systems, would minimize this human bias and pay more attention to scores based on hard results: achievement scores in math and reading."(p.12)
Negligence for numerous human/social factors→WMD often punish 'exceptional' individual.
Statistical systems require feedback.
-But in many cases in the real world, feedback doesn't happen because those voices are underrepresented-
"if the people being evaluated are kept in the dark, the thinking goes, they’ll be less likely to attempt to game the system."(p.14)
"the people running the WMDs don’t dwell on those errors. Their feedback is money, which is also their incentive."(p.18)
[Key terms]
Poisonous assumptions:One misleaded assumption that model morphs into abelief.No adjustment for our models
[Ch.1 BOMB PARTS What Is a Model?]
Transparent data vs opaque data"To create a model, then, we make choices about what’s important enough to include, simplifying the world into a toy version that can be easily understood and from which we can infer important facts and actions."(p.24)
Risk of too much simplification in the model.
"Our own values and desires influence our choices, from the data we choose to collect to the questions we ask. Models are opinions embedded in mathematics."(p.25)
No models are purely objective
"whether we’ve eliminated human bias or simply camouflaged it with technology. The new recidivism models are complicated and mathematical. But embedded within these models are a host of assumptions, some of them prejudicial."(p.28)
"But infact the model itself contributes to a toxic cycle and helps to sustain it. That’s a signature quality of a WMD."(p.30)
WMD elements:
Opacity/Scale/Damage
→Leap from one to another
[Key terms]
Model:-abstract representation of some process
-Whether it’s running in a computer program or in our head, the model takes what we know and uses it to predict responses in various situations. All of us carry thousands of models in our heads. They tell us what to expect, and they guide our decisions.-(23)
[Ch.2 SHELL SHOCKED My Journey of Disillusionment]
Structural source of fraud
"Their only glimpse of what lurked inside came from analyst ratings. And these analysts collected fees from the very companies whose products they were rating. Mortgage-backed securities, needless to say, were an ideal platform for fraud."(p.40)
Two false assumptions supported the system:
1. They are hedge in the risk because algorithms were made by 'specialits'
" Its purpose was only to optimize short-term profits for the sellers. And those sellers trusted that they’d manage to unload the securities before they exploded."(p.41)
2."The second false assumption was that not many people would default at the same time."(p.41)
"The math could multiply the horseshit, but it could not decipher it. This was a job for human beings."(p.43)
[Ch.4 PROPAGANDA MACHINE Online Advertising]
Targeted ad for vulnerables.-Vulnerability is worth gold-Bayesian analysis:
Rank the variables with the most impact on the desired outcome.
[Ch.5 CIVILIAN CASUALITIES Justice in the Age of Big Data]
Correlation between crime and poverty-Crime by rich?
"Cramping down on white-collar crime would require people with different tools and skills"(p.81)
Choices about the direction of attention
"While looking at WMDs, we ’re often faced with a choice between fairness and efficacy. Our legal traditions lean strongly toward fairness."(p.84)
"The question is whether we as a society are willing to sacrifice a bit of efficiency in the interest of fairness. Should we handicap the models, leaving certain data out?"(p.85)
"All too often they use data to justify the workings of the system but not to question or improve the system."(p.87)
"So instead of analyzing prisons and optimizing them, we deal with them as black boxes. Prisoners go in and disappear from our view."(p.88)
"Now the police have their eye on him. And if he behaves foolishly, as millions of other Americans do on a regular basis, if he buys drugs or gets into a bar room fight or carries an unregistered handgun, the full force of the law will fall down on him, and probably much harder than it would on most of us. After all, he’s been warned."(91)
[Question]
1. Vivid cases from chapter 1&4 reminded me of the ethical use of cutting edge technology.Recently I hear some discussions from tech industry that engineers raising their voices for fair use of technology.
I agree with O'Neil that transparency is a big issue.
If we, consumers can expect farmers for organic food or fair trade,
is it realistic for developers/engineers to assert their right for fair use of their products?
2.Reliability for 'assmption'.
Social norm in the system reinforce one's range of interpretation, and this would consequently lead to status quo(≒assumption)
The assumption reinforced through:
-'Scientific' data
-Daily conversation
-Experiences among a limited group
But how we can get rid of our assumptions?
Most of us never learn until we face a disaster that drastically changes our value.