We came across a tweet today promoting a Salon interview with Cathy O’Neil, author of the recent book “Weapons of Math Destruction," which chronicles how the biases written into algorithms distort society and people’s lives.
We’ve not read the book (yet), but were fascinated all the same by tales told from a data scientist, blogger, and former Wall Street quant and online marketing analyst, who left academia in 2007 to join the world of banking and finance. “[J]ust in the nick of time to get a front-row seat to the financial crisis,” O’Neill writes.
She finds that much of the mathematics and analytics in finance are used to create Big Lies, so she leaves for the fresh potential of startups, hoping to change the world….but instead finds the power of data she was working with became “kind of a weaponized mathematical algorithm...[targeting] poor people, who are being preyed upon.”
In one insightful part of the interview, O’Neill shares a true horror tale of trying to access the source code for New York’s “Value Added Model” program that grades teachers. She filed a Freedom of Information Act request to get it, but “was told I couldn’t [because] New York City had signed a contract with this place called VARK in Madison, Wisconsin” which kept the source code private -- even from the city and its dept. of education!
“Here are some scores, we’re not gonna explain them to you, but you should trust them. And by the way you can’t appeal them and you will not be given explanations for how to get better,” O’Neill explains. “It’s like you’re being put into a cult, but you don’t actually believe in it.”
Great interview, terrific article. Please consider reading it!