OneTrust_Square Banner_300x250_DD_ROS_01_19

Microsoft researcher Kate Crawford got some press at this time last year with her “six myths of big data,” among which was this item: “Big Data Doesn’t Discriminate.”

Over the past year, this idea of big data analytics and its potential for discriminatory harm has only gained steam. Our own resident VP of Research and Education, Omer Tene, published with Future of Privacy Forum ED Jules Polonetsky about the topic in a paper called “Judged by the Tin Man”; Michael Schrage wrote about it for Harvard Business Review, and the White House reports on big data highlighted potential discrimination as an issue to watch, among numerous other examples.

Perhaps it’s not surprising then that the two IAPP prize-winning papers at this year’s recently completed Privacy Law Scholars Conference both deal with big data, and the increasingly complicated systems that employ it.

Solon Barocas and Andrew Selbst’s paper, “Big Data’s Disparate Impact,” still in draft form, examines discrimination law in the U.S. and whether it can adequately handle the issues raised by big data. Danielle Citron and Frank Pasquale’s “The Scored Society,” published in Washington Law Review, looks at the legal complications created by systems using data and algorithms to include and exclude people from various programs. (To read about last year’s winners, click here.)

Selbst, currently with Public Citizen and about to embark on a clerkship with a judge on the third circuit, said the idea for their paper arose from a conversation he had on a New York City street with Barocas, currently with the Center for Information Technology Policy at Princeton.

With an interest in how new technology changes previous understandings of civil liberties, Selbst was primed to engage with Barocas’ dissertation work on the effects of big data on populations. “He was discovering that there are all these unintentional ways that discrimination could creep in that people wouldn’t think of right away, and I just said, ‘I think this breaks anti-discrimination law. I’m not sure the law can possibly handle that.’”

Barocas said he’s been working on big data’s indirect impacts since his master’s work in 2004, and then continued with his dissertation to look into data analysis, machine learning and the work scientists have been doing on non-discriminatory data mining models. “A lot of my work now is to translate these technical details into policy and philosophy,” he said. “It’s a really rich area.”

Working with Selbst, he said, “brought my insights to a more legal analysis.”

And all the activity surrounding the issues, including the White House reports, has been “really encouraging,” Barocas said. “We happened to finish our paper at the exact right moment.”

Really, the only question remaining is this: “We’re not sure how truly pessimistic to be,” Selbst said. In their initial thinking, they were fairly pessimistic, Selbst said, but “I have come to believe there is more reason for optimism, and we won’t know where we come down” until the paper is finished later this summer, incorporating feedback from PLSC and elsewhere.

Citron and Pasquale, for their parts, are fairly pessimistic, if not dire. “Individuals should be granted meaningful opportunities to challenge adverse decisions based on scores miscategorizing them,” they write. “Without such protections in place, systems could launder biased and arbitrary data into powerfully stigmatizing scores.”

The pair work together teaching law at the University of Maryland, where both are doing work on “black boxes,” those closed systems where data goes in and a decision comes out and it’s unclear, or certainly opaque, just how that decision was arrived at.

Citron wrote a paper exploring this issue first in 2007, “Technological Due Process,” which focused on a health and human services system in Colorado that was “a disaster.” Programmers coded in bad policy, with incorrect decision tables, and people were denied benefits like Medicaid and food stamps.

With no programmer notes, crashing systems, no audit trails, “there was no way to trace why a decision was made,” Citron said. “It was a failure of due process, with no chance to be heard.” Essentially, it was unauthorized rulemaking, with programmers doing real-life damage without even knowing it.

The question she raised was this: How do we update our understanding of due process for the 21st century?

So, when Washington Law Review asked her to write about artificial intelligence and the law, she naturally thought of Pasquale, with whom she had previously examined U.S. federal “fusion centers” and who was exploring these black box issues.

They decided to turn their gaze toward credit-scoring systems, combining Citron’s ideas about due process with Pasquale’s ideas about lack of insight into systems. “There are hundreds of ways that entities are scoring us in ways that we find very troubling,” she said.

“She was the person who got me into privacy law,” Pasquale said. “I’d never really written in the area until we worked on fusion centers.”

Instead, his black box work started out with his examination of Google’s search algorithms back in 2005/2006. People were complaining about their place in search results, and saying it was unfair. “Google would always say, ‘It’s about the quality of the user experience,’” Pasquale said, “and then I found that nobody could really get to the bottom of it. People just believed whatever Google said.” Back then, “people said, ‘You’re out of your mind. They’re not that powerful. Why are you even talking about this?’” Pasquale said. Now his work seems a little more relevant to folks.

Then he turned his eye to credit scores, which was particularly relevant during the housing crisis: “It seemed that so many times the credit score was setting up unfair games for people.”

Such is the nature of their paper. It’s particularly interesting to see the issue in the context of the original credit bureaus, Pasquale said, which would report on things like “effeminate gestures” or a “messy yard.” FCRA was meant to stop this kind of “disgusting insinuating innuendos,” he said. It was supposed to create a more scientific model for credit evaluation. In the process, however, it created a more opaque box.

Unfortunately, Citron and Pasquale will not be available to present their paper at the IAPP Privacy Academy in San Jose this fall (Citron is on book tour; Pasquale is hosting a big data conference of his own at UMaryland), but Barocas will be on hand for the event, speaking solo because of Selbst’s clerkship.

Written By

Sam Pfeifle


If you want to comment on this post, you need to login.


Board of Directors

See the esteemed group of leaders shaping the future of the IAPP.

Contact Us

Need someone to talk to? We’re here for you.

IAPP Staff

Looking for someone specific? Visit the staff directory.

Learn more about the IAPP»

Daily Dashboard

The day’s top stories from around the world

Privacy Perspectives

Where the real conversations in privacy happen

The Privacy Advisor

Original reporting and feature articles on the latest privacy developments

Privacy Tracker

Alerts and legal analysis of legislative trends

Privacy Tech

Exploring the technology of privacy

Canada Dashboard Digest

A roundup of the top Canadian privacy news

Europe Data Protection Digest

A roundup of the top European data protection news

Asia-Pacific Dashboard Digest

A roundup of the top privacy news from the Asia-Pacific region

Latin America Dashboard Digest

A roundup of the top privacy news from Latin America

IAPP Westin Research Center

Original works. Groundbreaking research. Emerging scholars.

Get more News »

IAPP Communities

Meet locally with privacy pros, dive deep into specialized topics or connect over common interests. Find your Community in KnowledgeNet Chapters, Sections and Affinity Groups.

IAPP Job Board

Looking for a new challenge, or need to hire your next privacy pro? The IAPP Job Board is the answer.

Join the Privacy List

Have ideas? Need advice? Subscribe to the Privacy List. It’s crowdsourcing, with an exceptional crowd.

Find a KnowledgeNet Chapter Near You

Talk privacy and network with local members at IAPP KnowledgeNet Chapter meetings, taking place worldwide.

Find more ways to Connect »

Find a Privacy Training Class

Two-day privacy training classes are held around the world. See the complete schedule now.

Privacy Core® e-learning Library Expands Again

Two innovative additions to our Privacy Awareness curriculum coming in April: Recognizing and Avoiding Social Engineering and Identifying Phishing Attacks.

Online Privacy Training

Build your knowledge. The privacy know-how you need is just a click away.

Upcoming Web Conferences

See our list of upcoming web conferences. Just log on, listen in and learn!

Train Your Team

Get your team up to speed on privacy by bringing IAPP training to your organization.

Let’s Get You DPO Ready

There’s no better time to train than right now! We have all the resources you need to meet the challenges of the GDPR.

Learn more »

CIPP Certification

The global standard for the go-to person for privacy laws, regulations and frameworks

CIPM Certification

The first and only privacy certification for professionals who manage day-to-day operations

CIPT Certification

The industry benchmark for IT professionals worldwide to validate their knowledge of privacy requirements

FIP Designation

Recognizing the advanced knowledge and issue-spotting skills a privacy pro must attain in today’s complex world of data privacy.

Certify Your Staff

Find out how you can bring the world’s only globally recognized privacy certification to a group in your organization.


The IAPP’S CIPP/E and CIPM are the ANSI/ISO-accredited, industry-recognized combination for DPO readiness. Learn more today.

Learn more about IAPP certification »

IAPP-OneTrust Website Scanning & Cookie Compliance Tool

Scan your website for cookies, tags, forms and policies and create a custom, dynamically updated cookie policy based on the results of your scans.

Are You Ready for the GDPR?

Check out the IAPP's EU Data Protection Reform page for all the tools and resources you need.

Privacy Vendor List

Find a privacy vendor to meet your needs with our filterable list of global service providers.

IAPP Communities

Meet locally with privacy pros, dive deep into specialized topics or connect over common interests. Find your Community in KnowledgeNet Chapters, Sections and Affinity Groups.

More Resources »

Global Privacy Summit 2017

What an amazing Summit! Looking for session presentations? Click through to the webpage and look in the session's description for a link to view slides.

Canada Privacy Symposium 2017

Early Bird discounts may be gone, but not your chance to catch this year's stellar lineup! Register today.

Asia Privacy Forum 2017

Join us in Singapore for exclusive networking and intensive education on data protection trends and challenges in the Asia Pacific region.

Privacy. Security. Risk. 2017

We're bringing the best of the best in privacy and infosecurity to sunny San Diego. Early registration for P.S.R. opens in May.

Europe Data Protection Congress 2017

Your source for European policy debate, multi-level strategic thinking and thought-provoking discussion. Registration opens in early June.

Sponsor an Event

Increase visibility for your organization—check out sponsorship opportunities today.

More Conferences »

Become a Member

Start taking advantage of the many IAPP member benefits today

Corporate Members

See our list of high-profile corporate members—and find out why you should become one, too

Renew Your Membership

Don’t miss out for a minute—continue accessing your benefits

Join the IAPP»