Privacy by proxy: Regulating inferred identities in AI systems

As artificial intelligence tools permeate more aspects of daily life, they often draw conclusions about people who have never directly interacted with them. Imagine a smart home assistant that notices someone's spouse arriving late each night and, without ever "knowing" them, starts recommending health supplements for insomnia.

This kind of "personalization by proxy" raises thorny questions: whose data is this and do the usual privacy rules even apply? In a world driven by ambient and relational data, privacy may no longer be solely an individual concern.

Traditional laws like the EU General Data Protection Regulation, California Consumer Privacy Act, and Colorado Privacy Act are built on individual rights, but AI inferences about nonusers strain these frameworks. Existing legal definitions of personal data and consent often struggle to account for inferences generated by AI, raising the question of whether privacy is evolving from an individual right toward a more collective understanding.

Inferred identities and expanding definitions of personal data

The GDPR expansively defines personal data as "any information which are related to an identified or identifiable natural person." In practice, this covers everything from names and email addresses to health data and religion, even when only inferred.

For example, regulators emphasize that if an AI system predicts someone's health condition from their shopping habits, those predictions must be treated as health data under the GDPR. The CCPA likewise defines personal information broadly and explicitly includes "inferences drawn from any of the information identified … to create a profile about a consumer reflecting the consumer's preferences, characteristics…"