When AI hallucinates facts about individuals: Personal data and accuracy under the GDPR

A generative artificial intelligence system can confidently say something about a person that is entirely false. This can lead to legal disputes under data protection law. 

In March 2025, NOYB filed a complaint with Norway's data protection authority, Datatilsynet, against OpenAI after ChatGPT generated an output that claimed a Norwegian man had murdered two of his children and had been sentenced to prison. The information was fabricated, but it mentioned the man's full name and included elements that resembled his real life, thereby, identifying him. The complaint is now being handled by Ireland's Data Protection Commission.

The growing number of legal disputes involving AI hallucinations shows that hallucinated outputs can have real-world consequences while simultaneously exposing providers and deployers of generative AI systems to significant legal risk. 

AI hallucinations — any instance in which an AI system generates new, factually incorrect information — are not reproductions of incorrect training data. Rather, the model generated entirely new information and content. This ability makes it valuable for many users and is based on the technical architecture. It is not a "bug" but an inherent feature of probabilistic models trained to produce coherent language.

In its complaint, NOYB alleged a violation of the EU General Data Protection Regulation's accuracy principle. If an AI system produces a false statement about a real person, many privacy lawyers, too, will jump to questions like: Is this output violating the accuracy principle under Article 5(1)(d) of the GDPR? Can the data subject request correction, deletion or even damages?

But the harder question comes one step earlier: Is every AI hallucinated statement about an identifiable person actually personal data?