Hallucinations in LLMs: Technical challenges, systemic risks and AI governance implications

We've come a long way in understanding large language models — but beneath the surface, they're still black boxes wrapped in probabilities and good intentions. The simplified notion that these systems are merely statistical engines predicting the next token masks a complex mechanism with characteristics that are still not fully understood.

Hallucination is one such feature — even the term itself is the subject of ongoing debate.

Regardless of the controversies, hallucination is commonly used to describe instances where an artificial intelligence system generates false, misleading or fabricated information that diverges from the original source content. More than that, it's often seen as a digital typo — something to patch, not ponder.

Not just bugs, signals of computation

Recent studies suggest hallucinations may not be mere bugs, but signatures of how these machines "think." This assertion comes from the idea that LLMs, as computable functions, cannot learn all computable ground truth functions. This inherent limitation ensures that some inputs will inevitably result in imperfect or inaccurate outputs, leading to deviations or "hallucinations."

Furthermore, the very architecture of transformers can be manipulated to produce specific, predefined — and potentially false — tokens by perturbing the input sequence, even with nonsensical prompts. This suggests that the capacity to generate divergent or fabricated information is not just a failure of learning specific data, but a characteristic tied to the model's operational mechanics and its inherent limits in perfectly mapping the vast space of language and knowledge.

Implications for AI governance