Model drift, data leaks and deepfakes: Rethinking AI governance in the age of autonomous risk

Artificial intelligence systems have rapidly transitioned from the lab to core business operations — bringing their associated risks along with them.

Not long ago, AI governance conversations centered on checklists for ethics, bias and model transparency. In 2025, governance teams find themselves confronted with real-time incidents that resemble those of cybersecurity or crisis management.

From AI models unpredictably drifting off course to employees inadvertently leaking data into chatbots, and even deepfakes duping staff — these risks are no longer hypothetical. They're happening, often in highly regulated environments where they were least expected.

The new wave of AI incidents

Consider a few examples that have prompted executives to rethink AI oversight. Early this year, an employee at a financial firm was tricked by a compelling deepfake video call impersonating senior management; the victim of the scam ended up wiring USD25 million to criminals before anyone realized the "boss" on the call wasn't real.

Around the same time, a leading tech company discovered its engineers unknowingly uploaded sensitive source code to an online AI chatbot, which prompted an immediate ban on employees using generative AI tools until proper safeguards were in place.

In a widely publicized prank, customers manipulated a car dealership's ChatGPT-powered assistant into "agreeing" to sell a more than USD60,000 SUV for USD1, exposing how easily a clever prompt can bypass an AI's intended rules. While no actual Chevy was sold for USD1, the incident forced the dealership — and its software vendor — to shut down the bot and acknowledge the need for stricter controls on AI interactions.