At a hearing earlier this week, European lawmakers urged the European Commission to use the full extent of its powers under the Digital Services Act to go after social platform X's Grok while pushing for a ban of apps that allow the creation of deepfaked, naked images.
The Internal Market and Consumer Protection hearing 26 Jan. came as regulators are upping their scrutiny of the Grok artificial intelligence chatbot's ability to generate undressed images of real people.
The European Commission announced 25 Jan. that it would investigate X under the DSA.
Deepfakes are an AI safety issue on which there seems to be international agreement. The U.S. banned them last year under the Take It Down Act and pressure has been mounting around the globe to restrict them. The U.K. sped up a portion of its Data (Use and Access) Act in mid-January criminalizing the creation of deepfakes. Regulators in Brazil, Canada, Hong Kong and the U.S. have all either expressed concern or opened probes into the platform. Those efforts continue despite Grok saying it has banned deepfakes of real people and limited the creation of images of naked people to paid subscribers.
The tension in the EU is whether apps that create naked images, sometimes called "nudification apps," can be outright banned under current digital laws.
"Is this covered in the existing legislation, or do we need to go further?" asked Slovakian MEP Veronika Cifrová Ostrihoňová. "I think we do, and actually ban these systems that make this possible, if we want to be speedy and strict in the future, and to have actual tools to act against the tech bros."
Executive Vice-President for Tech Sovereignty, Security and Democracy Henna Virkkunen said the Grok probe will look at whether the company took sufficient steps to prevent illegal content from being disseminated on its platform, including child sexual abuse material.
"Sexual deepfakes of women and children are a violent, unacceptable form of degradation. With this investigation, we will determine whether X has met its legal obligations under the DSA, or whether it treated rights of European citizens — including those of women and children — as collateral damage of its service," she said.
More than 50 lawmakers sent a letter to the Commission confirming whether AI apps that create nudity are banned under the DSA prior to the probe's announcement, Euractiv reports. And Parliament created a group dedicated to exploring legislation to do so if it is not, with Irish MEP Michael McNamara appointed lead rapporteur.
Representatives from the Commission's communications and digital technology departments stressed they would move quickly on the investigation and use their powers to leverage enforcement if warranted. But there are limits on what they can do under current law.
Head of the Digital Services and Platforms unit at DG Connect Prabhat Agarwal said his agency must abide by a strict framework for investigations, which is a substantial part of the DSA's text. Investigations can be lengthy because the Commission faces intense pushback from companies and their lawyers to prove whether they are operating within the correct framework for investigations, he said.
The Commission has learned ways to move investigations along more quickly, Agarwal said, but he could not promise when the Grok probe would be finished.
"We are challenged, not on the substance of whether Grok was generating illegal content, or whether this risk mitigation is effective or not, or whether this content is illegal, or whether this behavior is acceptable or not," he said. "It is whether we have used the right procedural vehicles in doing this."
The Commission had taken actions prior to its investigation, including asking Grok to put mitigations in place to reduce the number of naked images being produced, Agarwal said. Those efforts build on prior engagement after Grok made comments about the Holocaust, which spurred an investigation from the French government.
But the DSA only regulates the systems and processes of platforms, not individual content. If there are specific images that violate national laws, "then the fastest tool is for national authorities to send removal orders under Article 9 of the DSA, and these orders can be actioned within hours," Agarwal said.
As to whether nudifying apps can be banned outright, Agarwal said the Commission asked Google Play Store and Apple App Store on their policies about offering such apps, saying those uses are considered a systemic risk. But as to whether they are banned outright, that may fall under the AI Act's provisions, he said.
If so, that may be a tricky proposition for regulators. Laura Caroli, who helped negotiate the AI Act and was a policy advisor to co-rapporteur Brando Benifei, said the idea of an outright ban was never envisioned in the law.
"Bans were considered almost toxic, especially by the Member States, and it would have taken a massive cross-country scandal to even have them consider the possibility of going that far. Everybody was treading very carefully around bans, and focusing on the most pressing at the time," she wrote.
Caitlin Andrews is a staff writer for the IAPP.
