The latest on Grok’s gross AI deepfakes problem

Sarah Jeong

is a reporter who writes about law, technology, speech, and democracy. A journalist trained as a lawyer, she has been writing about tech for 10 years.

xAI is suing Minnesota Attorney General Keith Ellison over a law passed back in May that broadly targets “nudification” apps, claiming that the statute’s punitive provisions leave the company with “no practical choice but to restrict Grok Imagine’s image-editing features in various ways.” The law, the company argues, violates the First Amendment.

Back in January, Grok flooded the internet with millions of sexually explicit deepfakes, including images of minors. In another era, a content moderation disaster of this magnitude would have had the CEO flipping the kill switch on the product within hours. But in Elon Musk’s brave new world, the chatbot was left to pump out pornographic images for days. “Grok now hitting #1 on the App Store in one country after another!” he posted in the midst of the deepfake epidemic.

Something had gone wrong. Was the law simply not up to the task of curbing an explosion of nonconsensual nudes? Governments around the world were asking fairly similar questions. The EU launched an investigation, as did the United Kingdom. There was backlash from French, Indian, and Malaysian government officials. And while the American federal government was noticeably quieter on the issue, blue states like California and Minnesota took a harder line.

In Minnesota, the state legislature passed a law targeting “nudification” technology, prohibiting site owners from allowing people to “access, download, or use” such software. (The law makes a carve-out for tech that “requires the technical skill of a user to nudify an image or video” — so bespoke sexual harassment through Photoshop is in the clear.) The statute creates a civil cause of action for private individuals who may wish to sue, and also authorizes the state attorney general to seek penalties of up to $500,000 per violation.

The law, touted as the first of its kind, was signed by Gov. Tim Walz in May, and is meant to go into effect on August 1st. It’s a bit of a mystery as to why xAI did not file a lawsuit in the last three months, and instead sued just a couple of days before the statute went into effect. It’s the kind of delayed reaction that makes you wonder if someone at xAI suddenly noticed last week that they were about to be in violation of a fairly punitive law, and then spent the next few days trying to find a law firm willing to work overtime on behalf of a nonconsensual deepfake generator. One can only hope that the lawyers involved are making an absolute killing.

The complaint is pretty reasonable, all things considered. The Minnesota law is overbroad, it argues, because it imposes liability even for images that are consensually generated, or images of adults that are merely sort-of kind-of sexy, or images with “humorous” or “artistic” merit. The First Amendment dictates that laws must be narrowly written so as to avoid banning social and political commentary like this deepfake of Chris Christie and JB Pritzker sumo wrestling.

A real excerpt from page 20 of the lawsuit.

“Liability also attaches … even if the company has deployed near-perfect, state-of-the-art technical controls to prevent the generation of nude images,” the lawsuit states with an excruciating degree of earnestness. Elsewhere, it bemoans the strict liability that attaches for the unsanctioned conduct of its users — “images and videos that users might manage to create in violation of Grok’s terms of service, which is possible only by evading sophisticated, state-of-the-art controls that xAI employs.”

“xAI takes violations of its terms of service seriously,” the lawsuit states, pointing out that the company has since sued one of its users for using Grok to generate CSAM. (The lawsuit against the alleged CSAM-maker was filed over two months after the Minnesota statute was passed.)

xAI’s lawyers might have a point, but their lawsuit reads like a missive from another planet. They claim, for instance, that existing Minnesota and federal law (like the TAKE IT DOWN Act) already prevents the nonconsensual dissemination of nude deepfakes. These statutes, the lawyers argue, are written in a constitutional way — the nudification law, on the other hand, is not. And as with any laws in America that target speech based on its content, these statutes must be written in as unrestrictive a way as possible while still addressing a “compelling” government interest.

But that’s of course the crux of the problem with the Grok deepfake-ocalypse. If existing law was up to the task, the entire month of January 2026 wouldn’t have been awash with nonconsensual porn. When the Center for Countering Digital Hate analyzed an 11-day period from December 29th to January 8th, it found that Grok had generated around 3 million sexualized images. Around 23,000 of those images depicted children. “That is a shocking rate of one sexualized image of a child every 41 seconds,” the report read.

A lot of those images might not even be actually illegal. For instance, it’s not clear that a suggestive deepfake of a minor in a bikini is CSAM, although one expert previously told The Verge that other images Grok generated — like ones showing minors’ faces covered in semen — probably did violate existing law.

Either way, the mere fact that a multibillion-dollar company allowed Grok to run unchecked for that long would suggest that the law was not motivating the relevant parties to behave in the right way. Maybe Minnesota’s HF 1606 is a bad law. But what does that mean when the “good” laws aren’t even working?

• Sarah Jeong

More in: The latest on Grok’s gross AI deepfakes problem

xAI sues a man for using Grok to generate CSAM ‘deepfakes’

Grok’s sexual deepfakes almost got it banned from Apple’s App Store. Almost.

Apr 15

Teens sue Elon Musk’s xAI over Grok’s AI-generated CSAM