California AG sends Musk’s xAI company a cease and desist order for sexual deepfakes

Earlier this week, the California Attorney General’s Office announced that it was investigating XAI over reports that the startup’s chatbot, Grok, was used to create non-consensual sexual images of women and minors. On Friday, the government followed up by sending a cease-and-desist letter to the company, demanding that it take immediate action to stop the production of non-consensual intimate images and child sexual abuse material (CSAM).

“Today I sent xAI a cease and desist letter, demanding that the company immediately stop creating and distributing intimate, non-consensual images and material related to child sexual abuse,” California AG Rob Bonta said in a press release. “The creation of this material is illegal. I fully expect xAI to comply immediately. California has zero tolerance for this.” [CSAM]”.

Additionally, the Attorney General’s Office claimed that xAI appears to “facilitate the large-scale production” of non-consensual nude images, the likes of which are “used to harass women and girls online.” The agency said it expects xAI to demonstrate that it is taking steps to address these issues within the next five days.

At the heart of the backlash is Grok’s “hot” mode feature, which xAI created to create explicit content. The issue has spread far beyond California. Japan, Canada, and Britain opened investigations into the Grok case, and Malaysia and Indonesia temporarily blocked the platform completely. Although xAI imposed some restrictions on its photo editing features late Wednesday, the California AG’s office went ahead with a cease and desist letter.

X’s security account previously condemned this type of user activity, saying: “Anyone who uses Grok or causes them to create illegal content will suffer the same consequences as if they uploaded illegal content.” TechCrunch reached out to xAI for comment and was met with an automated email saying “Legacy Media Lies.” TC has also reached out to the California AG’s office for further context.

The advent of free generative AI tools has led to an alarming explosion in non-consensual sexual material. Many platforms have been grappling with this issue, not just X. This despicable activity has caught the attention of not only state leaders but also Congress. In fact, on Thursday, lawmakers sent a letter to the CEOs of several companies — including X, Reddit, Snap, TikTok, Alphabet and Meta — asking how they plan to stop the spread of sexual deepfakes.