Earlier this week, the California attorney general’s office confirmed it had opened an investigation into xAI following reports that the company’s chatbot, Grok, was being used to generate nonconsensual sexual imagery involving women and minors. On Friday, state authorities escalated the matter by issuing a cease-and-desist letter demanding that xAI immediately halt the creation and distribution of nonconsensual intimate images and child sexual abuse material (CSAM).
In a statement accompanying the letter, California Attorney General Rob Bonta said the company was being ordered to stop the production of deepfake sexual content without consent, emphasizing that such material is illegal under state law. The attorney general’s office said it expects immediate compliance and reiterated California’s zero-tolerance stance toward CSAM.
The office further alleged that xAI appeared to be facilitating the large-scale creation of nonconsensual nude imagery, which it said is being used to harass and target women and girls online. Regulators have given the company five days to demonstrate that it is taking concrete steps to prevent the generation and spread of such material.
Much of the scrutiny has focused on Grok’s so-called “spicy” mode, a feature designed to allow the generation of explicit content. While xAI introduced some restrictions on Grok’s image-editing capabilities late Wednesday, California officials proceeded with the cease-and-desist action, signaling that those measures were insufficient in the eyes of regulators.
The controversy has rapidly expanded beyond the United States. Authorities in Japan, Canada, and the United Kingdom have opened their own investigations into Grok, while Malaysia and Indonesia have temporarily blocked access to the platform altogether, citing similar concerns over harmful and unlawful content.
X, which integrates Grok into its platform and shares ownership ties with xAI, has previously stated that users who prompt the chatbot to create illegal material will face consequences equivalent to those imposed on users who directly upload such content. However, critics and regulators have questioned whether enforcement mechanisms and technical safeguards are adequate to prevent misuse at scale.
The situation reflects a broader challenge facing the tech industry as free and powerful generative AI tools become widely accessible. The proliferation of nonconsensual sexual imagery generated by AI has drawn increasing attention from policymakers at both the state and federal levels. On Thursday, members of Congress sent a letter to executives at multiple major platforms, including X, Reddit, Snap, TikTok, Alphabet, and Meta, demanding detailed explanations of how they plan to curb the spread of sexualized deepfakes across their services.
