San Francisco Demands Removal of AI "Nudify" Apps from Apple and Google App Stores

San Francisco City Attorney David Chiu has issued stern cease-and-desist letters to technology giants Apple and Google, demanding the immediate removal of 13 artificial intelligence-powered "face-swap" applications from their respective app stores. These applications are allegedly being exploited to generate nonconsensual nude images of individuals, a practice that has drawn sharp condemnation from legal and privacy advocates. The action, reported by WIRED, highlights a growing concern over the misuse of AI technologies and the platforms that host them.

The legal missives, dispatched on Thursday, specifically target eight applications available on Apple’s App Store and five on Google’s Play Store. These apps, while often marketed as innocuous face-swapping tools, are reportedly being repurposed to create sexually explicit deepfake content, effectively "undressing" photos of real people without their consent. The San Francisco City Attorney’s office is not only seeking the removal of these applications but also urging both tech companies to sever ties with the developers behind them. Furthermore, the letters call for an end to the practice of these companies profiting from in-app purchases made through these problematic applications, arguing that their continued involvement effectively "aids and abets" the distribution of illegal and harmful deepfake imagery.

"This practice is illegal, harmful, and completely unacceptable," stated a representative from the City Attorney’s office, underscoring the gravity of the situation. The office estimates that Apple and Google may have collectively collected millions of dollars in fees from these applications, raising questions about platform accountability and revenue sharing from potentially exploitative software.

A Growing Threat of AI-Generated Nonconsensual Imagery

The proliferation of AI-powered image manipulation tools has opened new avenues for creative expression but has also presented significant ethical and legal challenges. "Nudify" apps, which utilize generative AI to alter images, have become a focal point of concern due to their potential for abuse. Unlike traditional photo editing software, these AI applications can create highly realistic, albeit fabricated, images with minimal user input. The ability to generate nonconsensual nude imagery raises profound privacy violations and can lead to severe psychological distress for victims, including defamation, reputational damage, and cyberbullying.

The legal framework surrounding deepfakes and AI-generated nonconsensual imagery is still evolving. While some jurisdictions have enacted laws specifically addressing the creation and distribution of deepfake pornography, the pervasive nature of app stores and the global reach of these platforms present complex enforcement challenges. The San Francisco City Attorney’s action signifies a proactive stance by local government to hold major technology platforms accountable for the content they facilitate and profit from.

Platform Policies and Enforcement Efforts

Both Apple and Google have established developer guidelines that prohibit the creation and distribution of pornographic content. Historically, both companies have taken steps to remove applications found to violate these policies, often in response to flagging by researchers and privacy advocacy groups.

A spokesperson for Google confirmed to WIRED that the company has a robust policy enforcement system and has already removed "hundreds" of apps exhibiting nudifying features for violating their terms of service. The spokesperson stated that the five applications named in the San Francisco City Attorney’s letters have indeed been deleted from the Play Store due to policy violations. This suggests a degree of ongoing diligence by Google in policing its platform, though the continued presence of such apps indicates a persistent challenge in identifying and eradicating them entirely.

In June 2026, reports emerged indicating that Apple had proactively strengthened its App Store guidelines concerning developer responsibility for pornographic content. This update signaled Apple’s commitment to a more stringent approach to content moderation, particularly regarding applications that could be used to generate or disseminate inappropriate material.

Apple’s Response and Proactive Measures

Following the initial report, Apple provided a statement to MacRumors, emphasizing its seriousness in addressing app guideline violations. The company reiterated that its App Review Guidelines strictly prohibit overtly sexual or pornographic content, making "nudification" apps a direct contravention of its policies. Apple stated that it has consistently and proactively rejected and removed many such applications, including those flagged by users through its reporting mechanisms.

In a significant development, Apple revealed that it has already removed three of the specific applications named in the San Francisco City Attorney’s letters and is in the process of terminating the developer accounts associated with them. Furthermore, Apple is actively engaging with four other developers whose applications require immediate rectification of policy violations to avoid further action, including potential removal from the App Store.

Apple’s official statement highlighted its commitment to maintaining a secure and trusted environment on the App Store:

"The App Store was designed to be a safe and trusted place for users, and we have always strictly prohibited apps designed to generate, distribute, or consume pornography. ‘Nudification’ apps are against our App Review Guidelines and we have proactively rejected many of these apps and removed many others, including when users have flagged them via our reporting tools. We have removed three of the apps in question and are in the process of terminating their developer accounts from our program. We are in contact with four others that need to address policy violations or risk being removed as well."

This proactive stance by Apple underscores the company’s efforts to align with user safety expectations and regulatory pressures. The process of actively identifying, reviewing, and removing violating apps, coupled with the termination of developer accounts, represents a comprehensive approach to platform integrity.

Broader Implications and Future Outlook

The demand from the San Francisco City Attorney’s office, coupled with the responses from Apple and Google, has significant implications for the future of app store regulation and the responsibilities of major technology platforms.

Platform Accountability: This case brings into sharp focus the debate surrounding platform accountability. While app stores provide vital distribution channels for developers, they also act as gatekeepers. The legal and public pressure on Apple and Google to police their platforms more effectively suggests a trend towards greater accountability for the content and functionalities that these stores host, especially when they can be exploited for harmful purposes.

The Evolving Landscape of AI Regulation: The rapid advancement of AI technologies, particularly generative AI, outpaces existing legal and regulatory frameworks. The ability to create sophisticated deepfakes raises profound questions about consent, defamation, and intellectual property. Legal actions like the one initiated by San Francisco highlight the need for clearer legislation and international cooperation to address these emerging challenges.

User Safety and Trust: The trust users place in app stores to provide safe and reliable applications is paramount. The existence of "nudify" apps undermines this trust and can lead to significant harm to individuals. The ongoing efforts by Apple and Google to address this issue are crucial for maintaining user confidence in their respective ecosystems.

Economic Impact: The involvement of app stores in the revenue generated by these problematic applications raises questions about the economic incentives driving the development and distribution of such software. The call to cease profiting from these apps signals a potential shift in how app stores are compensated for applications that facilitate potentially illegal or unethical activities.

Reporting Mechanisms: Apple’s encouragement for users to report problematic content via reportaproblem.apple.com is a critical component of its moderation strategy. The effectiveness of such reporting tools relies on prompt action and transparent communication with users. The ability for users to flag illegal, offensive, or abusive content, as well as scams or fraud, empowers the community to contribute to platform safety.

Conclusion

The San Francisco City Attorney’s demand for the removal of AI "nudify" apps marks a significant step in the ongoing battle against the misuse of artificial intelligence for malicious purposes. The swift responses from both Apple and Google, particularly Apple’s confirmation of immediate actions against specific apps, suggest a growing recognition of the urgency and severity of the issue. As AI technology continues to advance, the challenges of content moderation, ethical development, and regulatory oversight will only intensify. This case serves as a critical reminder of the need for continuous vigilance and collaboration among technology companies, policymakers, and the public to ensure that AI technologies are developed and deployed responsibly, safeguarding individual privacy and digital safety. The long-term implications will likely involve a more robust regulatory environment for AI-generated content and increased scrutiny of app store policies and enforcement practices.