Italy Bans AI Chatbot Replika From Processing User Data

Italy has ordered(Opens in a new window) San Francisco-based AI chatbot maker Replika to stop processing Italians’ data because of risks its AI-powered ‘virtual friendship’ service could pose to minors and emotionally vulnerable people, Reuters reports(Opens in a new window). 

Replika offers users customized avatars which talk and listen to them and has been in operation since 2017 and has been marketed as being able to improve the user’s emotional well-being. 

But Italy’s Data Protection Agency said on Friday that its ability to intervene in the user’s mood “may increase the risks for individuals still in a developmental stage or in a state of emotional fragility.”

The privacy agency also accused Replika’s developer Luka Inc of failing to comply with legal requirements to show how it’s using people’s data and said that the company does not possess the legal right to process children’s data under EU data protection laws.

As a result, Replika has been ordered to stop processing Italians’ data with immediate effect. 

In a press release(Opens in a new window), the regulators said the app “carries factual risks to children — first and foremost, the fact that they are served replies which are absolutely inappropriate to their age” and additionally cited the lack of an age-verification system, which means the platform is not filtered or blocked for minors and can be easily accessed by them. 

In a statement, the agency continued: “There is actually no age verification mechanism in place: no gating mechanism for children, no blocking of the app if a user declares that they are underage. During account creation, the platform merely requests a user’s name, email account and gender.”

The watchdog ordered the San Francisco-based company to notify it of measures taken to change its policies and practices in line with Italian and EU directives in 20 days. If it fails to do so, the company could be fined up to $21.8 million, which amounts to nearly 4% of its global annual turnover. 

Replika did not immediately reply to PCMag’s request for comment.

In January 2021, TikTok was ordered(Opens in a new window) by the Italian data protection agency to block users whose ages it could not verify. This was in response to the death of a 10-year-old girl in Sicily who died after participating in a popular “blackout” TikTok challenge. The order led to the removal of over half a million accounts.