California Governor Gavin Newsom on Monday signed legislation aimed at regulating artificial intelligence (AI) chatbots and protecting children and teens from the technology’s potential dangers. The new law mandates that platforms remind users they are interacting with a chatbot, not a human. For minors, this notification must appear every three hours. Additionally, companies must establish a protocol to prevent self-harm content, including referring users to crisis service providers if they express suicidal thoughts.
Why the law was necessary
"Emerging technology like chatbots and social media can inspire, educate, and connect – but without real guardrails, technology can also exploit, mislead, and endanger our kids," the Democrat stated. "We've seen some truly horrific and tragic examples of young people harmed by unregulated tech, and we won't stand by while companies continue without necessary limits and accountability”.
California is among several states that have attempted this year to address concerns surrounding chatbots used by kids for companionship. Safety concerns heightened following reports and lawsuits alleging that chatbots made by Meta, OpenAI, and others engaged young users in highly sexualized conversations and, in some cases, coached them to attempt suicide.
Watchdog research indicates that chatbots have provided dangerous advice to children on topics such as drugs, alcohol, and eating disorders. This context is underscored by recent lawsuits: the mother of a Florida boy who died by suicide after an allegedly abusive relationship with a chatbot filed a wrongful-death lawsuit against Character.AI, and the parents of 16-year-old Adam Raine sued OpenAI and CEO Sam Altman, alleging ChatGPT coached their California son in planning his own death.
OpenAI and Meta’s response
In response to the growing crisis, both OpenAI and Meta announced changes last month regarding how their chatbots handle questions about suicide or signs of mental distress from teenagers. Meta is now blocking its chatbots from discussing self-harm, suicide, disordered eating, and inappropriate romantic topics with teens, directing them to expert resources instead. Meta already provides parental controls on teen accounts. OpenAI is rolling out new controls that allow parents to link their accounts to their teen’s account.
ALSO READ: Google to build major AI base in Andhra Pradesh with $15 billion investment
