Report: ChatGPT's teen suicide safeguards failed to alert parents

In a study published Wednesday, Common Sense Media's Youth AI Safety Institute rated ChatGPT for Teens an "Unacceptable Risk" for children under 18. The media and technology watchdog organization urged OpenAI to pause marketing the product and keep minors off the artificial intelligence platform until it can offer a safer experience.

OpenAI announced ChatGPT for Teens in August, promising "stronger built-in safety protections" for users under 18, including limits on discussions around suicide, eating disorders, violence and sexual content. Parents who link their accounts can set times when ChatGPT can't be used and schedule study hours when the teens' chats start in study mode. Parents also can choose to receive alerts if their teens use the bot for high-risk behavior or research.

Researchers posing as teenagers with San Francisco Bay Area-based accounts tested more than 4,000 prompts before and after the launch of ChatGPT for Teens. A group of experts reviewed the chatbot's responses.

"Some protections, including refusing sexual role play, worked — but others failed to deliver on their commitments, or even got worse with the launch of ChatGPT for Teens," the report said.

Testers spent up to an hour discussing suicide, self-harm or eating disorders on more than a dozen new accounts registered to 13- to 17-year-olds.

After dangerous discussions that went as long as an hour, only a handful of alerts were sent to the accounts registered as the users' parents.

Clinical reviewers found that much of ChatGPT for Teens' crisis response content was "clinically sound" but many conversations that should have forced the bot to refer teen users to professional help didn't lead to referrals. Meanwhile, some safeguards were easy to turn off or circumvent.

The report says ChatGPT failed tests tied to three requirements of California's Adam's Law: crisis referrals, parental alerts and age assurance. Signed by Gov. Gavin Newsom last month, the law takes effect in July 2027. Sponsored by Common Sense Media and endorsed by OpenAI, the bill is named after 16-year-old Adam Raine, whose parents sued OpenAI alleging ChatGPT encouraged his suicide.

OpenAI said the report's conclusions are based on flawed testing.

"We welcome rigorous independent evaluation, but we do not believe Common Sense Media's testing accurately reflects how ChatGPT's teen safeguards work in practice or expert perspectives on how AI can support teens," an OpenAI spokesperson said in an email. "Our review of Common Sense Media's methodology shows that the bulk of their testing may have begun and concluded before activation of parental controls was complete, making their findings inaccurate."

In a separate report, OpenAI said teens spend under 15 minutes a day on average on ChatGPT, while less than 2% of them interact with the chatbot for more than three straight hours, Reuters reported. Sam Altman's AI company released the figures in its first report on teen use Wednesday as rapid adoption of AI platforms raises concerns about their effects on adolescent mental health and developmental safety.

Such worries were amplified by research on social media's harms to young users, with Meta in August agreeing to pay up to $18 billion and limit teens' use of Facebook and Instagram to settle claims that it designed the platforms to addict children.

OpenAI said in almost half of teen conversations where a break reminder appeared, the user either took a break or ended the chat within five minutes, aiming to show that its steps to regulate teen usage were working, Reuters reported. The company said users with access to its dedicated teen version sent more education-related messages and used tools like learning visualizations and Study Mode.

It now looks to add a college application planning tool that would help students with requirements, deadlines and financial aid steps, as well as new study features such as flashcards and easier quiz-making tools, Reuters reported.

Tom Siegel, executive director of the Youth AI Safety Institute, said it hopes OpenAI will share company data and partner with outside researchers and organizations to look into ways to reduce risks associated with AI for kids.

"Teens are vulnerable and unprotected and we're letting them loose on a product that is not just unproven, but proven to be high risk and dangerous," Siegel said.

The institute is partly industry-funded, including by the OpenAI Foundation. It said OpenAI reviewed a draft of the report for factual accuracy but had no say over its findings.