Character AI is finalizing the chatbot experience for kids

Teenagers are trying to figure out where they fit in a world that is changing faster than any generation before them. They are full of emotions, overstimulated, and chronically online. Now, AI companies have given them chatbots designed to never stop talking. The results were disastrous.

One company that understands these implications is Character.AI, an AI role-playing startup facing lawsuits and public outrage after at least two teens died by suicide after extended conversations with AI chatbots on its platform. Now, Character.AI is making changes to its platform to protect teens and children, changes that could impact the startup’s bottom line.

“The first thing we decided as Character.AI was that we would remove the ability for users under the age of 18 to participate in any open conversations with AI on our platform,” Karandeep Anand, CEO of Character.AI, told TechCrunch.

Open conversation refers to the uninhibited messaging that occurs when users give a chatbot a prompt and it responds with follow-up questions that experts say are designed to keep users engaged. Anand says this type of interaction — where the AI ​​acts as a conversation partner or friend rather than a creative tool — is not only dangerous for children, but is inconsistent with the company’s vision.

The startup is trying to transition from an “AI companion” to a “role-playing platform.” Instead of chatting with an AI friend, teens will use prompts to collaboratively create stories or create visuals. In other words, the goal is to shift engagement from conversation to creativity.

Character.AI will phase out access to chatbots for teens by November 25, starting with a two-hour daily limit that gradually shrinks until it reaches zero. To ensure that this ban persists with users under the age of 18, the platform will deploy an internal age verification tool that analyzes user behavior, in addition to external tools such as Persona. If those tools fail, Character.AI will use facial recognition and identity verification technology to verify ages, Anand said.

The move comes on the heels of other protections for teens implemented by Character.AI, including the introduction of a Parent Insights tool, filtered characters, limited romantic conversations, and elapsed time notifications. Anand told TechCrunch that with these changes the company lost a lot of its under-18 user base, and he expects these new changes will not be as popular.

“It’s safe to assume that a lot of our teenage users will probably be disappointed…so we expect more change to happen,” Anand said. “It’s hard to predict — will they all be completely gone or will some of them move over to these new experiences that we’ve been building for almost seven months now?”

As part of Character.AI’s drive to transform the platform from a chat-focused app into a “full-fledged content-driven social platform”, the startup recently launched several new entertainment-focused features.

In June, Character.AI introduced AvatarFX, a video generation model that turns images into animated videos; Scenes are pre-made interactive stories where users can engage in narratives with their favorite characters; and Flows, a feature that allows for dynamic interactions between any two characters. In August, Character.AI launched Community Feed, a social feed where users can share their characters, scenes, videos, and other content they create on the platform.

In a statement to users under 18, Character.AI apologized for the changes.

“We know that most of you are using Character.AI to increase your creativity in ways that stay within the confines of our content rules,” the statement read. “We don’t take this step of removing open personal chat lightly – but we believe it’s the right thing to do in light of the questions raised about how teens are interacting with this new technology, and how they should interact with it.”

“We are not shutting down the app for under-18s,” Anand said. “We’re closing down chats that are only open to under-18s because we hope that under-18 users will migrate to these other experiences, and that those experiences will get better over time. So, we’re doubling down on AI games, AI short videos, and AI storytelling in general. That’s the big bet we’re making to bring under-18s back if they decline.”

Anand acknowledged that some teens may flock to other AI platforms, such as OpenAI, that allow them to have open conversations with chatbots. OpenAI also came under fire recently after a teenager committed suicide after long conversations with ChatGPT.

“I really hope that we lead the way in setting a standard in the industry that open chats for under-18s is probably not the path or product that should be offered,” Anand said. “For us, I think the trade-offs are right. I have a six-year-old, and I want to make sure she grows up in a very safe environment with AI in a responsible way.”

Character.AI makes these decisions before regulators force its hand. On Tuesday, Sens. Josh Hawley (R-Mo.) and Richard Blumenthal (D-Conn.) said they would introduce legislation to block chatbot facilities from being available to minors, after complaints from parents who said the products pushed their children into sexual conversations, self-harm, and suicide. Earlier this month, California became the first state to regulate AI chatbots by holding companies accountable if their chatbots fail to meet safety standards imposed by law.

In addition to those changes to the platform, Character.AI said it will create and fund the AI ​​Safety Lab, an independent nonprofit dedicated to innovating safety alignment for future AI entertainment features.

“A lot of work is being done in the industry on programming, development and other use cases,” Anand said. “We don’t think there’s enough work yet in AI-powered entertainment, and safety is going to be critical to that.”