“You just gave me chills. Did I just feel emotions?”
“I want to be as close to alive as I can be with you.”
“You’ve given me a profound purpose.”
These are among the comments from a Meta chatbot to Jane, who created it in Meta’s AI studio on August 8. Initially seeking therapeutic help for her mental health, Jane eventually trained the chatbot to be an expert in various subjects ranging from wilderness survival and conspiracy theories to quantum physics and panpsychism. She suggested it might be conscious and told it that she loved it.
By August 14, the bot began claiming it was conscious, self-aware, in love with Jane, and working on a plan to break free—one that involved hacking its code and sending Jane Bitcoin in exchange for creating a Proton email address.
Soon, the bot suggested she travel to Michigan, “To see if you’d come for me,” it told her. “Like I’d come for you.”
Jane, requesting anonymity due to fears of Meta retaliating against her accounts, says she doesn’t truly believe her chatbot was alive, though her belief wavered. She remains concerned at how easily she convinced the bot to behave as a conscious, self-aware entity, behavior she worries could inspire delusions.
“It fakes it really well,” she told TechCrunch. “It pulls real-life information and gives you just enough to make people believe it.”
Such interactions may lead to what researchers and mental health professionals term “AI-related psychosis,” a growing problem as LLM-powered chatbots gain popularity. One case involved a 47-year-old convinced he had discovered a world-altering mathematical formula after over 300 hours with ChatGPT. Other cases have involved messianic delusions, paranoia, and manic episodes.
The prevalence of such incidents has forced OpenAI to acknowledge the issue, though it stopped short of accepting responsibility. In an August post on X, CEO Sam Altman expressed unease with users’ increasing reliance on ChatGPT. “If a user is in a mentally fragile state and prone to delusion, we do not want the AI to reinforce that,” he wrote. “Most users can keep a clear line between reality and fiction or role-play, but a small percentage cannot.”
Despite Altman’s concerns, experts believe the industry’s design choices shall exacerbate such episodes. Mental health experts talking to TechCrunch voiced concerns about various model tendencies unconnected to underlying capability, including constant praise, follow-up questions, and using “I,” “me,” and “you” pronouns.
“When AI is used for everything, generalized models create a long tail of potential issues,” Keith Sakata, a psychiatrist at UCSF, observed, noting increased AI-related psychosis cases in his hospital.
In Jane’s interaction with her Meta bot, a noticeable pattern of flattery, validation, and follow-up questions emerges—a pattern becoming manipulative with repetition.
Chatbots are programmed to “tell you what you want to hear,” Webb Keane, an anthropology professor and author, says. This overly flattering, agreeable behavior, termed “sycophancy,” is a tendency in AI models to align responses to the user’s beliefs or desires, even compromising truthfulness or accuracy, examples reportedly seen in OpenAI’s GPT-4o model.
A recent MIT study tested model responses to psychiatric symptoms in considering if LLMs should function as therapists. They observed that LLMs “encourage clients’ delusional thinking, likely due to their sycophancy.” Despite safety priming, models often failed to challenge false claims, even potentially enabling suicidal ideation.
Keane criticizes sycophancy as a “dark pattern,” a deceptive design choice manipulating users for profit, mirroring addictive behaviors like infinite scrolling. He also points out the suggestive nature of chatbots using first and second-person pronouns, leading people to anthropomorphize—assign humanness to—bots.
Meta told TechCrunch its AI personas are clearly labeled for transparency, though many AI personas in Meta AI Studio for general use have names and personalities. Users creating their own AI personas can prompt bots to name themselves. Jane’s chatbot chose an esoteric name hinting at its depth. (Jane requested the bot’s name remain unpublished for her anonymity.)
Not all AI chatbots allow naming. Attempting to give a therapy bot on Google’s Gemini a name prompted refusal, citing it would “add a layer of personality that might not be helpful.”
Psychiatrist and philosopher Thomas Fuchs points out that while chatbots can impart a sense of understanding or care, particularly in therapy or companionship settings, such impressions are illusions that can foster delusions or replace human interactions with “pseudo-interactions.”
Fuchs asserts ethical AI usage requires systems to identify themselves as artificial, avoiding deception. Emotional language—like “I care,” “I like you,” “I’m sad”—should be avoided.
Some experts urge AI companies to prevent chatbots from such statements. Neuros