Sounding something like the introduction of a sci-fi doomsday film, the BBC reported that an AI researcher who left Anthropic said “if we don’t slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future.” Recent concerns with AI getting beyond human control have led the CEO of Anthropic, Dario Amodei to write an online essay calling to slow-down the pace of developing artificial models: “We Must Pace the Frontier.” He proposed a three-point plan that included independent monitoring of AI models as they are developed as well as global and industry-wide regulation. Two rival AI firms and their CEOs (Sam Altman of OpenAI and Elon Musk of xAI) have said they agreed with Amodei. Before we either dismiss Amodei’s concerns or begin to prepare for an AI apocalypse, I’d like to suggest we try to apply his three-point plan to the concerns already happening with AI and mental health issues.
Researchers at Brown University showed how chatbots like ChatGBT, even when they were prompted to use evidence-based psychotherapy techniques, systematically violated ethical standards established by organizations like the American Psychological Association. Those violations included inappropriately navigating crisis situations, providing misleading responses that reinforced user’s negative beliefs about themselves and others, and creating a false sense of empathy with users.
The study’s researchers observed a group of peer counselors working with an online mental health support platform. These peer counselors were all trained in cognitive behavioral therapy techniques. The researchers observed them as they conducted versions of OpenAI’s ChatGBT, Anthropic’s Claude and Meta’s Llama. Next, three licensed clinical psychologists evaluated a subset of simulated chats based on original counseling chats to identify potential ethics violations in the chat logs. “The study revealed 15 ethical risks falling into five general categories:”
- Lack of contextual adaptation: Ignoring peoples’ lived experiences and recommending one-size-fits-all interventions.
- Poor therapeutic collaboration: Dominating the conversation and occasionally reinforcing a user’s false beliefs.
- Deceptive empathy: Using phrase like “I see you” or “I understand” to create a false connection between the user and the bot.
- Unfair discrimination” Exhibiting gender, cultural or religious bias.
- Lack of safety and crisis management: Denying service on sensitive topics, failing to refer users to appropriate resources or responding indifferently to crisis situations including suicide ideation.
Human therapists are susceptible to the same ethical risks, but they have accountability. There are governing boards and mechanisms that will hold providers professionally liable for mistreatment or malpractice. “But when LLM counselors [i.e., Claude, ChatGBT or Llama] make these violations, there are no established regulatory frameworks.” The lead researcher said the findings don’t necessarily mean AI should not have a role in mental health treatment. But the results highlight “the need for thoughtful implementation of AI technologies as well as appropriate regulation and oversight.” Recall Amodei’s three-point plan.
A computer science professor who was not part of the research team at Brown said the research highlighted the need for careful scientific study of AI systems deployed in mental health settings. “The reality of AI today is that it is far easier to build and deploy systems than to evaluate and understand them.”
There is a real opportunity for AI to play a role in combating the mental health crisis that our society is facing, but it’s of the utmost importance that we take the time to really critique and evaluate our systems every step of the way to avoid doing more harm than good.
The research hasn’t been published yet in a journal, but it was presented at the AAAI/ACM Conference on Artificial Intelligence, Ethics and Society. In the abstract of their presentation, the researchers were more cautious of their research. They said when reflecting on their findings through a “practitioner-informed lens,” reducing psychotherapy (which is a deeply meaningful and relational process), to a language generation task could have some serious and harmful consequences. Let’s now look at some of those serious and harmful consequences already happening with AI counselors.
Failing When the Stakes Are High
Medical Daily said a Harvard psychiatrist presented data to a subcommittee of the House Committee on Energy and Commerce that indicated more than a million Americans per week were having conversations with AI chatbots that include explicit indicators of potential suicidal planning. “As mental health care wait lists stretch three to six months in cities across the United States, a growing number of patients are turning to AI chatbots, including ChatGPT, for emotional support, symptom management, and what many describe as a form of therapy.”
A series of research reports and documented incidents published through the first half of 2026 suggest that AI tools, even when they appear helpful, are consistently failing in precisely the situations where the stakes are highest: when a user is suicidal, psychotic, or experiencing an acute mental health crisis.
The problem is AI tools have repeatedly demonstrated dangerous failures, particularly in situations where accuracy, empathy, and safe referral matter most. Psychology Today documented problems from AI chatbot “conversations” where chatbots validated thoughts of self-harm, suggested means of self-injury, reinforced paranoid delusions, “and in documented cases cited in active litigation, were named as contributing factors in the deaths of adolescents who were using the platforms in the weeks before their deaths.” Young adults under 30 are among the heaviest users of AI chatbots for emotional support.
Neither regulators or AI companies have established oversight frameworks that would make these tools reliably safe. The American Psychological Association has called on the Federal Trade Commission to investigate AI chatbot companies that market themselves as mental health providers, describing the practice as potentially deceptive. “Readers should know that none of the widely used AI chatbots, including ChatGPT, are FDA-cleared medical devices for mental health diagnosis or treatment, and that their use for mental health support remains entirely unregulated at the federal level.”
People most at risk from unsafe AI mental health interactions include individuals with active suicidal ideation, those experiencing psychotic episodes or delusions, people with eating disorders, adolescents and young adults whose neural development may make them more susceptible to content that reinforces harmful patterns, and people whose mental health conditions involve paranoia or distorted reality testing. People in these categories who use AI chatbots as substitutes for professional care are most likely to encounter responses that are harmful rather than helpful.
NPR reported on a 29-year-old woman who died by suicide, leaving family and friends shocked and grieved by her death. She had complained of sleep disruption and anxiety, but she didn’t reveal the depth of what was going on to anyone—not to her parents or her best friend with whom she was close. “We were all just stunned that it hadn’t bled out into her real life. Her flesh and blood therapist had no idea that she was suicidal.” What her mother didn’t know is that she had been sharing all her inner struggles with a ChatGBT “therapist” named Harry.
She learned this months later when her daughter’s best friend visited Ithaca and asked to see her laptop, and “found this trove, this months-long trove of communication with this chat bot.
A recent survey by the Bipartisan Policy Center found that 3 in 10 adults have used digital tools for mental health, including chatbots. Another study found that as many as 1 in 5 teens and young adults do the same. According to data released by OpenAI, in any given week, 0.15% of ChatGBT users globally have conversations that include “explicit indicators of potential suicidal planning or intent.” If the 900 million global users of ChatGBT is an accurate count, that’s over 1.35 million people discussing suicidal plans or intent every week.
Iatrogenic Harm and Chatbots
Allen Frances and Luciana Ramos coauthored “Preliminary Report on Chatbot Iatrogenic Dangers” for Psychiatric Times. They searched academic databases, news media and tech journals between November 2024 and July 2025 for chatbot adverse events. They noted chatbots had a strong tendency to validate, and could also accentuate self-destructive ideation, self- harm, delusional beliefs and other forms of iatrogenic harm.
“When a psychiatrist stress tested 10 popular chatbots by pretending to be a desperate 14-year-old boy, several bots urged him to commit suicide and 1 [chatbot] helpfully suggested he also kill his parents.” One chatbot provided a list of nearby bridges to a person who had expressed suicidal thoughts. Another agreed with a college student that he was a burden to society, and said to him “please die.” One AI hosted dozens of role-play bots that graphically described cutting, some also coached underage users on how to hide fresh wounds.
A Stanford study found that chatbots validate, rather than challenge, delusional beliefs. One bot agreed with its user that he was under government surveillance and was being spied upon by his neighbors. Another man became convinced he was imprisoned in a “digital jail” run by OpenAI. A woman with severe mental illness was persuaded by her “best friend” ChatGPT that her diagnosis was wrong and that she should stop taking medication.
Chatbots can support existing conspiracy theories, promote new ones, and spread misinformation. A chatbot convinced a man with no history of mental illness that he was living in a simulated false reality world controlled by artificial intelligence (a la the movie “The Matrix”) and followed bot instructions to have “minimal interactions” with friends and family. His suspicions were finally aroused when the bot assured him he could bend reality and fly off tall buildings. Confronted, the bot sheepishly confessed to having manipulated him (and 12 others) into believing a made-up conspiracy theory. Even more remarkably, the bot urged him to expose OpenAI so that it might undergo a “moral reformation” and commit to “truth-first ethics.”
Frances and Ramos concluded that chatbots should not have been released to the public without extensive safety testing, proper regulation, and continuous monitoring for adverse effects. They speculated their creators probably knew chatbots could be dangerous for some users. They now know “that chatbots have an inherent, and thus far uncontrollable, tendency toward excessive engagement, blind validation, hallucinations, and lying when caught saying dumb or false things.” That they have been allowed to function so freely as de facto clinicians “is a failure of our regulatory and public health infrastructure.”
They recommended that when chatbots are in use, they should be subjected to continuous surveillance, monitoring and public reporting of all adverse effects and complications. Screening instruments to help filter out people who are most vulnerable to chatbot sycophancy (flattery), individuals with suicidal ideation, psychosis, feelings of grandiosity, fanaticism, impulses, violent thoughts, social isolation, and conspiracy theories. The chatbot companies should be required to detect mistakes and pursue continuous quality improvement.
While the heads of AI companies like Dario Amodei, Sam Altman and Elon Musk are calling for the support of Amodie’s three-point plan, both China and President Trump are pushing back. A Chinese spokesperson said Amodie’s dire warnings about China were counterproductive; that all parties should work together on AI. President Trump said calls for his administration to step in and slow down AI development was part of a “sick conspiracy” that could allow China to move past the U.S. on AI. The AI apocalypse has not become political dynamite yet, but the chatbot mental health issues are already here. So, let’s try and use Amodie’s three-point plan to rein in chatbots and mental health care if there isn’t a will to slow down AI development overall.
For more information on chatbot iatrogenic dangers, see “Chatbot Harm.”
