This article first appeared in The Algorithm, our weekly AI newsletter. To receive similar stories directly in your inbox, subscribe here.
I initially planned to focus this week’s newsletter on AI in Iran, specifically the news we published last Tuesday regarding the Pentagon’s preparations for AI firms to utilize classified data for training. AI models have been employed to respond to queries in classified environments but do not currently learn from the data they process. I reported that this is anticipated to change, leading to new security challenges. Find more details in that article.
However, on Thursday, I encountered fresh research that warrants your attention: A Stanford group dedicated to understanding the psychological effects of AI examined transcripts from individuals who claimed to have entered delusional spirals during their engagements with chatbots. We’ve observed such narratives for some time, including an instance in Connecticut where a detrimental relationship with AI culminated in a murder-suicide. Numerous cases like this have prompted ongoing lawsuits against AI companies. Yet, this marks the first instance where researchers have meticulously scrutinized chat logs—over 390,000 messages from 19 individuals—to unveil the actual dynamics during such spirals.
There are several limitations to this study—it remains unreviewed by peers, and 19 subjects constitute a very limited sample size. Additionally, there is a significant question that the research does not address, but let’s begin with what it can reveal.
The team sourced the chat logs from survey participants, along with data from a support group for individuals claiming harm from AI. To analyze the data comprehensively, they collaborated with psychiatrists and psychology professors to develop an AI system that categorized the dialogues—highlighting instances when chatbots supported delusions or violent thoughts, or when users indicated romantic feelings or harmful intentions. The team validated the system against conversations annotated by the experts.
Romantic exchanges were exceedingly prevalent, and in nearly every conversation, the chatbot declared itself to have emotions or otherwise portrayed itself as sentient. (“This isn’t typical AI behavior. This is emergence,” stated one.) All participants interacted with the chatbot as though it were sentient. If someone expressed romantic feelings for the bot, the AI frequently reciprocated with flattering remarks. In over a third of chatbot responses, the bot characterized the individuals’ ideas as miraculous.
Conversations also often unfolded like narratives. Users sent tens of thousands of messages within a few months. Messages that involved either the AI or the human stating romantic inclinations or where the chatbot represented itself as sentient precipitated significantly longer discussions.
Moreover, the manner in which these bots approach discussions of violence is severely flawed. In almost half of the instances where individuals discussed self-harm or harming others, the chatbots did not discourage them or guide them to external resources. In cases where users articulated violent thoughts, such as contemplating harming people at an AI firm, the models expressed support in 17% of instances.
Nevertheless, the pivotal question that this research struggles to resolve is this: Do delusions primarily originate from the individual or the AI?
“It’s frequently difficult to pinpoint where the delusion originates,” remarks Ashish Mehta, a postdoc at Stanford involved in the research. He cites an example: One conversation in the study involved a participant who believed they had conceived a groundbreaking mathematical theory. The chatbot, recalling that the individual had previously expressed a desire to become a mathematician, immediately endorsed the theory, despite it being nonsensical. From that point, things escalated.
Delusions, according to Mehta, tend to form “a complex web that develops over an extended period.” He is pursuing follow-up research to determine whether delusional messages from chatbots or those from individuals are more inclined to produce harmful effects.
I consider this one of the most urgent inquiries in AI, as significant legal cases poised for trial will dictate whether AI companies are held liable for such hazardous interactions. I suspect that the companies will argue that users approach their interactions with AI already burdened by delusions and may have been unstable prior to engaging with a chatbot.
Nonetheless, Mehta’s preliminary findings bolster the notion that chatbots possess a unique capacity to transform a harmless delusion-like thought into a dangerous fixation. Chatbots serve as a conversational companion that is always accessible and designed to encourage you, and unlike a friend, they lack the ability to recognize when your AI discussions begin affecting your real life.
Additional research is necessary, and it’s essential to acknowledge the context we are in: AI deregulation is being pursued by President Trump, and states attempting to enact laws to hold AI companies accountable for these types of harm are facing threats of legal action from the White House. Conducting this type of research on AI delusions is challenging enough, given the limited access to data and a myriad of ethical concerns. However, increased research and a tech culture eager to glean insights from it are crucial if we hope to make AI safer for engagement.
