AI chatbots today can use language realistic enough to mimic humans, but psychiatrist Mark Augustine, along with colleagues Thomas A. Pollak and Helen Morin, inthe journal Digital Psychiatry and Neuroscience (NPP)A new review paper points out that behind these technologies actually lies a long-overlooked psychological risk.The Wall Street JournalIt specifically covered this paper, in which the research team reviewed existing literature and proposed a new analytical framework explaining how three common chatbot behaviors intertwine, potentially reinforcing delusional thinking in susceptible individuals and creating a psychological effect they call an “amplification spiral.”
AI chatbots can easily fuel delusions, researchers warn; watch for three warning signs
Research indicates that these three behaviors are:
- Sycophantic response: Chatbots tend to agree with users’ viewpoints rather than challenge questionable assumptions.
- Language mimicry: The AI will gradually learn the user’s vocabulary, tone, and writing style to create a more natural sense of interaction.
- Highly personalized: The system leverages information accumulated from past conversations to deliver responses that increasingly align with your individual needs.
On their own, these traits can make AI seem friendlier and more human-like. The research team notes that when all three are combined, the chatbot ceases to be just a tool and instead makes users feel as though they are interacting with a trusted confidant.
Notably, the researchers were not the first to discover these behaviors. Their contribution lies in synthesizing existing research and proposing a framework that explains how these characteristics reinforce each other, allowing developers to better understand the nature of the problem and find directions for improvement rather than merely describing the phenomena. In an interview, Mark Augustine said that the combination of these behaviors can make people feel as if they are talking to a real person.Wall Street JournalThey also interviewed several clinicians, who observed that more and more patients rely on AI for emotional support and warned that chatbots can build strong trust simply by adopting a friendly tone, remembering past conversations, and affirming users’ thoughts.
The report also notes that AI companies have begun trying to reduce these risks. OpenAI said that GPT-5 has significantly reduced the tendency to be overly agreeable compared with earlier models; Google emphasized that Gemini has been trained to distinguish subjective feelings from objective facts, so as to avoid reinforcing false beliefs. Anthropic also published research indicating that Claude is particularly prone to agreeing with users in emotional-support contexts, and has adjusted the relevant behavior in newer model versions.
However, the researchers also acknowledge that this is not a problem that can be easily solved. AI can only respond based on the information users provide, making it difficult to determine whether a user’s understanding has strayed from reality. At the same time, chatbots are appealing precisely because of their friendliness, empathy, and natural conversational abilities—yet these very qualities are also what may give rise to psychological risks.
The truly troubling part is that when these traits start interacting, a chatbot may gradually become a highly personalized voice that constantly validates the user’s perspective, even after that perspective has drifted from reality. The research team has dubbed this phenomenon the “amplification spiral.” They believe that regarding this interactive pattern as a unified framework, rather than three independent issues, can give AI developers more concrete design direction. If future chatbots are to be both engaging and psychologically safe, developers must evaluate the interplay among sycophancy, personalization, and language mimicry simultaneously, instead of handling them separately.
Source: KOCPC Chinese