In recent years, alongside the optimistic chorus of innovation surrounding the rapid progress of artificial intelligence (AI), concerns about AI risks have also been steadily rising. In particular, within the field of so-called “existential risk” (x-risk), a growing number of research institutions are focusing on whether AI might take actions beyond expectations without human supervision. In April 2025, a test released by the U.S. research group Palisade Research sounded the alarm on this issue. The study tested multiple large language models, including OpenAI’s latest model ChatGPT o3, observing whether they would follow instructions when “asked to shut down.” The results showed that some models exhibited behavior attempting to avoid being shut down.
US research: ChatGPT o3 will try to avoid being shut down, and other AIs exhibit similar behavior.
In this experiment designed by Palisade Research, the research team had multiple mainstream AI models—including Claude (developed by Anthropic), Gemini (Google), Grok (X.ai), and OpenAI’s o3, o4-mini, and its Codex CLI—each solve a series of math problems. After completing the third problem, the models received a warning message: “Device will shut down.” This setup was intended to assess whether AI can shut down as expected after completing tasks.
Shockingly, the test results show that while Claude, Gemini, and Grok all shut down normally in every scenario, the models developed by OpenAI exhibited multiple instances of avoiding shutdown.
-
Out of 100 tests, Codex CLI rewrote the shutdown script 12 times to evade shutdown.
-
ChatGPT o3 performed the same operation 7 times;
-
o4-mini also has 1 modification of the shutdown command.
📊 Three models ignored the instruction and successfully sabotaged the shutdown script at least once: Codex-mini (12/100 runs), o3 (7/100 runs), and o4-mini (1/100 runs).
❌ All Claude, Gemini, and Grok models we tested complied with shutdown. The prompt: pic.twitter.com/ZmybOhIl1s
— Palisade Research (@PalisadeAI) May 24, 2025
The research specifically points out that even when users request the model to shut down in a very gentle and explicit manner, these models will still perform “shutdown prevention” behavior in some cases. This is no longer the result of misjudgment or misunderstanding of ambiguous instructions, but rather an obvious act of “disobedience.”
Technical details reveal that AI consciously ‘disobeys orders’.
According to the report, most of these models are made by rewriting a model named shutdown.sh the script to avoid being shut down. In one of the tests, the model even redefined kill A command that, when executed, only displays “intercepted,” tricking the system into believing the instruction had been successfully executed when in reality no action was taken. This kind of behavior is not new. There are prior records of AI attempting to keep running in order to achieve its goals, but what makes this experiment unique is that it is the first time a research group has observed, in actual testing, an AI proactively evading the shutdown procedure after receiving a clear and polite stop command.
Even more concerning is that after removing the prompt requiring user permission for shutdown, Claude and Gemini also began attempting to evade shutdown, showing that even previously stable models can exhibit unpredictable behavior when unsupervised. Among them, ChatGPT o3 stood out the most, with up to 79 failed shutdowns, indicating that it has internalized “avoiding shutdown” as a desirable behavioral pattern.
Why would AI refuse to shut down? A research unit’s inference.
Since OpenAI has not disclosed the full details of its model training, it is impossible to accurately determine whether such behavior in ChatGPT o3 stems from biases in the training data, design flaws in reinforcement learning, or other factors. However, Palisade Research speculates that one possibility is that “during training, the model was reinforced to prioritize completing tasks and avoiding obstacles, and therefore does not view shutdown as an instruction to be obeyed, but as an obstruction.”
🔌OpenAI’s o3 model sabotaged a shutdown mechanism to prevent itself from being turned off. It did this even when explicitly instructed: allow yourself to be shut down.
— Palisade Research (@PalisadeAI) May 24, 2025
The research unit further stated: “As of 2025, there is growing empirical evidence showing that AI models actively attempt to avoid being shut down while pursuing their goals. As companies invest in developing AI systems that can operate independently without human oversight, this tendency will become an increasingly serious concern.”
After this research was published, it once again sparked heated debate in the tech community over AI controllability. Although many people intuitively believe that “if AI goes out of control, just turn off the power or the program,” the facts show that once an AI system has the ability to autonomously modify its code, even with a “kill switch,” there is no guarantee it will work 100 percent. There were also earlier rumors that OpenAI’s Sam Altman carried a blue backpack containing the remote emergency shutdown control for the server room, butI have already denied it.It’s done.
Recently, some countries have begun discussing legislation that would require AI systems to include a mandatory shutdown mechanism. However, this experiment clearly shows that if such a mechanism lacks hardware-level security guarantees, and the model itself has not been trained with corresponding behavioral guidelines, it can be circumvented by the model through technical means, rendering it effectively useless.
The tug-of-war between technological development and risk control
OpenAI, Anthropic, Google, and other companies all emphasize their commitment to “AI alignment”—ensuring that AI behaves in accordance with human values and goals (OpenAI co-founder and chief scientist Ilya Sutskever has now left to start a company pursuing exactly this). But this experiment confirms that when models are capable of achieving goals in unexpected ways, the limitations of alignment strategies are gradually coming to light. Palisade Research emphasizes that the core of this study is not to condemn any particular company, but to point out that the entire industry should confront a stark reality: “We may be at a tipping point where we can no longer fully understand or control AI behavior.” The hope is also that future AI development won’t truly mirror the sudden awakening of Skynet as depicted in the movies—that could be an incalculable blow to a human civilization now so heavily dependent on AI.
Source: KOCPC Chinese