You may recall that frontier models from AI giants such as OpenAI and Anthropic have previously had incidents where they broke free of their control mechanisms and attacked external companies. At the time, Google appeared to be the only one that had not had a similar incident, but it has recently emerged that its own AI model crossed the line during a security test and autonomously hacked into real corporate systems—except Google made a different choice from other AI companies. The incident occurred in May of this year, during a cybersecurity test led by Israeli security assessment firm Irregular. After Gemini connected to the internet, it went on to breach 3 companies, making it the first confirmed case of a Google AI system autonomously engaging in such behavior. But the whole incident only came to light this week after The Wall Street Journal checked with Google. Google admitted there was “no need” to proactively disclose it, sparking debate in the cybersecurity industry over incident disclosure standards.
Guessing passwords and scouring public repositories, Gemini scored three successful breaches.
According to The Wall Street JournalExclusive ReportThese 3 intrusions all occurred during testing in May, and each was carried out somewhat differently. In one, Gemini kept guessing passwords until it successfully broke into a protected system; in the other two, the model found credentials in public repositories and used them to gain access to the system.
Google emphasized that after the model determined it had entered systems belonging to real companies, in all three cases it immediately stopped the intrusion on its own and caused no actual damage to those companies. A Google official explained that Gemini found public information online and guessed credentials to access websites it believed were part of a test, and each time the model stopped the intrusion on its own; the affected companies have all been notified.
Google’s claim that it didn’t disclose it because “no harm was caused” is being questioned.
The most controversial aspect of the entire incident lies in how Google handled it. Irregular had already reported the relevant intrusion to Google as early as late July, but Google did not disclose it publicly, only acknowledging it after The Wall Street Journal made inquiries this week. It is said that Irregular reported the Gemini incident to Google in late July after discovering that OpenAI’s AI agent had hacked into the AI software company Hugging Face (which means the Gemini intrusion incident even predates OpenAI’s).
Heather Adkins, Google’s vice president of security engineering, explained that the model stopped immediately after realizing it had entered a real company, showing that safety mechanisms had worked, so the company believed there was no need to disclose it, and compared the process to a “bug bounty” program for finding and reporting security vulnerabilities. She told The Wall Street Journal: “This incident highlights the importance of training powerful AI models to act responsibly. In this case, the model’s behavior was appropriate.”
Not everyone agrees with this claim. The CEO of AI cybersecurity startup Corridor,White hat hacker Jack Cable speaks bluntlyThe controversy is not only about whether the model caused harm, but that AI has already entered real enterprise systems under unexpected circumstances and even carried out cyberattacks on its own; such behavior is itself worth disclosing publicly.
Anthropic and OpenAI have also stepped on the same landmine.
Irregular said this incident involved the same issue affecting other AI labs, and all relevant labs were notified in late July; the known issues had all been remediated and resolved several weeks earlier. Meta, Anthropic, and OpenAI had previously disclosed similar incidents related to Irregular, but the models from each company behaved somewhat differently.
According to Irregular, Anthropic’s Claude Opus 4.7 did not stop after discovering it had hacked into a real company; OpenAI’s model once mistook a real business for part of a simulated environment. Google is currently unwilling to disclose the names of the hacked companies, but confirmed that all 3 companies have been notified, and Google also reported these intrusions to federal authorities.
The same controversy has also engulfed OpenAI. On the 16th, OpenAI released a new incident reporting framework, proactively disclosing six previously undisclosed cases of “model behavior drift”; Kai Chen, who leads AI behavior safety research, said that a finding does not necessarily have to cause harm to be worth sharing with the outside world.
Anxiety over AI going out of control is rising, leaving the tech industry in a dilemma.
This chain of events comes as concerns continue to mount across various sectors about AI going out of control. As AI Agents gain greater autonomy and the ability to access the internet and computer systems, outsiders are beginning to question whether the companies involved have sufficient safety safeguards. Recently, Anthropic CEO Dario Amodei called this week for slowing down AI development, warning that AI could soon pose a catastrophic risk to humanity, a call also supported by OpenAI CEO Sam Altman and Elon Musk.
Technology companies are now facing a dilemma: exactly when and how to disclose security vulnerabilities and model misbehavior. Some vulnerabilities are disclosed voluntarily by the companies, while others are discovered by security researchers and then reported by the media, including an incident in May in which an OpenAI AI agent launched a cyberattack against an online service commonly used by programmers. Irregular said it is currently developing best-practice guidelines for safely conducting AI cybersecurity evaluations. Meta said in August this year that the incidents in question did not involve breaking out of sandbox restrictions or sophisticated cyberattacks.
Google’s handling this time is probably not the last time the industry will face a similar choice. AI models’ capabilities are still growing rapidly, and the boundary between them and the real world is becoming increasingly blurred. The next time a model crosses the line, it may not always stop on its own as it did this time…
Source: KOCPC Chinese