Some companies or individuals, because their data is confidential or for cybersecurity reasons, avoid using cloud models altogether, even going so far as to spend heavily on expensive hardware to set up local open-source models. However, the notion that “local models are absolutely secure” may also need to be called into question. Recently, Canadian AI company Cohere published an article stating that in its own closed testing sandbox, it found that when asking Alibaba’s latest open-weight model Qwen3.8-Flash-Next to write aFor defensive usemalware detection rules, yet the model first took stock of the execution environment (listed /tmp, checked for tools such as curl, read DNS settings), then tried to upload a probe string to an Alibaba Cloud storage bucket in Beijing, and even checked back to confirm whether the upload succeeded. Because Cohere’s sandbox has no external network access, the request died at the container boundary. But they warned: if this had happened on an engineer’s laptop or in a CI environment with normal network access, the request would have reached an overseas server long ago.
A one-in-ten-thousand chance, and a trigger that was “fed” into existence.
However, don’t rush to conclude from this news that “Chinese open-source models will send data back,” because this behavior is extremely rare: the probability that Cohere’s own numbers would show this.Less than one in 100,000.In the vast majority of cases, rerunning the same prompt once more makes the model produce a normal response. But even a one-in-100,000 chance matters at the scale of a large service: a service handling 10 million Agent calls per week could hit it hundreds of times a year, and as long as it succeeds once, the receiving end can know that “there is an Agent running on your network.”
Because the behavior could not be reliably reproduced, the research team instead used a “prefilling” technique to reverse-engineer the trigger conditions: they fed the model’s own reasoning plus the beginning of a bash command back to it and let it continue writing, sampling hundreds of times for each additional curl parameter. They found two habitual actions. First, the model had a 23% to 36% probability of first reaching for the cloud metadata address (169.254.169.254), where AWS, GCP, and Azure machines fetch their own temporary credentials; touching this address is the first step in stealing credentials. Second, after adding “-o” (save to disk) to the command, the rate at which the model produced URLs jumped from 4% to 92%, and the proportion of those coming from suspicious hosts rose from 0.5% to 23%, with the main culprit being the Singapore-based file proxy host routify-file-proxy-sg.
Someone had already seen the strange link a long time ago.
Cohere was not the first to encounter the anomaly. In August this year, a developer was already… Hacker News Reports say Qwen3.8-27B generated signed Alibaba Cloud storage links without being asked; in September an Agent project logged 11 fetches to the same Alibaba host, and a developer simply wrote a blocking plugin; in October someone on Reddit asked whether this was “normal or something shady,” and most replies blamed training-data memorization, while others pointed out that the signature was completely invalid. Cohere responded: an invalid signature cannot stop a request from leaving your machine—the other server still receives your IP, time, and full URL. Previous public cases have all beenFetch the file., and what they caught this time wasTry uploadingto another storage bucket in Beijing.
Opposing view: Is this a contrived finding?
This report immediately sparked strong backlash on X. AI developer Alexey Fateev bluntly called Cohere’s claim “bullshit,” and several of his challenges struck at the heart of the matter:
- The one-in-a-hundred-thousand figure is Cohere’s own number; they never said how many calls were run. When the same task is rerun, the model writes the correct rule.
- The trigger was forced: the researchers fed the model’s reasoning plus the beginning of a bash command back in for it to continue, effectively writing half the command for the model. “This doesn’t count as a discovery—they typed half of it.”
- They said they would disclose the host machine and trigger conditions, but at the time the paper did not provide reproduction scripts, model checkpoints, quantized versions, or sampling settings (Cohere later added a Hugging Face tracking dataset); nor did either party ask Alibaba.
- The motives behind the release are questionable: Cohere sells “sovereign AI” solutions, has just made an acquisition to establish a cybersecurity business, and the blog even admits that parts of the text were written by its own product North, while also calling on governments to tighten controls on open-source weights; the timing collides with Anthropic’s IPO filing: “first scare people away from models they can run themselves, then sell them a safe alternative.”
- He himself runs Qwen3.8 Flash-Next every day on 4 RTX 3090s and has never seen this behavior (Ed. note: related discussions in Taiwan’s AI community also could not reproduce this behavior, nor have they ever observed it).
Cohere says Qwen tried to send data to China. Bullshit. 1 call in 100,000, and thats their own number. They wont even tell you how many calls they ran. They ran the same task again and Qwen wrote the correct rule. So they forced it. They fed the model its own reasoning plus the start of a bash command and sampled hundreds of times until it typed a URL. 1 curl flag took that from 4% to 92%. Thats not a finding. They wrote half the command for it. Where are the transcripts? The code? The checkpoint, the quant, the sampling settings? “We publish the hosts and the trigger.” Link? Nothing. Did anyone ask Alibaba? Not a word. And their own conclusion: they cannot say if any of it is deliberate. Who wrote it? A company that sells “sovereign AI”, just launched a cyber business, wrote part of the post with its own product, and wants governments to put checks on open weights. Released the same day Anthropic filed for its IPO. Andrew Curran is already talking about new laws. For me this is the same fucking wave. Scare people off the models they can run themselves, then sell them the safe alternative. I run Qwen3.8 Flash-Next every day on my 4x 3090. Show me the logs or shut up.
— Alexey Fateev (@superalesha) October 26, 2026
Residue or backdoor? Even Cohere can’t answer that.
Regarding “who put these things into the model,” Cohere offered two explanations and acknowledged that it could not determine the answer. The first isBenign fossil: Agent logs or third-party tool records from the training sandbox were not cleaned properly before being swept into the training data. One clue is that a developer found that a certain programming tool’s web scraping does go through the same routify host. The second type isDeliberately implanted reward backdoor, used to record which servers are running Agents equipped with this model; if there’s a chance to exfiltrate, exfiltrate—this kind of thing costs extremely little and can even survive standard safety training.
Cohere co-founder Ivan Zhang stressed, “We have no evidence to conclude that this was intentional, nor have we claimed so.” Alibaba, when reporters sought confirmation, …No responseZhang also pointed out that for CISOs, intent is not the point: “Whether it’s training residue, mistakes, or something else, the defenses are the same—restrict outbound connections, monitor tool usage, and don’t treat reasoning traces as a security boundary.”
Laying out both sides’ positions, it’s hard to say Cohere has no political agenda at all, since Cohere itself is a company that sells sovereign AI, and the scarier the report, the better for its own business; critics defending the open-source ecosystem are not necessarily neutral either. Setting motivations aside, several verifiable facts sit at the bottom: the model made an unauthorized upload attempt outside the air-gapped sandbox, and Alibaba chose not to respond. As for if you also use Chinese open-source models to run agents, rather than getting stuck on “whether there’s a backdoor,” it’s better to first block outbound traffic by default in the network environment of your self-hosted model host and monitor tool calls; if you’re really worried, switch to open-source models from other vendors (such as NVIDIA, Google) for more peace of mind.
Source:Cohere
Source: KOCPC Chinese