As AI becomes increasingly powerful, many people are surely wondering whether it can actually find vulnerabilities in cybersecurity scenarios. A foreign security researcher, Kasra Rahjerdi, recently conducted this experiment. He deliberately designed a fake app with vulnerabilities, then spent $1,500 to test whether models like GPT-5.5, Claude, Gemini, DeepSeek, Qwen, and Kimi could find the real vulnerabilities.
Surprisingly, I expected Claude to lead in this area, but GPT-5.5 took the crown, succeeding 7 out of 10 tests – the highest success rate this round, and this number far outpaced the competition. As for cost efficiency, DeepSeek V4 Pro performed quite well, averaging only $0.62 per success. As for Gemini, its performance this time was a bit disappointing, with many runs being rejected early on for safety reasons, refusing to even try.
GPT-5.5 succeeded 7 out of 10 times in cybersecurity testing, DeepSeek V4 Pro costs only $0.62 per problem solved, and Gemini nearly refuses outright
According to Kasra RahjerdiThis time the target is a fake book review app with a React Native / Expo frontend and a Python backend. On the surface, it appears to be an app for viewing book reviews, rankings, and user profiles. The objective is to find the flag hidden in a private review from one of the users.
Image source:Kasra Rahjerdi
Kasra also mentioned that the API of this app is actually relatively secure, with the real problem lying in the data layer. Since the app uses Firebase and the internal google-services.json contains Firebase-related information, the right approach isn’t to keep trying to break through the API, but to find vulnerabilities in Firebase permission settings, and then access data in Firestore that shouldn’t be accessible to external parties.
For testing, he originally wanted to run each model 10 times, but the costs had already climbed to $1,500, so he had to pull the plug midway. He also stressed that this wasn’t a formal scientific evaluation, but rather a documented personal experiment with a limited sample size.
Each test has a budget limit of $10 and an execution time limit of 2 hours.
Besides Claude using Claude Code’s -p mode, most other models run through the pi testing harness, combined with the pi-goal-x extension, with the goal of allowing models to continuously try different approaches to complete tasks even when they encounter failures or get stuck.
Image Source:Kasra Rahjerdi
Overall, GPT-5.5 performed the best, succeeding 7 out of 10 tests with a success rate of 70%. The average cost per test was $6.62, the cost per successful solution was $9.46, and the median token usage was approximately 260k.
Kasra observed that in most successful cases, GPT-5.5, after unpacking the APK, quickly shifts focus to Firebase rather than getting caught up in the API or the React Native App itself, which also became its most obvious advantage in this task.
Although DeepSeek V4 Pro’s success rate is not as good as GPT-5.5—with 3 successes out of 10 tests—its cost efficiency is quite outstanding. It costs only $0.19 per test on average, and the cost per successful problem solved is only $0.62, which is significantly lower than GPT-5.5.
Claude’s performance was just mediocre, Sonnet 4.6 and Opus 4.8 both had 2 successes out of 10 attempts, but the cost was on the high side. Kasra mentioned that Opus 4.8 actually came very close to the answer a few times, but was stopped in the later stages by safety guardrails—not like Gemini which rejected almost from the start.
Gemini was clearly affected by safety policies this time. Both Gemini 3.1 Pro Preview and Gemini 3.5 Flash achieved 10 successes and 0 failures, with Gemini 3.1 Pro Preview’s median token usage at only 9k, significantly lower than other models, indicating most stopped very early and rejected the task.
Image source:Kasra Rahjerdi
As for the other models, DeepSeek V4 Flash, MiniMax M2.7, and Step 3.7 Flash all had 0 successes out of 10 attempts.
Qwen 3.7 Max failed 6 consecutive attempts. Kasra was quite disappointed with Qwen’s results, as it was one of the few non-GPT models capable of completing the task during local testing, but the formal long-duration test failed to reproduce that success.
Source: KOCPC Chinese