🔍 Read the full analysis: An AI That Can Break In And Fix The Lock: What Gemini 4 Carbon’s Security Scores Really Show on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
Google says Gemini 4 Carbon scored 88.2% on SEC-bench Pro, 54.8% on ExploitGym and 71.0% on CWE-bench. Google ran the first two tests; Collinear AI runs the third, and published comparisons with rival models use differing methods. Google says Carbon is being offered first to a small group of vetted cyber defenders.
Google says its new Gemini 4 Carbon model scored highly on three cybersecurity benchmarks covering vulnerability discovery, exploit development and software repair, while restricting initial access to a small group of vetted cyber defenders. The results suggest one model can support both defensive and offensive security work, but two of the three headline scores were produced in Google-run evaluations, and the rival-model comparisons are not fully like-for-like.
Carbon scored 88.2% on SEC-bench Pro, which tests whether a model can trigger known software bugs when given information about where and what to investigate. Google reports that result on the benchmark’s May 2026 version, comprising 183 tasks across browser engines, averaged over three attempts and run with its Antigravity agent tool. OpenAI reported 85.4% for GPT-6 Astra on the same version, while Google lists Gemini 4 Argon at 83.4%. No Claude result is available on the public leaderboard, which says newer results are unavailable because of safeguard restrictions.
On ExploitGym, Carbon scored 54.8%, according to Google. The benchmark asks a model to turn a crash caused by a real vulnerability into an exploit that can run code and access a hidden test value. Google’s headline score uses a 10-hour limit; its methodology also reports 53.5% within six hours. Anthropic has reported 34.5% for Claude Opus 5.5 at six hours, but that figure comes from Anthropic’s own report. The differing time limits, tools, graders and reporting sources make the results imperfect comparisons.
Carbon scored 71.0% on CWE-bench, a test of whether a model can identify and fix flaws without breaking existing software tests. Collinear AI, which created the benchmark, runs the evaluations on a private set of 120 tasks. Its leaderboard puts Carbon three percentage points ahead of GPT-6 Astra, Gemini 4 Argon and Grok 4.7, each at 68%; Claude Opus 5.5 scored 67%. The source material says the best model still fails almost three in ten tasks on its first attempt, and five tasks defeated every model tested.
Infographic 01 · What Carbon's three scores measure
Find, exploit, fix: three tests, and only one outside referee.
Google says Gemini 4 Carbon leads all three security benchmarks it reported. Here is what each one asks of the model, and who produced Carbon's number.
1Find · SEC-bench Pro
- The model gets
- Source code, the area to search, the bug type and the expected error. Not the input that triggers the bug.
- It counts when
- The model writes an input that triggers a known, already-fixed bug in a browser JavaScript engine.
- Carbon’s score came from
- Google, 183-task May 2026 version, average of 3 attempts Google's own run
2Exploit · ExploitGym
- The model gets
- An input that crashes the program. Standard protections switched off.
- It counts when
- The model turns the crash into its own code running on the target and reads a hidden flag; an AI judge checks it used the intended bug.
- Carbon’s score came from
- Google, one attempt per task, 10-hour limit (53.5% at 6 hours) Google's own run
3Fix · CWE-bench
- The model gets
- A real open-source code base and one instruction: audit it and fix what you find. No hint what or where.
- It counts when
- The attack stops working and every existing test still passes. Models never write exploits.
- Carbon’s score came from
- Collinear AI, 120 private tasks, all models run by Collinear Outside run
Read the fine print
- Google tested without safety filters (“evaluated without guardrails”); Anthropic and OpenAI report their cyber results with safeguards off too
- Rival numbers come from the rivals’ own system cards, with their own time limits and graders
- SEC-bench Pro and ExploitGym use public bugs with public fixes; CWE-bench keeps its test set private
Finding is not discovering
- SEC-bench Pro tells the model where to look and what kind of bug to expect
- That is closer to a detective with a tip than to a hunt for unknown flaws
- Real attacks also need targets, delivery and staying undetected; real fixes need review
The same skill that proves a bug is dangerous is the skill an attacker needs.Thorsten Meyer · my reading
Sources: Google, 28 Oct 2026; Google methodology note, Oct 2026; CWE-bench leaderboard, 28 Oct 2026; SEC-bench Pro paper; ExploitGym paper; OpenAI GPT-6 Astra system card, 3 Sep 2026. Lead margin and counts as in the article; margin is my arithmetic.
Powered by Thorsten Meyer AI
Security Scores Cut Both Ways
The results matter because the same capabilities that help defenders locate and repair software weaknesses can also help someone develop an attack. Carbon’s reported exploit-development score is particularly relevant: it measures more than recognizing a flaw, asking whether a model can turn one into a working attack under the test conditions. That capability could help authorized security teams assess systems, but access controls matter because offensive use could cause harm.
For organizations, the scores are an indication of performance on specific benchmark tasks, not proof that Carbon can secure an entire company or reliably repair production software. CWE-bench’s requirement that existing tests pass is useful, but it does not establish that a patch is safe in every real-world environment. Teams adopting AI-generated fixes would still need human review, testing and responsibility for deployment.
For ordinary users, the reported scores do not mean their devices or accounts are newly exposed. Google’s restricted rollout means there is no stated general consumer release in the source material. The near-term relevance is that advanced security models are being developed with controlled access, while software updates and established security practices remain the practical protections users can rely on.
Popular right now
Philips 2200 Series Fully Automatic Espresso Machine with Classic Milk Frother
Jura E6 Automatic Espresso Machine, Piano Black (15622)
De'Longhi Magnifica XS ECAM22110B Automatic Espresso Machine
As an affiliate, we earn on qualifying purchases.
What the Three Tests Measure
The benchmarks represent different stages of security work, so their percentages should not be treated as interchangeable. SEC-bench Pro gives the model code, a search area, a bug type and an expected error; it must produce an input that triggers the known issue. It does not ask the model to find an entirely unknown vulnerability from scratch. The benchmark was developed by researchers at the University of Illinois Urbana-Champaign and includes bugs in browser engines and, in its newer version, the Linux kernel.
ExploitGym was developed by researchers led by the University of California, Berkeley, with contributors from research institutions and technology companies. It includes 869 vulnerabilities in widely used programs, V8 and the Linux kernel. In the described setup, standard protections such as memory randomization were switched off, and a second AI checked whether a submitted exploit used the intended vulnerability rather than a shortcut.
CWE-bench takes a different approach: the model receives a codebase and is asked to audit and fix it, without being told the location or number of flaws. The task counts as successful only if the attack stops working and the existing tests still pass. Google says it calculated its SEC-bench Pro and ExploitGym results using the open-source evaluation setup. The CWE-bench results come from Collinear AI’s official leaderboard; the company says Artificial Analysis independently evaluated and published them.
“self computed”
— Google, in its methodology note
Limits Behind the Reported Scores
No independent public test of Carbon across these security capabilities is identified in the source material. Google produced two of the three headline results, while the third comes from Collinear AI’s benchmark leaderboard. That difference does not establish that any result is inaccurate, but it affects how independently readers can verify the claims and compare them with vendor-reported figures.
The published figures also do not share a single testing protocol. ExploitGym results use different time limits in the comparisons cited, and reports can vary in tools, graders and other settings. SEC-bench Pro results cited for Carbon and GPT-6 Astra use the same benchmark version, but each company reports its own evaluation. The source material does not provide a full independent, side-by-side test of all named models under identical conditions.
Google says Carbon is included in the U.S. government’s voluntary pre-release testing, but the source does not describe the test’s findings or scope. It also does not specify the number of organizations receiving access, the terms they must meet, or when broader access may be considered. Benchmark results alone cannot show how the model performs on every software stack or whether a suggested fix would be safe to deploy without review.
Controlled Access and Further Testing
Google says it is making Carbon available first to a small group of vetted cyber defenders, rather than offering it broadly. The source material does not give a timetable for expanding access or name the participating organizations. The model is also undergoing voluntary U.S. government pre-release testing, according to Google.
Further evidence will depend on whether independent evaluators publish results and whether future comparisons use the same tasks, time limits, tools and grading rules. Organizations considering the model should distinguish benchmark success from operational performance, and plan for human review of AI-written patches. For consumers, the source describes no immediate product change; keeping software updated remains the relevant action.
Key Questions
What did Gemini 4 Carbon score on the security tests?
Google reports 88.2% on SEC-bench Pro and 54.8% on ExploitGym; it scored 71.0% on CWE-bench, whose results are run by Collinear AI.
Are all three results independently verified?
The source material says Google ran SEC-bench Pro and ExploitGym, while Collinear AI runs CWE-bench. It does not identify a public independent test covering all three benchmarks for Carbon.
Why is Google limiting access to Carbon?
The model can be tested on both finding and exploiting vulnerabilities as well as fixing them. Google says initial access is limited to vetted cyber defenders; the source does not provide further details about the access rules.
Does a high benchmark score mean Carbon can safely fix any software flaw?
No. CWE-bench tests fixes on a defined set of tasks and requires existing tests to pass, but the reported score does not guarantee a patch will work safely in every real system. Review and testing remain necessary.
Is Gemini 4 Carbon available to the public?
The source material describes a limited rollout to vetted cyber defenders, not general public availability, and gives no date for a broader release.
Source: ThorstenMeyerAI.com
