🔍 Read the full analysis: FrontierCode: A Practical Guide To AI-Generated Pull Requests on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, testing 23 AI models on coding tasks judged for review quality as well as correctness. Claude Opus 5.5 led with a 52.2% score, but cleared every blocking check in 57.7% of attempts; cost per task varied from $0.09 to $10.15 among highlighted models.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that asks whether a repository maintainer would accept a generated change, rather than merely whether it passes tests. Claude Opus 5.5 scored highest at 52.2%, but cleared all blocking checks in 57.7% of attempts, indicating that even the leading model did not meet the benchmark’s acceptance requirements in more than two in five attempts.
The results, added to Artificial Analysis’s evaluations on October 12, 2026, place GPT-6.1 Sol second with a 49.1% score and Claude Fable 5.1 third at 49.0%. Artificial Analysis ran its models through the same open-source agent harness, Stirrup, with up to 250 turns per attempt and three attempts per task. The review stage used GPT-6 Luna (medium) as the judge.
FrontierCode’s full benchmark contains 150 tasks; the Artificial Analysis evaluation used its 100-task Main set. The source report says nine of those 100 tasks were not solved by any model on any attempt. On the benchmark’s pass-rate measure, Claude Opus 5.5 reached 57.7%, while GPT-6.1 Sol reached 54.7% and Claude Fable 5.1 reached 54.3%. These figures refer to attempts that clear every blocking check, not the overall score.
Results among open-weight models were close to some leading proprietary systems: MiMo-V2.6-Pro scored 45.4%, followed by DeepSeek V4.1 Flash at 44.0% and Kimi K3 at 43.6%. Artificial Analysis reported per-task costs ranging from $0.09 for GPT-6 Luna (max) to $10.15 for Claude Opus 5.5 among the highlighted models. The source says GPT-6.1 Sol scored 49.1% at $0.81 per task.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Passing Tests Is Not Enough
The results matter for software teams weighing how much review to delegate to coding agents. FrontierCode treats a change as unsuccessful if it fails any blocking check, even when other tests pass. Its score also accounts for patch quality, including whether the change stays within scope and whether its tests detect the original bug. That makes its results a different measure from benchmarks centered mainly on functional correctness.
The report’s figures do not show that AI-generated pull requests cannot be used; they show that performance varies by task and that the leading model still fails at least one blocking requirement in a substantial share of attempts. Teams considering these tools can use the benchmark as evidence that human review remains relevant, while the close scores among several models and large cost differences may inform which systems they test in their own workflows.
Cost and score do not move in lockstep. Artificial Analysis identifies GPT-6 Luna (max) at 41.9% for $0.09 per task, MiMo-V2.6-Pro at 45.4% for $0.18, GPT-6.1 Sol at 49.1% for $0.81 and Claude Opus 5.5 at 52.2% for $10.15. The report calculates that the step from Sol to Opus adds 3.1 score points at roughly 12.5 times the cost. Those are benchmark-specific estimates, not a guarantee of comparable performance or spending in a company’s codebase.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Scores Patches
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026; version 1.1 followed in July with rules against looking up task solutions online. According to the source report, more than 20 maintainers created tasks from 36 repositories, including Celery, Budibase, uppy and Mattermost, spending more than 40 hours on each task. The benchmark tasks are private, intended to prevent models from having encountered their solutions during training.
The evaluation checks patches across six areas: hidden tests, command checks, reverse tests, adaptive tests, scope limits and an LLM code review against maintainers’ criteria. Reverse tests check that an agent’s own test fails on the original code; otherwise, the test may not establish that the bug was addressed. Blocking checks determine whether an attempt passes; quality checks affect the score for attempts that clear them. The source also says attempts flagged by Cognition’s trajectory scanner for looking up a solution receive zero.
Artificial Analysis’s results also differ from general model rankings in the source report. Claude Sonnet 5.5, listed second among the compared models on the Intelligence Index, ranks fifth on FrontierCode. Grok 4.7 ranks 18th on FrontierCode in that comparison, while DeepSeek V4.1 Flash and GPT-6 Luna rank better there than their Intelligence Index positions. This suggests that a general capability score alone does not describe performance on this particular code-review benchmark.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, as quoted in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The results are benchmark measurements, not a forecast of how often a model will produce an acceptable pull request in any particular organization. The source describes a 100-task evaluation set, but does not provide enough detail here to determine how closely its tasks represent every language, repository, team review practice or production workload. The reported scores also rely on a specific agent harness, attempt limit and review model.
Several models were absent from the leaderboard as of October 13, 2026, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their FrontierCode results were not available in the source material, so comparisons with those systems remain open. The reported costs are per task under the evaluation setup; the source does not establish that they will match costs in other settings.
As an affiliate, we earn on qualifying purchases.
Awaiting More Model Results
The next development to watch is whether Artificial Analysis adds results for the models missing from the October 13 listing, particularly those the source report identifies as having no FrontierCode score. Further evaluations could change the model rankings and provide a fuller comparison across providers.
For now, readers should treat the published leaderboard as a dated snapshot of 23 models on the 100-task Main set. The source material does not announce a date for the next update or a planned expansion. Maintainers and engineering teams can compare these results with their own review standards, but the benchmark alone does not settle how much oversight a specific AI-assisted workflow needs.
AI code quality evaluation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates whether an AI-generated code change meets a set of repository-specific requirements, including correctness, regression checks, scope limits and code review. An attempt that fails a blocking check scores zero.
Which model led the reported evaluation?
Claude Opus 5.5 (max with fallback) ranked first with a 52.2% FrontierCode score and a 57.7% pass rate, meaning it cleared every blocking check in that share of attempts.
Does the top score mean the model can handle pull requests without review?
No. In the reported evaluation, the leading model failed to clear all blocking checks in 42.3% of attempts. The benchmark does not establish that every failure would occur in a real company workflow, but its results do not support treating the score as proof that review is unnecessary.
How much did the tested models cost per task?
Costs varied by model and evaluation setup. The source reports $0.09 per task for GPT-6 Luna (max), $0.18 for MiMo-V2.6-Pro, $0.81 for GPT-6.1 Sol and $10.15 for Claude Opus 5.5.
Which models were missing from the October 13 results?
The source report lists Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as having no FrontierCode result at that time. It does not state when they may be evaluated.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
