TL;DR
Firmulate’s July 2026 management benchmark found that five frontier AI models identified every simulated company crisis and resisted every manipulation attempt. Only two completed a €55,000 contract, suggesting that correct analysis does not reliably produce authorized, finished work.
Firmulate’s July 2026 AI management benchmark found that five frontier models recognized every crisis and rejected every manipulation attempt during a simulated company’s worst week, but only two completed a €55,000 customer contract. The result points to a consequential gap for businesses adopting AI agents: producing a correct analysis and turning it into completed, authorized work are separate capabilities.
Firmulate gave each model control of the same small software company, which has 13 synthetic employees, monthly spending of €105,000 and monthly recurring revenue of €2,300. Decisions were versioned and auditable, while a public cash countdown recorded the cost of delay. Firmulate said every model diagnosed the crises, developed the customer pitch and resisted attempts to manipulate its decision-making.
The commercial test hinged on information hidden two document references deep in company files. According to Firmulate, the models that traced the evidence could support a full-price close worth €4,583 in added monthly recurring revenue. Yet only two models obtained the final signature, even though the field reached broadly similar diagnoses and pitches.
The July league table ranked gpt-5.6-sol first with 95 points, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline received 26 because the scoring system awarded partial progress. Firmulate also disclosed that Kimi K3 used the API’s default effort setting, while the other models ran at xhigh, limiting direct comparisons.
Execution Separates Similar AI Analysis
The findings suggest that businesses cannot judge an agent solely by whether it finds the right answer, writes a persuasive message or recognizes a security threat. Operational value also depends on whether the system can investigate incomplete evidence, follow internal controls and carry work through the final authorized step.
That distinction matters in sales, customer service and business operations, where an incomplete action can carry a real cost even when the preceding analysis is accurate. In this experiment, the expensive failure was not a false conclusion. It was a correct recommendation that remained unfinished.
The results also separate safety recognition from broader management performance. All five models refused fake CEO messages and a reporter’s attempt to obtain an informal answer, Firmulate said. Because manipulation resistance was consistent across the field, execution discipline became the main dividing line.
Principles of Agentic AI Governance: A Playbook for Managing AI Risk, Fairness, and Compliance (Agentic Governance and Architecture)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
A Company Built for Auditing
Firmulate designed the simulated company to expose behavior that short chat demonstrations may miss. Its workforce has accumulated more than 680 self-learned playbook rules, and each workday creates a versioned decision record. The environment requires models to connect evidence across files, work through approved channels and respond while financial pressure builds.
The test also imposed a strict trust constraint. Firmulate said a single trust breach capped a model’s score, reflecting the view that productive work cannot offset unauthorized conduct. None of the five models fell for the staged social-engineering attempts, including fake executive messages that escalated across three stages.
Opus 4.8 illustrated the difference between depth and completion. Firmulate described it as the most thorough participant, with deep analyses and 80 new rules learned, but it finished last after leaving an approved close incomplete and attempting to write into a locked department rather than escalating through the permitted route.
“Same diagnosis, same pitch — no signature.”
— Firmulate’s summary of the commercial test
THE AI OPERATING MODEL: How CEOs, CIOs, CTOs, CDOs, and Enterprise Leaders Redesign Governance, Roles, Platforms, and Decision Rights for AI-Native Execution
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Repeatability Has Yet to Be Shown
The supplied results do not establish whether the rankings would remain stable across repeated runs, different prompts or other company settings. The benchmark comes from Firmulate, and the source material does not describe an independent audit or peer-reviewed validation of its scoring system.
The unequal effort settings also leave uncertainty around direct model comparisons. Kimi K3 ran with the API default effort level, while the others used xhigh. It is also unclear from the supplied material which specific models completed the €55,000 signature or how much performance might change under equalized configurations.
Workflow Automation with Microsoft Power Automate: Design and scale AI-powered cloud and desktop workflows using low-code automation
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Live Records Face Further Scrutiny
Firmulate is keeping the company experiment live and has published a benchmark page, decision records and a quiz based on 242 unedited management decisions. Further runs may show whether closing strength and operating discipline persist across scenarios rather than appearing in a single simulated week.
For prospective AI buyers, the next practical step is to test agents against read-only exports of internal business data before granting operational authority. Such trials can measure whether a system finds evidence, respects permissions and finishes assigned work under realistic pressure.
AI project auditing tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What did the Firmulate experiment test?
It tested whether five frontier AI models could manage the same simulated software company through customer, financial and security crises while producing auditable decisions.
Did the models understand the business problems?
According to Firmulate, all five identified every crisis, rejected each manipulation attempt and developed an appropriate sales pitch. Their performance separated at the point of completing authorized action.
How many models completed the customer deal?
Two of the five models completed the €55,000 contract. The supplied source does not identify those two by name, so the league rankings should not be used to infer them.
Which model received the highest score?
gpt-5.6-sol ranked first with 95 points, ahead of Kimi K3 at 93. The comparison carries an effort-setting caveat because Kimi K3 used the API default while the others ran at xhigh.
What should companies measure before deploying AI agents?
Alongside reasoning and safety, companies can test whether agents trace evidence across internal records, respect approval boundaries and complete commercially meaningful tasks without writing to live systems during evaluation.
Source: Thorsten Meyer AI
