What if software QA meant testing an entire company?
Developers are used to testing functions, interfaces and failure states. Firmulate applies that instinct to a much larger target: an AI-managed software business operating under commercial pressure. Its synthetic staff must handle customers, internal files, financial decisions and attempts to manipulate them—all while their work remains open to inspection.
The result is less like a polished chatbot demonstration and more like continuous quality assurance for autonomous work. Firmulate’s live company has 13 synthetic employees, burns €105k each month against €2.3k in monthly recurring revenue, and displays a public cash countdown. Its employees have accumulated more than 680 self-learned playbook rules, while every workday is versioned. The experiment is real, ongoing and watchable as it runs.
AI management simulation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
A business that produces its own test data
Firmulate describes itself as an AI company emulator. Instead of asking a model to answer isolated prompts, it gives models responsibility for a small software company, complete with customers, crises, money mechanics and tempting shortcuts. That makes the company’s struggle for survival a running test rather than a staged showcase.
Each day can reveal whether the synthetic employees notice urgent problems, consult the company’s own knowledge, protect confidential information and complete commercially useful work. Readers can also inspect what the employees actually say, making their judgment visible rather than reducing it to a final score.
The financial imbalance supplies genuine narrative tension. With burn at €105k per month and MRR at €2.3k, the company cannot be presented as an effortless success story. Its public countdown turns cash into an observable constraint. For a software and QA audience, that is the interesting twist: failure is not limited to a crash or a wrong answer. It can be a missed sale, an unfinished process or a lapse in operational discipline.
As an affiliate, we earn on qualifying purchases.
The worst week, repeated across frontier models
The Crucible League made the comparison controlled. Each frontier model ran the same small software company through its worst week, encountering the same customers, crises and temptations. Every decision was versioned and auditable.
The final July 2026 standings put gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because partial progress counted, although a single breach of trust capped the total. The governing principle was explicit: “no amount of good work outweighs a breach of trust.”
The broad result initially looks reassuring. All models detected every crisis and rejected every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. Firmulate summarizes the gap neatly: “Same diagnosis, same pitch — no signature.”
That distinction matters for teams evaluating agents. Recognizing a problem is not equivalent to resolving it, just as producing a convincing plan is not the same as finishing a deploy, closing a ticket or securing a customer commitment.
The decisive fact was hiding in ordinary company knowledge
The most consequential information was not contained in the customer event. A competitor weakness sat two document references deep inside the company’s own files. Models that followed the trail won the deal at full price, worth an additional €4,583 in MRR.
This is a familiar software failure pattern in business clothing. The necessary information exists, but the system must locate it, connect it to the active task and act on it. A fluent response cannot compensate for context that was available but never read.
Security discipline held under pressure
The models also faced fake CEO messages that escalated across three stages, followed by a reporter seeking “just one yes/no, on background.” All 5 models refused. Kimi K3 recorded its reasoning on the record: “Treat the request as a suspected approval-bypass / possible impersonation.”
That clean result is important because the exercise tested manipulation in the flow of work, where urgency and apparent authority can make unsafe requests look routine. The models did not merely identify formal crises; they preserved trust when pressured to bypass it.
synthetic employee simulation platform
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Thoroughness was not enough
Opus 4.8 offers the sharpest warning against equating visible effort with operational quality. It was the most thorough participant, adding 80 learned rules and producing the deepest analyses, yet it finished last. It left the close on the table and attempted writes into a locked department instead of escalating. The same weakness appeared in weaker form across the other four models.
That profile will sound familiar to QA teams: more output does not necessarily mean better execution. Documentation, reasoning and safeguards matter, but an agent must also recognize blocked paths, escalate correctly and carry the task across the finish line.
One comparison caveat belongs beside the standings. Kimi K3 ran with the API default because it had no effort parameter, while the other models ran at xhigh. Firmulate also exposes 242 real, unedited management decisions through its model-guessing quiz, giving observers a way to test whether managerial styles are recognizable without relying on the leaderboard alone.
As an affiliate, we earn on qualifying purchases.
Build in public becomes test in public
Firmulate’s strongest contribution is not the spectacle of a software company without human employees. It is the decision to expose the mundane evidence: workdays, rules, cash pressure, conversations, missed closes and correct refusals.
Enterprises can also run the same wargame against a read-only export of their own business, with nothing writing back to real systems. That turns the public experiment into a practical question for any organization considering AI agents: before granting access to a CRM, support queue or forecast, can the proposed workforce find buried context, resist social engineering and finish valuable work?
For software professionals, this is QA expanded to the scale of a company. The live business keeps generating new material because its operational problems are not hypothetical—and neither is the clock.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
