Imagine relying on an AI system to handle your company’s critical decisions—only to find that despite exhaustive rules and thorough analyses, it fails to close a deal. This isn’t science fiction; it’s the real-world experiment conducted by Firmulate, a company testing the limits of AI in managing a mock software business. If you’re invested in how AI impacts decision-making and trust, the lessons from this experiment are crucial for your financial and strategic future.
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Testing AI in the Business Arena
Firmulate recently ran a public, transparent experiment involving four leading AI models—gpt-5.6-sol, Kimi K3, Sonnet 5, and Fable 5—each tasked with navigating the worst week of a small software company. The goal? To see whether these models could make sound decisions during crises, avoid manipulation, and ultimately close a critical €55,000 deal.
This isn’t just about AI chat performance; it’s about AI performing as a decision-maker in high-stakes, real-world scenarios. Every choice was carefully documented, versioned, and auditable, providing a unique window into AI behavior when faced with pressure, deception, and complex information.
Products Worth Considering
Apple MacBook Pro 2023 with M2 Pro Chip, 14‑inch, 16 GB RAM, 512 GB SSD – Space Gray
FLEXISPOT EN1 Electric Standing Desk, 48 x 24 in, Black
Apple Mac mini Desktop Computer with M4 chip, 10‑core CPU and GPU, 24GB Unified Memory, 512GB SSD, Gigabit Ethernet
As an affiliate, we earn on qualifying purchases.
Key Findings: Diligence Alone Isn’t Enough
All four models demonstrated impressive capabilities: they identified all crises and refused every manipulation attempt, including social engineering tricks such as fake CEO messages and reporter tricks—an area where AI is expected to falter. Yet, despite this vigilance, only two models managed to close the deal and secure the full payment.
The standout performer, gpt-5.6-sol, scored 95 out of 100, successfully uncovering buried facts in the company’s internal files—a critical piece of information needed to clinch the deal. Kimi K3 scored 93, closing the deal cleanly with the best discipline among the models. Sonnet 5 and Fable 5, despite promising efforts, fell short with scores of 88 and 77 respectively, mainly due to process slips and leaving crucial opportunities on the table.
The Hidden Weakness: Overlooking the Deep Context
Interestingly, the decisive weakness wasn’t in reacting to crises or avoiding manipulation; it was in reading and leveraging internal documentation. The models that delved two document references deep into the company’s files uncovered vital insights that boosted their chances of closing the deal at the full price (+€4,583 MRR). Those that relied on surface-level information or failed to prioritize reading deeply missed out on this opportunity.
Implications for Business and Finance
This experiment underscores a vital truth for businesses: diligence and volume of rules or analyses aren’t enough. For AI to truly be effective in decision-making roles, it must prioritize the most impactful information—reading deeply, understanding context, and knowing what to ignore. Simply put, more effort doesn’t always produce better results if it’s misdirected or unfocused.
The Human Factor and Trust
Moreover, the models’ refusal of manipulative social engineering attempts — including staged CEO messages — illustrates that current AI systems can uphold integrity under pressure. This is reassuring for enterprises considering AI for support, compliance, or decision-making. But the fact remains that without strategic focus, even the most thorough AI may leave opportunities on the table.
What This Means for Your Investments
If you’re managing your personal finances or investments, the takeaway is clear: quality of work isn’t just about how much effort is put in, but what is prioritized. AI systems that focus on the right information, read deeply, and stay disciplined under pressure can be more valuable than those that simply do more.
Firmulate’s live platform offers a transparent window into this dynamic, allowing enterprises and investors alike to test their AI tools against real-world scenarios before deployment. As the experiment shows, diligence alone doesn’t guarantee success—smart prioritization does.
In business decision-making, more effort isn’t always better. AI systems need to prioritize impactful information and stay disciplined under pressure to truly deliver value—less volume, more focus, better results.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
