Vals AI: Multi-Agent Teams Cost Up to 5x More for Rare Quality Gains

Splitting software engineering tasks across a team of coordinated language models sounds logical, but benchmark numbers show a steep cost penalty. According to an evaluation published on October 9 by Vals AI, multi-agent configurations cost between 1.8 and 5.1 times more than solitary agents while offering statistically significant quality gains in only one of four tested scenarios.

The benchmark, dubbed Vibe Code Bench, tasked agents with taking a technical specification and delivering a functional web application. Each output was deployed in a container via Docker Compose and validated using automated browser tests across 50 distinct applications. Researchers evaluated two frontier models—GPT-6 Sol and Claude Opus 5.5—at medium and maximum reasoning effort levels, running each setup once.

In team configurations, a coordinator agent could spawn up to five parallel sub-agents to delegate development and reconcile components. The lone clear winner was GPT-6 Sol running at medium reasoning effort: its test pass rate climbed from 77.6% to 84.9% (p = 0.005). The other three comparisons showed differences that failed the threshold for statistical significance.

The economic trade-off was sharp. For GPT-6 Sol on medium effort, the single-agent run averaged $1.22 per application, while the multi-agent team consumed $3.07. At maximum reasoning effort, solo GPT-6 Sol achieved an 89% score for $4.82, while the multi-agent setup reached 90.4% at $8.54. The team configuration did occasionally finish faster, trading raw token spend for wall-clock concurrency.

Delegation strategies also varied by model family. GPT-6 Sol acted as an aggressive project manager, instantly offloading independent modules to parallel sub-agents. Claude Opus 5.5 took a more bureaucratic approach, drafting shared interface contracts first and scheduling sub-agents sequentially.

The evaluation carries notable caveats: it tested a single benchmark focused purely on web development, and each of the eight configurations was executed only once. The results do not rule out multi-agent utility in other domains, but they do demonstrate that throwing autonomous sub-agents at code is far more reliable at multiplying API bills than improving software quality.

Source: habr

Primary source: Vals AI