MANY MINDEDAI Search Benchmark Reveals Cost Trade-offs for Developers
Artificial Analysis released the 'Search Index' benchmark, testing seven AI search API providers—Parallel, Exa, Firecrawl, You.com, Tavily, Keenable, and Brave—using GPT-5.6 Luna in the Stirrup framework. Each provider was evaluated across 25 runs per task using three standardized tests: DeepSearchQA (900 research questions), BrowseComp (200 hard-to-find facts), and AA-Omniscience (600 questions across six knowledge domains). The benchmark shows Parallel Search (advanced) reduces token usage by 40%+ compared to its Basic version, with per-task costs at $0.084 versus $0.11 for Basic. Parallel Search (turbo) also delivers 0.51 seconds response time versus 1.03 seconds for Basic. Crucially, the benchmark does not reduce knowledge costs for end users or expand public access to information. Its scoring system—where models achieve 65–75 points with search access versus 33 points without—does not guarantee improved knowledge retrieval for real-world applications. This tool helps developers compare API performance but remains a technical metric for internal use, not a pathway to free knowledge. The source explicitly states per-task costs increase while total costs decrease, meaning efficiency gains don't translate to lower public knowledge expenses. What matters next is whether developers adopt these efficiency metrics to build cost-effective tools—without implying public access improvements. The benchmark itself has no public knowledge access implications, as noted in the source's caveats.
Source: The Decoder