Share with your CTO
Across 107 enterprises, AI infrastructure spending is accelerating well ahead of the ability to measure it. Only 21% run AI in production at scale, yet 45% plan to evaluate AI-specialized clouds (CoreWeave, Lambda, and peers) that fewer than 2% currently use. Meanwhile, 83% of GPU-operating enterprises report utilization at or below 50%, and fewer than half track compute costs rigorously. A clear majority, 64%, plan to switch or add an infrastructure provider within twelve months, many within a single quarter.
What this means for your business
The pattern here has a name: spend-ahead-of-signal. Enterprises in the mid-market are committing capital to a re-platforming they haven’t yet validated, toward a category of specialized AI clouds they’ve barely touched, while the infrastructure already running sits half-idle and largely unmetered. Where you stand on this depends on one question: does your organization have a cost-per-workload view of its current AI compute, or are you flying on total spend and gut feel? The answer to that question is more diagnostic than your vendor roster.
The buying-criteria data is the sharpest thing in this report, and it deserves more attention than the utilization numbers. Integration with the existing stack ranked first at 41%, total cost of ownership second at 35%, and cost per million tokens dead last at 8%. That’s a rational hierarchy, except that 56% of respondents either track TCO only partially or can’t quantify it at all. Enterprises are telling vendors “we decide on total cost of ownership” while simultaneously admitting they can’t calculate it. That gap is what makes the switching wave dangerous: organizations are about to make foundational architecture moves on criteria they’re not yet equipped to measure honestly.
The memory-bandwidth finding in Finding 8 is the under-discussed one. As inference workloads scale, the binding constraint shifts from raw GPU compute to KV-cache memory capacity (the working memory a model needs to process long conversations or documents). Roughly one in five enterprises either don’t recognize this constraint or haven’t begun addressing it. That’s the next version of the same gap arriving before the current one closes. If your inference volumes are growing and your architecture review hasn’t touched memory hierarchy in the last two quarters, the utilization problem you have today will look cheap compared to the latency and cost problem coming. I’d revise that assessment only if the specialized-cloud evaluation wave in the next six months produces actual deployment data showing enterprises have figured out workload placement faster than past cycles suggest.
Concept deep-dive: GPU utilization
GPU utilization measures what fraction of a graphics processing unit’s compute capacity is actively doing useful work at any given time, think of it like seats filled on a flight that costs the same whether it’s half-empty or full. Low utilization means you’re paying for capacity that sits idle. For AI workloads, utilization tends to be low when models run in bursts, when teams over-provision to avoid queuing delays, or when there’s no orchestration layer routing jobs efficiently across available hardware. It’s the most direct proxy for infrastructure waste.
Based on reporting from The AI compute gap: Enterprises are buying infrastructure faster than they can measure what it costs, originally published 2026-07-16 15:16:00.
