Share with your CFO
AI inference costs, the continuous expense of running trained models against live data, now eat 85% of enterprise AI budgets according to AnalyticsWeek’s 2026 Inference Economics report, and 40% of companies are spending more than $10 million a year on AI overall. The billing structure is the problem: token counts, GPU hours, and API calls are unpredictable in ways cloud compute never was. A CDOTrends analysis argues that AI FinOps discipline is converging with data management, and that the data layer, not GPU optimization, is where the largest durable savings live.
What this means for your business
The companies most exposed here are the ones that approved AI budgets based on pilot costs and are now watching inference bills scale in ways the original business case didn’t model. Average GPU utilization across enterprise clusters sits at just 5% per Cast AI’s 2026 survey of 23,000 clusters, which means most organizations are paying for infrastructure they aren’t using while simultaneously getting surprised by the token bills they are generating. If your AI spend is growing faster than your AI headcount, that gap is almost certainly inference, not training.
The argument that data management is the underrated savings lever is the most interesting claim in this analysis, and it holds up under scrutiny. The core mechanism is straightforward: the RAG Context Tax, the cost explosion that happens when retrieval-augmented generation (a technique that feeds live company data into a model at query time to make answers more relevant) pulls in large volumes of unclassified, redundant, or irrelevant files, inflates every token count per query. Classifying and culling that unstructured data before it enters the AI pipeline shrinks both the storage bill and the inference bill simultaneously. Most FinOps conversations stop at the GPU layer because that’s where the line items are visible. The data layer is where the waste originates, and it rarely shows up on the same dashboard.
The piece is authored through a vendor-adjacent editorial lens, which tilts its emphasis toward data classification tooling as a savings mechanism rather than, say, model selection or contract renegotiation with inference providers, both of which can move the number faster in the short term. That framing doesn’t invalidate the argument, it just means the data management savings are presented as larger and more immediate than most enterprise data teams will find them on first attempt. Classifying petabytes of unstructured data is a multi-quarter program, not a quick win. The CFO who funds it based on projected inference savings needs a realistic timeline baked into the business case, not a best-case one.
The decision this reframes isn’t whether to adopt AI FinOps, that’s already happening in organizations at the $10 million spending threshold. It’s whether the data governance budget gets treated as an AI cost-reduction investment rather than a compliance overhead. If your finance team is tracking AI spend and your data team is tracking storage costs as separate line items with separate owners, you’re almost certainly double-counting the problem and underfunding the fix. The CFO who connects those two budget conversations owns a meaningful structural advantage over one who doesn’t, and the window before inference costs become a board-level topic is shorter than most finance leaders currently assume.
Concept deep-dive: RAG Context Tax
Retrieval-augmented generation works by pulling relevant company documents into a model’s context window at query time, so the model answers based on your data, not just its training. The RAG Context Tax is what happens when that retrieval process is undisciplined: bloated, unclassified file sets get ingested wholesale, inflating the token count per query, which multiplies across every inference call the organization runs. It’s the AI equivalent of photocopying an entire filing cabinet to find one memo, and billing by the page.
Based on reporting from How AI FinOps & Data Management Are Coming Together To Curb Outsize IT Spending, originally published 2026-09-21 00:43:00.
