Share with your CTO
VAST Data and AMD are betting that the next infrastructure battleground is inference efficiency, not raw training throughput. Their expanded partnership puts AMD’s 6th-generation EPYC processors inside VAST’s CBox and EBox platforms and pairs AMD Instinct GPUs with VAST’s AI Operating System for KV cache offloading at scale. Early tests on the MI355X claim a ninefold improvement in time-to-first-token and a 9.7x gain in token throughput under high-concurrency workloads, figures AMD hasn’t independently verified. DriveNets completes a three-way reference architecture targeting AI cloud builders.
What this means for your business
The CTOs most exposed to this story are the ones currently watching GPU utilization reports that don’t match the capital spend. Underused accelerators, slow model loading, and context-handling bottlenecks are margin problems disguised as performance problems, and the infrastructure decisions that create them are often already locked in. If your inference stack was designed around training-era assumptions, where compute was the constraint and storage was an afterthought, this partnership is pointing directly at the gap you’re probably already feeling.
The specific claim VAST is making, that inference is fundamentally a data problem rather than a compute problem, deserves scrutiny. KV cache (the stored computation of previous conversation turns that a model reuses to avoid redundant processing) becomes enormous at scale, and today most teams either recompute it constantly or keep it in expensive GPU memory. Offloading it to fast storage is the right architectural instinct. The 9x figures are self-reported under unspecified workload conditions, so treat them as directional, not contractual. But the underlying logic is sound, and the teams at Core42, Crusoe, and Vultr backing this architecture aren’t doing it on vendor faith alone.
The deeper shift this signals is that storage vendors are absorbing execution-layer responsibilities. VAST isn’t selling you a place to put data; it’s selling a combined data-and-execution fabric that competes with decisions your team probably made when it chose an orchestration stack. If you’re mid-cycle on an inference platform procurement or evaluating whether to extend an existing hyperscaler arrangement, the relevant question isn’t whether AMD beats NVIDIA on specs. It’s whether your current architecture treats KV cache as a first-class infrastructure concern, and if it doesn’t, that’s the budget line to reexamine before the next renewal, not after.
Concept deep-dive: KV Cache Offloading
When a large language model processes a conversation, it generates key-value pairs representing context from prior turns, essentially a running memory of what’s been said. Storing that in GPU memory is fast but expensive and space-limited. Offloading it to high-speed external storage frees GPU capacity for new computation without losing the context. Think of it as moving a document from your desk to a drawer that opens in milliseconds. At high concurrency, this directly determines how many simultaneous users a given GPU cluster can serve before quality or cost degrades.
Based on reporting from VAST Data, AMD Expand AI Infrastructure Tie-Up for Inference Optimisation, ETDatacenters, originally published 2026-07-28 00:10:00.
