Share with your CTO
VAST Data is betting that inference, not training, is where AI infrastructure money gets spent next, and it’s building the coalition to prove it. The company announced an expanded partnership with AMD pairing the VAST AI Operating System with AMD Instinct GPUs, 6th Gen EPYC CPUs, and Pensando Pollara 400 networking cards. The headline number: internal testing on an AMD Instinct MI355X showed 9x faster time-to-first-token and 9.7x more token throughput using VAST’s KV cache offloading under high-concurrency agentic workloads. Cloud providers Core42, Crusoe, Vultr, and TensorWave are already shipping on the stack.
What this means for your business
The companies whose AI infrastructure decisions this story touches most directly are those running or planning large-scale inference, meaning organizations past the pilot stage where GPU count was the only metric that mattered. If your team is still sizing infrastructure around training throughput, this announcement is a signal that the design assumptions are shifting. The question is whether your current storage and networking architecture was chosen when inference was an afterthought, because the gap between training-optimized and inference-optimized infrastructure is now measurable in dollars and latency.
The specific technical bet here is that KV cache, the mechanism by which inference systems store intermediate reasoning state to avoid recomputing it on every token, becomes a storage and data management problem at scale, not just a GPU memory problem. When agents run long multi-turn conversations or complex reasoning chains, the KV cache grows large and expensive to hold entirely in GPU HBM (the high-bandwidth memory on a GPU chip). VAST’s argument is that offloading that cache to fast NVMe-backed shared storage, connected via high-speed RDMA networking, is cheaper and faster than buying more GPUs to absorb the memory pressure. The 9x TTFT claim, if it holds in customer deployments, makes that argument hard to dismiss, though VAST self-reported the numbers and AMD explicitly did not independently verify them.
The coalition angle matters as much as the benchmarks. VAST is assembling a reference architecture with AMD, DriveNets, TensorMesh, and EmbeddedLLM that gives mid-tier AI cloud providers a validated, non-NVIDIA path to production inference. That’s a direct play on the opening AMD is creating in the market, and it narrows the gap between buying an H100 cluster and buying an Instinct cluster with something credible to run on it. For enterprise CTOs evaluating whether AMD-based infrastructure is ready for production workloads, a live ecosystem with named cloud operators carries more weight than any single benchmark.
The automated KV cache lifecycle management feature, which ties VAST’s data expiration policies to cached inference state, is the piece most likely to move a CISO from observer to sponsor on this conversation. Regulatory compliance around AI-generated data retention is an unsolved problem for most enterprises, and baking expiration into the storage layer rather than bolting it onto the application is architecturally cleaner. If your organization has an AI deployment that touches personal data, the question of where inference state lives and how long it persists is one that belongs in your next infrastructure review, not after your next audit.
Concept deep-dive: KV Cache Offloading
During inference, a language model generates text one token at a time and must remember the context of every previous token in the conversation. The KV cache stores that context so the model doesn’t recompute it from scratch with each new word, think of it as the model’s working notepad. That notepad lives in GPU memory, which is fast but scarce and expensive. Offloading it to high-speed shared storage frees GPU capacity for more concurrent requests, directly lowering cost per query and improving throughput at scale.
Based on reporting from VAST Data Expands Collaboration with AMD to Advance AI Infrastructure for the Inference Era, originally published 2026-07-24 15:22:00.
