Share with your CTO
Nvidia is betting that proprietary hardware wins the inference memory war. Its CMX platform pairs 64 BlueField-4 DPUs with roughly 9,600 TB of SSD storage per rack, extending KV cache capacity (the working memory that lets AI models handle long conversations without restarting from scratch) far beyond the 13.4 TB of on-chip HBM in a GB200 NVL72 rack. Nvidia claims up to 5x gains in both token throughput and power efficiency. The open alternative, the CXL standard, takes a different route: Marvell’s CXL switching hardware demonstrated a 4.8x inference throughput improvement and an 82.7% drop in time to first token using a 16 TB pooled DRAM approach.
What this means for your business
Your inference cost problem is fundamentally a memory problem. Compute is expensive and often idle because GPUs are waiting for KV cache data that doesn’t fit on-chip. The architectural choice now facing infrastructure teams is whether to solve that through Nvidia’s tightly integrated CMX stack or through CXL, which runs on the PCIe interconnect that nearly every server already supports. If your organization has already standardized on Nvidia hardware, CMX is the path of least friction. If you’re building a multi-vendor environment or facing pressure to reduce single-vendor dependency, Vera CPUs support CXL 3.1, meaning the choice isn’t binary even within an Nvidia footprint.
The CXL adoption numbers deserve scrutiny. Yole Research estimates that two-thirds of servers were CXL-capable by Q1 2025 but near zero percent are actually CXL-enabled, rising only to 13% by 2030. That gap between capability and deployment is where the real risk sits. CXL’s vendor-agnostic appeal is genuine, but “capable” hardware sitting unused means your organization bears the integration work that Nvidia’s STX reference architecture handles automatically for CMX. Google’s TPU 8i, which shares 331.8 TB of HBM coherently across 1,152 chips compared to Nvidia’s 20.7 TB coherent at rack scale, shows what pod-wide memory sharing can unlock for frontier model inference. That’s the architectural pressure Nvidia is responding to with CMX, and it’s real.
The procurement decision this reframes isn’t GPU quantity. It’s whether your next infrastructure refresh allocates budget toward modular memory offload capacity rather than raw accelerator count. CMX is additive and doesn’t require a new GPU purchase. CXL memory controllers like Marvell’s Structera X draw under 30 watts versus 150 to 700 watts for an incremental CPU or GPU. Either path changes the cost-per-token math before your next contract renewal with a hyperscaler or colocation provider. The vendor you’d revisit isn’t Nvidia specifically. It’s whoever currently holds your memory and networking budget.
Concept deep-dive: KV Cache
The KV cache is a GPU’s short-term scratch pad for an ongoing AI conversation. Every token the model processes generates key-value pairs that must be stored and retrieved to generate the next response. Think of it as the model’s working memory: lose it, and the system recomputes from scratch, wasting GPU cycles. As context windows grow from thousands to millions of tokens, the cache outgrows on-chip memory, and the battle to store and retrieve it cheaply is now the central cost driver in production inference.
Based on reporting from Nvidia, CXL, and the Battle to Improve AI Inference Economics, originally published 2026-07-17 15:47:00.
