The thing being moved
When a model answers a long question over a large document set, most of the work is not generating the answer. It is reading the context — turning every token of the prompt and its supporting material into the internal state the model attends over. That state is the key-value cache, and it lives in the accelerator’s memory, which is the scarcest and most expensive memory in the building. It grows with the length of the context and again with the number of concurrent sessions; the accelerator’s memory grows with neither. When it overflows, the fallback is to recompute it, and nobody wants that.
The architecture previewed this month moves the cache off the accelerator onto networked flash, so that state can be persisted, shared and streamed back when a related request arrives — the same bet processor designers made decades ago, that predicting what will be needed next is cheaper than stalling to find out. What has changed is the cost of the stall: not a core sitting idle, but a rack of accelerators.
For anyone paying an inference bill, the logic is sound. The fixed asset is the GPU. Requests served per GPU is the denominator of cost per token, and a cache that survives between sessions raises it.
The number, and what is missing from it
The announcement’s only figure is “up to 3X” more inference requests per GPU. It comes with no benchmark, no methodology, no model, no context length, no hit rate and no description of the test. A second vendor figure — a tripling of requests with a 65 percent reduction in infrastructure cost — has the same gap behind it.
The status matters more than the multiplier. This is a preview entering a design-partner stage aimed at cloud providers, with production scheduled months away and the vendors openly seeking feedback on what the gains might be. It is an approach to understand, not a product to price. The right posture is interest without a purchase order — and, when it does ship, a measurement on your workload with your context lengths and your reuse pattern before any of it enters a plan.
The part the storage vendors are not saying
Here is the angle that matters for an organisation rather than a cloud provider. The cache holds attention state derived from the prompt and everything retrieved to answer it. If your assistant reads contracts, the cache is a transformed copy of the contracts. If your agent works over customer records, the cache is derived from those records. Persist it on networked storage, share it across sessions and stream it between servers, and you have created a new tier of enterprise data — one that did not exist last year, that the storage vendor describes purely in terms of throughput, and that your data governance has never heard of.
This is the pattern we keep seeing as AI is integrated into existing estates: every optimisation of the inference path creates a new place where enterprise data lives. Vector indexes were the first. Retrieval caches were the second. Persisted model state is the third, and it is the least visible, because it does not look like data — it looks like performance infrastructure.
The questions are the familiar ones, asked of an unfamiliar object. What is in it, and what classification does that inherit? Who can read it — can one session’s cached state be reached by another user’s request? How long is it kept, and is that consistent with the retention rules on the source documents? Is it encrypted at rest on that networked flash, and where is that flash? None of these are hard to answer if they are asked at design time. All of them are awkward to answer after an auditor asks first.
Planning for it
Two practical conclusions. First, assume a working-memory tier is coming to your inference infrastructure, whether from this preview or the several like it, because the economics are too strong to resist — and size it from your own measurements of context length and reuse, not from a vendor multiplier. Second, put it on the data platform’s map now: classify it, set its retention, decide who can read it, and log access to it the way you would any other store derived from governed data.
The vendors are right that where the model keeps its memory decides what inference costs. They are quiet about the fact that it also decides where your data is. Both are true, and the second is the one that ends up in front of a regulator.
