How DeepSeek Splits Its KV Cache Between Host Memory and SSD

DeepSeek's paper on this model names prefill compute and KV cache storage as the two biggest costs left in deploying an agentic model at scale. I read the paper itself along with the deployment Q&A DeepSeek published alongside their open-sourced inference engine, DeepInfer, to see exactly how they manage that cache on SSD. Both are DeepSeek's technical writing, not secondhand coverage. They go into more depth than most of what's been written about this model.

Shrinking the cache before it ever gets stored

DeepSeek-V4.1-Flash splits its 40 Transformer layers into a 20-layer causal encoder and a 20-layer decoder. In a normal Transformer, every layer computes its own key-value pairs from its own hidden state. Here, the decoder layers skip that. Their main KV is projected directly from the encoder's final hidden state instead.

Each decoder layer still needs its own small sliding-window cache though. Reconstructing that exactly would mean running the last several thousand tokens back through all 20 decoder layers on every prefill. DeepSeek avoids that cost with a technique called bounded replay. Each prefill only recomputes the most recent 128 tokens for the sliding-window piece instead of reconstructing the full window. Together, these two pieces are what DeepSeek credits with nearly halving total prefill compute.

A second mechanism, Compressed Sparse Attention 2, shares KV and indexer data across layers rather than letting every layer keep its own copy. The whole global KV cache is also quantized to FP4. Combined, DeepSeek reports the always-in-HBM cache comes out to 890 bytes per token, 4x smaller than their prior model and 437x smaller than their original V1. None of this touches SSD. It's about not generating more cache than necessary in the first place.

Three tiers, split by how each one gets reused

What's left after that compression still has to live somewhere. DeepSeek's deployment Q&A lays out three tiers instead of treating the cache as one pool. In-flight cache sits in GPU memory during a request. The sliding-window cache stays in the host memory of whichever machine produced it. The main global KV cache goes to a dedicated SSD cluster running 3FS, DeepSeek's distributed file system.

The sliding-window cache skips SSD entirely, which makes it the interesting one. Its useful range is much smaller than its nominal window size, so instead of paying to persist and fetch it, DeepSeek rebuilds it on demand using the same bounded replay trick described above. Say a request lands on a machine without that cache locally, with no other machine holding it either. DeepSeek recomputes just the last 128 tokens and reconstructs an approximate version. It's the same technique, just saving a storage write here instead of prefill compute.

Where the three days and seven days come from

The retention numbers apply only to the third tier, the global KV cache on the 3FS cluster. DeepSeek's deployment Q&A gives the figures directly. That cluster provides roughly 100 PiB of SSD storage. Cached entries on it get a time to live of three days for agent traffic and seven days for chatbot traffic.

The split comes down to how often each kind of traffic reuses its cache. Chatbot conversations have a much heavier long tail. A small number of cached prefixes get hit again and again across many turns, so losing one costs more overall hit rate than losing an agent workload's cache entry would. DeepSeek measured this instead of guessing at it. A roughly three-day retention window costs chatbot traffic about 0.5 to 1 percent of its cache hit rate. That's presumably the reasoning behind giving chatbot traffic the longer seven-day window.

One thing is worth flagging here. DeepSeek's doc says there's no hard expiry timer in their cache-store code itself. Reclamation happens through 3FS's garbage collector doing least-recently-used eviction under a disk-usage watermark, one instance running per storage node. The doc still gives two separate fixed numbers for the two traffic types though. It doesn't spell out how a single watermark-based eviction policy produces two different TTLs. I'd treat the three days and seven days as DeepSeek's effective target more than a strict guarantee, until they document that mechanism in more detail.

The part worth reusing

The reusable idea here is splitting the cache by behavior instead of managing every entry the same way. The specific three-day and seven-day numbers matter less than that choice. A cache entry that's cheap to regenerate and only useful inside one active session doesn't belong on a shared, capacity-managed store at all. It should stay local, with a cheap recompute fallback for when it's missing. A cache entry that's expensive to regenerate and gets reused across many separate requests earns a dedicated tier with its own tuned retention policy. DeepSeek tuned theirs against a real hit-rate curve instead of picking a round number. I'd want the same kind of measurement before trusting any retention window on my own infrastructure.