KV Cache Offload Across Storage Tiers

The buffer is required: never put a synchronous write path from a GPU eviction stream directly on spindles. The create-heavy eviction stream sustained approximately 0.7 GB/s per 8-GPU host, while the HDD array’s durable write throughput ranged from roughly 9× below that rate at the stream’s actual commit sizes to approximately 692 MB/s only at ideal commit granularity. In matched windows, the synchronous HDD configuration absorbed just 10–18 GiB of KV writes and achieved a 19.0% hit rate—roughly one quarter of the workload’s reuse ceiling. Async write-back absorbed 89 GiB, but sacrificed durability and still reached only 64.3% reuse at concurrency 32 and 61.9% at concurrency 64, with TTFT at concurrency 64 roughly twice that of QLC. The same array reads at 6.34 GB/s in isolation; write commitment, not read bandwidth, is the constraint.

The benchmark ran on a single Supermicro inference host with 8x NVIDIA B200 GPUs, tensor parallelism 8, running a 70B parameter open-weights model at BF16 precision under vLLM 0.22.0 with LMCache 0.4.6. The workload was a multi-turn conversation replay with approximately 10,000-token prompts, 4 turns, and a 512-token completion limit; a configuration that produced an intrinsic prefix reuse ceiling of 71.7%. All remote storage arms ran over NFSv4.2 on 400GbE. The campaign comprised 144 runs across three storage configurations with zero failures. Full methodology is detailed in the technical brief.

Size the buffer to the eviction rate and the hot recall set, not to the capacity tier behind it. In this benchmark, a 30 GiB DRAM tier was sufficient to achieve full performance parity at the operating points tested. The eviction stream ran at approximately 0.7 GB/s sustained per 8-GPU host at 320 KiB per token. Where DRAM capacity is constrained, QLC flash is a viable alternative buffer layer; the Solidigm™ QLC arm in this study absorbed 84.6 GiB of KV writes in matched windows while holding the full reuse ceiling. The technical brief provides the full sizing methodology and two planning rules for production deployments.