Every token a model reads back is stored — the KV cache. It is usually the thing that runs a GPU out of memory first. Four measured moves make it smaller: share what repeats, keep the tokens that matter, prune the budget, and compress the rest. One budget, three receipts.
Where does the memory go, what does cutting it cost in accuracy, and which move is free?
Eight conversations share one 512-token system prompt. If each reserves the whole prompt, that is 504 blocks. Paged allocation stores the shared prefix once.
When memory is tight you keep a fixed number of tokens and drop the rest. Which ones you keep is the whole game: at the same 256-token budget the policy decides how much future attention survives.
Compression is a budget, not a free lunch. Keep fewer tokens and the output drifts. Drag the budget and watch the error knee appear.
At this budget the cache is 2.0× smaller (derived from kept fraction). Full FP16 KV is 128 KiB; here it is 64 KiB (derived).
Prune first, then store what is left in fewer bits. At a 64-token budget the KV cache drops from 128 KiB to 16 KiB — an 8× cut.
The layout is the cheap part to change: streaming paged attention matched dense attention on 9/9 parity cases at a max absolute error of 2.4e-7. Same answer, different memory layout.
Sharing the repeated system prompt cuts the block count 59.5% with no accuracy cost. Pruning and compression are where accuracy is spent, so they come second. And the choice of policy — not just the budget — moves retained attention from 16.5% to 96.4%.