Why KV cache reuse is harder for vision than for language

Anyone who has worked on LLM inference knows the standard optimization: KV caching. Each new token's keys and values are computed once, stored, and reused for the rest of the sequence. The savings are large, and the math is exact. The cache is not an approximation of what attention would have produced. It is what attention would have produced.

In theory the same trick should work for video vision transformers. Consecutive frames share most of their content, VideoMAE measured 90 to 95 percent temporal token redundancy in natural video, and the encoder is the expensive part of a video pipeline, so there is an enormous amount of room to reuse. Why not cache K and V across frames the way we cache them across tokens?

I spent several weeks on this question for a sparse-depth pipeline, read most of the relevant literature, and came out where the field has already converged. It does not work in the way the analogy suggests, and the reason is structural rather than a matter of engineering effort. It is worth walking through, because the cleanest way to see why a technique fails to port is to look hard at the assumption it quietly relied on where it came from.

The assumption that does most of the work

KV caching works in LLMs because LLM attention is causal. Token i only ever attends to tokens 0 through i, so once its key and value are computed, nothing a future token does can change them. The information that produced k_i and v_i is frozen by construction. Each new token extends the sequence without disturbing anything behind it, causality propagates cleanly down the stack, and the cache is exact because the values it stores are exact.

ViT encoders use global self-attention. Every patch token attends to every other patch token, in every layer. So if even one patch in the input changes, because something in the scene moved or the camera shifted a little, the correct K and V for all patches change with it. Every patch's layer-1 representation was computed by attending to the patch that just moved, every layer-2 representation was computed by attending to every layer-1 representation, and the dependency runs all the way down.

This is not an implementation artifact. It is the math. Global self-attention is, by definition, a function where every output depends on every input, and there is simply no causal partition to exploit.

What "vision KV cache reuse" actually means

The proposals in the literature do not claim exact caching. What they claim is that for patches whose input tokens are close to the previous frame's input tokens, the K and V vectors are approximately close to the previous frame's K and V, close enough to substitute without collapsing downstream accuracy.

That is a reasonable engineering proposition, but it is a different proposition. The technique has moved from caching, where you memoize a deterministic function, to approximating, where you substitute one value for another and hope the error stays small. And once you are approximating, accuracy floors, error bounds, and worst cases stop being someone else's problem and become things you have to analyze.

The most thoughtful work in this space is Eventful Transformers (Dutson et al., ICCV 2023). It introduces three coordinated mechanisms. A token gate compares each token against a stored reference and selects only the tokens whose change exceeds a threshold for recomputation. A token buffer fills the unchanged positions from storage after sparse processing. And sparse attention updates compute the QK⊤ changes selectively, from delta-based queries and keys. On ImageNet VID detection and EPIC-Kitchens action recognition, this gives a 2–4× speedup with minor accuracy loss, training-free.

That is a real, useful, deployable result, but notice where the savings come from. They come from skipping the expensive attention layers for static tokens, not from skipping the encoder. The patch projection still runs, the gate evaluation still runs, and the accuracy degrades under high motion, which is exactly the regime where the speedup was supposed to help most.

What the field has actually settled on

It helps to look at where the state of the art has landed in video depth estimation, one of the most demanding domains for both efficiency and accuracy. Video Depth Anything (CVPR 2025) recomputes the encoder per frame and adds temporal attention layers in the DPT head. FlashDepth (ICCV 2025) recomputes the encoder per frame and adds a Mamba recurrent module, around 1 percent of parameters, for temporal coherence. StableDPT and Online Video Depth Anything follow the same pattern: encoder recomputed, temporal logic in the decoder.

No recent video-depth method caches the encoder. The one partial exception, VeloDepth, runs the full encoder on keyframes and uses optical-flow-warped features for the frames in between, and it explicitly degrades under large motion and needs periodic full recomputation to recover.

This is the field's empirical answer to the original question. Recomputing the encoder is cheap enough relative to the cost of the errors that feature reuse introduces, and whatever you save by skipping the encoder gets eaten by the temporal-module complexity you then need to compensate for stale features.

What does port from LLM caching

Not everything from the LLM KV-cache literature is dead on arrival in vision. Several techniques carry over cleanly, because they operate on the cache's storage or shape rather than on whether the stored values are still correct.

Grouped Query Attention. Share K and V heads across query heads. It works in vision the same way it works in language: the K/V representation gets smaller and the attention output is mathematically identical up to the sharing. 4 to 8× KV memory reduction, at the cost of retraining.

KV quantization. Store the cached features in FP8 or INT4. 2 to 4× memory reduction at negligible accuracy cost, training-free.

Sliding window attention. Bound the temporal context. This is the analog of a fixed-window historical buffer in vision, the same math with a choice of how far back to remember.

Token eviction. Drop tokens that contribute little to later computation. Evict3R demonstrates this for vision, with a 50 percent peak-memory reduction, training-free.

All of these share one property. None of them require the cached values to still be correct. They reshape the cache, compress it, bound it, or prune it, and not one of them claims that yesterday's K and V are still the right answer.

Where the deeper insight lives

The wider lesson runs past attention. Whenever a technique is ported from one architecture to another, the first question to ask is what assumption it depended on. KV caching in LLMs depends on causality. Compiler memoization depends on referential transparency. Database materialized views depend on bounded update rates. Incremental compilation in a build system depends on the dependency declarations being honest.

When the assumption holds, the technique is exact and the savings are nearly free. When it is only approximately true, the technique becomes an approximation and a new responsibility appears, which is bounding the error. And when it fails outright, the technique has to be redesigned around a different assumption or given up for plain recomputation.

"Just cache it" fails in vision not because vision is harder than language, but because vision attention was designed under a different set of priors, and one of those priors, that any patch's representation can depend on any other patch's representation, is exactly the thing the caching assumption needed to be false. So the interesting engineering question stops being how to make vision caching work. It becomes what the cheapest way is to express the assumption that does hold in video, which is temporal coherence in the output rather than in the intermediate features. That question is the one that produced the recurrent-state, token-merge, and sparse-recompute architectures now winning benchmarks.