On knowing what changed in a vision pipeline

In a video pipeline, most of the scene between one frame and the next does not change, and the optimization seems to be sitting right there in front of you. Skip the parts that stayed the same, recompute only what moved, and pay only for the work that actually matters.

The idea looks obvious until you start to implement it. Then a quiet circularity shows up, the kind that takes a while to put into words and longer to accept. To skip the unchanged tokens you first need a way to identify them, and every way of identifying them is either too coarse to be safe or too expensive to be worth it.

There are three natural approaches, roughly in the order you would reach for them: pixel-level change detection, then feature-level change detection, then a learned change predictor. Each one fails for its own reason, and the reasons rhyme.

Pixels are noisier than expected

It would be convenient if "the scene has not changed" were a property of the pixels, so that a small L2 distance between corresponding patches in consecutive frames meant the patch was unchanged. Spend a few minutes on real footage and you run into what every video codec engineer already knows. Nothing in a video is ever truly static.

Camera sensors have shot noise. Auto-exposure and auto-white-balance shift the global statistics from one frame to the next. A handheld camera jitters below the level of a single pixel. Outdoor light changes continuously, and even on a tripod an indoor scene under fluorescents flickers at frequencies the sensor partly aliases. So the histogram of L2 distances for two consecutive frames of a "static" scene is not piled up at zero. It spreads across a wide band, and any threshold you pick will either wave through too many false positives, treating moving regions as static and quietly poisoning the downstream features, or reject too many true negatives, recomputing almost everything and giving the whole optimization away.

Codec engineers solved a version of this decades ago, with motion estimation, sub-pixel alignment, and rate-distortion optimization. H.264 and H.265 are industrial-grade change-detection systems in their own right. That is exactly the problem: a modern video encoder is not free, and even with hardware acceleration it carries real latency. Spending that much compute just to work out which patches changed already eats a sizable fraction of the work you were trying to avoid.

Features change in ways pixels do not

The natural fallback is to compare at the feature level. The pixels of a moving shadow may change while the high-level features stay roughly put, and it is the feature representation that the downstream task actually cares about. So you compare layer outputs frame to frame, gate on how similar they are, and recompute only when the features have drifted.

The logic is sound, and the implementation walks you straight into the trap. To know whether the features changed, you have to compute the features, and not computing the features was the entire point. You have replaced one full encoder pass with one full encoder pass plus a comparison step, which is strictly worse than where you started.

Real systems get around this by comparing features at an earlier, cheaper layer and using that to gate recomputation of the expensive later ones. Eventful Transformers (Dutson et al., ICCV 2023) does exactly this. The initial patch embeddings are always computed, and a thresholded gate decides which tokens get their self-attention layers redone. The savings are real, 2 to 4× on detection and action-recognition benchmarks, but they come from skipping the expensive attention layers for static tokens, not from skipping the encoder. The patch projection still runs, and so does the gate evaluation.

The gate also brings a failure mode of its own. Its decision is local, asking only whether this one token's input looks like its previous input. The consequence is global. Skipping a token's recomputation leaves its cached K and V as approximations, and that error then propagates into every other token's attention output. A local decision, a global cost. Accuracy degrades under heavy motion in a way that is hard to bound ahead of time, which is precisely the regime where you wanted the speedup most.

The learned-predictor approach

You could instead train a small separate network to predict which tokens are going to change, so that the system no longer leans on pixel statistics or on running the encoder halfway. This dedicated model takes the previous frame's features and the current frame's pixels and hands back a change mask.

In principle it works. In practice the predictor has to be accurate enough that its mistakes do not drag down downstream accuracy, and that bar quietly demands real capacity. Once you have spent that capacity, the predictor's cost starts competing with the encoder cost it was meant to save. Good predictors are not cheap. Cheap predictors are not good. You end up doing real work just to decide which work to skip, and the savings collapse.

The pragmatic version of this idea swaps the learned predictor for scene priors. MaskVD (2024) combines static masks (regions that are usually background in driving scenes) with dynamic masks (previous-frame detections plus a small motion buffer) to drop 80 percent of input patches while holding detection performance. It works because the prior is genuinely task-specific: in driving, the content that matters is mostly on the road, and the road behaves in a particular way. That same prior does not carry over to depth estimation, or to indoor agents, or to anything where the important content can turn up anywhere in the frame. You pay for the accuracy with an assumption about the data distribution, and the bill comes due the moment the distribution shifts.

The shape of the problem

This circularity, where skipping work needs information that itself takes work to produce, is not special to vision. It turns up almost anywhere people try to make a computation incremental.

LLM speculative decoding. A small draft model guesses the next few tokens and the big model verifies them. The cheap guesser is acceptable only because verification is cheap, a single forward pass over a handful of tokens, and that is because verification parallelizes. Take that away and the scheme falls over.

Build systems. Incremental builds work only when the dependency graph is declared honestly. One undeclared dependency, an environment variable, a clock-dependent test, an implicit include, and incremental compilation either corrupts silently or bails out to a clean build. Either way you did more work, not less.

Database materialized views. A view can be maintained incrementally only when its defining query supports differential evaluation. For a general query, refreshing the view is just a recomputation wearing a different name.

Cache coherence. CPU caches work because the hardware tracks every write at the granularity of a cache line, spending real silicon area and power on the bookkeeping. The cost did not disappear; it was paid once, in the architecture, so software never sees it.

The same trade-off sits under all of them. Avoiding repeated work requires tracking what changed, and the tracking is never free. The optimization only wins when that tracking cost is small next to the work it lets you skip. When the tracking cost creeps up toward the cost of the work itself, as it does for ViT encoder features, where "what changed" is a question you can only answer by running the encoder, plain recomputation wins.

The cheapest way to know

What I find striking is that the field's most successful answer has been to stop asking the question at all. Video Depth Anything, FlashDepth, StableDPT, Online Video Depth Anything: none of them try to work out which tokens changed. They recompute the encoder on every frame and put the temporal logic in the decoder, where it is both cheaper to run and easier to reason about. The encoder is treated as a stateless image-feature extractor, and temporal coherence comes from a small Mamba block or a few cross-attention layers downstream. Recomputation, it turns out, is cheap enough.

This is not a defeat for the original intuition. "Do not redo work you have already done" is still a good instinct. The correction is about where the cost actually lives. The encoder is expensive but stateless, so recomputing it has a bounded cost and a bounded latency. The temporal logic is cheap but stateful, so caching there is both safe, because the state is small and bounded, and useful, because it captures the cross-frame structure that actually matters.

When "what changed" is itself an expensive question, the cheapest move is often to stop asking it, recompute the parts that are cheap to recompute, and spend your caching effort where the math lets the cache stay small. Incremental computation is not wrong. It just has prerequisites, cheap change detection, bounded error propagation, or an architecture that respects locality, and when none of those hold, holding onto the optimization because it sounds right costs you both accuracy and engineering time.