gate-keep explorer

Gated KV-cache compression: keep top-budget tokens by attention-mass score, sketch the evicted tail into one latent memory key. Memory beats pure dropping.

← back to portfolio

Sinks (first 2 cols, StreamingLLM rule) always kept · scores = mean attention per key (H2O-style mass) · memory key = score-weighted barycenter of evicted tail, norm-preserved.

Before — full 16×32 attention

Token scores (lime = kept, red tick = evicted, first 2 = sinks)

After — kept + 1 latent memory key

kept token latent memory key evicted (dropped in baseline)
—
drop error (no memory)
—
memory error (ours)
—
gain %

Latent-summary math (simplified, mirrors gate_keep/compress.py)

w = softmax(s_evicted / temp)  →  k_bar = Σ wᵢ·Kᵢ, v_bar = Σ wᵢ·Vᵢ (weighted barycenter of tail)
k_mem = P·Pᵀ·k_bar · (‖k_bar‖/‖P·Pᵀ·k_bar‖) with random orthonormal sketch P (8-d, norm-preserved) — appended as one extra key.
Error = probe-query attention-output residual: ‖attn(K)·V − attn(K')·V'‖ / rms(V), probe q = mean(K). Same denominator for drop vs memory, so ordering is fair.