Linear attention: a memory that is learned
This is a draft. It can change.
Linear attention replaces the growing KV cache with one small matrix of fixed size. Each token reads the matrix, then updates it with one gradient step. Memory and compute stop growing with the length of the context. The price is that recall is less exact.
This post shows the idea with small matrices and a few lines of Python. Shapes are written in brackets, like [T, dk]: T rows, dk columns.
What softmax attention keeps
Take one attention head. A new token arrives with a query q, shape [1, dk]. The past is two matrices with one row per earlier token: keys K [T, dk] and values V [T, dv].
p = softmax(q @ K.T) # [1, dk] @ [dk, T] -> [1, T] one weight per past token
o = p @ V # [1, T] @ [T, dv] -> [1, dv]q has to touch every row of K, and the softmax needs all T scores at the same time. So the model keeps every past key and value. This is the KV cache. It grows by one row per token, and every new token reads all of it again.
Remove the softmax
Without the softmax, the output is two matrix products. We can move the brackets:
o = (q @ K.T) @ V = q @ (K.T @ V)Look at K.T @ V. Its shape is [dk, T] @ [T, dv] -> [dk, dv]. There is no T in the result, and there is no q in it. Call it S. It is built from the past only, so it exists before the new token arrives. The new token only does q @ S.
The loop
K.T @ V is a sum with one term per token. The term for one token is the outer product of its key and its value. So we can build S one token at a time:
S = zeros(dk, dv) # [dk, dv], never grows
for each new token:
S += outer(k, v) # [dk] x [dv] -> [dk, dv] write
o = q @ S # [1, dk] @ [dk, dv] -> [1, dv] readThree tokens, dk = dv = 2:
K [3, 2] V [3, 2]
[[1, 0], [[10, 1],
[0, 1], [20, 2],
[1, 1]] [30, 3]]
after token 0: S = [[10, 1], [ 0, 0]]
after token 1: S = [[10, 1], [20, 2]]
after token 2: S = [[40, 4], [50, 5]]
Token 1 has q = [1, 1]. It reads q @ S = [30, 3]. Softmax-free attention over tokens 0 and 1 gives the same answer: (q @ K[:2].T) @ V[:2] = [30, 3].
Each token reads S right after its own write. At that moment S holds tokens 0 to t and nothing later. That is how the loop stays causal without a mask.
What it saves
There is no KV cache. The only state is S, and it has the same size at token 10 and at token 1,000,000.
For one head with dk = dv = d, and one new token:
| softmax | linear | |
|---|---|---|
| state kept between tokens | K and V, 2·T·d numbers | S, d² numbers |
| numbers read per token | 2·T·d | d² |
| FLOPs per token | 4·T·d | 4·d² |
With d = 128 and T = 32,000, softmax reads about 8 million numbers per token. Linear attention reads 16,384. That is about 500 times less.
For a full prompt of T tokens, softmax attention builds a [T, T] matrix of scores and costs 4·T²·d FLOPs. Linear attention costs 4·T·d². The cost grows in a straight line with T. That is where the name comes from.
What it costs
Softmax makes the weights sharp. A score that is 3 higher gets about 20 times the weight (e³), not 3 times. Without it, picking one token out of thousands is less exact.
S is also small. It can hold about dk key-value pairs before they mix. Schlag, Irie and Schmidhuber (2021) measured this: with keys of 64 numbers, reads start to fail at 60 or more stored pairs.
There is a second problem. Writing twice under the same key does not replace the old value. It adds to it.
k = [0.6, 0.8]
write (k, [10, 1]): k @ S = [10, 1]
write (k, [30, 3]): k @ S = [40, 4] we wanted [30, 3]
The delta rule
S is a key-value store. So one thing must be true: after we store (k, v), reading with k returns v.
k @ S_new = v
Everything in this section comes from that one equation.
Every write in linear attention has the form S + outer(k, c), where c is the vector written under k:
S_new = S + outer(k, c)
Read the new memory with k:
k @ S_new = k @ S + k @ outer(k, c)
The second term is (k·k) * c. If k is normalized, k·k = 1, so k @ outer(k, c) = c.
k = [0.6, 0.8] c = [10, 1]
outer(k, c) = [[6, 0.6], [8, 0.8]]
k @ outer(k, c) = [0.6*6 + 0.8*8, 0.6*0.6 + 0.8*0.8] = [10, 1] = c
So the read is k @ S + c. We want the read to be v:
v = k @ S + c
c = v - k @ S
Put c back into the write:
S_new = S + outer(k, v - k @ S)
So the fix is one changed line in the loop:
S += outer(k, v) # before
S += outer(k, v - k @ S) # afterIn two steps:
pred = k @ S # [dk] @ [dk, dv] -> [dv] what S returns for this key now
err = v - pred # [dv] the gap to the new value
S += outer(k, err) # write the gap, not the valuec is err. Now the second write from the example above works:
S = [[6, 0.6], [8, 0.8]] k @ S = [10, 1]
new v = [30, 3] c = [20, 2]
S_new = [[18, 1.8], [24, 2.4]] k @ S_new = [30, 3] = v
This only works if the keys have length 1. So these models normalize each key before the write: k = k / k.norm().
It is a gradient step
The delta rule is one step of gradient descent on the loss ½ ||k @ S - v||², taken with respect to S (Gated DeltaNet, section 3.1). The step size is 1 in the code above.
So S is not only a cache. It is a small weight matrix, and the model trains it on its own context, one token at a time, during the forward pass. No backprop is running. The rule is written in closed form.
Two gates
Real layers add two learned numbers per token.
for t in range(T):
S = alpha[t][:, None] * S # [dk, 1] * [dk, dv] forget: shrink every row a little
pred = k[t] @ S # [dv]
err = v[t] - pred # [dv]
S += beta[t] * outer(k[t], err) # write the gap, with strength beta
o[t] = q[t] @ S # [dv]alphais the forget gate, shape[T, dk]. Near 0 clears the memory.betais the step size, shape[T]. Near 0 skips the token.
Both come from the token itself through trained weights and a sigmoid, so they are between 0 and 1.
q = x @ Wq # [T, dk]
k = F.normalize(x @ Wk, dim=-1) # [T, dk] length 1, so k·k = 1
v = x @ Wv # [T, dv]
alpha = torch.sigmoid(x @ Wa) # [T, dk]
beta = torch.sigmoid(x @ Wb).squeeze(-1) # [T]So two things are learned, at two speeds.
- Training learns
WaandWbby backprop, together withWq,Wk,Wv. This is where the model learns when to forget and how strongly to write. - At inference those weights are frozen.
Sis learned from the context, one token at a time, withalphaandbetadeciding each step.
Fast weights, and the link to continual learning
This idea is from 1992. Schmidhuber described two networks: a slow network with trained weights, and a fast network whose weights the slow network rewrites at every step. In 2021, Schlag, Irie and Schmidhuber showed that linear attention is this scheme. Wq, Wk, Wv, Wa, Wb are the slow weights. They are trained by backprop and then frozen. S is the fast weight matrix. It changes at every token.
A 2025 primer by Irie and Gershman puts RetNet, Mamba2, GLA and DeltaNet in one family. They differ only in the write line, and each write line is one gradient step on a small local loss.
So here is a model that learns after training has ended. That is why people who work on continual learning look at it. But there is a limit. S is set to zero at the start of every sequence. The model learns inside one context and forgets it at the end. The 2021 paper and the primer do not study forgetting across sessions.
Papers
- Schmidhuber, “Learning to Control Fast-Weight Memories”, Neural Computation 4(1), 1992.
- Schlag, Irie, Schmidhuber, “Linear Transformers Are Secretly Fast Weight Programmers”, arXiv 2102.11174, 2021.
- Yang, Kautz, Hatamizadeh, “Gated Delta Networks: Improving Mamba2 with Delta Rule”, arXiv 2412.06464, 2024.
- Irie, Gershman, “Fast weight programming and linear transformers: from machine learning to neurobiology”, arXiv 2508.08435, 2025.