Linear attention: a memory that is learned

attention
Remove the softmax and the KV cache becomes one small matrix. Each token reads it, then updates it with a gradient step.
Published

November 20, 2026

Note

This is a draft. It can change.

Linear attention replaces the growing KV cache with one small matrix of fixed size. Each token reads the matrix, then updates it with one gradient step. Memory and compute stop growing with the length of the context. The price is that recall is less exact.

This post shows the idea with small matrices and a few lines of Python. Shapes are written in brackets, like [T, dk]: T rows, dk columns.

What softmax attention keeps

Take one attention head. A new token arrives with a query q, shape [1, dk]. The past is two matrices with one row per earlier token: keys K [T, dk] and values V [T, dv].

p = softmax(q @ K.T)      # [1, dk] @ [dk, T] -> [1, T]    one weight per past token
o = p @ V                 # [1, T]  @ [T, dv] -> [1, dv]

q has to touch every row of K, and the softmax needs all T scores at the same time. So the model keeps every past key and value. This is the KV cache. It grows by one row per token, and every new token reads all of it again.

Remove the softmax

Without the softmax, the output is two matrix products. We can move the brackets:

o = (q @ K.T) @ V   =   q @ (K.T @ V)

Look at K.T @ V. Its shape is [dk, T] @ [T, dv] -> [dk, dv]. There is no T in the result, and there is no q in it. Call it S. It is built from the past only, so it exists before the new token arrives. The new token only does q @ S.

The loop

K.T @ V is a sum with one term per token. The term for one token is the outer product of its key and its value. So we can build S one token at a time:

S = zeros(dk, dv)                 # [dk, dv], never grows
for each new token:
    S += outer(k, v)              # [dk] x [dv] -> [dk, dv]    write
    o  = q @ S                    # [1, dk] @ [dk, dv] -> [1, dv]    read

Three tokens, dk = dv = 2:

K  [3, 2]         V  [3, 2]
[[1, 0],          [[10, 1],
 [0, 1],           [20, 2],
 [1, 1]]           [30, 3]]

after token 0:  S = [[10, 1], [ 0, 0]]
after token 1:  S = [[10, 1], [20, 2]]
after token 2:  S = [[40, 4], [50, 5]]

Token 1 has q = [1, 1]. It reads q @ S = [30, 3]. Softmax-free attention over tokens 0 and 1 gives the same answer: (q @ K[:2].T) @ V[:2] = [30, 3].

Each token reads S right after its own write. At that moment S holds tokens 0 to t and nothing later. That is how the loop stays causal without a mask.

What it saves

There is no KV cache. The only state is S, and it has the same size at token 10 and at token 1,000,000.

For one head with dk = dv = d, and one new token:

softmax linear
state kept between tokens K and V, 2·T·d numbers S, d² numbers
numbers read per token 2·T·d d²
FLOPs per token 4·T·d 4·d²

With d = 128 and T = 32,000, softmax reads about 8 million numbers per token. Linear attention reads 16,384. That is about 500 times less.

For a full prompt of T tokens, softmax attention builds a [T, T] matrix of scores and costs 4·T²·d FLOPs. Linear attention costs 4·T·d². The cost grows in a straight line with T. That is where the name comes from.

What it costs

Softmax makes the weights sharp. A score that is 3 higher gets about 20 times the weight (e³), not 3 times. Without it, picking one token out of thousands is less exact.

S is also small. It can hold about dk key-value pairs before they mix. Schlag, Irie and Schmidhuber (2021) measured this: with keys of 64 numbers, reads start to fail at 60 or more stored pairs.

There is a second problem. Writing twice under the same key does not replace the old value. It adds to it.

k = [0.6, 0.8]
write (k, [10, 1]):   k @ S = [10, 1]
write (k, [30, 3]):   k @ S = [40, 4]     we wanted [30, 3]

The delta rule

S is a key-value store. So one thing must be true: after we store (k, v), reading with k returns v.

k @ S_new = v

Everything in this section comes from that one equation.

Every write in linear attention has the form S + outer(k, c), where c is the vector written under k:

S_new = S + outer(k, c)

Read the new memory with k:

k @ S_new = k @ S + k @ outer(k, c)

The second term is (k·k) * c. If k is normalized, k·k = 1, so k @ outer(k, c) = c.

k = [0.6, 0.8]    c = [10, 1]
outer(k, c)     = [[6, 0.6], [8, 0.8]]
k @ outer(k, c) = [0.6*6 + 0.8*8, 0.6*0.6 + 0.8*0.8] = [10, 1] = c

So the read is k @ S + c. We want the read to be v:

v = k @ S + c
c = v - k @ S

Put c back into the write:

S_new = S + outer(k, v - k @ S)

So the fix is one changed line in the loop:

S += outer(k, v)                  # before
S += outer(k, v - k @ S)          # after

In two steps:

pred = k @ S                      # [dk] @ [dk, dv] -> [dv]    what S returns for this key now
err  = v - pred                   # [dv]                       the gap to the new value
S   += outer(k, err)              # write the gap, not the value

c is err. Now the second write from the example above works:

S = [[6, 0.6], [8, 0.8]]          k @ S = [10, 1]
new v = [30, 3]                   c = [20, 2]
S_new = [[18, 1.8], [24, 2.4]]    k @ S_new = [30, 3] = v

This only works if the keys have length 1. So these models normalize each key before the write: k = k / k.norm().

It is a gradient step

The delta rule is one step of gradient descent on the loss ½ ||k @ S - v||², taken with respect to S (Gated DeltaNet, section 3.1). The step size is 1 in the code above.

So S is not only a cache. It is a small weight matrix, and the model trains it on its own context, one token at a time, during the forward pass. No backprop is running. The rule is written in closed form.

Two gates

Real layers add two learned numbers per token.

for t in range(T):
    S    = alpha[t][:, None] * S          # [dk, 1] * [dk, dv]   forget: shrink every row a little
    pred = k[t] @ S                       # [dv]
    err  = v[t] - pred                    # [dv]
    S   += beta[t] * outer(k[t], err)     # write the gap, with strength beta
    o[t] = q[t] @ S                       # [dv]
  • alpha is the forget gate, shape [T, dk]. Near 0 clears the memory.
  • beta is the step size, shape [T]. Near 0 skips the token.

Both come from the token itself through trained weights and a sigmoid, so they are between 0 and 1.

q = x @ Wq                                  # [T, dk]
k = F.normalize(x @ Wk, dim=-1)             # [T, dk]   length 1, so k·k = 1
v = x @ Wv                                  # [T, dv]
alpha = torch.sigmoid(x @ Wa)               # [T, dk]
beta  = torch.sigmoid(x @ Wb).squeeze(-1)   # [T]

So two things are learned, at two speeds.

  • Training learns Wa and Wb by backprop, together with Wq, Wk, Wv. This is where the model learns when to forget and how strongly to write.
  • At inference those weights are frozen. S is learned from the context, one token at a time, with alpha and beta deciding each step.

Papers

  • Schmidhuber, “Learning to Control Fast-Weight Memories”, Neural Computation 4(1), 1992.
  • Schlag, Irie, Schmidhuber, “Linear Transformers Are Secretly Fast Weight Programmers”, arXiv 2102.11174, 2021.
  • Yang, Kautz, Hatamizadeh, “Gated Delta Networks: Improving Mamba2 with Delta Rule”, arXiv 2412.06464, 2024.
  • Irie, Gershman, “Fast weight programming and linear transformers: from machine learning to neurobiology”, arXiv 2508.08435, 2025.