Tinkerings
  • Drafts

Drafts

Posts I am still working on. They can change or have mistakes.

Linear attention: a memory that is learned

attention

Remove the softmax and the KV cache becomes one small matrix. Each token reads it, then updates it with a gradient step.

Nov 20, 2026
No matching items

Tinkerings