Posts I am still working on. They can change or have mistakes.
Remove the softmax and the KV cache becomes one small matrix. Each token reads it, then updates it with a gradient step.