#attention

Attention Residuals: How Kimi Rethinks Depth-Wise Information Flow in LLMs

Kimi's Attention Residuals paper proposes replacing fixed residual connections with learned softmax …

Attention Mechanisms Compared: Standard, Linear, and Flash

A deep dive comparing standard softmax attention, linear attention, and Flash Attention: their math, …