#residual-connections

Attention Residuals: How Kimi Rethinks Depth-Wise Information Flow in LLMs

Kimi's Attention Residuals paper proposes replacing fixed residual connections with learned softmax …