Gated DeltaNet

Gated Delta Networks: Improving Mamba2 with Delta Rule

• Mamba2: ${\mathbf{S}}_t={\alpha }_t{\mathbf{S}}_{t-1}+{\mathbf{v}}_t{\mathbf{k}}_t^{\top }$, where ${\alpha }_t\in (0,1)$.

• Gated Delta Rule:

  $${\mathbf{S}}_t={\mathbf{S}}_{t-1}\left({\alpha }_t\left(\mathbf{I}-{\beta }_t{\mathbf{k}}_t{\mathbf{k}}_t^{\top }\right)\right)+{\beta }_t{\mathbf{v}}_t{\mathbf{k}}_t^{\top }$$