AutoRound

Summary

• Trainable quantization parameters:
  • Weight clipping $\alpha ,\beta$
  • Rounding value adjustment $\mathbf{V}\in {\mathbb{R}}^{\text{shape}(\mathbf{W})}$

$$\tilde{\mathbf{W}}=s\times \text{clip}\left(\lfloor \frac{\mathbf{W}}{s}+z+\mathbf{V}\rceil \right)$$ $$s=\frac{\max \left(\mathbf{W}\right)\times \alpha -\min \left(\mathbf{W}\right)\times \beta }{2^{\text{bit}}-1}$$ $$\underset{\alpha ,\beta ,\mathbf{V}}{\min }{\Vert \mathbf{WX}-\tilde{\mathbf{W}}\mathbf{X}\Vert }_F^2$$ $${\mathbf{W}}_{t+1}={\mathbf{W}}_t-{\text{lr}}_t\times \text{sign}({\mathbf{g}}_t)$$