QuIP

Summary

• LDLQ: a more efficient OPTQ.
• Incoherence Processing: transform weights and activations with random orthogonal matrices via 2-factor Kronecker.

Methodology

LDLQ

$${\hat{\mathbf{W}}}_k=\mathcal{Q}\left({\mathbf{W}}_k+\left({\mathbf{W}}_{1:(k-1)}-{\hat{\mathbf{W}}}_{1:(k-1)}{\mathbf{a}}_k\right)\right){\mathbf{W}}_k\in {\mathbb{R}}^{n\times 1}{\mathbf{a}}_k\in {\mathbb{R}}^{k-1}$$ $$\hat{\mathbf{W}}=\mathcal{Q}\left(\mathbf{W}+\left(\mathbf{W}-\hat{\mathbf{W}}\right)\mathbf{U}\right)\mathbf{U}\text{ is upper triangular.}$$

The optimal $\mathbf{U}$ satisfies the LDL decomposition of $\mathbf{H}$:

$$\mathbf{H}=\left(\mathbf{U}+\mathbf{I}\right)\mathbf{D}{\left(\mathbf{U}+\mathbf{I}\right)}^{\top }$$

This is equivalent to the OPTQ algorithm while being more efficient, requires no matrix inversion and only one Cholesky decomposition.

Quantization with Incoherence Processing

• Quantization is executed under the rotated domain.
• Therefore, both hessian $\mathbf{H}$ and weight $\mathbf{W}$ are multiplied by random orthogonal matrices $\mathbf{U},\mathbf{V}$.