SliceGPT

Summary

• Static channel-wise weight pruning by utilizing computational invariance.
• Apply PCA (eigen decomposition) to calibration dataset to obtain transformation matrix ${\mathbf{Q}}_l$.
• ${\mathbf{Q}}_l$ is merged into linear weights.
• LayerNorm is converted to RMSNorm.

$$\begin{array}{c}\Vert \mathbf{x}{\Vert }_2=\sqrt{\sum _i{x}_i^2}=\sqrt{{\mathbf{x}}^{\top }\mathbf{x}}\\ ⟹{\Vert \mathbf{Q}\mathbf{x}\Vert }_2=\sqrt{{\mathbf{x}}^{\top }{\mathbf{Q}}^{\top }\mathbf{Q}\mathbf{x}}=\Vert \mathbf{x}{\Vert }_2\\ ⟹\text{RMSNorm}(\mathbf{XQ}){\mathbf{Q}}^{\top }=\text{RMSNorm}(\mathbf{X})\end{array}$$