Learning Generalizable Mixed-Precision Quantization via Attribution Imitation

Abstract

In this paper, we propose a generalizable mixed-precision quantization (GMPQ) method for efficient inference. Conventional methods require the consistency of datasets for bitwidth search and model deployment to guarantee the policy optimality, leading to heavy search cost on challenging large-scale datasets in realistic applications. On the contrary, our GMPQ searches the mixed-quantization policy that can be generalized to large-scale datasets with only a small amount of data, so that the search cost is significantly reduced without performance degradation. Specifically, we observe that locating network attribution correctly is general ability for accurate visual analysis across different data distribution. Therefore, despite of pursuing higher accuracy and lower model complexity, we preserve attribution rank consistency between the quantized models and their full-precision counterparts via capacity-aware attribution imitation for generalizable mixed-precision quantization strategy search, where the capacity of quantized networks is considered to fully utilize the network capacity without insufficiency. Since slight noise in attribution is amplified by discrete ranking operations with significant rank errors, mimicking the attribution ranks of the full-precision models obstructs the quantized networks to correctly locate the attribution. To address this, we further present a robust generalizable mixed-precision quantization method to smooth the attribution for rank error alleviation by hierarchical attribution partitioning, which efficiently partitions the attribution pixels in high spatial resolution and assigns the same attribution value for pixels within a group. Moreover, we propose dynamic capacity-aware attribution imitation to adjust the concentration degree of the attribution according to sample hardness, so that sufficient model capacity is achieved with full utilization for each image. Extensive experiments on image classification and object detection show that our GMPQ and R-GMPQ obtain competitive accuracy-complexity trade-offs with significantly reduced search cost compared to the state-of-the-art mixed-precision networks.

Appendices

Appendix

A. Visualization of Optimal Quantization Policy

We searched the quantization policy on different small datasets with various architectures via the presented GMPQ. Figure 11 demonstrates the optimal bitwidth allocation for weights and activations of each layer, where ResNet18 was compressed and the policy was searched on various small datasets including CIFAR-10 (Krizhevsky et al., 2009), Cars (Krause et al., 2013), Flowers (Nilsback and Zisserman, 2008), Aircraft (Maji et al., 2013), Pets (Parkhi et al., 2012) and Food (Bossard et al., 2014). Figure 12 depicts the obtained quantization strategy searched on CIFAR-10 with MobileNet-V2 (Sandler et al., 2018), ResNet18 (He et al., 2016) and ResNet50 architectures. The BOPs limit was set to 7.4G, 15.3G and 30.7G for MobileNet-V2, ResNet18 and ResNet50.

Fig. 11

The visualization of the optimal quantization policy searched on different small datasets including CIFAR-10, Cars, Flowers, Aircraft, Pets and Food

Fig. 12

The visualization of the optimal quantization policy searched on CIFAR-10 by our GMPQ. We evaluated our method with MobileNet-V2, ResNet18 and ResNet50 on ImageNet for image classification

For quantization policy searched on different small datasets, the optimal bitwidth allocation varies significantly although the complexity of the obtained model is close to each other. It is observed that activations are usually assigned with higher bitwidths than weights in most quantization policy, indicating that the classification performance and attribution rank consistency are more sensitive to activation quantization than weight quantization. The bitwidth distribution of weights and activations obtained on Cars, Aircraft, Food, and CIFAR-10 is similar, which also achieves better generalization performance on largescale datasets compared with that searched on Flowers and Pets. For the Flowers and Pets datasets, the optimal quantization policy is similar to uniform quantization in fixed-precision networks, which also leads to worse accuracy-complexity trade-offs due to the lack of generalization ability.

For quantization policy for different architectures, it is observed that Layer 7, 12 and 17 in ResNet18 containing residual connections require the larger bitwidth compared with their corresponding regular branches. Since MobileNet-V2 is very compact, it receives higher bitwidths allocations than other network architectures. On the contrary, ResNet50 is compressed with lower bitwidths due to the significant redundancy compared with MobileNet-V2.

Table 9 Top-1 accuracy (%) and BOPs (G) on ImageNet of the mixed-precision networks searched on different small datasets across various network architectures

B. Accuracy of Quantization Policy Searched on Different Small Datasets

In this section, we show the top-1 accuracy and BOPs on ImageNet of our GMPQ with the quantization policy searched on different small datasets including CIFAR-10, Cars, Flowers, Aircraft, Pets and Food. The applied network architectures contain MobileNet-V2, ResNet-18 and ResNet-50, and more accuracy-complexity trade-offs for ResNet-18 are demonstrated in Fig. 10b. Table 9 illustrates the accuracy and the complexity on ImageNet, where those of full-precision networks are also provided. The search cost is significantly reduced across various architectures compared with conventional mixed-precision quantization methods shown in Table 5, while the accuracy is only degraded slightly. The accuracy of quantization policy searched on CIFAR-10 achieves the highest, because the gap of object category between CIFAR-10 and ImageNet is the smallest compared with other datasets. Although the discrepancy of object class distribution between ImageNet and the small datasets such as Aircraft is non-negligible, the accuracy of the mixed-precision networks is still comparable with state-of-the-art approaches shown in Table 5 due to the attribution rank preservation.

C. Explanation of the Generalization Risk (9)

As visualized in Fig. 3 of the manuscript, quantized networks with lower capacity tend to acquire more concentrated attribution although the attribution rank remains similar, where the networks focus on smaller regions to avoid capacity insufficiency for image representation. To further demonstrate the soundness of the observation, we report the entropy of attribution for networks in different bitwidths that reveals the attribution concentration. The entropy E is defined as follows:

Fig. 13

The relation between the attribution entropy and the model complexity, and they are significantly positively correlated

$$\begin{aligned} ~~~~~~~~~~~~~~~~E=\sum _{i,j}-M_{ij}[y_x] \log M_{ij}[y_x] \end{aligned}$$

(15)

Large entropy indicates more diverse attribution and vice versa. Figure 13 shows the average attribution entropy and BOPs across the validation set of ImageNet dataset for ResNet18 in networks quantized by different optimal quantization policies (searched on ImageNet). The correlation coefficient is 0.733 between the attribution entropy and network BOPs, which verifies the observation that networks with smaller capacity acquire more concentrated attribution. For the value of p in (9), excessively large p for attribution imitation leads to over-concentrated attribution. Therefore, the networks focus on small image regions with little information, and the network capacity is not fully utilized for feature representation. On the contrary, extremely small p for attribution imitation results in attribution divergence, and focusing on large image regions causes the capacity insufficiency in the forward pass. Therefore, we require the attribution rank of quantized and full-precision networks to be similar, while the attribution concentration is adjusted according to the network capacity. We also conducted ablation studies to show the effectiveness of the definition shown in (9). We leverage two other functions to acquire p based on the average bitwidth of the networks in the following:

$$\begin{aligned}&p=\frac{1}{L}\sum _{k=1}^{L}[Q_{w}^0/\left( \sum _{i=1}^{N_w^k}\pi _{w,i}^kq_{w,i}^k\right) ]^{\frac{1}{2}}\cdot [Q_{a}^0/\left( \sum _{i=1}^{N_a^k}\pi _{a,i}^kq_{a,i}^k\right) ]^{\frac{1}{2}}\nonumber \\&p=\frac{1}{L}\sum _{k=1}^{L}[Q_{w}^0/\left( \sum _{i=1}^{N_w^k}\pi _{w,i}^kq_{w,i}^k\right) ]^{2}\cdot [Q_{a}^0/\left( \sum _{i=1}^{N_a^k}\pi _{a,i}^kq_{a,i}^k\right) ]^{2} \end{aligned}$$

(16)

where the concavity is different for these functions. Table 10 shows the accuracy-complexity trade-off for ResNet18 on the validation set of ImageNet, where the linear form shown in (8) of the manuscript achieves the best performance.

Table 10 The accuracy-complexity trade-offs for different definition of p in (9)

D. Accuracy During the Compression Policy Search

We optimized the supernet containing all bitwidth selections with and without the generalization risk shown in (7) respectively, where we leveraged the training set from CIFAR-10 for policy search and validation set from ImageNet for evaluation. Meanwhile, we also directly utilized the training set of ImageNet for optimizing the supernet, and report the accuracy curve for reference as the baseline. We leveraged ResNet18 as the backbone architecture, and the BOPs budget was set as 7.5G. Evaluating the acquired quantization policy requires extremely high cost because we have to finetune the quantized models until convergence. Therefore, we evaluated the searched quantization policies from different experimental settings every 10 epochs during the search process, where the quantization policy with the largest importance weight is selected for evaluation. We plot the accuracy curve in Fig. 14, where our the objective with the generalization risk consistently outperforms the one without the generalization risk. The advantages of our method become more significant when the search process gradually converges. Meanwhile, the gap between our method and the optimal compression policy acquired by searching with ImageNet is small. The results can empirically verify the higher generalization ability of our method.

Fig. 14

The classification accuracy for quantization policies acquired during the search process

Table 11 The accuracy-complexity trade-off across different sample size from CIFAR-10 for search

E. Influence of the Sample Size of Datasets for Policy Searching

In order to analysis the influence of dataset sample size for policy searching, we searched the mixed-precision quantization policies with different sample sizes on CIFAR-10. The data amount is set to be 20%, 40%, 60%, 80% and 100% of the original training set, and we report the accuracy-complexity trade-offs on the ImageNet. Moreover, we also demonstrate the performance of the optimal quantization policy that is obtained by searching on full training datasets of ImageNet. The networks quantized with acquired policies are finetuned by the datasets for evaluation. The results bring us following conclusions:

• Utilizing extremely small amount of data (e.g. $\leqslant 40\%$) from CIFAR-10 usually leads to the over-fitting for quantization policy. Since the accuracy gap between the acquired quantization policy and the optimal one is large, the quantization policy search faces the over-fitting problem.

• Enlarging the size of the dataset for quantization policy search can alleviate the over-fitting of the acquired bitwidth assignment between policy search and model deployment, since we observe that the model achieves similar accuracy for optimal quantization policy and those searched on the full training set of CIFAR-10.

F. Formulation of Rank Errors caused by Attribution Noise

The attribution acquired by Grad-cam contains noise Selvaraju et al. (2017); Sundararajan et al. (2017) which changes the attribution value slightly. However, the errors on the attribution map are significantly amplified by the ranking operation, which deviates the attribution rank of full-precision networks from the correct one obviously in attribution imitation. The generalization risk shown in (8) can be expanded as:

$$\begin{aligned} \mathcal {R}_G(\varvec{W},\mathcal {Q},\varvec{x})=&\,\sum _{i,j}||r(M_{q, ij}[y_x])-r(M_{f, ij}[y_x])||_2^2\nonumber \\ =&\,\sum _{i,j}||r(M_{q, ij}[y_x])-r(M_{f, ij}[y_x]+\delta _{ij})||_2^2 \nonumber \\&\quad +||r(M_{f, ij}[y_x]+\delta _{ij})-r(M_{f, ij}[y_x])||_2^2 \end{aligned}$$

(17)

where $\delta _{ij}$ means the noise of the attribution satisfying Gaussian distribution with zero mean and $\sigma _{ij}$ standard deviation. The cross term in the expansion is regarded to be zero for omission because there is no statistical correlation between the attribution value and the noise. The first term in (17) is the objective that we aim to optimize, and the second term can be represented as the KL-divergence between distribution of the two ranking variables. The KL-divergence can be written as:

$$\begin{aligned}&D_{KL}(p(r(M)=k)||p(r(M+\delta )=k))\nonumber \\ {}&=\int _M p(r(M)=k)\log \frac{p(r(M)=k)}{p(r(M+\delta )=k)}dM\nonumber \\&\quad =\int _M p(r(M)=k)\cdot \frac{\partial p(r(M)=k)}{\partial M}\cdot \frac{\delta }{M}dM\nonumber \\&\quad =C_0\delta \end{aligned}$$

(18)

where we omit the subscript i and j for simplification. All variables related to M and k is deterministic when optimizing (17), and we treat them as a constant $C_0$. Minimizing the second term in (17) equals to minimizing the KL-divergence shown in (18), which is also equivalent to minimizing the standard deviation $\sigma$ in the Gaussian distribution of $\delta$. As semantically similar pixels usually have feature importance in similar distribution, we smooth attribution of these pixels by averaging their attribution value. Therefore, the standard deviation of their noise can be reduced since they are i.i.d. In conclusion, leveraging semantically similar pixels for attribution smoothing can reduce rank errors caused by attribution noise, which provides accurate guidance for quantized models to locate attribution correctly.

G. Ablation Study w.r.t. the Number of Pixels in Attribution Rank Preservation

Since many attribution pixels have the extremely low value (less than $10^{-2}$), imitating these pixels on full-precision networks for quantized ones cannot bring sufficient supervision. The reason is that noise makes most contribution to the attribution pixel value in these cases. Therefore, we only select the top pixels based on their attribution values when minimizing the attribution distance between quantized and full-precision networks, so that the informative localization ability instead of noise in the full-precision networks is mimicked by quantized ones. In order to assign the optimal value of k for selecting top-k pixels in attribution imitation learning, we conducted ablation studies with respect to k and report the accuracy-complexity trade-offs in Table 12. Small k fails to acquire sufficient information on the full-precision attribution, and large k brings much noise in attribution imitation. Both of them degrade the trade-off between the accuracy and model complexity.

Table 12 The accuracy-complexity trade-off across different numbers of top-k pixels for attribution rank preservation

H. Details of Small Datasets for Quantization Policy Search

We introduce the datasets that we carried experiments on. For quantization policy search, we employed the small datasets including CIFAR-10 (Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009), Cars (Krause et al., 2013), Flowers (Nilsback and Zisserman, 2008), Aircraft (Maji et al., 2013), Pets (Parkhi et al., 2012) and Food (Bossard et al., 2014). CIFAR-10 contains 60, 000 images divided into 10 categories with equal number of samples, and CIFAR-100 contains the same number of images which are evenly distributed in 100 classes. Flowers has 8,189 images spread over 102 flower categories. Cars includes 16, 185 samples with 196 types at the level of maker, model and year, and Aircraft contains 10, 200 collected images with 100 samples for each of the 102 aircraft model variants. Pet was created with 37 dog and cat categories with 200 images for each class, and Food contains 32, 135 high-resolution food photos of menu items from the 6 restaurants.

I. Rank Errors for Different Settings for R-GMPQ

Because the full-precision attribution can be affected by noise in network training, the attribution rank may fail to reflect true region importance especially for attribution pixels with similar values. Therefore, we leverage the smoothing techniques to eliminate the noise in the attribution. Since the rank of the true attribution without noise is intractable, we randomly sampled five seeds to train the full-precision networks and used their average attribution as the approximated true attribution. We have reported the attribution rank difference on ImageNet with ResNet18 in Table 13. By comparing Table 13 with Table 2, we know that low attribution rank difference leads to better accuracy-complexity trade-offs.

Table 13 The average rank difference between quantized attribution and approximated true attribution with different smoothing techniques and settings

J. Explanation of Attribution Similarity for Generalizable Quantization Policy Search

Let us assume that $Q_D$ and $Q_S$ are respectively the optimal quantization policies searched on the data in deployment and on our tractable small datasets. The generalization ability of the acquired quantization policy can be demonstrated by the difference between the expected loss of models quantized by $Q_D$ and $Q_S$:

$$\begin{aligned} ~~~~~~~~~~~~J=||L(Q_D,X_{val})-L(Q_S,X_{val})|| \end{aligned}$$

(19)

where $X_{val}$ represents the distribution of validation data in deployment. L(Q, X) means the loss function of the neural networks with the quantization policy Q on the dataset X. Smaller J indicates higher generalization ability of our policy $Q_S$ because the loss is more similar to that of the model quantized by $Q_D$. We expand J as follows:

$$\begin{aligned} J=&\,||L(Q_D,X_{val})-L(R,X_{val})+L(R,X_{val})\nonumber \\&-L(R,X_{sma})+L(R,X_{sma})-L(Q_S,X_{sma})\nonumber \\&+L(Q_S,X_{sma})-L(Q_S,X_{val})||\nonumber \\ \leqslant&\, ||L(Q_D,X_{val})-L(R,X_{val})||+||L(R,X_{sma})\nonumber \\&-L(Q_S,X_{sma})||+||\big (L(R,X_{val})-L(R,X_{sma})\big )\nonumber \\&+\big (L(Q_S,X_{sma})-L(Q_S,X_{val})\big )||\nonumber \\ =&\,J_1+J_2+J_3 \end{aligned}$$

(20)

The first term $J_1$ is the intractable loss gap for validation data in deployment caused by quantization, and it can be regarded as a constant $C_0$. The second term $J_2$ corresponds to the loss gap for training data of our small datasets caused by quantization, and we have minimized this term in our method by optimizing the task risk. The third term $J_3$ can be rewritten as follows:

$$\begin{aligned} J_3&=||\int _{X_{sma}}^{X_{val}}\frac{\partial L(R,X)}{\partial X}dX-\int _{X_{sma}}^{X_{val}}\frac{\partial L(Q_S,X)}{\partial X}dX||\nonumber \\&\leqslant \int _{X_{sma}}^{X_{val}}||\frac{\partial L(R,X)}{\partial X}-\frac{\partial L(Q_S,X)}{\partial X}||dX \end{aligned}$$

(21)

where $\partial L(R,X)/\partial X$ and $\partial L(Q_S,X)/\partial X$ demonstrate the attribution of full-precision and quantized models respectively. Since we require the similar attribution between quantized and full-precision models, $J_3$ is also minimized so that the generalization ability of the acquired quantization policy is enhanced.