CN110647990A - A tailoring method of deep convolutional neural network model based on grey relational analysis - Google Patents

Abstract

The invention discloses a method for cutting a deep convolutional neural network model based on grey relational analysis, comprising: performing data augmentation on target data to obtain more training data; Data is trained, and a set of model parameters that fit the training data is obtained as an experimental model for tailoring; gray correlation analysis is used to quantify the importance of each convolution kernel in the experimental model, and the importance of each convolution kernel is obtained. based on the quantized value of the importance of the convolution kernel, the importance of all convolutions is obtained, and the least important convolution kernel is used as the target convolution kernel; the target convolution kernel and the target convolution kernel are The convolution kernel of the next layer related to the convolution kernel is repeatedly cropped until the stopping condition is satisfied. It realizes the advantages of accurately finding the convolution kernel that has the least impact on the accuracy after being trimmed, increasing the model trimming ratio while ensuring the accuracy, and speeding up the inference operation speed of the new model after trimming.

Description

A tailoring method of deep convolutional neural network model based on grey relational analysis

technical field

The invention relates to the field of neural networks, in particular to a method for cutting a deep convolutional neural network model based on grey relational analysis.

Background technique

Convolutional neural networks have made remarkable theoretical and technological breakthroughs in image classification, target detection and image segmentation, and have more market-acceptable recognition accuracy. However, the huge amount of computation and storage of convolutional neural networks makes it difficult to apply to embedded terminal devices with limited computing power and storage space. Therefore, tailoring the model structure, accelerating the model inference speed, and reducing the model storage capacity are of great significance for the popularization and application of convolutional neural networks. At present, there are various techniques to improve the operation speed of neural networks, such as designing compact model networks, model distillation, low-rank decomposition, model quantization, and model tailoring. As a method for improving computing speed with convenient implementation, good accuracy retention and obvious acceleration effect, model clipping technology has received more and more attention.

In the model cropping method, the key part is to evaluate the importance of the convolution kernel. Whether the convolution kernel that has the least impact on the result after cropping can be accurately found determines whether the accuracy of the cropped model can be maintained, and also determines whether the cropping algorithm can achieve the maximum The inference speed is improved by multiples and the volume is compressed by multiples.

Convolution kernel evaluation methods can be divided into two categories: data-driven and parameter-driven according to the evaluation object.

The parameter-driven method can consume less time in the process of model trimming, but it has a relatively large impact on the accuracy of the model and cannot achieve a high speed improvement. The parameter-driven method directly examines the model parameter W, and evaluates the importance of the convolution kernel channel according to the sum of the parameter values of each channel or whether the parameter value is greater than the threshold. Since it is only necessary to traverse and visit the convolution kernel parameter W once and perform a simple summation calculation when evaluating the importance, no additional calculation process and repeated process are required, so the cutting process takes less time.

Data-driven methods can keep the accuracy of the model better after large-scale cropping. The data-driven method needs to use the training set data and model parameters to obtain the activation value of each layer of the network, and evaluate the importance of the convolution kernel according to some data characteristics of the activation value. Since each picture needs to be input into the network one by one to obtain the activation values of each layer, the amount of calculation is relatively large, and the calculation time is relatively long during the cropping process. However, since the activation values generated by a large number of training set images contain more information, selecting the corresponding convolution kernel as the cropping target according to the activation value can more accurately find the most redundant convolution kernel channels.

Different convolution kernel evaluation methods need to be sorted according to the data information characteristics of different objects. The simpler the information contained in the object, the less computation is required during the cutting process, and the processing time is short, but after cutting, it has a great impact on the model accuracy and cannot be achieved. Large cropping ratio; the richer the information contained in the object, the greater the amount of calculation and the long processing time during the cropping process, but the impact on the model accuracy after cropping is small, and a large cropping ratio can be achieved. A good convolution kernel evaluation method needs to cut as many convolution kernels as possible under the guarantee of accuracy, so as to improve the running speed of the model in the inference stage, so as to ensure the real-time needs of the embedded platform.

SUMMARY OF THE INVENTION

The purpose of the present invention is to, in view of the above problems, to propose a cutting method of a deep convolutional neural network model based on grey relational analysis, so as to accurately find the convolution kernel that has the least impact on the accuracy after being cut, and in the case of ensuring the accuracy It has the advantages of increasing the model clipping ratio and speeding up the inference operation speed of the new model after clipping.

To achieve the above purpose, the technical solution adopted in the embodiment of the present invention is:

A method for tailoring a deep convolutional neural network model based on grey relational analysis, comprising:

Perform data augmentation on the target data to obtain more training data;

The untrained initial network model is trained using the training data, and a set of model parameters fitting the training data is obtained as an experimental model for tailoring;

Use gray correlation analysis to quantify the importance of each convolution kernel in the experimental model, and obtain the quantified value of the importance of each convolution kernel;

The importance of all convolutions is obtained based on the quantized value of the importance of the convolution kernel, and the least important convolution kernel is used as the target convolution kernel;

The target convolution kernel and the next layer of convolution kernels related to the target convolution kernel are repeatedly cropped until the stopping condition is satisfied.

Further, the target data is picture data.

Further, the data augmentation includes: horizontal flipping or brightness fine-tuning.

Further, the untrained initial network model is trained using the training data, as follows:

The untrained initial network model is trained by using the stochastic gradient descent method with the training data, so that the value of the loss function reaches the global minimum.

Further, the stop condition is floating-point operand FLOPs;

Among them, L is the total number of layers of the neural network, i is the serial number of the neural network layer, h, w and c are the height, width and depth of the input feature map of the current layer, n is the depth of the output feature map, and k is the convolution kernel size of.

Further, the gray correlation analysis is used to quantify the importance of each convolution kernel in the experimental model, and the quantified value of the importance of each convolution kernel is obtained, including:

Converting the feature layer of each convolution kernel in the experimental model into a two-dimensional matrix through global average pooling;

Combine the two-dimensional matrix transformed by the global average pooling of all convolution kernel feature layers into an analysis matrix;

selecting a reference sequence within the analysis matrix;

Calculate the correlation coefficient of the comparison sequence of the analysis matrix and the reference sequence, respectively;

calculating the importance of the reference sequence based on the correlation coefficient;

Based on the importance of the reference sequence, a convolution kernel channel importance quantization value is obtained.

Further, obtaining the convolution kernel channel importance quantization value based on the importance of the reference sequence, including:

adding a regularization function on the number of layers based on the importance of the reference sequence;

Based on the regularization function, the convolution kernel channel importance quantization value independent of the number of layers is obtained

Further, the correlation coefficient is:

代表第k个比较序列与参考序列之间的关联度，ρ是分辨系数，

和

为分析矩阵的元素。in,

represents the degree of association between the k-th comparison sequence and the reference sequence, ρ is the resolution coefficient,

and

is the element of the analysis matrix.

Further, the calculating the importance of the reference sequence based on the correlation coefficient includes:

The average correlation degree between the comparison sequence and the reference sequence is obtained based on the correlation coefficient. If the average correlation degree is higher, the features extracted by the convolution kernel channel represented by the reference sequence and the features extracted by the other channels are more Similar, that is, the convolution kernel channel represented by the reference sequence is less important.

Further, the said

where i represents the number of layers, j represents the serial number of the convolution kernel channel, represents the importance of the jth convolution kernel channel.

This embodiment has the following beneficial effects:

In this embodiment, gray correlation analysis is used to quantify the importance of each convolution kernel in the experimental model, so that the importance of each convolution kernel is judged according to the quantized value, and the target volume to be cut is selected according to the importance Accumulate the kernel, trim the target convolution kernel and related convolution kernels, so as to accurately find the convolution kernel that has the least impact on the accuracy after being trimmed, increase the model trimming ratio while ensuring the accuracy, and speed up the new model after trimming. The advantage of inference operation speed.

The method of this implementation uses grey relational analysis to evaluate the importance of the convolution kernel, and cuts out the convolution kernel that contributes little to the result, thereby reducing the amount of computation and improving the inference speed; the convolution kernels of all layers need to be evaluated, and the difference between The importance of the convolution kernels can be compared with each other, eliminating the need to pre-set the clipping ratio for each layer before clipping; the FLOTs ratio before and after model clipping meets the requirements as the signal for clipping stop, and the clipping can be stopped in time after the speed improvement effect meets the requirements. Avoid excessive drop in accuracy; it is universal and can be widely used in various networks and their variants that are currently common; The method of quantitatively measuring the correlation between factors can accurately find the two most closely related factors among multiple factors, and is suitable for finding the correlation between convolution kernels as a cutting basis, so that even if a large proportion of most convolutions are cut out The kernel can still keep the model accuracy basically unchanged; while maintaining the accuracy, the percentage of clipping is increased to speed up the running speed of the model in the inference phase.

The technical solutions of the present invention will be further described in detail below through the accompanying drawings and embodiments.

Description of drawings

1 is a flowchart of a method for cutting a deep convolutional neural network model based on grey relational analysis according to an embodiment of the present invention;

2 is a schematic block diagram of a method for tailoring a deep convolutional neural network model based on grey relational analysis according to an embodiment of the present invention;

3 is a schematic diagram of a current layer convolution kernel channel and a next layer related channel clipping of a method for clipping a deep convolutional neural network model based on gray correlation analysis according to an embodiment of the present invention;

4 is a schematic diagram of residual network trimming of a trimming method for a deep convolutional neural network model based on grey relational analysis according to an embodiment of the present invention;

FIG. 5 is a flowchart for evaluating the importance of convolution kernels in the method for trimming a deep convolutional neural network model based on grey relational analysis according to an embodiment of the present invention.

Detailed ways

The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, but not to limit the present invention.

As shown in Figure 1, a method for tailoring a deep convolutional neural network model based on grey relational analysis includes:

S101: perform data augmentation on the target data to obtain more training data;

S102: Use the training data to train the untrained initial network model, and obtain a set of model parameters that fit the training data as an experimental model for tailoring;

S103: quantify the importance of each convolution kernel in the experimental model by using grey correlation analysis to obtain a quantified value of the importance of each convolution kernel;

S104: Obtain the importance of all convolutions based on the quantized value of the importance of the convolution kernel, and use the least important convolution kernel as the target convolution kernel;

S105: Repeat trimming of the target convolution kernel and the next layer of convolution kernels related to the target convolution kernel until a stopping condition is satisfied.

In a specific implementation manner, the target data may be picture data. Data augmentation, including: horizontal flip or brightness fine-tuning, etc.

In a specific embodiment, the untrained initial network model is trained using the training data, as follows:

The untrained initial network model is trained by using the stochastic gradient descent method with the training data, so that the value of the loss function reaches the global minimum.

In a specific implementation, the stop condition is floating-point operand FLOPs;

Among them, L is the total number of layers of the neural network, i is the serial number of the neural network layer, h, w and c are the height, width and depth of the input feature map of the current layer, n is the depth of the output feature map, and k is the convolution kernel size of.

In a specific embodiment, the gray correlation analysis is used to quantify the importance of each convolution kernel in the experimental model, and the quantified value of the importance of each convolution kernel is obtained, including:

Converting the feature layer of each convolution kernel in the experimental model into a two-dimensional matrix through global average pooling;

Combine the two-dimensional matrix transformed by the global average pooling of all convolution kernel feature layers into an analysis matrix;

selecting a reference sequence within the analysis matrix;

Calculate the correlation coefficient of the comparison sequence of the analysis matrix and the reference sequence, respectively;

calculating the importance of the reference sequence based on the correlation coefficient;

Based on the importance of the reference sequence, a convolution kernel channel importance quantization value is obtained.

In a specific embodiment, the obtaining of the convolution kernel channel importance quantization value based on the importance of the reference sequence includes:

adding a regularization function on the number of layers based on the importance of the reference sequence;

Based on the regularization function, the convolution kernel channel importance quantization value independent of the number of layers is obtained

Further, the correlation coefficient is:

代表第k个比较序列与参考序列之间的关联度，ρ是分辨系数，

和

为分析矩阵的元素。in,

represents the degree of association between the k-th comparison sequence and the reference sequence, ρ is the resolution coefficient,

and

is the element of the analysis matrix.

In a specific embodiment, the calculating the importance of the reference sequence based on the correlation coefficient includes:

The average correlation degree between the comparison sequence and the reference sequence is obtained based on the correlation coefficient. If the average correlation degree is higher, the features extracted by the convolution kernel channel represented by the reference sequence and the features extracted by the other channels are more Similar, that is, the convolution kernel channel represented by the reference sequence is less important.

In specific embodiments, the

代表第j个卷积核通道的重要性。where i represents the number of layers, j represents the serial number of the convolution kernel channel,

represents the importance of the jth convolution kernel channel.

In a specific application scenario,

In this embodiment, the data-driven model clipping method is improved, and gray correlation analysis is used to quantify the importance of the convolution kernel channels, and the unimportant convolution kernel channels are deleted to reduce the amount of calculation and speed up the operation. The entire algorithm flow is mainly composed of Data augmentation, pre-training, evaluation of convolution kernels, clipping stop conditions, etc. as shown in picture 2.

The specific implementation steps are:

(1) Data augmentation, including random horizontal flipping and random brightness fine-tuning, is performed on the image data to obtain more training data, so that the model can better fit various environmental conditions. Horizontal flip is represented as:

I(a,b)=I(Weight-a,b),

where I(a,b) represents the image and Weight is the length of the image.

(2) The initial network model is trained using the augmented data generated in step (1) to obtain a set of model parameters that fit the training data as an experimental model. Use stochastic gradient descent (SGD) to train model parameters, and the parameter update method is:

where w _j represents the jth parameter in the model parameter W, α is the learning rate, m is the batch size, and x ⁽ⁱ⁾ and y ⁽ⁱ⁾ are the images and labels of the training set, respectively.

转变成形状是1×n_i的特征向量V_i ^l；将训练集中m张图片逐一输入网络得到形状为m×n_i的张量T_i，将m作为灰色关联分析的序列数，n_i作为因素个数；将第一个卷积核通道的特征作为参考序列。逐个计算参考序列和比较序列对应元素的绝对差值，分别计算每个比较序列和参考序列的关联系数；重复上述步骤，将第二个卷积核通道的特征作为参考序列计算其重要性，以此类推获得所有卷积核通道的重要性；经过正则化后最终得到一个与层数无关的卷积核通道重要性量化值。如图5所示。(3) Use grey relational analysis to quantify the importance of each convolution kernel in the experimental model obtained in step (2): first, use global average pooling to convert the i-th shape of the original shape h _i ×w _i ×ci _i Layer Feature Map

Transform into a feature vector V _i ^l with a shape of 1×n _i ; input m pictures in the training set into the network one by one to obtain a tensor T _i with a shape of m×n _i , take m as the sequence number of grey relational analysis, and n _i as The number of factors; the feature of the first convolution kernel channel is used as the reference sequence. Calculate the absolute difference between the corresponding elements of the reference sequence and the comparison sequence one by one, and calculate the correlation coefficient of each comparison sequence and the reference sequence respectively; repeat the above steps, and use the feature of the second convolution kernel channel as the reference sequence to calculate its importance, with The importance of all convolution kernel channels is obtained by analogy; after regularization, a convolution kernel channel importance quantization value independent of the number of layers is finally obtained. As shown in Figure 5.

(3.1) Global average pooling:

The output feature of each layer is a three-dimensional matrix, which is not convenient to process and analyze directly. The feature layer is transformed into a 2D matrix by a global average pooling operation. The specific process is as follows:

If the output feature of the i-th layer of the l-th picture is Then the global average pooling process is:

为池化后的特征向量。in

is the feature vector after pooling.

(3.2) Obtain the analysis matrix:

最后向量合并为矩阵T_i：In order to accurately obtain the importance of each convolution kernel channel, it needs to be analyzed according to the information of m pictures in the entire dataset. That is, for the output features of each layer, it is necessary to send m pictures into the convolutional neural network to obtain the output features of this layer and perform global average pooling to obtain m pictures.

Finally the vectors are merged into a matrix T _i :

是向量V_i ¹的第一个向量值。矩阵T_i作为分析矩阵，将m作为分析的序列数，n_i作为分析的因素个数。in

is the first vector value of vector V _i ¹ . The matrix T _i is used as the analysis matrix, m is the number of sequences analyzed, and n _i is the number of factors analyzed.

(3.3) Select the reference sequence:

为：Before each calculation, one factor (convolution kernel channel) needs to be selected as the reference sequence, and the calculation result is the average correlation between other factors and the reference factor, so the whole calculation process needs to select each factor as the reference factor in turn. When the n _i th convolution kernel is used as a reference factor, the n _{i th} row of the matrix T _i needs to be placed in the first row to get

for:

(3.4) Calculate the correlation coefficient:

Calculate the correlation coefficient for each compared sequence and the reference sequence separately as:

代表第k个比较序列与参考序列之间的关联度，ρ是分辨系数。in

represents the degree of association between the k-th comparison sequence and the reference sequence, and ρ is the resolution coefficient.

(3.5) Calculate the reference sequence importance:

The higher the average degree of correlation between the comparison sequence and the reference sequence, the more similar the features extracted by the convolution kernel channel represented by the reference sequence to the features extracted by the rest of the channels, that is, the lower the importance of the convolution kernel channel:

代表第j个卷积核通道的重要性，其重要程度与值的大小成反比。in

Represents the importance of the jth convolution kernel channel, and its importance is inversely proportional to the magnitude of the value.

(3.6) Calculate the importance of the convolution kernel channel:

其中i代表层数，j代表卷积核通道的序号：The channels between different layers have different ranges of eigenvalues due to different input distributions, so when comparing the importance of different layers, an L2 regularization function about the number of layers needs to be added. Finally, a convolution kernel channel importance quantization value independent of the number of layers is obtained

where i represents the number of layers, and j represents the serial number of the convolution kernel channel:

(4) Obtain the quantized value of importance for all convolution kernels according to the calculation, and take the least important convolution kernel as the clipping object; clip the corresponding channel of the target convolution kernel and the convolution kernel of the next layer related to it. Train to make up for the loss of performance; repeat the operation until the clipping stop condition is finally met; and finally train to recover the accuracy loss of the model during clipping. As shown in Figure 3.

(5) The speed of calculation is proportional to the amount of calculation. The indicator of the calculation amount is the number of floating-point operations. The more convolution kernel channels in the convolution operation and the larger the input feature map, the larger the number of floating-point operations. The invention reduces the calculation amount by cutting the number of channels of the convolution kernel, and speeds up the operation speed of the inference stage. The stopping condition of the trimming process is that the ratio of FLOPs between the new model and the initial model meets the requirement of improving the operation speed.

Among them, L is the total number of layers of the neural network, i is the serial number of the neural network layer, h, w and c are the height, width and depth of the input feature map of the current layer, n is the depth of the output feature map, and k is the convolution kernel size of.

The solid-line box in Figure 4 represents the convolution operation. The parameters in the box represent the width of the convolution kernel, the height of the convolution kernel, the number of input channels, and the number of convolution kernels. There are two types of residual blocks. The first skip connection process does not have a convolution operation, as shown in the dotted boxes B and D in the figure. The residual block only cuts the first and second convolution processes, and the third The convolution process is not cut to ensure that the channels before and after the skip connection are consistent. The second residual block skip connection process has convolution operation, as shown in the dotted boxes A and C in the figure, the third convolution layer can theoretically be trimmed, as long as the number of convolution kernels of the skip connection can be changed at the same time, but This changes the input dimension of the next residual block, so both residual blocks only crop 2 convolutional layers.

Finally, it should be noted that the above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, for those skilled in the art, the The technical solutions described in the foregoing embodiments may be modified, or some technical features thereof may be equivalently replaced. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. a cutting method based on the deep convolutional neural network model of grey relational analysis, is characterized in that, comprises: Perform data augmentation on the target data to obtain more training data; The untrained initial network model is trained using the training data, and a set of model parameters fitting the training data is obtained as an experimental model for tailoring; Use gray correlation analysis to quantify the importance of each convolution kernel in the experimental model, and obtain the quantified value of the importance of each convolution kernel; The importance of all convolutions is obtained based on the quantized value of the importance of the convolution kernel, and the least important convolution kernel is used as the target convolution kernel; The target convolution kernel and the next layer of convolution kernels related to the target convolution kernel are repeatedly cropped until the stopping condition is satisfied. 2 . The method for cropping a deep convolutional neural network model based on grey relational analysis according to claim 1 , wherein the target data is picture data. 3 . 3 . The method for cropping a deep convolutional neural network model based on grey relational analysis according to claim 2 , wherein the data augmentation comprises: horizontal flipping or brightness fine-tuning. 4 . 4. the clipping method of the deep convolutional neural network model based on grey relational analysis according to claim 1, is characterized in that, the initial network model without training utilizes described training data to carry out training, is: The untrained initial network model is trained by using the stochastic gradient descent method with the training data, so that the value of the loss function reaches the global minimum. 5. the clipping method of the deep convolutional neural network model based on grey relational analysis according to claim 1, is characterized in that, described stop condition is floating-point operand FLOPs;

Among them, L is the total number of layers of the neural network, i is the serial number of the neural network layer, h, w and c are the height, width and depth of the input feature map of the current layer, n is the depth of the output feature map, and k is the convolution kernel size of. 6. the clipping method of the deep convolutional neural network model based on grey relational analysis according to claim 1, is characterized in that, described utilizes grey relational analysis to carry out the quantification of importance to each convolution kernel in described experimental model , get the quantized value of the importance of each convolution kernel, including: Converting the feature layer of each convolution kernel in the experimental model into a two-dimensional matrix through global average pooling; Combine the two-dimensional matrix transformed by the global average pooling of all convolution kernel feature layers into an analysis matrix; selecting a reference sequence within the analysis matrix; Calculate the correlation coefficient of the comparison sequence of the analysis matrix and the reference sequence, respectively; calculating the importance of the reference sequence based on the correlation coefficient; Based on the importance of the reference sequence, a convolution kernel channel importance quantization value is obtained. 7. The cutting method of the deep convolutional neural network model based on grey relational analysis according to claim 6, wherein the described importance based on the reference sequence, obtains the convolution kernel channel importance quantization value, including : adding a regularization function on the number of layers based on the importance of the reference sequence; Based on the regularization function, the convolution kernel channel importance quantization value independent of the number of layers is obtained

8. the clipping method of the deep convolutional neural network model based on grey relational analysis according to claim 6 or 7, is characterized in that, The correlation coefficient is:

in,

represents the degree of association between the k-th comparison sequence and the reference sequence, ρ is the resolution coefficient,

and

is the element of the analysis matrix. 9. The method for tailoring a deep convolutional neural network model based on grey relational analysis according to claim 8, wherein the calculating the importance of the reference sequence based on the relational coefficient comprises: The average correlation degree between the comparison sequence and the reference sequence is obtained based on the correlation coefficient. If the average correlation degree is higher, the features extracted by the convolution kernel channel represented by the reference sequence and the features extracted by the other channels are more Similar, that is, the convolution kernel channel represented by the reference sequence is less important. 10. The cutting method of the deep convolutional neural network model based on grey relational analysis according to claim 9, wherein the

where i represents the number of layers, j represents the serial number of the convolution kernel channel,

represents the importance of the jth convolution kernel channel.

Images