CN114881203A - Model reasoning method, device and electronic device - Google Patents

Abstract

The present disclosure provides a model inference method, device and electronic device, and relates to the technical fields of artificial intelligence such as deep learning and computer vision. The specific implementation scheme is: performing equivalent transformation on the input image data of the target model to obtain an input matrix; The processing priority of the row data of the input matrix relative to the column data; according to the data processing method, the first weight matrix of the target model and the input matrix are multiplied to obtain a model inference result. A weight matrix is obtained by performing sparse processing on the second weight matrix obtained by training the target model based on the preset sparsity.

Description

Model reasoning method, device and electronic device

technical field

The present disclosure relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning and computer vision, and in particular to a model inference method, device and electronic device.

Background technique

With the rapid development of artificial intelligence, in the fields of computer vision such as image classification and recognition, the real-time requirements of neural network model inference are getting higher and higher. Therefore, it is obvious to realize lightweight, high-performance and low-energy inference acceleration on the mobile terminal. critical.

In order to solve the problem that the neural network model has a large amount of data storage and high computational complexity, the weight matrix of the neural network model can usually be sparsely processed by pruning, and the input weight matrix can be sparsed based on the neural network model. The image data is subjected to convolution calculation to obtain the model inference result.

SUMMARY OF THE INVENTION

The present disclosure provides a model inference method, device and electronic device.

According to a first aspect of the present disclosure, there is provided a model inference method, comprising:

Perform equivalent transformation on the input image data of the target model to obtain the input matrix;

determining a data processing mode of the input matrix based on the data amount of the input picture data, where the data processing mode is used to represent the processing priority of the row data of the input matrix relative to the column data;

According to the data processing method, the first weight matrix of the target model and the input matrix are multiplied to obtain a model inference result, and the first weight matrix is obtained by training the target model based on the preset sparsity. The second weight matrix of is obtained by sparse processing.

According to a second aspect of the present disclosure, there is provided a model inference device, comprising:

The conversion module is used to perform equivalent conversion on the input image data of the target model to obtain the input matrix;

a determining module, configured to determine a data processing mode of the input matrix based on the data amount of the input picture data, where the data processing mode is used to represent the processing priority of the row data of the input matrix relative to the column data;

a multiplication processing module, configured to perform multiplication processing on the first weight matrix of the target model and the input matrix according to the data processing method to obtain a model inference result, where the first weight matrix is based on a preset sparsity The second weight matrix obtained by training the target model is obtained by sparse processing.

According to a third aspect of the present disclosure, there is provided an electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform any of the methods of the first aspect.

According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform any one of the methods of the first aspect.

According to a fifth aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements any one of the methods of the first aspect.

The technology according to the present disclosure solves the problem of relatively low model inference efficiency, and improves the efficiency of model inference.

It should be understood that what is described in this section is not intended to identify key or critical features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of drawings

The accompanying drawings are used for better understanding of the present solution, and do not constitute a limitation to the present disclosure. in:

1 is a schematic flowchart of a model inference method according to a first embodiment of the present disclosure;

Fig. 2 is a schematic diagram of the calculation principle of a first weight matrix and an input matrix;

Fig. 3 is a schematic diagram of the processing principle of performing column first and then row on the input matrix;

Fig. 4 is a schematic diagram of the processing principle of performing first row and then column on the input matrix;

Fig. 5 is the data storage format schematic diagram of the first weight matrix;

6 is a schematic structural diagram of a model inference apparatus according to a second embodiment of the present disclosure;

7 is a schematic block diagram of an example electronic device used to implement embodiments of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

first embodiment

As shown in FIG. 1 , the present disclosure provides a model inference method, including the following steps:

Step S101: Perform equivalent transformation on the input image data of the target model to obtain an input matrix.

In this embodiment, the model inference method relates to the technical field of artificial intelligence, in particular to the technical field of deep learning and computer vision, and can be widely used in image processing scenarios such as image recognition and segmentation. The model inference method of the embodiment of the present disclosure may be executed by the model inference apparatus of the embodiment of the present disclosure. The model inference apparatus of the embodiment of the present disclosure may be configured in any electronic device to execute the model inference method of the embodiment of the present disclosure. The electronic device may be a server or a terminal device, which is not specifically limited here.

The target model can be a neural network model, which can include at least one convolution kernel, each convolution kernel corresponds to at least one channel, and the weight matrix of the target model can be constructed by training the weights of each convolution kernel during convolution calculation . When the image is input to the target model, the target model can perform the convolution calculation on the weight matrix and the input image data to perform model inference, and obtain the model inference result, thereby realizing image processing.

The input picture data can be the pixel data of the input picture, the input picture data can be the pixel data of one picture, or the pixel data of two pictures, or even the pixel data of multiple pictures, among which, the input picture of the target model can be It is a grayscale image or a color image, which is not specifically limited here.

When the input picture of the target model is a color picture, or the number of input pictures of the target model is at least two, the input picture data is three-dimensional data. For example, for a color picture, its pixel data can be C1×H×W. Three-dimensional data, C1 is the channel number of the color picture, H is the height of the color picture, and W is the width of the color picture.

When the input image of the target model includes multiple images and the size of each input image is the same, the input image data can be C2×H×W three-dimensional data, and C2 can be the sum of the number of channels of the multiple input images .

The data layout of the input image data of the target model can be equivalently converted in the im2col method or in other ways. Taking im2col as an example, im2col uses the determinant to perform equivalent conversion on the three-dimensional data to optimize the convolution operation, such as converting the three-dimensional image The data C2×H×W is transformed into a two-dimensional matrix of C2×C3.

In an optional implementation manner, the data of each channel in the input picture data can be expanded into a row of data, and the data of multiple channels can be arranged in rows, which can be arranged in a two-dimensional matrix to obtain the input matrix. Among them, the number of rows of the input matrix is the total number of channels of the input image data, and the number of columns of the input matrix is H×W.

In this step, by equivalently converting the input image data of the target model into a matrix, the convolution operation can be optimized and converted into a matrix multiplication operation, thereby simplifying the inference operation of the model.

Step S102: Determine a data processing mode of the input matrix based on the data amount of the input picture data, where the data processing mode is used to represent the processing priority of the row data of the input matrix relative to the column data.

In this step, the data amount of the input picture data may be the number of pixels of the data picture data, for example, for the input picture data of C2×H×W, the data amount is equal to C2*H*W.

The data processing method of the input matrix can be used to characterize the processing priority of the row data of the input matrix relative to the column data, which can include two cases, one is that the processing priority of the row data is higher than the processing priority of the column data, the other is The processing priority for column data is higher than the processing priority for row data.

The processing priority of row data is higher than the processing priority of column data, indicating that all row data is processed first, and then the column data is processed, while the processing priority of column data is higher than that of row data, indicating that all column data is processed first, and then row data is processed. .

The row and column access of the input matrix can be adaptively performed based on the data amount of the input picture data, so as to determine the calculation method of the input matrix.

In this step, the row and column access of the input matrix is adaptively performed based on the data amount of the input image data. When the amount of data is small, since the row data may be less, the data of one row of the input matrix can be quickly retrieved by means of data prefetching. Take out for model inference calculation, so the calculation method of input matrix first row and then column can realize faster model inference calculation. However, when there is a large amount of data, since there may be more row data, faster model inference calculation can be achieved by calculating the input matrix in columns first and then rows.

Step S103: According to the data processing method, multiply the first weight matrix of the target model and the input matrix to obtain a model inference result, and the first weight matrix is based on the preset sparsity. The second weight matrix obtained by model training is obtained by sparse processing.

In this step, the first weight matrix of the target model may be the weight matrix after sparse processing, and the target model may be trained by techniques such as migration learning, automated deep learning platform AutoDL, hyperparameter tuning, etc., to obtain the result obtained by training on the business data set The weight value of the target model is obtained, and the second weight matrix is obtained.

Among them, the number of rows of the second weight matrix is the total number of channels of the target model, the number of columns is the number of convolution kernels, and the total number of channels of the target model is usually equal to the total number of channels of the input image data.

The second weight matrix can be sparsed based on the preset sparsity. The sparse processing refers to increasing the proportion of zero values in the weight matrix, that is, the sparse processing is to change the non-zero values in the weight matrix into zeros. The target model is compressed to reduce the effective number of weights in the weight matrix of the target model. The preset sparsity may be set according to the actual situation. Generally, in order to improve the speed of model inference as much as possible and improve the real-time performance of model inference, the preset sparsity may be greater than 50%.

An unstructured pruning technique may be used to sparse the second weight matrix, or a structured pruning technique may be used to sparse the second weight matrix, which is not specifically limited here.

Among them, the unstructured pruning technique can refer to the convolution factor in the random pruning convolution kernel, that is, the weight value, that is, unstructured pruning allows clipping at any position of the weight matrix. In an optional embodiment, the second weight matrix may be sparsed according to the importance factor of each effective weight in the second weight matrix, for example, the relatively unimportant effective weights that are arranged after the effective weights are reset. zero. The structured pruning technique is to convolve the channel and the convolution kernel, such as setting the entire weight group (corresponding to the entire convolution kernel) of a row in the weight matrix to zero.

On the basis of obtaining the first weight matrix, the first weight matrix is stored, and then in the model inference process, the first weight matrix and The input matrix is multiplied to obtain the model inference result.

Figure 2 is a schematic diagram of the calculation principle of the first weight matrix and the input matrix. As shown in Figure 2, the left figure represents the first weight matrix, and each square represents a weight. When the square contains a numerical value, the square represents an effective weight. , the value in the square represents the weight value of the valid weight, when the square does not include a value, the square represents an invalid weight, and its weight value is zero. The right figure in Figure 2 represents the input matrix, and the number of columns of the first weight matrix is equal to the number of rows of the input matrix. In this way, the first weight matrix and the input matrix can be multiplied to obtain the model inference result.

In the process of multiplying the first weight matrix of the target model and the input matrix, the relevant data of the input matrix can be continuously obtained by using the data prefetching technology through the cache memory, and the embedded assembly instruction can be used to perform the first weighting process. Vector multiplication of the rows of the matrix with the columns of the input matrix to obtain the matrix multiplication result.

The data in the matrix multiplication result can accumulate a bias term to obtain the model inference result. The model inference results can then be sent to a post-processing module for image processing.

In this embodiment, an input matrix is obtained by performing equivalent transformation on the input picture data of the target model; based on the data amount of the input picture data, a data processing method of the input matrix is determined, and the data processing method is used to represent The processing priority of the row data of the input matrix relative to the column data; according to the data processing method, the first weight matrix of the target model and the input matrix are multiplied to obtain a model inference result, the The first weight matrix is obtained by performing sparse processing on the second weight matrix obtained by training the target model based on the preset sparsity. In this way, the model inference speed after model compression can be improved, thereby improving the real-time performance of the model inference.

Optionally, the step S102 specifically includes:

In the case where the data amount of the input picture data is greater than a preset threshold, it is determined that the data processing mode is a first processing mode, and the first processing mode is used to represent the processing priority of the column data of the input matrix greater than The processing priority of row data;

When the size of the input picture data is less than or equal to the preset threshold, determine that the data processing mode is a second processing mode, and the second processing mode is used to represent the processing priority of the row data of the input matrix Greater than the processing priority of column data.

In this embodiment, a preset threshold can be set. For example, the preset threshold is 128k. When the data amount is less than or equal to 128k, it can indicate that the data amount of the input picture data is relatively small, and the data processing method of the input matrix is determined to be first and then later. Columns, otherwise the data processing method of the input matrix is determined as columns first and then rows. In this way, the row and column access of the input matrix can be adaptively performed according to the data amount of the input picture data.

Optionally, the step S103 specifically includes:

When the data processing method is the first processing method, acquire target column data in the input matrix, and associate each row data in the first weight matrix with the target column in the input matrix respectively performing vector multiplication processing on the data to obtain a model inference result, and the target column data is at least part of the column data in the input matrix;

When the data processing mode is the second processing mode, for the target row data in the first weight matrix, all column data in the input matrix are acquired, and all column data in the first weight matrix are The target row data is vector multiplied with each column of data in the first weight matrix to obtain a model inference result, and the target row data is any row data in the first weight matrix.

In this embodiment, when it is determined that the data processing method of the input matrix is the processing method of the column first and then the row, the column data can be processed first, and then the row data can be processed, that is, some column data of the input matrix are obtained. Column data, each row of data in the first weight matrix is vector multiplied with these column data in the input matrix, to obtain some data in the model inference result. In the case where the multiplication process of the first weight matrix and the input matrix is not completed, continue to obtain column data that has not been calculated in the input matrix, and for these column data, compare each row of data in the first weight matrix with the column data in the input matrix. Perform vector multiplication processing to obtain other data in the model inference result, until the multiplication processing of the first weight matrix and the input matrix is completed, that is, the processing of all row data of the input matrix is finally completed. In this way, the matrix multiplication of the first weight matrix and the input matrix, that is, the model inference calculation, can be realized.

FIG. 3 is a schematic diagram of the processing principle of performing column first and then row processing on the input matrix. As shown in FIG. 3 , when the input matrix 301 (the 4×8 matrix on the left side of the equal sign shown in FIG. 3 ) is obtained, the first column to the first column In the case of 4 columns of column data (target column data 3011), each row of data (represented by 3021, 3022, 3023 and 3024 respectively) in the first weight matrix 302 (the 4×4 matrix in FIG. 3 ) is respectively Perform vector multiplication with the target column data in the input matrix to obtain some data in the model inference result 303 (the 4×8 matrix on the right side of the equal sign as shown in Figure 3), which are represented by 3031, 3032, 3033 and 3034 respectively. . Wherein, the processing can be performed successively according to the calculation principle of the ID 1, ID 2, ID 3 and ID 4 (the digital ID on the left side of the input matrix in FIG. 3).

In the case where it is determined that the data processing method for the input matrix is the row-first-column processing method, the row data may be processed first, and then the column data may be processed. For a row of data in the first weight matrix, obtain all column data in the input matrix, and perform vector multiplication of the row data in the first weight matrix with each column of data in the first weight matrix to obtain the model inference result. some data. In the case where the multiplication process of the first weight matrix and the input matrix is not completed, continue to perform another row of data in the first weight matrix with each column of data in the first weight matrix. The vector multiplication process is performed to obtain other data in the model inference result, until the multiplication process of the first weight matrix and the input matrix is completed, that is, the processing of all row data of the input matrix is finally completed. In this way, the matrix multiplication of the first weight matrix and the input matrix, that is, the model inference calculation, can be realized.

FIG. 4 is a schematic diagram of the processing principle of the first row and then the column of the input matrix. As shown in FIG. 4 , the calculation principle represented by the mark 1 and the mark 2 indicates that: for the first weight matrix 401 (4×4 shown in FIG. 4 ) The row data 4011 of the first row in the matrix), perform vector multiplication of the row data with each column data in the input matrix 402 (the 4×8 matrix on the left side of the equal sign as shown in Figure 4) to obtain the model inference result Some data in 403 (the 4×8 matrix on the right side of the equal sign shown in Figure 4) are represented by 4031 and 4032, respectively. The calculation principle represented by the identifier 3 and the identifier 4 indicates that: for the row data 4012 of the second row in the first weight matrix 401, the row data and each column data in the input matrix 402 are respectively subjected to vector multiplication processing to obtain the model inference result. Other data in 403 are represented by 4033 and 4034 respectively.

Optionally, the first weight matrix is represented by first information, and the first information includes position information of effective weights in the first weight matrix, weight data, and information on the number of effective weights in each row, the Step S103 specifically includes:

For each row in the first weight matrix, obtain weight data and position information of the effective weight of the row in the first weight matrix based on the quantity information of the effective weight of the row in the first weight matrix;

Based on the position information of the effective weight of the row, obtain the data corresponding to the position information of each column in the input matrix;

Perform vector multiplication processing on the weight data of the effective weight of the row in the first weight matrix and the data corresponding to the position information in each column of the input matrix to obtain a model inference result.

In this implementation manner, in order to reduce the data storage amount of the target model, the first weight matrix may be represented by first information, where the first information includes position information of effective weights in the first weight matrix, weight data, and each row The quantity information of the effective weights, that is, only the relevant information of the effective weights in the first weight matrix can be stored.

The position information of the effective weights in the first weight matrix can be represented by the actual position of each effective weight in the first weight matrix, for example, the effective weight A is located in the first row and the first column. The position information of the valid weights in the first weight matrix can also be represented by the position difference between each adjacent valid weights determined in a preset manner, for example, according to the arrangement order of columns from left to right and rows from top to bottom, The position difference between every two adjacent effective weights is stored.

Correspondingly, when the first weight matrix and the input matrix are multiplied, the first weight matrix and the input matrix may be multiplied according to the positions of the effective weights in the first weight matrix. For example, for the first row in the first weight matrix, the weight data of the effective weights in the first row and the data corresponding to the effective weight positions in each column in the input matrix can be multiplied and accumulated to obtain the model inference result.

In the implementation process, for each row in the first weight matrix, based on the quantity information of the effective weight of the row in the first weight matrix, the weight data and position information of the effective weight of the row in the first weight matrix can be obtained, and the The weight data of the effective weight of the row in the first weight matrix and the data corresponding to the position information in each column of the input matrix are subjected to vector multiplication processing to obtain a model inference result.

As shown in Figure 2, for the first row in the first weight matrix, the weight data of its effective weights are located in the 1st, 3rd, 4th, 5th, 7th, 9th, 10th, and 12th columns, respectively, then the first weight matrix can be The valid data in this row is vector multiplied with the data in the 1st, 3rd, 4th, 5th, 7th, 9th, 10th, and 12th rows in each column of the input matrix to obtain the model inference result.

In this embodiment, the matrix multiplication process of the input matrix is completed according to the quantity information, position and weight data of the effective weights in the first weight matrix, that is, the model inference calculation is completed, the computational complexity of the target model can be reduced, and the model can be further improved. Inference speed.

Optionally, before the step S103, it further includes:

Based on the size of the data cache unit and the size of the vector processing unit, the input matrix is subjected to block processing for row data to obtain a plurality of data processing blocks;

The step S103 specifically includes:

According to the data processing method, vector multiplication is performed on each row of data in the first weight matrix of the target model and the column data constructed by the plurality of data processing blocks to obtain a model inference result.

In this implementation manner, the input matrix can be processed in rows and columns to adapt to the size of the hardware cache of a central processing unit (Central Processing Unit, CPU) and the size of data processed by instructions, thereby improving data access efficiency and data calculation efficiency.

In an optional implementation manner, the cache may be an L1 cache, and its data cache unit, that is, the cache line cacheline, may be 64. In order to improve the data access efficiency, the input matrix may be processed in blocks of 64 for the row data. Faster reading of input matrix data.

In addition, embedded assembly instructions can be used to implement vector multiplication of the row data of the first weight matrix and the column data of the input matrix. In order to adapt to the size of the vector processing unit, the input matrix can be divided into blocks by integer multiples of the vector processing unit. For example, when the neon assembly instruction is used to process data, the vector processing unit is 8, and the input matrix can be processed in blocks in units of integer multiples of 8.

In order to adapt to the data boundary of the input matrix, if the data boundary is not enough to perform the block processing with 8, the block processing can also be performed in units of 4, or the block processing can be performed in units of 1.

For example, for a row of data with 125 columns, it can be divided into 6 data processing blocks, with 64, 32, 16, 8, 4, and 1 columns respectively.

Correspondingly, data processing blocks can be continuously obtained through cache and data prefetching technology to obtain relevant data of the input matrix, and embedded assembly instructions can be used to combine the row data of the first weight matrix with the data processing blocks constructed by multiple data processing blocks. The column data is multiplied by vectors to obtain the model inference result. In this way, data access efficiency and data calculation efficiency can be improved, and the efficiency of model inference can be further improved.

The process of model inference of the target model based on the first weight matrix obtained after compression is described in detail above. The compression process of the target model, that is, the acquisition process of the first weight matrix, is described in detail below.

Optionally, the method further includes:

Obtain second information obtained after the target model is trained, the second information includes the second weight matrix and the importance factor of each effective weight in the second weight matrix;

Based on the preset sparsity, the second weight matrix is sparsed according to the importance factor of each effective weight in the second weight matrix to obtain the first weight matrix of the target model.

In this embodiment, unstructured pruning can be performed on the target model.

The importance factor of each effective weight in the second weight matrix can represent the importance of the effective weight in the model inference process.

There are various ways to obtain the second information. For example, techniques such as migration learning, automated deep learning platform AutoDL, and hyperparameter tuning can be used to train the target model, so as to train on the business data set to obtain the weight value of the target model, and to obtain the first Two weight matrices, and training can also obtain the importance factor of each effective weight in the second weight matrix. For another example, the second information sent by other electronic devices after training the target model may be received.

In the case where the second information is obtained, the effective weights in the second weight matrix can be sorted in descending order of the importance factors of each effective weight in the second weight matrix, and based on the preset sparsity, the effective weights can be sorted. After the effective weight is reset to zero, the second weight matrix is sparsely processed to obtain the first weight matrix. For example, if the sparsity of the second weight matrix is 40% and the preset sparsity is 60%, 20% of the effective weights in the second weight matrix need to be reset to zero.

When the first weight matrix is obtained, model inference can be performed based on the first weight matrix.

In this embodiment, based on the preset sparsity, the second weight matrix is sparsed according to the importance factor of each effective weight in the second weight matrix. In this way, unstructured pruning can be realized. , by cutting the effective weight according to the importance factor of the effective weight, under the premise of ensuring the accuracy of the model inference, a high model compression rate can be achieved.

Optionally, the first weight matrix is characterized by first information, and the method further includes:

storing the weight value of each effective weight in the first weight matrix to obtain weight data;

storing the position difference between every two adjacent valid weights determined in the first weight matrix according to a preset method to obtain position information;

storing the effective weight quantity corresponding to each channel in the first weight matrix to obtain quantity information;

The first information includes the weight data, location information, and the quantity information.

In this embodiment, the first weight matrix may be stored by using the first information.

The first information may store the effective weights in the first weight matrix through different arrays, that is, the first information may include multiple arrays, and the first information may also store the effective weights in the first weight matrix through a data object, that is, the first information It can also be a data object, which is not specifically limited here.

In an optional embodiment, the first weight matrix may be traversed, and the weight value of the effective weight, the address difference between the current non-zero weight value and the previous non-zero weight value, and the weight quantity of the effective weight of the corresponding channel are sequentially stored. in different arrays.

Fig. 5 is a schematic diagram of the data storage format of the first weight matrix. As shown in Fig. 5, the left picture is the first weight matrix. The first weight matrix can be traversed in the order 501 of rows from left to right and columns from top to bottom , store the address difference between the current non-zero weight value and the previous non-zero weight value in the "offset" array, store the number of effective weights of the corresponding channel in the "nonzero_num" array, and store the weight value of the effective weight in the "nonzero_num" array. Stored in the "val" array.

Among them, if the row label corresponding to the current non-zero weight value is greater than the row label of the previous non-zero weight value, the position difference is greater than 0, and if the row label corresponding to the current non-zero weight value is smaller than the row label of the previous non-zero weight value, The position difference is less than 0. If the row label corresponding to the current non-zero weight value is equal to the row label of the previous non-zero weight value, the position difference is equal to 0.

Additionally, the actual position of the first effective weight in the first weight matrix can be stored, so that based on the actual position and the "offset" array, the actual position of each effective weight in the first weight matrix can be determined.

In this embodiment, the relevant information of the effective weights in the first weight matrix is stored by the first information, and the position difference of every two adjacent effective weights in the first weight matrix is stored, so that the data storage capacity of the model can be reduced, and the data storage capacity of the model can be further reduced. Improve the compression ratio of the model.

Optionally, based on the preset sparsity, according to the importance factor of each effective weight in the second weight matrix, sparse the second weight matrix to obtain the first weight of the target model. matrix, including:

Based on the preset sparsity, according to the importance factor of each effective weight in the second weight matrix, sparse the second weight matrix to obtain a third weight matrix;

Quantizing the third weight matrix to obtain the first weight matrix.

In this embodiment, after the third weight matrix is obtained by sparse processing, the effective weights in the third weight matrix can be quantized, and the first weight matrix with low bits of integer type can be obtained through quantization, which can further reduce the complexity of the model. The amount of data storage and the compression ratio of the model are improved.

Second Embodiment

As shown in FIG. 6, the present disclosure provides a model inference apparatus 600, including:

The conversion module 601 is used to perform equivalent conversion on the input picture data of the target model to obtain an input matrix;

A determination module 602, configured to determine a data processing mode of the input matrix based on the data amount of the input picture data, where the data processing mode is used to represent the processing priority of the row data of the input matrix relative to the column data;

A multiplication processing module 603, configured to perform multiplication processing on the first weight matrix of the target model and the input matrix according to the data processing method to obtain a model inference result, where the first weight matrix is based on a preset sparseness The second weight matrix obtained by training the target model is obtained by sparse processing.

Optionally, the determining module 602 includes:

A first determination unit, configured to determine that the data processing mode is a first processing mode when the data amount of the input picture data is greater than a preset threshold, and the first processing mode is used to characterize the input matrix. The processing priority of column data is greater than that of row data;

a second determining unit, configured to determine that the data processing mode is a second processing mode when the size of the input picture data is less than or equal to the preset threshold, and the second processing mode is used to represent the input matrix The processing priority of row data is higher than that of column data.

Optionally, the multiplication processing module 603 is specifically used for:

When the data processing method is the first processing method, acquire target column data in the input matrix, and associate each row data in the first weight matrix with the target column in the input matrix respectively performing vector multiplication processing on the data to obtain a model inference result, and the target column data is at least part of the column data in the input matrix;

When the data processing mode is the second processing mode, for the target row data in the first weight matrix, all column data in the input matrix are acquired, and all column data in the first weight matrix are The target row data is vector multiplied with each column of data in the first weight matrix to obtain a model inference result, and the target row data is any row data in the first weight matrix.

Optionally, the first weight matrix is represented by first information, and the first information includes position information of effective weights in the first weight matrix, weight data, and information on the number of effective weights in each row, the The multiplication processing module 603 is specifically used for:

For each row in the first weight matrix, obtain weight data and position information of the effective weight of the row in the first weight matrix based on the quantity information of the effective weight of the row in the first weight matrix;

Based on the position information of the effective weight of the row, obtain the data corresponding to the position information of each column in the input matrix;

Perform vector multiplication processing on the weight data of the effective weight of the row in the first weight matrix and the data corresponding to the position information in each column of the input matrix to obtain a model inference result.

Optionally, the device further includes:

a block processing module for performing block processing on the row data based on the size of the data cache unit and the size of the vector processing unit to obtain a plurality of data processing blocks;

The multiplication processing module 603 is specifically configured to perform vector multiplication processing on each row data in the first weight matrix of the target model and the column data constructed by the multiple data processing blocks according to the data processing method , get the model inference result.

Optionally, the device further includes:

an acquisition module, configured to acquire second information obtained after the target model is trained, the second information includes the second weight matrix and the importance factor of each effective weight in the second weight matrix;

A sparse processing module, configured to perform sparse processing on the second weight matrix according to the importance factor of each effective weight in the second weight matrix based on the preset sparsity, to obtain the first value of the target model. weight matrix.

Optionally, the first weight matrix is characterized by first information, and the device further includes:

a first storage module, configured to store the weight value of each effective weight in the first weight matrix to obtain weight data;

a second storage module, configured to store the position difference between each two adjacent effective weights determined in the first weight matrix in a preset manner to obtain position information;

a third storage module, configured to store the effective weight quantity corresponding to each channel in the first weight matrix to obtain quantity information;

The first information includes the weight data, location information, and the quantity information.

Optionally, the thinning processing module is specifically used for:

Based on the preset sparsity, according to the importance factor of each effective weight in the second weight matrix, sparse the second weight matrix to obtain a third weight matrix;

Quantizing the third weight matrix to obtain the first weight matrix.

The model inference apparatus 600 provided by the present disclosure can implement the various processes implemented by the model inference method embodiments, and can achieve the same beneficial effects. To avoid repetition, details are not described here.

In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information involved are all in compliance with relevant laws and regulations, and do not violate public order and good customs.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

7 shows a schematic block diagram of an example electronic device that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 7 , the electronic device 700 includes a computing unit 701 that can be programmed according to a computer program stored in a read only memory (ROM) 702 or loaded into a random access memory (RAM) 703 from a storage unit 708 . Various appropriate actions and processes are performed. In the RAM 703, various programs and data necessary for the operation of the device 700 can also be stored. The computing unit 701 , the ROM 702 , and the RAM 703 are connected to each other through a bus 704 . An input/output (I/O) interface 705 is also connected to bus 704 .

Various components in the electronic device 700 are connected to the I/O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc. etc.; and a communication unit 709, such as a network card, modem, wireless communication transceiver, and the like. The communication unit 709 allows the device 700 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

Computing unit 701 may be various general-purpose and/or special-purpose processing components with processing and computing capabilities. Some examples of computing units 701 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the model inference method. For example, in some embodiments, the model inference method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 708 . In some embodiments, part or all of the computer program may be loaded and/or installed on device 700 via ROM 702 and/or communication unit 709 . When a computer program is loaded into RAM 703 and executed by computing unit 701, one or more steps of the model inference method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the model inference method by any other suitable means (eg, by means of firmware).

Various implementations of the systems and techniques described herein above may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips system (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, performs the functions/functions specified in the flowcharts and/or block diagrams. Action is implemented. The program code may execute entirely on the machine, partly on the machine, partly on the machine and partly on a remote machine as a stand-alone software package or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a distributed system server, or a server combined with blockchain.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, no limitation is imposed herein.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure should be included within the protection scope of the present disclosure.

Claims (19)

1. A method of model inference, comprising:

performing equivalent transformation on input picture data of a target model to obtain an input matrix;

determining a data processing mode of the input matrix based on the data volume of the input picture data, wherein the data processing mode is used for representing the processing priority of row data relative to column data of the input matrix;

and according to the data processing mode, multiplying the first weight matrix of the target model and the input matrix to obtain a model reasoning result, wherein the first weight matrix is obtained by performing sparsification processing on a second weight matrix obtained by training the target model based on preset sparsity.

2. The method of claim 1, wherein the determining a data processing manner of the input matrix based on the data amount of the input picture data comprises:

determining that the data processing mode is a first processing mode under the condition that the data volume of the input picture data is larger than a preset threshold value, wherein the first processing mode is used for representing that the processing priority of the column data of the input matrix is larger than the processing priority of the row data;

and under the condition that the size of the input picture data is smaller than or equal to the preset threshold, determining that the data processing mode is a second processing mode, wherein the second processing mode is used for representing that the processing priority of the row data of the input matrix is larger than the processing priority of the column data.

3. The method according to claim 2, wherein said multiplying the first weight matrix of the target model and the input matrix according to the data processing manner to obtain a model inference result comprises:

under the condition that the data processing mode is the first processing mode, acquiring target column data in the input matrix, and performing vector multiplication processing on each row of data in the first weight matrix and the target column data in the input matrix respectively to obtain a model reasoning result, wherein the target column data is at least part of column data in the input matrix;

and under the condition that the data processing mode is the second processing mode, acquiring all column data in the input matrix aiming at target row data in the first weight matrix, and performing vector multiplication processing on the target row data in the first weight matrix and each column data in the first weight matrix respectively to obtain a model reasoning result, wherein the target row data is any row data in the first weight matrix.

4. The method of claim 1, wherein the first weight matrix is characterized by first information, the first information includes position information of effective weights in the first weight matrix, weight data, and number information of effective weights per row, and the multiplying process is performed on the first weight matrix of the target model and the input matrix to obtain a model inference result, and includes:

for each row in the first weight matrix, acquiring weight data and position information of the effective weights of the row in the first weight matrix based on the quantity information of the effective weights of the row in the first weight matrix;

acquiring data of each column in the input matrix corresponding to the position information based on the position information of the effective weight of the row;

and carrying out vector multiplication on the weight data of the effective weight of the row in the first weight matrix and the data of each column corresponding to the position information in the input matrix to obtain a model reasoning result.

5. The method according to claim 1, wherein before the multiplying the first weight matrix of the target model and the input matrix according to the data processing manner to obtain the model inference result, the method further comprises:

based on the size of a data cache unit and the size of a vector processing unit, performing block processing on the input matrix aiming at row data to obtain a plurality of data processing blocks;

the multiplying the first weight matrix of the target model and the input matrix according to the data processing mode to obtain a model inference result, including:

and according to the data processing mode, carrying out vector multiplication processing on each row of data in the first weight matrix of the target model and the column data constructed by the plurality of data processing blocks to obtain a model reasoning result.

6. The method of claim 1, further comprising:

acquiring second information obtained after the target model is trained, wherein the second information comprises the second weight matrix and an importance factor of each effective weight in the second weight matrix;

and based on preset sparsity, performing sparsification processing on the second weight matrix according to the importance factor of each effective weight in the second weight matrix to obtain a first weight matrix of the target model.

7. The method of claim 6, wherein the first weight matrix is characterized by first information, the method further comprising:

storing the weight value of each effective weight in the first weight matrix to obtain weight data;

storing the position difference between every two adjacent effective weights determined according to a preset mode in the first weight matrix to obtain position information;

storing the effective weight quantity corresponding to each channel in the first weight matrix to obtain quantity information;

the first information includes the weight data, location information, and the quantity information.

8. The method of claim 6, wherein the obtaining the first weight matrix of the target model by performing sparsification on the second weight matrix according to an importance factor of each effective weight in the second weight matrix based on a preset sparsity degree comprises:

based on a preset sparsity, performing sparsification processing on the second weight matrix according to the importance factor of each effective weight in the second weight matrix to obtain a third weight matrix;

and quantizing the third weight matrix to obtain the first weight matrix.

9. A model inference apparatus, comprising:

the conversion module is used for carrying out equivalent conversion on input picture data of the target model to obtain an input matrix;

the determining module is used for determining a data processing mode of the input matrix based on the data volume of the input picture data, wherein the data processing mode is used for representing the processing priority of row data relative to column data of the input matrix;

and the multiplication processing module is used for multiplying the first weight matrix of the target model and the input matrix according to the data processing mode to obtain a model reasoning result, and the first weight matrix is obtained by performing sparsification processing on a second weight matrix obtained by training the target model based on a preset sparsity.

10. The apparatus of claim 9, wherein the means for determining comprises:

the first determining unit is used for determining that the data processing mode is a first processing mode under the condition that the data volume of the input picture data is larger than a preset threshold value, wherein the first processing mode is used for representing that the processing priority of column data of the input matrix is larger than the processing priority of row data;

and the second determining unit is used for determining that the data processing mode is a second processing mode under the condition that the size of the input picture data is smaller than or equal to the preset threshold, wherein the second processing mode is used for representing that the processing priority of the row data of the input matrix is larger than the processing priority of the column data.

11. The apparatus according to claim 10, wherein the multiplication processing module is specifically configured to:

under the condition that the data processing mode is the first processing mode, acquiring target column data in the input matrix, and performing vector multiplication processing on each row of data in the first weight matrix and the target column data in the input matrix respectively to obtain a model reasoning result, wherein the target column data is at least part of column data in the input matrix;

and under the condition that the data processing mode is the second processing mode, acquiring all column data in the input matrix aiming at target row data in the first weight matrix, and performing vector multiplication processing on the target row data in the first weight matrix and each column data in the first weight matrix respectively to obtain a model reasoning result, wherein the target row data is any row data in the first weight matrix.

12. The apparatus according to claim 9, wherein the first weight matrix is characterized by first information, the first information includes position information of effective weights in the first weight matrix, weight data, and number information of effective weights per row, and the multiplication processing module is specifically configured to:

for each row in the first weight matrix, acquiring weight data and position information of the effective weights of the row in the first weight matrix based on the quantity information of the effective weights of the row in the first weight matrix;

acquiring data of each column in the input matrix corresponding to the position information based on the position information of the effective weight of the row;

and carrying out vector multiplication on the weight data of the effective weight of the row in the first weight matrix and the data of each column corresponding to the position information in the input matrix to obtain a model reasoning result.

13. The apparatus of claim 9, further comprising:

the block processing module is used for carrying out block processing on the input matrix aiming at the row data based on the size of a data cache unit and the size of a vector processing unit to obtain a plurality of data processing blocks;

and the multiplication processing module is specifically configured to perform vector multiplication processing on each row of data in the first weight matrix of the target model and the column data constructed by the plurality of data processing blocks according to the data processing mode to obtain a model inference result.

14. The apparatus of claim 9, further comprising:

an obtaining module, configured to obtain second information obtained after the target model is trained, where the second information includes the second weight matrix and an importance factor of each effective weight in the second weight matrix;

and the sparsification processing module is used for sparsifying the second weight matrix based on a preset sparsity according to the importance factor of each effective weight in the second weight matrix to obtain a first weight matrix of the target model.

15. The apparatus of claim 14, wherein the first weight matrix is characterized by first information, the apparatus further comprising:

the first storage module is used for storing the weight value of each effective weight in the first weight matrix to obtain weight data;

the second storage module is used for storing the position difference between every two adjacent effective weights determined according to a preset mode in the first weight matrix to obtain position information;

the third storage module is used for storing the effective weight quantity corresponding to each channel in the first weight matrix to obtain quantity information;

the first information includes the weight data, location information, and the quantity information.

16. The apparatus according to claim 14, wherein the sparsification processing module is specifically configured to:

based on a preset sparsity, performing sparsification processing on the second weight matrix according to the importance factor of each effective weight in the second weight matrix to obtain a third weight matrix;

and quantizing the third weight matrix to obtain the first weight matrix.

17. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-8.

19. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-8.

Images