CN114861907A - Data computing method, device, storage medium and device - Google Patents

Abstract

The application relates to the field of data calculation, and provides a data calculation method, a data calculation device, a storage medium and data calculation equipment. The method comprises the following steps: acquiring a target activation value and a target weight of a floating point type to be involved in a target matrix multiplication operation; respectively carrying out quantization processing on the target activation value and the target weight to obtain a quantization activation value of a fixed point type corresponding to the target activation value and a quantization weight of a fixed point type corresponding to the target weight; performing the target matrix multiplication operation by using the quantization activation value and the quantization weight; and performing inverse quantization processing on the result of the target matrix multiplication operation according to the target activation value or the quantization mode of the target weight to obtain target output data of a floating point type. By quantizing the limited number of target activation values and target weights meeting the conditions, the method and the device reduce the resources and time consumed during calculation, save the video memory, and do not reduce the calculation precision.

Description

Data computing method, device, storage medium and device

technical field

The present application relates to the field of data computing, and more particularly to a data computing method, apparatus, storage medium and device.

Background technique

In some data calculation scenarios, the data is usually input into the model corresponding to the scenario, and the result corresponding to the data is obtained through the calculation of each layer in the model. Among them, the activation values involved in the calculation of each layer in the model are usually floating-point data.

For example, when generating images or texts through a generative model, the target text is obtained by inputting the generative target and latent variables into the model. The activation value involved in the calculation of each layer in the model is usually the corresponding 32-bit floating-point type data, and the target text is obtained after the operation is performed through the model. Among them, the activation value can refer to the input data or output data of each layer in the model.

Existing data calculation methods often quantify all floating-point type data involved in the calculation into fixed-point type data to speed up calculation and improve efficiency. However, calculating after quantizing floating-point type data into fixed-point type data sacrifices calculation accuracy. If all floating-point type data are quantized into fixed-point type data for calculation, the precision of the final result will have a large deviation .

SUMMARY OF THE INVENTION

In this context, the embodiments of the present application are expected to provide a data calculation method, device, storage medium and device, which quantify a limited number of target activation values and target weights to be involved in a matrix multiplication operation, that is, a limited number of floating Instead of converting all floating-point type data to fixed-point type data, point type data is converted to fixed-point type data, which improves calculation efficiency and ensures calculation accuracy.

In a first aspect of the present application, a data calculation method is provided, comprising:

Obtaining a target activation value and a target weight, wherein the target activation value and the target weight are to be subjected to a target matrix multiplication operation and both are floating-point data within a first preset threshold range;

Perform quantization processing on the target activation value and the target weight, respectively, to obtain a quantized activation value corresponding to the target activation value and a quantized weight corresponding to the target weight, wherein the quantized activation value and the quantized activation value are obtained. The weights are all fixed-point type data within the second preset threshold range;

Using the quantized activation value and the quantized weight to perform the target matrix multiplication operation;

Perform inverse quantization processing on the result of the multiplication operation of the target matrix according to the quantization method of the target activation value or the target weight to obtain target output data, where the target output data is a floating point type within a first preset threshold range data.

In an embodiment of the present application, it is applied to a neural network model;

Wherein, the neural network model includes at least one multiplication operation of the target matrix;

The target matrix multiplication operation is one of the following matrix multiplications:

QKV matrix multiplication of attention computation layer;

Mapping matrix multiplication of the mapping layer;

The first fully connected matrix multiplication of the feedforward neural network layer;

A second fully connected matrix multiplication of the feedforward neural network layer.

In an embodiment of the present application, in the attention calculation layer, the target activation value is the input data of the attention calculation layer, and the target weight includes the query weight of the attention calculation layer, the key word weight and value weight;

Quantizing the target weight, including:

Perform quantization processing according to the respective channel dimensions of the query weight, the keyword weight, and the value weight, to obtain a quantized query weight corresponding to the query weight, a quantized keyword weight corresponding to the keyword weight, and a quantized value weight corresponding to the value weight;

The performing the target matrix multiplication operation using the quantized activation value and the quantized weight includes:

Use the quantized activation value and the quantized query weight, the quantized keyword weight, and the quantized value weight to perform matrix multiplication operations respectively;

Perform inverse quantization processing on the result of the multiplication operation of the target matrix according to the quantization method of the target activation value or the target weight to obtain target output data, including:

The results of the three matrix multiplication operations are respectively subjected to inverse quantization processing according to the quantization method of the target activation value or the target weight;

The attention is calculated according to a preset rule by using three inverse-quantized matrix multiplication results as output data of the attention calculation layer.

In an embodiment of the present application, when the attention calculation layer is a masked multi-head attention calculation layer, the channel dimension of the weight is the attention head dimension of the weight.

In an embodiment of the present application, quantization processing or inverse quantization processing is performed through a preset fusion operator;

The fusion operator includes a quantization fusion operator and an inverse quantization fusion operator;

The quantization fusion operator is used to perform fusion calculation on the data calculation before the quantization processing of the target activation value and the quantization processing of the target activation value;

The inverse quantization fusion operator is used to perform fusion calculation on the data after inverse quantization processing and the inverse quantization processing.

In an embodiment of the present application, each target matrix multiplication corresponds to a quantization fusion operator and/or an inverse quantization fusion operator;

Wherein, the quantization fusion operator corresponding to the QKV matrix multiplication is used to perform fusion calculation on the normalization processing and the quantization processing of the activation value after the normalization processing; the inverse quantization fusion operator corresponding to the QKV matrix multiplication For the inverse quantization process of the operation result of the QKV matrix multiplication and the addition of the offset term for fusion calculation;

The quantization fusion operator corresponding to the mapping matrix multiplication is used to perform fusion calculation on the permutation conversion processing and the quantization processing of the activation value after the permutation conversion processing; the inverse quantization fusion operator corresponding to the mapping matrix multiplication is used to combine the The inverse quantization processing of the operation result of the mapping matrix multiplication, the addition of the offset terms, and the addition of the residuals are used for fusion calculation;

The quantization fusion operator corresponding to the first fully connected matrix multiplication is used to perform fusion calculation on the normalization processing and the quantization processing of the activation value after the normalization processing; the inverse corresponding to the first fully connected matrix multiplication The quantization fusion operator is used to perform fusion calculation by inverse quantization processing of the operation result of the first fully connected matrix multiplication, addition of offset terms, and activation operation;

The inverse quantization fusion operator corresponding to the second fully connected matrix multiplication is used for inverse quantization processing of the operation result of the second fully connected matrix multiplication, addition of offset items, and normalization processing to perform fusion calculation.

In an embodiment of the present application, if the neural network model is in a parallel training state, the target activation value and the target weight are respectively quantized according to the parallel training mode of the neural network model;

When the parallel training mode of the neural network model is data parallelism, the quantization of the channel dimension is performed on the target weights that meet the conditions, and the tensor dimension is performed on the target activation values and the target weights that do not meet the conditions. quantification processing;

When the parallel training mode of the neural network model is model parallelism, the target activation value and the target weight are respectively subjected to quantization processing of the data block dimension, wherein the target activation value and the data block of the target weight are Dimensions are quantified differently.

In an embodiment of the present application, performing the quantization processing of the channel dimension on the target weights that meet the conditions includes:

Obtain the attention head dimension of the target weight;

The target weight is divided into data blocks according to the attention head dimension to obtain each target weight sub-block;

Each target weight sub-block is quantized separately.

In an embodiment of the present application, the quantization processing of the data block dimension is performed on the target activation value and the target weight respectively, including:

obtaining the parallel scale of the model and the attention head dimension of the target weight;

The target activation value is divided into data blocks according to the scale of the model parallelism to obtain each target activation value sub-block; and the target weight is divided according to the scale of the model parallelism and the attention head dimension according to the data block. , get each target weight sub-block;

Perform quantization processing on each of the target activation value sub-blocks and each of the target weight sub-blocks respectively.

In an embodiment of the present application, performing data partitioning on the target weight according to the parallel scale of the model and the attention head dimension includes:

The product of the attention head dimension of the target weight and the parallel scale of the model is used as a divisor for data partitioning.

In a second aspect of the present application, a data computing device is provided, comprising:

an acquisition module, configured to acquire a target activation value and a target weight, wherein the target activation value and the target weight are to be subjected to a target matrix multiplication operation and both are floating-point data within a first preset threshold range;

a quantization module, configured to perform quantization processing on the target activation value and the target weight, respectively, to obtain a quantized activation value corresponding to the target activation value and a quantized weight corresponding to the target weight, wherein the quantization The activation value and the quantization weight are both fixed-point type data within the second preset threshold range;

a calculation module configured to perform the target matrix multiplication operation using the quantized activation value and the quantized weight;

an inverse quantization module, configured to perform inverse quantization processing on the result of the multiplication operation of the target matrix according to the quantization method of the target activation value or the target weight to obtain target output data, and the target output data is a first preset Floating point data within the threshold range.

In a third aspect of the present application there is provided a computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the method as described in the first aspect.

In a fourth aspect of the present application, there is provided a computing device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the computer program when executed The method described in the first aspect.

Compared with the prior art, according to the data calculation method, device, storage medium and device of the embodiments of the present application, a limited number of target activation values and target weights to be involved in the matrix multiplication operation are quantized, that is, a limited number of floating-point Converting type data to fixed-point type data, instead of converting all floating-point type data to fixed-point type data, reduces the resources and time consumed during calculation, saves video memory, and does not reduce calculation accuracy, bringing a better experience to users .

Description of drawings

The above and other objects, features and advantages of exemplary embodiments of the present application will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present application are shown by way of example and not limitation, wherein:

FIG. 1 is a schematic diagram of an application scenario of a data calculation method according to some embodiments of the present application;

2 is a schematic flowchart of a data calculation method according to an embodiment of the application;

Fig. 3 is a GPT model quantization calculation diagram of an embodiment of the application;

Fig. 4 is a GPT model inverse quantization calculation diagram of an embodiment of the application;

5 is a schematic diagram of quantization cutting during parallel training of models according to another embodiment of the present application;

6 is a schematic structural diagram of a data computing device according to an embodiment of the present application;

7 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present application;

FIG. 8 is a schematic structural diagram of a computing device according to an embodiment of the present application.

In the drawings, the same or corresponding reference numerals denote the same or corresponding parts.

Detailed ways

The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and implement the present application, but do not limit the scope of the present application in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Those skilled in the art know that the embodiments of the present application can be implemented as a system, apparatus, device, method or computer program product. Accordingly, the present disclosure may be embodied in entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

It should be noted that terms involved in various embodiments or drawings of the present application:

float: float type data;

FP16/FP32: 16-bit/32-bit floating point data;

Per-tensor: each tensor;

Per-channel: each channel;

Per-block: Every data block.

At present, the data calculation method for a model constructed based on a neural network usually needs to perform calculation on 32-bit data, which puts a large pressure on the bandwidth of the data calculation and reduces the calculation performance.

To this end, the embodiment of the present application provides a data calculation method, which can quantify a limited number of target activation values and target weights to be involved in the matrix multiplication operation under the premise that the impact on the data calculation result is small. Processing, that is, converting a limited amount of floating-point type data to fixed-point type data, instead of converting all floating-point type data to fixed-point type data, reduces the bandwidth pressure of data computing, improves the computing power of data computing devices, and guarantees The precision of the data calculation results.

The data calculation method provided in the embodiments of the present application can be applied to a neural network model based on artificial intelligence. Artificial intelligence (AI) is to use a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, Theories, methods, techniques, and applied systems for perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning/deep learning.

In the embodiments of the present application, the artificial intelligence software technologies mainly involved include the above-mentioned natural language processing technologies and deep learning and other directions.

For example, it may involve deep learning (Deep Learning) in machine learning (Machine learning, ML), including various artificial neural networks (artificial neural networks).

First, the executive body of the embodiment of the present application is introduced. The data computing method provided in this application can be executed by a data computing device. A neural network model applicable to the data computing method may be deployed in the data computing device, and the data computing device may be a server, where the server may be an independent physical server, or a server cluster or distributed server composed of multiple physical servers It can also provide basic cloud services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Cloud servers for computing services. The data computing device may be a server, and the terminal may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected through wired or wireless communication, which is not limited in this application.

The data computing device may have the capability of implementing automatic sentence generation and translation techniques in natural language processing techniques, and the like.

The data computing device may have machine learning (Machine Learning, ML) capabilities. ML is a multi-field interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or realize human learning behaviors to acquire new knowledge or skills, and to reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications are in all fields of artificial intelligence. Machine learning and deep learning often include techniques such as artificial neural networks.

In the embodiments of the present application, the neural network model applying the above data calculation method mainly involves the application of various artificial neural networks, such as sequence generation through the neural network model.

It should be noted that this embodiment of the present application does not limit the type of model for data calculation by this method, and the model may be any type of model. In a possible implementation manner, the model may be a Recurrent Neural Network (RNN) Model.

Next, the server is used as the execution body, and the data calculation method provided by the embodiment of the present application is introduced in combination with the actual application scenario.

Referring to FIG. 1 , the figure shows a schematic diagram of an application scenario of a data calculation method provided by an embodiment of the present application. As shown in FIG. 1 , the application scenario includes a server 101, and the server 101 executes the data calculation method provided by the embodiment of the present application.

In this embodiment of the present application, when data calculation is required for input data, the server 101 may input the input data into a model for data calculation, so as to determine output data corresponding to the input data through the model.

The application scenarios of this application include speech analysis, speech noise reduction, speech translation, text translation, text recognition, sequences and other scenarios.

In the case of a speech analysis scenario, the model may be a speech analysis model, the input data may be speech data to be subjected to speech analysis, and the determined output data corresponding to the input data may be data for which speech analysis has been completed. When it is a speech noise reduction scene, the model may be a speech noise reduction model, the input data may be the speech data to be denoised, and the output data corresponding to the determined input data may be the speech after noise reduction of the speech data to be denoised data.

When it is a sentence translation scenario, the model may be a sentence translation model, then the input data may correspond to the sentence data of the first language to be translated, and the output data corresponding to the determined input data may be the translation of the second language. statement data. When applied to a sequence generation scenario, the model can be a sequence generation model, the input data can be data including the sequence to be generated, the output data corresponding to the determined input data is a sequence obtained by performing data calculation according to the sequence to be generated, etc. ,No longer.

After inputting the input data to the model, the model processes the input data to obtain the activation value flowing in each layer. The activation value can be the activation value input to a neural network layer or the activation value output from a neural network layer, and the weight The value can be a weight value inherent to a neural network layer. In the process of activation value flow, the target activation value and target weight can be obtained, quantized, and then the quantized target activation value and target weight can continue to perform the matrix multiplication operation to be involved.

The principles and spirit of the present application are explained in detail below with reference to several representative embodiments of the present application.

For example, referring to FIG. 1 , the server 101 may take the input activation value of the linear overlay in the model as the target activation value, and obtain the target activation value, which is a one-dimensional sequence [-0.127, 0.126, 0.08, -0.07].

After acquiring the target activation value, the server 101 may quantify the target activation value, and determine the quantized activation value corresponding to each target activation value, which is recorded as the quantized activation value. The quantized activation value after quantization is fixed-point type data. Specifically, in this example, the target activation value is quantized as an integer value.

In this example, the target activation value can be quantified into the data range [-127, 127] by scaling up by a factor of 1000. Thus, the determined quantization activation value is: "-127, 126, 8, -7", and a one-dimensional sequence [-127, 126, 8, -7] is obtained.

From this, the quantized activation values and the quantized weights are calculated forward.

After the calculation is completed, the result of the forward calculation can be inversely quantized according to the quantization method of the target activation value or the target weight, and the corresponding output result can be obtained, so as to reduce the activation value and the weight after the quantization of the forward calculation. The effect of the calculated output. That is to say, the difference between the result obtained by calculating the activation value and the target weight and the result obtained by calculating the quantized activation value and the quantized weight is within an allowable range.

In this embodiment, the result of forward calculation can be scaled by 1000 times according to the quantization method to realize inverse quantization. As shown in Figure 1, after the forward calculation is performed according to the quantization activation value and the quantization weight, the one-dimensional sequence is obtained as [102, 103, -76, 8]. The data in the sequence can be inversely quantized and scaled by 1000 times to obtain The sequence [0.102, 0.103, -0.76, 0.08], which is recorded as the corresponding output result.

After the calculation of the input data is completed through the model, the output data corresponding to the input data is determined.

In this example, under the premise of not affecting the data calculation results as much as possible, by quantizing the target activation value and target weight into integer values with fewer digits, the bandwidth pressure of data calculation is reduced and the calculation of data calculation equipment is improved. ability.

It should be noted that, although the above examples only specifically describe how to quantize the target activation value and inversely quantify the output data, in some embodiments of the present application, the target weight is also quantized, and the quantization weight and quantization are also quantized. The activation value is processed according to the original calculation method of activation value and weight; for example, inputting a target activation value will perform matrix multiplication calculation with the target weight, then when the data calculation method of this application is used to calculate the data, it will be The target activation value and the target weight are respectively quantized, then the quantized activation value and the quantized weight are subjected to matrix multiplication calculation, and the calculation result is used as the corresponding output data, or after the calculation result is inversely quantized, as corresponding output data.

It can be understood that the quantization method of the corresponding weight is similar to the quantization method of the corresponding activation value, that is, a quantization coefficient is used to process the corresponding weight to obtain the quantized corresponding weight.

Various technical solutions of the present application will be described below with reference to several specific embodiments.

The following describes a method for data calculation according to an exemplary embodiment of the present application with reference to FIG. 2 in conjunction with the application scenario of FIG. 1 . It should be noted that the above application scenarios are only shown for the convenience of understanding the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect. On the contrary, the embodiments of the present application can be applied to any applicable scenarios.

Next, the data computing method provided by the embodiment of the present application will be introduced by taking the server as the above-mentioned data computing device and taking a scenario generated by natural language as an example. Wherein, the above-mentioned model is deployed in the server, and the model may be a model obtained by completing training, and the data calculation method is an inference process for the model.

Referring to Figure 2, the data calculation method includes:

Step S110, obtaining a target activation value and a target weight, wherein the target activation value and the target weight are to be multiplied by a target matrix and both are floating-point data within a first preset threshold range;

The data calculation method provided by the embodiments of the present application quantifies each target activation value and target weight in the neural network model, and the quantization of each target activation value and target weight can be divided into multiple groups, a group of target activation values and target weights Weights can include multiple target activations and target weights.

It should be noted that a set of target activation values and target weights should participate in the same matrix multiplication operation. That is, this embodiment determines whether the activation value is the target activation value according to whether the activation value is to be subjected to matrix multiplication operation with the weight. Similarly, whether the weight is to be subjected to matrix multiplication operation with the activation value determines whether the weight is the target weight. ; In addition, if the activation value and the weight are to be subjected to a matrix multiplication operation, the activation value and the weight are determined as a set of target activation values and target weights. For example, if activation value 1 is to perform matrix multiplication operation 1 with weight 1, then activation value 1 and weight 1 are a set of target activation values and target weights.

It can be understood that, since the natural activation values and weights in the neural network model are floating-point data, the present embodiment does not have any effect on whether the data types of the activation values and weights in the neural network model are floating-point data. make judgments. That is, the activation values and weights in the neural network model are automatically considered to be floating-point data. Those skilled in the art may further determine whether the activation value and weight to be involved in the matrix multiplication operation are floating-point data according to the actual application scenario, which is not limited in this embodiment.

The floating-point data is a real number used to indicate a decimal, and the first preset threshold range may be a threshold range of floating-point data recognized in the prior art, for example, may be 1.8E-308˜1.8E+308.

It should be noted that data of floating-point type can express more information, that is, the result of calculation based on data of floating-point type will be more accurate, however, because data of floating-point type will take more time to calculate and computing resources, thus requiring quantization of floating-point data. Considering that if all floating-point data in the neural network model are quantized, the accuracy of the final output result of the model will be greatly lost. Therefore, in the embodiment, the target activation value and target weight that meet certain conditions are obtained. Quantization processing.

The reason is that the matrix multiplication calculation of floating-point data will take extremely high computing resources and time, that is, it has a very high computational complexity, and compared with matrix multiplication calculations, other multiplication and addition calculations cost less computational resources . Therefore, only quantizing the weights and activation values that need to be calculated by matrix multiplication can achieve better resource saving effect, that is, the model calculation speed and cost resources after quantization are greatly reduced, and the loss of accuracy is also small.

Based on the above principles, the inventor found that the neural network model includes at least one matrix multiplication operation that requires the participation of weights and activation values. Thus, in one embodiment, the target weight and the target activation value can be determined based on the matrix multiplication operation, that is, the matrix multiplication operation involving the activation value and the weight is determined as the target matrix multiplication operation; specifically, the target matrix multiplication operation The operation is one of the following matrix multiplications:

QKV matrix multiplication of attention computation layer;

Mapping matrix multiplication of the mapping layer;

The first fully connected matrix multiplication of the feedforward neural network layer;

A second fully connected matrix multiplication of the feedforward neural network layer.

In one embodiment, the generative unsupervised pre-training model (GPT) includes all the target matrix multiplication operations listed in the foregoing embodiment.

It should be noted that the GPT model includes multiple stacked decoder-encoders (transformer-decoder), each transformer-decoder has the same structure, such as including mask multi-head attention layer, mapping layer, feedforward neural network layer, residual connection layer and normalization layer, thus, each transformer-decoder includes each target matrix multiplication operation listed in the above embodiment.

Step S120, quantizing the target activation value and the target weight, respectively, to obtain a quantization activation value corresponding to the target activation value and a quantization weight corresponding to the target weight, wherein the quantization activation value and The quantization weights are all fixed-point type data within the second preset threshold range;

As described in the application scenario section, quantizing floating-point data means converting floating-point data to fixed-point data. The conversion formula between floating-point data and fixed-point data is as follows:

x_out=(clamp(round(x/scale+zero_point),quant_min,quant_max)-zero_point)*scale, where x_out represents the fixed-point type data obtained after quantization processing, x is the floating-point type data before quantization processing, quant_min , quant_max respectively represent the maximum and minimum values that can be represented by fixed-point type data under a specific bit width, and scale is a scaling factor (for example, an example of an application scenario that expands by 1000 times).

In the quantization process, the most important thing is to calculate the scale coefficient. The calculation method of the Scale coefficient is the ratio of the statistical range of the tensor to the fixed-point range. Therefore, how to count the range of the tensor becomes the key to the quantification accuracy. It is necessary to take care of as many numbers as possible from the range, and from the precision Distinguish numbers that are as close to each other as possible. Commonly used methods for calculating scale include methods such as max and percentage.

In addition, the quantization method can be divided into symmetric quantization and asymmetric quantization according to whether zero_point is 0 or not. Zero_point refers to the value that the 0 of the floating-point number is mapped to the fixed-point number. If the 0 of the floating-point number is mapped to the 0 of the fixed-point number, it is symmetric quantization, otherwise it is asymmetric quantization. In deep learning, since activation values and weights basically conform to a normal distribution with a mean value of 0, the embodiment of the present application adopts symmetric quantization to achieve the purpose of saving computation.

The fixed-point type data is data in which the decimal point is implied in a certain fixed position, and the second preset threshold range may be the threshold range of the fixed-point type data recognized by the prior art, for example, it may also be 1.8E-308～ 1.8E+308.

In order to describe in more detail which activation values and weights need to be quantified in the neural network model, the following will refer to Figure 3 to introduce how to quantify each target activation value and target weight in the GPT model as an example. Among them, float input is the input of floating-point data: that is, the output activation value of the embedding layer. Since the embedding layer does not involve the matrix multiplication operation of the weight and the activation value, even if the quantization process is performed, the performance cannot be improved, and the memory usage is relatively high. At least, this embodiment does not perform quantization processing on the activation value and weight of the embedding layer, so as to ensure the accuracy of data calculation performed by the model.

In the attention calculation layer, the target activation value is input data of the attention calculation layer, and the target weight includes query weight, keyword weight and value weight of the attention calculation layer;

Quantizing the target weight, including:

Perform quantization processing according to the respective channel dimensions of the query weight, the keyword weight, and the value weight, to obtain a quantized query weight corresponding to the query weight, a quantized keyword weight corresponding to the keyword weight, and a quantized value weight corresponding to the value weight;

The performing the target matrix multiplication operation using the quantized activation value and the quantized weight includes:

Use the quantized activation value and the quantized query weight, the quantized keyword weight, and the quantized value weight to perform matrix multiplication operations respectively;

Perform inverse quantization processing on the result of the multiplication operation of the target matrix according to the quantization method of the target activation value or the target weight to obtain target output data, including:

The results of the three matrix multiplication operations are respectively subjected to inverse quantization processing according to the quantization method of the target activation value or the target weight;

The attention is calculated according to a preset rule by using three inverse-quantized matrix multiplication results as output data of the attention calculation layer.

Referring to FIG. 3 , the query-scale corresponds to the query quantization coefficient in the transformer-decoder structure. The activation value query of the input attention calculation layer will perform matrix multiplication operations with the query weight Q, keyword weight K, and value weight V of the attention calculation layer respectively (ie, the QKV multiplication of the attention calculation layer, shown in Figure 3). The Gemm shown stands for matrix multiplication). It can be seen from FIG. 3 that the inputs of the above-mentioned three matrix multiplications are substantially the same activation value. Therefore, in this embodiment, the quantization processing operations of the respective inputs of the above-mentioned three matrix multiplications are combined into one, thereby reducing the number of quantization steps. Guaranteed quantization accuracy.

In addition, in some embodiments, the attention computation layer may be a masked multi-head attention computation layer, that is, the attention computation layer includes multiple attention heads. Therefore, in order to improve the granularity of quantization and reduce the quantization error, in one embodiment, if the dimension of the target weight is (head_num, size_per_head), the head dimension, that is, size_per_head, is regarded as the channel dimension, and the target weight is determined according to the channel dimension. The quantization processing is performed separately, that is, the target weight is first divided into each target weight sub-block according to the attention head dimension, and then the quantization processing is performed separately.

Since the two matrix multiplications of Q*K and attention attn*v have no weight, they are all multiplication of activation values, which cannot reduce the video memory occupation. Therefore, this embodiment does not perform quantization processing on the data participating in the two matrix multiplications.

Continuing to refer to FIG. 3, consider that Out_scale/fc1_scale and fc2_scale correspond to the scaling factor of the project matrix multiplication of the deocder structure (that is, the mapping matrix multiplication of the mapping layer), the fc1 matrix multiplication (that is, the first of the feedforward neural network layer). The scaling factor of the fully connected matrix multiplication) and the scaling factor of the fc2 matrix multiplication (that is, the second fully connected matrix multiplication of the feedforward neural network layer), this embodiment performs normal quantization operations to save computing resources and time.

After introducing how to determine the target activation value and target weight, and how to perform quantization processing on each target activation value and target weight, step S130 is executed next, and the target matrix multiplication is performed by using the quantized activation value and the quantized weight. operation.

In order to further improve the calculation efficiency, in an embodiment of the present application, quantization processing is performed by a preset fusion operator; the fusion operator includes a quantization fusion operator;

The quantization fusion operator is used to perform fusion calculation on the data calculation before the quantization processing of the target activation value and the quantization processing of the target activation value.

Specifically, each target matrix multiplication can correspond to a quantization fusion operator;

Wherein, the quantization fusion operator corresponding to the QKV matrix multiplication is used to perform fusion calculation on the normalization process and the quantization process of the activation value after the normalization process; layernorm and the input queue scaling factor query-scale (equivalent to Quant in Figure 4).

The quantization fusion operator corresponding to the described mapping matrix multiplication is used for the quantization processing of the activation value after the permutation conversion processing and the permutation conversion processing to carry out fusion calculation; Transpose permutation conversion processing and activation value Quant in the dashed-line frame in Figure 4 quantization processing.

The quantization fusion operator corresponding to the first fully connected matrix multiplication is used for normalization processing and the quantization processing of the activation value after the normalization processing to carry out fusion calculation; the first fully connected matrix multiplication FC1 in Figure 4 The layer normalization LayerNorm processing in the dashed box before Gemm and the quantization processing of the activation value Quant.

The inverse quantization fusion operator corresponding to the second fully connected matrix multiplication is used to perform fusion calculation with the activation processing and the quantization processing of the activation value after the activation processing; the second fully connected matrix in the dotted line frame in Figure 4 Activation GELU processing before multiplying fc2_scale and quantization processing of activation values according to fc2_scale.

After the target matrix multiplication operation is performed using the quantization activation value and the quantization weight, in order to ensure the accuracy of the final calculation result, step S140 also needs to be executed, according to the quantization method of the target activation value or the target weight. The result of the target matrix multiplication operation is subjected to inverse quantization processing to obtain target output data, where the target output data is floating-point data within a first preset threshold range.

The inverse quantization process is the inverse operation of the quantization process. For example, the quantization process enlarges the data by 1000 times, and the inverse quantization process reduces the data by 1000 times.

In addition, since some target weights are quantized in the channel dimension, in some embodiments, the quantization method of the target weight is not the same as the target activation value. The quantization method of the target weight performs inverse quantization processing on the result of the multiplication operation of the target matrix.

Similar to the quantization embodiment, in an embodiment of the present application, inverse quantization processing is also performed through a preset fusion operator;

The fusion operator is an inverse quantization fusion operator;

The inverse quantization fusion operator is used to perform fusion calculation on the data after inverse quantization processing and the inverse quantization processing.

In an embodiment of the present application, each target matrix multiplication also corresponds to an inverse quantization fusion operator;

Wherein, the inverse quantization fusion operator corresponding to the QKV matrix multiplication is used for the inverse quantization processing of the operation result of the QKV matrix multiplication and the addition of the offset term for fusion calculation; After the matrix multiplication operation is completed, this embodiment fuses the inverse quantization process with the addition of the bias term bias, for example, the inverse quantization deQuant in FIG. 4 and the query weight Q bias term addition Qbias.

The inverse quantization fusion operator corresponding to the mapping matrix multiplication is used for the inverse quantization processing of the operation result of the mapping matrix multiplication and the addition of offset terms and the addition of residuals for fusion calculation; specifically, for example, in FIG. 4 The inverse quantization deQuant, the mapping bias term addition Proj Bias and the Add input residual addition.

The inverse quantization fusion operator corresponding to the first fully connected matrix multiplication is used to perform fusion calculation on the inverse quantization processing of the operation result of the first fully connected matrix multiplication, the addition of offset terms, and the activation operation; specifically, For example, the inverse quantization deQuant in Figure 4, the addition of the bias term FC1 Bias&act, and the GELU activation operation (shown in Figure 3).

The inverse quantization fusion operator corresponding to the second fully connected matrix multiplication is used for the inverse quantization processing of the operation result of the second fully connected matrix multiplication and the addition and normalization of the offset items to perform fusion calculation; Say, for example, inverse quantization deQuant, bias term addition FC2Bias&act and normalization Add&Norm in Figure 4.

In addition, the quantization fusion operator may be obtained by fusing the quantization processing with the original kernel function calculated before the quantization processing; the inverse quantization fusion operator may be the original kernel calculated after the inverse quantization processing. The function is obtained after the inverse quantization process is fused.

In the embodiments of the present application, when performing quantization inference, the processes of quantization and inverse quantization are integrated into the front and rear operators, thereby reducing access to video memory and improving performance. Since the weight and its quantization coefficient are known, this embodiment can perform quantization on the weight in advance.

Some neural network models, such as the GPT large model, have the problem that the memory of a single graphics card is insufficient due to the huge amount of parameters, and all models cannot be loaded. In order to solve the problem of insufficient video memory, the weight can be cut by tensor tensor, and the cut tensor can be stored on different graphics cards, so as to achieve the purpose of loading a larger model.

Specifically, in one embodiment, if the neural network model is in a parallel training state, the target activation value and the target weight are respectively quantized according to the parallel training mode of the neural network model;

When the parallel training mode of the neural network model is data parallelism, the quantization of the channel dimension is performed on the target weights that meet the conditions, and the tensor dimension is performed on the target activation values and the target weights that do not meet the conditions. Quantization processing. Wherein, the matching condition may be that the target weight is the weight output by multiple attention heads.

In one embodiment, the quantization processing of the channel dimension is performed on the target weights that meet the conditions, including:

Obtain the attention head dimension of the target weight;

The target weight is divided into data blocks according to the attention head dimension to obtain each target weight sub-block;

Each target weight sub-block is quantized separately.

When the model is trained in parallel, for weight quantization, the data inside the weight channel is usually allocated to different graphics cards, which makes it impossible to obtain the statistical range of the data in the channel normally, and thus cannot calculate the scale coefficient of the weight; for the activation value, Since different machines will calculate different parts of the activation value, the overall range of the activation value cannot be counted, and the activation value scale coefficient cannot be calculated.

In order to solve the problem that the activation value and the scale of the weight cannot be counted across machines due to the parallel training of the model, the embodiment of the present application proposes a method for quantizing the granularity of per-block data blocks.

When the parallel training mode of the neural network model is model parallelism, the target activation value and the target weight are respectively subjected to quantization processing of the data block dimension, wherein the target activation value and the data block of the target weight are Dimensions are quantified differently.

In one embodiment, the quantization processing of the data block dimension is respectively performed on the target activation value and the target weight, including:

obtaining the parallel scale of the model and the attention head dimension of the target weight;

The target activation value is divided into data blocks according to the scale of the model parallelism to obtain each target activation value sub-block; and the target weight is divided according to the scale of the model parallelism and the attention head dimension according to the data block. , get each target weight sub-block;

Perform quantization processing on each of the target activation value sub-blocks and each of the target weight sub-blocks respectively.

In one embodiment, performing data partitioning on the target weight according to the parallel scale of the model and the attention head dimension includes:

The product of the attention head dimension of the target weight and the parallel scale of the model is used as a divisor for data partitioning.

Per-block has a finer quantization granularity than per-channel. For the activation value, the block processing is performed according to the scale of model parallelism. Assuming that the parallel size of the model is K and the number of data of the original activation value is n, then per-tensor The quantization granularity of per-block is n, and the quantization granularity of per-block is n/K; for the weight, it is also divided into blocks according to the scale of model parallelism. The number of heads of attention of the weight is n, then the quantization granularity of per-channel is n/h, and the quantization granularity of per-block is n/(h*K).

Referring to Figure 5, in this figure, a rectangle represents the quantization process of per-tensor, (a rectangle represents a tensor), and the horizontal division into two represents the quantization process of per-channel, horizontal + vertical one point Four represents the per-block quantization process.

In Figure 5, the parameters at the beginning of quant represent the quantization of activation values, and the parameters at the end of weight represent the quantization of weights.

The choice of the three quantization processes depends on the training method. In the case of data parallelism, this embodiment adopts the quantization method of activation value per-tensor and weight per-channel. Therefore, under data parallelism, the beginning of quant is the whole rectangle, and the end of weight is divided into two parts. 's rectangle.

Under model parallelism, the first is the model parallelization process of attention. Due to the parallelism of the model, the weight of QKV performs a tensor split operation, and the split operation is not in the channel dimension, so it is expressed as vertical The two dividing lines of , become a rectangle divided into four. Quant_out is the result of calculating the attention of the output of the three-matrix multiplication of QKV. In the case of tensor-split, it is also cut vertically. Similarly, out_weight also needs to be cut vertically until the calculation of attention is completed (that is, the node represented by quant_fc1) .

Similar to the model parallel process of attention, the calculation process of the feedforward neural network layer ffn is also the same. FC1_weight and FC2_weight become a quantization process that is divided into four per-blocks under the dual effects of model parallelism and per_channel quantization. The matrix multiplication result of fc1, quant_fc2, has changed from the original per-tensor quantization to a vertical split into two. The per-block quantization of , until the end of the ffn calculation process, that is, the output result of the fc2 matrix multiplication, is restored to the per-tensor quantization process.

For example, a matrix [[1,2,3,4],[5,6,7,8]], the quantization method of per-tensor quantizes the 8 values in the matrix together; the quantization method of per-channel is Quantize [1,2,3,4] and quantize [5,6,7,8]; for per-block quantization, first divide [1,2,3,4] into two data blocks block: [1,2] and [3,4], then quantize [1,2] and quantize [3,4], and then repeat the above operations for [5,6,7,8].

According to the data calculation method of the embodiment of the present application, a limited number of target activation values and target weights to be involved in the matrix multiplication operation are quantized, that is, a limited number of floating-point type data are converted into fixed-point type data, instead of all floating-point type data. Converting type data to fixed-point type data reduces the resources and time consumed in computing, saves video memory, does not reduce computing accuracy, and brings a better experience to users. In addition, in some embodiments, quantization, inverse quantization, and front and rear operators are also combined to further improve computational efficiency. In addition, in some embodiments, the quantization of the data block granularity is performed on the model in the parallel training state, since the data quantization is performed at a finer granularity, the quantization error is effectively reduced, and the communication overhead between the machines is avoided, and the training efficiency is improved.

After the method of the exemplary embodiment of the present application is introduced, next, referring to FIG. 6 , the apparatus for data calculation according to the exemplary embodiment of the present application is described. The apparatus 60 includes:

The acquisition module 610 is configured to acquire a target activation value and a target weight, wherein the target activation value and the target weight are to be subjected to a target matrix multiplication operation and are both floating-point data within a first preset threshold range;

The quantization module 620 is configured to perform quantization processing on the target activation value and the target weight, respectively, to obtain a quantization activation value corresponding to the target activation value and a quantization weight corresponding to the target weight, wherein the Both the quantization activation value and the quantization weight are fixed-point type data within the second preset threshold range;

a calculation module 630, configured to perform the target matrix multiplication operation by using the quantized activation value and the quantized weight;

The inverse quantization module 640 is configured to perform inverse quantization processing on the result of the multiplication operation of the target matrix according to the quantization method of the target activation value or the target weight to obtain target output data, and the target output data is the first pre- Set the floating-point data within the threshold range.

In an embodiment of the present application, the device 60 is applied in a neural network model;

Wherein, the neural network model includes at least one multiplication operation of the target matrix;

The target matrix multiplication operation is one of the following matrix multiplications:

QKV matrix multiplication of attention computation layer;

Mapping matrix multiplication of the mapping layer;

The first fully connected matrix multiplication of the feedforward neural network layer;

A second fully connected matrix multiplication of the feedforward neural network layer.

In an embodiment of the present application, in the attention calculation layer, the target activation value is the input data of the attention calculation layer, and the target weight includes the query weight of the attention calculation layer, the key word weight and value weight;

The quantization module 620 is further configured to perform quantization processing according to the respective channel dimensions of the query weight, the keyword weight and the value weight, to obtain a quantified query weight corresponding to the query weight, The quantized keyword weight corresponding to the keyword weight and the quantized value weight corresponding to the value weight;

The computing module 630 is further configured to perform matrix multiplication operations with the quantized activation value and the quantized query weight, the quantized keyword weight, and the quantized value weight, respectively;

The inverse quantization module 640 is further configured to perform inverse quantization processing on the results of the three matrix multiplication operations respectively according to the quantization method of the target activation value or the target weight; and

The attention is calculated according to a preset rule by using three inverse-quantized matrix multiplication results as output data of the attention calculation layer.

In an embodiment of the present application, when the attention calculation layer is a masked multi-head attention calculation layer, the channel dimension of the weight is the attention head dimension of the weight.

In an embodiment of the present application, quantization processing or inverse quantization processing is performed through a preset fusion operator;

The fusion operator includes a quantization fusion operator and an inverse quantization fusion operator;

The quantization fusion operator is used to perform fusion calculation on the data calculation before the quantization processing of the target activation value and the quantization processing of the target activation value;

The inverse quantization fusion operator is used to perform fusion calculation on the data after inverse quantization processing and the inverse quantization processing.

In an embodiment of the present application, each target matrix multiplication corresponds to a quantization fusion operator and/or an inverse quantization fusion operator;

Wherein, the quantization fusion operator corresponding to the QKV matrix multiplication is used to perform fusion calculation on the normalization processing and the quantization processing of the activation value after the normalization processing; the inverse quantization fusion operator corresponding to the QKV matrix multiplication For the inverse quantization process of the operation result of the QKV matrix multiplication and the addition of the offset term for fusion calculation;

The quantization fusion operator corresponding to the mapping matrix multiplication is used to perform fusion calculation on the permutation conversion processing and the quantization processing of the activation value after the permutation conversion processing; the inverse quantization fusion operator corresponding to the mapping matrix multiplication is used to combine the The inverse quantization processing of the operation result of the mapping matrix multiplication, the addition of the offset terms, and the addition of the residuals are used for fusion calculation;

The quantization fusion operator corresponding to the first fully connected matrix multiplication is used to perform fusion calculation on the normalization processing and the quantization processing of the activation value after the normalization processing; the inverse corresponding to the first fully connected matrix multiplication The quantization fusion operator is used to perform fusion calculation by inverse quantization processing of the operation result of the first fully connected matrix multiplication, addition of offset terms, and activation operation;

The inverse quantization fusion operator corresponding to the second fully connected matrix multiplication is used for inverse quantization processing of the operation result of the second fully connected matrix multiplication, addition of offset items, and normalization processing to perform fusion calculation.

In an embodiment of the present application, if the neural network model is in a parallel training state, the target activation value and the target weight are respectively quantized according to the parallel training mode of the neural network model;

When the parallel training mode of the neural network model is data parallelism, the quantization of the channel dimension is performed on the target weights that meet the conditions, and the tensor dimension is performed on the target activation values and the target weights that do not meet the conditions. quantification processing;

When the parallel training mode of the neural network model is model parallelism, the target activation value and the target weight are respectively subjected to quantization processing of the data block dimension, wherein the target activation value and the data block of the target weight are Dimensions are quantified differently.

In an embodiment of the present application, the quantization module 620 is further configured to obtain the attention head dimension of the target weight; the target weight is divided into data blocks according to the attention head dimension to obtain each target weight sub-dimension. block; quantize each target weight sub-block separately.

In an embodiment of the present application, the quantization module 620 is further configured to obtain the scale of the model parallelism and the attention head dimension of the target weight; and perform the target activation value according to the scale of the model parallelism. Data is divided into blocks to obtain each target activation value sub-block; and according to the scale of the model parallel and the attention head dimension, the data is divided into blocks according to the target weight to obtain each target weight sub-block; each target is activated The value sub-block and each of the target weight sub-blocks are separately quantized.

In an embodiment of the present application, performing data partitioning on the target weight according to the parallel scale of the model and the attention head dimension includes:

The product of the attention head dimension of the target weight and the parallel scale of the model is used as a divisor for data partitioning.

According to the data computing device of the embodiment of the present application, a limited number of target activation values and target weights to be involved in the matrix multiplication operation are quantized, that is, a limited number of floating-point type data are converted into fixed-point type data, instead of all floating-point type data. Converting type data to fixed-point type data reduces the resources and time consumed in computing, saves video memory, does not reduce computing accuracy, and brings a better experience to users. In addition, in some embodiments, quantization, inverse quantization, and front and rear operators are also combined to further improve computational efficiency. In addition, in some embodiments, the quantization of the data block granularity is performed on the model in the parallel training state, since the data quantization is performed at a finer granularity, the quantization error is effectively reduced, and the communication overhead between the machines is avoided, and the training efficiency is improved.

After introducing the method and apparatus of the exemplary embodiment of the present application, next, the computer-readable storage medium of the exemplary embodiment of the present application will be described with reference to FIG. A computer program (ie a program product) is stored on the computer, and when the computer program is run by the processor, it will implement the steps described in the above method embodiments, for example, obtain the target activation value and the target weight, wherein the target The activation value and the target weight are to be multiplied by the target matrix, and both are floating-point data within the range of the first preset threshold; quantization processing is performed on the target activation value and the target weight, respectively, to obtain the target activation value and the target weight. The quantization activation value corresponding to the activation value and the quantization weight corresponding to the target weight, wherein the quantization activation value and the quantization weight are both fixed-point type data within a second preset threshold range; the quantization activation value is used Carry out the target matrix multiplication operation with the quantization weight; perform inverse quantization processing on the result of the target matrix multiplication operation according to the quantization method of the target activation value or the target weight to obtain target output data, and the target output The data is floating-point data within the range of the first preset threshold; the specific implementation of each step will not be repeated here.

It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random Access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media will not be repeated here.

After introducing the method, apparatus and storage medium of the exemplary embodiment of the present application, next, referring to FIG. 8 , the apparatus for data computing of the exemplary embodiment of the present application will be described.

8 illustrates a block diagram of an exemplary computing device 80, which may be a computer system or server, suitable for use in implementing embodiments of the present application. The computing device 80 shown in FIG. 8 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

As shown in FIG. 8, components of computing device 80 may include, but are not limited to, one or more processors or processing units 801, system memory 802, and a bus 803 connecting different system components (including system memory 802 and processing unit 801).

Computing device 80 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computing device 80, including both volatile and nonvolatile media, removable and non-removable media.

System memory 802 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 8021 and/or cache memory 8022 . Computing device 80 may further include other removable/non-removable, volatile/non-volatile computer system storage media. For example only, the ROM 8023 may be used to read and write to a non-removable, non-volatile magnetic medium (not shown in Figure 8, commonly referred to as a "hard disk drive"). Although not shown in Figure 8, a disk drive may be provided for reading and writing to removable non-volatile magnetic disks (eg "floppy disks"), as well as removable non-volatile optical disks (eg CD-ROM, DVD- ROM or other optical media) to read and write optical drives. In these cases, each drive may be connected to bus 803 through one or more data media interfaces. System memory 802 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of various embodiments of the present application.

A program/utility 8025 having a set (at least one) of program modules 8024, which may be stored, for example, in system memory 802, and such program modules 8024 include, but are not limited to: an operating system, one or more application programs, other program modules As well as program data, each or some combination of these examples may include an implementation of a network environment. Program modules 8024 generally perform the functions and/or methods of the embodiments described herein.

Computing device 80 may also communicate with one or more external devices 804 (eg, keyboards, pointing devices, displays, etc.). Such communication may take place through input/output (I/O) interface 805 . Also, the computing device 80 may communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN), and/or a public network such as the Internet) through a network adapter 806 . As shown in FIG. 8 , network adapter 806 communicates with other modules of computing device 80 (eg, processing unit 801 , etc.) through bus 803 . It should be appreciated that, although not shown in FIG. 8 , other hardware and/or software modules may be used in conjunction with computing device 80 .

The processing unit 801 executes various functional applications and data calculations by running the programs stored in the system memory 802, for example, obtains target activation values and target weights, wherein the target activation values and the target weights are to be performed on the target matrix. Multiplication operation and both are floating-point type data within the first preset threshold range; respectively perform quantization processing on the target activation value and the target weight to obtain a quantized activation value corresponding to the target activation value and a quantization activation value corresponding to the target activation value. The quantization weight corresponding to the target weight, wherein the quantization activation value and the quantization weight are both fixed-point type data within a second preset threshold range; the quantization activation value and the quantization weight are used to perform the target matrix multiplication operation; perform inverse quantization processing on the result of the multiplication operation of the target matrix according to the quantization method of the target activation value or the target weight to obtain target output data, and the target output data is a floating value within the first preset threshold range. Point type data. The specific implementation manner of each step is not repeated here.

It should be noted that although several units/modules or sub-units/modules of the data computing device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. Indeed, according to embodiments of the present application, the features and functions of two or more units/modules described above may be embodied in one unit/module. Conversely, the features and functions of one unit/module described above may be further subdivided to be embodied by multiple units/modules.

Furthermore, although the operations of the methods of the present application are depicted in the figures in a particular order, this does not require or imply that the operations must be performed in the particular order, or that all illustrated operations must be performed to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined to be performed as one step, and/or one step may be decomposed into multiple steps to be performed.

Although the spirit and principles of the present application have been described with reference to several specific embodiments, it should be understood that the present application is not limited to the specific embodiments of the application, nor does the division of aspects mean that features in these aspects cannot be combined to perform Benefit, this division is only for convenience of presentation. This application is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims (13)

1. A method of data computation, comprising:

acquiring a target activation value and a target weight, wherein the target activation value and the target weight are to be subjected to target matrix multiplication and are floating point type data within a first preset threshold range;

respectively carrying out quantization processing on the target activation value and the target weight to obtain a quantization activation value corresponding to the target activation value and a quantization weight corresponding to the target weight, wherein the quantization activation value and the quantization weight are both fixed point type data within a second preset threshold range;

performing the target matrix multiplication operation by using the quantization activation value and the quantization weight;

and performing inverse quantization processing on the result of the target matrix multiplication operation according to the quantization mode of the target activation value or the target weight to obtain target output data, wherein the target output data is floating point type data within a first preset threshold range.

2. The data computation method of claim 1, applied in a neural network model;

wherein the neural network model comprises at least one of the target matrix multiplication operations;

the target matrix multiplication operation is one of the following matrix multiplications:

QKV matrix multiplication of the attention calculation layer;

mapping matrix multiplication of a mapping layer;

a first fully-connected matrix multiplication of the feedforward neural network layer;

a second fully-connected matrix multiplication of the feedforward neural network layer.

3. The data calculation method according to claim 2, wherein in the attention calculation layer, the target activation value is input data of the attention calculation layer, and the target weight includes a query weight, a keyword weight, and a value weight of the attention calculation layer;

performing quantization processing on the target weight, including:

respectively carrying out quantization processing according to the channel dimensions of the query weight, the keyword weight and the value weight to obtain a quantization query weight corresponding to the query weight, a quantization keyword weight corresponding to the keyword weight and a quantization value weight corresponding to the value weight;

the performing the target matrix multiplication operation by using the quantization activation value and the quantization weight includes:

matrix multiplication is carried out on the quantization activation value, the quantization inquiry weight, the quantization keyword weight and the quantization value weight respectively;

performing inverse quantization processing on the result of the target matrix multiplication according to the quantization mode of the target activation value or the target weight to obtain target output data, including:

performing inverse quantization processing on the results of the three matrix multiplication operations according to the quantization modes of the target activation values or the target weights respectively;

and calculating attention according to a preset rule by adopting three matrix multiplication results subjected to inverse quantization, and taking the attention as output data of the attention calculation layer.

4. The data calculation method of claim 3, wherein when the attention calculation layer is a mask multi-head attention calculation layer, a channel dimension of a weight is a head-of-interest dimension of the weight.

5. The data calculation method according to claim 2, wherein quantization processing or inverse quantization processing is performed by a preset fusion operator;

the fusion operator comprises a quantitative fusion operator and an inverse quantitative fusion operator;

the quantization fusion operator is used for performing fusion calculation on data calculation before quantization processing of a target activation value and quantization processing of the target activation value;

and the inverse quantization fusion operator is used for performing fusion calculation on the data calculation after inverse quantization processing and the inverse quantization processing.

6. The data calculation method of claim 5, wherein each target matrix multiplication corresponds to one quantized fusion operator and/or one inverse quantized fusion operator;

the quantization fusion operator corresponding to the QKV matrix multiplication is used for performing fusion calculation on normalization processing and quantization processing of the activation value after the normalization processing; the inverse quantization fusion operator corresponding to the QKV matrix multiplication is used for performing fusion calculation by adding an offset term and inverse quantization processing of the operation result of the QKV matrix multiplication;

the quantization fusion operator corresponding to the mapping matrix multiplication is used for performing fusion calculation on permutation conversion processing and the quantization processing of the activation value after the permutation conversion processing; the inverse quantization fusion operator corresponding to the mapping matrix multiplication is used for performing fusion calculation on inverse quantization processing, bias term addition and residual addition of the operation result of the mapping matrix multiplication;

the quantization fusion operator corresponding to the first full-connection matrix multiplication is used for performing fusion calculation on normalization processing and the quantization processing of the activation value after the normalization processing; the inverse quantization fusion operator corresponding to the first full-connection matrix multiplication is used for performing fusion calculation on inverse quantization processing, bias term addition and activation operation of the operation result of the first full-connection matrix multiplication;

and the inverse quantization fusion operator corresponding to the second full-connection matrix multiplication is used for performing fusion calculation on the inverse quantization processing, the addition of the bias terms and the normalization processing of the operation result of the second full-connection matrix multiplication.

7. The data calculation method according to any one of claims 2 to 5, wherein if the neural network model is in a parallel training state, the target activation value and the target weight are respectively subjected to quantization processing according to a parallel training mode of the neural network model;

when the parallel training mode of the neural network model is data parallel, carrying out channel dimension quantization processing on the target weight meeting the condition, and carrying out tensor dimension quantization processing on the target activation value and the target weight not meeting the condition;

and when the parallel training mode of the neural network model is model parallel, respectively carrying out data block dimension quantization processing on the target activation value and the target weight, wherein the data block dimensions quantization modes of the target activation value and the target weight are different.

8. The data computing method of claim 7, wherein the performing quantization processing of channel dimensions on the target weights that meet conditions comprises:

obtaining the attention head dimension of the target weight;

partitioning the target weight into blocks according to the attention head dimension to obtain each target weight sub-block;

and respectively carrying out quantization processing on each target weight subblock.

9. The data calculation method of claim 7, wherein the performing quantization processing on the target activation values and the target weights respectively for data block dimensions comprises:

acquiring the parallel scale of the model and the attention head dimension of the target weight;

performing data partitioning on the target activation value according to the parallel scale of the model to obtain each target activation value sub-block; performing data blocking on the target weight according to the parallel scale of the model and the dimension of the attention head to obtain each target weight sub-block;

and respectively carrying out quantization processing on each target activation value sub-block and each target weight sub-block.

10. The data computing method of claim 7, wherein the data partitioning of the target weights according to the parallel scale of the model and the care-head dimension comprises:

and taking the product of the concerned head dimension of the target weight and the parallel scale of the model as a divisor for partitioning data.

11. A data computing apparatus, comprising:

the acquisition module is configured to acquire a target activation value and a target weight, wherein the target activation value and the target weight are to be subjected to target matrix multiplication and are floating point type data within a first preset threshold range;

the quantization module is configured to perform quantization processing on the target activation value and the target weight respectively to obtain a quantization activation value corresponding to the target activation value and a quantization weight corresponding to the target weight, wherein the quantization activation value and the quantization weight are both fixed point type data within a second preset threshold range;

a calculation module configured to perform the target matrix multiplication operation using the quantization activation value and the quantization weight;

and the inverse quantization module is configured to perform inverse quantization processing on the result of the target matrix multiplication according to the quantization mode of the target activation value or the target weight to obtain target output data, wherein the target output data is floating point type data within a first preset threshold range.

12. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method of any one of claims 1-10.

13. A computing device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1-10 when executing the computer program.

Images