CN112364981B - Differentiable searching method and device for mixed precision neural network - Google Patents

Abstract

The invention discloses a differentiable searching method and a differentiable searching device for a mixed precision neural network. The method comprises the following steps: acquiring an initialization hyper-network; the super network comprises a plurality of sub-networks, and each sub-network carries a super parameter; updating the hyper-parameters based on a differentiable search method to obtain a first hyper-network; hardware performance evaluation is carried out on sub-networks contained in the first super-network, and the super-parameters of the first super-network are updated according to the evaluation result to obtain a second super-network; judging whether an updating termination condition is met, and if so, determining the second hyper-network as a target neural network; otherwise, returning to and executing the operation of updating the hyper-parameters based on the differentiable search method to obtain the first hyper-network. By using the method, automatic model quantification can be performed, and a neural network can be searched and constructed for a specific hardware platform.

Description

A Differentiable Search Method and Device for Mixed Precision Neural Network

technical field

Embodiments of the present invention relate to the field of computer technology, and in particular to a differentiable search method and device for a mixed-precision neural network.

Background technique

Deep learning can automatically learn useful features, without relying on feature engineering, and has achieved results that surpass other algorithms in tasks such as image recognition, video understanding, and natural language processing. This success is largely due to the emergence of new neural network structures, such as ResNet, Inception, DenseNet, MobileNet, etc.

Neural Architecture Search (Neural Architecture Search, NAS) is a technology for automatically designing neural networks. It can automatically design high-precision and high-performance network structures based on data sets through algorithms, which can match or even exceed the level of human experts in certain tasks. , and can discover some network structures that humans have not proposed before. Compared with the traditional manual design of network structure and hyperparameters, neural architecture search can effectively reduce the design and use costs of neural networks.

However, as the complexity of actual tasks increases, it is necessary to design a larger and deeper neural network, and to deploy the model more widely on different hardware platforms. In addition, although traditional NAS methods based on reinforcement learning and evolutionary algorithms can design high-precision and high-performance networks, the search algorithm itself requires too much computing resources, which is not conducive to the large-scale promotion and application of NAS methods. .

Therefore, it is an urgent technical problem to propose a neural network architecture search method that has fast search speed, consumes less computing and memory resources, can automatically perform model quantization, and can search for specific hardware platforms.

Contents of the invention

Embodiments of the present invention provide a differentiable search method and device for a mixed-precision neural network, capable of automatic model quantization, and capable of searching and constructing a neural network for a specific hardware platform.

In the first aspect, an embodiment of the present invention provides a differentiable search method for a mixed-precision neural network, including:

Obtain an initialized supernetwork; the supernetwork includes a plurality of subnetworks, and each subnetwork carries hyperparameters;

updating the hyperparameters based on a differentiable search method to obtain a first hypernetwork;

performing hardware performance evaluation on the subnetworks included in the first supernetwork, updating hyperparameters of the first supernetwork according to the evaluation results, and obtaining a second supernetwork;

Judging whether the update termination condition is satisfied, if so, determining the second supernetwork as the target neural network; otherwise, returning to the operation of updating the hyperparameters based on the differentiable search method to obtain the first supernetwork.

In the second aspect, the embodiment of the present invention also provides a differentiable search device for a mixed-precision neural network, including:

An acquisition module, configured to acquire an initialized supernetwork; the supernetwork includes a plurality of subnetworks, and each subnetwork carries hyperparameters;

A first update module, configured to update the hyperparameters based on a differentiable search method to obtain a first hypernetwork;

The second update module is configured to evaluate the hardware performance of the sub-networks included in the first super-network, update the hyper-parameters of the first super-network according to the evaluation results, and obtain a second super-network;

The judging module is used to judge whether the update termination condition is satisfied, and if so, determine the second hypernetwork as the target neural network; otherwise, return to execute the update of the hyperparameters based on the differentiable search method to obtain the first supernetwork operation of the network.

In a third aspect, an embodiment of the present invention also provides a computer device, including:

one or more processors;

storage means for storing one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors are made to implement the differentiable search method for a mixed-precision neural network in any embodiment of the present invention.

In a fourth aspect, an embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and is characterized in that, when the program is executed by a processor, the mixed-precision neural network as provided in any embodiment of the present invention is implemented. Differentiable search methods for networks.

Embodiments of the present invention provide a differentiable search method and device for a mixed-precision neural network. First, an initial supernetwork is obtained. The supernetwork includes a plurality of subnetworks, and each subnetwork carries hyperparameters. Updating the hyperparameters to obtain a first hypernetwork, and then evaluating the hardware performance of the subnetworks included in the first hypernetwork, updating hyperparameters of the first supernetwork according to the evaluation results, and obtaining a second supernetwork network, and finally determine whether the update termination condition is met, and if so, then determine the second supernetwork as the target neural network; otherwise, return to execute the hyperparameter update based on the differentiable search method to obtain the first supernetwork operate. Using the above technical solution, automatic model quantization can be performed, and a neural network can be searched and constructed for a specific hardware platform.

Description of drawings

FIG. 1 is a schematic flowchart of a differentiable search method for a mixed-precision neural network provided in Embodiment 1 of the present invention;

FIG. 2 is a schematic flowchart of a differentiable search method for a mixed-precision neural network provided in Embodiment 2 of the present invention;

3 is a schematic structural diagram of a differentiable search device for a mixed-precision neural network provided by Embodiment 3 of the present invention;

FIG. 4 is a schematic structural diagram of a computer device provided by Embodiment 4 of the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, but not to limit the present invention. In addition, it should be noted that, for the convenience of description, only some structures related to the present invention are shown in the drawings but not all structures.

Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe various operations (or steps) as sequential processing, many of the operations may be performed in parallel, concurrently, or simultaneously. In addition, the order of operations can be rearranged. The process may be terminated when its operations are complete, but may also have additional steps not included in the figure. The processing may correspond to a method, function, procedure, subroutine, subroutine, or the like. In addition, the embodiments and the features in the embodiments of the present invention can be combined with each other under the condition of no conflict.

The term "comprising" and its variants used in the present invention are open to include, ie "including but not limited to". The term "based on" is "based at least in part on". The term "one embodiment" means "at least one embodiment."

Embodiment one

Fig. 1 is a schematic flow chart of a differentiable search method for a mixed-precision neural network provided by Embodiment 1 of the present invention. This method is applicable to the situation of searching a high-performance neural network. device, wherein the device can be implemented by software and/or hardware, and generally integrated on a computer device.

As shown in Figure 1, a differentiable search method for a mixed-precision neural network provided by Embodiment 1 of the present invention includes the following steps:

S110. Acquire an initialized supernetwork; the supernetwork includes multiple subnetworks, and each subnetwork carries hyperparameters.

In this embodiment, the initializing supernetwork may be a hypernetwork with initial value settings, for example, the initializing supernetwork may be obtained from the search space.

A sub-network can be understood as a sub-network of the aforementioned initialized super-network, a super-network can be composed of multiple sub-networks, and a sub-network can include multiple network layers.

The subnetwork carries hyperparameters, wherein the hyperparameters are continuously differentiable, and the hyperparameters of a subnetwork can be a space set composed of multiple sets of vectors. Exemplarily, the hyperparameters of a network layer in a subnetwork can be represented by Two sets of different vectors, one set of vectors can be composed of the influence factors of the convolution kernel configuration parameters of the layer network, and the other set of vectors can be formed of the influence factors of the quantized bit values of the layer network. The number of influencing factors of the two groups of vectors can be determined according to the number of convolution kernel configuration parameters and the number of quantization bit values respectively. If the number of convolution kernel configuration parameters of the layer network is three, the corresponding The influence factor of the convolution kernel configuration parameters of the layer network is also three.

Wherein, the convolution kernel configuration parameters may include one or more of the following: the number of convolution kernels in each layer, the convolution kernel height for each convolution kernel in each layer, and the convolution kernel height for each convolution kernel in each layer. The convolution kernel width, each layer is for the stride height of each convolution kernel, and each layer is for the stride width of each convolution kernel. The convolution kernel configuration parameters of each network layer can be a vector composed of any number of parameters selected from the above parameters.

Wherein, the quantization bit value may include one or more of the following: the data bit width of each layer of convolutional neural network feature map, the data bit width of each layer of convolutional neural network weight, and the data bit width of each layer of activation function width. The quantization bit value of each network layer may be a vector composed of at least one value arbitrarily selected from the above values.

The influence factor of the convolution kernel configuration parameter can represent the degree of influence of each parameter in the convolution kernel configuration parameter. For example, the greater the influence factor corresponding to a parameter, it indicates that the parameter is in all convolutions of the network layer. The greater the influence in the kernel configuration parameters; correspondingly, the influence factor of the quantization bit value can represent the degree of influence on the quantization bit value. For example, the greater the influence factor corresponding to a certain value in the quantization bit value, it indicates The value is more influential among all quantized bit values of this network layer.

S120. Update hyperparameters based on a differentiable search method to obtain a first hypernetwork.

In this embodiment, the differentiable search method may be a neural network search method, which may be used to receive the accuracy feedback of the neural network. Updating hyperparameters based on a differentiable search method to obtain the first supernetwork can be understood as updating hyperparameters carried by several subnetworks sampled from the supernetwork based on a differentiable search method.

Among them, sampling several subnetworks from the supernetwork may include sampling using gumbel-softmax. Exemplarily, hyperparameters are converted into probability vectors through gumbel-softmax, and multiple subnetworks are sampled from the supernetwork according to the size of the probability vector , the probability vector can be understood as a vector composed of the product of the influence factor of the convolution kernel configuration parameters of each layer of the network and the influence factor of the quantization bit value. The greater the probability.

Based on the differentiable search method to update the hyperparameters, obtaining the first hypernetwork can be understood as: first keep the hyperparameters carried by the subnetwork unchanged, train the above subnetwork on the training set, and update the weight parameters of the subnetwork, Keeping the weight parameters unchanged, the trained sub-network is propagated forward on the verification set to obtain the value of the network target loss function, and the hyperparameters are derived according to the value of the network target loss function to obtain the gradient value, and according to the gradient The value updates the hyperparameters of the subnetwork.

Among them, the training set is used to fit the model, and the classification model is trained by setting the parameters of the classifier. When combined with the verification set later, different values of the same parameter will be selected to fit multiple classifiers.

Among them, the function of the verification set is to use each model to verify on the verification set data in order to update the hyperparameters after training multiple models through the training set, and update the hyperparameters based on the loss on the verification set.

The first hypernetwork may be a new hypernetwork composed of updated hyperparameters after hyperparameters are updated through a differentiable search method.

S130. Evaluate the hardware performance of the subnetworks included in the first supernetwork, and update hyperparameters of the first supernetwork according to the evaluation results to obtain a second supernetwork.

In this embodiment, before hardware performance evaluation, the first supernetwork needs to be sampled to obtain several second sampled subnetworks, wherein the sampling method is the same as that in step S120.

This step can be realized by an evolutionary algorithm. The evolutionary algorithm in this embodiment can be used to receive and process non-differentiable hardware feedback. Based on this, the hyperparameters of the first hypernetwork can be updated. The hardware feedback can be understood as the evaluation of the hardware evaluation model. Feedback of hardware performance indicators for each layer of the network in the sub-network.

Wherein, the hardware evaluation model can be used to evaluate the hardware performance of the network. Exemplarily, the hardware performance evaluation model can evaluate the hardware performance of the second sampling sub-network in the first supernetwork. The hardware performance evaluation can be understood as evaluating the hardware performance index of the target hardware.

The hardware evaluation model can be used to evaluate the hardware performance index of the neural network model on the target hardware. Specifically, the hardware evaluation model can be used to evaluate the hardware performance index of the second sampling sub-network model on the target hardware. The hardware performance index can include the following One or more of, for example, power consumption, latency, and model size.

Among them, the evaluation methods can include the following two methods:

Method 1: Deploy several second sampling sub-network models to the target hardware to obtain hardware performance indicators;

Method 2: Determine the hardware performance index according to the convolution kernel configuration parameters and quantization bit values of several second sampling sub-network models.

The hyperparameters of the first hypernetwork are updated according to the results of the evaluation, wherein the evaluation results can be understood as the evaluation results of the hardware performance indicators of the second sampling subnetwork, if one or more of the hardware performance indicators of the second sampling subnetwork The evaluation result of is optimal, which indicates that the convolution kernel configuration parameters and quantization bit values of the second sampling sub-network are optimal. It should be noted that the second sampling sub-network can include many network layers, and can be specific to a certain The convolution kernel configuration parameters and quantization bit values of each network layer are optimal.

After determining the optimal second sampling sub-network, the value of the influence factor corresponding to the configuration parameters of the convolution kernel of the specific network layer inside it and the quantization bit value can be increased. After the value of the influence factor is increased, the optimal The second sampling sub-network can be changed into a new sub-network, and the super-network where it is located is correspondingly updated to a new super-network.

The second supernetwork may be a network after updating the influence factor of the convolution kernel configuration parameter and the influence factor of the quantization bit value of the optimal second sampling subnetwork inner network layer.

S140 , judging whether the update termination condition is satisfied, and if so, execute step 150 ; otherwise, return to execute step 120 .

In this embodiment, the update termination condition may be a condition for terminating the update of the super network, that is, a condition for the neural network architecture to stop searching. Specifically, the termination update condition can include the following two: first, the termination update condition can be when the verification set feedback and hardware feedback of the subnetwork both meet the preset indicators; second, the termination update condition can be when the supernetwork is updated When the number of times reaches the set threshold.

Among them, the verification set feedback can be understood as the feedback on the accuracy of the sub-network model in the differentiable search method, and the hardware feedback can be understood as the feedback on the results of hardware performance evaluation.

Wherein, the preset index may be a value set by the user according to the actual situation before using the method provided in this embodiment. When the hyperparameters of the updated hypernetwork meet the preset indicators, stop updating the hyperparameters of the hypernetwork, and determine the hypernetwork as the target neural network, which can be understood as a hypernetwork that meets user needs .

Wherein, the set threshold can be the number of updates of the hyper-parameters of the hypernetwork set by the user. When the number of updates reaches the set threshold, the neural network construction method provided in this embodiment can automatically stop updating the hyper-parameters, and set The hypernetwork obtained by the last update is determined as the target neural network.

If the second supernetwork does not meet the update termination condition, continue to execute steps S120 and S130, update the supernetwork accordingly, and finally determine whether the updated supernetwork meets the update termination condition. This loops until the updated supernetwork meets the update termination condition, and then stops updating the supernetwork.

Step 150, determining the second supernetwork as the target neural network.

Embodiment 1 of the present invention provides a differentiable search method for a mixed-precision neural network. First, an initial supernetwork is obtained; the supernetwork includes a plurality of subnetworks, and each subnetwork carries hyperparameters; secondly, based on the differentiable search method, the The hyperparameters are updated to obtain a first supernetwork; then, hardware performance evaluation is performed on the subnetworks included in the first supernetwork, and the hyperparameters of the first supernetwork are updated according to the evaluation results to obtain a second supernetwork ;Finally judge whether the update termination condition is satisfied, if so, determine the second supernetwork as the target neural network; otherwise, return to the operation of updating the hyperparameters based on the differentiable search method to obtain the first supernetwork . Using the above method can effectively reduce the cost of neural network design and use. Compared with the traditional neural network architecture search method, the search speed is significantly improved; in addition, compared with the full-precision neural architecture search method, it can search for lower complexity Lightweight network.

Further, the supernetwork is a convolutional neural network, and the subnetwork includes at least one layer of network; the hyperparameters include the influence factors of the convolution kernel configuration parameters of each layer of the network and the influence factors of quantized bit values, and the hyperparameters are continuously differentiable of.

Specifically, the supernetwork may be a convolutional neural network, and the subnetwork may include one or more layers of networks, and the number of network layers of the subnetwork and supernetwork is not specifically limited here.

Among them, the better the hardware feedback and verification set feedback results of the subnetwork, the better the convolution kernel configuration parameters and quantization bit values of the subnetwork, and the corresponding hyperparameters in the subnetwork can be increased to obtain a new updated super network.

It should be noted that hyperparameters can be continuously differentiable for validation set feedback. Exemplarily, the loss function of the neural network can be derived for hyperparameters, and then gradient descent can be used to iteratively optimize the hyperparameters.

Further, based on the differentiable search method to update the hyperparameters, the way to obtain the first hypernetwork can be: sampling the subnetworks included in the initial hypernetwork to obtain at least one first sampling subnetwork; The sampling sub-network is trained; the verification set is propagated forward in the first sampling sub-network after training to obtain the target loss function value; the differential calculation is performed on the target loss function value to obtain the gradient value; the hyperparameter is updated according to the gradient value, Get the first hypernet.

Wherein, after sampling the sub-networks included in the initialization super-network, one or more sampling sub-networks can be obtained according to the sampling probability, and the sampling sub-networks can be understood as the sub-networks sampled from the initialization super-network using gumbel-softmax.

Training the first sampling sub-network based on the training set may include: training one or more first sampling sub-networks several times on the training set to update the weight parameters of these first sampling sub-networks, and then maintain the weight parameters constant.

Wherein, the target loss function value may be a function that needs to be optimized eventually. The differential calculation of the target loss function value can be understood as the derivation calculation of the target loss function on the hyperparameters to obtain the gradient value of the hyperparameters. The hyperparameters can then be iteratively optimized using gradient descent.

Exemplarily, the existing network can be verified on the verification set to obtain the value of the network loss function; then this value is derived from the above hyperparameters to obtain the gradient value; finally, the hyperparameters are updated according to the gradient value .

Further, the process of training the first sampling sub-network based on the training set may be: training the first sampling sub-network based on the training set, so as to update the weight parameters of the first sampling sub-network.

Wherein, the weight parameter is updated to a new weight parameter after each training, and the first sampling sub-network is trained through the training set, and the weight parameter of the trained first sampling sub-network can be updated.

Further, performing hardware performance evaluation on the sub-networks included in the first super-network, updating the hyperparameters of the first super-network according to the evaluation results, and obtaining the second super-network, including: sampling the sub-networks included in the first super-network, Obtain at least one second sampling sub-network; perform hardware evaluation on at least one second sampling sub-network to obtain an optimal second sampling sub-network; set the convolution kernel configuration parameters of each layer network in the optimal second sampling sub-network The impact factor and the impact factor of the quantized bit value are increased to obtain the second supernetwork.

Wherein, the hardware evaluation of the second sampling sub-network may evaluate the hardware performance index of the second sampling sub-network through a hardware evaluation model, and the sub-network with the best evaluation result after evaluation is the optimal second sampling sub-network.

It should be noted that the obtained optimal second sampling subnetwork has the best hardware performance index, which indicates that the hyperparameters of the optimal second sampling subnetwork, that is, the convolution kernel configuration parameters and quantization bit values are optimal. Here, the configuration parameter of the convolution kernel and the quantization bit value may be the hyperparameters of each layer of the network in the optimal second sampling sub-network.

After the optimal convolution kernel configuration parameter and quantization bit value are determined, the influence factor of the optimal convolution kernel configuration parameter and the influence factor of the optimal quantization bit value may be increased. The increased hyperparameters cause the corresponding optimal subnetwork to be updated, and the hypernetwork of the optimal subnetwork is also updated.

Further, performing hardware evaluation on at least one second sampling subnetwork includes: deploying at least one second sampling subnetwork on target hardware to obtain hardware performance indicators; or, according to the convolution kernel of at least one second sampling subnetwork The size of configuration parameters and quantized bit values determine hardware performance indicators.

Among them, there are two ways to evaluate the hardware of the sampling sub-network. The first way may include directly deploying the sampled second sampling sub-networks on the target hardware, so as to obtain hardware performance indicators. Here It can be understood as obtaining the hardware performance index of each layer network of the plurality of second sampling sub-networks.

Wherein, the second way may include estimating the hardware performance index of each layer of the second sampling sub-network according to the network layers of the second sampling sub-network and the corresponding relationship table during the neural network architecture search. The obtained hardware performance index can be used to characterize the pros and cons of the convolution kernel configuration parameters and quantization bit values in the second sampling subnetwork.

Further, before performing hardware evaluation on at least one second sampling sub-network, it also includes: establishing a correspondence table between convolution kernel configuration parameters and quantization bit value selection of at least one second sampling sub-network and hardware performance indicators; Correspondingly, according to the convolution kernel configuration parameters of each layer of the at least one second sampling sub-network and the size selection of the quantization bit value to determine the hardware performance index, including: according to the convolution kernel configuration parameters of at least one second sampling sub-network and The size of the quantized bit value is selected to find the corresponding hardware performance index from the correspondence table.

Wherein, the correspondence table may include a plurality of convolution kernel configuration parameters of the second sampling sub-network and the corresponding relationship between the size selection of the quantization bit value and the hardware performance index, and the hardware performance index may be estimated for all the second sampling sub-networks Hardware performance metrics for all layers of the network. Exemplarily, the convolution kernel configuration parameters and quantization bit values of a certain network layer of a second sampling sub-network can have multiple combinations, and the selection of each combination can have its corresponding hardware performance index, and there is a relationship between them One-to-one correspondence relationship, the one-to-one correspondence relationship is established as a correspondence relationship table.

It should be noted that the establishment of the correspondence table is to deploy a large number of different second sampling subnetworks with known structures to the target hardware before updating the hyperparameters of the second sampling subnetwork and before neural architecture search. The hardware performance index of each second sampling sub-network, and then establish a corresponding relationship table based on the determined hardware performance index.

When evaluating the hardware of the second sampling sub-network, according to the network layer of the second sampling sub-network to be estimated and the above-mentioned correspondence lookup table, any one or Multiple.

Based on the selection of the convolution kernel configuration parameters and quantization bit values of each layer of the network in the second sampling sub-network, one can directly find the combination of convolution kernel configuration parameters and quantization bit values of the layer network from the correspondence table. A corresponding hardware performance index. Exemplarily, there can be multiple convolution kernel configuration parameters in this layer network, and there can also be multiple quantization bit values in this layer network. The convolution kernel configuration parameters and quantization bit values can be arbitrarily combined into one group, namely There can be many combinations, and each combination has its corresponding hardware performance index.

Further, after obtaining the second supernetwork, it also includes: determining the hardware resources required by the second supernetwork and the preset limited resources; if the required hardware resources exceed the preset limited resources, then reducing the quantization bit value of each subnetwork , so that the required hardware resources are less than or equal to the preset limited resources.

Wherein, the required hardware resources may be understood as the performance of the subnetwork on the target hardware. Exemplarily, the required hardware resources may include power consumption and delay.

Wherein, the preset limited resource can be understood as a resource limit value of the updated hypernetwork model on the target hardware, that is, the hardware resources required by the updated hypernetwork cannot exceed the preset limited resource. Preset limited resources can be customized by users according to actual needs before the hypernetwork is updated.

Exemplarily, when the network model runs on the target hardware, it will definitely generate power consumption. Different network models have different power consumption. Generally, the network model with higher complexity consumes more power. Before updating the parameters, set a power consumption limit in advance, such as screening out the network above 10W, then the entire update process will be guided to search for a super network with low power consumption that is satisfactory to the user.

When the required hardware resources exceed the preset limited resources, the quantization bit value of each layer network in each sub-network can be reduced at the same time. For example, the quantization bit value of each layer network can be reduced from 10 bits to 8 bits, then reduced to 6 bits, and then reduced successively, so that the required hardware resources are less than or equal to the preset limited resources. Reducing the quantization bit value can also be understood as a way of reducing the bit width bit value of the network layer.

Embodiment two

FIG. 2 is a schematic flowchart of a differentiable search method for a mixed-precision neural network provided by Embodiment 2 of the present invention. With the development of artificial intelligence applications, the demand for deploying neural network models to edge hardware devices (such as mobile phones, Internet of Things devices, etc.) is getting higher and higher. However, the hardware deployment of neural networks often faces many hardware resource constraints, such as power consumption and delay, so it is necessary to design a lightweight network that can maintain model accuracy and have excellent performance in hardware implementation. Model quantization is a neural network compression optimization technology, which can represent the data in the neural network with a lower bit width, which can greatly reduce the amount of calculation and complexity of the model while ensuring the accuracy of the model without too much loss. Can be deployed to some resource-constrained hardware platforms. Currently, model quantization can be divided into manual model quantization and automated model quantization. In the manual model quantization method, human experts need to rely on experience and repeated experiments to determine the quantized bit value of each layer of the network. Although significant optimization has been achieved, it requires a lot of manpower and time costs, and it is easy to fall into a local optimum. excellent. Automated model quantization regards the low-bit quantization of the neural network as an optimization problem, and uses various optimization algorithms to automatically quantify each layer of the network, which has better results and lower costs than manual model quantization.

Traditional reinforcement learning-based NAS methods and evolutionary algorithm-based NAS methods consume a lot of time, computing, and memory resources during the search process. This implementation provides a differentiable search method for mixed-precision neural networks, which can perform automatic model quantization and search for a specific hardware platform. This method has the advantages of high performance, fast speed, and low resource consumption.

As shown in FIG. 2 , the differentiable search method for the mixed-precision neural network provided in this embodiment adopts a search strategy combining the differentiable search method and the evolutionary algorithm. First, obtain the hyperparameters of the hypernetwork, that is, obtain the hyperparameters of the initialized hypernetwork. Based on the hyperparameters of the hypernetwork, use gumbel-softmax to sample several subnetworks, that is, the first sampling subnetwork, and then use the differentiable search method to analyze the hyperparameters of the hypernetwork Update, including: keep the hyperparameters of the sub-networks unchanged, train these sub-networks several times on the training set, and update the weight parameters of these sub-networks, obtain the trained sub-networks, and send the trained sub-networks to The network verification set evaluator performs verification evaluation, and after the evaluation, the verification set feedback can be obtained, and the hyperparameters of the super network are updated according to the verification set feedback to obtain the first super network.

After updating the hyperparameters of the initialized hypernetwork based on the differentiable search method, the evolutionary algorithm is used to update the hyperparameters of the hypernetwork, including: based on the updated hyperparameters of the hypernetwork, the gumbel-softmax is used to sample several subnetworks again. Second sampling sub-network; then according to the hardware deployment of the model or query the lookup table, that is, the corresponding relationship table, use the hardware evaluation model to evaluate the hardware performance index of the second sampling sub-network on the target hardware, obtain hardware feedback, and perform the second sampling according to the hardware feedback Sub-network updates update the super-network hyperparameters.

The above two update methods are used alternately and cyclically to update the hyperparameters of the hypernetwork. Among them, for the accuracy feedback of the validation set, the hyper-parameters of the hyper-network are updated in a gradient-based manner, and for hardware feedback, an evolutionary algorithm is used to update the hyper-parameters of the hyper-network.

Among them, the differentiable search method is used to receive the network accuracy feedback, based on which the hyper-parameters of the hyper-network are updated. The network validation set evaluator is used to evaluate the loss/accuracy of the trained sub-network on the validation set.

A differentiable search method for a mixed-precision neural network provided in this embodiment explores the network architecture space. Compared with other methods, this method uses a gradient-based differentiable search algorithm to explore the search space, while searching for the architecture. Combined with mixed precision quantization, it has better performance and efficiency in practical task applications. In the network search process, this method adds actual hardware feedback as constraint information, and uses evolutionary algorithms to process non-differentiable hardware feedback information. It can search for architectures that meet hardware resource requirements, and it is easier to search for global optimal solutions. The method can achieve high accuracy in computer vision tasks using very little computing resources, and provides a basis for the design of automated network architectures that can be applied in practice by individual researchers, small companies, and university research teams. convenient.

The differentiable search method of the mixed-precision neural network provided by the embodiment of the present invention adopts a differentiable search method to make the search space continuous and differentiable, and regards the selection of quantized bit values of each layer of the network as a neural network architecture search problem, A gradient-based algorithm is used to search for architectural parameters, namely convolution kernel configuration parameters and network layer quantization bit values. On the one hand, the gradient-based differentiable search algorithm significantly improves the search speed compared with the reinforcement learning and evolutionary algorithms used in traditional neural network architecture search. On the other hand, the searched network is a quantized mixed-precision network, which consumes significantly less computing and memory resources than a full-precision neural network, and provides the possibility of deep learning on some resource-constrained hardware.

Embodiment Three

Fig. 3 is a schematic structural diagram of a differentiable search device for a mixed-precision neural network provided by Embodiment 3 of the present invention, which is applicable to the case of searching a high-performance neural network, wherein the device can be implemented by software and/or hardware, and Generally integrated on computer equipment.

As shown in Figure 3, the device includes:

An acquisition module 310, configured to acquire an initialized supernetwork; the supernetwork includes a plurality of subnetworks, and each subnetwork carries hyperparameters;

The first update module 320 is configured to update the hyperparameters based on a differentiable search method to obtain a first hypernetwork;

The second update module 330 is configured to evaluate the hardware performance of the first supernetwork, update the hyperparameters of the first supernetwork according to the evaluation result, and obtain the second supernetwork;

The judging module 340 is used to judge whether the update termination condition is satisfied, and if so, determine the second supernetwork as the target neural network; otherwise, return to the operation of updating hyperparameters based on the differentiable search method to obtain the first supernetwork .

In this embodiment, the device is firstly used to obtain the initialization supernetwork through the acquisition module; the supernetwork includes a plurality of subnetworks, and each subnetwork carries hyperparameters; secondly, through the first update module, it is used to The hyperparameters are updated to obtain the first hypernetwork; then the second update module is used to evaluate the hardware performance of the subnetworks included in the first supernetwork, and update the hyperparameters of the first supernetwork according to the evaluation results to obtain The second supernetwork; pass afterwards; finally pass through the judging module, used to judge whether the update termination condition is satisfied, if so, then determine the second supernetwork as the target neural network; otherwise, return to execute the differentiable search method based on all The above hyperparameters are updated to obtain the operation of the first hypernetwork.

This embodiment provides a differentiable search device for a mixed-precision neural network, capable of automatic model quantization, and capable of searching and constructing a neural network for a specific hardware platform.

Further, the supernetwork is a convolutional neural network, and the subnetwork includes at least one layer of network; the hyperparameters include the influence factor of the convolution kernel configuration parameters of each layer of the network and the influence factor of the quantized bit value, and The hyperparameters are continuously differentiable.

Further, the first updating module 320 is also configured to: sample the subnetworks included in the initialization supernetwork to obtain at least one first sampling subnetwork; train the first sampling subnetwork based on the training set; The verification set is propagated forward in the first sampling sub-network after training to obtain the target loss function value; the differential calculation is performed on the target loss function value to obtain the gradient value; the hyperparameter is updated according to the gradient value to obtain The first super network.

Further, the training module is configured to train the first sampling sub-network based on the training set, so as to update weight parameters of the first sampling sub-network.

Further, the second update module 330 is further configured to sample the subnetworks included in the first supernetwork to obtain at least one second sampled subnetwork; perform hardware evaluation on the at least one second sampled subnetwork to obtain The optimal second sampling subnetwork; increasing the influence factor of the convolution kernel configuration parameters and the influence factor of the quantization bit value of each layer of the network in the optimal second sampling subnetwork to obtain a second supernetwork.

Further, the hardware evaluation module is configured to deploy the at least one second sampling subnetwork on the target hardware to obtain hardware performance indicators; or, according to the convolution kernel configuration parameters and quantization bit values of the at least one second sampling subnetwork The size determines the hardware performance metrics.

Further, before the hardware evaluation module evaluates the hardware, the second update module 330 is also used to: establish the corresponding relationship between the configuration parameters of the convolution kernel of the at least one second sampling sub-network and the size selection of the quantization bit value and the hardware performance index Table; according to the convolution kernel configuration parameters of each layer of the at least one second sampling sub-network and the size of the quantized bit value, the corresponding hardware performance index is searched from the correspondence table.

Further, the second update module 330 is also used to determine the hardware resources required by the second supernetwork and the actual available preset limited resources; if the required hardware resources exceed the preset limited actual available resources, reduce The quantized bit value of each sub-network makes the required hardware resources less than or equal to the preset limit and actual available resources.

The above-mentioned differentiable search device for mixed-precision neural network can execute the differentiable search method for mixed-precision neural network provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

Embodiment Four

FIG. 4 is a schematic structural diagram of a computer device provided by Embodiment 4 of the present invention. As shown in FIG. 4 , the computer device provided by Embodiment 4 of the present invention includes: one or more processors 41 and a storage device 42; there may be one or more processors 41 in the device, and one processor is used in FIG. 4 41 is an example; the storage device 42 is used to store one or more programs; the one or more programs are executed by the one or more processors 41, so that the one or more processors 41 implement The differentiable search method of the mixed precision neural network described in any one of the examples.

The computer device may further include: an input device 43 and an output device 44 .

The processor 41, the storage device 42, the input device 43 and the output device 44 in the computer equipment may be connected via a bus or in other ways, and connection via a bus is taken as an example in FIG. 4 .

The storage device 42 in the computer equipment, as a computer-readable storage medium, can be used to store one or more programs, and the programs can be software programs, computer executable programs and modules, as described in the first or second embodiment of the present invention Provide program instructions/modules corresponding to the method (for example, the modules in the device shown in FIG. 3 include: a first update module 320 and a second update module 330). The processor 41 executes various functional applications and data processing of the terminal device by running the software programs, instructions and modules stored in the storage device 42, that is, realizes the differentiable search method of the mixed precision neural network in the above method embodiment.

The storage device 42 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created according to the use of the device, and the like. In addition, the storage device 42 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage devices. In some examples, the storage device 42 may further include memories that are remotely located relative to the processor 41, and these remote memories may be connected to the device through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The input device 43 can be used to receive input numbers or character information, and generate key signal input related to user settings and function control of the device. The output device 44 may include a display device such as a display screen.

And, when one or more programs included in the above-mentioned device are executed by the one or more processors 41, the programs perform the following operations:

Obtain an initialized supernetwork; the supernetwork includes a plurality of subnetworks, and each subnetwork carries hyperparameters;

updating the hyperparameters based on a differentiable search method to obtain a first hypernetwork;

performing hardware performance evaluation on the subnetworks included in the first supernetwork, updating hyperparameters of the first supernetwork according to the evaluation results, and obtaining a second supernetwork;

Judging whether the update termination condition is satisfied, if so, determining the second supernetwork as the target neural network; otherwise, returning to the operation of updating the hyperparameters based on the differentiable search method to obtain the first supernetwork.

Embodiment five

Embodiment 5 of the present invention provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, it is used to perform a differentiable search method for a mixed-precision neural network. The method includes:

Obtain an initialized supernetwork; the supernetwork includes a plurality of subnetworks, and each subnetwork carries hyperparameters;

updating the hyperparameters based on a differentiable search method to obtain a first hypernetwork;

performing hardware performance evaluation on the subnetworks included in the first supernetwork, updating hyperparameters of the first supernetwork according to the evaluation results, and obtaining a second supernetwork;

Judging whether the update termination condition is satisfied, if so, determining the second supernetwork as the target neural network; otherwise, returning to the operation of updating the hyperparameters based on the differentiable search method to obtain the first supernetwork.

Optionally, when the program is executed by the processor, it can also be used to implement the differentiable search method of the mixed precision neural network provided by any embodiment of the present invention.

The computer storage medium in the embodiments of the present invention may use any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, Random Access Memory (RAM), read-only memory (Read Only Memory, ROM), Erasable Programmable Read Only Memory (Erasable Programmable Read Only Memory, EPROM), flash memory, optical fiber, portable CD-ROM, optical storage device, magnetic storage device, or any suitable combination of the above. A computer readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a data signal carrying computer readable program code in baseband or as part of a carrier wave. Such propagated data signals may take many forms, including but not limited to: electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. .

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, Radio Frequency (RF), etc., or any suitable combination of the foregoing.

Computer program code for carrying out the operations of the present invention may be written in one or more programming languages, or combinations thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional A procedural programming language, such as the "C" language or similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as through the Internet using an Internet service provider). connect).

Note that the above are only preferred embodiments of the present invention and applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present invention, and the present invention The scope is determined by the scope of the appended claims.

Claims (7)

1. A differential search method of a mixed-precision neural network is characterized by comprising the following steps:

acquiring an initialization hyper network; the super network comprises a plurality of sub networks, and each sub network carries a super parameter;

updating the hyper-parameters based on a differentiable search method to obtain a first hyper-network;

sampling sub-networks included in the first super-network to obtain at least one second sampling sub-network;

establishing a corresponding relation table of the size selection of the convolution kernel configuration parameters and the quantization bit values of the at least one second sampling sub-network and the hardware performance indexes;

deploying the at least one second sampling sub-network to target hardware to obtain a hardware performance index; or selecting a corresponding hardware performance index from the corresponding relation table according to the convolution kernel configuration parameters and the quantization bit values of at least one second sampling sub-network to obtain an optimal second sampling sub-network;

increasing the influence factors of the convolution kernel configuration parameters and the influence factors of the quantized bit values of each layer network in the optimal second sampling sub-network to obtain a second super-network;

judging whether an updating termination condition is met, and if so, determining the second hyper-network as a target neural network; otherwise, returning to execute the operation of updating the hyper-parameters based on the differentiable search method to obtain the first hyper-network.

2. The method of claim 1, wherein the super network is a convolutional neural network, and wherein the sub-network comprises at least one layer of network; the super-parameters include an influence factor of a convolution kernel configuration parameter and an influence factor of a quantization bit value of each layer network, and are continuously differentiable.

3. The method of claim 2, wherein updating the hyper-parameter based on a differentiable search method to obtain a first hyper-network comprises:

sampling sub-networks included in the initialization hyper-network to obtain at least one first sampling sub-network;

training the first sampling subnetwork based on a training set;

forward propagating the verification set in the trained first sampling sub-network to obtain a target loss function value;

carrying out differential calculation on the target loss function value to obtain a gradient value;

and updating the hyper-parameters according to the gradient values to obtain a first hyper-network.

4. The method of claim 3, wherein training the first sampling sub-network based on a training set comprises:

training the first sampling sub-network based on a training set to update weight parameters of the first sampling sub-network.

5. The method of claim 2, after obtaining the second hyper-network, further comprising:

determining hardware resources required by the second hyper-network and preset limited resources;

and if the required hardware resource exceeds the preset limited resource, reducing the quantized bit value of each sub-network so that the required hardware resource is less than or equal to the preset limited resource.

6. A differential search apparatus for a hybrid precision neural network, comprising:

an acquisition module for acquiring an initialized hyper-network; the super network comprises a plurality of sub-networks, and each sub-network carries a super parameter;

the first updating module is used for updating the hyper-parameters based on a differentiable searching method to obtain a first hyper-network;

the second updating module is used for evaluating the hardware performance of the first hyper-network and updating hyper-parameters of the first hyper-network according to an evaluation result to obtain a second hyper-network;

the judging module is used for judging whether the updating termination condition is met or not, and if so, determining the second hyper-network as a target neural network; otherwise, returning to execute updating the hyper-parameters based on a differentiable search method to obtain the operation of the first hyper-network;

the second updating module is further configured to sample subnetworks included in the first super network to obtain at least one second sampling subnetwork; performing hardware evaluation on the at least one second sampling sub-network to obtain an optimal second sampling sub-network; increasing the influence factors of the convolution kernel configuration parameters and the influence factors of the quantized bit values of each layer of the optimal second sampling sub-network to obtain a second super-network;

the hardware evaluation module is used for deploying the at least one second sampling sub-network to target hardware to obtain a hardware performance index; or determining a hardware performance index according to the convolution kernel configuration parameters and the sizes of the quantized bit values of the at least one second sampling sub-network;

before the hardware evaluation module evaluates the hardware, the second updating module is further configured to establish a correspondence table between the size selection of the convolution kernel configuration parameters and the quantized bit values of the at least one second sampling subnetwork and the hardware performance index; and selecting to search the corresponding hardware performance index from the corresponding relation table according to the convolution kernel configuration parameter and the quantized bit value of each layer of at least one second sampling sub-network.

7. The apparatus of claim 6, wherein the super network is a convolutional neural network, and wherein the sub-network comprises at least one layer of network; the super-parameters include an influence factor of a convolution kernel configuration parameter and an influence factor of a quantization bit value of each layer network, and are continuously differentiable.

Images