WO2021007812A1 - Deep neural network hyperparameter optimization method, electronic device and storage medium - Google Patents

Abstract

A deep neural network hyperparameter optimization method, an electronic device, and a storage medium. The method comprises: using a multi-task conditional neural network model to replace an existing Gaussian process model to implement a data processing process of hyperparameter optimization; determining parameters, sequentially selecting training sets of a decoder and an encoder according to the specified parameters to perform network training, prediction, screening, and evaluation; adding candidate points of each task obtained after evaluation and target function values into observation sets of corresponding tasks; and carrying out network training, prediction, screening, and evaluation again, repeating the aforementioned steps until the maximum number of iterations is reached, and finding the point, i.e. the hyperparameter, when the target function values in the observation sets of each task are the largest, thereby solving the problems in the prior art of complex covariance calculations and the like when the Gaussian process is used to implement hyperparameter optimization, and simultaneously achieving the hyperparameter optimization of multiple tasks.

Description

A deep neural network hyperparameter optimization method, electronic equipment and storage medium Technical field

The invention relates to a hyperparameter optimization method, in particular to a deep neural network hyperparameter optimization method, electronic equipment and storage medium.

Background technique

At present, many of the network models of deep learning are not learned during the training process, but are set directly before the training starts. These parameters are called neural network hyperparameters. Hyperparameter optimization has always been a problem that limits the performance of network models. At present, neural network models are becoming more and more complex, and there are more and more types of hyperparameters, which often results in failure to select appropriate hyperparameter combinations according to the relationship between hyperparameters. At present, the most commonly used hyperparameter optimization method is the Bayesian optimization method. This method can find a good decision combination from the decision space under the actual evaluation times as much as possible. The main idea is to construct a proxy model of the entire problem process based on historical data. Real problems are evaluated, and the next sampling point is determined by the uncertainty of the proxy model prediction, and an approximate optimal solution is found after constant iteration. At present, the main proxy model in Bayesian optimization is Gaussian Process (GPs), but generally only a single problem is optimized at a time, or runs in parallel at the expense of hardware to ensure full use of data and information within a certain period of time. Single-task learning learns from scratch independently, ignoring the relevant information of other similar tasks to in-depth study of data characteristics, and often encounters problems such as large noise, high data dimensionality, or small amount of data that have a greater impact on the results. A large amount of observation data is usually required to train a sufficiently accurate single-task agent model, but it is difficult to meet the requirements in real life, which leads to certain limitations of the model trained on the data, which results in the model prediction is not accurate enough. As the amount of data increases, the computational complexity of the covariance function in the Gaussian process increases exponentially, leading to problems such as high computational cost and long running time.

Summary of the invention

In order to overcome the shortcomings of the prior art, one of the objectives of the present invention is to provide a deep neural network hyperparameter optimization method, which can solve the problems of high calculation cost and long running time for hyperparameter optimization in the prior art.

The second object of the present invention is to provide an electronic device that can solve the problems of high calculation cost and long running time when optimizing hyperparameters in the prior art.

The third object of the present invention is to provide a computer storage medium, which can solve the problems of high calculation cost and long running time when optimizing hyperparameters in the prior art.

One of the objectives of the present invention is achieved by the following technical solutions:

A deep neural network hyperparameter optimization method, including the following steps:

Parameter setting steps: set the number of training tasks, the objective function of each task, the number of initial values of each task, and the maximum number of iterations;

Model training steps: Obtain the observation set of each task according to the number of initial values of each task and the objective function, and train the network model according to the observation set of each task to obtain a multi-task conditional neural network model;

Prediction step: predict any point in the unknown area according to the multi-task conditional neural network model to obtain the objective function value of each point corresponding to each task;

Screening step: Screen the objective function value of each task according to the particle swarm algorithm and all points in the unknown area to obtain candidate points for each task;

Iterative steps: Substitute the candidate points of each task into the multi-task conditional neural network model to calculate the true value, and add the candidate points of each task and the calculated true value to the observation set of each task to form each The new observation set of the task, and then according to the new observation set, the model training step, the prediction step, the screening step and the iteration step are executed in sequence; until the maximum number of iterations is reached, the observation set formed by the last iteration is selected from the observation set corresponding to the largest response value The parameter combination of as a hyperparameter combination.

Further, the multi-task neural conditional network model learns from each other during the training of the conditional neural network model through multi-task learning, and then forms the multi-task conditional neural network model.

Furthermore, the multi-task conditional neural network model combines the output layers of the conditional neural network model of multiple tasks with a similarity network layer to form a multi-task conditional neural network model; the similarity network layer is composed of a fully connected network.

Further, the model training step specifically includes:

Step S11: Set the number of model training iterations, the maximum number of algorithm iterations, and the minimized loss function; the minimized loss function is the sum of the minimized loss functions of all tasks; the minimized loss function of each task is: minimize the negative conditional logarithm Probability

Step S12: randomly select the corresponding initial value from the observation set of each task according to the preset ratio, generate the decoder training set of the corresponding task, and use the observation set of each task as the encoder training set of the corresponding task;

Step S13: Perform network model training according to the decoder training set and encoder training set of each task, and update the network parameters of the conditional neural network model of each task and the parameters of the similarity network layer according to the minimized loss function;

Step S14: When the network model training reaches the number of model training iterations, return to perform step S12 to step S14 in sequence; until the maximum number of algorithm iterations is reached, a multi-task conditional neural network model is obtained.

Further, the step S13 further includes: updating the network parameters of the conditional neural network model and the parameters of the similarity network layer of each task according to the minimized loss function and using the Adam optimizer back propagation algorithm.

Further, the generation process of the observation set is as follows: First, the objective function of each task is generated according to the objective function of each task and the number of initial values, and the corresponding objective function value is obtained according to the initial value and objective of each task. The function value is the observation value of each task.

Further, the screening step is: calculating the collection function EI of each task according to the value of the objective function for each task at each point in the unknown area, and using the collection function EI of each task as the fitness of the particle swarm algorithm Function, and then select the point with the largest function value of the collection function EI according to the particle swarm algorithm as the candidate point for each task.

The second objective of the present invention is achieved by adopting the following technical solutions:

An electronic device, including a memory, a processor, and a computer program stored on the memory and running on the processor. The processor implements a deep neural network as one of the objectives of the present invention when the processor executes the program Steps of hyperparameter optimization method.

The third objective of the present invention is achieved by the following technical solutions:

A computer-readable storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the steps of a deep neural network hyperparameter optimization method adopted as one of the objectives of the present invention are realized.

Compared with the prior art, the present invention has the following beneficial effects:

The present invention implements the training of the model by using the conditional neural network model instead of the Gaussian model, and combines it with multi-task learning to form a multi-task conditional neural network model, and applies the multi-task conditional neural network model to the Bayesian algorithm framework, In turn, multi-task hyperparameter optimization is realized, so that the calculation cost and running time cost are reduced in the process of hyperparameter optimization.

Description of the drawings

FIG. 1 is a flowchart of a method for optimizing hyperparameters of a deep neural network provided by the present invention;

Figure 2 is a network structure diagram of the conditional neural network model provided by the present invention;

Figure 3 is a network structure diagram of a multi-task conditional neural network model provided by the present invention;

Figure 4 is a network training diagram of a multi-task conditional neural network model provided by the present invention;

FIG. 5 is a schematic diagram of the prediction of the prediction point x ^* in the multi-task conditional neural network model provided by the present invention;

Fig. 6 is a flowchart of a hyperparameter optimization method for applying a multi-task conditional neural network model to a Bayesian decision algorithm provided by the present invention;

Fig. 7 is a schematic diagram of the extended data set provided by the present invention.

Detailed ways

Hereinafter, the present invention will be further described with reference to the drawings and specific implementations. It should be noted that, provided that there is no conflict, the following embodiments or technical features can be combined to form new embodiments. .

Example one:

The present invention proposes a method that uses Conditional Neural Processes (CNPs) to replace Gaussian Processes (GPs), and CNPs combine the characteristics of random processes and neural networks, and are inspired and used by the flexibility of Gaussian processes Gradient descent is used for neural network training to realize the hyperparameter optimization method.

The conditional neural network process is parameterized by the neural network when learning the known observation data. The conditional neural network model is trained by randomly sampling the data set and following the gradient steps to maximize the random number given a random observation set. Set the conditional likelihood.

The general method of hyperparameter optimization in the prior art is to use the Gaussian process for model training, and this patent uses a conditional neural network model with stronger learning ability to replace the traditional Gaussian process, avoiding the calculation of complex covariance functions . By combining the advantages of current multi-task learning, a multi-task conditional neural network process (Multi-task Conditional Neural Processes in One-to-many case, OMc-MTCNPs) is proposed on the basis of the conditional neural network model and applied to In the Bayesian decision algorithm (Bayesian), the hyperparameters are optimized.

In the multi-task conditional neural network model, the multi-task conditional neural network model can make full use of the known small amount of data to enhance the generalization performance of each single task. Applying the multi-task conditional neural network model to the framework of the Bayesian decision algorithm can find a better approximate optimal solution in the hyperparameter optimization process, and solve the long running time and high computational cost of the existing hyperparameter optimization processing. problem.

For example, as shown in Figure 1, for M tasks (that is, each task is a deep network model, and each model has a deep network hyperparameter optimization training set). Among them, each task has multiple hyperparameters, and the multi-task conditional neural network model can iteratively find a better hyperparameter combination corresponding to each task according to the respective training set and training results of each task. When the optimal hyperparameter combination is found, the unknown point can be accurately predicted under the condition that the optimal hyperparameter exists.

In order to better explain how the multi-task conditional neural network model optimizes hyperparameters, the present invention first introduces the model training process of the conditional neural network model and the model training process of the multi-task conditional neural network model.

1. For conditional neural network model (CNPs model):

1. The conditional neural network model uses two stages of learning data to alleviate the problem that a small amount of data cannot be effectively trained: the first stage is to learn the statistical distribution information of the training data, the second stage only uses part of the training data and the first stage to learn The distribution information of the joint fits a specific function.

有一个未知表达式的函数f:X→Y，将X输入到函数f中得到输出

其中y _i＝f(x _i)。 For example, suppose we have n data sets with different inputs

There is a function f:X→Y with an unknown expression, input X into the function f to get the output

Where y _i =f(x _i ).

Among them, the first stage: Figure 2 shows the process structure diagram of the conditional neural network. The conditional neural network is composed of two parts, namely the encoder h and the decoder g, both of which are composed of neural networks. Among them, the encoder h mainly learns the mapping relationship between the data, that is, parameterized conditional probability; the decoder g uses the learned information to perform predictive regression, that is, calculates the conditional probability.

Among them, the second stage: fitting function p: X→Y, the decoder g finally makes f and p as close as possible.

CNPs在观测值

上参数化f(T|O,T)，利用这种方式放弃了随机过程的数学计算，但增加了灵活性和扩展性。 One advantage of the conditional neural network model is that there is no need to set a Gaussian prior, but to obtain distribution information from the data directly through the neural network. If we have a set of m unobserved data

CNPs in observations

The above parameterization f(T|O,T), in this way, the mathematical calculation of the random process is abandoned, but the flexibility and scalability are increased.

2. Mathematical analysis of conditional neural network model

Let P _θ be a distribution on the random variable f(x),x∈T, where θ is the set of all parameters that define P _θ . Then arrange the known observation set O and the unknown observation set T in a certain order to obtain O'and T', that is:

P _θ (f(T)|O,T)=P _θ (f(T)|O,T')=P _θ (f(T)|O',T). Among them, T={x1,x2,x3,...,xn}. Therefore, P _θ (f(T)|O _, T)=∏ _x∈T P _θ (f(T)|O _, x)(1), that is, to transform each combination of x in T, That is, it is possible to decompose P _θ and perform a separate conditional distribution calculation for each input x.

As shown in Figure 2, it represents the structure of the conditional neural network model, where r _i =h _θ (x _i ,y _i )

为整合所有学习到的信息并映射到单个x上。在大部分实验中

用来表示加权平均，即式子(2)可以等价于r＝(r ₁+r ₂+…+r _n-1+r _n)/n。 Among them, h _θ : X×Y→R ^d and g _θ : X×R ^d →R ^e are neural networks. The encoder h in formula (2) extracts the known data information to obtain r, and then uses the formula ( 3) It is integrated and used as one of the decision factors for the encoder g to predict each x. Formula (4) is the mathematical representation of the prediction made by the encoder g. Among them, the symbol

To integrate all learned information and map it to a single x. In most experiments

Used to express the weighted average, that is, formula (2) can be equivalent to r=(r ₁ +r ₂ +...+r _n-1 +r _n )/n.

3. Model parameter training and prediction

将其作为模型训练的编码器h。 First select a data set O, and randomly select a subset from the data set O according to a preset ratio as the encoder h for model training, and use the data set O as the decoder g for model training. That is, the parameters in the training P _θ are updated by the error between the target predicted value of X in the data set O and the real target value, and the obtained parameters can make the predicted result more accurate. When selecting a subset, generally speaking, N～uniform[1,...,n] is usually set, that is, the first N elements of the data set O are selected to obtain the subset

Use it as the encoder h for model training.

Therefore, to train the conditional neural network model based on the data set, the steps are as follows:

Step A1: First set the learning rate, the number of model training iterations, and the maximum number of algorithm iterations. Before a model is trained, these hyperparameters can be given based on historical experience to facilitate the start of model training. Among them, the learning rate is also a hyperparameter, which is mainly used to set the neural network model, how to learn, through the optimization of the hyperparameter, set it to the neural network model to achieve model training.

Step A2: according to a preset ratio O randomly selected subset of the data set from O, _N. Generally speaking, N～uniform[1,...,n]. That is, the first N data are selected from the data set O as the subset _ON .

Step A3: Use the data set O as the training set of the decoder g, and the subset _ON as the training set of the encoder h.

其中，

另外，该最小化损失函数是通过最小化附条件对数概率得到的，其计算公式是常规公式，本发明不对计算公式进行改进。 Step A4: Use the minimized negative conditional logarithmic probability as the minimized loss function, that is:

among them,

In addition, the minimized loss function is obtained by minimizing the conditional logarithmic probability, and its calculation formula is a conventional formula, and the present invention does not improve the calculation formula.

Step A5: Use the Adam optimizer update algorithm to update the parameters of the multi-condition network model according to the minimized loss function; return to step A2 to step A5 after each model training iteration number, continue to select subsets, model training and parameters Update, and so on.

Step A6: When the maximum number of iterations of the algorithm is reached, the training ends, and the conditional neural network model of the task is obtained.

After the model training is completed, input the unknown point x ^* into the decoder g of the conditional neural network model, and the unknown point x ^* can be predicted by the conditional neural network model to obtain the objective function value, such as the objective function The predicted mean and variance of the value.

2. Multi-task conditional neural network model:

The Gaussian process is suitable for occasions with small sample size, but it has the disadvantage of high computational complexity of the covariance function; and the above-mentioned conditional neural network model can only perform model training on a single task. For multi-task model training, the present invention expands the conditional neural network model and expands it into a multi-task conditional neural network model (Multi-task Conditional Neural Processes in One-to-many case, OMC-MTCNPs) Through multi-task learning, not only can the problem of insufficient data be alleviated, but also the learning ability of a single task can be improved through mutual learning between tasks, and the complicated covariance calculation process can also be avoided.

In other words, the present invention combines multi-task learning with the strong learning and predictive capabilities of the conditional neural network model, proposes a multi-task conditional neural network model, and applies the multi-task conditional neural network model to Bayesian decision making The optimization framework of the algorithm is used to optimize the hyperparameters, and then obtain the optimal hyperparameters. Both the conditional neural network model and the multi-task conditional neural network model have strong scalability. For some simple low-dimensional data, a fully connected layer can achieve better performance, and for some complex high-dimensional data problems, we can use Convolutional layer to efficiently extract data information.

1. First, the present invention provides the network structure of the multi-task conditional neural network model, as shown in Fig. 3.

代表训练集。其中，每个任务同时采样相同的集合X，Y _l代表点集合X在第l个任务上的目标函数值的集合。 Suppose there are M tasks,

Represents the training set. Among them, each task samples the same set X at the same time, and Y _l represents the set of objective function values of the point set X on the l-th task.

In other words, the multi-task conditional neural network model combines the conditional neural network models of multiple tasks, and combines the output layer of each conditional neural network model with the similarity network layer k to form the multi-task conditional neural network. Network model.

和方差矩阵

用神经网络连接起来，则整个模型输出的均值矩阵m和方差矩阵v分别由

和

表示。w _m和b _m分别表示均值连接项的权值矩阵和偏置矩阵，w _v和b _v分别表示方差连接项的权值矩阵和偏置矩阵。也就是说，每个任务最终输出经过相似性网络层k后都与其他任务相关。 However, for the similarity network layer k is composed of a fully connected network, it mainly learns the similarity measure between each task by updating parameters, and combines the similarity information to perform a certain linear combination of the prediction results of each task to obtain the final the result of. In other words, the mean matrix of each conditional neural network model output by the similarity network layer k

Sum variance matrix

Connected by a neural network, the mean matrix m and variance matrix v output by the entire model are respectively determined by

with

Said. w _m and b _m respectively represent the weight matrix and the bias matrix of the mean connection term, and w _v and b _v represent the weight matrix and the bias matrix of the variance connection term, respectively. In other words, the final output of each task is related to other tasks after passing through the similarity network layer k.

The multi-task conditional neural network model uses the experience learned from similar related tasks to effectively spread the information of one task to the network model of other tasks through an appropriate loss function, and realize the information sharing between tasks. Due to the characteristics of the multi-task model, the scale of observation data is increased by M times, and each task can use the information shared by other tasks, avoiding the task to learn from scratch from its own data, and it can also update these experiences from the network. Integration in the process of dealing with similar problems. Figure 3 only draws the addition of a fully connected layer to represent the relevance of tasks, but if you deal with complex problems, the network structure or the number of layers in the similarity network layer k can be flexibly changed according to different tasks to meet requirements. When the relationship between tasks is difficult to express, by adding more hidden layers, the similarity network layer k can learn the mathematical distribution characteristics of all data. These characteristics are represented in hidden nodes. When the network is forwarded, it can be Activate these nodes to encourage tasks to learn from each other.

2. Mathematical analysis of multi-task conditional neural network model

和未观测数据Tl。 Suppose we test M tasks, for the lth task in the multi-task conditional neural network model, there are n different input observation data

And unobserved data Tl.

f _l表示第l任务的目标函数，O _l＝{X _l,Y _l＝f _l(X _l)}为第l个任务的观测集合，L _l为对应第l个任务的损失函数，θ为所有定义Q的参数向量。 Let Q _lθ be the probability distribution on the function f _l :X _l ←Y _l , namely

f _l represents the objective function of the lth task, O _l = {X _l , Y _l = f _l (X _l )} is the observation set of the lth task, L _l is the loss function corresponding to the lth task, and θ is All parameter vectors that define Q.

In the conditional neural network model of each task of the multi-task conditional neural network model, the mathematical structure is similar to the aforementioned formulas (2), (3) and (4).

结构，

代表

的参数，其含义与单任务的条件神经网络模型相似，

即为第l个任务的输出经过相似性网络层处理后的最终结果。 For the first task dependency network layer k use

structure,

representative

The parameters of, whose meaning is similar to that of a single-task conditional neural network model,

It is the final result after the output of the lth task is processed by the similarity network layer.

3. The training process of the multi-task conditional neural network model is as follows:

In the multi-task conditional neural network model, the input observation data of each task is the same, that is, it is set as X=X ₁ =X ₂ =...=X _M. Since a priori knowledge means to ensure that the X input to each task is consistent, the mutual results obtained in this way are related, otherwise noise will be generated, such as x1 input task 1, different values x2 input task 2, the results obtained are not There is no certain connection. Therefore, the input observation data of each task is set to be the same in this application.

During training, we will appropriately adjust the training data ratio σ of the encoder selected from the training data set. Therefore, the multi-task conditional neural network model prevents overfitting under the premise of fully learning the observation data, and at the same time guarantees the uncertainty of the model in the learning decision space X. Under the premise of ensuring randomness, a subset of the data set is randomly selected multiple times according to the set data ratio as the training data set of the encoder, which can make the generalization effect of the model better.

Similarly, when training the model and updating the parameters, it is also necessary to minimize the loss function.

则最小化负条件对数概率： make

Then minimize the negative conditional logarithmic probability:

其中w _l为第l任务损失函数的权重。为了均衡每个任务的比重，我们将权重设置为1。

Where w _l is the weight of the loss function of the lth task. In order to balance the weight of each task, we set the weight to 1.

Figure 4 shows the model training of the multi-task conditional neural network model. When the multi-task conditional neural network model is trained, the conditional neural network model of each task will be trained accordingly, and the conditional neural network model of each task can be Independently learn related data set features for other tasks.

That is, first train the conditional neural network model of each task, and then combine the tasks into a whole for training, and update the parameters of the conditional neural network model of each task at the same time, and reverse the loss function under the minimized loss function. To update the network parameters of the conditional neural network model of each task, that is, to distribute the information of different tasks to the parameters of the conditional neural network model of each task. At the same time, it is also necessary to update the parameters in the similarity network layer k, and each parameter tends to show the correlation between tasks.

The specific steps of multi-task conditional neural network model training are as follows:

Step B1: Set network parameters such as the number of model training iterations, the maximum number of algorithm iterations, and the minimized loss function. The minimized loss function is the sum of the loss functions of each task.

Step B2: Select the corresponding decoder training set O and encoder training set O ^σn for each task according to the preset ratio for model training.

Step B3: Update the parameters of the conditional neural network model of each task and the parameters of the similarity network layer k according to the minimized loss function and the Adam optimizer back propagation update algorithm, that is, the dashed line and the dashed box related parameters in FIG. 4.

进行模型训练以及参数更新等，以此类推。 Step B4: When the model training reaches the number of model training iterations, return to step B2 to step B4, and randomly select the encoder training set again in turn

Perform model training and parameter update, and so on.

Step B5: When the maximum number of iterations of the algorithm is reached, the network training is completed, and the multi-task conditional neural network model is obtained.

After the multi-task conditional neural network model is trained, it can predict any unknown point x _*, and predict the objective function value of the unknown point on each task, such as predicting the mean and variance of the unknown point on each task.

As shown in Figure 5, the decoder of the conditional neural network model of each task inputs the unknown point x _* , and the corresponding training set is still input on the encoder training set. This part is to read the training set information and make subsequent predictions Help out. Since the information between tasks is shared through the similarity network layer k, the predicted value output by the conditional neural network model of each task is significantly better than the result predicted by a single task.

Therefore, in order to realize the optimization of hyperparameters, the present invention provides a hyperparameter optimization method based on a multi-task conditional neural network model. The multi-task conditional neural network model is applied to the framework of the Bayesian decision algorithm to realize the hyperparameter optimization. As shown in FIG. 6, according to the foregoing, the multi-task conditional neural network model has excellent predictive performance. In the present invention, it is applied as a proxy model to the framework of the Bayesian decision algorithm to optimize the hyperparameters .

The method specifically includes the following steps:

Step S1: Setting of network parameters. For example, determine the dimension d of the parameters, the number n of initial data for each task (usually n=11×d-1), the number of tasks M, the maximum number of iterations T, and the objective function f _{l of the lth} task. .

Step S2: Obtain the initial value set of each task according to the dimension of the parameters and the number of initial data of each task. For example, the initial value set can be expressed as

For example: the dimension of the parameter is 2, that is, the two-dimensional parameter [x1,x2], where the range of x1 is [0,6], and the range of x2 is [6,10]. Suppose that 4 initial points are to be generated, that is, x1 takes 2,4; x2 takes 8,9. Finally, the initial points [2,8],[2,9],[4,8],[4,9] are obtained. That is to say, divide evenly in the decision space, and then find points in each small part, and finally get the data of the initial point, and generate the initial point set.

观测集为

Step S3: Input each initial value in the initial value set of each task into the objective function of the corresponding task, calculate the objective function value of the corresponding task, and then according to the initial value of each task and the corresponding objective function value Form the observation set for each task. That is, the objective function value is

The observation set is

Step S4: Select a subset of the corresponding task from the observation set of each task according to the preset ratio, use the observation set as the decoder training section, and use the subset as the encoder training set for network model training, and then train to obtain Multi-task conditional neural network model. Among them, the subset is

和方差σ _l＝1:M(x)。 Step S5: Predict the predicted value of any point x in each task in the unknown area according to the multi-task conditional neural network model, that is, input the arbitrary point into the conditional neural network model of each task to obtain the corresponding objective function value . Such as mean

The sum variance σ _l=1:M (x).

In this way, each task corresponds to multiple unknown points and corresponding objective function values, where the unknown points are the combination of parameters of the task.

由于每个任务均在多个参数组合，因此根据每个任务的所有未知点与目标函数值，以及粒子群算法对未知点进行筛选，进而筛选出每个任务的最优点，也即是最优参数组合。 Step S6: Use the particle swarm algorithm to find the optimal parameter combination for each task. The optimal parameter combination is

Since each task is combined with multiple parameters, the unknown points are filtered according to all the unknown points and objective function values of each task, as well as the particle swarm algorithm, and then the best advantage of each task, that is, the optimal Parameter combination.

Further, to screen unknown points, the present invention also uses the collection function EI to screen. Since the collection function EI has the expected improvement characteristics of oil volume, the collection function EI is used as the fitness function of the particle swarm algorithm. It calculates the collection function EI of each task according to the value of the objective function of each task at each point in the unknown area, and uses the collection function EI of each task as the fitness function of the particle swarm algorithm, and then selects it according to the particle swarm algorithm The point with the largest function value of the collection function EI is used as the candidate point of each task, and the candidate point of each task is the optimal parameter combination of the corresponding task.

然后再进行模型训练、预测、筛选等。 Step S6: Input the optimal parameter combination of each task into the objective function of the corresponding task to calculate the true value to obtain the objective function value, and add the optimal parameter combination and objective function value of each task to the observation of the corresponding task Concentrate and form a new observation set corresponding to the task, and then continue to perform steps S3 to S6 according to the new observation set. For example, for the mth task, add the calculated point and the objective function value to the observation set

Then perform model training, prediction, and screening.

As shown in Figure 7, when the optimal parameter combination of each task is found, the optimal parameter combination of each task is actually evaluated, and then the objective function value corresponding to the optimal parameter combination of each task is calculated. That is to say, the optimal parameter combination of each task and the corresponding objective function value are added to the observation set of the corresponding task, and then the model training, prediction, screening, and evaluation operations are performed again. The real evaluation here refers to: after the decision variables are obtained, the decision variables are substituted into the test problem and recalculated, that is, the optimal parameter combination is input into the objective function of the corresponding task for prediction, and the corresponding objective function value is obtained .

即为每个任务的最优超参数。 Step S7: By analogy, the above steps are executed in turn, when the maximum number of iterations of the algorithm T is reached, the new observation set formed by each task is to find the point corresponding to the maximum value of the objective function of each task, that is Is a combination of parameters:

That is the optimal hyperparameter for each task.

According to the above optimized hyperparameters of each task, the multi-task conditional neural network model can accurately predict the objective function value of any point on each task, such as the mean value and the variance.

In order to further illustrate the above hyperparameter optimization method, the present invention also provides related experimental tests, which further illustrate the effectiveness and accuracy of the hyperparameter optimization of the present invention relative to the prior art. The specific data are referred to as follows:

Error in the MNIST data set! The reference source was not found. Test the multi-classification performance of the Lenet-5 network on the Fashion-MNIST dataset.

This experiment sets up two kinds of hyper-parameter problems: the first one sets two variable hyper-parameters of learning rate and bias, and the other hyper-parameters are set to default fixed values; the second one is to add the dropout value on the basis of the first one . In the table, use Q1 and Q2 to represent these two problems, and show the mean and standard deviation of the accuracy of the three tasks on the test set (data in parentheses).

This experiment uses two proxy models, the conditional neural network model and the multi-task conditional neural network model, to optimize the hyperparameters in the framework of the Bayesian decision algorithm. That is, to find a hyperparameter combination that can make the network prediction accuracy high, and then can verify the effectiveness and accuracy of the multi-task conditional neural network model given in this paper for the hyperparameter optimization problem based on the experimental results. In addition, since the neural network may get different results each time it is run, in order to eliminate errors, all accuracy values are the average of the results of ten runs.

Table 1 The mean and standard deviation of the accuracy of MNIST tested by neural network

Table 2 The mean and standard deviation of the accuracy of the neural network test Fashion-MNIST

From Table 1 and Table 2, the hyperparameter performance of the multi-task conditional neural network model (OMc-MTCNPs model) is better than the hyperparameter optimization based on the conditional neural network model (CNPs) and the traditional hyperparameter optimization based on the Gaussian process. Therefore, the hyperparameter optimization method based on the multi-task conditional neural network model proposed by the present invention is better.

That is to say, the present invention adopts the conditional neural network model as the agent model in the Bayesian decision algorithm framework, replacing the traditional Gaussian process, realizing single-task hyperparameter optimization, and adopting the conditional neural network model on the basis of The similarity network layer is extended to a multi-task conditional neural network model, which realizes multi-task hyperparameter optimization. By proposing a similarity network layer in multi-task learning, it is different from the general regularization constraint item in that it is composed of a neural network, which can parameterize the similarity between tasks and share different information learned by each task to improve Generalization performance of each task.

In addition, because the present invention uses the conditional neural network model to replace the traditional Gaussian process to build a model for a single task, and the conditional neural network model is composed of a multi-layer special network structure, it can avoid the complex covariance of the traditional Gaussian process The problem of calculation, and the learning and fitting function of the conditional neural network model is also stronger than the traditional Gaussian process, and there is no need to set a suitable prior distribution in advance (because the Gaussian process needs to set the data set distribution model (Gaussian distribution ), and then use the data set to update the parameters in the Gaussian distribution, and it may not be suitable for the current data set. Conditional neural network model is to set a specific network structure, only need to update the parameters, it has strong adaptability) Therefore, the multi-task conditional neural network model formed by combining the conditional neural network model with multi-task learning greatly improves the learning ability of the model, thereby realizing hyperparameter optimization, and avoiding computational complexity.

The present invention also provides an electronic device, which includes a memory, a processor, and a computer program that is stored in the memory and can be run on the processing. The processor implements the deep neural network described in the article when the program is executed. Steps of hyperparameter optimization method.

The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of a deep neural network hyperparameter optimization method as described in the text are realized.

The foregoing embodiments are only preferred embodiments of the present invention, and cannot be used to limit the scope of protection of the present invention. Any insubstantial changes and substitutions made by those skilled in the art on the basis of the present invention belong to the present invention. The scope of protection required.

Claims (9)

A method for optimizing hyperparameters of a deep neural network is characterized in that it comprises the following steps: Parameter setting steps: set the number of training tasks, the objective function of each task, the number of initial values of each task, and the maximum number of iterations; Model training steps: Obtain the observation set of each task according to the number of initial values of each task and the objective function, and train the network model according to the observation set of each task to obtain a multi-task conditional neural network model; Prediction step: predict any point in the unknown area according to the multi-task conditional neural network model to obtain the objective function value of each point corresponding to each task; Screening step: Screen the objective function value of each task according to the particle swarm algorithm and all points in the unknown area to obtain candidate points for each task; Iterative steps: Substitute the candidate points of each task into the multi-task conditional neural network model to calculate the true value, and add the candidate points of each task and the calculated true value to the observation set of each task to form each The new observation set of the task, and then according to the new observation set, the model training step, the prediction step, the screening step and the iteration step are executed in sequence; until the maximum number of iterations is reached, the observation set formed by the last iteration is selected from the observation set corresponding to the largest response value The parameter combination of as a hyperparameter combination. The method for optimizing deep neural network hyperparameters according to claim 1, characterized in that: the multi-task neural conditional network model learns from each other during the training of the conditional neural network model through multi-task learning, Then a multi-task conditional neural network model is formed. The method for optimizing deep neural network hyperparameters according to claim 2, wherein the multi-task conditional neural network model combines the output layers of the conditional neural network model of multiple tasks with a similarity network layer to form a multi-task Conditional neural network model; the similarity network layer is composed of a fully connected network. The method for optimizing hyperparameters of a deep neural network according to claim 1, wherein the model training step is specifically: Step S11: Set the number of model training iterations, the maximum number of algorithm iterations, and the minimized loss function; the minimized loss function is the sum of the minimized loss functions of all tasks; the minimized loss function of each task is the minimized negative conditional log probability ； Step S12: randomly select the corresponding initial value from the observation set of each task according to the preset ratio, generate the decoder training set of the corresponding task, and use the observation set of each task as the encoder training set of the corresponding task; Step S13: Perform network model training according to the decoder training set and encoder training set of each task, and update the network parameters of the conditional neural network model of each task and the parameters of the similarity network layer according to the minimized loss function; Step S14: When the network model training reaches the number of model training iterations, return to perform step S12 to step S14 in sequence; until the maximum number of algorithm iterations is reached, a multi-task conditional neural network model is obtained. The method for optimizing hyperparameters of a deep neural network according to claim 4, wherein: The step S13 further includes: updating the network parameters of the conditional neural network model and the similarity network layer of each task according to the minimized loss function and using the Adam optimizer back propagation algorithm. The method for optimizing hyperparameters of a deep neural network according to claim 1, characterized in that: the generation process of the observation set is as follows: firstly, the objective function of each task is generated according to the objective function of each task and the number of initial values. The corresponding objective function value is obtained, and the observation value of each task is obtained according to the initial value of each task and the objective function value. The method for optimizing the hyperparameters of a deep neural network according to claim 1, characterized in that: the screening step is: calculating the acquisition function EI of each task according to the objective function value of each task at each point in the unknown area , And use the collection function EI of each task as the fitness function of the particle swarm algorithm, and then select the point with the largest value of the collection function EI as the candidate point of each task according to the particle swarm algorithm. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and capable of running on the processor, characterized in that: the processor executes the program when the program is executed as in any one of claims 1-7 The steps of a deep neural network hyperparameter optimization method. A computer-readable storage medium with a computer program stored thereon, wherein the computer program implements the deep neural network hyperparameter optimization method according to any one of claims 1-7 when the computer program is executed by a processor A step of.

Images