CN118554555A - MASAC algorithm-based distributed photovoltaic voltage reactive power control method for power distribution network - Google Patents

Abstract

The invention discloses a distributed photovoltaic voltage reactive power control method for a distribution network based on a MASAC algorithm, which belongs to the field of power system automation technology and artificial intelligence reinforcement learning. First, a distribution network voltage reactive power decentralized control framework taking into account distributed PV is constructed, and the voltage reactive power decentralized control problem of the distribution network is converted into a Markov game model; then a MASAC algorithm is constructed to solve the Markov game model, and an Actor and Critic neural network is constructed for each intelligent agent, and the neural network is trained in a centralized training manner to obtain a distributed photovoltaic voltage reactive power control model for the distribution network, and then the model is used to realize online control of the distribution network voltage, complete online control of the distribution network voltage, and realize decentralized regulation of photovoltaic inverters. The invention can reduce communication requirements and computing burdens, improve grid voltage stability, and has wide applicability and flexibility.

Description

A distributed photovoltaic voltage and reactive power control method for distribution network based on MASAC algorithm

Technical Field

The present invention belongs to the field of power system automation technology and artificial intelligence reinforcement learning, and specifically relates to a distributed photovoltaic voltage reactive power control method for a distribution network based on a MASAC algorithm.

Background Art

As the penetration of distributed energy resources, especially PV (photovoltaic), continues to increase, they are becoming increasingly important for meeting the growing electricity demand and protecting the environment. However, the unpredictability and volatility of these resources pose many technical challenges to system operators. In particular, the overvoltage problem caused by reverse current caused by excessive PV penetration is of particular concern under low load conditions. In order to improve the voltage profile of the distribution network, reactive voltage control (VVC) is an effective tool that can be used to control the reactive power set point of traditional reactive sources or newly added inverters to regulate the voltage of the distribution network. The VVC method regulates the voltage by utilizing the reactive power absorption/injection capabilities of capacitor banks and smart inverters. From the perspective of control strategies, voltage regulation methods can be divided into four categories: centralized control, local control, distributed control, and decentralized control. Centralized control requires the establishment of fast communication channels, which is costly; local control and distributed control are subject to the need for coordination between agents, which may not be feasible in many applications. Finally, decentralized control combines the advantages of distributed and centralized control methods and utilizes partition control and interval coordination. Traditional decentralized control models require system topology and parameters, which are difficult to obtain in actual distribution systems, especially when a large number of indoor photovoltaic units are popular. Therefore, developing a decentralized control method that does not rely on the precise parameters of the distribution network will overcome the above challenges.

Deep reinforcement learning, as one of the most commonly used machine learning-based methods, has attracted widespread attention because they can learn optimal control strategies. In deep reinforcement learning methods, the correlation between actions and states is obtained through continuous interaction with the environment. Therefore, it reduces the dependence on obtaining complete information about system parameters. A well-trained deep reinforcement learning agent can provide adaptable actions for any new dynamics in the real world. Studies have used a reactive voltage control framework based on deep reinforcement learning for voltage regulation in distribution networks. However, these methods require communication links and centralized processing to make decisions based on system status, so they may not be suitable for large-scale power grids with thousands of PVs. A multi-agent deep reinforcement learning (MADRL) method was proposed, which uses the multi-agent deep deterministic policy gradient (MADDPG) algorithm for voltage regulation. This method is only performed in the centralized training phase, while the decentralized execution phase reduces the necessity of real-time communication between numerous distributed resources. However, they have poor exploration efficiency and are not suitable for solving multi-agent decision-making scenarios in large-scale power systems.

Summary of the invention

In view of the problems existing in the prior art, the present invention provides a distributed photovoltaic voltage reactive power control method for a distribution network based on a MASAC algorithm, which is better applied to photovoltaic regulation of large-scale power systems.

In order to solve the above technical problems, the present invention provides the following technical solutions: a distributed photovoltaic voltage reactive power control method for distribution network based on MASAC algorithm, comprising the following steps:

S1. Construct a decentralized control framework for voltage and reactive power of distribution network with distributed photovoltaics, and transform the problem of decentralized control of voltage and reactive power of distribution network into a Markov game model.

The framework of decentralized control of voltage and reactive power in distribution network includes: minimizing the active power loss of distribution network over a period of time as the goal, using distributed photovoltaic inverters as decision variables, and using a preset voltage range as a constraint;

The Markov game model includes: state space: a set of local observation values consisting of the net injection of active/reactive power of the photovoltaic inverters and the voltage amplitude of the photovoltaic inverters included in each agent, action space: a set of actions consisting of the reactive output of the photovoltaic inverters controlled by all agents; reward function: constructed by the active loss cost and the voltage limit penalty; state transfer process: the state of the agent follows the power flow calculation constraints of the distribution network and is updated according to the state transfer probability distribution;

S2. Construct the MASAC algorithm to solve the Markov game model, and construct Actor and Critic neural networks for each agent. The Actor neural network determines the strategy of the agent, and the Critic neural network is used to determine the value of the strategy.

The neural network is trained in a centralized training manner to obtain a distributed photovoltaic voltage reactive control model for the distribution network. This model is then used to realize online control of the distribution network voltage and achieve decentralized regulation of the photovoltaic inverter.

Furthermore, in the aforementioned step S1, the objective is to minimize the active power loss of the distribution network within a period of time, and specifically, the objective function is constructed as follows:

Wherein, T represents the optimization time period, and P _loss (t) represents the active network loss at time t.

Furthermore, in the aforementioned step S1, the constraint condition is:

Where V _k (t) is the voltage of PV inverter k at time t, V and They are the preset voltage lower and upper limits respectively.

Furthermore, in the aforementioned step S1, the state space S is a set of local observation values si _,t of all photovoltaic inverters at time t, si _,t is a local observation value of agent i at time t, _si,t = ( _pi , _qi , _vi ), _pi , _qi and _vi represent the net injection amount of active/reactive power and the node voltage amplitude of the photovoltaic inverter where agent i is located, respectively;

The action space A is the set of actions a _i,t consisting of the reactive output of the PV inverters controlled by all agents at time t, and Q _PV,i,t is the reactive output of the PV inverter controlled by agent i at time t.

Furthermore, in the aforementioned step S1, the reward function is as follows:

In the formula, R(t) is the reward value at time t, P _loss (t) is the active power loss at time t, and the function is a 0-1 discriminant function. When the voltage V _k (t) of the k node satisfies the upper and lower limits V , When function f is 0, otherwise it is 1, σ ₁ is the unit active power loss cost, and σ ₂ is the voltage limit penalty factor.

Furthermore, in the aforementioned step S1, the state transfer process uses the PYPOWER power flow calculation tool to establish the distribution network environment, uses the runpf function to perform power flow calculation, and the power flow calculation constraints include power balance constraints and power flow constraints; the state transfer probability distribution of the intelligent agent is P(s′|s,a), which means that after the intelligent agent takes action a _t according to the current state S _t , the probability of the environment transferring from S _t to S′ _t under the action of action a _t .

Furthermore, the aforementioned step S2 includes the following sub-steps:

S201. Construct an actor network of each agent based on the Actor network. The strategy of the actor network of each agent is as follows:

in, For each agent’s action at a specific time point t, Determined by the Actor network; i represents the index of the agent, and the state vector of agent i at time t is expressed as The strategy of each agent is denoted as It is a strategy based on compressed Gaussian distribution;

S202. Each agent iteratively updates the entropy of the joint strategy π(a _t |s _t ) based on maximizing the expected return and the strategy, and the entropy H(π) is as follows:

Where H(π _i ) is the entropy of each local strategy, representing the randomness of the strategy and quantifying the uncertainty in the system; N is the number of agents;

S203, in the strategy evaluation stage, train the Critic network parameter θ to reduce the Bellman residual:

J _Q (θ) is the objective function of the Critic network parameter θ, which trains the network parameters by minimizing this function; represents the expectation of the state-action pairs produced by the current policy, which is calculated under the distribution of the current state s _t and action a _t , D is the experience replay buffer, which stores the previous ones for training; Q(s _t , a _t ) represents the action value function, γ is the discount factor used to calculate the present value of future rewards, and its value is between 0 and 1; r(s _t , a _t ) is the immediate reward obtained by taking action a _t in state s _t ; V _θ is the value estimate of the value function network parameterized by parameter θ for the next state s _t+1 ; α represents the temperature parameter, which is the entropy regularization coefficient, which weighs the relationship between reward and entropy to encourage exploration.

The parameters of the Critic network are optimized using stochastic policy gradient as follows:

Where:

Among them, r is the instant reward value, φ _i is the strategy parameter of each agent,

S204, strategy formulation phase, the Actor network objective function is as follows:

In the formula, represents the optimal joint strategy, Q(s _t ,a _t ) represents the action value function, α represents the temperature parameter; π′ is the target strategy;

S205. The strategy of each agent is trained by minimizing the expected entropy of the actions produced by its actor network, as shown in the following formula:

The stochastic gradient descent method is used to update the policy parameters φ _i of each agent, and α is updated as follows:

Where H' is the target entropy, which is an equivalent vector composed of hyperparameters;

Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:

1. Reduce communication requirements and computational burden: The multi-agent deep reinforcement learning method of the present invention can be executed in a decentralized manner, significantly reducing the communication requirements between agent networks. Especially in complex power systems containing a large number of distributed energy resources, this advantage reduces the computational burden brought by centralized methods, thereby improving the overall efficiency and reliability of the system.

2. Improve grid voltage stability: By coordinating and controlling the reactive power set point of the photovoltaic inverter, the present invention effectively improves the voltage curve of the distribution network and enhances the voltage stability of the grid. This is particularly important for dealing with the instability and volatility of solar power generation.

3. Wide applicability and flexibility: The present invention does not rely on system modeling, which enables it to be flexibly applied to a variety of distribution network configurations without the need for detailed understanding of system topology or parameters. This increases the scope of application of the method, especially for distribution networks where it is difficult to obtain accurate system data.

4. Optimization of reinforcement learning algorithm: The developed MASAC algorithm has a strong exploration capability and can effectively find the best action plan for the intelligent agent. Compared with the traditional maximum entropy-based soft Q learning method, this invention avoids the potential complexity and instability problems and enhances the stability and reliability of the algorithm.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart of the method of the present invention.

FIG. 2 is a schematic diagram of training results of a MASAC algorithm provided by an embodiment.

FIG3 is a schematic diagram of test results of a MASAC algorithm provided by an embodiment.

FIG4 is a schematic diagram of reactive power voltage control robustness test results provided by an embodiment.

FIG5 is a schematic diagram showing the effect of reactive voltage control on all node voltages provided by an embodiment of the present invention.

DETAILED DESCRIPTION

In order to better understand the technical content of the present invention, specific embodiments are given and described as follows in conjunction with the accompanying drawings.

Various aspects of the invention are described herein with reference to the accompanying drawings, in which many illustrative embodiments are shown. The embodiments of the invention are not limited to those described in the accompanying drawings. It should be understood that the invention is implemented by any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed in the invention are not limited to any implementation. In addition, some aspects disclosed in the invention may be used alone or in any appropriate combination with other aspects disclosed in the invention.

As shown in FIG1 , the present invention provides a distributed photovoltaic voltage reactive power control method for a distribution network based on a MASAC algorithm, comprising the following steps:

S1. Construct a decentralized control framework for voltage and reactive power of distribution network with distributed photovoltaics, and transform the problem of decentralized control of voltage and reactive power of distribution network into a Markov game model.

The framework of decentralized control of voltage and reactive power in distribution network includes: minimizing the active power loss of distribution network over a period of time as the goal, using distributed photovoltaic inverters as decision variables, and using a preset voltage range as a constraint;

The Markov game model includes: state space: a set of local observation values consisting of the net injection of active/reactive power of the photovoltaic inverters included in each intelligent agent and the voltage amplitude of the photovoltaic inverter; action space: a set of actions consisting of the reactive output of the photovoltaic inverters controlled by all intelligent agents; reward function: constructed by the active loss cost and the voltage over-limit penalty; state transfer process: the state of the intelligent agent follows the power flow calculation constraints of the distribution network and is updated according to the state transfer probability distribution.

(1) Objective function: The goal of voltage and reactive power control is to minimize the active power loss of the distribution network within a period of time while ensuring that the node voltage does not exceed the limit. The objective function is as follows:

Wherein, T represents the optimization time period, and P _loss (t) represents the active network loss at time t.

(2) Decision variables: The voltage and reactive power decentralized control object of the distribution network is the distributed PV inverter. The decentralized control of the distribution network is achieved by adjusting the reactive power output Q _PV of the distributed PV inverter.

(3) Constraints: The most important factor in the decentralized control of voltage and reactive power in distribution networks is the node voltage constraint. During the control process, it is necessary to ensure that the node voltage is within the specified limit range V and Inside.

(4) State space: The state space S represents the state of all agents at time t, and si _,t represents the local observation value of agent i at time t, _si,t = ( _pi , _qi , _vi ), where _pi , _qi and _vi represent the net injection of active/reactive power and the node voltage amplitude of the local node where agent i is located, respectively, and S is the set of all node states.

(5) Action space: The action space A represents the set of actions of all agents at time t, and a _i,t represents the reactive output Q _PV,i,t of the PV inverter controlled by agent i at time t.

(6) Reward function: The reward function equation is used to measure the actions taken by the agent. All agents share the same reward function, which consists of two parts: active power loss cost and voltage limit penalty, as follows:

In the above formula, R(t) is the reward value at time t, P _loss (t) is the active power loss at time t, and the function is a 0-1 discriminant function. When the voltage V _k (t) of the k node satisfies the upper and lower limits V , When the function f is 0, otherwise it is 1. The upper and lower limits of the voltage amplitude are set to 1.05 and 0.95, _σ1 is the unit active power loss cost, and _σ2 is the voltage limit penalty factor.

(7) State transfer process: The state transfer process after the action of the intelligent agent is sent to the environment must strictly follow the power flow calculation constraints of the distribution network, including power balance constraints and power flow constraints. The present invention uses the PYPOWER power flow calculation tool to build the distribution network environment, and uses the runpf function to perform power flow calculation. This function will automatically meet the power balance constraints and power flow constraints when calculating the power flow. The state transfer probability distribution of the intelligent agent is represented by P(s′|s,a), which represents the probability that the environment will transfer from S _t to S′ _t under _the action of action a _{t after the intelligent agent takes action a t according} to the current state S t.

S2. Construct the MASAC algorithm to solve the Markov game model, and construct Actor and Critic neural networks for each agent. The Actor neural network determines the strategy of the agent, and the Critic neural network is used to determine the value of the strategy.

The neural network is trained in a centralized training manner to obtain a distributed photovoltaic voltage reactive control model for the distribution network. This model is then used to realize online control of the distribution network voltage and achieve decentralized regulation of the photovoltaic inverter.

In the present invention, the main innovation of multi-agent MASAC lies in its centralized training and decentralized execution process. In the training phase, the Critic network is centrally trained by using global information, while in the execution phase, each agent uses its local observations as individual inputs to formulate its own control strategy in a decentralized manner, that is, using a compressed Gaussian distribution function to generate continuous actions. In the method proposed in the present invention, the strategy is trained as a trade-off between maximizing entropy and expected returns. This helps to avoid the problem of premature convergence, which is necessary to achieve global optimality. The strategy of the actor network of each agent in the MASAC framework can be expressed as follows:

in, For each agent’s action at a specific time point t, Determined by the Actor network; i represents the index of the agent, and the state vector of agent i at time t is expressed as The strategy of each agent is denoted as It is a strategy based on compressed Gaussian distribution.

Each agent has its own independent policy, which is updated in each iteration to maximize the trade-off between expected reward and the entropy of the policy. The entropy of the policy represents the randomness of the policy and is a quantification of the uncertainty in the system. The entropy H(π) of the joint policy π(a _t |s _t ) can be expressed as follows:

Where H(π _i ) is the entropy of each local strategy, and N is the number of agents;

In the strategy evaluation phase, the Critic network parameters θ are trained to reduce the Bellman residual:

Where J _Q (θ) is the objective function of the Critic network parameter θ, which trains the network parameters by minimizing this function; represents the expectation of the state-action pairs produced by the current policy, which is calculated under the distribution of the current state s _t and action a _t . D is the experience replay buffer, which stores previous experience (state, action, reward, etc.) for training; Q(s _t , a _t ) represents the action value function, γ is the discount factor used to calculate the present value of future rewards, and its value is between 0 and 1; r(s _t , a _t ) is the immediate reward obtained by taking action a _t in state s _t ; V _θ is the value estimate of the next state s _t+1 by the value function network parameterized by parameter θ; α represents the temperature parameter, which is the entropy regularization coefficient, which weighs the relationship between reward and entropy to encourage exploration.

During the optimization process, the stochastic policy gradient is used to optimize the parameters of the Critic network:

Where:

In the formula, r is the instant reward value, φ _i is the strategy parameter of each agent,

In the strategy formulation phase, the Actor network goal of the MASAC algorithm can be expressed as:

Where π′ is the target strategy, represents the optimal joint strategy, Q(s _t ,a _t ) represents the action-value function, α represents the temperature parameter, and the strategy of each agent is parameterized as φ _i , which aims to reduce the expected entropy through training.

Specifically, each agent’s policy is trained with the objective of minimizing the expected entropy of actions produced by its actor network, as shown below:

The stochastic gradient descent method is used to update the policy parameters φ _i of each agent. Finally, α can be updated as follows:

Where H' is the target entropy, which is an equivalent vector composed of hyperparameters. Actor and Critic neural networks are trained for all agents, and the minimum value of the Q function is considered in the objective function to minimize the overestimation of the state value.

The MASAC method proposed in this paper aims to optimize the reactive power output of the agent to adjust the voltage of the distribution network node, and treats each PV as an agent for control. In the training phase, the action of each agent is provided to the centralized Critic network to calculate the reward and send it to the agent for strategy update. The offline training process is shown in the following table:

Step 4: Deploy the reinforcement learning agent trained in step 3, and use distributed execution to complete the online control of the distribution network voltage, thereby achieving decentralized control of each PV.

When the agents are properly trained, they only take local states as input and take actions without communicating with a centralized controller. The following table shows the decentralized distributed online execution process:

In order to evaluate the performance of the proposed voltage control framework, simulation experiments on a modified IEEE 34 bus test system were conducted. Twelve aggregated PV inverters were added at different nodes with a total power generation capacity of 1576kW. Since the maximum load demand was 1756kW, the maximum solar PV power generation accounted for about 90% of the total peak load. The performance of the decentralized agent was tested under different load and PV curves than the training dataset. In addition, the reactive power of the inverter was controlled to ensure that its operating power factor was not less than 0.9 recommended by the manufacturer. As a power flow solver, PYPOWER was used, connected with a Python interface as a learning and testing environment.

Each agent is trained for 500 episodes to learn the optimal control strategy to find the optimal behavior to deal with the voltage violation scenario. Both the Actor network and the Critic network are composed of a fully connected neural network, which consists of an input layer, an output layer, and a hidden layer. Its parameters are shown in the following table.

In the initial stage of training, individuals randomly explore the decision space of the environment, and eventually converge and provide the optimal action after a specific event occurs, as shown in Figure 2. After the training stage, each trained agent only needs its local state to provide the optimal action to solve the voltage regulation problem. Through these simulation experiments, the present invention verifies the effectiveness and adaptability of decentralized agents under different load and photovoltaic curve conditions.

Figure 3 depicts the voltage fluctuation at a node of the test system under the training model and the base case. It can be observed that in the base case scenario, there is a voltage violation according to the voltage standard limit, while there is no voltage violation under the control of the proposed MASAC algorithm. In addition, in the base case, the voltage variation is larger than that of the proposed training model.

Figure 4 shows the voltage fluctuations of the tested nodes under the trained model and the basic scenario. The results show that the proposed method has a better performance. Finally, Figure 5 describes the voltage changes of all 34 nodes at the 20th minute. The results show that the proposed trained model has a better voltage distribution in terms of voltage changes and violations than the basic case.

Although the present invention has been described above with preferred embodiments, it is not intended to limit the present invention. A person skilled in the art of the present invention may make various modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the definition of the claims.

Claims (7)

1. A power distribution network distributed photovoltaic voltage reactive power control method based on MASAC algorithm is characterized by comprising the following steps:

S1, constructing a voltage reactive power decentralization control framework of a power distribution network of a record and distributed photovoltaic, and converting the voltage reactive power decentralization control problem of the power distribution network into a Markov game model;

the reactive power decentralization control frame of distribution network voltage includes: the method comprises the steps of taking active power loss of a power distribution network in a period of time as a target, taking a distributed photovoltaic inverter as a decision variable and taking a preset voltage range as a constraint condition;

The Markov game model comprises: state space: and the intelligent agent comprises a local observation value set formed by the net injection quantity of active/reactive power of the photovoltaic inverter and the voltage amplitude of the photovoltaic inverter, and an action space: an action set formed by reactive output quantity of all the photovoltaic inverters controlled by the intelligent agent; bonus function: constructed from active loss cost, and voltage out-of-limit penalty; state transition process: the state of the intelligent agent follows the load flow calculation constraint of the power distribution network and is updated according to the state transition probability distribution;

S2, constructing MASAC algorithm to solve a Markov game model, constructing an Actor and a Critic neural network for each agent, wherein the Actor neural network determines the strategy of the agent, and the Critic neural network is used for judging the value of the strategy;

training is carried out by adopting a neural network in a centralized training mode to obtain a distributed photovoltaic voltage reactive control model of the power distribution network, then the model is utilized to realize the on-line control of the power distribution network voltage, the on-line control of the power distribution network voltage is completed, and the decentralization regulation and control of the photovoltaic inverter are realized.

2. The distributed photovoltaic voltage reactive power control method of a power distribution network based on MASAC algorithm as claimed in claim 1, wherein in step S1, the objective of minimizing active power loss of the power distribution network in a period of time is to construct an objective function as follows:

Where T represents the optimization time period and P _loss (T) represents the active network loss at time T.

3. The distributed photovoltaic voltage reactive power control method of a power distribution network based on MASAC algorithm as claimed in claim 1, wherein in step S1, the constraint condition is:

Wherein V _k (t) is the voltage of the photovoltaic inverter k at time t, V and Respectively a preset lower voltage limit and an upper voltage limit.

4. The method for distributed photovoltaic voltage reactive power control of a power distribution network based on MASAC algorithm as claimed in claim 1, wherein in step S1, the state space S is a set of local observations S _i,t of all photovoltaic inverters at time t, S _i,t is a local observation of an agent i at time t, and S _i,t＝(p_i,q_i,v_i),p_i,q_i and v _i respectively represent the net injection amount and the node voltage amplitude of active/reactive power of the photovoltaic inverter in which the agent i is located;

The action space a is a set of actions a _i,t composed of the reactive output of the photovoltaic inverter controlled by all agents at time t, and Q _PV,i,t is the reactive output of the PV inverter controlled by agent i at time t.

5. The distributed photovoltaic voltage reactive power control method of a power distribution network based on MASAC algorithm as claimed in claim 1, wherein in step S1, the reward function is as follows:

Wherein R (t) is a reward value at time t, P _loss (t) is an active loss at time t, a function Is a 0-1 discriminant function, when the voltage V _k (t) of the k node meets the upper limit V and the lower limit V,The time function f is 0, otherwise is 1, sigma ₁ is the unit active loss cost, and sigma ₂ is the voltage out-of-limit penalty factor.

6. The method for reactive power control of power distribution network distributed photovoltaic voltage based on MASAC algorithm according to claim 1, wherein in step S1, the state transition process uses PYPOWER power flow calculation tool to build the environment of the power distribution network, and runpf function is used to perform power flow calculation, and the power flow calculation constraint comprises power balance constraint and power flow constraint; the state transition probability distribution of the agent is P (S '|s, a), which indicates the probability that the environment transitions from S _t to S' _t under the action of action a _t after the agent takes action a _t according to the current state S _t.

7. A distributed photovoltaic voltage reactive control method of a power distribution network based on MASAC algorithm as claimed in claim 1, wherein step S2 comprises the following sub-steps:

S201, constructing an Actor network of each intelligent agent based on the Actor network, wherein the strategy of the Actor network of each intelligent agent is as follows:

wherein, For each action taken by the agent at a particular point in time t,Determining by an Actor network; i represents the index of agent, and the state vector of agent i at time t is expressed asThe policy of each agent is noted asIs a strategy based on compressed gaussian distribution;

S202, each agent combines entropy H (pi) of a strategy pi (a _t|s_t) based on the entropy iteration update of the maximum expected return and the strategy, and the following formula is shown in the specification:

Wherein H (pi _i) is entropy of each local strategy, represents randomness of the strategy and is the quantification of uncertainty in the system; n is the number of the intelligent agents;

S203, training Critic network parameters theta in a strategy evaluation stage, and reducing Bellman residual errors:

j _Q (θ) is an objective function of Critic network parameter θ that trains network parameters by minimizing the function; representing the desire for state-action pairs generated by the current strategy, calculated under the distribution of current state s _t and action a _t, D is an empirical playback buffer that stores previous training; q (s _t,a_t) represents an action value function, gamma is a discount factor, and is used for calculating the present value of future rewards, and the value of the present value is between 0 and 1; r (s _t,a_t) is the immediate prize obtained by taking action a _t in state s _t; v _θ is the value estimate of the next state s _t+1 by the network of cost functions parameterized by the parameter θ; alpha represents that the temperature parameter is an entropy regularization coefficient that balances the relationship between rewards and entropy to encourage exploration;

parameters of the Critic network are optimized by using random strategy gradients, and the following formula is adopted:

Wherein:

Wherein r is an instant rewarding value, phi _i is a policy parameter of each agent,

S204, in a strategy making stage, an Actor network objective function is expressed as follows:

In the method, in the process of the invention, Representing the optimal combination strategy, Q (s _t,a_t) representing an action value function, and alpha representing a temperature parameter; pi' is the target policy;

S205, the policy of each agent is trained by minimizing the expected entropy of actions generated by its actor network, as shown in the following formula:

And updating the strategy parameters phi _i of each agent by adopting a random gradient descent method, wherein alpha is updated as follows:

Wherein H' is a target entropy, and the target entropy is an equivalent vector consisting of super parameters;

s206, training an Actor and Critic neural network for all agents, and taking the minimum value of the Q function in the objective function.