CN118886689A - A safe power dispatching method based on deep reinforcement learning - Google Patents

Abstract

The invention discloses a safe power dispatching method based on deep reinforcement learning, which belongs to the field of power dispatching and comprises the following steps: s1, training a scheduling model according to countermeasure learning by adopting a Markov decision process; s2, training a main expert model based on an expert rule base interactive tuning mode; and S3, outputting a final power dispatching scheme according to the main expert model optimization dispatching model. Based on the Rainbow DQN algorithm, two novel training modes are provided, namely, safety constraint guarantee learning is carried out by training an countermeasure model and a scheduling model in a manner based on countermeasure learning, and a main expert model is obtained by training in a manner based on expert rule base interactive optimization. Through a large number of experiments performed on the virtual simulation power grid environment, the effectiveness of the safety power dispatching algorithm based on the deep reinforcement learning is verified, a certain degree of disturbance and attack can be resisted, and the high-efficiency and robust power dispatching capability of the power system is improved.

Description

A safe power dispatching method based on deep reinforcement learning

Technical Field

The present invention belongs to the field of electric power dispatching, and specifically relates to a safe electric power dispatching method based on deep reinforcement learning.

Background Art

Power dispatching is an effective management method that aims to ensure the safe and stable operation of the power grid, reliable power supply, and the orderly progress of various types of power production work. The specific work content of power dispatching includes judging the safety and economic operation status of the power grid based on the data information fed back by various information collection devices or the information provided by monitoring personnel, combined with the actual operation parameters of the power grid (such as voltage, current, frequency, load, etc.), comprehensively considering the development of various production work. The dispatching center issues operating instructions through telephone or automatic system, directing on-site operators or automatic control systems to make adjustments, such as adjusting generator output, load distribution, and switching capacitors and reactors, so as to ensure the continuous safe and stable operation of the power grid.

The rapid development of new energy and the continuous growth of peak load have brought new challenges to the dispatching capacity of the power generation side. New energy refers to renewable energy such as wind power and photovoltaic power, which can be obtained through natural capabilities and has the characteristics of intermittent and randomness. Not every region has the ability to build a wind power plant or a photovoltaic plant, so it is necessary to interconnect the power grid quickly and on a large scale. With the increase of various flexible devices or electronic devices, the regulation scale of the entire power system has become larger, and the uncertainty and volatility have also increased. The instantaneous safety constraints at every moment need to be met in the grid dispatching. Once a constraint is not met, the entire power grid may collapse, causing large-scale power outages and huge economic losses. Therefore, the entire power system needs to have efficient and robust dispatching capabilities.

In recent years, deep reinforcement learning algorithms have been widely used in the field of power dispatching. Power dispatching through deep reinforcement learning is a research direction with great application value and prospects. Reinforcement learning has been applied to economic dispatch problems involving virtual power plants, facilitating the rapid decomposition of dispatch instructions for virtual power plants. This technology can also be applied to the self-dispatching of wind farms and energy storage systems to achieve faster dispatching decisions.

Summary of the invention

In view of the above-mentioned deficiencies in the prior art, the safe power dispatching method based on deep reinforcement learning provided by the present invention solves the problems of poor efficiency and robustness of the existing power dispatching methods.

In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a safe power dispatching method based on deep reinforcement learning, comprising the following steps:

S1, train the scheduling model based on adversarial learning using Markov decision process;

S2, training the main expert model based on interactive tuning of the expert rule base;

S3. Output the final power dispatching plan based on the dispatching model optimized by the main expert model.

Further: S1 specifically includes:

The action space, state space and reward function are defined through the Markov decision process, the behavior of the scheduling model and the adversarial model in the virtual simulation power grid environment is simulated, the network parameters of the scheduling model and the adversarial model are updated through the Rainbow DQN algorithm, and the scheduling model is trained through adversarial learning;

The strategy defined by the Markov decision process is as follows: based on the observed data in the virtual simulation power grid environment, a power dispatching plan is generated through a dispatching model or an adversarial model;

The observation space defined by the Markov decision process includes the grid information of node voltage value, line current value, generator output power and load power demand, and the state space The specific expression is:

In the formula, represents the voltage value of all nodes, Indicates the current value of all lines, represents the output power of all generators, Indicates the power demand of all loads;

The action space defined by the Markov decision process includes all permutations and combinations of power dispatch actions. The specific expression is:

In the formula, Indicates the adjustment value of the generator output power, Indicates the switch operation status, Indicates load adjustment;

The reward function defined by the Markov decision process is specifically based on the goal of power grid operation, which includes power supply reliability, economic benefits and robustness. The expression of the reward function R _t is specifically:

In the formula, Indicates power supply reliability, used to measure voltage stability, frequency deviation, represents the voltage of the ith node, represents the rated voltage of each node in the power grid, N represents the total number of nodes in the power grid, represents the current of the jth line, represents the maximum allowable current of the jth line, M represents the total number of lines in the power grid, Represents economic benefits, used to measure the cost of power generation, is the cost coefficient of the kth generator, G represents the total number of generators, represents the adjustment value of the output power of the kth generator, Represents environmental benefits and is used to measure pollutant emissions. is the emission coefficient of the kth generator, Represents robustness, which is used to measure the ability to resist failure. is the impact assessment value of the lth fault, F represents the total number of faults, represents the weight of the first indicator, represents the weight of the second indicator, represents the weight of the third indicator, Indicates the weight of the fourth indicator.

Further: The expression for updating the network parameters of the scheduling model and the adversarial model through the Rainbow DQN algorithm is specifically:

In the formula, For having Atomic vector The projection on represents the number of atoms on the discrete support, is the target distribution, is the KL divergence, is the probability of time difference error, To adjust right The control parameters of the degree of influence.

Further: In S1, the security cost of the Markov decision process according to Status settings;

Security costs The specific expression is:

In the formula, represents the first weight coefficient, represents the second weight coefficient, represents the third weight coefficient, Indicates the voltage safety cost, which is used to reflect the degree to which the voltage deviates from the rated value. Indicates the allowable voltage deviation range, Indicates the current safety cost, which is used to reflect whether the line current exceeds the limit of safe operation. Represents the frequency security cost, which is used to measure the frequency deviation of the power grid. Indicates the current grid frequency. Indicates the rated frequency of the power grid, Indicates the allowable frequency deviation range, Represents the maximum value function.

The beneficial effect of the above further scheme is: according to the characteristics of the power system, the scheduling model is trained by the Rainbow DQN algorithm using adversarial learning and expert rule base interactive tuning to further improve the optimization effect of the power scheduling scheme.

Furthermore, the objective function of adversarial learning training for the scheduling model is specifically as follows:

In the formula, Indicates the current state. represents the actions of the scheduling model, represents disturbance, It indicates the state to which the system transfers after executing the scheduling and disturbance. represents the strategy of the scheduling model, represents the strategy of adversarial model, and All are obtained through the state space, Acquired through action space;

Indicates that the adversarial model tries to find the most unfavorable perturbation , It means that the scheduling model learns how to adjust the scheduling strategy under the most unfavorable scenario generated by the adversarial model. Represents the average of all possible state transitions.

Further: S2 comprises the following sub-steps:

S21, setting a master expert model and a slave expert model, the master expert model includes basic knowledge and preliminary optimization capabilities of power dispatching, and the slave expert model is generated based on a copy of the master expert model;

S22, initializing the master expert model and the slave expert model;

S23. Optimize the output of the slave expert model through the expert rule base, and then train the master expert model.

The beneficial effect of the above further scheme is: the present invention adopts an interactive tuning method based on the expert rule base constructed by power experts to train the main expert model, each slave expert model corresponds to a power expert, and the main expert model is used for unified updating and management, further improving the optimization effect of the power dispatching scheme.

Further: S23 includes the following sub-steps:

S231, outputting an initial power dispatching plan through the dispatching model, inputting the initial power dispatching plan into the expert model, and obtaining an adjusted power dispatching plan;

S232, performing fault business analysis on the adjusted power dispatching plan through an expert rule base, and generating feedback results of the adjusted power dispatching plan;

S233, sending the feedback result to the slave expert model through the virtual simulation power grid environment, and updating the model parameters of the slave expert model according to the feedback result;

S234. Feedback the model parameters updated from the expert model to the main expert model according to a preset period to train the main expert model.

The beneficial effects of the above further scheme are: using the expert rule base to interactively tune and adjust the master expert model to make it more consistent with the professional judgment of power experts. Using environmental feedback to adjust the slave expert model to make it more consistent with the actual situation of power grid operation. Regularly feeding back the updates of the slave expert model to the master expert model to ensure that the master expert model can gradually absorb the knowledge and experience of various power experts.

Further: In S234, the expression of the loss function Loss for training the main expert model is specifically:

In the formula, is the expert confidence score of the expert rule base for the i-th adjusted power dispatching plan, is the predicted score of the ith adjusted power dispatch plan from the expert model, and N is the number of dispatch plans.

Further: S3 includes the following sub-steps:

S31, inputting the observed values in the virtual simulation power grid environment into the dispatching model to generate a power dispatching plan, wherein the observed values include node voltage, line current, generator output power and load demand;

S32, sending the power dispatching plan to the main expert model to generate an optimized power dispatching plan;

S33, performing fault business analysis on the optimized power dispatching plan through an expert rule base, and generating feedback results of the optimized power dispatching plan;

S34. Determine whether the optimized power dispatching plan is qualified according to the feedback results of the optimized power dispatching plan. If so, use the optimized power dispatching plan as the final power dispatching plan; if not, optimize the main expert model according to the feedback results of the optimized power dispatching plan and return to S32.

The beneficial effects of the present invention are:

(1) The present invention provides a safe power dispatching method based on deep reinforcement learning. By learning the knowledge related to power dispatching and combining the characteristics and disadvantages of deep reinforcement learning, the present invention can effectively deal with various safety issues in power dispatching. Based on the Rainbow DQN algorithm, the present invention proposes two novel training methods, namely, training the adversarial model and the dispatching model based on adversarial learning to perform safety constraint assurance learning, and training the main expert model based on the interactive tuning of the expert rule base. The present invention proposes a safe power dispatching algorithm based on deep reinforcement learning. The safety cost of the Markov decision process Taking voltage safety cost, current safety cost and frequency safety cost into consideration improves the safety of power grid operation. Theoretical analysis shows that this method has potential advantages in terms of safety and dispatch performance.

(2) The purpose of this invention is to expand the problem into safe power dispatching, including load management, fault detection and isolation, emergency plans, etc., based on various safety standards and rules in power dispatching and in combination with the characteristics of the power grid system, and finally establish a safe power dispatching algorithm based on deep reinforcement learning. It is used to balance supply and demand and reduce the pressure of peak load on the system through demand response, load transfer and peak-to-valley filling during peak power consumption periods; it is also used to quickly detect and isolate fault areas, prevent fault expansion, and reduce the impact on the power system; it is also used to formulate emergency plans to ensure rapid response and power restoration in the event of an emergency.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a safe power dispatching method based on deep reinforcement learning according to the present invention.

FIG2 is a schematic diagram of adversarial learning training based on a Markov decision process.

FIG3 is a schematic diagram of the master-slave expert model training.

FIG4 is a flow chart of power dispatching plan generation and adjustment.

DETAILED DESCRIPTION

The specific implementation modes of the present invention are described below to facilitate those skilled in the art to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

As shown in FIG1 , in one embodiment of the present invention, a safe power dispatching method based on deep reinforcement learning includes the following steps:

S1, train the scheduling model based on adversarial learning using Markov decision process;

S2, training the main expert model based on interactive tuning of the expert rule base;

S3. Output the final power dispatching plan based on the dispatching model optimized by the main expert model.

The S1 is specifically:

The action space, state space and reward function are defined through the Markov decision process, the behavior of the scheduling model and the adversarial model in the virtual simulation power grid environment is simulated, the network parameters of the scheduling model and the adversarial model are updated through the Rainbow DQN algorithm, and the scheduling model is trained through adversarial learning;

The strategy defined by the Markov decision process is as follows: based on the observed data in the virtual simulation power grid environment, a power dispatching plan is generated through a dispatching model or an adversarial model;

The observation space defined by the Markov decision process includes the grid information of node voltage value, line current value, generator output power and load power demand, and the state space The specific expression is:

In the formula, represents the voltage value of all nodes, Indicates the current value of all lines, represents the output power of all generators, Indicates the power demand of all loads;

The action space defined by the Markov decision process includes all permutations and combinations of power dispatch actions. The specific expression is:

In the formula, Indicates the adjustment value of the generator output power, Indicates switch operation status, such as line on/off, Indicates load adjustment;

The reward function defined by the Markov decision process is specifically based on the goal of power grid operation, which includes power supply reliability, economic benefits and robustness. The expression of the reward function R _t is specifically:

In the formula, Indicates power supply reliability, used to measure voltage stability, frequency deviation, represents the voltage of the ith node, It represents the rated voltage of each node in the power grid. It is a fixed value and represents the normal operating voltage of the power grid. N represents the total number of nodes in the power grid. represents the current of the jth line, represents the maximum allowable current of the jth line, M represents the total number of lines in the power grid, Represents economic benefits, used to measure the cost of power generation, is the cost coefficient of the kth generator, G represents the total number of generators, represents the adjustment value of the output power of the kth generator, Represents environmental benefits and is used to measure pollutant emissions. is the emission coefficient of the kth generator, Represents robustness, which is used to measure the ability to resist failure. is the impact assessment value of the lth fault, F represents the total number of faults, represents the weight of the first indicator, represents the weight of the second indicator, represents the weight of the third indicator, Indicates the weight of the fourth indicator. The indicator weight can be adjusted according to actual needs.

As shown in FIG2 , in this embodiment, the task of training the scheduling model and the adversarial model is modeled as a reinforcement learning problem, namely, a Markov decision process. The key components of reinforcement learning, such as the strategy, action space, observation space, and reward function defined by the Markov decision process, are as follows: The strategy refers to the power scheduling plan generated by the scheduling model or the adversarial model based on the observation of the model in the virtual simulated power grid environment; To counter the strategy, is the scheduling strategy, is the action output by the adversarial strategy, The action output by the scheduling strategy, The current state, action, reward, and next state of the interaction between the adversarial strategy and the environment. , , , ) is the current state, action, reward, and next state of the interaction between the dispatching strategy and the environment; the action space contains all possible combinations of power dispatching actions; the observation space includes grid information such as node voltage, line current, generator output power, load demand, etc.; the reward function is based on the goals of grid operation, such as power supply reliability: ensuring that the grid does not fail during the power supply process, such as overload, undervoltage, etc.; economic benefits: minimizing power generation costs and optimizing the output of generator sets; environmental benefits: minimizing pollution emissions; robustness: being able to withstand a certain degree of disturbance and attack and maintain normal operation.

The expression for updating the network parameters of the scheduling model and the adversarial model through the Rainbow DQN algorithm is as follows:

In the formula, For having Atomic vector The projection on represents the number of atoms on the discrete support, is the target distribution, is the KL divergence, is the probability of time difference error, To adjust right In this embodiment, atoms are discrete points used to represent the distribution of rewards, through which the algorithm can approximate the distribution of rewards, rather than just the expected rewards.

In this embodiment, the Rainbow DQN algorithm is used to train the scheduling model according to the characteristics of the power system by using adversarial learning and interactive tuning of the expert rule base to further improve the optimization effect of the power scheduling scheme. The Rainbow DQN algorithm combines six independent improvements to the DQN (Deep Q Learning) algorithm by the reinforcement learning community and achieves good results.

In S1, the security cost of the Markov decision process in S1 according to Status settings;

Security costs The specific expression is:

In the formula, represents the first weight coefficient, represents the second weight coefficient, It represents the third weight coefficient, which reflects the contribution of each safety indicator to the total safety cost. It represents the voltage safety cost, which is used to reflect the degree to which the voltage deviates from the rated value and has a direct impact on the stability of power grid operation. Indicates the allowable voltage deviation range, Indicates the current safety cost, which is used to reflect whether the line current exceeds the limit of safe operation. Excessive current may cause line damage. Indicates the frequency safety cost, which is used to measure the frequency deviation of the power grid. Frequency instability will affect the normal operation of the equipment. Indicates the current grid frequency. Indicates the rated frequency of the power grid, Indicates the allowable frequency deviation range, Represents the maximum value function.

In this embodiment, the security cost It is related to the St+1 state. For example, if a transmission line is suddenly interrupted at time t+1, the cost will be very high. Therefore, the worst-case training can be performed in advance. Adversarial learning is usually used to learn the safety constraints for the worst-case of N-1 line interruption. Through this adversarial training, the dispatching model can learn how to optimize the dispatching strategy in various worst-case scenarios, so as to master the safety measures for extreme situations such as N-1 line interruption in advance. The ultimate goal is to enable power dispatching to maintain safe operation and meet the safety requirements of the power grid in the most unfavorable conditions. The objective function of adversarial learning training for the dispatching model and the adversarial model can be expressed as the following Minimax optimization problem:

In the formula, is the current state; To schedule the actions of the model; Actions against the model, i.e. generated perturbations, such as line interruptions; The state that the system transfers to after executing the scheduling and disturbance; The strategy for scheduling the model; To combat the model, and All are obtained through the state space, Obtained through action space.

Maximize the section Indicates that the adversarial model tries to find the most unfavorable perturbation , for example, simulating N-1 outage scenarios to find the line outage combination that has the greatest impact on grid security. This is partly to foresee the worst-case scenario in advance.

Minimize the part It means that the dispatch model learns how to adjust the dispatch strategy under the most unfavorable scenario generated by the adversarial model to minimize the safety cost of the system, that is, to ensure the safe operation of the power grid under the most unfavorable circumstances.

expect It represents the average of all possible state transitions, comprehensively considering different perturbations and scheduling combinations to evaluate the overall performance.

The S2 comprises the following sub-steps:

S21, setting a master expert model and a slave expert model, the master expert model includes basic knowledge and preliminary optimization capabilities of power dispatching, and the slave expert model is generated based on a copy of the master expert model;

S22, initializing the master expert model and the slave expert model. In this embodiment, the scheduling model is used to initialize the expert model to ensure that the expert model and the scheduling model have the same basic knowledge, which reduces problems that may be caused by knowledge mismatch, such as hallucinations and inconsistencies.

S23. Optimize the output of the slave expert model through the expert rule base, and then train the master expert model.

As shown in FIG3 , in this embodiment, the present invention trains the master expert model by interactive tuning based on the expert rule base constructed by power experts. Each slave expert model corresponds to a power expert. The master expert model is used for unified updating and management to further improve the optimization effect of the power dispatching scheme.

The S23 comprises the following sub-steps:

S231, outputting an initial power dispatching plan through the dispatching model, inputting the initial power dispatching plan into the expert model, and obtaining an adjusted power dispatching plan;

S232, performing fault business analysis on the adjusted power dispatching plan through an expert rule base, and generating feedback results of the adjusted power dispatching plan;

Among them, the expert rule base is set up by power experts, which includes various characteristic information of the power grid, such as: the actual operating parameters, operating environment, warning information and faulty equipment information of the power grid when a power grid fault occurs. The expert rule base is used to perform fault business analysis on the power dispatching plan, and judge the abnormal power grid operating parameters, abnormal environment, abnormal warning information and abnormal equipment when the fault occurs. When any abnormal information causes a power grid equipment failure, it is predicted whether the power dispatching plan will cause a power grid failure.

S233, sending the feedback result to the slave expert model through the virtual simulation power grid environment, and updating the model parameters of the slave expert model according to the feedback result;

S234. Feedback the model parameters updated from the expert model to the main expert model according to a preset period to train the main expert model.

In this embodiment, the master expert model is interactively tuned using the expert rule base to make it more consistent with the professional judgment of power experts. The slave expert model is adjusted using environmental feedback to make it more consistent with the actual situation of power grid operation. The updates of the slave expert model are regularly fed back to the master expert model to ensure that the master expert model can gradually absorb the knowledge and experience of various power experts.

In S234, the expression of the loss function Loss for training the main expert model is specifically:

In the formula, is the expert confidence score of the expert rule base for the i-th adjusted power dispatching plan, is the predicted score of the ith adjusted power dispatch plan from the expert model, and N is the number of dispatch plans.

In this embodiment, the training goal of the expert model is to maximize the consistency between the confidence selected by the expert rule base and the prediction of the expert model, which is achieved by minimizing the relevant loss function.

The S3 comprises the following sub-steps:

S31, inputting the observed values in the virtual simulation power grid environment into the dispatching model to generate a power dispatching plan, wherein the observed values include node voltage, line current, generator output power and load demand;

S32, sending the power dispatching plan to the main expert model to generate an optimized power dispatching plan;

S33, performing fault business analysis on the optimized power dispatching plan through an expert rule base, and generating feedback results of the optimized power dispatching plan;

S34. Determine whether the optimized power dispatching plan is qualified according to the feedback results of the optimized power dispatching plan. If so, use the optimized power dispatching plan as the final power dispatching plan; if not, optimize the main expert model according to the feedback results of the optimized power dispatching plan and return to S32.

In this embodiment, the power dispatching scheme outputted by the expert model optimization dispatching model is adopted, and the optimized power dispatching scheme is subjected to fault business analysis through the expert rule base. If qualified, the scheme becomes the final power dispatching scheme. If unqualified, the expert model is further trained and optimized according to the feedback results to improve its performance in future tasks.

The beneficial effects of the present invention are as follows: the present invention provides a safe power dispatching method based on deep reinforcement learning, which effectively addresses various safety issues in power dispatching by learning relevant knowledge about power dispatching and combining the characteristics and disadvantages of deep reinforcement learning. The present invention is based on the Rainbow DQN algorithm and proposes two novel training methods, namely, training the adversarial model and the dispatching model based on adversarial learning to perform safety constraint assurance learning, and training the main expert model based on the interactive tuning of the expert rule base. The present invention proposes a safe power dispatching algorithm based on deep reinforcement learning, and the safety cost of the Markov decision process Taking voltage safety cost, current safety cost and frequency safety cost into consideration improves the safety of power grid operation. Theoretical analysis shows that this method has potential advantages in terms of safety and dispatch performance.

The purpose of this invention is to expand the problem into safe power dispatching, including load management, fault detection and isolation, emergency plans, etc., based on various safety standards and rules in power dispatching, combined with the characteristics of the power grid system, and finally establish a safe power dispatching algorithm based on deep reinforcement learning. It is used to balance supply and demand and reduce the pressure of peak load on the system through demand response, load transfer and peak-to-valley filling during peak power consumption; it is also used to quickly detect and isolate fault areas, prevent fault expansion, and reduce the impact on the power system; it is also used to formulate emergency plans to ensure rapid response and power restoration in the event of an emergency.

In the description of the present invention, it is necessary to understand that the orientation or positional relationship indicated by the terms "center", "thickness", "upper", "lower", "horizontal", "top", "bottom", "inner", "outer", "radial", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", and "third" are used only for descriptive purposes, and cannot be understood as indicating or implying the relative importance or the number of implicitly specified technical features. Therefore, the features defined by "first", "second", and "third" may explicitly or implicitly include one or more of the features.

Claims (9)

1. The safe power scheduling method based on deep reinforcement learning is characterized by comprising the following steps of:

S1, training a scheduling model according to countermeasure learning by adopting a Markov decision process;

s2, training a main expert model based on an expert rule base interactive tuning mode;

And S3, outputting a final power dispatching scheme according to the main expert model optimization dispatching model.

2. The safe power scheduling method based on deep reinforcement learning according to claim 1, wherein S1 specifically is:

Defining action space, state space and rewarding function through Markov decision process, simulating the behavior of the scheduling model and the countermeasure model in virtual simulation power grid environment, updating network parameters of the scheduling model and the countermeasure model through Rainbow DQN algorithm, and performing countermeasure learning training on the scheduling model;

The strategy defined by the Markov decision process is specifically: generating a power dispatching scheme through a dispatching model or an countermeasure model based on observed data in the virtual simulation power grid environment;

The observation space defined by the Markov decision process comprises the voltage value of the node, the current value of the line, the output power of the generator and the grid information of the power demand of the load, and the state space The expression of (2) is specifically:

in the formula, Representing the voltage values of all the nodes,Representing the current values of all the lines,Representing the output power of all of the generators,Representing the power requirements of all loads;

the action space defined by the Markov decision process comprises a permutation and combination of all power dispatching actions, and the action space The expression of (2) is specifically:

in the formula, Representing an adjustment value of the output power of the generator,Indicating the state of operation of the switch,Indicating load adjustment;

The markov decision process defines a reward function, specifically a grid-based goal, which includes power reliability, economic benefit, and robustness, and the expression of the reward function R _t is specifically:

in the formula, Represents the power supply reliability, is used for measuring voltage stability and frequency deviation,Representing the voltage at the i-th node,Representing the rated voltage of each node in the power grid, N representing the total number of nodes in the power grid,Representing the current of the j-th line,Represents the maximum allowable current of the jth line, M represents the total number of lines in the grid,Represents economic benefit, is used for measuring the power generation cost,And G represents the total number of generators,Represents an adjustment value of the output power of the kth generator,Represents environmental benefit, is used for measuring pollutant emission,Is the emission coefficient of the kth generator,Robustness is represented, which is used to measure the resistance to faults,For the impact evaluation value of the first failure, F represents the total number of failures,A first index weight is represented as a first index weight,A second index weight is represented as a function of the second index weight,A third index weight is represented as a function of the first index weight,Representing a fourth index weight.

3. The safe power scheduling method based on deep reinforcement learning according to claim 2, wherein the expression for updating the network parameters of the scheduling model and the countermeasure model by the Rainbow DQN algorithm is specifically:

in the formula, To have the following characteristicsVector of atomsThe projection onto the surface of the lens,Representing the number of atoms on the discrete support,In order to achieve a distribution of the objects,For the KL divergence, the average value of the power supply is calculated,As a probability of a time difference error,To adjustFor a pair ofA control parameter of the degree of influence of (a).

4. A method of secure power scheduling based on deep reinforcement learning according to claim 3, wherein in S1 the security cost of the markov decision processAccording toSetting a state;

safety cost The expression of (2) is specifically:

in the formula, A first weight coefficient is represented and a second weight coefficient is represented,A second weight coefficient is represented and is used to represent,A third weight coefficient is represented and is used to represent,Representing a voltage safety cost, for reflecting the extent to which the voltage deviates from the nominal value,Indicating the range of voltage deviations allowed,Represents a current safety cost, for reflecting whether the line current exceeds the limit of safe operation,Represents the frequency safety cost, is used for measuring the frequency deviation condition of the power grid,Representing the current grid frequency,Representing the nominal frequency of the electrical network,Indicating the range of allowed frequency deviations,Representing the maximum function.

5. The deep reinforcement learning-based safe power scheduling method according to claim 4, wherein the objective function for performing the countermeasure learning training on the scheduling model is specifically the following formula:

in the formula, The current state is indicated and the current state is indicated,The actions of the scheduling model are represented,The disturbance is indicated as such,Indicating the state to which the system transitions after performing scheduling and perturbation,The policy of the scheduling model is represented,The strategy of the challenge model is represented,AndAre all acquired through the state space,Acquiring through an action space;

6. The safe power scheduling method based on deep reinforcement learning according to claim 1, wherein S2 comprises the following sub-steps:

S21, setting a master expert model and a slave expert model, wherein the master expert model comprises basic knowledge and preliminary optimization capability of power dispatching, and the slave expert model is generated according to a copy of the master expert model;

s22, initializing a master expert model and a slave expert model;

S23, optimizing output of the slave expert model through an expert rule base, and further training the master expert model.

7. The deep reinforcement learning-based safe power scheduling method according to claim 6, wherein S23 comprises the following sub-steps:

S231, outputting an initial power dispatching scheme through a dispatching model, and inputting the initial power dispatching scheme into a slave expert model to obtain an adjusted power dispatching scheme;

S232, performing fault service analysis on the adjusted power dispatching scheme through an expert rule base, and generating a feedback result of the adjusted power dispatching scheme;

S233, sending a feedback result to the slave expert model through the virtual simulation power grid environment, and updating model parameters of the slave expert model according to the feedback result;

s234, feeding back model parameters updated from the expert model to the main expert model according to a preset period, and training the main expert model.

8. The safe power scheduling method based on deep reinforcement learning according to claim 7, wherein in S234, the expression of the Loss function Loss for training the main expert model is specifically:

in the formula, Scoring expert certainty of the ith adjusted power scheduling scheme for an expert rule base,For the predictive scoring of the ith adjusted power schedule from the expert model, N is the number of schedule.

9. The deep reinforcement learning-based safe power scheduling method according to claim 7, wherein S3 comprises the following sub-steps:

s31, inputting an observed value in a virtual simulation power grid environment into a scheduling model to generate a power scheduling scheme, wherein the observed value comprises node voltage, line current, generator output power and load demand;

S32, sending the power dispatching scheme to a main expert model, and generating an optimized power dispatching scheme;

s33, performing fault service analysis on the optimized power dispatching scheme through an expert rule base, and generating a feedback result of the optimized power dispatching scheme;

S34, judging whether the optimized power dispatching scheme is qualified according to a feedback result of the optimized power dispatching scheme, and if so, taking the optimized power dispatching scheme as a final power dispatching scheme; if not, the main expert model is optimized according to the feedback result of the optimized power dispatching scheme, and the S32 is returned.