CN117318169A - Active distribution network dispatching method based on deep reinforcement learning and new energy consumption - Google Patents

Abstract

The present invention provides an active distribution network dispatching method that takes into account new energy consumption based on deep reinforcement learning, and belongs to the technical field of power system energy dispatching. Including: establishing a simulation model based on the regional distribution network structure, obtaining historical photovoltaic output data, wind power output historical data, village power load historical data and factory power load historical data in the regional distribution network structure, applying VMD data preprocessing technology, PCC correlation analysis algorithm and BiLSTM model predict the next 24 hours of data as a training data set; define the simulation model required for Markov decision-making in reinforcement learning; train the simulation model and repeatedly train using 24 hours of real-time prediction data ; Apply the trained agent to distribution network dispatching. At each dispatching moment, input the corresponding state space S(t) into the Q neural network in the Rainbow algorithm to obtain the value of each executable action in the current state space. Q function controls the action strategy of energy storage equipment.

Description

Active distribution network dispatching method based on deep reinforcement learning and new energy consumption

Technical field

The present invention relates to the technical field of power system energy dispatching, and in particular to an active distribution network dispatching method based on deep reinforcement learning and considering new energy consumption.

Background technique

With the rapid development of social economy, the demand for energy is increasing day by day. The extensive use of traditional energy sources such as coal not only reduces their stock, but also causes serious environmental pollution. Distributed power generation using new energy can effectively reduce dependence on traditional energy and bear part of the power supply pressure. The main sources of new energy are wind energy, photovoltaics, etc. While they have environmental protection advantages, they also have problems such as unstable power generation caused by weather changes and other factors. The regional distribution network integrates distributed energy systems, energy storage systems and loads, and is connected to the active distribution network to form a balance between supply and demand. Commonly used regional distribution network dispatching schemes include traditional methods, heuristic methods and methods based on reinforcement learning. Traditional methods include nonlinear programming, quadratic programming, Newton's method, etc. These methods are simple to calculate, but difficult to deal with complex problems; heuristic methods include genetic algorithms, simulated annealing algorithms, particle swarm algorithms, etc., which have complex algorithms. , model dependence, easy to fall into local optimality and other problems. The scheduling strategy based on reinforcement learning simulates the human learning process and finds the optimal strategy through continuous interaction with the environment, and currently achieves good application results. However, the above methods are often greatly affected by the dynamics of the power system and the intermittency of new energy sources, resulting in certain errors between the dispatch results and the actual conditions. At the same time, due to problems such as unstable output of new energy, the phenomenon of "abandoning wind and light" is becoming increasingly serious. Therefore, when dispatching regional distribution networks, we must not only consider the cost-effectiveness of electricity prices, but also consider the issue of new energy consumption.

Contents of the invention

In view of this, the present invention provides an active distribution network dispatching method based on deep reinforcement learning that takes into account new energy consumption, constructs a learning model to implement optimal dispatching of energy storage equipment, and reduces the intermittency and uncertainty of new energy. The impact brought by new energy sources will be fully and reasonably consumed.

The technical solutions adopted by the embodiments of the present invention to solve the technical problems are:

An active distribution network dispatching method based on deep reinforcement learning and considering new energy consumption, including:

Step S1, establish a simulation model based on the regional distribution network structure, obtain historical photovoltaic output data, wind power output historical data, village power load historical data and factory power load historical data in the regional distribution network structure, and apply VMD data to predict The processing technology, PCC correlation analysis algorithm and BiLSTM model predict the photovoltaic output forecast data, wind power output forecast data, village power load forecast data and factory power load forecast data in the next 24 hours as training data for the reinforcement learning dispatch model. set;

Step S2, define the simulation model required for Markov decision-making in reinforcement learning;

Step S3: Use the Rainbow algorithm to train the simulation model defined in step S2, divide 24 hours into 24 scheduling moments, and apply 24 hours of real-time prediction data to repeatedly train until the final reward function reaches convergence;

Step S4: Apply the agent trained in step S3 to regional distribution network dispatch. At each dispatch moment, input the corresponding state space S(t) into the Q neural network in the Rainbow algorithm to obtain the current state space. The Q function of each executable action is compared; the optimal action is selected by comparing the Q function of each action, thereby controlling the action strategy of the energy storage device.

Preferably, the step S1 includes:

Step S11, obtain 4 types of data _sequences Data type, village electricity consumption data type, factory electricity consumption data type;

Step S12, apply data preprocessing technology to perform VMD decomposition on the cleaned data sequence X(t) _l to obtain K solid-state mode components where k represents the kth solid-state mode component obtained after VMD decomposition;

Step S13, step S13, for each solid-state mode component Perform PCC correlation analysis, calculate the correlation coefficient between feature components, filter out IMF components with low correlation, and extract K' IMF components with high correlation;

Step S14, apply the BiLSTM model to process the K' highly correlated IMF components, extract features and predict prediction components

Step S15, superimpose Get forecast data/>

Preferably, the step S12 includes:

Step S121, for the preprocessed data _sequence

In the formula, K is the number of solid-state mode components that need to be decomposed, {u _k } and {ω _k } respectively correspond to the k-th mode component and the central frequency after decomposition, δ(t) is the Dirac function, * is the convolution operation , represents the differential at time t, j represents the imaginary unit, |||| ₂ represents the second normal form function, and st represents the constraint;

Step S122, in order to solve the model proposed in step S121, a penalty factor α and a Lagrangian multiplier operator λ are introduced to convert the constrained problem into an unconstrained problem and obtain an augmented Lagrangian expression:

The parameters u _k , ω _k and λ are iteratively updated through the multiplier alternating direction method. The update formula is:

in and/> represents X(t) _l , u _i (t), λ(t) and/> The Fourier transform of , n is the number of iterations, γ is the noise tolerance;

Step S123, for the given judgment accuracy e>0, if it satisfies Then stop iteration, otherwise return to step S122;

Step S124, the iteration ends, and K u components are obtained, which are the IMF components, denoted as

Preferably, the step S13 includes:

Step S131, use Pearson correlation coefficient PCC to calculate each solid-state mode component Carry out correlation analysis and use the data sequence X(t) _l as the target value to calculate the correlation coefficient r _i . The calculation formula of PCC is:

Among them, IMF _i ^t,l represents the i-th component, i=1,2,...,K; Cov(IMF _i ^t,l ,X(t) _l ) is the relationship between IMF _i ^t,l and X(t) _l Covariance, Ver[IMF _i ^t,l ] and Ver[X(t) _l ] are the variances of IMF _i ^t,l and X(t) _l respectively;

Step S132: Screen out K' IMF components with high correlation according to the correlation analysis result r _i obtained in step S131, K' ≤ K.

Preferably, the step S132 selects K' high-correlation IMF components based on the correlation analysis results r _i . The screening method includes:

Select the IMF components corresponding to the first K' r _i in order from high to low r _i .

Preferably, step S14 applies the BiLSTM model to process the K' highly correlated IMF components, extract features and obtain predicted components. In the process, the BiLSTM model corresponding to time t is:

Input gate:

Forgetting Gate:

Memory unit:

Output gate:

Hidden state:

In the formula, i _t represents the input gate, f _t represents the forgetting gate, c _t represents the memory unit, o _t represents the input gate, h _t represents the hidden state; tanh and sigmoid represent the activation function; W _hi , W _hf , W _hc , W _ho respectively represents the weight coefficient of h _t-1 in the feature extraction process of the input gate, forgetting gate, memory unit and output gate; W _xi , W _xf , W _xc and W _xo respectively represent the input gate, forgetting gate, memory unit and output. In the process of door feature extraction The weight coefficient; b _i , b _f , b _c , _{and bo} represent the bias values in the feature extraction process of the input gate, forget gate, memory unit, and output gate respectively; h _t-1 represents the hidden state at the previous moment; c _t-1 represents the memory unit when it is not updated.

Preferably, the step S2 includes:

Step S21, define the environmental state space S(t). The S(t) consists of 24-hour village power load data, factory power load data, wind power output data, photovoltaic output data and the real-time power storage status of energy storage equipment. Composed of five parts;

Step S22: Define the action space of the intelligent agent. The action space includes three actions: charging, discharging, and idle of the energy storage device;

Step S23, define a reward function for controlling the action of the energy storage device;

Step S24, the agent interacts with the main power grid, and the regional distribution network is connected to the main power grid through a public connection point. When the output of all new energy sources in the regional distribution network can meet all load requirements and the energy storage device has When it is full, the remaining electricity from the new energy source is fed back to the main grid; when the output of the new energy source and the energy storage equipment cannot meet the full load demand, power is purchased from the main grid.

Preferably, in step S21, at the future scheduling time t, the agent obtains the photovoltaic output prediction data at time t from the environment. Wind power output forecast data/> Village electricity load forecast data/> Factory electricity load forecast data/> The power state E _t of the energy storage device at time t. These five state information constitute the environmental state space. It is the scheduling time within the next 24 hours;

In the step S22, the action strategy set A of the energy storage device includes:

Among them, a _I represents the charging action strategy of the energy storage device, specifically charging the energy storage device through photovoltaic output, wind power output or the main grid, and a _O represents the discharge action strategy of the energy storage device, specifically the energy storage device. Energy equipment discharges to village electrical equipment, factory electrical equipment or the main grid, a _N represents the idle action strategy of the energy storage equipment;

Further, under the condition that physical constraints are met, a dynamic model is used to represent the energy storage device, which is specifically expressed as:

Among them, E _t represents the power of the energy storage device at time t, which satisfies E _min < E _t < E _max , where E _min and E _max represent the maximum capacity and minimum capacity of the energy storage device respectively; P _t represents the The charging and discharging power of the energy storage device, P _t <0 means that the energy storage device is in a discharge state, P _t > 0 means that the energy storage device is in a charging state; ζ and η respectively represent the charging efficiency and Discharge efficiency;

In step S23, under the condition that the physical constraints of the energy storage device are met, the reward function is set to:

In the formula, k _O is the discharge reward factor, k _I is the charging reward factor, and n is the penalty factor;

In step S24, the grid power balance limit is set, and the power balance relationship is:

P _balance (t)＝P _renew (t)-P _load (t)

P _grid (t)=P _balance (t)+P _E (t)

In the formula, P _renew (t) is the total power generation of new energy in the regional distribution network at time t, P _load (t) is the total power consumed by the loads in the regional distribution network at time t; P _balance (t) is the total power consumption of the loads in the regional distribution network at time t. The difference between the supply and demand of the total power generated by new energy and the total power consumed by the load. P _balance (t) > 0 means there is excess power generated by new energy in the regional distribution network. P _balance (t) < 0 means the There is insufficient new energy power generation in the regional distribution network; when P _E (t) > 0, P _E (t) represents the discharge power of the energy storage equipment; when P _E (t) < 0, P _E (t) represents the energy storage equipment Charging power; P _grid (t) is the mutual transmission of electric power between the regional distribution network and the main grid, P _grid (t) is a regular expression that the regional distribution network transmits to the main grid, P _grid (t) If it is negative, it means that the main grid transmits power to the regional distribution grid.

Preferably, the step S3 includes:

Step S31: Construct a neural network with a hidden layer and two fully connected layers, and interfere with all fully connected layer parameters by adding a Gaussian distributed noise term to replace the ε-greedy (random-greedy) exploration method of DQN. By adding noise to the parameters of the fully connected layer, the exploration ability of the algorithm is effectively enhanced, and the original calculation formula y=wx+b is modified into:

y＝(μ ^w +σ ^w ⊙N ^w )x+μ ^b +σ ^b ⊙N ^b

In the variant formula, the weight w and error b in the formula y=wx+b are converted into random noise ε that obeys the normal distribution of the mean μ and the variance σ and the Gaussian distribution, where ε is the random noise ε of each round of training. The constants generated in , N ^b , μ ^b , σ ^w , N ^w , and μ ^w are all parameters;

Step S32, add a competition network in front of the output layer Q network, and decompose the Q function of the output layer into the sum of the value function V and the advantage function H, that is, Q = V + H, where V represents the reward value caused by the state, H Represents the reward value obtained after the energy storage device performs charging, discharging, and idle actions. Due to the state constraints of the Q network, the V value is updated first, and then the H value is adjusted. The Q function formula is:

In the formula, a _t is the action strategy, θ is the network layer parameter of the Q function, ω is the value function network layer parameter, υ is the advantage function network layer parameter,/> is the average advantage function, a′ _t is all possible actions produced in state s _t ;

Step S33, build two Q networks as the output layer of the neural network to decouple the action selection a _t and the V value of the selected action. The first Q network is used to select the best action in the current state, and the second Q network is used to select the best action in the current state. Q network is used to evaluate charging and discharging actions;

Step S34, use a multi-step learning strategy to obtain instant rewards by interacting with the environment. The reward formula is:

In the formula, n is the stride length, θ is the neural network parameter, d is the discount rate, and R is the return value;

Step S35: Use the priority experience playback pool PR to customize a fixed-capacity experience pool, and put each set of data (s _t , a _t , r _t , s _t+1 ) after training of the agent into the In the experience pool, the training data error δi is calculated at the same time, and different error priorities are assigned to re-enter the neural network for training. The specific sampling priority formula is as follows:

ρ _i =|δ _i |+ε

δ _i =|Q(s _t ,a _t )-Q'(s _t ,a _t )|

In the formula, _Pi is the correlation value, ε is the noise factor that prevents _Pi from being 0, β is the annealing factor used to adjust the priority, and δ _i is the error value caused by a set of experiences during training.

It can be seen from the above technical solutions that the active distribution network dispatching method based on deep reinforcement learning and new energy consumption provided by the embodiment of the present invention first establishes a simulation model based on the regional distribution network structure to obtain the photovoltaic output in the regional distribution network structure. Historical data, wind power output historical data, village power load historical data and factory power load historical data, apply VMD data preprocessing technology, PCC correlation analysis algorithm and BiLSTM model to predict photovoltaic output forecast data and wind power output in the next 24 hours Forecast data, village power load forecast data and factory power load forecast data are used as training data sets for the reinforcement learning scheduling model; define the simulation model required for Markov decision-making in reinforcement learning; use the Rainbow algorithm to simulate the simulation defined in step S2 The model is trained; the trained agent is applied to distribution network dispatching. At each dispatching moment, the corresponding state space S(t) is input into the Q neural network in the Rainbow algorithm, and each possible state in the current state space is obtained. Execute the Q function of the action; select the optimal action by comparing the Q function of each action to control the action strategy of the energy storage device. The present invention constructs a learning model to implement optimal scheduling of energy storage equipment, reduce the impact of the intermittency and uncertainty of new energy, and achieve full and reasonable consumption of the electricity produced by new energy.

Description of drawings

Figure 1 is a flow chart of the active distribution network dispatching method taking into account new energy consumption based on deep reinforcement learning according to the present invention.

Figure 2 is a structural diagram of the regional distribution network in the present invention.

Figure 3 is a diagram of the energy scheduling model of deep reinforcement learning in the present invention.

Figure 4 is a flow chart of the prediction method based on VMD and BiLSTM in the present invention.

Figure 5 is a schematic diagram of the predicted electric power values of new energy sources and loads in the present invention.

Figure 6 is a schematic flowchart of the overall flow of the active distribution network dispatching method based on deep reinforcement learning and taking into account new energy consumption according to the present invention.

Figure 7 is a schematic diagram of changes in rewards obtained during training in the invention.

Fig. 8 is a schematic diagram of the change in power state of the energy storage device in the present invention.

Detailed ways

The technical solutions of the embodiments of the present invention are explained and described below, but the following embodiments are only preferred embodiments of the present invention and are not all of them. Based on the examples in the implementation mode, other embodiments obtained by those skilled in the art without making creative efforts shall fall within the protection scope of the present invention.

In view of the problems existing in the regional distribution network proposed by the background technology, the present invention intends to integrate the prediction algorithm into dispatching to form a "prediction + dispatching" framework, which can effectively improve the accuracy and stability of dispatching in the future.

The present invention provides an energy storage dispatching strategy based on deep reinforcement learning that takes into account new energy consumption, learns an optimal strategy, ensures the normal operation of each participating part in the power grid, and obtains a reasonable amount of electricity generated by new energy sources. Consumptive. As shown in Figure 1, the implementation steps of the active distribution network dispatching method provided by the present invention include:

Step S1, establish a simulation model based on the regional distribution network structure (as shown in Figure 2), and obtain historical photovoltaic output data, wind power output historical data, village power load historical data, and factory power load historical data in the regional distribution network structure. , applying VMD data preprocessing technology, PCC correlation analysis algorithm and BiLSTM model to predict the photovoltaic output forecast data, wind power output forecast data, village power load forecast data and factory power load forecast data in the next 24 hours as a reinforcement learning dispatch model The training data set (refer to Figure 5);

Step S2, define the simulation model required for Markov decision-making in reinforcement learning;

Step S3: Use the Rainbow algorithm to train the simulation model defined in step S2. Divide 24 hours into 24 scheduling moments, schedule once per hour, and use 24 hours of real-time prediction data to train repeatedly until the final reward function reaches convergence. ;

Step S4: Apply the agent trained in step S3 to regional distribution network dispatching. At each dispatching moment, input the corresponding state space S(t) into the Q neural network in the Rainbow algorithm to obtain each state space under the current state space. A Q function that can perform actions; by comparing the Q function of each action, the optimal action is selected to control the action strategy of the energy storage device.

Referring to Figure 4, the steps of establishing a training data set in step S1 include:

Step S11, obtain 4 kinds of data sequences X(t _{) l} and perform data cleaning preprocessing ₍ the method of data cleaning for data sequence and jump data, replaced by the average value of several nearby data), where l∈{1,2,3,4}, l indicates the type of data sequence, and the type of data sequence includes photovoltaic data type, wind data Type, village power consumption data type, factory power consumption data type;

Step S12, apply the VMD model to perform VMD decomposition on the preprocessed data sequence X(t) _ll to obtain K solid-state mode components where k represents the kth solid-state mode component obtained after VMD decomposition;

Step S121, for the preprocessed data _sequence

In the formula, K is the number of solid-state mode components that need to be decomposed (a positive integer), {u _k } and {ω _k } respectively correspond to the k-th mode component and the center frequency after decomposition, and δ(t) is the Dirac function. , * is the convolution operation, represents the differential at time t, j represents the imaginary unit, |||| ₂ represents the second normal form function, and st represents the constraint;

Step S122: In order to solve the model proposed in step S121, the penalty factor α (used to reduce the influence of Gaussian noise) and the Lagrangian multiplier operator λ are introduced to convert the constrained problem into an unconstrained problem and obtain the augmented Lagrangian expression:

The parameters u _k , ω _k and λ are iteratively updated through the multiplier alternating direction method. The update formula is:

in and/> represents X(t) _l , u _i (t), λ(t) and/> The Fourier transform of , n is the number of iterations, γ is the noise tolerance, which meets the fidelity requirements of signal decomposition;

Step S123, for the given judgment accuracy e>0, if it satisfies Then stop iteration, otherwise return to step S122;

Step S124, the iteration ends, and K (K×l) u components are obtained, that is, IMF components, denoted as

Step S13, for each solid-state mode component Perform PCC correlation analysis, calculate the correlation coefficient between feature components, screen out IMF components with low correlation, and extract K' IMF components with high correlation; specifically:

Step S131, use Pearson correlation coefficient PCC to calculate each solid-state mode component Carry out correlation analysis and calculate the correlation coefficient r _i based on the target value of the original data signal quantity X(t) _l . The calculation formula of PCC is:

Among them, IMF _i ^t,l represents the i-th component, i=1,2,...,K; Cov(IMF _i ^t,l ,X(t) _l ) is the relationship between IMF _i ^t,l and X(t) _l Covariance, Ver[IMF _i ^t,l ] and Ver[X(t) _l ] are the variances of IMF _i ^t,l and X(t) _l respectively;

Step S132: According to the correlation analysis result r _i obtained in step S131, K' IMF components with high correlation are selected and input into the BiLSTM prediction model, where K'≤K. Here, the screening method can be used: select the IMF components corresponding to the first K' _{r i in} order from high to low;

Step S14, apply the BiLSTM model to process K' highly correlated IMF components, extract features and predict the predicted components Among them, the BiLSTM model corresponding to time t is:

Input gate:

Forgetting Gate:

Memory unit:

Output gate:

Hidden state:

In the formula, i _t represents the input gate, f _t represents the forgetting gate, c _t represents the memory unit, o _t represents the input gate, h _t represents the hidden state; tanh and sigmoid represent the activation function; W _hi , W _hf , W _hc , W _ho respectively represents the weight coefficient of h _t-1 in the feature extraction process of the input gate, forgetting gate, memory unit and output gate; W _xi , W _xf , W _xc and W _xo respectively represent the input gate, forgetting gate, memory unit and output. The weight coefficient of IMF _k ^t,l during the feature extraction process of the gate; b _i , b _f , b _c , and _bo respectively represent the offset values in the feature extraction process of the input gate, forgetting gate, memory unit, and output gate; h _t-1 represents the hidden state at the previous moment; c _t-1 represents the memory unit before updating.

Step S15, superimpose Get forecast data/>

Referring to Figure 3, the specific implementation of defining the simulation model in step S2 includes:

Step S21, define the environmental state space S(t). S(t) consists of five elements: 24-hour village power load data, factory power load data, wind power output data, photovoltaic power output data, and real-time power storage status of energy storage equipment. Partly composed;

In step S21, at the future scheduling time t, the agent obtains the photovoltaic output prediction data at time t from the environment. Wind power output forecast data/> Village electricity load forecast data/> Factory electricity load forecast data/> The power state E _t of the energy storage device at time t. These five state information constitute the environmental state space. It is the scheduling time within the next 24 hours;

In step S22, the action space at each moment includes the three states of charging, discharging, and idle of the energy storage device. The action strategy set A of the energy storage device includes:

Among them, a _I represents the charging action strategy of the energy storage device, specifically charging the energy storage device through photovoltaic output (branch 1), wind power output (branch 2) or the main grid, and a _O represents the discharge action strategy of the energy storage device. , specifically the energy storage equipment discharges to the village power equipment (branch 1), factory power equipment (branch 2) or the main grid, a _N represents the idle action strategy of the energy storage equipment;

Furthermore, under the condition that the physical constraints are met, the action strategy for the energy storage device needs to optimize the charge and discharge time and charge and discharge amount of the energy storage device. A dynamic model is used to represent the energy storage device, which is specifically expressed as:

Among them, E _t represents the power of the energy storage device at time t, which satisfies E _min < E _t < E _max , where E _min and E _max represent the maximum capacity and minimum capacity of the energy storage device respectively; P _t represents the charge and discharge of the energy storage device. Power, P _t <0 represents the energy storage device in the discharge state, P _t >0 represents the energy storage device in the charging state; ζ and η represent the charging efficiency and discharge efficiency of the energy storage device respectively;

Step S23, define a reward function for controlling the action of the energy storage device; the agent selects the action of the energy storage device through the reward function. The reward function mainly considers the action reward. Under the condition that the physical constraints of the energy storage device are met, the reward function is set as :

In the formula, k _O is the discharge reward factor, k _I is the charging reward factor, and n is the penalty factor;

Step S24, the agent interacts with the main grid, and the regional distribution network is connected to the main grid through public connection points. When the output of all new energy sources (wind power and photovoltaic) in the regional distribution network can meet all load demands and the energy storage equipment has When full, the remaining electricity from new energy sources is fed back to the main grid; when the output of new energy sources and energy storage equipment cannot meet the full load demand, power is purchased from the main grid. In step S24, the grid power balance limit is set, and the power balance relationship is:

P _balance (t)＝P _renew (t)-P _load (t) (10)

P _grid (t)＝P _balance (t)+P _E (t) (11)

In the formula, P _renew (t) is the total power generation of new energy in the regional distribution network at time t, P _load (t) is the total power consumed by the loads in the regional distribution network at time t; P _balance (t) is the total power consumption of the loads in the regional distribution network at time t. The difference between the supply and demand of the total power generated by new energy and the total power consumed by the load. P _balance (t) > 0 means there is excess power generated by new energy in the regional distribution network. P _balance (t) < 0 means the There is insufficient new energy power generation in the regional distribution network; when P _E (t) > 0, P _E (t) represents the discharge power of the energy storage equipment; when P _E (t) < 0, P _E (t) represents the energy storage equipment Charging power; P _grid (t) is the mutual transmission of electric power between the regional distribution network and the main grid, P _grid (t) is a regular expression that the regional distribution network transmits to the main grid, P _grid (t) If it is negative, it means that the main grid transmits power to the regional distribution grid.

The specific implementation of step S3 includes:

Step S31, construct a neural network with a hidden layer and two fully connected layers, and interfere with all fully connected layer parameters plus a Gaussian distributed noise term to replace DQN's ε-greedy (random-greedy) exploration method. Adding noise to the parameters of the fully connected layer effectively strengthens the exploration ability of the algorithm. The original calculation formula y=wx+b is modified into:

y＝(μ ^w +σ ^w ⊙N ^w )x+μ ^b +σ ^b ⊙N ^b (12)

In the variant formula, the weight w and error b in the formula y=wx+b are converted into random noise ε that obeys the normal distribution of the mean μ and the variance σ, and obeys the Gaussian distribution, where ε is the random noise ε of each round of training. The constants generated in , N ^b , μ ^b , σ ^w , N ^w , and μ ^w are all parameters;

Step S32, add a competition network in front of the output layer Q network, and decompose the Q function of the output layer into the sum of the value function V and the advantage function H, that is, Q = V + H, where V represents the reward value caused by the state, H Indicates the reward value obtained after the energy storage device performs charging, discharging, and idle actions. Due to the state constraints of the Q network, the V value is updated first, and then the H value is adjusted. In actual operation, in order to limit the values of H and V, To keep it within a reasonable range, the average value of H is generally limited to 0. The Q function formula is:

In the formula, a _t is the action strategy, θ is the network layer parameter of the Q function, ω is the network layer parameter of the value function, υ is the network layer parameter of the advantage function,/> is the average advantage function, a′ _t is all possible actions produced in state s _t ;

Step S33, build two Q networks as the output layer of the neural network, and decouple the action selection a _t and the V value of the selected action. The first Q network is used to select the best action in the current state, and the second Q network is used to select the best action in the current state. Used to evaluate charging and discharging actions, the problem of overestimation error can be effectively solved by using the output results of two networks.

In step S34, a multi-step learning strategy is used to obtain instant rewards by interacting with the environment. Therefore, the Q value in the early stage of training can be more accurately estimated, thereby speeding up training. The reward formula is:

In the formula, n is the stride length, θ is the neural network parameter, d is the discount rate, and R is the reward value; refer to Figure 7 for the reward situation;

Step S35: Use the prioritized experience playback pool PR (Prioritized Reply) to customize a fixed-capacity experience pool, and combine each set of data (s _t , a _t , r _t , s _t+1 ) after training of the agent Put it into the experience pool, calculate the training data error δ _i at the same time, assign different error priorities and re-enter the neural network for training. (The meaning of assigning error priorities is for sorting, according to the interval where the error value is located. Give priority, then arrange the data according to the priority, and select the data with the first few priorities as training data to reduce the amount of training data and improve training efficiency and quality.) The specific sampling priority formula is as follows:

δ _i =|Q(s _t ,a _t )-Q'(s _t ,a _t )| (16)

In the formula, _Pi is the correlation value, ε is the noise factor that prevents _Pi from being 0, β is the annealing factor used to adjust the priority, and δ _i is the error value caused by a set of experiences during training.

Compared with the prior art, the present invention has the following advantages:

(1) Combine VMD decomposition with the deep learning model BiLSTM, and use PCC correlation analysis to streamline the IMF components, effectively improve calculation efficiency, and solve the instability of photovoltaic and wind power generation, improving prediction accuracy.

(2) Integrate prediction algorithms into scheduling to form a "prediction + scheduling" framework, which can effectively improve the accuracy and stability of scheduling in the future.

(3) Implement the deep reinforcement learning algorithm Rainbow to optimize the scheduling of energy storage equipment, effectively absorbing the power generated by new energy sources.

This invention uses the energy storage scheduling strategy of deep reinforcement learning. The deep learning model extracts high-order data features from high-dimensional, continuous space. At the same time, combined with VMD data preprocessing technology, it has a better understanding of wind power generation and photovoltaic power generation containing uncertainty. Good expression ability and feature mining ability. Reinforcement learning solves continuous decision-making problems and can optimize scheduling strategies through repeated trial and error exploration in dynamic and uncertain environments. Among them, in this invention, for the effective consumption of new energy power generation, VMD decomposition and deep learning model BiLSTM (bidirectional long short-term memory network) are applied to predict wind power, photovoltaic, and load; the deep reinforcement learning Rainbow algorithm is used for energy storage The optimal dispatch of the system can effectively solve the problems of low energy utilization efficiency, reduced dispatch performance and high operating costs due to the uncertainty of renewable energy, energy flow and load diversity.

In the prediction process, PCC correlation analysis, machine learning and data mining tasks are applied. The use of PCC can help exclude features that are weakly correlated with the target variable, thereby reducing the data analysis resources occupying the feature space and improving the analysis efficiency of the model. accuracy.

What is disclosed above is only the preferred embodiment of the present invention. Of course, it cannot be used to limit the scope of the present invention. Those of ordinary skill in the art can understand all or part of the processes for implementing the above embodiments and make decisions according to the claims of the present invention. Equivalent changes still fall within the scope of the invention.

Claims (9)

1. An active power distribution network scheduling method based on deep reinforcement learning and new energy consumption is characterized by comprising the following steps:

step S1, a simulation model is established according to a regional power distribution network structure, photovoltaic output historical data, wind output historical data, village power utilization load historical data and factory power utilization load historical data in the regional power distribution network structure are obtained, VMD data preprocessing technology, PCC correlation analysis algorithm and BiLSTM model are applied, photovoltaic output prediction data, wind output prediction data, village power utilization load prediction data and factory power utilization load prediction data in the future 24 hours are predicted, and the photovoltaic output prediction data, the wind output prediction data, the village power utilization load prediction data and the factory power utilization load prediction data are used as training data sets of a reinforcement learning scheduling model;

step S2, defining a simulation model required by Markov decision in reinforcement learning;

step S3, training the simulation model defined in the step S2 by adopting a Rainbow algorithm, dividing 24 hours into 24 scheduling moments, and repeatedly training by applying 24 hours of real-time prediction data until the final reward function is converged;

step S4, applying the agent trained in the step S3 to regional power distribution network scheduling, and inputting a corresponding state space S (t) into a Q neural network in a Rainbow algorithm at each scheduling moment to obtain a Q function of each executable action in the current state space; and selecting an optimal action by comparing the Q functions of each action, thereby controlling the action strategy of the energy storage equipment.

2. The active power distribution network scheduling method based on deep reinforcement learning and new energy consumption according to claim 1, wherein the step S1 comprises:

step S11, obtaining 4 data sequences X (t) _l Performing data cleaning pretreatment, wherein l is {1,2,3,4}, and l indicates the type of a data sequence, wherein the type of the data sequence comprises a photovoltaic data type, a wind power data type, a village power data type and a factory power data type;

step S12, applying data preprocessing technique to the cleaned data sequence X (t) _l VMD decomposition is performed to obtain K solid state mode componentsWherein k represents a kth solid state mode component obtained after decomposition of the VMD;

step S13, for each of the solid mode componentsPerforming PCC correlation analysis, calculating correlation between characteristic componentsScreening IMF components with low correlation degree, and extracting K' IMF components with high correlation degree;

s14, processing the K' IMF components with high correlation by using the BiLSTM model, extracting features and predicting predicted components

Step S15, superpositionDeriving predictive data->

3. The active power distribution network scheduling method based on deep reinforcement learning and new energy consumption according to claim 2, wherein the step S12 includes:

step S121, for the data sequence X (t) after preprocessing _l Assuming each mode has a limited bandwidth with a center frequency, K modes are now sought, minimizing the sum of the estimated bandwidths for each mode, with the specific model:

wherein K is the number of solid mode components to be decomposed, { u _k }、{ω _k The k mode component and the center frequency after decomposition are respectively corresponding, delta (t) is a dirac function, x is convolution operation,represents differentiating the time tJ represents an imaginary unit, I ₂ Representing a two-norm function, and s.t. representing constraint conditions;

step S122, introducing a penalty factor alpha and a Lagrange multiplier lambda to solve the model proposed in the step S121, and converting the constraint problem into a non-constraint problem to obtain an augmented Lagrange expression:

iterative updating of parameter u by multiplier alternating direction method _k 、ω _k And lambda, updating the formula:

wherein the method comprises the steps ofAnd->Representing X (t) _l 、u _i (t), lambda (t) and +.>N is the number of iterations and γ is the noise tolerance;

step S123, for a given judgment precision e>0, if it meetsStopping the iteration, otherwise returning to execute the step S122;

step S124, after iteration, obtaining K u components, namely the IMF components, which are recorded as

4. The active power distribution network scheduling method based on deep reinforcement learning and new energy consumption according to claim 3, wherein the step S13 includes:

step S131, for each of the solid state mode components, using the Pearson correlation coefficient PCCPerforming a correlation analysis with said data sequence X (t) _l For the target value, calculate the correlation coefficient r _i The calculation formula of PCC is:

wherein, IMF _i ^t,l Representing the i-th component, i=1, 2, …, K; cov (IMF) _i ^t,l ,X(t) _l ) Is IMF _i ^t,l And X (t) _l Covariance of Ver [ IMF ] _i ^t,l ]And Ver [ X (t) _l ]IMF respectively _i ^t,l And X (t) _l Is a variance of (2);

step S132, according to the correlation analysis result r obtained in the step S131 _i And screening out K 'IMF components with high correlation, wherein K' is less than or equal to K.

5. The active power distribution network scheduling method based on deep reinforcement learning and new energy consumption according to claim 4, wherein the step S132 is based on the correlation analysis result r _i Screening K' IMF components with high correlationThe selection method comprises the following steps:

according to r _i The first K' r are selected from the order from high to low _i The corresponding IMF component.

6. The active power distribution network scheduling method based on deep reinforcement learning and new energy consumption according to claim 5, wherein the step S14 processes the K' IMF components with high correlation by using a BiLSTM model, extracts features and obtains predicted componentsIn the process, the BiLSTM model corresponding to the time t is as follows:

an input door:

forgetting the door:

a memory unit:

output door:

hidden state:

wherein i is _t Representing the input gate, f _t Indicating forgetful door c _t Representing a memory cell, o _t Represents the input gate, h _t Representing a hidden state; tanh, sigmoid represents an activation function; w (W) _hi 、W _hf 、W _hc 、W _ho Respectively represent h in the characteristic extraction process of the input gate, the forgetting gate, the memory unit and the output gate _t-1 Weight coefficient of (2); w (W) _xi 、W _xf 、W _xc 、W _xo Respectively represent IMF in the characteristic extraction process of input gate, forget gate, memory unit and output gate _k ^t,l Weight coefficient of (2); b _i 、b _f 、b _c 、b _o Respectively representing bias values in the characteristic extraction process of the input gate, the forgetting gate, the memory unit and the output gate; h is a _t-1 Representing the hidden state of the previous moment; c _t-1 Representing the memory cells when not updated.

7. The active power distribution network scheduling method based on deep reinforcement learning and new energy consumption according to claim 6, wherein the step S2 comprises:

step S21, defining an environment state space S (t), wherein the S (t) consists of five parts, namely village power load data, factory power load data, wind power output data, photovoltaic output data and real-time power storage state of energy storage equipment for 24 hours;

step S22, defining an action space of the intelligent agent, wherein the action space comprises three actions of charging, discharging and idling of the energy storage equipment;

step S23, defining a reward function for controlling the action of the energy storage device;

step S24, the intelligent agent interacts with a main power grid, the regional power distribution network is connected with the main power grid through a public connection point, and when all new energy output in the regional power distribution network can meet all load demands and the energy storage equipment has full electric quantity, the residual electric quantity of the new energy is fed back to the main power grid; and purchasing electricity from the main power grid when the new energy output and the energy storage equipment cannot meet all load requirements.

8. The active power distribution network scheduling method based on deep reinforcement learning and new energy consumption according to claim 7, wherein the method comprises the following steps:

in step S21, at a future scheduling time t, the agent acquires photovoltaic output prediction data at time t from the environmentWind power output prediction data->Village electricity load prediction data +.>Factory electrical load prediction dataEnergy storage device t moment state of charge E _t These five state information make up the environmental state spaceScheduling time within 24 hours in the future;

in the step S22, the action policy set a of the energy storage device includes:

wherein a is _I Representing a charging action strategy of the energy storage device, in particular charging the energy storage device by photovoltaic output, wind output or a main power grid, a _O Representing a discharging action strategy of the energy storage equipment, in particular that the energy storage equipment discharges to village electric equipment, factory electric equipment or a main power grid, a _N Representing an idle action strategy of the energy storage device;

further, under the condition that the physical constraint condition is met, the energy storage equipment is represented by adopting a dynamic model, which is specifically represented as follows:

wherein E is _t Representing the electric quantity of the energy storage equipment at the t moment, and meeting E _min <E _t <E _max Here E _min And E is _max Representing a maximum capacity and a minimum capacity of the energy storage device, respectively; p (P) _t Representing the charge and discharge power of the energy storage device, P _t < 0 represents that the energy storage device is in a discharge state, P _t > 0 represents that the energy storage device is in a charged state; ζ and η represent a charging efficiency and a discharging efficiency of the energy storage device, respectively;

in the step S23, the reward function is set to:

wherein k is _O To discharge the reward factor, k _I A charging reward factor, n is a penalty factor;

in the step S24, a power balance limit of the power grid is set, and the power balance relationship is as follows:

P _balance (t)＝P _renew (t)-P _load (t)

P _grid (t)＝P _balance (t)+P _E (t)

wherein P is _renew (t) is the total power generation power of new energy sources in the regional power distribution network at the moment t, P _load (t) the total power consumption of the load in the power distribution network in the region at the moment t; p (P) _balance (t) is the difference between the total power of the new energy power generation and the total power of the load power consumption, P _balance (t)>0 represents surplus power of new energy generation in the regional power distribution network, and P _balance (t)<0 represents that the new energy power generation power in the regional power distribution network is insufficient; p (P) _E (t)>At 0, P _E (t) represents the discharge power of the energy storage device, P _E (t)<At 0, P _E (t) represents energy storage device charging power; p (P) _grid (t) transmitting electric power for the regional distribution network and the main power grid, P _grid (t) regular representing the delivery of the regional distribution network to the main grid, P _grid (t) negative indicating that the main power grid distributes power to the areaAnd (5) network power transmission.

9. The active power distribution network scheduling method based on deep reinforcement learning and new energy consumption according to claim 8, wherein the step S3 comprises:

step S31, constructing a neural network of a hidden layer and two full-connection layers, and adding a Gaussian-distributed noise item to all full-connection layer parameters to interfere so as to replace an exploration mode of epsilon-greedy (random-greedy) of the DQN; the exploring capability of an algorithm is effectively enhanced by adding noise to the full-connection layer parameters, and the original front-term calculation formula y=wx+b is changed into the following formula:

y＝(μ ^w +σ ^w ⊙N ^w )x+μ ^b +σ ^b ⊙N ^b

in the variant formula, the weights w and the errors b in the formula y=wx+b are converted into a normal distribution obeying the mean μ and the variance σ, and a random noise epsilon obeying the gaussian distribution, where epsilon is a constant produced in each training round, N ^b 、μ ^b 、σ ^w 、N ^w 、μ ^w All are parameters;

step S32, adding a competing network before the Q network of the output layer, decomposing the Q function of the output layer into a sum of a cost function V and a dominance function H, i.e., q=v+h, where V represents a reward value caused by a state, H represents a reward value obtained after the energy storage device performs charging, discharging, and idle actions, and because of a constraint of the Q network in a state, updating the V value preferentially, and readjusting the H value, where the Q function formula is:

in the method, in the process of the invention,a _t is an action strategy, θ is a network layer parameter of the Q function, ω is a cost function network layer parameter, v is a dominance function network layer parameter, +.>As the mean value of the dominance function, a' _t To be in state s _t All possible actions generated in (a);

step S33, two Q networks are built as output layers of the neural network, and actions are selected as a _t The V value of the action is selected for decoupling, wherein the first Q network is used for selecting the optimal action in the current state, and the second Q network is used for evaluating the charge and discharge actions;

step S34, using a multi-step learning strategy, obtaining instant rewards through interaction with the environment, wherein a rewards formula is as follows:

wherein n is stride length, θ is neural network parameter, d is discount rate, and R is return value;

step S35, custom defining a fixed-capacity experience pool by using the priority experience playback pool PR, and training each group of data (S _t ,a _t ,r _t ,s _t+1 ) Put into the experience pool and calculate training data error delta _i Different error priorities are given to be re-sent into the neural network for training, and the specific sampling priority is given by the following formula:

ρ _i ＝|δ _i |+ε

δ _i ＝|Q(s _t ,a _t )-Q'(s _t ,a _t )|

wherein P is _i Is a correlation value, ε is the prevention of P _i A noise factor of 0, β is an annealing factor for adjusting priority, δ _i Is an error value caused by a group of experience in training.