CN116628448B - Sensor management method based on deep reinforcement learning in extended goals - Google Patents

Abstract

The application discloses a sensor management method based on deep reinforcement learning in an extended target, which comprises the following steps: modeling is conducted on an ellipse expansion target, and a virtual interaction environment with deep reinforcement learning is constructed according to an expansion target filtering algorithm; establishing a TD3 algorithm intelligent agent; the virtual interaction environment is interacted with the TD3 algorithm intelligent agent to obtain sensor control data, and the sensor control data is stored as a sample in an experience playback pool; based on the experience playback pool, extracting samples, training an intelligent agent of a TD3 algorithm, and deciding a sensor path planning optimal action through the trained intelligent agent; and (3) acting the optimal action on the sensor, and obtaining the sensor position after the sensor is subjected to state transfer, so as to obtain the sensor measurement value of the expansion target at the current moment, and carrying out filtering prediction and updating to carry out tracking estimation of the expansion target. The application optimizes the tracking effect of the ellipse expanding target as a whole.

Description

Sensor management method based on deep reinforcement learning in extended goals

Technical field

The present invention relates to the technical field of sensor intelligent management, and in particular to a sensor management method based on deep reinforcement learning in extended targets.

Background technique

Sensor management refers to the process of ultimately achieving goals by controlling the degrees of freedom in a sensor system to meet certain constraints and optimize certain performance measures. With the emergence of modern high-resolution sensors, the target can produce multiple measurements from multiple measurement sources at a certain time, so more state characteristics of the target can be estimated, such as the shape profile of the target. This type of problem is called Extended target tracking issues. Generally, there are two main categories of sensor management methods to obtain the best measurement information for the purpose of optimizing target tracking performance. One is the sensor management method based on task theory. This method uses specific task requirements, such as the variance of the state or the related information. The measurement of the target state distribution is used to formulate the corresponding sensor control strategy. However, such methods are difficult to meet the requirements of systems where multiple task requirements exist simultaneously. The other type is a sensor management method based on information theory, which establishes an evaluation function (such as Kullback-Leibler divergence, Rényi divergence, etc.) through some measure of the information gain between two probability density functions, and then maximizes the information gain. Solve the sensor control strategy under the criterion. Sensor control methods based on information theory can maximize the overall information gain of a system containing multiple tasks.

Sensor management in traditional target tracking is usually studied under the theoretical framework of a partially observable Markov decision process. Sensor management is usually performed in a discrete action space, because each decision needs to be based on the established response. Under the evaluation criteria, evaluate all actions in the achievable sensor control scheme. Therefore, traditional methods cannot handle the dimension explosion and computational complexity problems caused by the sharp increase in the action space.

In recent years, deep reinforcement learning has become a new research hotspot in the field of artificial intelligence. The cross-fusion of deep reinforcement learning and extended target tracking problems provides a new way to realize intelligent decision-making in sensor control. The Deep Q-Network (DQN) algorithm is a pioneering work in the field of deep reinforcement learning. On the basis of DQN, a series of improved algorithms such as Double DQN (DDQN), Dueling DQN and Double Dueling DQN (D3QN) have been proposed. Although DQN and its improved algorithms have good application effects, they still cannot handle the problem of continuous action space. Therefore, a classic continuous control algorithm has emerged in the field of deep reinforcement learning, the deep deterministic policy gradient (DDPG) algorithm, which is an important algorithm applied to complex and continuous control. However, the DDPG algorithm has the problem that the Critic network overestimates the Q value.

Among the existing sensor management methods in the field of target tracking, when the task of sensor management is to control the position of the sensor platform to optimize the performance of target tracking, due to the two types of sensor management methods based on task theory and information theory, both need to establish specific Research is conducted under task optimization or certain optimization criteria, so control decisions are mainly based on discrete sensor action spaces, and the entire action space needs to be traversed in each decision-making process. When all actions to be decided in the degree of freedom space need to be considered, , traditional sensor control methods will face the problem of sharp decline in efficiency caused by dimensionality explosion, and when higher degrees of freedom in decision-making are required, traditional sensor control methods will be helpless.

Contents of the invention

In order to solve the above technical problems, the present invention proposes a sensor management method based on deep reinforcement learning in extended targets, which expands the traditional sensor management decision space from discrete action space to continuous action space, and establishes a joint optimization extension based on the elliptical extended target tracking estimation effect. The target kinematic state and extended state (contour information) are the scientific reward and return mechanism of the target. Based on the deep reinforcement learning algorithm, the intelligent agent learning optimal control strategy is established to realize intelligent decision-making of sensor control through artificial intelligence.

In order to achieve the above objectives, the present invention provides a sensor management method based on deep reinforcement learning among extended goals, including:

Model the elliptical extended target and build a virtual interactive environment with deep reinforcement learning based on the extended target filtering algorithm;

Establish TD3 algorithm intelligent agent;

The virtual interactive environment interacts with the TD3 algorithm agent to obtain sensor control data, and the sensor control data is stored as a sample in an experience playback pool; samples are extracted based on the experience playback pool to train the TD3 algorithm The agent uses the trained agent to decide the optimal action for sensor path planning;

The optimal action is applied to the sensor, and the sensor position is obtained after state transition of the sensor, thereby obtaining the extended target sensor measurement value at the current moment, and filtering prediction and update are performed to perform tracking and estimation of the extended target.

Preferably, modeling for the elliptical expansion target includes:

Set the state of extended target tracking at time k as: ζ _k = (x _k ,X _k ), where x _k represents the kinematic state of the target, and X _k represents the expanded state of the target;

The approach to modeling is:

Among them, w _k is the zero-mean Gaussian process noise, v _k is the zero-mean Gaussian measurement noise, x _s,k (π) is the sensor position at the current moment, Mapping system state evolution,/> For measurement mapping, x _k+1 represents the kinematic state of the target at k+1 moment,/> Represents multiple measurement values at time k.

Preferably, the expansion state of the expansion target at time k is modeled as an elliptical shape, which is described by a positive definite symmetric matrix X _k as:

Among them, θ _k is the direction angle of the elliptical shape, σ _k,1 and σ _k,2 are the major axis and minor axis of the elliptical shape respectively.

Preferably, a virtual interactive environment with deep reinforcement learning is constructed according to the extended target filtering algorithm, including:

Based on the neural network to fit the value function and policy function in the deep reinforcement learning, the deep reinforcement learning algorithm is used to control the sensor through the exploration and utilization mechanism, an intelligent sensor control system is established, and the virtual interaction is constructed through the intelligent sensor control system Environment; the extended target filtering algorithm includes a prediction process and an update process.

Preferably, the prediction process is:

Among them, F _k|k-1 is the state transition matrix, I _d is the d-dimensional identity matrix, P _k|k-1 is the prediction covariance matrix, D _k|k-1 is the covariance matrix of zero-mean Gaussian process noise, x _k|k-1 is the one-step prediction value, x _k-1|k-1 is the filter update value at time k-1, and P _k-1|k-1 is the corresponding covariance matrix.

Preferably, the TD3 algorithm agent includes:

Actor network: used to select actions based on status;

Target Actor network: used to select actions based on the status again based on the results obtained by the Actor network;

Critic network: used to evaluate the actions selected by the Actor network;

Target Critic network: used to evaluate the action selected by the Actor network again based on the results obtained by the Critic network.

Preferably, obtaining the sensor control data includes:

After the agent takes action and acts on the virtual interactive environment at any time, the sensor moves from the state x _s, k at time k to the state x _{s,k+1 at time k+1} , and is evaluated through the reward function The reward value R _k+1 is obtained, and then the agent continuously improves the strategy based on the reward value, and finally learns the optimal strategy to determine the actions of the sensor at each moment.

Preferably, the method of constructing the reward function includes:

Define the extended target k time prior probability distribution and posterior probability distribution to obey the multivariate Gaussian distribution, obtain the Gauss Wasserstein distance between the prior probability distribution and the posterior probability distribution, and construct the above based on the Gauss Wasserstein distance reward function.

Preferably, the reward function is:

Among them, a _k,0 indicates that the sensor is in a stationary state at the current moment.

Compared with the existing technology, the present invention has the following advantages and technical effects:

The method of the present invention uses a random matrix to model the expanded state of the elliptical expanded target, which can effectively estimate the target motion state and expanded state, and then uses the setting of the evaluation function in a sensor management method similar to information theory to construct and apply it to deep reinforcement learning TD3 The reward function in the algorithm, this reward function comprehensively considers the joint optimization of the target motion state and contour information (extended state). After using the TD3 algorithm to effectively control the sensor in the continuous action space, compared with sensorless control, it can not only It is more accurate in estimating the position of the target center of mass, and at the same time, it is also more accurate in estimating the target contour information, so the tracking effect of the elliptical extended target is optimized as a whole.

Description of the drawings

The drawings that form a part of this application are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an improper limitation of this application. In the attached picture:

Figure 1 is a schematic diagram of the sensor control trajectory in the embodiment of the present invention;

Figure 2 is an error curve diagram of the semi-major axis and semi-minor axis of the elliptical expansion target according to the embodiment of the present invention;

Figure 3 is a centroid estimation error graph according to the embodiment of the present invention;

Figure 4 is a schematic diagram of the extended target GW distance controlled by the Changan device according to the embodiment of the present invention;

Figure 5 is a flow chart of a sensor management method based on deep reinforcement learning in the expanded target of the embodiment of the present invention;

Figure 6 is a schematic diagram of the network connection relationships in the TD3 algorithm agent according to the embodiment of the present invention.

Detailed ways

It should be noted that, as long as there is no conflict, the embodiments and features in the embodiments of this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and embodiments.

It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and, although a logical sequence is shown in the flowchart, in some cases, The steps shown or described may be performed in a different order than here.

The present invention proposes a sensor management method based on deep reinforcement learning in extended goals, as shown in Figure 5, including:

Model the elliptical extended target and build a virtual interactive environment with deep reinforcement learning based on the extended target filtering algorithm;

(1) Extended target tracking problem description:

When tracking an extended target, in addition to tracking the motion state of the target center of mass, it also tracks the extended state of the target, that is, the evolution of the target's shape over time. The state of extended target tracking at time k is expressed as: ζ _k = (x _k , X _k ), where x _k represents the kinematic state of the target and obeys the multivariate Gaussian distribution. X _k represents the extended state of the target and obeys the inverse Wishart distribution. The system is modeled as:

Among them, w _k is zero-mean Gaussian process noise, and v _k is zero-mean Gaussian measurement noise. x _s,k (π) is the sensor position at the current moment, is the system state evolution map,/> is a measurement mapping.

The extended state of the extended target at time k is modeled as an elliptical shape and is described by a positive definite symmetric matrix X _k as:

Among them, θ _k is the direction angle of the elliptical shape, σ _k,1 and σ _k,2 are the major axis and minor axis of the elliptical shape respectively.

(2) Extended target filtering algorithm:

The extended target filtering algorithm is implemented under the framework of Bayesian filtering algorithm and consists of a prediction process and an update process. Each process is divided into prediction and update of motion state and extended state:

1) Forecasting process:

One-step prediction of motion status, since it obeys the multivariate Gaussian distribution, its mean and covariance matrix are as follows:

Among them, F _k|k-1 is the state transition matrix, I _d is the d-dimensional identity matrix, P _k|k-1 is the covariance matrix, and D _k|k-1 is the covariance matrix of zero-mean Gaussian process noise.

One-step prediction of extended state:

v _k|k-1 ＝e ^-T/τ v _k-1|k-1 (6)

Among them, v _k|k-1 and V _k|k-1 are respectively the degrees of freedom and the inverse scale matrix in the inverse Wishart distribution obtained at time k based on the posterior prediction at time k-1, T represents the sampling time, and τ is the time attenuation Constant, d represents the dimension of the target extended state, v _k-1|k-1 and V _k-1|k-1 represent the posterior of k-1 moment, that is, the degree of freedom and inverse of k-1 moment obtained after iterative update. Scale matrix, E[X _k|k-1 ] represents the mathematical expectation of X _k|k-1 .

2)Update process:

Movement status updates:

in, Represents the center of mass measurement,/> represents the corresponding scattering matrix. W _k|k-1 represents the system gain matrix, ε _k is the innovation part of the system measurement, and S _k|k-1 represents the covariance matrix of the innovation part.

Update on extension status:

v _k|k ＝v _k|k-1 +n _k (16)

Among them, n _k is the measurement number.

It is assumed that the extended target moves in a straight line at a uniform speed, and the system equation is established according to formula (1). At the same time, the shape of the extended target is modeled as an ellipse by formulas (2)-(3). The environment for interacting with the reinforcement learning agent is set by the extended objective filtering algorithm of Equations (4)-(19). For the interactive environment, that is, input the posterior estimate of the extended target motion state at time k-1 and the covariance matrix x _k-1|k-1 , P _k-1|k-1 and the posterior estimate v of the extended state _k-1|k-1 ,V _k-1|k-1 , and the real-time position of the sensor, the posterior value x _k|k ,P _k|k at time k is obtained through the filtering algorithm of formula (4)-(19), v _k|k ,V _k|k .

The extended target tracking problem has been studied in different fields such as radar and computer vision. Its performance depends on the relative geometry of the observer (measurement sensor) and the moving target, so the task of sensor management is selected as sensor trajectory planning. According to the framework shown in Figure 1, first the elliptical extended target is modeled and a virtual interactive environment with deep reinforcement learning is constructed based on the extended target tracking algorithm. The neural network is used to fit the value function and strategy function in reinforcement learning, and deep reinforcement is used. The learning algorithm performs sensor control through mechanisms such as exploration and utilization to build an intelligent sensor control system.

The network structure is constructed according to the TD3 algorithm, which contains a total of 6 networks (as shown in Figure 6): Actor network, target Actor network, two Critic networks and the target Critic network. The network training algorithm is represented by formulas (20)-(27) ) is given. So far, the two main subjects, namely the environment and the reinforcement learning agent, have been built. The next step is to train the agent, and finally obtain the optimal strategy and then perform target tracking based on sensor measurements. The target Actor network has the same function as the Actor network, and similarly, the target Critic network has the same function as the Critic network. When updating network parameters, it will involve calculating the action taken in the next state and the value of the state action after the action is taken in the next state. The purpose of setting the target network is to suppress the Q caused by reusing the original network, that is, "bootstrapping" The problem of too high value. In addition, two Critic networks and a target Critic network are set up because when updating the network, the operation of maximizing will also lead to the problem of overestimation of the Q value, which can be effectively suppressed by selecting a smaller value for update through two different Critic networks.

The DDPG algorithm is an important deep reinforcement learning algorithm for dealing with complex, continuous control problems. However, the DDPG algorithm will have the problem of overestimating the Q value in the Critic network. Therefore, to address this problem, the Double Delay Deep Deterministic Policy Gradient (TD3) algorithm is used. The DDPG algorithm is optimized through three parts, effectively suppressing the problem of excessive Q value. Therefore, in order to improve the performance optimization of the deep reinforcement learning algorithm for extended target tracking, the present invention performs sensor control in an environment using continuous tasks based on the TD3 algorithm.

(3)TD3 algorithm

Reinforcement learning problems contain two subjects: the agent and the environment. In the extended target sensor control based on deep reinforcement learning, the interactive environment is the elliptical extended target filter algorithm, and the reinforcement learning agent is trained to perform intelligent sensor control. Reinforcement learning problems can often be modeled using a Markov decision process (MDP), represented as a quintuple is a finite set of states, that is, all possible states that the agent explores in the environment. Use s to represent the state at the current moment, s′ to represent the state at the next moment, and the specific state is the position of the sensor in the coordinate system. /> is a limited action set, that is, a set of all possible actions that an agent can take based on the current state. Let a represent the currently taken action. At this time, it is the sensor path planning. After fixing the sensor speed, the sensor can choose the direction angle of movement. . P is the state transition function, which is the probability of the sensor transitioning from the current state s to the next state s′. /> It is a reward function, which represents the expected reward that the sensor can obtain after taking action based on the current position status. γ is the discount factor, which represents the proportion of the value of future expected rewards at the current moment.

The process of interaction between the agent and the environment is that after the agent takes action a _k and acts on the environment at time k, the sensor moves from the state x _s,k at time k to the state x _{s,k+1 at time k+1} . According to the reward function The reward value R _k+1 is obtained through evaluation, and then the agent continuously improves the strategy based on the reward value, and finally learns the optimal strategy to determine the sensor's action a _k at each moment.

In MDP, the value function includes two types: state value function and action value function (Q function). In the TD3 algorithm, the action value function is used, which means that under the guidance of the policy function π, the agent takes action a according to the state s. Expected rewards to be obtained. Reinforcement learning algorithms can be divided into two categories according to whether the environment model is known: model-based methods and model-free methods. Since most of the actual problem environments are complex and unknown, This makes it difficult to model the environment, so the model-free method is more widely used. Model-free methods can be divided into policy-based methods, value-based methods, and the Actor-Critic (AC) method that combines the two.

The TD3 algorithm for deep reinforcement learning is a model-free AC-based method that uses neural networks to fit the value function and policy function. Among them, Actor represents a network based on the policy function, which is used to select actions based on the state, and Critic represents a network based on the value function, which is used to evaluate the actions selected by the Actor network. Assume that the parameters are θ ^μ and Actor network and Critic network, the target Actor network with parameters θ ^μ′ and the parameter /> and/> The two-target Critic network. The Critic network is updated using gradient descent. The update formula is described as:

Updating the Actor network through gradient descent can be described as:

Among them, N is the number of batch samples sampled for each learning, and α and β are the learning rate. For the target network using soft update, the update formula can be described as:

θ ^μ′ ←τθ ^μ +(1-τ)θ ^μ′ (27)

The agent outputs the sensor action based on the current sensor position at k time according to the current agent's network parameters, and then obtains the sensor k+1 time position based on this action, and brings the k+1 time position into the filtering environment to obtain the sensor measurement value. This is used to perform extended target tracking pseudo-update, so that the reward value of the action taken at time k can be obtained according to the designed reward function given by formula (29). The data is divided into (x _s,k ,a _k ,r _k+1 ,x stored in the form of _s,k+1 ). Many moments will be involved in target tracking, that is, k∈{0,1,2...T}, until the end of the tracking moment. In this process, the environment will continue to interact with the reinforcement learning agent, and many pieces of data in the form of (x _s,k ,a _k ,r _k+1 ,x _s,k+1 ) will be obtained, which will be stored in In the experience replay pool, set a certain experience replay pool capacity. When the data exceeds this capacity, the old data will be discarded and new data will be used to fill the experience replay pool.

For the construction of the reward function in deep reinforcement learning: select the information gain between the extended target prior probability density and the posterior probability density to design the reward function, and use the Gaussian Wasserstein distance to measure this information gain. At this time, the Gauss-Wasserstein (GW) distance between the prior probability density and the posterior probability density is used to comprehensively evaluate the extended target tracking estimation effect. The larger the GW distance, the greater the information gain. The reward function is used to guide the enhancement. Learning the optimal strategy for the agent can avoid the problem of sparse rewards that make it difficult for the agent to converge.

(4)Design of reward function

Since target tracking involves many moments, that is, k∈{0,1,2...T}, at each moment, a small batch of data is extracted from the experience replay pool, and the network update method according to the TD3 algorithm is formula (20)-( 27) Update network parameters. Continuously extract data from the experience replay pool, set a certain number of training times, iteratively update the network parameters, and finally converge to the optimum. The control agent makes the optimal strategy based on the sensor position at k time to obtain the sensor action.

In elliptical extended target tracking, the position component and extended state of the motion state can be defined as a multivariate Gaussian distribution, which is used to describe the overall effect of extended target tracking, expressed as: N _x ~ N (m _x ,s∑ _x ), where , m _x is determined by the position component of x _k , s∑ _x represents the range, s is the scaling factor, take s=1, and ∑ _x is determined by the random matrix X _k . Then it is defined that the prior probability distribution and posterior probability distribution at moment k of the extended target both obey the multivariate Gaussian distribution, which is described as: The Gaussian Wasserstein distance between the two is:

Based on this, the reward function is:

Among them, a _k,0 indicates that the sensor is in a stationary state at the current moment.

The reinforcement learning agent trained at each moment is used to finally select its action at the next moment based on the sensor position at the current moment to obtain the best measurement. The new measurement is obtained according to formulas (9)-(10), and finally brought into Go to the filtering algorithm (6)-(19) to obtain the final extended target tracking result. Through iteration of the above steps, the optimal sensor path is finally obtained to optimize the overall performance of elliptical extended target tracking.

Figure 2 shows the estimation error between the semi-major axis and the semi-minor axis of the target expanded state estimation and the real target after intelligent planning of the sensor trajectory based on the TD3 algorithm in extended target tracking. It can be seen that the estimation error of the semi-major axis All are below 1.0, the estimation errors of the semi-minor axis are all below 0.5, and the shape estimation of the extended target is relatively accurate.

Figure 3 shows the centroid estimation error between the extended target state estimate and the true target state under the sensorless control scheme and the TD3-based control scheme. As can be seen from Figure 3, after the sensor control method based on deep reinforcement learning is added to the extended target, the centroid tracking estimate of the extended target is more accurate.

Figure 4 shows the Gauss Wasserstein distance between the extended target estimate and the real extended target under the sensorless control scheme and the TD3-based control scheme. This indicator comprehensively considers the overall performance of the target motion state and extended state estimation. As can be seen from Figure 4, adding the sensor control method based on deep reinforcement learning to the extended target improves the overall performance of the extended target tracking. Not only does the center of mass estimation become more accurate, but the tracking estimate of the target contour information is also closer to the target. True shape.

This invention uses a random matrix to model the expanded state of the elliptical expanded target, which can effectively estimate the target motion state and expanded state, and then uses the setting of the evaluation function in a sensor management method similar to information theory to construct the deep reinforcement learning TD3 algorithm. The reward function in The target center of mass position is more accurately estimated, and the target contour information is also more accurately estimated, so the tracking effect of the elliptical extended target is optimized as a whole.

The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application. All are covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims (8)

1. The sensor management method based on deep reinforcement learning in the extended goal is characterized by: Model the elliptical extended target and build a virtual interactive environment with deep reinforcement learning based on the extended target filtering algorithm; Establish TD3 algorithm intelligent agent; The virtual interactive environment interacts with the TD3 algorithm agent to obtain sensor control data, and the sensor control data is stored as a sample in an experience playback pool; samples are extracted based on the experience playback pool to train the TD3 algorithm The agent uses the trained agent to decide the optimal action for sensor path planning; Wherein, obtaining the sensor control data includes: After the agent takes action and acts on the virtual interactive environment at any time, the sensor moves from the state x _s, k at time k to the state x _{s,k+1 at time k+1} , and is evaluated through the reward function Obtain the reward value R _k+1 , and then the agent continuously improves the strategy based on the reward value, and finally learns the optimal strategy to determine the actions of the sensor at each moment; The optimal action is applied to the sensor, and the sensor position is obtained after state transition of the sensor, thereby obtaining the extended target sensor measurement value at the current moment, and filtering prediction and update are performed to perform tracking and estimation of the extended target. 2. The sensor management method based on deep reinforcement learning in the extended target according to claim 1, characterized in that modeling the elliptical extended target includes: Set the state of extended target tracking at time k as: ζ _k = (x _k ,X _k ), where x _k represents the kinematic state of the target, and X _k represents the expanded state of the target; The approach to modeling is: Among them, w _k is zero-mean Gaussian process noise, v _k is zero-mean Gaussian measurement noise, x _{s, k} (π) is the sensor position at the current moment, f _k : is the system state evolution mapping, h _k :/> For measurement mapping, x _k+1 represents the kinematic state of the target at k+1 moment,/> Represents multiple measurement values at time k. 3. The sensor management method based on deep reinforcement learning in the extended target according to claim 2, characterized in that the extended state of the extended target at time k is modeled as an elliptical shape, and is described by a positive definite symmetric matrix X _k as: Among them, θ _k is the direction angle of the elliptical shape, σ _k,1 and σ _k,2 are the major axis and minor axis of the elliptical shape respectively. 4. The sensor management method based on deep reinforcement learning in the extended target according to claim 1, characterized in that, according to the extended target filtering algorithm, a virtual interactive environment with deep reinforcement learning is constructed, including: Based on the neural network to fit the value function and policy function in the deep reinforcement learning, the deep reinforcement learning algorithm is used to control the sensor through the exploration and utilization mechanism, an intelligent sensor control system is established, and the virtual interaction is constructed through the intelligent sensor control system Environment; the extended target filtering algorithm includes a prediction process and an update process. 5. The sensor management method based on deep reinforcement learning in the extended target according to claim 4, characterized in that the prediction process is: Among them, F _k|k-1 is the state transition matrix, I _d is the d-dimensional identity matrix, P _k|k-1 is the prediction covariance matrix, D _k|k-1 is the covariance matrix of zero-mean Gaussian process noise, x _k|k-1 is the one-step prediction value, x _k-1|k-1 is the filter update value at time k-1, and P _k-1|k-1 is the corresponding covariance matrix. 6. The sensor management method based on deep reinforcement learning in the extended target according to claim 1, characterized in that the TD3 algorithm agent includes: Actor network: used to select actions based on status; Target Actor network: used to select actions based on the status again based on the results obtained by the Actor network; Critic network: used to evaluate the actions selected by the Actor network; Target Critic network: used to evaluate the action selected by the Actor network again based on the results obtained by the Critic network. 7. The sensor management method based on deep reinforcement learning in the extended target according to claim 1, characterized in that the method of constructing the reward function includes: Define the extended target k time prior probability distribution and posterior probability distribution to obey the multivariate Gaussian distribution, obtain the Gauss Wasserstein distance between the prior probability distribution and the posterior probability distribution, and construct the above based on the Gauss Wasserstein distance reward function. 8. The sensor management method based on deep reinforcement learning in the extended target according to claim 7, characterized in that the reward function is: Among them, a _k,0 indicates that the sensor is in a stationary state at the current moment.