CN116165886A - Multi-sensor intelligent cooperative control method, device, equipment and medium - Google Patents

Abstract

The invention discloses a multi-sensor intelligent cooperative control method, a device, equipment and a medium, wherein the method comprises the following steps: establishing a reinforcement learning agent model, wherein a state space of the reinforcement learning agent model comprises a global comprehensive situation expression and a single sensor state embedded representation, and an action space of the reinforcement learning agent model comprises action output values abstracted by different tasks executed by a plurality of sensors; training the reinforcement learning intelligent agent model through sampling, and guiding the reinforcement learning intelligent agent model to learn through rewarding modeling engineering, wherein the training comprises centering training and decentralizing execution; the reinforcement learning agent model obtained through loading training cooperatively controls a plurality of sensors. The invention realizes cross-means cross-region cooperative control of multiple sensors by using reinforcement learning technology.

Description

Multi-sensor intelligent collaborative control method, device, equipment and medium

Technical Field

The present invention belongs to the field of artificial intelligence technology, and in particular relates to a multi-sensor intelligent collaborative control method, device, equipment and medium.

Background Art

At present, the electromagnetic spectrum space presents the characteristics of strong confrontation. Faced with a large number of complex radiation sources with different characteristics, when conducting detection activities on key target signals in the network and electronic space, it is necessary to coordinate and dispatch multiple sensors of the same or different types (such as ultra-short wave, microwave, electronic and portable radars, etc.) across regions, give full play to their respective advantages, realize the instant discovery and positioning of multiple targets, and meet the needs of long-term continuous tracking.

Traditional manual decision-making and control methods have the problems of high latency, low fault tolerance, and are unable to comprehensively consider the situation of high-dimensional complex signals and the optimal coordinated scheduling of multi-sensor resources. In addition, most key target signals have short burst characteristics, making it difficult to achieve effective and continuous positioning and tracking of radiation sources.

Deep reinforcement learning faces the problem of sparse reward signals and low sampling rate, which leads to low sample utilization, slow learning speed and even difficulty in training convergence. This shortcoming is particularly prominent in a complex electromagnetic spectrum environment with strong confrontation and multiple radiation sources. In addition, since some key signals have characteristics such as concealment, suddenness, and phases, and different signals have different characteristics that cannot be detected by a single means (i.e. a single type of sensor), the task of locating and tracking radiation sources is obscure and difficult to disassemble.

Summary of the invention

The purpose of the present invention is to overcome the defects of the prior art and provide a multi-sensor intelligent collaborative control method, device, equipment and medium, which can collaboratively control multiple sensors through artificial intelligence technology in a complex multi-radiation source environment, thereby realizing instant detection and positioning of multiple targets and meeting the needs of long-term continuous tracking of targets.

The object of the present invention is achieved through the following technical solutions:

A multi-sensor intelligent collaborative control method, the method comprising:

Establishing a reinforcement learning agent model corresponding to each sensor, wherein the state space of the reinforcement learning agent model includes a global comprehensive situation expression and a single sensor state embedding representation, and the action space of the reinforcement learning agent model includes action output values abstracted from different tasks performed by multiple sensors;

Training the reinforcement learning agent model through sampling, and guiding the reinforcement learning agent model to learn through reward shaping engineering, wherein the training includes centralized training and decentralized execution;

The trained reinforcement learning agent model collaboratively controls multiple sensors.

Furthermore, the training of the reinforcement learning agent model by sampling specifically includes:

Open multiple sampling threads to perform independent sampling in multiple parallel simulation environments or parallel real environments with different configuration scenarios;

Put the sampled data into the sampled experience buffer pool;

When the training conditions are met, the sampled data is taken out from the sampled experience cache pool to update the agent model according to the corresponding reinforcement learning algorithm centralized training, and then the updated model parameters are placed in the model parameter cache pool.

Furthermore, guiding the learning of the reinforcement learning agent model through the reward shaping project specifically includes:

Set up human-design rewards, end-game rewards, and curiosity rewards;

The artificially designed rewards include giving corresponding reward values when a preset task is completed or when a task fails;

The end-game mode reward includes awarding rewards based on overall signal detection effectiveness;

The curiosity reward includes giving a reward when an unknown space is explored.

Furthermore, the confidence region strategy optimization method used in the method to establish the reinforcement learning agent model specifically includes:

The confidence region strategy optimization algorithm is simplified using a first-order approximation form, and the confidence region strategy optimization algorithm is simplified as follows:

The corresponding constraints are:

为旧策略的优势函数；Where π is the new strategy; π _old is the old strategy; S is the state; a is the action,

is the advantage function of the old strategy;

State-action-value function:

State value function:

Advantage function:

为新旧策略的KL散度的平均值；Where γ is the attenuation factor,

is the average value of the KL divergence of the new and old strategies;

The simplified confidence region strategy optimization algorithm is approximated using the Monte Carlo method to obtain

用于表示新旧策略的比率，得到make

Used to express the ratio of the new and old strategies, we get

The constraints of the confidence region policy optimization algorithm are approximated as r _t (θ)∈[1-∈,1+∈], where ∈ is the clip coefficient. Then the objective function of the confidence region policy optimization algorithm with constraints can be expressed as the unconstrained objective function:

Adding the objective function of the state value function and the entropy of the policy model to the unconstrained objective function yields the complete objective function:

C₁、C₂分别为预先设置的对应项的系数。in,

C ₁ and C ₂ are preset coefficients of corresponding items respectively.

Furthermore, the centralized training and decentralized execution specifically include:

A central controller collects the global states of all agents and makes unified decisions;

Each sensor performs its own task asynchronously according to its current state.

Furthermore, the method also includes guiding the learning of the reinforcement learning agent model based on an expert strategy of generative adversarial imitation learning.

Furthermore, the expert strategy based on generative adversarial imitation learning guides the learning of the reinforcement learning agent model specifically including:

Repeat the following steps until the optimal strategy is obtained:

Use the reinforcement learning agent model corresponding to the current sensor to interact with the environment to obtain the agent generation trajectory;

The agent-generated trajectory and the demonstration trajectory are input into the discriminator together and the discriminator parameters are updated in a supervised learning manner;

The updated discriminator outputs a new discrimination reward function;

The updated reward function is used to provide a reward signal to further update the agent strategy.

On the other hand, the present invention also provides a multi-sensor intelligent collaborative control device, the device comprising:

An agent model building module, wherein the agent model building module builds a reinforcement learning agent model corresponding to each sensor, wherein the state space of the reinforcement learning agent model includes a global comprehensive situation expression and a single sensor state embedding representation, and the action space of the reinforcement learning agent model includes action output values abstracted from different tasks performed by multiple sensors;

An agent model training module, wherein the agent model training module trains the reinforcement learning agent model by sampling and guides the reinforcement learning agent model to learn by reward shaping engineering, wherein the training includes centralized training and decentralized execution;

A sensor control module, wherein the sensor control module collaboratively controls multiple sensors through a trained reinforcement learning agent model.

On the other hand, the present invention also provides a computer device, which includes a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement any one of the above-mentioned multi-sensor intelligent collaborative control methods.

On the other hand, the present invention also provides a computer-readable storage medium, in which a computer program is stored. The computer program is loaded and executed by a processor to implement any of the above-mentioned multi-sensor intelligent collaborative control methods.

The beneficial effects of the present invention are:

(1) The present invention realizes the coordinated control of multiple sensors by establishing a reinforcement learning intelligent agent model, which enables multiple sensors to have corresponding processing capabilities for complex signal situations in complex working environments.

(2) The present invention utilizes reinforcement learning technology and generative adversarial imitation learning technology based on expert knowledge to achieve continuous positioning and tracking of key signals in complex electromagnetic spectrum environments.

(3) The present invention can control cross-regional multiple sensors to asynchronously execute their respective tasks, has the ability to capture and locate short burst signals, and has a certain degree of continuous positioning and tracking capability.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a multi-sensor intelligent collaborative control method provided by an embodiment of the present invention;

FIG2 is a schematic diagram of a state space of an embodiment of the present invention;

FIG3 is a logic diagram of the operation of the intelligent agent training module according to an embodiment of the present invention;

FIG4 is a schematic diagram of the architecture of an intelligent agent training system according to an embodiment of the present invention;

FIG5 is a diagram of a distributed architecture framework according to an embodiment of the present invention;

FIG6 is a schematic diagram of a multi-agent learning system according to an embodiment of the present invention;

7 is a schematic diagram of an expert strategy guided learning method for generative adversarial imitation learning according to an embodiment of the present invention;

FIG8 is a structural block diagram of a multi-sensor intelligent collaborative control device provided in an embodiment of the present invention.

DETAILED DESCRIPTION

The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

Traditional manual decision-making and control methods have the problems of high latency, low fault tolerance, and are unable to comprehensively consider the situation of high-dimensional complex signals and the optimal coordinated scheduling of multi-sensor resources. In addition, most key target signals have short burst characteristics, making it difficult to achieve effective and continuous positioning and tracking of radiation sources.

Deep reinforcement learning faces the problem of sparse reward signals and low sampling rate, which leads to low sample utilization, slow learning speed and even difficulty in training convergence. This shortcoming is particularly prominent in a complex electromagnetic spectrum environment with strong confrontation and multiple radiation sources. In addition, since some key signals have characteristics such as concealment, suddenness, and phases, and different signals have different characteristics that cannot be detected by a single means (i.e. a single type of sensor), the task of locating and tracking radiation sources is obscure and difficult to disassemble.

In order to solve the above technical problems, the following embodiments of the multi-sensor intelligent collaborative control method, device, equipment and medium of the present invention are proposed.

Example 1

This embodiment aims at locating and tracking the signals of key radiation sources in a complex electromagnetic spectrum environment, and uses reinforcement learning technology to achieve cross-means and cross-regional collaborative control of multiple sensors, optimize resource scheduling and decision-making speed, improve positioning precision and accuracy, and further improve the continuous positioning capability of some short burst signal radiation sources.

Referring to FIG. 1 , FIG. 1 is a flow chart of a multi-sensor intelligent collaborative control method provided by this embodiment, and the method specifically includes the following steps:

Step 1: Establish a reinforcement learning agent model corresponding to each sensor.

Specifically, the state space of the reinforcement learning agent model includes the global comprehensive situation expression and the embedded representation of the state of a single sensor, and the action space of the reinforcement learning agent model includes the action output values abstracted from different tasks performed by multiple sensors.

The state space of the reinforcement learning agent model in this embodiment is composed of the state vectors of sensors, targets, and signals. In this solution, the state space is an N×D dimensional matrix. N represents the number of elements in the current environment, including sensors, targets, and signals. When the number of environmental elements is less than N, the remaining positions are padded with 0. D represents the length of the vector, and the vectors between different elements will be padded to the same length by padded with 0.

The following part of the present invention will describe the specific implementation method by means of the coordinated control of three detection device sensors in three different areas. It should be understood that the number of sensors and areas here is not intended to limit the present invention.

Referring to Figure 2, as shown in Figure 2, it is a schematic diagram of the state space of this embodiment. The state space of this embodiment consists of two parts: a global comprehensive situation expression and a single sensor state embedding representation, wherein the global neutralization situation expression includes the target, signal, and global operation and maintenance status, respectively. In the target comprehensive situation embedding representation, 0 represents that the current intelligent agent has not successfully located the signal, so there is no longitude and latitude, and 1 represents that the target is located; taking three detection sensors in three different areas as an example, since positioning requires at least two sensors in different areas, the sensor ID supporting positioning, 0 represents invalid data, 1 represents that the sensor between areas (1,2) is positioned, 2 represents that the sensor between areas (1,3) is positioned, 3 represents that the sensor between areas (2,3) is positioned, and 4 represents that the sensors between areas (1,2,3) are all positioned. The signal comprehensive situation embedding representation consists of all the signal situations currently detected, specifically including frequency, bandwidth, amplitude, azimuth, and key signal identification. The discrete embedding representation of the global operation and maintenance status consists of the working status of all current sensors, including task type, task status, current wide-scan frequency band (fill 0 if none), and current control frequency point (fill 0 if none). The embedding representation of a single sensor status includes the current sensor signal situation data (frequency, bandwidth, amplitude, direction, key signal identification), historical fixed-frequency data (frequency, bandwidth, identification), the current working status of the sensor (task type, task status, current wide-scan frequency band, current control frequency point), and the set of wide-scan frequency points of similar sensors (composed of a binary array, the first dimension is the frequency, and the second dimension is whether it is the signal identification scanned by the sensor itself).

This embodiment takes the ultrashort wave detection sensor as an example, and the action space of the reinforcement learning agent is shown in Table 1. For the continuous actions therein, the corresponding specific action output value is obtained by sampling from the corresponding Gaussian distribution.

Table 1. Action space of reinforcement learning agent

Step 2: Train the reinforcement learning agent model through sampling, and guide the reinforcement learning agent model to learn through reward shaping engineering. The training includes centralized training and decentralized execution.

The multi-sensor intelligent collaborative control method for radiation source positioning and tracking consists of a task planning module and an agent training module. The task planning module gives different sensor collaboration schemes by loading different agent models, such as resource priority, positioning priority, tracking priority, etc.

Agent training is the core module of this embodiment, which mainly trains the corresponding agent by interacting with simulation software or real environment.

3 and 4 , FIG3 is a running logic diagram of the intelligent agent training module of the present embodiment, and FIG4 is a schematic diagram of the architecture of the intelligent agent training system of the present embodiment.

The agent training of this embodiment is mainly divided into a sampling part and a training part, in which the blue side is the radiation source side, and its corresponding blue side trajectory generation submodule (what trajectory the radiation source moves with) and the blue side decision submodule (when the signal carried by the radiation source is switched on and off) are generated by rules; the red side is the sensor agent side. The red side model is an agent model, and the blue side model is a rule model. In this embodiment, based on the gRPC distributed architecture, a distributed sampling centralized training scheme is adopted for the agent training module to solve the low sampling rate problem faced by deploying PPO reinforcement learning agents in a complex electromagnetic spectrum environment. Referring to Figure 5, as shown in Figure 5, it is a distributed architecture framework diagram of this embodiment. Multiple sampling threads are opened to perform independent sampling in multiple parallel simulation environments with different configuration scenarios. The sampled data is then uniformly placed in the sampling experience cache pool. When the training conditions are met, the sampled data is taken out from the sampling experience cache pool to update the agent model according to the centralized training of the corresponding reinforcement learning algorithm, and then the updated model parameters are placed in the model parameter cache pool. The sampling module will periodically take out the latest model parameters from the model parameter cache pool to update the agent model, thereby updating the agent strategy, and use the updated strategy for sampling.

At this point, the reinforcement learning training technology based on distributed sampling centralized training has completed the cycle of sampling, training and model iteration, and has greatly improved the learning efficiency of intelligent agents by maximizing the use of physical hardware resources to increase the sampling rate.

Aiming at the difficulty of cross-regional multi-sensor collaborative control and taking into account the characteristics of cross-regional multi-sensors performing their respective tasks asynchronously, a centralized training and decentralized execution (CTDE) semi-multi-agent PPO reinforcement learning algorithm was proposed based on the idea of multi-intelligence reinforcement learning.

Reinforcement learning algorithms generally have a certain degree of volatility, which has a great impact on the training process and the final training performance. In order to ensure that the policy model improves monotonically during optimization, the Trust Region Policy Optimization (TRPO) algorithm uses KL divergence to measure the difference between the new and old policies, constructs a target formula similar to the natural gradient method, and uses this goal to continuously optimize the policy to effectively prevent large fluctuations caused by noise in the policy gradient, and uses the conjugate gradient method to reduce the amount of calculation of the Fisher information matrix. However, as a second-order method, the TRPO algorithm still requires a lot of computational cost. The PPO algorithm further simplifies the objective function based on TRPO, uses a first-order approximation form, and speeds up the training while ensuring accuracy.

First, the objective function of the TRPO algorithm can be organized as:

The corresponding constraints are:

为旧策略的优势函数，定义如下(γ为衰减因子)：Where π is the new strategy; π _old is the old strategy; s is the state; a is the action,

is the advantage function of the old strategy, defined as follows (γ is the decay factor):

State-action-value function:

State value function:

Advantage function:

为新旧策略的KL散度的平均值。

is the average value of the KL divergence between the new and old strategies.

Although TRPO uses the conjugate gradient method to reduce the amount of computation required to solve the complex and constrained objective function, it still requires a lot of computational cost and the algorithm efficiency is low. For this reason, the PPO algorithm further optimizes the objective function.

In practical applications, the expected calculation is often approximated using the Monte Carlo method, so the objective function of the TRPO algorithm becomes:

用于表示新旧策略的比率，则TRPO算法的目标函数进一步变为：make

Used to represent the ratio of the new and old strategies, the objective function of the TRPO algorithm further becomes:

The constraints of TRPO are approximated as r _t (θ) ∈ [1-ε, 1+ε], where ε is the clip coefficient. Then the constrained TRPO objective function can be expressed as the unconstrained objective function:

The common gradient descent method is then used to solve the problem. While maintaining the advantage of the TRPO algorithm in improving the strategy steadily, the first-order method greatly reduces the amount of calculation and improves the efficiency of the algorithm. The objective function of the state value function (VF) and the entropy (S) of the strategy model are further added to the final objective function, so the complete objective function of PPO becomes:

C₂分别为对应项的系数，在本实施例中默认C₁＝0.001，C₂＝0。in,

C ₂ are coefficients of the corresponding items. In this embodiment, C ₁ =0.001 and C ₂ =0 by default.

Target signal positioning and tracking in a complex electromagnetic spectrum environment requires collaboration between multiple sensors across regions. There are three common frameworks for multi-agent reinforcement learning: centralized training and centralized execution (CTCE), centralized training and decentralized execution (DTCE), and decentralized training and decentralized execution (DTDE). For centralized training, a central controller is required to collect the global state of all agents and make unified decisions, so centralized training can ensure better collaboration. However, the sensors located in different areas have different observation ranges and different numbers and types of observed signals, so each sensor asynchronously performs its own tasks according to its current state. Referring to Figure 6, Figure 6 is a schematic diagram of the multi-agent learning system of this embodiment. This embodiment uses a centralized training and decentralized execution (DTCE) framework to deploy a multi-agent learning system.

In order to further utilize expert prior knowledge (i.e., expert strategy) to accelerate agent learning and further solve the problem of low sample utilization rate of reinforcement learning methods, the present invention proposes an expert strategy guided learning method based on generative adversarial imitation learning (GAIL). Inspired by the successful application of nonlinear loss functions in generative adversarial networks and inverse reinforcement learning, generative adversarial imitation learning can learn strategies directly from expert trajectories. The specific implementation method is shown in Figure 7.

在供给有限的示范数据集情况下，逆强化学习很容易过拟合。因此采用一个损失函数正则化器ψ(c)来避免过拟合。如果分配给专家示范中的动作状态对较小的损失值，ψ(c)会对损失函数施加轻微惩罚，反之施加较大惩罚。于是本发明中，逆强化学习的目标函数就可以表示为：The goal of inverse reinforcement learning is to fit a loss function c from the potential function family C: ^RS×A ={c:S×A→R}, which minimizes the expected cumulative loss on the trajectory demonstrated by the expert and is larger for the trajectory generated by any other strategy. Considering that in a complex electromagnetic spectrum environment with multiple radiation sources, there is a huge set of loss functions due to the high dimensionality of the environment.

In the case of a limited supply of demonstration data sets, inverse reinforcement learning is prone to overfitting. Therefore, a loss function regularizer ψ(c) is used to avoid overfitting. If the action state assigned to the expert demonstration has a small loss value, ψ(c) will impose a slight penalty on the loss function, otherwise a larger penalty will be imposed. Therefore, in the present invention, the objective function of inverse reinforcement learning can be expressed as:

Further, ψ(c) takes the form of expectation of expert data, which can be expressed as:

in

And define the occupancy of the strategy as:

Where γ is the decay factor in the Markov decision process. Occupancy can be interpreted as the distribution of state-action pairs encountered by the agent when interacting with the environment under strategy π. Therefore, the decayed return expectation of the trajectory generated by interacting with the environment under different strategies can be expressed as:

It can be proved that firstly recovering a loss function through inverse reinforcement learning and then learning the strategy through reinforcement learning can be expressed as:

是ψ(c)的凸共轭形式；ρ_π和

是策略π和专家策略π_E的占用度。Where λ is the weight of the policy entropy H(π);

is the convex conjugate form of ψ(c); ρ _π and

is the occupancy of strategy π and expert strategy π _E.

In GAIL, the loss function is set as:

c(s,a)＝log(D(s,a))

Where D:S×A→(0,1) is a discriminator. c(s,a) provides a reward signal for updating the agent's strategy. It can be further proved that:

Finally, the expert strategy-guided learning method based on Generative Adversarial Imitation Learning (GAIL) can be summarized as solving the saddle point of the above equation.

的形式被训练用于将专家策略采样轨迹τ_E中的状态动作对(s,a)～τ_E和智能体生成轨迹τ_agent中的状态动作对(s,a)～τ_agent区分开；生成器(即智能体策略π)通过最大化E_π[log(D(s,a))]让鉴别器将智能体状态动作对“误判”为专家采样轨迹状态动作对；

为策略的γ衰减因果熵。The discriminator D(s,a) is constructed by minimizing

The form of is trained to distinguish the state-action pairs (s, a) ~ τ _E in the expert strategy sampling trajectory τ _E and the state-action pairs (s, a) ~ τ _agent in _{the agent-generated trajectory τ agent} ; the generator (i.e., the agent strategy π) maximizes E _π [log(D(s, a))] to make the discriminator "misjudge" the agent state-action pair as the expert sampling trajectory state-action pair;

is the γ-decay causal entropy of the policy.

Referring to FIG. 7 , FIG. 7 is a schematic diagram of the expert strategy guided learning method for generative adversarial imitation learning in this embodiment. In each new iteration round: (1) the generator uses the current sensor agent strategy to interact with the environment to obtain the agent generation trajectory; (2) the agent generation trajectory and the demonstration trajectory are input into the discriminator together and the discriminator parameters are updated in a supervised learning manner; (3) the updated discriminator outputs a new identification reward function; (4) the updated reward function is used to provide a reward signal to further update the agent strategy (i.e., the generator). The above steps are repeated continuously, and the generator and discriminator continuously optimize their respective performance through adversarial training until the ideal strategy is learned.

This embodiment solves the problem that the positioning and tracking task is difficult to disassemble, and guides the intelligent agent to learn through reward shaping engineering based on the idea of hierarchical reinforcement learning. The specific reward method is shown in Table 2.

Table 2 Reward Summary

Among them, rewards 1-7 are artificially designed rewards, which are set around finding signals, locating targets, and controlling and detecting balance. The specific reward values and calculation processes will be continuously adjusted during the actual experiment. Reward 8 is a final mode reward, and the reward calculation is based on the results of the overall signal detection efficiency evaluation. Reward 9 is more special and is a curiosity reward. From the perspective of the direction of the agent's exploration of the state space in reinforcement learning, the following rewards and punishments are additionally given so that the agent has a better search state space. Traditional reinforcement learning algorithms have extremely low sample utilization in an environment with sparse feedback, and the learning speed is slow and difficult to converge. In the present invention, sensors in different areas in most states are in a signal search state, and only when the target signal is scanned can the corresponding positive reward be obtained. In addition, in some scenarios, there is a group of states with a high transition probability, but the transition rate between this group of states and other states is low. Therefore, a curiosity reward is designed to give additional rewards to states with low exploration rates to encourage the exploration of unknown spaces, or to avoid some spaces from being unexplored.

Specifically, the reward and punishment r(S _known ,S _novel ) is designed as a curiosity reward function, and the agent that explores the unknown state is rewarded according to the currently explored state space and the currently unexplored state space. First, in the agent training, a dedicated memory storage module is designed to record the probability of each state s _t appearing during the training process: φ(s _t ; m), where m represents the memory storage module parameters, and the memory storage network parameters are updated after training. After the curiosity reward is enabled, each time the agent takes an action and moves to a new state, the corresponding reward and punishment are given: Asign(k-φ(s _t ,m _-1 )), where A is the reward weight; k is the reward threshold. When the corresponding state transition probability is greater than k, it is punished, otherwise it is rewarded.

Step 3: The trained reinforcement learning agent model collaboratively controls multiple sensors.

Different intelligent agent models are loaded through the task planning module to give different sensor coordination schemes to achieve coordinated control of multiple sensors. In a complex multi-radiation source environment, by executing different working modes (wideband scanning, fixed-frequency control, direction finding and positioning), the radiation source target can be instantly discovered and continuously positioned and tracked.

The multi-sensor intelligent collaborative control method provided in this embodiment has the ability to process complex signal situations. In response to the need for positioning and tracking of key radiation source signals in complex electromagnetic spectrum environments, deep learning is combined with reinforcement learning to achieve comprehensive processing of the global comprehensive situation (target comprehensive situation, signal comprehensive situation, global operation and maintenance status) and the status of a single sensor. By fine-grained settings of the action space of different sensors and the internal representation of a single action, the reinforcement learning technology and the generative adversarial imitation learning technology based on expert knowledge are comprehensively utilized to achieve continuous positioning and tracking of key signals in complex electromagnetic spectrum environments. It has the ability to capture and locate short burst signals, and has a certain degree of continuous positioning and tracking capabilities.

Example 2

Referring to FIG. 8 , FIG. 8 is a structural block diagram of a multi-sensor intelligent collaborative control device provided in this embodiment, and the device specifically includes:

An agent model building module, which builds a reinforcement learning agent model corresponding to each sensor. The state space of the reinforcement learning agent model includes a global comprehensive situation expression and a single sensor state embedding expression. The action space of the reinforcement learning agent model includes action output values abstracted from different tasks performed by multiple sensors.

Agent model training module: The agent model training module trains the reinforcement learning agent model through sampling and guides the learning of the reinforcement learning agent model through reward shaping engineering. The training includes centralized training and decentralized execution;

The sensor control module collaboratively controls multiple sensors through the trained reinforcement learning agent model.

The multi-sensor intelligent collaborative control device provided in this embodiment realizes the collaborative control of multiple sensors by establishing a reinforcement learning intelligent agent model, which enables multiple sensors to have corresponding processing capabilities for complex signal situations in complex working environments. Utilizing reinforcement learning technology and generative adversarial imitation learning technology based on expert knowledge, continuous positioning and tracking of key signals in complex electromagnetic spectrum environments can be achieved. It can control the asynchronous execution of tasks between multiple sensors across regions, has the ability to capture and locate short burst signals, and has a certain continuous positioning and tracking capability.

Example 3

This preferred embodiment provides a computer device, which can implement the steps in any embodiment of the multi-sensor intelligent collaborative control method provided in the embodiments of the present application. Therefore, the beneficial effects of the multi-sensor intelligent collaborative control method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

Example 4

Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention provides a storage medium, which stores multiple instructions, and the instructions can be loaded by a processor to execute the steps of any embodiment of the multi-sensor intelligent collaborative control method provided in the embodiment of the present invention.

The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

Since the instructions stored in the storage medium can execute the steps in any multi-sensor intelligent collaborative control method embodiment provided in the embodiments of the present invention, the beneficial effects that can be achieved by any multi-sensor intelligent collaborative control method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. The intelligent cooperative control method for the multiple sensors is characterized by comprising the following steps of:

establishing a reinforcement learning agent model corresponding to each sensor, wherein a state space of the reinforcement learning agent model comprises global comprehensive situation expression and single sensor state embedded representation, and an action space of the reinforcement learning agent model comprises action output values abstracted by different tasks executed by a plurality of sensors;

training the reinforcement learning intelligent agent model through sampling, and guiding the reinforcement learning intelligent agent model to learn through rewarding modeling engineering, wherein the training comprises centering training and decentralizing execution;

the reinforcement learning agent model obtained through training cooperatively controls a plurality of sensors.

2. The multi-sensor intelligent cooperative control method according to claim 1, wherein the training of the reinforcement learning agent model by sampling specifically includes:

starting a plurality of sampling threads to independently sample in a plurality of parallel simulation environments or parallel real environments with different configuration scenes respectively;

uniformly placing the sampling data into a sampling experience buffer pool;

when the training conditions are met, sampling data are taken out from the sampling experience buffer pool, the intelligent body model is updated according to the corresponding reinforcement learning algorithm centralized training, and then the updated model parameters are put into the model parameter buffer pool.

3. The multi-sensor intelligent cooperative control method of claim 1, wherein the guiding the reinforcement learning agent model learning through a bonus modeling project specifically comprises:

setting artificial design rewards, final office mode rewards and curiosity rewards;

the artificial design rewards comprise corresponding rewards when a preset task is completed or the task fails;

the final pattern rewards include awarding rewards based on overall signal detection effectiveness;

the curiosity rewards include awarding rewards when an unknown space is explored.

4. The multi-sensor intelligent cooperative control method according to claim 1, wherein the confidence region strategy optimization method adopted when the method establishes the reinforcement learning agent model specifically comprises the following steps:

simplifying a confidence region strategy optimization algorithm by using a first-order approximation form, wherein the confidence region strategy optimization algorithm is simplified as follows:

the corresponding constraint conditions are:

wherein pi is a new policy; pi _old Is an old strategy; s is the state; a is the action of the device,

is a dominance function of the old strategy;

state action value function:

state value function:

dominance function:

wherein, gamma is an attenuation factor,

the average value of KL divergence of the new strategy and the old strategy;

approximating the simplified confidence region strategy optimization algorithm by using a Monte Carlo method to obtain

Order the

For representing the ratio of new strategy to old strategy, get

Approximating the constraint of the confidence region policy optimization algorithm to r _t (θ)∈[1-∈,1+∈]Where e is a clip coefficient, the constrained confidence region policy optimization algorithm objective function may represent an unconstrained objective function:

adding the entropy of the objective function of the state value function and the entropy of the strategy model to the unconstrained objective function to obtain a complete objective function:

wherein ,

C ₁ 、C ₂ the coefficients of the corresponding items are preset respectively.

5. The multi-sensor intelligent cooperative control method of claim 1, wherein the performing of the centering training and the decentralizing specifically comprises:

collecting global states of all agents and making a unified decision by a central controller;

and each sensor asynchronously executes each task according to the current state of the sensor.

6. The multi-sensor intelligent cooperative control method of claim 1, further comprising guiding learning of the reinforcement learning agent model based on generating expert policies against imitation learning.

7. The multi-sensor intelligent cooperative control method of claim 1, wherein the learning based on the expert strategy generating the anti-imitation learning to guide the reinforcement learning agent model specifically comprises:

repeating the following steps until an optimal strategy is obtained:

interaction between the reinforcement learning agent model corresponding to the current sensor and the environment is utilized to obtain an agent generation track;

the intelligent agent generates a track and an demonstration track to be input into the discriminator together and updates the parameters of the discriminator in a supervised learning mode;

the updated discriminator outputs a new discrimination reward function;

the agent policy is further updated with the updated reward function to provide a reward signal.

8. A multi-sensor intelligent cooperative control device, the device comprising:

the system comprises an intelligent body model building module, a multi-sensor intelligent body model processing module and a multi-sensor intelligent body model processing module, wherein the intelligent body model building module builds a reinforcement learning intelligent body model corresponding to each sensor, a state space of the reinforcement learning intelligent body model comprises global comprehensive situation expression and single sensor state embedded representation, and an action space of the reinforcement learning intelligent body model comprises action output values abstracted by different tasks executed by a plurality of sensors;

the intelligent body model training module is used for training the reinforcement learning intelligent body model through sampling and guiding the reinforcement learning intelligent body model to learn through rewarding modeling engineering, and the training comprises centering training and decentralizing execution;

and the sensor control module cooperatively controls the plurality of sensors through the reinforcement learning intelligent body model obtained through training.

9. A computer device, characterized in that it comprises a processor and a memory in which a computer program is stored, which computer program is loaded and executed by the processor to implement the multi-sensor intelligent cooperative control method according to any of claims 1 to 7.

10. A computer readable storage medium, characterized in that the storage medium has stored therein a computer program that is loaded and executed by a processor to implement the multi-sensor intelligent cooperative control method according to any one of claims 1 to 7.

Images