CN115618497A - An airfoil optimization design method based on deep reinforcement learning - Google Patents

Abstract

The present invention proposes an airfoil optimization design method based on deep reinforcement learning. This method is different from supervised learning, but has the characteristics of independent learning strategy and the largest long-term reward. It is an optimization method closer to intelligence, and It has the portability and the universality of the strategy. If the design conditions are changed within a certain range, such as incoming flow Mach number, Reynolds number, etc., the strategy obtained from the original optimization can still provide the initial optimization direction, and the optimization goal can be significantly improved within a very short number of steps.

Description

An airfoil optimization design method based on deep reinforcement learning

technical field

The invention belongs to the field of aircraft design, and proposes an airfoil optimization design method based on deep reinforcement learning.

Background technique

As the main component of the aircraft, the airfoil not only improves the lift required for flight, but also ensures the stability and operability of the aircraft. With the understanding and exploration of the aerodynamic performance of the airfoil, multiple airfoil libraries have also been established accordingly. At present, there is no super airfoil that fully satisfies all design states. It is necessary to initially select a benchmark airfoil that meets the requirements from the airfoil library based on factors such as design use and working conditions. Based on the benchmark airfoil, it is continuously improved in combination with design requirements. base airfoil. For improving the reference airfoil, the initial method is to use the wind tunnel to conduct repeated experiments, which will consume huge material and manpower.

Since the beginning of the 21st century, the advancement of science and technology has injected new vitality into Computational Fluid Dynamics (CFD) technology, and CFD technology has quickly become the main means of analyzing and solving fluid problems. The use of computational fluid dynamics technology can greatly shorten the optimization cycle and save a lot of manpower and material resources, which allows designers to perform repeated calculations until the desired airfoil is achieved, such as the extensive application of evolutionary and genetic algorithms in aerodynamic optimization problems. However, the evolutionary and genetic algorithms have a low utilization rate for the large amount of computational data generated in the optimization process.

The application of machine learning in fluid mechanics has developed rapidly in recent years. At present, the most common method of machine learning in optimization design is the rapid performance prediction method based on response surface, which uses existing data to construct the mapping between input and output to speed up the optimization process. Partially replacing the "role" played by CFD technology in the optimization process belongs to supervised learning. Unlike supervised learning, reinforcement learning builds a mapping from the state parameters of the environment to action parameters, and its purpose is to iteratively update the strategy for interacting with the environment to obtain the maximum cumulative reward. At this time, it is different from the real output that is required to be predicted in supervised learning, but corresponds to the reward of a certain action. The problem of airfoil design is not to pursue the optimum of a certain performance alone, but to achieve the optimum of the overall performance after repeated weighing and comparison. It is a complex system engineering, which often needs to be manually modified by combining the experience and understanding of the engineer himself. . The training process of reinforcement learning is very similar to the process of engineers accumulating experience. Analogous to the traditional trial-and-error method, the agent responsible for making actions in reinforcement learning continuously takes different "actions" during the training process and observes the benefits of the design results for a period of time in the future (or after taking a series of actions), Update the policy to take action. As the payoff increases, the agent can be considered to gain the same design experience as the engineer to a certain extent. In terms of airfoil optimization, Li Runze used reinforcement learning to optimize the pressure distribution of the airfoil to reduce the resistance of the transonic airfoil. Viquerat conducted optimization attempts with constraints and without constraints. In the initial stage, there is a possibility that the agent cannot find its direction and falls into a local optimum. In general, applying reinforcement learning to airfoil optimization in aircraft aerodynamic optimization design can effectively improve optimization efficiency and has a wide range of application scenarios.

Contents of the invention

Due to the low optimization efficiency of the current CFD-based method, in order to improve the efficiency of airfoil optimization, the present invention proposes an airfoil optimization design method based on deep reinforcement learning. This method is different from supervised learning, but has an independent learning strategy. The biggest feature of long-term rewards is a more intelligent optimization method with transferability, or the universality of strategies.

Technical scheme of the present invention is:

Described a kind of airfoil optimization design method based on depth reinforcement learning, comprises the following steps:

Step 1: Use the free-form surface deformation method to parameterize the geometry of the airfoil: establish a free-form surface deformation control frame around the reference airfoil, establish the mapping relationship between the control frame and the airfoil, and obtain a new airfoil by changing the position of the control frame points type;

Step 2: Establish an optimized design model, and confirm the single design goals and constraints according to the flight requirements, where the design goals and constraints are various aerodynamic parameters of the airfoil, such as lift coefficient, drag coefficient and airfoil thickness, etc., and the design goals and constraints expressed in mathematical expressions. A general single-objective optimization problem can be written in the following mathematical form:

Minimize: f(x)

subject to:g _w (x)≥0,w=1,2,···,W

h _r (x)=0,r=1,2,...,R

Among them, x is the design variable, f(x) is the objective function, g _w (x) is the inequality constraint, there are W in total; h _r (x) is the equality constraint, there are R in total.

Step 3: Establish a reward function according to the optimization objective and constraint conditions, where the total reward value is obtained by the linear sum of the reward values of each aerodynamic parameter, where the reward value is increased when the target is reached, the reward value is not increased if the constraint is satisfied, and the reward value is reduced if the constraint is not satisfied. At the same time, the target reward value and constraint reward value are multiplied by different coefficients to balance the magnitude difference between the target and constraint.

将翼型中有关设计目标和约束条件的气动参数作为状态，包括翼型的升力系数、阻力系数、最大厚度和力矩系数。将基准机翼的气动参数作为初始状态s₀。Step 4: Establish an agent, including a policy model π and a value function model. The policy function model can output an action strategy, and the value function model can output an advantage estimate and a value function. Both the policy model and the value function model use a two-layer hidden layer Artificial neural network model, the number of hidden layer nodes is 64. Initialize policy model parameters θ ₀ and value function model parameters

The aerodynamic parameters related to design goals and constraints in the airfoil are taken as the state, including the lift coefficient, drag coefficient, maximum thickness and moment coefficient of the airfoil. The aerodynamic parameters of the reference wing are taken as the initial state s ₀ .

Step 5: The current agent is given an action a by the policy model according to the state and reward value, which is the new design variable.

Step 6: Perform actions on the airfoil to get a new airfoil.

Step 7: Establish a structured grid for the new airfoil, use the open source solver CFL3D to perform numerical simulation of the flow around the airfoil, the main control equation is the Reynolds average N-S equation, the turbulence model is the k-ωSST model, and the lift coefficient of the airfoil is calculated , drag coefficient, maximum thickness and moment coefficient, confirm that the aerodynamic parameters of the new airfoil are in the new state.

Step 8: Calculate the reward value from the calculated aerodynamic parameters according to the reward function in step 3.

Step 9: Repeat steps 5-8 for a total of e-1 times from the current policy model to obtain the trajectory and reward value {r _e } including the state and action obtained in each cycle, trajectory τ={s ₀ ,a ₀ ,· ··,s _e-1 ,a _e-1 ,s _e }, where s ₀ and a ₀ are the initial state and action, s _e is the e-step state, s _e-1 is the e-1 step state, a _{e- 1} is e-1 step action.

Step 10: Repeat steps 5-9 a total of n-1 times based on the current policy model to obtain n trajectories and reward values.

Step 11: According to the obtained n trajectory parameters and reward values, calculate the advantage estimate based on the value function model of the current agent, that is, the difference between the expected reward of each action a and the mean value of the expected reward of all possible actions in this state value.

优化目标为损失函数最小，用优化后的参数更新策略模型和价值函数模型，得到新的策略模型和价值函数模型。Step 12: Construct a loss function based on the advantage estimate, trajectory and reward value, and optimize the policy model parameters θ and value function model parameters based on the stochastic gradient descent algorithm

The optimization goal is to minimize the loss function, and the optimized parameters are used to update the strategy model and value function model to obtain a new strategy model and value function model.

Step 13: Repeat steps 5-12 until the loss function is no longer reduced, and the training is completed.

Beneficial effect

1. The airfoil optimization design method based on deep reinforcement learning proposed by the present invention has the characteristics of self-learning strategy and long-term reward maximum. Compared with genetic algorithm and other algorithms, only a large amount of calculation data is used as the evaluation standard of optimization goals and constraints, and the depth reinforcement Learning and trying to learn the experience gained in the optimization process can improve the efficiency of data utilization, reduce the amount of calculation, and improve the efficiency of optimization.

2. The airfoil optimization design method based on deep reinforcement learning proposed by the present invention, the strategy model obtained by deep reinforcement learning optimization has a certain degree of universality or transferability: if the design conditions are changed within a certain range, such as incoming flow Mach number, Reynolds number, etc. The strategy obtained from the original optimization can still provide the initial optimization direction, and the optimization goal can be significantly improved within a short number of steps, which is a feature and advantage that traditional optimization methods do not have.

Additional aspects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and comprehensible from the description of the embodiments in conjunction with the following drawings, wherein:

Fig. 1 is the overall flow chart of the airfoil optimization method of the present invention.

Figure 2 is the control frame of the free-form surface deformation method.

Figure 3 is a schematic diagram of the airfoil design space.

Figure 4 is a schematic diagram of a fully connected network.

Figure 5 shows the reinforcement learning optimization process.

Figure 6 shows the overall view and partial view of the mesh of the airfoil.

Fig. 7 is the airfoil optimization resistance convergence curve of the proximal optimization strategy (PPO) in this paper.

Fig. 8 is the airfoil optimization drag convergence curve of the comparison strategy non-dominated sorting genetic algorithm (NSGA-II).

Figure 9 is the airfoil optimization drag convergence curve using the pre-trained PPO strategy.

detailed description

Embodiments of the present invention are described in detail below, and the embodiments are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

Step 1: In this example, RAE2822 is used as the reference airfoil, and the free-form surface deformation method (FFD) is used to parameterize the reference airfoil. The control frame of the free-form surface deformation method is shown in Figure 2. The design variables are controlled by the upper and lower airfoils It consists of 14 points, and the perturbation range of each design point in the y direction is ±0.02. The design space of the airfoil is shown in Figure 3.

Step 2: The design goal is to minimize the resistance of the airfoil in the design state, while satisfying three constraints: (1) the lift coefficient does not decrease, (2) the absolute value of the moment coefficient of the airfoil does not increase, (3) the airfoil The maximum thickness does not decrease. The mathematical expression of the optimization design problem is as follows:

Minimize: C _d

Subject to: C _l ≥ C _l0

|C _m |≤|C _m0 |

t≥t ₀

Among them, C _d is the drag coefficient of the airfoil, C _l is the lift coefficient of the airfoil, C _l0 is the lift coefficient of the reference airfoil, C _m is the moment coefficient of the airfoil, C _m0 is the moment coefficient of the reference airfoil, and t is the airfoil , t ₀ is the thickness of the reference airfoil.

Step 3: Construct the reward function according to the design goals and constraints in step 2 as follows:

If C _d -C _d0 <0, reward _cd =100×(C _d -C _d0 ).

If C _l -C _l0 <0 then reward _cl =10×(C _l -C _l0 ), otherwise let reward _cl =0.

If C _m -C _m0 <0, set reward _cm =100×(C _m -C _m0 ), otherwise set reward _cm =0.

If tt ₀ <0, set reward _t =50×(tt ₀ ), otherwise set reward _t =0.

The final reward value reward = reward _cd + reward _cl + reward _cm + 100 reward _t .

where reward _cd is the reward value of the drag coefficient, reward _cl is the reward value of the lift coefficient, reward _cm is the reward value of the moment coefficient, reward _t is the reward value of the airfoil thickness, and 10, 50 and 100 are used to balance the magnitude between the target and the constraint gap.

Step 4: Establish an agent, including a policy model π and a value function model. The policy function model outputs an action strategy, while the value function model outputs an advantage estimate and a value function. The models all use an artificial neural network model with two hidden layers, based on PyTorch Write, the number of hidden layer nodes is 64.

将翼型的升力系数、阻力系数、最大厚度和力矩系数作为状态，这里将基准机翼的升力系数、阻力系数、最大厚度和力矩系数作为初始状态s₀。强化学习优化流程中各部分关系如图5所示。Artificial neural network is a kind of feedforward neural network, which connects multiple neuron nodes together, and the output of each layer is used as the input of the next layer. Through weight calculation, it forms a computing structure with grouping level. Schematic diagram of fully connected network As shown in Figure 4. Initialize policy model parameters θ ₀ and value function model parameters

Take the lift coefficient, drag coefficient, maximum thickness and moment coefficient of the airfoil as the state, and here take the lift coefficient, drag coefficient, maximum thickness and moment coefficient of the reference wing as the initial state s ₀ . The relationship between various parts in the reinforcement learning optimization process is shown in Figure 5.

Step 5: The current agent is given an action by the policy model according to the state and reward value, which is the new airfoil design variable.

Step 6: Perform actions on the airfoil to get a new airfoil.

Step 7: Establish a C-shaped structured grid for the two-dimensional new airfoil. Figure 6 shows the overall and partial grid of the airfoil. The calculation condition is Ma=0.734, Re=6.5×10 ⁶ , α=2.79°, where Ma is the Mach number of the incoming flow, Re is the Reynolds number of the incoming flow, and α is the angle of attack of the incoming flow. When the influence of physical force and external heating is not considered, the open source solver CFL3D is used for numerical simulation calculation of the flow around the airfoil. The main control equation is the Reynolds average NS equation, and the turbulence model is the k-ωSST turbulence model. Drag coefficient c _d and moment coefficient c _m , lift coefficient c _l and airfoil thickness t. Confirm that the aerodynamic parameters of the new airfoil are in the new state.

Step 8: Calculate the reward value from the calculated c _d , _cm , c _l and t according to the reward function in step 3.

Step 9: Repeat steps 5-8 for a total of e-1 times from the current policy model to obtain the trajectory and reward value {r _e } including the state and action obtained in each cycle, trajectory τ={s ₀ ,a ₀ ,· ··,s _e-1 ,a _e-1 ,s _e }, where s ₀ and a ₀ are the initial state and action, s _e is the e-step state, s _e-1 is the e-1 step state, a _{e- 1} is e-1 step action.

Step 10: Repeat steps 5-9 based on the current policy model to obtain a total of n trajectories and reward values.

Step 11: According to the obtained trajectory parameters, calculate the advantage estimate based on the current value function model. The advantage estimate is the interpolation between the expected reward of each action a and the average value of the expected reward of all actions in this state. If the advantage estimate is greater than 0, it means The action a is better than average, otherwise inferior to average.

优化目标为损失函数最小，用优化后的参数更新策略模型和价值函数模型，得到新的策略模型和价值函数模型。Step 12: Construct a loss function based on the advantage estimate, trajectory and reward value, and optimize the policy model parameters θ and value function model parameters based on the stochastic gradient descent algorithm

The optimization goal is to minimize the loss function, and the optimized parameters are used to update the strategy model and value function model to obtain a new strategy model and value function model.

Step 13: Repeat steps 5-12 until the loss function is no longer reduced, and the training is completed.

Proximal Policy Optimization (PPO) is a deep reinforcement learning algorithm. Figure 7 shows the optimal resistance convergence curve of the proximal optimization strategy in this paper. It can be seen that the airfoil resistance has been significantly improved. Compared with the initial airfoil RAE2822, the resistance is reduced by about 50 frames, which is about 30% improved. In order to verify the optimization effect of PPO in the single-objective constrained aerodynamic optimization design, the non-dominated sorting genetic algorithm (NSGA-II) is used here as a comparison algorithm. The population size of NSGA-II is 100, the evolution algebra is 100, and the optimization convergence The curve is shown in Figure 8; there is still a gap between the optimization effect of PPO and NSGA-II, so the pre-training method is used to improve the optimization effect of PPO. Pre-training is based on models that have been trained in similar problems, which can provide a better initial point for optimization. Pre-training can avoid falling into local optimum and get better optimization results.

The network is pre-trained, and the optimization convergence curve of the pre-trained PPO is shown in Figure 9. The optimization results obtained by the two optimization methods and the required calculation amount are compared in Table 1. It can be observed that the optimized convergence trend is significantly more stable than the genetic algorithm, and the convergence trend of the two is the same. In the case of similar resistance optimization effects, the calculation amount of the pre-trained PPO strategy in this paper is 0.05 times that of the genetic algorithm, and the calculation amount has been significantly reduced. decrease. As shown in Table 2, when the fixed calculation amount is 500, the resistance optimization effect of the PPO strategy and the pre-trained PPO strategy in this paper is also far better than the genetic algorithm. It can be seen that the airfoil optimization design method based on deep reinforcement learning of the present invention can significantly reduce the amount of calculation compared with the genetic algorithm, and at the same time, it is not easy to fall into local optimum after pre-training.

Table 1

Table 2

Next, change the design conditions to illustrate that the strategy obtained by reinforcement learning has certain universality: keep the basic airfoil unchanged, and the design state changes from Ma=0.734, Re=6.5×10 ⁶ , α=2.79° to 1# respectively: Ma=0.72, Re=1×10 ⁷ , α=2.79°, 2#: Ma=0.76, Re=8×10 ⁶ , α=2.79° and 3#: Ma=0.6, Re=6.5×10 ⁶ , α = 2.79°. In order to verify the effectiveness of the strategy, the existing strategy is directly used to optimize the three states for 5 steps, 10 steps, 15 steps and 20 steps, and observe the change of resistance. The results are shown in Table 3. It can be seen that in the three design states, after 20 steps of optimization, the resistance of 1# decreased by 7.4%; the resistance of 2# decreased by 3.7%; the resistance of 3# decreased by 6.8%, and the optimization of the three design states The resulting constraints all meet the requirements. All in all, reinforcement learning has learned a certain universal strategy, which can provide direction and preliminary optimization results for optimization.

table 3

Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be construed as limitations to the present invention. Variations, modifications, substitutions, and modifications to the above-described embodiments are possible within the scope of the present invention.

Claims (8)

1. A wing section optimization design method based on deep reinforcement learning is characterized in that: the method comprises the following steps:

step 1: carrying out geometrical parameterization on the airfoil profile by using a free-form surface deformation method: establishing a free-form surface deformation control frame around the reference wing profile, establishing a mapping relation between the control frame and the wing profile, and obtaining a new wing profile by changing the position of a control frame point;

step 2: establishing an optimized design model, and confirming a single design target and a constraint condition according to flight requirements;

and step 3: establishing a reward function according to the optimization target and the constraint condition;

and 4, step 4: establishing an agent comprising a strategy model pi and a value function model, wherein the strategy function model outputs an action strategy, and the value function model can output advantage estimation and a value function; initializing a policy model parameter θ ₀ And cost function model parameters

Taking aerodynamic parameters of airfoil design target and constraint condition as states, wherein the aerodynamic parameters corresponding to the reference airfoil are taken as initial states s ₀ ；

And 5: the current intelligent agent gives actions according to the state and the reward value by a strategy model to obtain new wing section design variables;

and 6: performing action on the wing profile to obtain a new wing profile;

and 7: establishing a structured grid for the obtained new airfoil profile, carrying out numerical simulation on the flow around the new airfoil profile, and calculating to obtain the pneumatic parameters of the new airfoil profile as a new state;

and 8: calculating to obtain a reward value according to the reward function in the step 3 by using the pneumatic parameters obtained by calculation in the step 7;

and step 9: repeating the steps 5-8 for e-1 times by the current strategy model loop to obtain the track containing the state and the action obtained by each loop and the reward value { r } _e }, trace τ = { s = ₀ ，a ₀ ，…，s _e-1 ，a _e-1 ，s _e In which s is ₀ And a ₀ For initial states and actions, s _e Is in the state of e step, s _e-1 Is in the e-1 step state, a _e-1 Step e-1;

step 10: repeating the step 5-9 for n-1 times based on the current strategy model to obtain n tracks and reward values;

step 11: calculating advantage estimation based on a value function model of the current agent according to the obtained n track parameters and the reward values;

step 12: constructing a loss function based on the dominance estimates, the trajectories and the reward values,optimizing a policy model parameter θ and a cost function model parameter

The optimization target is that the loss function is minimum, and the strategy model and the value function model are updated by the optimized parameters to obtain a new strategy model and a new value function model;

step 13: and (5) circularly repeating the steps 5-12 until the loss function is not reduced any more, and finishing the training.

2. The airfoil profile optimization design method based on the deep reinforcement learning as claimed in claim 1, wherein: in step 2, the design objective and the constraint condition are aerodynamic parameters of the airfoil profile.

3. The airfoil optimization design method based on deep reinforcement learning according to claim 2, characterized in that: in step 2, the design goal is to minimize the resistance of the airfoil profile in the design state and simultaneously satisfy three constraint conditions: the lift coefficient is not reduced, (2) the absolute value of the moment coefficient of the airfoil is not increased, and (3) the maximum thickness of the airfoil is not reduced;

the mathematical expression for the optimization design problem is as follows:

Minimize：C _d

Subject to：

|C _m |≤|C _m0 |

t≥t ₀

wherein C _d Is the airfoil drag coefficient, C _l Is the wing profile lift coefficient,

Lift coefficient, C, of the reference airfoil _m Moment coefficient of airfoil profile, C _m0 As the moment coefficient of the reference airfoil profile, t as the thickness of the airfoil profile, t ₀ Is the thickness of the reference airfoil.

4. The airfoil optimization design method based on deep reinforcement learning according to claim 3, characterized in that: in step 3, the total reward value is obtained by linear summation of the reward values of the pneumatic parameters, wherein the reward value is increased when the target is reached, the reward value is not increased when the constraint is met, the reward value is reduced when the constraint is not met, and different coefficients are multiplied between the reward value of the target and the reward value of the constraint to balance the magnitude difference between the target and the constraint.

5. The airfoil profile optimization design method based on the deep reinforcement learning as claimed in claim 4, wherein: constructing the reward function according to the design goals and constraint conditions in step 2 is as follows:

if C is present _d -C _d0 If < 0, reward _cd ＝100×(C _d -C _d0 )；

If it is not

Then

Otherwise, the instruction reward _cl ＝0；

If C is _m -C _m0 If < 0, command reward _cm ＝100×(C _m -C _m0 ) Otherwise, command reward _cm ＝0；

If t-t ₀ If < 0, command reward _t ＝50×(t-t ₀ ) Otherwise, command reward _t ＝0；

Final prize value reward = reward _cd +reward _cl +reward _cm +100reward _t ；

Wherein reward _cd Reward value of drag coefficient _cl Reward value for coefficient of lift _cm Rewarded for the value of the moment coefficient reward _t The value is awarded for the airfoil thickness.

6. The airfoil profile optimization design method based on the deep reinforcement learning as claimed in claim 1, wherein: in step 4, the strategy model and the value function model both use an artificial neural network model with two hidden layers, and the number of nodes of the hidden layers is 64.

7. The airfoil optimization design method based on deep reinforcement learning according to claim 1, characterized in that: and 7, performing numerical simulation on the new airfoil profile by using an open source solver CFL3D, wherein the main control equation is a Reynolds average N-S equation, and the turbulence model is a k-omega SST model, and calculating to obtain the pneumatic parameters of the new airfoil profile.

8. The airfoil optimization design method based on deep reinforcement learning according to claim 1, characterized in that: the advantage estimate is the difference between the expected reward for each action a and the mean of the expected rewards for all possible actions in that state.

Images