CN119728454B - Network flow simulation method, device, equipment and medium for large model training - Google Patents

Abstract

The invention relates to the technical field of computers, and provides a network flow simulation method, device, equipment and medium for large model training, wherein the method comprises the steps of obtaining configuration information of a user; the method comprises the steps of defining a network topological structure and training parameters of a large model to be trained according to configuration information, generating a communication load matrix based on the training parameters and the network topological structure, wherein the communication load matrix is used for representing calculation time and data transmission requirements of each network node in the network topological structure, and executing flow simulation according to the communication load matrix to simulate network flow in a model training process of the large model to be trained. The network topology structure and the training parameters of the model are defined through the configuration information of the user, so that the user can flexibly adjust the network structure to adapt to the simulation of the network flow of training clusters with different scales and structures, and the flexibility and the applicability of the simulation of the network flow of model training are improved.

Description

Network traffic simulation method, device, equipment and medium for large model training

Technical Field

The present invention relates to the field of computer technology, and in particular to a network traffic simulation method, device, equipment and medium for large model training.

Background Art

With the rapid development of deep learning and artificial intelligence technologies, training large-scale machine learning models has become a core requirement for promoting technological progress in various industries. Such models usually contain hundreds of millions or even hundreds of billions of parameters, and the training process relies on distributed computing clusters. In the distributed training process, network communication plays a key role, responsible for data synchronization between nodes and the transmission of model parameters. Efficient network communication requires the network topology to have high bandwidth, low latency, and good load balancing capabilities to ensure that data transmission during training can be carried out in a timely and stable manner.

In the existing technology, general network simulation tools are used to simulate distributed training, which has achieved certain results in terms of network performance in distributed training environments, but there are still significant deficiencies in the specific needs of large-scale model training. This is mainly reflected in the definition and configuration of the network topology structure, which is difficult for users to flexibly adjust, limiting its applicability in training clusters of different sizes.

Summary of the invention

The present invention provides a network traffic simulation method, device, equipment and medium for large-scale model training, which are used to solve the defect that in the prior art, a general network simulation tool is used to simulate distributed training. According to the specific needs of large-scale model training, it is difficult for users to flexibly adjust the definition and configuration of the network topology structure, which limits the applicability of network simulation in training clusters of different scales.

The present invention provides a network traffic simulation method for large model training, comprising the following steps:

Get the user's configuration information;

Define the network topology and the training parameters of the large model to be trained according to the configuration information;

Generate a communication load matrix based on the training parameters and the network topology; the communication load matrix is used to characterize the computing time and data transmission requirements of each network node in the network topology;

Traffic simulation is performed according to the communication load matrix to simulate the network traffic during the model training process of the large model to be trained.

According to the network traffic simulation method for large model training provided by the present invention, the network topology structure and the training parameters of the large model to be trained are defined according to the configuration information, including:

Determine the number of nodes according to the network configuration information in the configuration information, and set the network nodes according to the number of nodes; the network configuration information includes node configuration information;

Configuring key parameters of the network nodes based on the node configuration information to define a network topology; the key parameters include a connection mode, link bandwidth, and link delay between the network nodes;

The training parameters of the large model to be trained are defined according to the model configuration information in the configuration information; the training parameters include model parameter scale, training parallel strategy and training batch size.

According to the network traffic simulation method for large model training provided by the present invention, the training parameters also include hidden layer size, number of model layers, number of attention heads of the attention mechanism, hidden layer size of the forward feedback network, vocabulary size and sequence length;

The training parallel strategies include data parallelism, tensor parallelism and pipeline parallelism;

The training batch size includes a global batch size and a micro batch size.

According to the network traffic simulation method for large model training provided by the present invention, the communication load matrix is generated based on the training parameters and the network topology structure, including:

Based on the training parameters and the network topology, determine the traffic pattern of model parameter synchronization and gradient transmission during the training process of the large model to be trained;

A communication load matrix is generated according to the traffic pattern; the communication load matrix includes forward calculation duration, forward communication duration and backward calculation duration.

According to the network traffic simulation method for large model training provided by the present invention, the traffic simulation is performed according to the communication load matrix to simulate the network traffic in the model training process of the large model to be trained, including:

Determine a computing process set and a communication process set according to the communication load matrix; the computing process set includes a forward computing process set and a backward computing process set;

Inputting the computing process set and the communication process set into a global pipeline scheduler for allocation; the global pipeline scheduler determines the pipeline stage according to the forward execution times and the backward execution times of the network nodes;

Execute the target communication process corresponding to the pipeline stage to simulate the network traffic during the model training process of the large model to be trained.

According to the network traffic simulation method for large model training provided by the present invention, the target communication process corresponding to the pipeline stage is executed, including:

Obtain a preset algorithm set, and select a target algorithm combination from the algorithm set; the algorithm set includes multiple algorithm combinations, each of which is obtained by combining a congestion control algorithm and a load balancing algorithm; the target algorithm combination is any one of the multiple algorithm combinations;

Based on the target algorithm combination, executing the target communication process corresponding to the pipeline stage;

Return and execute the step of selecting a target algorithm combination from the algorithm set until the target algorithm combination is the last one in the algorithm set.

According to the network traffic simulation method for large model training provided by the present invention, after executing the target communication process corresponding to the pipeline stage, it also includes:

Collecting performance indicators during the target communication process; the performance indicators include network performance indicators and training performance indicators; the network performance indicators include link utilization, end-to-end delay and network throughput, and the training performance indicators include single training iteration time, training efficiency and communication overhead ratio;

The network topology structure and the training process of the large model to be trained are evaluated according to the performance indicators to obtain performance evaluation information.

The present invention also provides a network traffic simulation device for large model training, comprising the following modules:

Information acquisition module, used to obtain user configuration information;

A simulation configuration module, used to define the network topology and the training parameters of the large model to be trained according to the configuration information;

A load generation module, used to generate a communication load matrix based on the training parameters and the network topology; the communication load matrix is used to characterize the computing time and data transmission requirements of each network node in the network topology;

The simulation module is used to perform traffic simulation according to the communication load matrix to simulate the network traffic during the model training process of the large model to be trained.

The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the network traffic simulation method for large model training as described in any one of the above is implemented.

The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a network traffic simulation method for large model training as described in any one of the above.

The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the network traffic simulation method for large model training as described in any one of the above.

The network traffic simulation method, device, equipment and medium for large model training provided by the present invention define the network topology structure and the training parameters of the large model to be trained through the user's configuration information, generate a communication load matrix based on the network topology structure and the training parameters of the large model to be trained, perform traffic simulation according to the communication load matrix, simulate the network traffic in the model training process of the large model to be trained, and realize the simulation of the network traffic in the model training process, so as to optimize and adjust the network configuration of the large model training process according to the simulation results. By defining the network topology structure and the training parameters of the model through the user's configuration information, the user can flexibly adjust the network structure to adapt to the simulation of the network traffic of training clusters of different scales and structures, thereby improving the flexibility and applicability of the simulation of the network traffic for model training.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 is a flow chart of a network traffic simulation method for large model training provided by the present invention.

FIG. 2 is a schematic diagram of a simulation flow of network traffic provided by the present invention.

FIG3 is a schematic diagram of the structure of a network traffic simulation device for large model training provided by the present invention.

FIG. 4 is a schematic diagram of the structure of an electronic device provided by the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

The embodiments of the present invention provide a network traffic simulation method, device, equipment and medium for large-scale model training, which can solve the deficiencies of existing general network simulation tools in network traffic simulation for large-scale model training, and solve the problems of inflexible network topology configuration, inaccurate simulation of specific traffic patterns, insufficient performance indicator analysis dimensions and limited algorithm testing support.

Specifically, an embodiment of the present invention provides a network traffic simulation method for large model training. FIG1 is a flow chart of the network traffic simulation method for large model training provided by the present invention. As shown in FIG1 , the method includes the following steps:

Step 100, obtaining user configuration information;

Step 200, defining the network topology and the training parameters of the large model to be trained according to the configuration information;

Step 300, generating a communication load matrix based on the training parameters and the network topology; the communication load matrix is used to characterize the computing time and data transmission requirements of each network node in the network topology;

Step 400, performing traffic simulation according to the communication load matrix to simulate the network traffic during the model training process of the large model to be trained.

The user's configuration information is obtained, and the network topology structure and the training parameters of the large model to be trained are defined according to the user's configuration information, wherein the user's configuration information includes the network configuration information of the network structure and the training configuration information of the large model to be trained.

Furthermore, when defining the network topology structure and the training parameters of the large model to be trained, the network topology structure is defined according to the network configuration information, and the training parameters of the large model to be trained are defined according to the training configuration information.

The network topology is a network structure used to support network communication functions during the distributed training process of the large model to be trained. It is responsible for data synchronization between computing nodes and the transmission of model parameters. The network topology includes multiple computing nodes, which are also called network nodes or communication nodes. The data transmission requirements of each network calculation are determined based on the training process of the large model to be trained, and the training process of the large model to be trained is determined based on the training parameters.

A communication load matrix is generated based on the training parameters and the network topology structure, and the communication load matrix is used to characterize the computing time and data transmission requirements of each network node in the network topology structure. Among them, the computing time of the network node includes forward computing time, communication time and backward computing time. The forward computing time is the computing time corresponding to the forward computing process in the model training process, and the backward computing time is the computing time corresponding to the backward computing process in the model training process. Optionally, the generated communication load matrix includes the communication load matrix between each network node in the network topology structure.

The forward calculation time and the backward calculation time are the calculation times obtained by abstractly modeling the model calculation amount, and the communication time is the communication time obtained by modeling the bandwidth delay of the network link.

Furthermore, the forward calculation process mainly includes data preparation, model application, layer-by-layer calculation and result output, and the backward calculation process mainly includes loss calculation, gradient calculation, parameter update and iterative loop.

Among them, data preparation is the preprocessing of sample data required for model training, such as normalization and format conversion; model application is the transfer of training sample data obtained in the data preparation stage to the model; layer-by-layer calculation means that each layer in the model processes the input data to generate intermediate outputs; result output refers to the process of obtaining predicted values when the model performs tasks such as classification or regression. The output result of the model is usually one or more vectors, which represent the prediction results of the model.

Furthermore, loss calculation is after the forward propagation of the model. The model network will calculate the loss value, that is, the error, based on the model output and the true label. The loss value is used to measure the gap between the model prediction value and the true value; gradient calculation is to use the loss obtained by forward calculation to calculate the gradient of all model parameters, that is, the derivative of the loss value with respect to the model parameters. This process is back propagation; parameter update is to update the model weights and biases through the gradient descent algorithm after calculating the gradient of each model parameter. The basic principle of gradient descent is: if the gradient of the loss value (that is, the error) points in a certain direction, then adjust the model parameters in the opposite direction to reduce the loss value of the model; the iterative cycle is during the model training process. The model will perform multiple forward propagations, calculate losses and back propagate errors, and update the model weights and biases. A complete forward propagation and back propagation process is called an iteration, that is, an "epoch". The model training process usually goes through multiple epochs until the model converges to a lower loss value or reaches the preset maximum number of iterations.

Traffic simulation is performed according to the generated communication load matrix to simulate the network traffic in the model training process of the large model to be trained. The process of performing traffic simulation includes traffic generation and injection, data collection and analysis, and result verification and optimization stages.

In the traffic generation and injection stage, the corresponding data packets and sending intervals are mainly generated according to the data synchronization and parameter transfer process in the training of the large model to be trained, and the data packets are sent according to the sending interval, thereby simulating the network traffic of data synchronization and parameter transfer in the training process of the large model to be trained.

During the data collection and analysis phase, performance indicators of the large model to be trained are collected to determine whether the network performance and model performance meet the requirements, so that the network configuration can be adjusted according to the simulation results during the actual training of the large model.

In this embodiment, the network topology and the training parameters of the large model to be trained are defined through the user's configuration information, a communication load matrix is generated based on the network topology and the training parameters of the large model to be trained, and traffic simulation is performed according to the communication load matrix to simulate the network traffic in the model training process of the large model to be trained, so as to achieve the simulation of the network traffic in the model training process, so as to optimize and adjust the network configuration of the large model training process according to the simulation results. By defining the network topology and the training parameters of the model through the user's configuration information, the user can flexibly adjust the network structure to adapt to the simulation of the network traffic of training clusters of different sizes and structures, thereby improving the flexibility and applicability of the simulation of the network traffic for model training.

Optionally, the user's configuration information is determined based on the user's demand for large model training, including network configuration information and model configuration information, the network configuration information is used to configure the network topology, and the model configuration information is used to configure the training parameters of the large model to be trained. Based on this, step 200 may also include:

Step 201, determining the number of nodes according to the network configuration information in the configuration information, and setting the network nodes according to the number of nodes; the network configuration information includes node configuration information;

Step 202, configuring key parameters of the network nodes based on the node configuration information to define a network topology; the key parameters include a connection mode, link bandwidth, and link delay between the network nodes;

Step 203, defining the training parameters of the large model to be trained according to the model configuration information in the configuration information; the training parameters include model parameter scale, training parallel strategy and training batch size.

The network configuration information includes the number of nodes and the node configuration information of each node. When defining the network topology, the number of nodes is determined according to the network configuration information in the configuration information, and then the network nodes are set according to the number of nodes. The network configuration information also includes the node configuration information of each node to be set, and the key parameters of each network node are configured based on the node configuration information, thereby defining the network topology.

Optionally, the key parameters of the network nodes include at least the connection mode, link bandwidth and link delay between the network nodes. The network topology structure includes multiple network nodes, and the key parameters of different network nodes may be the same or different.

Furthermore, the training parameters of the large model to be trained are defined according to the model configuration information in the configuration information, and the training parameters include model parameter scale, training parallel strategy and training batch size.

Optionally, the training parameters also include hidden layer size, number of model layers, number of attention heads of the attention mechanism, hidden layer size of the feedforward network, vocabulary size, and sequence length. Training parallel strategies include data parallelism, tensor parallelism, and pipeline parallelism, and training batch size includes global batch size and micro batch size.

General network simulation tools lack sufficient accuracy and flexibility when simulating complex and specific communication modes in the process of large model training, such as parameter synchronization and gradient transmission, which makes it difficult for simulation results to fully reflect the network communication requirements in actual training. In this embodiment, a communication load matrix is generated according to the traffic patterns of parameter synchronization and gradient transmission in the process of model training, so that the simulation of network traffic can reflect the network communication requirements in the actual model training process.

Based on this, step 300 may also include:

Step 301, based on the training parameters and the network topology, determining the traffic mode of model parameter synchronization and gradient transmission during the training process of the large model to be trained;

Step 302, generating a communication load matrix according to the traffic pattern; the communication load matrix includes forward calculation duration, forward communication duration and backward calculation duration.

Based on the training parameters of the large model to be trained and the defined network topology, the traffic pattern of model parameter synchronization and gradient transmission during the training process of the large model to be trained is determined, and then a communication load matrix is generated according to the traffic pattern.

The generated communication load matrix includes forward calculation time, forward communication time and backward calculation time, where the communication load matrix includes the forward calculation time of each flow, the forward communication time of each flow, and the backward calculation time of each flow. The communication load matrix is generated according to the network topology and the training parameters of the large model to be trained. The communication load matrix represents the calculation time of each network node and the data transmission requirements between network nodes, reflecting the specific traffic pattern of model parameter synchronization and gradient transmission during model training.

Optionally, when simulating the network traffic in the model training process of the large model to be trained, the forward computing process and the communication process in the model training process are allocated based on the global pipeline scheduler. Step 400 may also include:

Step 401, determining a computing process set and a communication process set according to the communication load matrix; the computing process set includes a forward computing process set and a backward computing process set;

Step 402, inputting the computing process set and the communication process set into a global pipeline scheduler for allocation; the global pipeline scheduler determines the pipeline stage according to the forward execution times and the backward execution times of the network nodes;

Step 403, executing the target communication process corresponding to the pipeline stage to simulate the network traffic during the model training process of the large model to be trained.

When simulating network traffic, the computing process set and the communication process set are determined according to the generated communication load matrix, and the computing process set and the communication process set are input into the global pipeline scheduler for allocation. The global pipeline scheduler determines the pipeline stage according to the forward execution times and backward execution times of the network nodes.

The computing process set includes a forward computing process set and a backward computing process set. The computing process set also includes the execution status of each network node, which is used to characterize whether the next step of the network node is to perform forward communication or backward communication. The global pipeline scheduler determines the forward execution times and backward execution times of the network node according to the computing process set, thereby determining the pipeline stage. The forward execution times of the network node can be determined according to the number of forward computing processes in the computing process set executed by the network node, and the backward execution times of the network node can be determined according to the number of backward computing processes in the computing process set executed by the network node.

Furthermore, the target communication process corresponding to the pipeline stage is executed to simulate the network traffic in the model training process of the large model to be trained. The target communication process is the communication process corresponding to the pipeline stage in the communication process set.

Optionally, the forward calculation set contains the forward calculation process of each flow, and the communication process set contains the communication process of each flow. The global pipeline scheduler allocates the forward calculation process and the communication process in order according to the model training process of the large model to be trained. The pipeline stages include data loading, preheating, single forward and single backward, cooling, and gradient synchronization.

General network simulation tools are mainly based on general communication protocols, and do not support training parameter customization and batch customization for large models. It is difficult to capture the communication characteristics of high-frequency words and small data volumes in large model training, and lack built-in support for training-related performance indicators. In order to simulate specific communication requirements in large-scale training, secondary development and customization are usually required, which increases the difficulty and development cost of network traffic simulation. Large model network traffic is affected by the training parallel strategy, and there are some special collective communication modes and traffic modes that general network simulation tools cannot reproduce. In addition, after traffic simulation, some network tuning algorithms such as congestion control algorithms or load balancing algorithms cannot be tested. In terms of performance analysis, it mainly focuses on the network level, lacks comprehensive performance evaluation during training, and has limited testing and evaluation of different tuning algorithms, which limits the optimization of traffic simulation at the network transmission layer.

Based on this, the network traffic simulation method provided by the embodiment of the present invention can realize the test evaluation of different algorithms during the network traffic simulation process. When executing the communication process, a congestion control algorithm and a load balancing algorithm are adopted. Specifically, in step 403, the target communication process corresponding to the pipeline stage is executed, and it can also include:

Step 413, obtaining a preset algorithm set, and selecting a target algorithm combination from the algorithm set; the algorithm set includes multiple algorithm combinations, each of which is obtained by combining a congestion control algorithm and a load balancing algorithm; the target algorithm combination is any one of the multiple algorithm combinations;

Step 423, based on the target algorithm combination, executing the target communication process corresponding to the pipeline stage;

Step 433, return and execute the step of selecting a target algorithm combination from the algorithm set; until the target algorithm combination is the last one in the algorithm set.

When executing the communication process to simulate the network traffic in the model training process, first obtain a preset algorithm set, which includes multiple algorithm combinations, and any algorithm combination is obtained by combining at least one congestion control algorithm and one load balancing algorithm. Select a target algorithm combination from the algorithm set, and the target algorithm combination is any one of the multiple algorithm combinations in the algorithm set.

In one embodiment, based on a plurality of different congestion control algorithms and a plurality of different load balancing algorithms, the congestion control algorithms and the load balancing algorithms are combined in pairs to obtain a plurality of algorithm combinations to form an algorithm set.

Further, based on the selected target algorithm combination, the target communication process corresponding to the pipeline stage is executed, and the step of selecting the target algorithm combination from the algorithm set is returned and executed until the selected target algorithm combination is the last one in the algorithm set.

Optionally, based on the same communication process, the same communication process is executed separately based on different algorithm combinations. In this way, for each communication process, each communication process is executed separately based on different algorithm combinations. Alternatively, based on the same algorithm combination, all communication processes are executed, and then the next algorithm combination is selected to repeat the execution of all communication processes. In this way, for each algorithm combination, all communication processes are executed separately based on the algorithm combination.

In the large-scale distributed training process, due to memory limitations, it is necessary to introduce three-dimensional hybrid parallel technology. Affected by the three-dimensional hybrid parallel technology, the traffic pattern of the training iteration is solidified and divided into stages such as data loading, preheating, single forward and single reverse, cooling and gradient synchronization. In the network traffic simulation, this embodiment implements the pipeline operation process of calculating each batch of data blocks in turn as the batch data is input during the training iteration, and realizes tensor parallel communication during the calculation process, and executes point-to-point communication to pass to the next computing node. In this process, the end-to-end network performance of each flow is collected and visualized to evaluate the network performance. In addition, the analysis dimension of the performance indicators is mainly concentrated at the network level, lacking the problem of comprehensive performance evaluation of the training process. This embodiment can realize the comprehensive evaluation of the model training process based on network performance and model performance.

Based on this, in step 423, after executing the target communication process corresponding to the pipeline stage based on the target algorithm combination, the following may also be included:

Step 404, collecting performance indicators during the target communication process; the performance indicators include network performance indicators and training performance indicators; the network performance indicators include link utilization, end-to-end delay and network throughput, and the training performance indicators include single training iteration time, training efficiency and communication overhead ratio;

Step 405: Evaluate the network topology and the training process of the large model to be trained according to the performance index to obtain performance evaluation information.

The performance indicators of the target communication process are collected, including network performance indicators and training performance indicators. The network performance indicators further include link utilization, end-to-end delay and network throughput. The training performance indicators include single training iteration time, training efficiency and communication overhead ratio.

The network topology and the training process of the large model to be trained are evaluated according to the collected performance indicators to obtain corresponding performance evaluation information. Optionally, based on the performance evaluation information, the user's configuration information can be optimized to achieve the optimization of the network topology for model training.

In one embodiment, referring to the simulation process of network traffic shown in FIG2 , first, the user's configuration information is obtained, including network configuration information and training configuration information, and the network topology is defined according to the network configuration information, including setting the number of nodes, and configuring the link bandwidth and link delay between nodes, etc., and the training parameters of the large model to be trained are configured according to the training configuration information, including setting the model parameter scale, training parallel strategy and training batch size.

Based on the configured network topology and training parameters, the communication load matrix is generated, and then the simulation environment is initialized. According to the algorithm combination of the congestion control algorithm and the load balancing algorithm, the network traffic simulation is performed to simulate the network traffic during the model training process of the large model to be trained. The performance indicators of the network traffic during the simulation process are monitored and collected. The performance indicators include network performance indicators and training performance indicators. The network performance indicators are used to evaluate the network performance of the model training, and the training performance indicators are used to evaluate the training performance of the model training.

In one embodiment, the network configuration information includes the number of nodes, and further includes the configuration information of the network topology, specifically including the number of GPUs in a single server of the network topology, the number of servers, the number of network switches, the number of network links, the server type and the switch number. Generally, for the setting of network nodes, 8 network nodes are connected to an in-machine switch, and the 8 nodes are respectively connected to the leaf layer switches. The link bandwidth is also configurable within a certain range to meet the simulation of network environments from small clusters to ultra-large-scale data centers. The link delay is also configurable within a certain range, for example, it is configurable between 0.001ms and 10ms, which is used to simulate links of different lengths, thereby reproducing the cluster distribution in different geographical locations.

In some embodiments, the discrete event simulation (DES) method is used to accurately simulate the transmission process of data packets in the network, and the end-to-end completion time, link occupancy, and bandwidth utilization of the traffic are recorded. When the pipeline executes the communication process, it determines which stage of the pipeline it is currently in based on the forward execution times and backward execution times of the current network node, and then executes the communication process corresponding to the stage. In any communication process, the communication mode and next state of the current network node, as well as the next network node, are obtained, where the communication mode includes no communication, one-way communication, two-way communication, and sleep waiting.

Furthermore, when executing the communication process, if the communication mode of the current network node is no communication, the next node of the current network node is backward calculation; if the communication mode of the current network node is one-way communication, a data packet is sent to the next network node, and the next state of the current network node is backward calculation; if the communication mode of the current network node is two-way communication, the current network node and the previous network node send data packets to each other, and the next state of the current network node is forward calculation; if the communication mode of the current network node is sleep waiting, the data packet is waited for to arrive, and the next state of the current network node is backward calculation. At any time during the model training process, the communication modes of different nodes in the network topology can be the same or different.

Optionally, after the simulation of network traffic is completed, a simulation report can be generated. The simulation report includes the collected performance indicators and evaluation information of the user configuration information based on the performance indicators. The evaluation information can be used to guide the user to adjust the configuration information.

In this embodiment, by customizing the configuration of network topology and model training parameters, it is possible to support diversified simulation of network traffic during model training. Moreover, based on the training parallel strategy, a three-dimensional hybrid parallel training process for large model training can be implemented, which conforms to the traffic pattern of large model training, and is conducive to the accurate and flexible simulation of complex and specific communication modes in the large model training process, ensuring that the simulation results can fully reflect the network communication requirements in the actual model training.

Furthermore, in the process of simulating network traffic, the end-to-end performance indicators of each traffic in the simulation process can be visualized, so as to obtain accurate network performance indicators and model training performance indicators in the model training process without a real cluster. It also supports the testing of the load balancing algorithm and congestion control algorithm of the network transmission layer, as well as the switching test of different algorithms, so as to obtain the performance indicators of different algorithms in the large-scale distributed training process, with strong adaptability.

The network traffic simulation device for large model training provided by the present invention is described below. The network traffic simulation device for large model training described below and the network traffic simulation method for large model training described above can be referenced to each other.

3, the network traffic simulation device for large model training provided by an embodiment of the present invention includes:

The information acquisition module 10 is used to acquire the user's configuration information;

A simulation configuration module 20, used to define the network topology and the training parameters of the large model to be trained according to the configuration information;

A load generation module 30, configured to generate a communication load matrix based on the training parameters and the network topology; the communication load matrix is used to characterize the computing time and data transmission requirements of each network node in the network topology;

The simulation module 40 is used to perform traffic simulation according to the communication load matrix to simulate the network traffic during the model training process of the large model to be trained.

In one embodiment, the simulation configuration module 20 is further used for:

Determine the number of nodes according to the network configuration information in the configuration information, and set the network nodes according to the number of nodes; the network configuration information includes node configuration information;

Configuring key parameters of the network nodes based on the node configuration information to define a network topology; the key parameters include a connection mode, link bandwidth, and link delay between the network nodes;

The training parameters of the large model to be trained are defined according to the model configuration information in the configuration information; the training parameters include model parameter scale, training parallel strategy and training batch size.

In one embodiment, the training parameters further include hidden layer size, number of model layers, number of attention heads of the attention mechanism, hidden layer size of the feedforward network, vocabulary size, and sequence length;

The training parallel strategies include data parallelism, tensor parallelism and pipeline parallelism;

The training batch size includes a global batch size and a micro batch size.

In one embodiment, the load generation module 30 is further configured to:

Based on the training parameters and the network topology, determine the traffic pattern of model parameter synchronization and gradient transmission during the training process of the large model to be trained;

A communication load matrix is generated according to the traffic pattern; the communication load matrix includes forward calculation duration, forward communication duration and backward calculation duration.

In one embodiment, the simulation module 40 is further used for:

Determine a computing process set and a communication process set according to the communication load matrix; the computing process set includes a forward computing process set and a backward computing process set;

Inputting the computing process set and the communication process set into a global pipeline scheduler for allocation; the global pipeline scheduler determines the pipeline stage according to the forward execution times and the backward execution times of the network nodes;

Execute the target communication process corresponding to the pipeline stage to simulate the network traffic during the model training process of the large model to be trained.

In one embodiment, the simulation module 40 is further used for:

Obtain a preset algorithm set, and select a target algorithm combination from the algorithm set; the algorithm set includes multiple algorithm combinations, each of which is obtained by combining a congestion control algorithm and a load balancing algorithm; the target algorithm combination is any one of the multiple algorithm combinations;

Based on the target algorithm combination, executing the target communication process corresponding to the pipeline stage;

Return and execute the step of selecting a target algorithm combination from the algorithm set until the target algorithm combination is the last one in the algorithm set.

In one embodiment, the simulation module 40 is further used for:

Collecting performance indicators during the target communication process; the performance indicators include network performance indicators and training performance indicators; the network performance indicators include link utilization, end-to-end delay and network throughput, and the training performance indicators include single training iteration time, training efficiency and communication overhead ratio;

The network topology structure and the training process of the large model to be trained are evaluated according to the performance indicators to obtain performance evaluation information.

FIG4 illustrates a schematic diagram of the physical structure of an electronic device. As shown in FIG4 , the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the steps of the network traffic simulation method for large model training, for example, including:

Get the user's configuration information;

Define the network topology and the training parameters of the large model to be trained according to the configuration information;

Generate a communication load matrix based on the training parameters and the network topology; the communication load matrix is used to characterize the computing time and data transmission requirements of each network node in the network topology;

Traffic simulation is performed according to the communication load matrix to simulate the network traffic during the model training process of the large model to be trained.

In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the steps of the network traffic simulation method for large model training provided by the above methods, for example, including:

Get the user's configuration information;

Define the network topology and the training parameters of the large model to be trained according to the configuration information;

Generate a communication load matrix based on the training parameters and the network topology; the communication load matrix is used to characterize the computing time and data transmission requirements of each network node in the network topology;

Traffic simulation is performed according to the communication load matrix to simulate the network traffic during the model training process of the large model to be trained.

In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the network traffic simulation method for large model training provided by the above methods, for example, including:

Get the user's configuration information;

Define the network topology and the training parameters of the large model to be trained according to the configuration information;

Generate a communication load matrix based on the training parameters and the network topology; the communication load matrix is used to characterize the computing time and data transmission requirements of each network node in the network topology;

Traffic simulation is performed according to the communication load matrix to simulate the network traffic during the model training process of the large model to be trained.

The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (9)

1. A network traffic simulation method for large model training, characterized by comprising: Get the user's configuration information; Defining the network topology and the training parameters of the large model to be trained according to the configuration information; the training parameters include the model parameter scale, the training parallel strategy and the training batch size; Generate a communication load matrix based on the training parameters and the network topology; the communication load matrix is used to characterize the computing time and data transmission requirements of each network node in the network topology; Performing traffic simulation according to the communication load matrix to simulate the network traffic during the model training process of the large model to be trained; The generating of a communication load matrix based on the training parameters and the network topology structure comprises: Based on the training parameters and the network topology, determine the traffic pattern of model parameter synchronization and gradient transmission during the training process of the large model to be trained; A communication load matrix is generated according to the traffic pattern; the communication load matrix includes forward calculation duration, forward communication duration and backward calculation duration. 2. The network traffic simulation method for large model training according to claim 1 is characterized in that the network topology structure and the training parameters of the large model to be trained are defined according to the configuration information, including: Determine the number of nodes according to the network configuration information in the configuration information, and set the network nodes according to the number of nodes; the network configuration information includes node configuration information; Configuring key parameters of the network nodes based on the node configuration information to define a network topology; the key parameters include a connection mode, link bandwidth, and link delay between the network nodes; The training parameters of the large model to be trained are defined according to the model configuration information in the configuration information. 3. The network traffic simulation method for large model training according to claim 1 is characterized in that the training parameters also include hidden layer size, number of model layers, number of attention heads of the attention mechanism, hidden layer size of the forward feedback network, vocabulary size and sequence length; The training parallel strategies include data parallelism, tensor parallelism and pipeline parallelism; The training batch size includes a global batch size and a micro batch size. 4. The network traffic simulation method for large model training according to claim 1 is characterized in that the traffic simulation is performed according to the communication load matrix to simulate the network traffic in the model training process of the large model to be trained, including: Determine a computing process set and a communication process set according to the communication load matrix; the computing process set includes a forward computing process set and a backward computing process set; Inputting the computing process set and the communication process set into a global pipeline scheduler for allocation; the global pipeline scheduler determines the pipeline stage according to the forward execution times and the backward execution times of the network nodes; Execute the target communication process corresponding to the pipeline stage to simulate the network traffic during the model training process of the large model to be trained. 5. The network traffic simulation method for large model training according to claim 4, characterized in that the target communication process corresponding to the pipeline stage is executed, comprising: Obtain a preset algorithm set, and select a target algorithm combination from the algorithm set; the algorithm set includes multiple algorithm combinations, each of which is obtained by combining a congestion control algorithm and a load balancing algorithm; the target algorithm combination is any one of the multiple algorithm combinations; Based on the target algorithm combination, executing the target communication process corresponding to the pipeline stage; Return and execute the step of selecting a target algorithm combination from the algorithm set until the target algorithm combination is the last one in the algorithm set. 6. The network traffic simulation method for large model training according to claim 4 is characterized in that after executing the target communication process corresponding to the pipeline stage, it also includes: Collecting performance indicators during the target communication process; the performance indicators include network performance indicators and training performance indicators; the network performance indicators include link utilization, end-to-end delay and network throughput, and the training performance indicators include single training iteration time, training efficiency and communication overhead ratio; The network topology structure and the training process of the large model to be trained are evaluated according to the performance indicators to obtain performance evaluation information. 7. A network traffic simulation device for large model training, characterized by comprising: Information acquisition module, used to obtain user configuration information; A simulation configuration module, used to define the network topology and the training parameters of the large model to be trained according to the configuration information; the training parameters include the model parameter scale, the training parallel strategy and the training batch size; A load generation module, used to generate a communication load matrix based on the training parameters and the network topology; the communication load matrix is used to characterize the computing time and data transmission requirements of each network node in the network topology; A simulation module, used to perform traffic simulation according to the communication load matrix, and simulate the network traffic in the model training process of the large model to be trained; The load generation module is further used for: The generating of a communication load matrix based on the training parameters and the network topology structure comprises: Based on the training parameters and the network topology, determine the traffic pattern of model parameter synchronization and gradient transmission during the training process of the large model to be trained; A communication load matrix is generated according to the traffic pattern; the communication load matrix includes forward calculation duration, forward communication duration and backward calculation duration. 8. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the network traffic simulation method for large model training as described in any one of claims 1 to 6 when executing the computer program. 9. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the network traffic simulation method for large model training as described in any one of claims 1 to 6 is implemented.