CN117938715A - Abnormality detection method, device, electronic device and readable medium of RDMA system - Google Patents

Abstract

The present disclosure provides an abnormality detection method, device, electronic device and readable medium of an RDMA system, wherein the abnormality detection method of the RDMA system includes: abstractly modeling the workspace of the RDMA system to generate a search space of the RDMA system; determining the workload according to the dimension of the search space; determining the conditional characteristics of any workload triggering the abnormality of the RDMA system to generate a minimum functional feature set of the abnormality; for any minimum functional feature set, calling the counter of the RDMA system to determine the abnormality index of the workload. Through the embodiments of the present disclosure, active detection of performance abnormalities of the RDMA system is achieved, and the performance and stability of the RDMA system in practical applications are improved.

Description

Abnormality detection method, device, electronic device and readable medium of RDMA system

Technical Field

The present disclosure relates to the field of communication technology, and in particular to an abnormality detection method, device, electronic device and readable medium for an RDMA system.

Background technique

In related technologies, RDMA hardware vendors conduct a large number of tests, but due to the large number of parties involved, the RDMA system may still experience various performance anomalies, seriously affecting the performance and stability of the RDMA network.

However, existing RDMA performance tests mainly rely on simple benchmark tools or test known applications, which cannot fully cover potential application workloads and make it difficult to detect performance anomalies.

It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field.

Summary of the invention

The purpose of the present disclosure is to provide an abnormality detection method, device, electronic device and readable medium for an RDMA system, which are used to overcome the problem of poor reliability of the RDMA system caused by the limitations and defects of the related art at least to a certain extent.

According to a first aspect of an embodiment of the present disclosure, a method for detecting anomalies in an RDMA system is provided, comprising: abstractly modeling a workspace of the RDMA system to generate a search space of the RDMA system; determining a workload according to a dimension of the search space; determining conditional features for triggering anomalies in the RDMA system by any of the workloads to generate a minimum functional feature set of the anomaly; and for any of the minimum functional feature sets, calling a counter of the RDMA system to determine an abnormality indicator of the workload.

In an exemplary embodiment of the present disclosure, it further includes:

The workload mode and/or the workload selection result are adjusted based on the iterative calculation result of the abnormal indicator using a statistical algorithm.

In an exemplary embodiment of the present disclosure, adjusting the workload mode and/or the workload selection result based on the iterative calculation result of the abnormal indicator by a statistical algorithm includes:

Calling an annealing algorithm in the statistical algorithm to iteratively calculate a count difference between the count value of the workload and the count value of a new workload;

determining a ratio value between the count difference value and the count value of the workload;

The workload mode and/or the workload selection result is adjusted according to the ratio value.

In an exemplary embodiment of the present disclosure, adjusting the mode of the workload and/or the selection result of the workload according to the ratio value includes:

If it is determined that the ratio value is less than zero, transferring the workload to the new workload;

If it is determined that the ratio value is greater than zero, then calculating the opposite of the ratio value using an exponential function with a natural constant e as the base;

The calculation result of the exponential function is determined as the probability of transferring the workload to the new workload.

In an exemplary embodiment of the present disclosure, adjusting the mode of the workload and/or the selection result of the workload according to the ratio value includes:

The request mode of the RDMA system and/or the allocation information of the buffer of the RDMA system are adjusted according to the ratio value.

In an exemplary embodiment of the present disclosure, abstract modeling of the working space of the RDMA system to generate the search space of the RDMA system includes:

Abstracting the workspace of the RDMA system to determine a memory area corresponding to the workspace, and registering the memory area;

Creating the queue pair and setting the transmission type of the queue pair;

Create a work queue element based on the registered memory area and queue pair with the transfer type set;

The completion queue is created according to the work queue element to generate a search space of the RDMA system.

In an exemplary embodiment of the present disclosure, determining the workload according to the dimension of the search space includes:

Parsing the topology, host memory source, transmission mode and message mode included in the dimensions of the search space;

The traffic between hosts of the RDMA system is generated according to the topology, the host memory source, the transmission mode and the message mode, and the workload corresponding to the traffic is determined.

According to a second aspect of an embodiment of the present disclosure, there is provided an abnormality detection device for an RDMA system, comprising:

A modeling module, configured to perform abstract modeling on the working space of the RDMA system to generate a search space of the RDMA system;

a determination module configured to determine a workload according to a dimension of the search space;

The determination module is configured to determine the conditional features of any of the workloads triggering the RDMA system abnormality, so as to generate a minimum functional feature set of the abnormality;

The determination module is configured to call a counter of the RDMA system to determine an abnormality indicator of the workload for any of the minimum functional feature sets.

According to a third aspect of the present disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the methods described above based on instructions stored in the memory.

According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a program is stored, and when the program is executed by a processor, the abnormality detection method of the RDMA system as described in any one of the above items is implemented.

The disclosed embodiments generate a search space for the RDMA system by abstractly modeling the workspace of the RDMA system, and then determine the workload according to the dimension of the search space, determine the conditional features for triggering an abnormality in the RDMA system by any workload, and generate an abnormal minimum functional feature set. Finally, for any minimum functional feature set, call the counter of the RDMA system to determine the abnormal index of the workload, thereby realizing active detection of performance anomalies of the RDMA system and improving the performance and stability of the RDMA system in practical applications.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

FIG1 is a schematic diagram showing an exemplary system architecture of an anomaly detection solution for an RDMA system to which an embodiment of the present invention can be applied;

FIG2 is a flow chart of an abnormality detection method for an RDMA system in an exemplary embodiment of the present disclosure;

FIG3 is a flow chart of another abnormality detection method for an RDMA system in an exemplary embodiment of the present disclosure;

FIG4 is a flow chart of another abnormality detection method for an RDMA system in an exemplary embodiment of the present disclosure;

FIG5 is a flow chart of another abnormality detection method for an RDMA system in an exemplary embodiment of the present disclosure;

FIG6 is a flow chart of another abnormality detection method for an RDMA system in an exemplary embodiment of the present disclosure;

FIG7 is a flow chart of another abnormality detection method for an RDMA system in an exemplary embodiment of the present disclosure;

FIG8 is a flow chart of another abnormality detection method for an RDMA system in an exemplary embodiment of the present disclosure;

FIG9 is a system architecture diagram of an abnormality detection solution for an RDMA system in an exemplary embodiment of the present disclosure;

FIG10 is a schematic diagram of a search space (working space) of an anomaly detection scheme for an RDMA system in an exemplary embodiment of the present disclosure;

FIG11 is a flow chart of an abnormality detection scheme for an RDMA system in an exemplary embodiment of the present disclosure;

FIG12 is a block diagram of an abnormality detection device for an RDMA system in an exemplary embodiment of the present disclosure;

FIG. 13 is a block diagram of an electronic device in an exemplary embodiment of the present disclosure.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art. The described conditional features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

In addition, the accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and/or processor devices and/or microcontroller devices.

FIG. 1 is a schematic diagram showing an exemplary system architecture of an anomaly detection solution for an RDMA system to which an embodiment of the present invention can be applied.

As shown in Fig. 1, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

It should be understood that the number of terminal devices, networks and servers in FIG1 is only illustrative. According to implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, desktop computers, etc.

In some embodiments, the abnormality detection method of the RDMA system provided in the embodiments of the present invention is generally executed by the server 105. Accordingly, the abnormality detection device of the RDMA system is generally set in the terminal device 103 (or the terminal device 101 or 102). In other embodiments, some terminals may have functions similar to those of the server device to execute the method.

The technical terms involved in the embodiments of the present disclosure are explained and described below, specifically as follows:

MFS: Minimum Feature Set, minimum feature set or minimum feature set.

MR: Memory Region. Applications need to register memory regions first to make them accessible to RDMA. Use functions to register MRs. MRs have a starting address and length, defining a continuous memory space. RNIC can directly access registered MRs without CPU involvement.

RNIC: RDMA Network Interface Card, RDMA network card.

QP: Queue Pair, queue pair, use ibv_create_qp to create QP. QP represents a "connection" between the application and RNIC. QP is an abstraction of point-to-point communication, including sending queues and receiving queues. Each QP needs to be configured with a transmission mode, such as reliable connection RC.

WQE: Work Queue Element, work request. To send/receive messages, you need to construct a WQE and deliver it to the QP. The WQE contains a scatter/gather list that specifies a set of MR memory buffers involved in the transmission and delivers the WQE to the QP sending queue.

Among them, scatter/gather is used to describe the operation of reading from or writing to the Channel.

Scatter: When reading from a Channel, the data read is written to multiple buffers during the read operation. Therefore, the Channel scatters the data read from the Channel to multiple buffers.

Gather: Write data from multiple buffers into the same Channel. Therefore, Channel gathers data from multiple buffers and sends them to Channel. Scatter/gather is often used in situations where the transmitted data needs to be processed separately. For example, when transmitting data consisting of a message header and a message body, you can scatter the message header and the message body into different buffers for easy processing.

CQ: Completion Queue, completion queue, CQ is used to receive completion notifications and determine whether WQE is completed. Create CQ and query CQ. After obtaining the completion notification, the WQE memory buffer can be reused.

RC: Connection-oriented reliable service.

UC: Connection-oriented unreliable service.

UD: Datagram-oriented unreliable service.

RD: Connectionless (similar to UDP) reliable service.

RDMA: Remote Direct Memory Access, is created to solve the delay of server-side data processing during network transmission. RDMA transfers data directly to the computer's storage area through the network, quickly moving data from one system to the remote system memory without causing any impact on the operating system. This does not require much computer processing power, eliminates the overhead of external memory copying and context switching, and frees up memory bandwidth and CPU cycles to improve application system performance.

The exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

FIG. 2 is a flowchart of an abnormality detection method for an RDMA system in an exemplary embodiment of the present disclosure.

Referring to FIG. 2 , the abnormality detection method of the RDMA system may include:

Step S202: abstractly modeling the working space of the RDMA system to generate a search space of the RDMA system.

Step S204: determine the workload according to the dimension of the search space.

In an exemplary embodiment of the present disclosure, the workload may be computer devices such as servers, terminals, and intermediate nodes, but is not limited thereto.

Step S206: determining the conditional features of any of the workloads triggering the RDMA system exception, so as to generate a minimum functional feature set of the exception.

In an exemplary embodiment of the present disclosure, the necessary conditions for triggering the anomaly are extracted by running a minimum feature set algorithm (MFS), and the extracted feature set is used to avoid repeated searches of known anomaly areas, thereby accelerating the search process. The MFS algorithm tests each feature of the anomaly one by one to determine which features are necessary.

In an exemplary embodiment of the present disclosure, if an exception is triggered using an RC queue pair, it is tested whether the exception can be reproduced when using a UC queue. If the exception cannot be reproduced, it indicates that the RC queue is one of the MFS features of the exception, and the MFS outputs the necessary feature combination that triggers the exception.

Step S208: for any of the minimum functional feature sets, call the counter of the RDMA system to determine an abnormality indicator of the workload.

In an exemplary embodiment of the present disclosure, the counters include performance counters and/or diagnostic counters. The performance counters mainly reflect the performance of the RDMA subsystem, such as throughput, latency, CPU usage, etc. The diagnostic counters mainly reflect errors or abnormal events inside the RNIC, such as cache miss, internal congestion, etc., but are not limited to this.

The disclosed embodiments generate a search space for the RDMA system by abstractly modeling the workspace of the RDMA system, and then determine the workload according to the dimension of the search space, determine the conditional features for triggering an abnormality in the RDMA system by any workload, and generate an abnormal minimum functional feature set. Finally, for any minimum functional feature set, call the counter of the RDMA system to determine the abnormal index of the workload, thereby realizing active detection of performance anomalies of the RDMA system and improving the performance and stability of the RDMA system in practical applications.

Next, each step of the abnormality detection method of the RDMA system is described in detail.

In an exemplary embodiment of the present disclosure, as shown in FIG3 , it further includes:

Step S302: adjusting the workload mode and/or the workload selection result based on the iterative calculation result of the abnormal indicator using a statistical algorithm.

In an exemplary embodiment of the present disclosure, as shown in FIG4 , adjusting the workload mode and/or the workload selection result based on the iterative calculation result of the abnormal indicator by a statistical algorithm includes:

Step S402: calling the annealing algorithm in the statistical algorithm to iteratively calculate the count difference between the count value of the workload and the count value of the new workload.

Step S404: determine a ratio between the count difference and the count value of the workload.

Step S406: adjusting the workload mode and/or the workload selection result according to the ratio value.

In an exemplary embodiment of the present disclosure, the workload mode includes an access mode, a message mode, a request mode, etc., but is not limited thereto. For example, a mode in which RDMA access does not require confirmation from the remote machine at all, or a mode in which RDMA access requires the participation of the remote machine's CPU. The workload mode determines the generated RDMA traffic.

In an exemplary embodiment of the present disclosure, the workload selection result includes whether to move the workload causing the abnormality to a new workload.

In an exemplary embodiment of the present disclosure, as shown in FIG5 , the mode of adjusting the workload and/or the selection result of the workload according to the ratio value includes:

Step S502: If it is determined that the ratio value is less than zero, the workload is transferred to the new workload.

In an exemplary embodiment of the present disclosure, the above ratio is recorded as ΔE. If ΔE<0, the workload is transferred to the new workload.

Step S504: If it is determined that the ratio value is greater than zero, the opposite of the ratio value is calculated using an exponential function with the natural constant e as the base.

Step S506: Determine the calculation result of the exponential function as the probability of transferring the workload to the new workload.

In an exemplary embodiment of the present disclosure, if ΔE>0, exp(-ΔE/T) is calculated as the probability of transferring the workload to the new workload.

In an exemplary embodiment of the present disclosure, as shown in FIG6 , the mode of adjusting the workload and/or the selection result of the workload according to the ratio value includes:

Step S602: adjusting the request mode of the RDMA system and/or the allocation information of the buffer of the RDMA system according to the ratio value.

In an exemplary embodiment of the present disclosure, as shown in FIG7 , abstract modeling of the working space of the RDMA system to generate the search space of the RDMA system includes:

Step S702: abstract the workspace of the RDMA system to determine a memory area corresponding to the workspace, and register the memory area.

Step S704: create the queue pair and set the transmission type of the queue pair.

Step S706: Create a work queue element according to the registered memory area and the queue pair with the transmission type set.

Step S708: Create the completion queue according to the work queue element to generate a search space of the RDMA system.

In an exemplary embodiment of the present disclosure, as shown in FIG8 , determining the workload according to the dimension of the search space includes:

Step S802, parsing the topology structure, host memory source, transmission mode and message mode included in the dimensions of the search space.

Step S804: Generate traffic between hosts of the RDMA system according to the topology, the host memory source, the transmission mode, and the message mode, and determine the workload corresponding to the traffic.

In an exemplary embodiment of the present disclosure, the transmission mode includes: QP type such as RC, UC, UD, etc.; the number of QPs; the operation code type; and the use of WQE.

The abnormality detection scheme of the RDMA system disclosed in the present invention is described in detail below with reference to FIG. 9 to FIG. 11 .

In an exemplary embodiment of the present disclosure, as shown in FIG. 9 , a system architecture of an anomaly detection solution for an RDMA system 900 is provided. The RDMA system 900 includes a workload engine, an anomaly monitor, a workload generator, and the like.

In an exemplary embodiment of the present disclosure, the workload engine is responsible for establishing RDMA traffic.

In an exemplary embodiment of the present disclosure, anomaly monitoring detects performance bottlenecks and a minimum necessary anomaly feature set of MFS based on traffic and PFC pause frames.

In an exemplary embodiment of the present disclosure, the workload generator obtains indicators such as hardware count to determine the working mode of the test.

In an exemplary embodiment of the present disclosure, the above steps are repeated for multiple rounds, and finally a feature set of RDMA performance anomalies, that is, a minimum functional feature set, is obtained.

In an exemplary embodiment of the present disclosure, as shown in FIG. 10 , a search space 1000 of an anomaly detection solution for an RDMA system is provided. Based on the dimension parameters set in the search space, corresponding network traffic is generated between hosts through an RDMA network interface card, and is specifically modeled in the following four dimensions:

(1) Topology: How traffic flows into/out of the RNIC and to/from other server hard-commodity components.

(2) Memory allocation settings include configuring and dividing storage areas.

(3) Transmission settings: (a) QP type (RC, UC, UD); (b) Number of QPs; (c) Opcode type; (d) WQE usage. You can also set Opcodes such as SEND, WRITE, etc.

(4) Message mode: You can flexibly set the message size and define the order of a series of messages.

Furthermore, the workload engine in the search space 1000 can combine and test different RDMA operations in the search space, abstractly model the workspace, and simulate the application's operations on RDMA, so as to quickly search for feature sets of RDMA performance anomalies.

In an exemplary embodiment of the present disclosure, as shown in FIG11 , the abnormality detection scheme of the RDMA system includes the following steps:

Step S1102: The workload setter constructs an abstract RDMA search space, including:

(1) Initialize S: workspace settings;

(2) Initialization temperature: annealing algorithm temperature indicator;

(3) Initialize N: workload dimension.

Step S1104: The workload engine generates RDMA traffic according to the workload pattern of the workload setter.

Step S1106: The abnormality monitor determines whether there is an abnormality based on the RDMA system indicators.

Step S1108: trigger an exception.

Step S1110: The anomaly monitor generates a minimum anomaly set using the MSF algorithm.

Step S1112: The workload setter obtains a minimum abnormal feature set.

The above steps S1104 to S1112 are iterated repeatedly, and finally a feature set of all detected RDMA performance anomalies is output.

Step S1114: The workload setter obtains the RDMA system counter.

Step S1116: The workload setter uses an annealing algorithm to calculate the energy function.

Step S1118: The annealing algorithm calculated by the workload setter selects whether to accept the workload based on the energy value and the annealing temperature.

Step S1120: The workload setter determines whether to update the annealing parameter temperature according to the number of iterations.

Step S1122: The workload setter determines the next workload pattern according to the RDMA system (subsystem or main system) counter and the current search space.

Based on the above steps S1102 to S1122, a search space is constructed from the developer's perspective, and a comprehensive search space is constructed by analyzing the standard verbs library and the design decisions that developers can make, such as request patterns, how to allocate RDMA buffers, etc. Key abstractions in RDMA programming are constructed, including memory regions (MRs), queue pairs (QPs), work queues (WQEs), and completion queues (CQs).

Among them, the dimensions that affect performance are: host topology, memory allocation, transmission settings, and message mode. Four key dimensions that can affect the performance of the RDMA subsystem. These abstractions will have different effects on the performance of RDMA applications under different settings. By simulating different workloads and using performance counters, anomalies can be discovered.

The following is a detailed description of each modeling module:

(1) Memory Region (MR): The application needs to register the memory region first to make it accessible to RDMA. Use the function to register the MR. The MR has a start address and length, defining a continuous memory space. The RNIC can directly access the registered MR without CPU involvement.

(2) Queue Pair (QP): Use ibv_create_qp to create a QP. QP represents a "connection" between the application and the RNIC. QP is an abstraction of point-to-point communication, including a send queue and a receive queue. Each QP needs to be configured with a transmission mode, such as a reliable connection RC.

(3) Work Queue Element (WQE): To send/receive messages, you need to construct a WQE and deliver it to the QP. The WQE contains a scatter/gather list that specifies a set of MR memory buffers that participate in the transmission. The WQE is delivered to the QP sending queue.

(3) Completion Queue (CQ): CQ is used to receive completion notifications and determine whether the WQE is completed. Create a CQ and query the CQ to obtain the completion notification. After obtaining the notification, the WQE memory buffer can be reused.

Among them, the specific workload is generated according to the search space dimension, which can be modeled according to the following four dimensions:

(1) Topology: How traffic flows into/out of the RNIC and to/from other server hard-commodity components.

(2)Memory allocation settings.

(3) Transmission settings: (a) QP type (RC, UC, UD), you can set Opcode such as SEND, WRITE, etc.; (b) Number of QPs; (c) Opcode type; (d) WQE usage.

(4) Message mode: The message size can be flexibly set, and the order of a series of messages can be defined) through the RDMA network interface card to generate corresponding network traffic between hosts.

After completing the construction of the search space, the search process of the RDMA system uses a simulated annealing algorithm to effectively search for application workloads that may trigger performance anomalies by driving the extreme values of hardware counters.

The process of searching for RDMA performance anomalies using the annealing algorithm is as follows:

(1) Use the workload setter to select an initial random workload, i.e., the initial state.

(2) The workload engine generates RDMA traffic based on the workload pattern of the workload setter and sends it to the RDMA system.

(3) The anomaly detector performs anomaly detection based on the anomaly indicators of the RDMA system to determine whether an anomaly is triggered.

(4) If the anomaly detector detects a new anomaly, the anomaly monitor: runs the minimum feature set algorithm (MFS) to extract the necessary conditions to trigger the anomaly. The extracted feature set is used to avoid repeated searches of known anomaly areas, thereby speeding up the search process. The MFS algorithm tests each feature of the anomaly one by one to determine which features are necessary. For example: If an anomaly is triggered by an RC queue pair, test whether the anomaly can be reproduced using UC. If not, it indicates that RC is one of the MFS features of the anomaly, and MFS outputs the necessary feature combination to trigger the anomaly.

(5) The workload setter obtains the minimum anomaly feature set.

(6) The workload setter obtains the RDMA system counters. The counter values of the workload points are checked. The counter interface is obtained through the RNIC, including performance counter values and diagnostic counters. (These counter data do not rely on proprietary knowledge and do not require access to the internal implementation details of the RNIC and other hardware).

(7) The workload setter uses an annealing algorithm to calculate the energy function. In this disclosure, the smaller the performance counter, the better, and the larger the diagnostic counter, the better as the optimization goal. Calculate the energy difference ΔE between the current point and the new point counter value. For the performance counter (to reduce its value as much as possible), ΔE is defined as: ΔE = (new counter value - old counter value) / old counter value. For the diagnostic counter (to increase its value as much as possible), ΔE is defined as: ΔE = (old counter value - new counter value) / new counter value. The performance counter mainly reflects the performance of the RDMA subsystem, such as throughput, latency, CPU usage, etc. The goal of this disclosure is to find workloads that cause performance anomalies (reduced throughput, increased latency, etc.). Therefore, if any workload makes the performance counter value smaller, it means that the performance is reduced and it is likely to trigger an anomaly. Therefore, for the performance counter, the smaller its value, the more likely it is to find a performance anomaly. The diagnostic counter mainly reflects errors or abnormal events inside the RNIC, such as cache miss, internal congestion, etc. These counters usually only rise when there is a problem with the RNIC. Therefore, if any workload makes the diagnostic counter value larger, it means that the internal anomaly of the RNIC has increased and there is a high probability of a problem. So for diagnostic counters, the larger the value, the more likely it is to find performance anomalies.

(8) The workload setter uses the annealing algorithm to calculate the energy function result and decide whether to accept it: if the energy change indicates that the counter value has been optimized, specifically ΔE<0, the new point is accepted; if the energy change indicates a deterioration, ΔE>0, the new workload is accepted with probability exp(-ΔE/T).

Furthermore, as the number of iterations increases, it is determined whether to change the temperature parameters of the annealing algorithm. As the number of search rounds increases, the probability acceptance metric is gradually reduced to make the search more focused.

(9) The workload setter determines whether to update the annealing temperature parameter based on the number of iterations.

(10) The workload setter determines the next workload pattern based on the RDMA subsystem counter and the current search space. The workload engine generates RDMA traffic based on the workload setter workload pattern and sends it to the RDMA system.

In summary, the above steps are repeated and iterated, and finally a feature set of all detected RDMA performance anomalies is output.

In an exemplary embodiment of the present disclosure, the embodiment of the present disclosure maintains a list of performance anomalies, each of which is a minimum fault region (corresponding to MFS), such as an area in the search space, which causes the performance anomaly. The search starts with a random workload in the search space, and the algorithm of the embodiment of the present disclosure measures the value of the counter. In each iteration of SA, the embodiment of the present disclosure changes the workload in the search dimension. The embodiment of the present disclosure uses an anomaly monitor to test whether the new workload causes the performance anomaly. If so, the embodiment of the present disclosure runs the MFS algorithm of the embodiment of the present disclosure to determine the area belonging to the anomaly in the entire search space. The embodiment of the present disclosure adds the new anomaly to the collection and changes the current workload to a random workload. If the new workload does not trigger the performance anomaly, the embodiment of the present disclosure measures the point by comparing the value of the counter and decides whether to move the current workload to the new workload. For efficient search, the embodiment of the present disclosure always skips the workload belonging to the existing performance anomaly.

Furthermore, the detection of abnormal conditions disclosed in the present invention includes: in each round of iteration, the anomaly detector determines whether the new workload triggers an RDMA performance abnormality.

Abnormal detection condition 1: If a pause frame is generated and the proportion of the pause frame exceeds 0.1%, it is determined that an abnormal bottleneck is likely to occur.

Anomaly detection condition two: Each (remote network interface card) has its own specified maximum number of bits per second and maximum number of packets per second, which can be verified through simple benchmark tests. If the workload's throughput is less than 20% of these upper limits, it means that performance is likely limited by other bottlenecks in the RDMA (remote direct memory access) subsystem.

Furthermore, the processing process of extracting feature sets by the anomaly detector disclosed in the present invention is as follows:

If a new anomaly occurs, the anomaly monitor runs the minimum feature set algorithm (MFS) to extract the necessary conditions to trigger the anomaly. The extracted feature set is used to avoid repeated searches of known anomaly areas, thereby speeding up the search process. The MFS algorithm tests each feature of the anomaly one by one to determine which features are necessary. For example: If an anomaly is triggered by an RC queue pair, test whether the anomaly can also be reproduced using UC. If not, it indicates that RC is one of the MFS features of the anomaly. MFS outputs the necessary feature combination that triggers the anomaly. After multiple rounds of repeated iterations, the feature set of all detected RDMA performance anomalies is finally output.

Corresponding to the above method embodiment, the present disclosure also provides an abnormality detection device for an RDMA system, which can be used to execute the above method embodiment.

FIG. 12 is a block diagram of an abnormality detection device for an RDMA system in an exemplary embodiment of the present disclosure.

Referring to FIG. 12 , an abnormality detection device 1200 of an RDMA system may include:

The modeling module 1202 is configured to perform abstract modeling on the working space of the RDMA system to generate a search space of the RDMA system.

The determination module 1204 is configured to determine the workload according to the dimension of the search space.

The determination module 1204 is configured to determine the conditional features of any of the workloads triggering the RDMA system abnormality, so as to generate a minimum functional feature set of the abnormality.

The determination module 1204 is configured to call the counter of the RDMA system to determine the abnormal index of the workload for any of the minimum functional feature sets.

In an exemplary embodiment of the present disclosure, the abnormality detection device 1200 of the RDMA system is further configured as follows:

The workload mode and/or the workload selection result are adjusted based on the iterative calculation result of the abnormal indicator using a statistical algorithm.

In an exemplary embodiment of the present disclosure, the abnormality detection device 1200 of the RDMA system is further configured as follows:

Calling an annealing algorithm in the statistical algorithm to iteratively calculate a count difference between the count value of the workload and the count value of a new workload;

determining a ratio value between the count difference value and the count value of the workload;

The workload mode and/or the workload selection result is adjusted according to the ratio value.

In an exemplary embodiment of the present disclosure, the abnormality detection device 1200 of the RDMA system is further configured as follows:

If it is determined that the ratio value is less than zero, transferring the workload to the new workload;

If it is determined that the ratio value is greater than zero, then calculating the opposite of the ratio value using an exponential function with a natural constant e as the base;

The calculation result of the exponential function is determined as the probability of transferring the workload to the new workload.

In an exemplary embodiment of the present disclosure, the abnormality detection device 1200 of the RDMA system is further configured as follows:

The request mode of the RDMA system and/or the allocation information of the buffer of the RDMA system are adjusted according to the ratio value.

In an exemplary embodiment of the present disclosure, the modeling module 1202 is further configured to:

Abstracting the workspace of the RDMA system to determine a memory area corresponding to the workspace, and registering the memory area;

Creating the queue pair and setting the transmission type of the queue pair;

Create a work queue element based on the registered memory area and queue pair with the transfer type set;

The completion queue is created according to the work queue element to generate a search space of the RDMA system.

In an exemplary embodiment of the present disclosure, the determination module 1204 is further configured to:

Parsing the topology, host memory source, transmission mode and message mode included in the dimensions of the search space;

The traffic between hosts of the RDMA system is generated according to the topology, the host memory source, the transmission mode and the message mode, and the workload corresponding to the traffic is determined.

Since the functions of the device 1200 have been described in detail in the corresponding method embodiments, the present disclosure will not elaborate on them here.

It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the conditional features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the conditional features and functions of one module or unit described above can be further divided into being concretized by multiple modules or units.

In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

The electronic device 1300 according to this embodiment of the present invention is described below with reference to Fig. 13. The electronic device 1300 shown in Fig. 13 is only an example and should not bring any limitation to the functions and application scope of the embodiment of the present invention.

As shown in Fig. 13, the electronic device 1300 is presented in the form of a general-purpose computing device. The components of the electronic device 1300 may include, but are not limited to: the at least one processing unit 1310, the at least one storage unit 1320, and a bus 1330 connecting different system components (including the storage unit 1320 and the processing unit 1310).

The storage unit stores program codes, which can be executed by the processing unit 1310, so that the processing unit 1310 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 1310 can perform the method shown in the embodiment of the present disclosure.

The storage unit 1320 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 13201 and/or a cache storage unit 13202 , and may further include a read-only storage unit (ROM) 13203 .

The storage unit 1320 may also include a program/utility 13204 having a set (at least one) of program modules 13205, such program modules 13205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

Bus 1330 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

The electronic device 1300 may also communicate with one or more external devices 1340 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1300, and/or communicate with any device that enables the electronic device 1300 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input/output (I/O) interface 1350. Furthermore, the electronic device 1300 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and/or public networks, such as the Internet) via a network adapter 1360. As shown, the network adapter 1360 communicates with other modules of the electronic device 1300 via a bus 1330. It should be understood that, although not shown in the figure, other hardware and/or software modules may be used in conjunction with the electronic device 1300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps according to various exemplary implementations of the present invention described in the above "Exemplary Method" section of the present specification.

The program product for implementing the above method according to an embodiment of the present invention can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and concept of the present disclosure are indicated by the claims.

Claims (10)

1. An anomaly detection method for an RDMA system, comprising:

abstract modeling a workspace of the RDMA system to generate a search space of the RDMA system;

Determining a workload according to the dimension of the search space;

Determining a condition feature of any of the workloads triggering the RDMA system exception to generate a minimum set of functional features of the exception;

And for any minimum functional feature set, calling a counter of the RDMA system to determine an abnormality index of the workload.

2. The RDMA system anomaly detection method as recited in claim 1, further comprising:

And adjusting the mode of the workload and/or the selection result of the workload based on the iterative calculation result of the abnormal index by a statistical algorithm.

3. The RDMA system anomaly detection method of claim 2, wherein adjusting the workload pattern and/or the workload selection result based on the iterative computation of the anomaly metrics by a statistical algorithm comprises:

Invoking an annealing algorithm in the statistical algorithm to iteratively calculate a count difference between the count value of the workload and the count value of a new workload;

determining a ratio value between the count difference value and the count value of the workload;

and adjusting the mode of the workload and/or the selection result of the workload according to the proportion value.

4. The RDMA system anomaly detection method of claim 3, wherein adjusting the workload pattern and/or the workload selection result according to the scale value comprises:

if the ratio value is determined to be smaller than zero, transferring the workload to the new workload;

If the ratio value is determined to be greater than zero, calculating the opposite number of the ratio value by using an exponential function based on a natural constant e;

the calculation of the exponential function is determined as a probability of transferring the workload to the new workload.

5. The RDMA system anomaly detection method of claim 3, wherein adjusting the workload pattern and/or the workload selection result according to the scale value comprises:

And adjusting the request mode of the RDMA system and/or the allocation information of the buffer area of the RDMA system according to the proportion value.

6. The RDMA system anomaly detection method of any of claims 1-5, wherein abstractly modeling a workspace of the RDMA system to generate a search space of the RDMA system comprises:

Abstracting a working space of the RDMA system to determine a memory area corresponding to the working space, and registering the memory area;

creating the queue pair and setting the transmission type of the queue pair;

Creating a work queue element according to the registered memory area and the queue pair provided with the transmission type;

Creating the completion queue from the work queue element to generate a search space for the RDMA system.

7. The RDMA system anomaly detection method of any of claims 1-5, wherein determining a workload from the dimensions of the search space comprises:

analyzing a topological structure, a host memory source, a transmission mode and a message mode which are included in the dimension of the search space;

And generating traffic among hosts of the RDMA system according to the topological structure, the host memory source, the transmission mode and the message mode, and determining the workload corresponding to the traffic.

8. An abnormality detection apparatus for an RDMA system, comprising:

A modeling module configured to abstract model a workspace of the RDMA system to generate a search space of the RDMA system;

A determining module arranged to determine a workload from the dimensions of the search space;

the determining module is configured to determine a condition feature of any of the workloads that triggers the RDMA system exception to generate a minimum set of functional features of the exception;

the determining module is configured to invoke a counter of the RDMA system to determine an abnormality indicator of the workload for any of the minimum feature sets.

9. An electronic device, characterized by comprising:

a memory; and

A processor coupled to the memory, the processor configured to perform the anomaly detection method of the RDMA system of any of claims 1-7 based on instructions stored in the memory.

10. A computer readable storage medium having stored thereon a program which, when executed by a processor, implements the anomaly detection method of the RDMA system of any of claims 1-7.