WO2021102617A1 - Multi-public cloud computing platform-oriented cluster monitoring system and monitoring method therefor - Google Patents

Abstract

The present invention provides a multi-cloud computing platform-oriented cluster monitoring system, comprising: a data acquisition subsystem, configured to acquire basic resource data, task running state and consumption, and overall resource usage monitoring data of each cloud computing platform according to given indicators, and provide an interface for a scheduling system to call to obtain real-time monitoring data to guide scheduling; a data processing subsystem, configured to perform a series of processing on cluster monitoring data uploaded by each data subsystem by means of an RPC request and transfer same to a backend for storage, and provide an interface for data display and for a bill system; and an alarm subsystem, configured to process and analyze the monitoring data according to an alarm strategy, determine an alarm level, and send alarm information. The present invention can facilitate better viewing the overall task running condition, improve the resource utilization, make computing resources persistent to facilitate bill audit and account checking, automatically solve some abnormalities, and reduce manual participation.

Description

Multi-public cloud computing platform-oriented cluster monitoring system and monitoring method thereof Technical field

The invention belongs to the technical field of multi-public cloud computing scheduling, and specifically relates to a cluster monitoring system and a monitoring method for multi-public cloud computing platforms, which can be used in a multi-cloud computing platform cluster as a multi-computing cluster resource state, computing Task status monitoring alarm system and data support system of other related systems.

Background technique

Cloud computing is a pay-per-use model that provides usable, convenient, and on-demand network access and enters a configurable computing resource sharing pool (resources include networks, servers, storage, application software, and services) , These resources can be provided quickly, with little management effort or little interaction with service providers.

With the development of technologies such as the Internet and cloud computing, more and more shared cloud computing resources are available for selection. Based on computing cost and regionalization considerations, computing task scheduling across multiple public clouds has become a trend. At the same time, the complexity of cloud computing itself is also increasing. In view of cost and complexity considerations, public cloud providers cannot provide complete monitoring indicators and monitoring data for each computing resource, so users can only rely on limited monitoring data. Make a decision.

Each cloud computing provider will provide visual resource monitoring for users to have a rough grasp of resource usage in a certain period of time and provide thresholds based on resource data to perform alarm actions. An effective monitoring system can manage the cloud computing platform agilely, thereby ensuring the availability and security of the entire scheduling computing process under the continuous iteration of the cloud platform and user computing platform.

The current public cloud monitoring system has problems mainly in the following aspects:

1. Cloud computing providers only provide basic monitoring indicators and monitoring data of computing resources. The computing platform is mainly based on large-scale computing-intensive tasks, which will use a large amount of CPU resources for computing. Cloud providers only provide basic resource monitoring such as CPU, memory, and network of computing nodes, but these basic monitoring data are not enough. To meet the needs of computing platforms. Currently, computing platforms on the market are based on kubernetes or mesos, and need to monitor the real-time running status of each computing task on the platform, as well as monitoring indicators such as task resource requests and actual usage. Currently, cloud providers are unable to complete the support for these monitoring indicators.

2. Based on cost considerations, cloud providers cannot customize the collection of monitoring indicators and cannot perform more detailed analysis of monitoring data, and thus cannot feed back the scheduling system to adjust strategies in real time. In 1 it is explained that cloud providers cannot provide some monitoring indicators required by cloud computing platforms, and cloud providers cannot allow users to collect and display these indicators in a certain way. The existing monitoring data is stored on the cloud and there is no way to obtain historical data for a certain period of time for more detailed analysis, and then the resource usage cannot be checked, which makes it impossible to check the resource usage bill. If the data that the scheduling system relies on for policy adjustment cannot be obtained, there is no real-time scheduling strategy, and a single scheduling strategy may cause unnecessary waste of resources.

3. The built-in resource monitoring views are scattered and cannot provide a user-customizable unified view. Moreover, the monitoring methods of various cloud providers are different from each other. It is difficult to have a unified way to process the resource monitoring data of each cloud. Different from the traditional operation and maintenance monitoring system, the computing platform is more concerned with the overall operating status of a certain cluster rather than the basic resource status of a certain machine, such as the overall resource allocation rate of the cluster, the resource request rate, and the consumption speed of computing tasks. Cloud provides not only the collection of certain indicators but also the integration of basic resources is difficult to achieve, which also cannot provide an intuitive view of the resource status.

4. The alarm system can only be set separately for a certain resource or a certain type of resource and cannot classify the alarm. Cloud providers can provide threshold alarms for basic resources, but cannot classify these alarms. When there are too many alarm information, it is easy to cause critical alarm information to be ignored and cannot be processed in a timely manner. In addition, the existing alarm strategy cannot meet the alarm requirements of the cloud computing platform at all, and thus cannot fully grasp the runtime state of the computing platform.

Summary of the invention

technical problem

The solution to the problem

Technical solutions

In view of the above technical problems, the present invention provides a cluster monitoring system for multiple public cloud computing platforms and a monitoring method thereof, which realizes the acquisition of monitoring data of computing clusters on multiple public clouds, and supports the collection and monitoring of monitoring indicators that the computing platform depends on. Data analysis, dumping, and feedback to the scheduling system and reconciliation system; support custom warning classification and different alarm methods; support resource monitoring views for the aggregation of various cloud monitoring data.

The specific technical solutions are as follows:

The cluster monitoring system of the multi-cloud computing platform includes three subsystems:

The data collection subsystem is responsible for collecting basic resource data, task operation status and consumption, overall resource usage status and other monitoring data of each cloud computing platform according to the established indicators, and provides an interface for the scheduling system to call to obtain real-time monitoring data to guide scheduling;

The data processing subsystem is responsible for the cluster monitoring data uploaded by each data subsystem through RPC request, performs a series of processing and dumps it to the back-end for storage, and provides interfaces for data display, billing system and other functions;

The alarm subsystem is responsible for processing and analyzing monitoring data according to the alarm strategy, confirming the alarm level and sending alarm information. For the following low-level alarms, the subsystem can process and recover by itself according to the preset method.

The data acquisition subsystem can be divided into three modules according to the type of data to be collected, namely:

In the cluster node information collection module, due to the randomness of the scheduling system to distribute computing tasks, it needs to cooperate with the number of tasks to monitor whether there are enough machines to meet the task scheduling. It is very necessary to observe the number of nodes in each cluster in real time. This system is responsible for collecting the actual number of nodes in the computing cluster nodes on each cloud computing provider, the planned number of nodes, and the maximum number of supported nodes. A unified application interface layer is implemented on this module to interface with the basic monitoring interfaces of various cloud providers;

Cluster computing resource information collection module. The cluster runs computationally intensive tasks. Increasing the CPU usage can greatly save computing costs. This module is responsible for collecting the CPU information of each computing cluster, including the total number of CPUs and task request usage. The number of CPUs, the number of CPUs actually used by the task. On this module, by deploying third-party plug-ins (Heapster, Metrics-Server, Prometheus) in the cluster, a unified resource interface encapsulation is implemented for these plug-ins, and the number of CPUs used by task requests and the number of CPUs actually used by tasks can be obtained in real time. For the total number of CPUs, we implemented the acquisition interface according to the situation of each cloud;

Calculating task status collection module, this module collects all task information in the cluster in real time, and classifies tasks through a classification sub-module (such as Running, Pending, Evicted, ImagePull, PodInitializing, etc.). These data will provide a powerful alarm system data support.

In addition, the data collection subsystem will provide a cluster state definition function. We analyze the various states (such as expansion state, shrinking state, stable state, full load state, etc.) that will appear during the entire cluster calculation process. These states can be more It intuitively reflects the current runtime situation of the cluster, and will form a monitoring information to guide the scheduling according to the needs of the scheduling system.

The data processing subsystem can be divided into three modules according to different functions, namely:

The monitoring information aggregation module is responsible for processing the monitoring data uploaded by the data collection subsystem, performing a series of processing and analysis on the cluster information and task operation information on each cloud, and doing different levels of aggregation and then dumping for display and auditing. ；

The bill information processing module, according to the requirements of the bill, processes the data required for bill analysis and stores it in the time series database every minute;

The task transfer module is responsible for automatically transferring tasks to clusters with low load based on the monitoring information of each cluster and the task information of the scheduling system, thereby shortening the waiting time of tasks and improving the utilization of computing resources.

The alarm subsystem can be divided into three modules according to different alarm types, namely:

The alarm strategy processing module is responsible for implementing the data processing logic of the alarm strategy. Different alarm strategies will have different processing logic;

The cluster resource usage alarm module is responsible for executing the alarm strategy logic related to the cluster information, classifying the strategy processing results, and then selecting different channels to send the alarm information according to the severity of the alarm;

The computing task running status alarm module is responsible for executing the alarm strategies related to the computing task. Due to the diversity of task status, this module will define the priority of various status detections and perform sequential detection according to the priority, and perform tasks in abnormal conditions. Carry out sending alarm information processing. At the same time, this module defines relative solutions for some low-level exceptions, and will execute the solutions while sending alarm information.

The monitoring method of the cluster monitoring system for multiple public cloud computing platforms is specifically coordinated by three subsystems. The steps for each subsystem are described below:

The data collection subsystem is distributed and runs on the computing cluster of the cloud provider. The specific steps are as follows:

(1) Through a unified interface layer function, the collection of cluster node information, cluster resource information, and computing task status information is performed synchronously, which are described below:

(1.1) Cluster node information collection module. First, the unified interface layer of this module completes the access to the cloud provider, and then uses the unified interface layer to obtain cluster node information, that is, the actual number of nodes, the planned number of nodes, and the maximum number of supported nodes; and finally Into the memory for temporary storage.

(1.2) Cluster computing resource information collection module. First of all, the unified interface layer of this module completes the encapsulation of cluster resource collection plug-ins, supporting various plug-ins such as Heapster, Metrics-Server, and Prometheus; secondly, confirm the plug-ins supported by the cluster (provided by different clouds). The plug-ins supported by vendors are different); finally, the total number of CPUs, the number of CPUs requested by the task, and the number of CPUs actually used by the task are obtained through the unified interface layer. Finally, it is temporarily saved in the memory.

(1.3) The computing task state information collection module first obtains all task information of the current cluster through the native interface layer of the cluster computing engine; secondly, it uses a state classifier to classify all states and extract key information; finally, it is temporarily stored in the memory.

(2) When the above steps (1.1), (1.2), and (1.3) complete a round of collection, the cluster state definition function will load the monitoring data in the memory to analyze and determine the current state of the cluster.

(3) Extract the latest monitoring data and upload it to the data processing subsystem through RPC request.

(4) Streamline and process monitoring data according to the requirements of the dispatch system, and store the latest results for the dispatch system to call.

The data processing subsystem receives the monitoring data information and performs corresponding processing. The specific steps are as follows:

(1) Receive monitoring data pushed by each cluster;

(2) Analyze monitoring information according to different dimensions, such as gathering computing node information, CPU usage information, etc. according to cloud provider dimensions, and persisting the processed data to the database;

(3) According to the requirements of the billing system, process the billing system in a recognizable way and store it in the designated time series database;

(4) Pull the real-time task data of the scheduling system, analyze the monitoring data information of each cluster uniformly, and screen the transferable tasks and the corresponding number for task transfer.

The alarm subsystem processes the monitored data information according to the alarm strategy logic and the specific steps are as follows:

(1) Execute alarm strategy logic related to cluster information according to monitoring data information, and execute alarm actions based on abnormal levels.

(2) According to the monitoring data information, execute the alarm strategy related to the calculation task, and execute the alarm action according to the abnormal level. For low-level exceptions with preset solutions, execute the solutions corresponding to the exceptions.

The beneficial effects of the invention

Beneficial effect

The cluster monitoring system and monitoring method for multiple public cloud computing platforms provided by the present invention have the following technical effects:

(1) Through the integration of the computing platform interfaces of major cloud providers, through further processing to eliminate the differences between each cloud, provide standard format monitoring resource data, accelerate the deployment of new clusters; complete the plug-ins for collecting various resources of the cluster The support interface of, can ensure the uniformity of the format of the monitoring data on the basis of ensuring the diversity of the cluster plug-in; the monitoring classification function can display the current cluster task running in the foreground according to the running progress, so that the user can better observe the overall task running situation .

(2) Through a series of monitoring data analysis modules, the cluster monitoring data can be analyzed and processed, and then feedback the scheduling system to adjust the scheduling strategy and improve the resource utilization rate. And analyze and study scheduling data and monitoring data, adjust task distribution, estimate batch task completion time, automatically transfer tasks, and shorten task waiting time.

(3) Centralized cluster monitoring view, you can see the runtime status of each cluster including (node information, CPU information, task information, etc.); the aggregated resource view, you can see the overall resources at the cloud provider level Information; the persistence of computing resources facilitates bill auditing and reconciliation.

(4) Dynamically monitor the resource utilization rate, task status, etc. of each cluster, and use different channels to send formatted alarm information according to the severity of the abnormality; and some preset solutions can automatically solve some abnormal problems and reduce manual participation .

Brief description of the drawings

Description of the drawings

Figure 1 is a diagram of the overall architecture of the monitoring system of the present invention;

Figure 2 is a system structure diagram of the data acquisition subsystem of the present invention;

Figure 3 is a system structure diagram of the data processing subsystem of the present invention;

Figure 4 is a system structure diagram of the alarm subsystem of the present invention;

Figure 5 is a flow chart of the implementation of the entire system of the present invention.

Invention embodiment

Embodiments of the present invention

The technical solutions of the present invention will be further described in detail below through the accompanying drawings and embodiments.

Figure 1 is the overall architecture of the system, used to illustrate the relationship between the various subsystems and the flow of data:

The cluster collection subsystem runs on the cluster clusters of each cloud, and uploads the collected monitoring data to the remote data processing system. After a series of analysis, the data is dumped and sent to the alarm subsystem. The alarm subsystem is based on The alarm strategy processes the monitoring data and then performs alarm actions.

Figure 2 is the architecture diagram of the data acquisition subsystem, combined with this figure to illustrate how this system works.

First, complete support for cloud provider computing clusters, resource plug-ins, and scheduling engine APi; secondly, according to different modules, call the unified interface layer function of the module to complete the collection of basic monitoring data, especially for task information, a task classifier is required. Task information is subdivided; then the monitoring data is gathered, standardized processing, and additional processing related data, waiting for S12 (scheduling system) to call and guide the scheduling strategy; finally upload the monitoring data to the data processing subsystem S11 .

Figure 3 is the architecture diagram of the data processing subsystem, combined with this figure to illustrate how this system works.

First, the subsystem receives the monitoring data uploaded by each cluster, and after each cluster has uploaded the data once or within a certain period of time, the subsystem will preprocess the data to determine whether the data is valid or not expired, etc., if the data is valid, Make a copy and send it to the alarm subsystem (S21); secondly, the aggregation module and the billing information processing module begin to process the monitoring data at the same time, and the aggregation module processes the monitoring information according to different dimensions (such as cloud provider, computing engine type, Task type, etc.), processed and stored in the library and displayed in the front-end view at the same time. The billing information processing module filters computing resource information (the total number of CPUs, the number of requests, the number of nodes, etc.) according to the billing system requirements, and processes them according to the dimensions of the cloud provider, and then stores them in the time series database for the billing system (S23) to call. Then the task transfer module pulls the computing task information from the scheduling system and combines the collected monitoring data to perform migration data calculation for tasks according to the principle of saving task queuing time and maximizing resource utilization, such as tasks that have been waiting for a long time in a batch of clusters Transfer to a cluster where other resources are relatively idle, and then call the migration interface (S22) of the scheduling system to complete the migration action.

Figure 4 is the architecture diagram of the alarm subsystem, combined with this figure to illustrate how this system works.

First obtain the overall monitoring data of the cluster from the data collection subsystem, and the alarm strategy processing module will process the data according to the strategy logic of the subsystem; then send it to the respective alarm module according to different alarm types, and the cluster resource usage alarm module receives the alarm. After the data is collected, the monitoring data is differentiated according to the level of the alarm abnormality; after the calculation task running state alarm module receives the alarm data, it sends the alarm data and finds whether there is a preset solution according to the alarm, and executes the solution if there is one and informs the relevant personnel .

Figure 5 is a schematic diagram of the implementation of the entire system, the specific operation process is as follows:

Step 1: Connect the new cluster to the cluster and confirm whether the cloud provider is already supported. If not, support for the cloud provider needs to be completed;

Step two, confirm whether the cluster resource plug-in and whether the plug-in is already supported, if it is supported, just deploy the collection subsystem directly, if not, you need to complete the support for the plug-in before deploying;

Step 3: Collect cluster monitoring data. The three types of data are collected in parallel, and after collecting task information, they are classified using a classifier;

Step four, standardize the monitoring data, the purpose is to facilitate storage and analysis;

Step 5: Analyze and cache the data required by the scheduling system for the scheduling system to call;

Step 6, upload monitoring data to the data processing subsystem, and the data acquisition subsystem process ends;

Step 7. The data processing system preprocesses the monitoring data uploaded by each cluster to confirm the validity;

Step 8. Process the bill-related data according to the billing system requirements and store it in the time series database for use by the billing system;

Step 9: Gather the bill information according to different dimensions and store it in the database;

Step 10, pull task information and monitor data to complete task analysis and transfer;

Step 11, the front-end updates the display in real time, and the data processing subsystem flow ends;

Step 12: The alarm processing module receives the monitoring data and starts processing according to the alarm strategy. This step starts to be executed after step 7;

Step 13: The cluster resource usage alarm module differentiates the alarm level and sends alarm information;

Step 14. The computing task running state alarm module sends alarm information according to the alarm level;

Step 15: When searching for solutions with task status alarm presets, if there are solutions, execute the solutions and notify relevant personnel;

Step 16, the whole process ends.

Claims (7)

The cluster monitoring system for multiple public cloud computing platforms is characterized by three subsystems: The data collection subsystem is responsible for collecting basic resource data, task operation status and consumption, and overall resource usage monitoring data of each cloud computing platform according to the established indicators, and provides an interface for the scheduling system to call to obtain real-time monitoring data to guide scheduling; The data processing subsystem is responsible for the cluster monitoring data uploaded by each data subsystem through RPC request, performs a series of processing and dumps it to the back-end for storage, and provides interfaces for data display and billing system functions; The alarm subsystem is responsible for processing and analyzing monitoring data according to the alarm strategy, confirming the alarm level and sending alarm information. For the following low-level alarms, the alarm subsystem can process and recover by itself according to the preset method. The cluster monitoring system for multiple public cloud computing platforms according to claim 1, wherein the data collection subsystem includes three modules, which are: The cluster node information collection module is responsible for collecting the actual number of nodes, the planned number of nodes, and the maximum number of supported nodes of the computing cluster nodes on each cloud computing provider; The cluster node information collection module implements a unified application interface layer, which is used to interface with the basic monitoring interfaces of various cloud providers; The cluster computing resource information collection module is responsible for collecting the CPU information of each computing cluster, including the total number of CPUs, the number of CPUs used by task requests, and the number of CPUs actually used by the task; the cluster computing resource information collection module deploys third-party plug-ins in the cluster, A unified resource interface package is implemented for these plug-ins, and the number of CPUs requested by the task and the number of CPUs actually used by the task can be obtained in real time; for the total number of CPUs, the acquisition interface is implemented according to the conditions of each cloud; The computing task status collection module collects all task information in the cluster in real time, classifies tasks through a classification sub-module, and the task information provides data support for the alarm system. The cluster monitoring system oriented to multiple public cloud computing platforms according to claim 2, wherein the data collection subsystem also includes a cluster state definition function, and various states that will appear in the cluster computing process are intuitively reflected Based on the current runtime situation of the cluster, a piece of monitoring information to guide the scheduling is formed according to the requirements of the scheduling system. The cluster monitoring system for multiple public cloud computing platforms according to claim 1, wherein the data processing subsystem includes three modules, respectively: The monitoring information aggregation module is responsible for processing the monitoring data uploaded by the data collection subsystem, performing a series of processing and analysis on the cluster information and task operation information on each cloud, and doing different levels of aggregation and then dumping for display and auditing. ； The bill information processing module, according to the requirements of the bill, processes the data required for bill analysis and stores it in the time series database every minute; The task transfer module is responsible for automatically transferring tasks to clusters with low load based on the monitoring information of each cluster and the task information of the scheduling system. The cluster monitoring system for multiple public cloud computing platforms according to claim 1, characterized in that, in the alarm subsystem, the alarm type includes three modules, namely: The alarm strategy processing module is responsible for implementing the data processing logic of the alarm strategy. Different alarm strategies have different processing logic; The cluster resource usage alarm module is responsible for executing the alarm strategy logic related to the cluster information, classifying the strategy processing results, and then selecting different channels to send the alarm information according to the severity of the alarm; The computing task running status alarm module is responsible for executing the alarm strategy related to the computing task. The priority of various status detection is defined and the order detection is performed according to the priority, and the task in the abnormal state is sent alarm information processing; at the same time, for Some low-level exceptions define relative solutions, and the solutions will be executed at the same time as the alarm information is sent. The monitoring method of a cluster monitoring system oriented to multiple public cloud computing platforms according to any one of claims 1 to 5, characterized in that it comprises the following steps: The data collection subsystem is distributed and runs on the computing cluster of the cloud provider. The specific steps include: (1) Through a unified interface layer function, the collection of cluster node information, cluster resource information, and computing task status information is performed synchronously; (2) The above steps (1) When a round of collection is completed, the cluster state definition function will load the monitoring data in the memory to analyze and determine the current state of the cluster; (3) Extract the latest monitoring data and upload it to the data processing subsystem through RPC request; (4) Streamline and process monitoring data according to the requirements of the dispatch system, and store the latest results for the dispatch system to call; The data processing subsystem receives the monitoring data information and performs corresponding processing. The specific steps are as follows: (1) Receive monitoring data pushed by each cluster; (2) Analyze the monitoring information according to different dimensions, and persist the processed data to the database; (3) According to the requirements of the billing system, process the billing system in a recognizable way and store it in the designated time series database; (4) Pull the real-time task data of the scheduling system, analyze the monitoring data information of each cluster in a unified manner, filter the transferable tasks and the corresponding number for task transfer; The alarm subsystem processes the monitored data information according to the alarm strategy logic and the specific steps are as follows: (1) Execute alarm strategy logic related to cluster information according to monitoring data information, distinguish abnormal levels and execute alarm actions; (2) Execute alarm strategies related to computing tasks based on monitoring data information, distinguish abnormal levels and perform alarm actions; for low-level abnormalities with preset solutions, execute the corresponding solutions for the abnormalities. The monitoring method for a cluster monitoring system oriented to multiple public cloud computing platforms according to claim 6, characterized in that the step (1) of the data collection subsystem specifically includes the following steps: (1.1) Cluster node information collection module. First, the unified interface layer of this module completes the access to the cloud provider, and then uses the unified interface layer to obtain cluster node information, that is, the actual number of nodes, the planned number of nodes, and the maximum number of supported nodes; and finally Temporarily save in memory; (1.2) Cluster computing resource information collection module. First of all, the unified interface layer of this module completes the packaging of cluster resource collection plug-ins; secondly, confirm the plug-ins supported by the cluster; finally obtain the total number of CPUs and the number of CPUs requested by the task through the unified interface layer , The number of CPU actually used by the task; it is finally stored in the memory temporarily; (1.3) The computing task state information collection module first obtains all task information of the current cluster through the native interface layer of the cluster computing engine; secondly, it uses a state classifier to classify all states and extract key information; finally, it is temporarily stored in the memory.

Images