CN111723112B - Data task execution method, device, electronic device and storage medium - Google Patents

Abstract

The embodiment of the invention provides a data task execution method, a device, electronic equipment and a storage medium, wherein the method comprises the following steps: determining the data volume of a target data query task and the resource consumption cost of the target data query task under each preset calculation engine; and determining a target computing engine for executing the target data query task according to the data volume and the resource consumption cost, and executing the target data query task according to the target computing engine. According to the data task execution method, the device, the electronic equipment and the storage medium, the data query task is executed by selecting the matched computing engine from the computing engines through the data volume of the target data query task and the resource consumption cost of the target data query task under the preset computing engines, so that the influence of the unmatched computing engines on the execution of the data query task is avoided, and the execution efficiency and the stability of the data query task are improved.

Description

Data task execution method, device, electronic device and storage medium

technical field

The present invention relates to the technical field of data processing, in particular to a data task execution method, device, electronic equipment and storage medium.

Background technique

The task processing forms of big data are mainly divided into batch task processing and real-time task processing. Batch task processing is also called Hive offline task processing due to processing time issues. Real-time task processing technology realizes faster response time for data processing. There are different implementation methods for real-time task processing. Real-time task processing also includes Storm streaming computing and Spart real-time interactive computing.

Whether it is batch task processing or real-time task processing, it can be called a computing engine. Different computing engines have different application scenarios. Offline task processing is suitable for processing data with a large amount of data and rarely updated data, but the disadvantages are also obvious, such as slow response time, poor interactivity, and single programming method. Although real-time task processing has a fast response time, most real-time computing engines often have relatively high hardware requirements due to their dependence on memory, and their tolerance for data volume is not high enough.

Existing data platforms all support hybrid computing engines for batch task processing and real-time processing. For example, real-time and offline computing engines such as Hive, Spark, Storm, and Kylin can be supported on the same data platform. Each computing engine has specific Application scenarios. Therefore, before the data query task is executed, a specific computing engine will be manually specified. At this time, a computing engine will be selected for the task, and the data computing engine cannot be dynamically selected. As a result, an unsuitable computing engine may affect the running of the data query task and reduce the data rate. Execution efficiency and stability of query tasks.

Contents of the invention

Aiming at the problems existing in the prior art, embodiments of the present invention provide a data task execution method, device, electronic equipment, and storage medium.

In a first aspect, an embodiment of the present invention provides a data task execution method, including:

Determine the data volume of the target data query task, and the resource consumption cost of the target data query task under each preset computing engine;

A target computing engine for executing the target data query task is determined according to the data amount and the resource consumption cost, and the target data query task is executed according to the target computing engine.

Further, the target data query task is obtained by parsing the target query statement, and is composed of multiple subtasks. Correspondingly, determining the resource calculation cost of the target data query task under each preset computing engine includes:

Determine the amount of IO read and write data, CPU resources and memory resources required by each subtask in each computing engine in the target data query task, and the currently available IO bandwidth, CPU resources and memory resources of the data platform;

According to the amount of IO read and write data, CPU resources, and memory resources required by each subtask under each computing engine, as well as the currently available IO bandwidth, CPU resources, and memory resources of the data platform, determine the target data query task in each computing engine The resource consumption cost under .

Further, the target data query is determined according to the amount of IO read and write data, CPU resources, and memory resources required by each subtask under each computing engine, and the currently available IO bandwidth, CPU resources, and memory resources of the data platform. Resource consumption costs of tasks under each computing engine, including:

Determine the sum of the ratio of the IO read and write data volume of each subtask under each computing engine to the current available IO bandwidth of the data platform, the sum of the ratio of CPU resources to the currently available CPU resources of the data platform, and the ratio of memory resources to the current available data platform The sum of the ratios of memory resources;

The resource consumption cost of the target data query task under each computing engine is determined according to each sum.

Further, the determining the target computing engine for executing the target data query task according to the data amount and the resource computing cost includes:

When it is determined that the amount of data exceeds the first threshold, it is determined that the Hive computing engine is the target computing engine;

When it is determined that the amount of data does not exceed the first threshold, the computing engine corresponding to the minimum resource computing cost of each computing engine is determined as the target computing engine.

Further, before determining the target computing engine for executing the target data query task according to the data amount and the resource consumption cost, the method further includes:

Obtain an execution cache table corresponding to the target data query task;

Execute the target data query task according to the calculation engine prestored in the execution cache table.

Further, the execution cache table includes the calculation engine execution table and data cache valid time information, and accordingly, executing the target data query task according to the calculation engine pre-stored in the execution cache table includes:

After determining that the target data query task is cached according to the effective time information of the data cache, the pre-stored computing engine is parsed from the computing engine execution table to execute the target data query task.

In a second aspect, an embodiment of the present invention provides a data task execution device, including:

A determining module, configured to determine the data volume of the target data query task, and the resource consumption cost of the target data query task under each preset computing engine;

An execution module, configured to determine a target computing engine for executing the target data query task according to the data amount and the resource consumption cost, and execute the target data query task according to the target computing engine.

In a third aspect, an embodiment of the present invention provides a data task execution system, which is characterized in that it includes:

A task parser for determining a target data query task;

A task compiler, configured to determine the data volume of the target data query task, and the resource consumption cost of the target data query task under each preset computing engine;

A computing engine selector, configured to determine a target computing engine for executing the target data query task according to the amount of data and the resource consumption cost, and execute the target data query task according to the target computing engine;

The metadata repository is used to store data required for executing the target data query task.

In a fourth aspect, an embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and operable on the processor. When the processor executes the program, the above-mentioned data task execution is realized. method steps.

In a fifth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above data task execution method are implemented.

The data task execution method, device, electronic equipment, and storage medium provided by the embodiments of the present invention, through the data volume of the target data query task and the resource consumption cost of the target data query task under the preset calculation engines, from each calculation engine Select a matching computing engine to execute data query tasks, avoid unmatched computing engines from affecting the execution of data query tasks, and improve the execution efficiency and stability of data query tasks.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

FIG. 1 is a flow chart of an embodiment of a data task execution method of the present invention;

FIG. 2 is a schematic diagram of a complete flow of the data task execution method of the present invention;

3 is a structural diagram of an embodiment of the data task execution device of the present invention;

4 is a structural diagram of an embodiment of the data task execution system of the present invention;

FIG. 5 is a structural diagram of an embodiment of the electronic device of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

Figure 1 shows a data task execution method provided by an embodiment of the present invention, including:

S11. Determine the data volume of the target data query task, and the resource consumption cost of the target data query task under each preset computing engine;

S12. Determine a target computing engine for executing the target data query task according to the data volume and resource consumption cost, and execute the target data query task according to the target computing engine.

With regard to step S11 and step S12, it should be noted that, in the embodiment of the present invention, the target data query task is formed by parsing and converting the data query sentence input through the system interface. The data query statement includes SQL statement, SPARQL statement, DQL statement, etc. Therefore, the data query task obtained by parsing and converting these statements may be SQL task, SPARQL task, DQL task, etc.

The method of this embodiment is explained below with the SQL task, specifically as follows:

Parse the SQL statement and convert it into a task tree form. There are multiple tree nodes in the task tree, and each tree node corresponds to a subtask. In the process of parsing the SQL statement, some primary analysis work including syntax checking is performed, and after the syntax analysis is passed, it is determined that the target SQL task is qualified.

Each subtask in the SQL task needs to process the underlying data when it is executed, and the underlying data can be calculated to obtain the data volume, so the target SQL task is calculated to obtain the corresponding data volume.

In the embodiment of the present invention, the underlying data corresponding to the SQL task can be stored in the metadata repository, which also includes the name of the data table related to the SQL task, the columns and partitions of the table and its attributes, table attributes, the directory where the data of the table is located, and so on.

In the embodiment of the present invention, the calculation engine includes Hive (offline), Spark (real-time), Storm (flow), Kylin calculation, etc. For different computing engines, the resource consumption cost of the target SQL task under each computing engine is different. Therefore, the resource consumption cost of the target SQL task is calculated to obtain the resource consumption cost of each calculation engine.

Next, according to the amount of data obtained above and the resource consumption cost of each computing engine, the target computing engine for executing the target SQL task can be determined through the preset computing engine selection strategy, and the target SQL task can be executed according to the target computing engine.

In this regard, it should be noted that the calculation engine selection strategy is based on the data volume and the resource consumption cost under each calculation engine as filtering conditions, and one calculation engine is determined from each calculation engine as the target calculation engine for executing the target SQL task.

According to a data task execution method provided by an embodiment of the present invention, a matching computing engine is selected from each computing engine based on the data volume of the target data query task and the resource consumption cost of the target data query task under each preset computing engine. Execute data query tasks, avoid unmatched computing engines from affecting the execution of data query tasks, and improve the execution efficiency and stability of data query tasks.

In a further embodiment of the method in the above embodiment, it is mainly an explanation of the resource consumption cost of determining the target SQL task under each preset computing engine, specifically as follows:

S111. Determine the amount of IO read and write data, CPU resources, and memory resources required by each subtask in the target SQL task under each computing engine, as well as the currently available IO bandwidth, CPU resources, and memory resources of the data platform;

S112. According to the amount of IO read and write data, CPU resources and memory resources required by each subtask under each computing engine, and the currently available IO bandwidth, CPU resources and memory resources of the data platform, determine the target SQL task under each computing engine resource consumption cost.

Regarding step S111 and step S112, it should be noted that, in this embodiment of the present invention, since it is mentioned in the above embodiment that the target SQL task includes multiple subtasks. Analysis of each subtask can obtain the corresponding IO read and write data volume, CPU resources and memory resources, as well as the current available IO bandwidth, CPU resources and memory resources of the data platform. Here, the data platform is the computing environment required to perform data query tasks.

Determine the sum of the ratio of the IO read and write data volume of each subtask under each computing engine to the current available IO bandwidth of the data platform, the sum of the ratio of CPU resources to the currently available CPU resources of the data platform, and the ratio of memory resources to the current available data platform The sum of the ratios of the memory resources, and determine the resource consumption cost of the target data query task under each computing engine according to each sum.

In a further embodiment of the method in the above embodiment, a resource consumption cost acquisition formula is used to determine the resource consumption cost of the target SQL task under each computing engine.

Wherein, the resource consumption cost acquisition formula includes:

C _{compute-engine} is resource consumption cost;

IO _i is the amount of IO read and write data required by the computing engine for the i-th subtask in the target SQL task, and IO _usable is the currently available IO bandwidth of the data platform;

CPU _i is the CPU resource required by the computing engine for the i-th subtask in the target SQL task, and CPU _useable is the CPU resource currently available on the data platform;

Mem _i is the memory resource required by the computing engine for the i-th subtask in the target SQL task, and Mem _{usable is} the memory resource currently available on the data platform.

It should be noted that the amount of IO read and write data, CPU resources, and memory resources required by each subtask under different computing engines may be different. For this reason, the resource consumption cost of the target SQL task under each computing engine is different.

In the further embodiment of the method in the above embodiment, the process of determining the target computing engine for executing the target SQL task according to the amount of data, resource computing cost and preset computing engine selection strategy is mainly explained. Here, because the target SQL The data volume of the task and the resource consumption cost of the target SQL task under the computing engine are both numerical values. For this reason, the calculation engine selection strategy is mainly used to judge the value, so as to determine the matching computing engine as the target computing engine of the target SQL task. details as follows:

The compute engine selection strategy includes:

When it is determined that the data volume of the SQL task exceeds the first threshold, the Hive computing engine is determined as the target computing engine.

When it is determined that the data volume of the SQL task does not exceed the first threshold, the computing engine corresponding to the minimum resource computing cost of each computing engine is determined as the target computing engine.

In this regard, it should be noted that the Hive computing engine is suitable for offline computing scenarios with a large amount of data, and the task execution time is relatively long. Computing engines such as Spark and Storm are suitable for real-time computing scenarios, and their data processing time is short. However, when the computing resource demand is large and the computing resources are insufficient, the execution of the SQL task will directly fail.

Here, the first threshold is a boundary value for judging whether the SQL task is a task with a large amount of data. For example, tasks with a data volume exceeding 10T use the hive computing engine.

In a further embodiment of the method of the above-mentioned embodiment, before the target computing engine for executing the target SQL task is determined according to the data volume of the SQL task and the resource consumption cost of the SQL task under each computing engine, the calculation related to the execution of the target SQL task The explanation of the engine selection process is as follows:

Get the execution cache table corresponding to the target SQL task.

In the above embodiment, it is mentioned that the underlying data corresponding to the SQL task can be stored in the metadata repository, which also includes the name of the data table related to the SQL task, the columns and partitions of the table, and its attributes, table attributes, the directory where the data of the table is located, and so on. In addition, the metadata storage repository also includes an execution cache table of the SQL task, and the execution cache table includes information of a computing engine used by a certain SQL task within the effective cache time.

After it is determined that the data information on the execution cache table is valid, the target SQL task is executed according to the calculation engine pre-stored in the execution cache table.

In a further embodiment of the method in the above embodiment, the execution cache table includes a calculation engine execution table and data cache validity time information, and the calculation engine execution table includes the calculation engine used by the SQL task within the cache validity time. The data cache valid time information includes the cache valid time of the SQL task.

For this reason, after determining that the target SQL task is cached according to the effective time information of the data cache, the pre-stored calculation engine execution target SQL task is obtained by parsing from the calculation engine execution table.

In addition, after determining that the data information in the execution cache table is invalid, then determine the data volume of the target SQL task and the resource consumption cost of the target SQL task under each preset computing engine, and execute the target SQL according to the target computing engine Task.

It should be noted that after determining the target computing engine for executing the target data query task according to the data volume and resource consumption cost, and executing the target data query task according to the target computing engine, the target computing engine needs to be stored in the execution cache table in the metadata in the data repository.

In the embodiment, since the execution cache table is stored in the metadata repository, the system can directly access the metadata repository before determining the target computing engine for executing the target SQL task according to the data volume and resource consumption cost, so as to facilitate The execution cache table quickly selects the appropriate computing engine.

Combining the contents of the above-mentioned embodiments, Fig. 2 shows a schematic diagram of the overall flow of the data task execution method provided by an embodiment of the present invention, referring to Fig. 2, the details are as follows:

The SQL task is cached in the execution cache table. If the Hive computing engine is cached in the execution cache table, the SQL task will be executed using the Hive computing engine; if the Spark computing engine is cached in the execution cache table, the Spark computing engine will be used to execute the SQL. Task.

If the SQL task is not cached on the execution cache table and the data volume exceeds the first threshold, the SQL task will be executed using the Hive computing engine.

If the amount of data does not exceed the first threshold, calculate the resource consumption cost of the SQL task under the Hive and Spark computing engines.

If the resource consumption cost under the Hive computing engine is relatively small, the Hive computing engine is used to execute the SQL task.

If the cost of resource consumption under the Spark computing engine is relatively small, the Spark computing engine is used to execute the SQL task.

Fig. 3 shows a data task execution device provided by an embodiment of the present invention, including a determination module 21 and an execution module 22, wherein:

A determining module 21, configured to determine the data volume of the target data query task, and the resource consumption cost of the target data query task under each preset computing engine;

The execution module 22 is configured to determine a target computing engine for executing the target data query task according to the data volume and resource consumption cost, and execute the target data query task according to the target computing engine.

In a further embodiment of the device in the above embodiment, the determination module is specifically used to:

Determine the amount of IO read and write data, CPU resources, and memory resources required by each subtask in each computing engine in the target data query task, as well as the currently available IO bandwidth, CPU resources, and memory resources of the data platform;

According to the amount of IO read and write data, CPU resources, and memory resources required by each subtask under each computing engine, as well as the current available IO bandwidth, CPU resources, and memory resources of the data platform, determine the target data query task under each computing engine Resource consumption costs.

In a further embodiment of the device in the above embodiment, according to the amount of IO read and write data, CPU resources and memory resources required by each subtask under each computing engine, and the currently available IO bandwidth, CPU resources and memory resources of the data platform, Determine the resource consumption cost of the target data query task under each computing engine, including:

Determine the sum of the ratio of the IO read and write data volume of each subtask under each computing engine to the current available IO bandwidth of the data platform, the sum of the ratio of CPU resources to the currently available CPU resources of the data platform, and the ratio of memory resources to the current available data platform The sum of the ratios of memory resources;

The resource consumption cost of the target data query task under each computing engine is determined according to each sum.

In a further embodiment of the device in the above embodiment, the resource consumption cost acquisition formula is used to determine the resource consumption cost of the target data query task under each computing engine, wherein the resource consumption cost acquisition formula includes:

C _{compute-engine} is resource consumption cost;

IO _i is the amount of IO read and write data required by the computing engine for the i-th subtask in the target data query task, and IO _usable is the currently available IO bandwidth of the data platform;

CPU _i is the CPU resource required by the computing engine for the i-th subtask in the target data query task, and CPU _useable is the CPU resource currently available on the data platform;

Mem _i is the memory resource required by the computing engine for the i-th subtask in the target data query task, and Mem _{usable is} the memory resource currently available on the data platform.

In a further embodiment of the device in the above embodiment, the execution module is specifically used to:

When it is determined that the amount of data exceeds the first threshold, it is determined that the Hive computing engine is the target computing engine;

When it is determined that the amount of data does not exceed the first threshold, the computing engine corresponding to the minimum resource computing cost of each computing engine is determined as the target computing engine.

In a further embodiment of the device in the above embodiment, before determining the target computing engine for executing the target data query task according to the amount of data, the resource computing cost, and the preset computing engine selection strategy, an acquisition module is also included for :

Obtain an execution cache table corresponding to the target data query task;

Execute the target data query task according to the pre-stored calculation engine in the execution cache table.

In a further embodiment of the device in the above embodiment, the execution cache table includes the calculation engine execution table and data cache valid time information, and correspondingly, the acquisition module executes the target data query task according to the calculation engine pre-stored in the execution cache table , specifically for:

After determining that the target data query task is cached according to the effective time information of the data cache, the pre-stored calculation engine is obtained by parsing from the calculation engine execution table to execute the target data query task.

Since the principle of the apparatus described in the embodiment of the present invention is the same as that of the method described in the foregoing embodiments, more detailed explanations will not be repeated here.

It should be noted that, in the embodiment of the present invention, a hardware processor (hardware processor) may be used to implement related functional modules.

The data task execution device provided by the embodiment of the present invention selects a matching computing engine from each computing engine to execute the data task based on the data volume of the target data query task and the resource consumption cost of the target data query task under the preset computing engines. Query tasks, avoid unmatched computing engines from affecting the execution of data query tasks, and improve the execution efficiency and stability of data query tasks.

Figure 4 shows a schematic structural diagram of a data task execution system provided by an embodiment of the present invention, referring to Figure 4, the system includes a task parser 31, a task compiler 32, a computing engine selector 33 and a metadata repository 34, of which:

A task parser 31, configured to determine a target data query task;

The task compiler 32 is used to determine the data volume of the target data query task, and the resource consumption cost of the target data query task under each preset computing engine;

A computing engine selector 33, configured to determine a target computing engine for executing the target data query task according to the amount of data and the resource consumption cost, and execute the target data query task according to the target computing engine;

The metadata repository 34 is configured to store data required for executing the target data query task.

Since the principle of the system described in this embodiment of the present invention is the same as that of the method described in the foregoing embodiments, more detailed explanations will not be repeated here.

It should be noted that, in the embodiment of the present invention, relevant functional units may be implemented by a hardware processor (hardware processor).

The data task execution system provided by the embodiment of the present invention selects a matching computing engine from each computing engine to execute the data according to the data volume of the target data query task and the resource consumption cost of the target data query task under each preset computing engine. Query tasks, avoid unmatched computing engines from affecting the execution of data query tasks, and improve the execution efficiency and stability of data query tasks.

FIG. 5 illustrates a schematic diagram of the physical structure of an electronic device. As shown in FIG. 5, the electronic device may include: a processor (processor) 41, a communication interface (Communications Interface) 42, a memory (memory) 43 and a communication bus 44, Wherein, the processor 41 , the communication interface 42 , and the memory 43 communicate with each other through the communication bus 44 . The processor 41 can call the logical instructions in the memory 43 to perform the following method: determine the data volume of the target data query task, and the resource consumption cost of the target data query task under each preset calculation engine; calculate according to the data volume and resource The cost determines the target computing engine for executing the target data query task, and executes the target data query task according to the target computing engine.

In addition, the above-mentioned logic instructions in the memory 43 can be implemented in the form of software function units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes. .

An embodiment of the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method provided by the above-mentioned embodiments is implemented, for example, including: determining the target data query The data volume of the task, and the resource consumption cost of the target data query task under the preset computing engines; determine the target computing engine for executing the target data query task according to the data volume and resource computing cost, and execute the target data query according to the target computing engine Task.

Through the above description of the implementations, those skilled in the art can clearly understand that each implementation can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic discs, optical discs, etc., including several instructions to make a computer device (which may be a personal computer, server, or network device, etc.) execute the methods described in various embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims (8)

1. A data task execution method, characterized in that, comprising: Determine the data volume of the target data query task, and the resource consumption cost of the target data query task under each preset computing engine; determining a target computing engine for executing the target data query task according to the amount of data and the resource consumption cost, and executing the target data query task according to the target computing engine; The target data query task is obtained by parsing the target query statement and consists of multiple subtasks. Correspondingly, determine the resource consumption cost of the target SQL task under each preset computing engine, including: Determine the amount of IO read and write data, CPU resources, and memory resources required by each subtask in each computing engine in the target data query task, as well as the currently available IO bandwidth, CPU resources, and memory resources of the data platform; According to the amount of IO read and write data, CPU resources, and memory resources required by each subtask under each computing engine, as well as the current available IO bandwidth, CPU resources, and memory resources of the data platform, determine the target data query task under each computing engine resource consumption costs; Determine the target computing engine to execute the target data query task according to the amount of data and resource computing cost, including: When it is determined that the amount of data exceeds the first threshold, it is determined that the Hive computing engine is the target computing engine; When it is determined that the amount of data does not exceed the first threshold, determine the computing engine corresponding to the minimum resource consumption cost under each computing engine as the target computing engine; The first threshold is a boundary value for judging whether the SQL task is a task with a large amount of data. 2. The data task execution method according to claim 1, characterized in that, according to the required IO read and write data volume, CPU resources and memory resources under each computing engine according to each subtask, and the currently available data platform IO bandwidth, CPU resources and memory resources, determine the resource consumption cost of the target data query task under each computing engine, including: Determine the sum of the ratio of the IO read and write data volume of each subtask under each computing engine to the current available IO bandwidth of the data platform, the sum of the ratio of CPU resources to the currently available CPU resources of the data platform, and the ratio of memory resources to the current available data platform The sum of the ratios of memory resources; The resource consumption cost of the target data query task under each computing engine is determined according to each sum. 3. The data task execution method according to claim 1, further comprising: Obtain an execution cache table corresponding to the target data query task; Execute the target data query task according to the calculation engine prestored in the execution cache table. 4. The data task execution method according to claim 3, wherein the execution cache table includes a calculation engine execution table and data cache valid time information, and correspondingly, executes the Target data query tasks, including: After determining that the target data query task is cached according to the data cache valid time information, the pre-stored computing engine is obtained by parsing from the computing engine execution table, and the target data query task is executed according to the pre-stored computing engine. 5. A data task execution device, characterized in that, comprising: A determining module, configured to determine the data volume of the target data query task, and the resource consumption cost of the target data query task under each preset computing engine; An execution module, configured to determine a target computing engine for executing the target data query task according to the amount of data and the resource consumption cost, and execute the target data query task according to the target computing engine; In the process of determining the resource consumption cost of the target data query task under each preset calculation engine, the determining module is specifically used for: Determine the amount of IO read and write data, CPU resources, and memory resources required by each subtask in each computing engine in the target data query task, as well as the currently available IO bandwidth, CPU resources, and memory resources of the data platform; According to the amount of IO read and write data, CPU resources, and memory resources required by each subtask under each computing engine, as well as the current available IO bandwidth, CPU resources, and memory resources of the data platform, determine the target data query task under each computing engine resource consumption costs; In the process of determining the target computing engine that executes the target data query task according to the data volume and resource consumption cost, the execution module is specifically used for: When it is determined that the amount of data exceeds the first threshold, it is determined that the Hive computing engine is the target computing engine; When it is determined that the amount of data does not exceed the first threshold, determine the computing engine corresponding to the minimum resource computing cost under each computing engine as the target computing engine; The first threshold is a boundary value for judging whether the SQL task is a task with a large amount of data. 6. A data task execution system, comprising: A task parser for determining a target data query task; A task compiler, configured to determine the data volume of the target data query task, and the resource consumption cost of the target data query task under each preset computing engine; A computing engine selector, configured to determine a target computing engine for executing the target data query task according to the amount of data and the resource consumption cost, and execute the target data query task according to the target computing engine; a metadata repository, configured to store data required for executing the target data query task; In the process of determining the resource consumption cost of the target data query task under each preset computing engine, the task compiler is specifically used for: Determine the amount of IO read and write data, CPU resources, and memory resources required by each subtask in each computing engine in the target data query task, as well as the currently available IO bandwidth, CPU resources, and memory resources of the data platform; According to the amount of IO read and write data, CPU resources, and memory resources required by each subtask under each computing engine, as well as the current available IO bandwidth, CPU resources, and memory resources of the data platform, determine the target data query task under each computing engine resource consumption costs; In the process of determining the target computing engine that executes the target data query task according to the data volume and resource consumption cost, the computing engine selector is specifically used for: When it is determined that the amount of data exceeds the first threshold, it is determined that the Hive computing engine is the target computing engine; When it is determined that the amount of data does not exceed the first threshold, determine the computing engine corresponding to the minimum resource computing cost under each computing engine as the target computing engine; The first threshold is a boundary value for judging whether the SQL task is a task with a large amount of data. 7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein the processor implements any of claims 1 to 4 when executing the program. The steps of the data task execution method described in the item. 8. A non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the data task execution method according to any one of claims 1 to 4 is implemented A step of.

Images