CN118550934B - Trino resource group management method and Trino resource group management device - Google Patents
Trino resource group management method and Trino resource group management device Download PDFInfo
- Publication number
- CN118550934B CN118550934B CN202411025606.0A CN202411025606A CN118550934B CN 118550934 B CN118550934 B CN 118550934B CN 202411025606 A CN202411025606 A CN 202411025606A CN 118550934 B CN118550934 B CN 118550934B
- Authority
- CN
- China
- Prior art keywords
- resource
- resource group
- maximum
- group
- resources
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a Trino resource group management method and a Trino resource group management device, which belong to the technical field of data processing, and the method comprises the following steps: constructing a metadata table corresponding to the resource group, and setting the maximum usable resource and the resource growth proportion of the resource group; receiving a latest query request submitted by a client, and counting the first resource usage total amount of all queries currently running on a resource group; determining the maximum resource which can be applied by the resource group to the resource manager of the resource group so as to allocate capacity expansion resources for the resource group; the total amount of second resource usage for all queries currently running on the resource group is counted to determine whether to execute the latest query request on the resource group. The resource group in the application can automatically expand the capacity of the resource when the resource is lacking, and can contract the capacity of the resource when the resource is idle, so that the resource of the resource group has elastic property, and the requirement of dynamically adjusting the resource limit of the resource group in real time according to the service condition of the resource can be effectively met.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a Trino resource group management method and apparatus.
Background
In the current big data and cloud computing age, a distributed SQL query engine is used as a key infrastructure and is important for realizing efficient and extensible data analysis. Trino (original name PrestoSQL), which is a leading open source distributed SQL query engine, takes an important role in the field of interactive analysis query with excellent performance, flexibility and wide support for multiple data sources, and one of the core advantages is an innovative Resource management mechanism, namely, efficient management and Resource isolation of query operation through Resource groups (Resource groups).
The resource group is used as a core abstract unit in Trino architecture, and aims to ensure that different user groups or query types can fairly and reasonably share cluster resources through predefined resource limitation and fine scheduling strategies. However, there is a significant challenge in the prior art resource group configuration: the resource limitation parameters thereof typically need to be preset prior to the start of Trino service processes, which can only be adjusted during service operation by manually modifying the database configuration file.
The static and manual configuration mode is used for a system administrator unfamiliar with Trino internal working principles and resource management mechanisms, so that the operation difficulty and the maintenance cost are increased undoubtedly, and meanwhile, the problems of uneven resource allocation, reduced inquiry performance, system breakdown and the like can be caused by improper resource configuration, so that the user experience and the service continuity are seriously affected. Therefore, how to provide a more intelligent and automatic resource group configuration and adjustment scheme becomes the focus of common attention of Trino communities and users.
Disclosure of Invention
The invention aims to provide a Trino resource group management method and a Trino resource group management device, which are used for solving the problem that the resource limitation in Trino resource groups is difficult to dynamically adjust in the prior art.
In order to achieve the above purpose, the present application adopts the following technical scheme:
The application discloses a Trino resource group management method, which comprises the following steps:
Constructing a metadata table corresponding to a resource group, and setting the maximum usable resource and the resource growth proportion of the resource group in the metadata table;
Receiving a latest query request submitted by a client, and counting the first resource usage total of all queries currently running on the resource group before executing the latest query request;
Determining the maximum resource which can be applied to a resource group resource manager by the resource group according to the maximum available resource, the resource increasing proportion and the total first resource use amount, so that the resource group resource manager allocates capacity-expanding resources for the resource group according to the maximum applicable resource;
And counting the second resource use total amount of all queries currently running on the resource group, and determining whether to execute the latest query request on the resource group according to the second resource use total amount and the resource limit value of the resource group after capacity expansion.
Preferably, the metadata table is further provided with a weight of the resource group and a maximum idle time of the resource.
Preferably, the method further comprises:
when any inquiry operation on the resource group is completed, counting the latest resource usage total amount of all inquiries currently operated on the resource group, comparing the latest resource usage total amount with the initial resource amount of the resource group, and if the time that the latest resource usage total amount is smaller than the initial resource amount exceeds the maximum idle time of the resources, recovering the capacity-expanding resources allocated to the resource group.
Preferably, the determining the maximum resource applicable to the resource manager of the resource group according to the maximum available resource, the resource increment ratio and the total amount of the first resource use includes:
And comparing the total first resource use amount with an initial resource limit value of the resource group, and calculating the product of the resource increment proportion and the initial resource limit value when the total first resource use amount is larger than the initial resource limit value and the initial resource limit value is smaller than the maximum usable resource to obtain the maximum resource which can be applied to a resource manager of the resource group by the resource group.
Preferably, before the resource group resource manager allocates capacity expansion resources to the resource group according to the maximum applicable resources, the method further comprises:
And sending a resource application request to a resource group resource manager, wherein the resource application request comprises the maximum resource applicable by the resource group.
Preferably, the allocating capacity expansion resources for the resource group according to the applicable maximum resources includes:
Sequencing all the resource groups applied to the capacity expansion resources according to the sequence from small weight to large weight, and sequentially recovering the capacity expansion resources in the resource groups with idle resources according to the sequence, wherein the capacity expansion resources do not exceed the maximum applicable resources of the corresponding resource groups;
After recovery, sorting all the resource groups applying for the capacity expansion resources according to the sequence from big to small of the weight, and judging whether the residual resources of the resource manager of the resource groups are smaller than the maximum resources applicable to the resource group with the maximum weight;
when the residual resources are not smaller than the maximum resources applicable to the resource group with the maximum weight, allocating capacity expansion resources for the resource group with the maximum weight;
And when the residual resources are smaller than the maximum resources applicable to the resource group with the maximum weight, the residual resources are all distributed to the resource group with the maximum weight, and the difference value between the residual resources and the maximum resources applicable to the resource group with the maximum weight is recorded to wait for the next resource distribution.
Preferably, the determining whether to execute the latest query request on the resource group according to the second resource usage total amount and the resource limit value after the capacity expansion of the resource group includes:
comparing the total amount of the second resource usage with the resource limit value of the resource group after capacity expansion;
and executing the latest query request on the resource group when the total amount of the second resource usage is smaller than the resource limit value after the capacity expansion of the resource group.
A Trino resource group management device, comprising:
the configuration module is used for constructing a metadata table corresponding to the resource group, and setting the maximum usable resource and the resource growth proportion of the resource group in the metadata table;
the statistics module is used for receiving the latest query request submitted by the client and counting the first resource usage total amount of all queries currently running on the resource group before executing the latest query request;
the computing module is used for determining the maximum resource which can be applied to the resource group resource manager by the resource group according to the maximum available resource, the resource increasing proportion and the total first resource use amount, so that the resource group resource manager allocates capacity-expanding resources for the resource group according to the maximum applicable resource;
And the judging module is used for counting the second resource use total amount of all queries currently running on the resource group and determining whether to execute the latest query request on the resource group according to the second resource use total amount and the resource limit value after the capacity expansion of the resource group.
An electronic device comprising a memory and a processor, the memory to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a Trino resource group management method as in any one of the above.
A computer readable storage medium storing a computer program which when executed by a computer implements a Trino resource group management method as claimed in any one of the preceding claims.
The invention has the following beneficial effects:
In the invention, the resource group can automatically expand the capacity of the resource when the resource is lacking, and can contract the capacity of the resource when the resource is idle, so that the resource of the resource group has elastic characteristics, and the requirement of dynamically adjusting the resource limitation of the resource group in real time according to the service condition of the resource can be effectively met.
Drawings
In order to more clearly illustrate the embodiments of the application or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the application, and that other drawings can be obtained according to these drawings without inventive faculty for a person skilled in the art.
FIG. 1 is a first flowchart of a Trino resource group management method according to an embodiment of the present invention;
FIG. 2 is a second flowchart of a Trino resource group management method provided by an embodiment of the present invention;
Fig. 3 is a schematic structural diagram of a Trino resource group management device according to an embodiment of the present invention;
fig. 4 is a schematic diagram of an electronic device for implementing a Trino resource group management method according to an embodiment of the present application.
Detailed Description
In order to make the technical scheme of the application clearer, the application is further described in detail below with reference to the attached drawings and specific embodiments. The terms "first," "second," and the like in the claims and the description of the application, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order, and it is to be understood that the terms so used may be interchanged, if appropriate, merely to describe the manner in which objects of the same nature are distinguished in the embodiments of the application by the description, and furthermore, the terms "comprise" and "have" and any variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
As shown in fig. 1, a Trino resource group management method includes the following steps:
S110, constructing a metadata table corresponding to the resource group, and setting the maximum usable resource and the resource growth proportion of the resource group in the metadata table;
S120, receiving a latest query request submitted by a client, and counting the first resource usage total amount of all queries currently running on the resource group before executing the latest query request;
s130, determining the maximum resource which can be applied to a resource group resource manager by the resource group according to the maximum available resource, the resource increasing proportion and the total first resource use amount, so that the resource group resource manager allocates capacity-expanding resources for the resource group according to the maximum applicable resource;
S140, counting second resource usage total amount of all queries currently running on the resource group, and determining whether to execute the latest query request on the resource group according to the second resource usage total amount and the resource limit value after the capacity expansion of the resource group.
In this embodiment, the construction of the metadata table corresponding to the resource group is performed in a database, where the Trino connectable data storage systems include multiple types, such as a relational database MySQL, a NoSQL database, and a hadop HDFS, where the selected connection database is a MySQL database, specifically, in the MySQL database, a max_memory_ LIMIT VARCHAR (128) NULL field for representing the maximum available resource of the resource group is added by a ALTER TABLE resource _groups add_memory_ LIMIT VARCHAR (128) NULL field for the metadata table, a resource_size_limit field for representing the maximum available resource of the resource group is added by a ALTER TABLE resource _ groups ADD resource _ WEIGHT INT NULL field for representing the resource_size of the resource group, the weight can be used for resource reclamation, and also can be used for resource allocation, and a max_size limit field for representing the resource growth proportion of the resource group when the resource group is applied for expansion is added by a ALTER TABLE resource _ groups ADD growth _ fraction DOUBLE NULL table is added, the maximum available resource is not added by a ALTER TABLE resource _ groups ADD resource _ WEIGHT INT NULL field for representing the resource group weight, and the maximum available resource is considered to be used for the resource of the maximum idle time (the maximum available resource is not to be used for the resource), and the maximum available resource is set for the maximum available resource, and the idle time is not limited by a value of the maximum available resource, and the maximum available resource is set for the resource is considered to be used for the maximum idle time.
Then Trino receives the latest query request a from the client, which may be the command line interface Trino CLI, may be a JDBC driver, or may be a Python client, where before executing the latest query request a, trino calculates the total first resource usage amount for all the resources currently running on the resource group, and determines the maximum resource that the resource group can apply to the resource group resource manager ResourceGroupResourceManager by combining the maximum available resource of the resource group, the resource growth ratio and the total first resource usage amount, and sends a resource application request to the resource group resource manager, where the resource application request includes the maximum resource that the resource group can apply to, and the resource group resource manager allocates an expanded resource for the resource according to the maximum resource that the resource group can apply to, where the maximum resource that the resource group can apply to refers to the maximum expected resource amount declared when the resource group applies to expand, and the expanded resource refers to the additional resource that the resource group can actually allocate to the resource group, and the maximum resource that the resource group can be allocated to the resource group and the maximum resource that the resource group can not apply to the maximum capacity.
Then, counting the amount of the resources used by all the queries currently running on the resource group again to obtain a second total amount of the use of the resources, because the resource request is sent from the resource group to the resource manager of the resource group, which is not instant, but a period of time is needed, during which the amount of the resources used by all the queries currently running on the resource group will change correspondingly, and determining whether to execute the latest query request A on the resource group according to the total amount of the use of the second resource and the limited value of the resources after the expansion of the resource group, the limited value of the resources refers to the total amount of the resources currently available for the resource group, and because the resource group can apply for automatic expansion of the resources, if not limiting the resource request, one resource group can apply for the whole Trino resources.
Further, the determining whether to execute the latest query request on the resource group according to the second resource usage total amount and the resource limit value after the capacity expansion of the resource group includes:
comparing the total amount of the second resource usage with the resource limit value of the resource group after capacity expansion;
and executing the latest query request on the resource group when the total amount of the second resource usage is smaller than the resource limit value after the capacity expansion of the resource group.
The second resource usage total amount is compared with the resource limit value after the capacity expansion of the resource group, if the second resource usage total amount is smaller than the resource limit value after the capacity expansion of the resource group, the condition that the resource group has unused resources is indicated, the latest query request A is executed on the resource group, and because the resource group only runs new queries when the residual usable resources exist, and meanwhile, as long as the unused resources exist in the resource group, the new queries can be run without considering the size of the unused resources, and if the unused resources do not exist in the resource group, the new queries are not run.
Further, when any query operation on the resource group is completed, counting the latest resource usage total amount of all queries currently operated on the resource group, comparing the latest resource usage total amount with the initial resource amount of the resource group, and if the time that the latest resource usage total amount is smaller than the initial resource amount exceeds the maximum idle time of the resources, recovering the capacity-expanding resources allocated to the resource group.
When one query is run on the resource group, statistics is carried out on all the resources currently running on the resource group to obtain the latest resource usage total amount, if the latest resource usage total amount is smaller than the initial resource amount of the resource group, the situation that the expanded resource group is idle is indicated, at this moment, the resource group starts timing, the current time is recorded as T 1, the resource group has an initial resource amount, the initial resource amount is fixed and can not be recycled by the resource group manager, after that, the latest resource usage total amount is continuously calculated, if the latest resource usage total amount obtained by calculation is smaller than the initial resource amount of the resource group in a period of time, the current time is recorded as T 2, if T 2-T1 > max_idle_limit, the resource group is marked as idle, the resource group is waited for recycling of the expanded resource, and in the period of time, if the latest resource usage total amount calculated in the period of time is not smaller than the initial resource amount, the latest resource usage total amount is recorded as the latest resource T 1, and the latest resource is recorded as the latest resource until the latest resource usage total amount is smaller than the initial resource amount of the initial resource is recorded as T 1.
In this embodiment, the resource group can automatically perform the capacity expansion when the resource is absent, and can perform the capacity reduction when the resource is idle, so that the resource of the resource group has the elastic characteristic, and the requirement of dynamically adjusting the resource limitation of the resource group in real time according to the use condition of the resource can be satisfied.
Example 2
As shown in fig. 2, a Trino resource group management method includes:
S210, constructing a metadata table corresponding to a resource group, and setting the maximum usable resource and the resource growth proportion of the resource group in the metadata table;
s220, receiving a latest query request submitted by a client, and counting the first resource usage total amount of all queries currently running on the resource group before executing the latest query request;
S230, comparing the first resource usage total amount with an initial resource limit value of the resource group;
S240, when the total amount of the first resource is larger than the initial resource limit value and the initial resource limit value is smaller than the maximum usable resource, calculating the product of the resource increase proportion and the initial resource limit value to obtain the maximum resource which can be applied by the resource group to the resource group resource manager;
S250, sending a resource application request to the resource group resource manager, wherein the resource application request comprises the maximum resource applicable to the resource group, so that the resource group resource manager allocates capacity expansion resources for the resource group according to the maximum applicable resource;
s260, counting second resource usage total amount of all queries currently running on the resource group, and determining whether to execute the latest query request on the resource group according to the second resource usage total amount and the resource limit value after the capacity expansion of the resource group.
In embodiment 2, after counting the total amount of first resource usage of all queries currently running on the resource group, the total amount of first resource usage is compared with an initial resource limit value of the resource group, if the total amount of first resource usage is greater than the initial resource limit value and the initial resource limit value is less than the maximum available resource of the resource group, the product of the resource growth ratio of the resource group and the initial resource limit value thereof is calculated, thereby obtaining the maximum resource that the resource group can apply to the resource group resource manager, and then a resource application request is sent to the resource group resource manager.
Further, the allocating capacity expansion resources for the resource group according to the applicable maximum resources includes:
Sequencing all the resource groups applied to the capacity expansion resources according to the sequence from small weight to large weight, and sequentially recovering the capacity expansion resources in the resource groups with idle resources according to the sequence, wherein the capacity expansion resources do not exceed the maximum applicable resources of the corresponding resource groups;
After recovery, sorting all the resource groups applying for the capacity expansion resources according to the sequence from big to small of the weight, and judging whether the residual resources of the resource manager of the resource groups are smaller than the maximum resources applicable to the resource group with the maximum weight;
when the residual resources are not smaller than the maximum resources applicable to the resource group with the maximum weight, allocating capacity expansion resources for the resource group with the maximum weight;
And when the residual resources are smaller than the maximum resources applicable to the resource group with the maximum weight, the residual resources are all distributed to the resource group with the maximum weight, and the difference value between the residual resources and the maximum resources applicable to the resource group with the maximum weight is recorded to wait for the next resource distribution.
After the resource group resource manager receives the resource application request, firstly, arranging all the resource groups for which the capacity expansion resources are applied in Trino according to the order from small to large, and sequentially recovering the capacity expansion resources of the resource groups with the idle resources according to the order, namely, recovering the capacity expansion resources of the resource groups with the minimum weight and the idle resources, then recovering the capacity expansion resources of the resource groups with the second minimum weight and the idle resources, and so on, until the capacity expansion resources in all the resource groups with the idle resources are recovered, then arranging all the resource groups for which the capacity expansion resources are being applied according to the order from large to small, and allocating the resources with the priority of the residual resources of the resource group resource manager as the resource groups with the large weight, if the residual resources are not smaller than the maximum resources applicable by the resource groups with the maximum weight, directly allocating the maximum resources applicable by the resource groups with the maximum weight, if the residual resources are still remaining, performing capacity expansion resource allocation for the resource groups with the second maximum weight, and allocating the residual resources with the maximum weight to the maximum resources applicable by the resource groups, and recording the maximum resources which can be applied by the maximum resources.
When the resource group receives the capacity-expanding resource allocated by the resource group resource manager, the second total resource usage amount is obtained by counting the resource amounts used by all the queries currently running on the resource group again, and the remaining steps are the same as those in embodiment 1, so that no description is repeated.
In the embodiment, the resource manager of the resource group is used for recovering and distributing the resources of the resource group, so that the resource limit of the resource group can be dynamically adjusted in real time, and the resource utilization rate is improved.
Example 3
As shown in fig. 3, a Trino resource group management device includes:
the configuration module is used for constructing a metadata table corresponding to the resource group, and setting the maximum usable resource and the resource growth proportion of the resource group in the metadata table;
the statistics module is used for receiving the latest query request submitted by the client and counting the first resource usage total amount of all queries currently running on the resource group before executing the latest query request;
the computing module is used for determining the maximum resource which can be applied to the resource group resource manager by the resource group according to the maximum available resource, the resource increasing proportion and the total first resource use amount, so that the resource group resource manager allocates capacity-expanding resources for the resource group according to the maximum applicable resource;
And the judging module is used for counting the second resource use total amount of all queries currently running on the resource group and determining whether to execute the latest query request on the resource group according to the second resource use total amount and the resource limit value after the capacity expansion of the resource group.
The device can execute the method provided by all the embodiments of the present invention, and has corresponding functional modules and beneficial effects for executing the method, and technical details not described in detail in the present embodiment can be seen in the method provided by all the embodiments of the present invention.
Example 4
As shown in fig. 4, a computer device is provided, including a memory, a processor and a computer program stored on the memory, the processor executing the computer program to implement the steps of one of the Trino resource group management methods described above.
The computer device may be a server or a terminal. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a Trino resource group management method.
The present embodiment also provides a computer readable storage medium having stored thereon a computer program or instructions which when executed by a processor performs the steps of a Trino resource group management method as described above.
The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network interface card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device.
Computer program instructions for carrying out operations of the present invention may be assembly instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as SMALLTALK, C ++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present invention are implemented by personalizing electronic circuitry, such as programmable logic circuitry, field Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), with state information for computer readable program instructions, which can execute the computer readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium having the instructions stored therein includes an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The foregoing examples illustrate only a few embodiments of the invention and are described in detail herein without thereby limiting the scope of the invention. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the invention, which are all within the scope of the invention. Accordingly, the scope of protection of the present invention is to be determined by the appended claims.
Claims (9)
1. A Trino resource group management method, comprising the steps of:
Constructing a metadata table corresponding to a resource group, and setting the maximum usable resource and the resource growth proportion of the resource group in the metadata table;
Receiving a latest query request submitted by a client, and counting the first resource usage total of all queries currently running on the resource group before executing the latest query request;
Determining the maximum resource which can be applied to a resource group resource manager by the resource group according to the maximum available resource, the resource increasing proportion and the total first resource use amount, so that the resource group resource manager allocates capacity-expanding resources for the resource group according to the maximum applicable resource;
counting second resource usage total amount of all queries currently running on the resource group, and determining whether to execute the latest query request on the resource group according to the second resource usage total amount and the resource limit value of the resource group after capacity expansion;
wherein the determining the maximum resource applicable to the resource manager of the resource group according to the maximum available resource, the resource increasing proportion and the total first resource usage amount includes:
And comparing the total first resource use amount with an initial resource limit value of the resource group, and calculating the product of the resource increment proportion and the initial resource limit value when the total first resource use amount is larger than the initial resource limit value and the initial resource limit value is smaller than the maximum usable resource to obtain the maximum resource which can be applied to a resource manager of the resource group by the resource group.
2. The method of claim 1, wherein the metadata table is further provided with weights of the resource groups and maximum idle time of the resources.
3. The method for managing a set of Trino resources according to claim 2, wherein said method further comprises:
when any inquiry operation on the resource group is completed, counting the latest resource usage total amount of all inquiries currently operated on the resource group, comparing the latest resource usage total amount with the initial resource amount of the resource group, and if the time that the latest resource usage total amount is smaller than the initial resource amount exceeds the maximum idle time of the resources, recovering the capacity-expanding resources allocated to the resource group.
4. The method of claim 1, wherein before the resource manager allocates capacity-expanding resources to the resource group according to the applicable maximum resources, the method further comprises:
And sending a resource application request to a resource group resource manager, wherein the resource application request comprises the maximum resource applicable by the resource group.
5. The method of claim 4, wherein the allocating capacity-expanding resources for the resource group according to the applicable maximum resources comprises:
Sequencing all the resource groups applied to the capacity expansion resources according to the sequence from small weight to large weight, and sequentially recovering the capacity expansion resources in the resource groups with idle resources according to the sequence, wherein the capacity expansion resources do not exceed the maximum applicable resources of the corresponding resource groups;
After recovery, sorting all the resource groups applying for the capacity expansion resources according to the sequence from big to small of the weight, and judging whether the residual resources of the resource manager of the resource groups are smaller than the maximum resources applicable to the resource group with the maximum weight;
when the residual resources are not smaller than the maximum resources applicable to the resource group with the maximum weight, allocating capacity expansion resources for the resource group with the maximum weight;
And when the residual resources are smaller than the maximum resources applicable to the resource group with the maximum weight, the residual resources are all distributed to the resource group with the maximum weight, and the difference value between the residual resources and the maximum resources applicable to the resource group with the maximum weight is recorded to wait for the next resource distribution.
6. A method of managing a resource group according to claim 3, wherein said determining whether to execute said latest query request on said resource group based on said second total amount of resource usage and said resource limit value after said expansion of said resource group comprises:
comparing the total amount of the second resource usage with the resource limit value of the resource group after capacity expansion;
and executing the latest query request on the resource group when the total amount of the second resource usage is smaller than the resource limit value after the capacity expansion of the resource group.
7. A Trino resource group management apparatus, comprising:
the configuration module is used for constructing a metadata table corresponding to the resource group, and setting the maximum usable resource and the resource growth proportion of the resource group in the metadata table;
the statistics module is used for receiving the latest query request submitted by the client and counting the first resource usage total amount of all queries currently running on the resource group before executing the latest query request;
the computing module is used for determining the maximum resource which can be applied to the resource group resource manager by the resource group according to the maximum available resource, the resource increasing proportion and the total first resource use amount, so that the resource group resource manager allocates capacity-expanding resources for the resource group according to the maximum applicable resource;
The judging module is used for counting second resource usage total amount of all queries currently running on the resource group and determining whether to execute the latest query request on the resource group according to the second resource usage total amount and the resource limit value after the capacity expansion of the resource group;
the determining the maximum resource applicable to the resource manager of the resource group according to the maximum available resource, the resource increment proportion and the total first resource usage amount comprises the following steps:
And comparing the total first resource use amount with an initial resource limit value of the resource group, and calculating the product of the resource increment proportion and the initial resource limit value when the total first resource use amount is larger than the initial resource limit value and the initial resource limit value is smaller than the maximum usable resource to obtain the maximum resource which can be applied to a resource manager of the resource group by the resource group.
8. An electronic device comprising a memory and a processor, the memory configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a Trino resource group management method as in any one of claims 1-6.
9. A computer-readable storage medium storing a computer program, wherein the computer program causes a computer to execute a Trino resource group management method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411025606.0A CN118550934B (en) | 2024-07-30 | 2024-07-30 | Trino resource group management method and Trino resource group management device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411025606.0A CN118550934B (en) | 2024-07-30 | 2024-07-30 | Trino resource group management method and Trino resource group management device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118550934A CN118550934A (en) | 2024-08-27 |
| CN118550934B true CN118550934B (en) | 2024-10-15 |
Family
ID=92448526
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202411025606.0A Active CN118550934B (en) | 2024-07-30 | 2024-07-30 | Trino resource group management method and Trino resource group management device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118550934B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109710397A (en) * | 2017-10-26 | 2019-05-03 | 阿里巴巴集团控股有限公司 | Data processing method, device, storage medium, processor and system |
| CN112711506A (en) * | 2019-10-24 | 2021-04-27 | 阿里巴巴集团控股有限公司 | Method and device for adjusting application instances in resource group, storage medium and processor |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8577885B2 (en) * | 2010-12-09 | 2013-11-05 | International Business Machines Corporation | Partitioning management of system resources across multiple users |
| WO2018014351A1 (en) * | 2016-07-22 | 2018-01-25 | 华为技术有限公司 | Method and apparatus for resource configuration |
| US10541855B2 (en) * | 2017-02-03 | 2020-01-21 | Ribbon Communications Operating Company, Inc. | Resource allocation methods and apparatus |
| CN116860427A (en) * | 2022-03-28 | 2023-10-10 | 华为云计算技术有限公司 | Resource recovery methods and devices |
| CN115328662A (en) * | 2022-09-13 | 2022-11-11 | 国网智能电网研究院有限公司 | A process thread resource management control method and system |
| CN116048794A (en) * | 2022-12-30 | 2023-05-02 | 北京奥星贝斯科技有限公司 | Method and device for executing query task, storage medium and electronic equipment |
-
2024
- 2024-07-30 CN CN202411025606.0A patent/CN118550934B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109710397A (en) * | 2017-10-26 | 2019-05-03 | 阿里巴巴集团控股有限公司 | Data processing method, device, storage medium, processor and system |
| CN112711506A (en) * | 2019-10-24 | 2021-04-27 | 阿里巴巴集团控股有限公司 | Method and device for adjusting application instances in resource group, storage medium and processor |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118550934A (en) | 2024-08-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20240111586A1 (en) | Multi-policy intelligent scheduling method and apparatus oriented to heterogeneous computing power | |
| US11113158B2 (en) | Rolling back kubernetes applications | |
| CN108337109B (en) | Resource allocation method and device and resource allocation system | |
| US10430332B2 (en) | System and method for performance tuning of garbage collection algorithms | |
| US11119813B1 (en) | Mapreduce implementation using an on-demand network code execution system | |
| US11347684B2 (en) | Rolling back KUBERNETES applications including custom resources | |
| CN101645022A (en) | Work scheduling management system and method for a plurality of colonies | |
| CN103019651A (en) | Parallel processing method and device for complex tasks | |
| WO2022141727A1 (en) | Resource deployment system and method based on cloud cost | |
| US12038879B2 (en) | Read and write access to data replicas stored in multiple data centers | |
| US11144359B1 (en) | Managing sandbox reuse in an on-demand code execution system | |
| WO2013185175A1 (en) | Predictive analytics for resource provisioning in hybrid cloud | |
| CN114598706B (en) | Storage system elastic expansion method based on Serverless function | |
| CN119201476B (en) | Model reasoning optimization method, device, equipment, storage medium and program product | |
| CN115221186A (en) | Data query method, system and device and electronic equipment | |
| CN109542841A (en) | The method and terminal device of data snapshot are created in cluster | |
| CN116627661A (en) | Method and system for scheduling computing power resources | |
| CN112000674A (en) | A method and system for monitoring and processing abnormal data of gas station based on big data | |
| CN112286688A (en) | Memory management and use method, device, equipment and medium | |
| CN113934525B (en) | Hadoop cluster task scheduling method based on positive and negative feedback load scheduling algorithm | |
| US11138215B2 (en) | Method and system for implementing parallel database queries | |
| CN110798492B (en) | Data storage method and device and data processing system | |
| CN118550934B (en) | Trino resource group management method and Trino resource group management device | |
| CN113037851B (en) | Method for cloud mobile phone system super-resolution based on storage implementation | |
| CN115562841A (en) | Cloud video service self-adaptive resource scheduling system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |