CN111756828B - Data storage method, device and equipment - Google Patents
Data storage method, device and equipment Download PDFInfo
- Publication number
- CN111756828B CN111756828B CN202010567658.6A CN202010567658A CN111756828B CN 111756828 B CN111756828 B CN 111756828B CN 202010567658 A CN202010567658 A CN 202010567658A CN 111756828 B CN111756828 B CN 111756828B
- Authority
- CN
- China
- Prior art keywords
- disk
- data object
- node
- resource domain
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data storage method, wherein in a disk resource domain strategy, disk resources of nodes are divided into more than two disk resource domains, and the corresponding relation between objects and the disk resource domains is set. Therefore, when the disk mapping is performed, the target disk resource domain corresponding to the object is determined first, then the hash algorithm is utilized to determine which disk in the target disk resource domain the object is specifically mapped to, and finally the mapping relation between the object and the disk is obtained. Therefore, the method avoids the problem that the object is randomly mapped to any disk in the node by setting the disk resource domain policy, and realizes the destination mapping of the object so that the object can only be mapped to the corresponding disk resource domain. The flexibility of resource allocation is improved, and the storage performance of the distributed storage system is fully exerted. In addition, the application also provides a data storage device, equipment and a readable storage medium, and the technical effects of the data storage device and the equipment correspond to those of the method.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data storage method, apparatus, device, and readable storage medium.
Background
The shaepdog is an emerging distributed storage system of an open source community, adopts a completely symmetrical structure, does not have a central node similar to metadata service, and provides storage service as a whole outside by means of interconnection of a large number of common PC servers through a network.
Unlike other distributed storage designs, the shaepdog has no metadata information, i.e. node position information stored by the object is not recorded, and in the data storage process, the shaepdog calculates the mapping relation from the object to the storage position through a hash algorithm.
When calculating the mapping relation between the object and the disk, the disadvantage of calculating the storage position of the object through the hash algorithm is that: according to the hash algorithm, the objects are randomly distributed on any disk in the node, so that the objects cannot be organized according to a certain rule to be mapped purposefully. For example, assuming that there are 4 disks per node, the object may be randomly distributed on any one of the 4 disks according to the hash algorithm, and the storage range of the object cannot be limited to only disk 1 and disk 2 of the node.
Therefore, the current distributed storage system determines the mapping relation of the objects to the disks through a hash algorithm, the objects are randomly distributed on any disk of the node, and the resource distribution mode is excessively dead, so that the storage performance of the distributed storage system is affected.
Disclosure of Invention
The purpose of the application is to provide a data storage method, a device, equipment and a readable storage medium, which are used for solving the problems that a current distributed storage system determines the mapping relation of an object to a disk through a hash algorithm, a resource allocation mode is excessively dead, and the storage performance of the distributed storage system is affected. The specific scheme is as follows:
in a first aspect, the present application provides a data storage method, including:
determining a data object to be stored;
determining a target node to which the data object is mapped, and acquiring a disk resource domain policy of the target node, wherein the disk resource of the target node is divided into more than two disk resource domains, and the disk resource domain policy comprises a corresponding relation between the data object and the disk resource domain and also comprises a corresponding relation between a disk and the disk resource domain;
determining the mapping relation between the data object and the disk by utilizing a consistent hash algorithm on a target disk resource domain corresponding to the data object;
and storing the data object according to the mapping relation between the data object and the disk.
Preferably, the determining, by using a consistent hash algorithm, a mapping relationship between the data object and the disk in the target disk resource domain corresponding to the data object includes:
constructing a hash ring according to the target disk resource domain corresponding to the data object;
calculating a hash value of the name of the data object by using a consistent hash algorithm;
determining the position of the data object in the hash ring according to the size of the hash value;
and determining the mapping relation between the data object and the disk according to the position of the data object in the hash ring.
Preferably, before the obtaining the disk resource domain policy of the target node, the method further includes:
and setting a disk resource domain policy of the target node, and dividing the high-performance disk and the low-performance disk into different disk resource domains.
Preferably, the storing the data object according to the mapping relationship between the data object and the disk includes:
and determining storage position information of the data object according to the mapping relation between the data object and the disk, and storing the data object according to the storage position information, wherein the storage position information comprises a disk resource domain number, a disk number and a virtual node number.
Preferably, the determining the target node to which the data object is mapped includes:
acquiring a node resource domain policy of a current cluster, wherein node resources of the current cluster are divided into more than two node resource domains, and the node resource domain policy comprises a corresponding relation between a data object and the node resource domains and also comprises a corresponding relation between a node and the node resource domains;
and determining the mapping relation between the data object and the node by utilizing a consistent hash algorithm on a target node resource domain corresponding to the data object to obtain a target node mapped by the data object.
Preferably, before the acquiring the node resource domain policy of the current cluster, the method further includes:
setting node resource domain strategies of the current cluster, and dividing nodes in different fault domains into the same node resource domain.
In a second aspect, the present application provides a data storage device comprising:
an object determination module: for determining a data object to be stored;
the strategy acquisition module is used for: the method comprises the steps of determining a target node to which the data object is mapped, and obtaining a disk resource domain policy of the target node, wherein the disk resource of the target node is divided into more than two disk resource domains, and the disk resource domain policy comprises a corresponding relation between the data object and the disk resource domain and also comprises a corresponding relation between a disk and the disk resource domain;
the mapping relation determining module: the mapping relation between the data object and the disk is determined by utilizing a consistent hash algorithm on a target disk resource domain corresponding to the data object;
and a storage module: and the data object is stored according to the mapping relation between the data object and the disk.
In a third aspect, the present application provides a data storage device comprising:
a memory: for storing a computer program;
a processor: for executing the computer program to implement the steps of the data storage method as described above.
In a fourth aspect, the present application provides a readable storage medium having stored thereon a computer program for implementing the steps of the data storage method as described above when executed by a processor.
The data storage method provided by the application comprises the following steps: determining a data object to be stored; determining a target node to which the data object is mapped, and acquiring a disk resource domain policy of the target node, wherein the disk resource of the target node is divided into more than two disk resource domains, and the disk resource domain policy comprises a corresponding relation between the data object and the disk resource domains and also comprises a corresponding relation between the disk and the disk resource domains; on a target disk resource domain corresponding to the data object, determining the mapping relation between the data object and the disk by utilizing a consistent hash algorithm; and storing the data object according to the mapping relation between the data object and the disk.
Therefore, the method divides the disk resources of the node into more than two disk resource domains in the disk resource domain policy, and sets the corresponding relation between the data object and the disk resource domain in the disk resource domain policy. Therefore, when mapping between the data object and the disk is performed, the target disk resource domain corresponding to the data object is determined first, then the hash algorithm is utilized to determine which disk in the target disk resource domain the data object is specifically mapped to, and finally the mapping relation between the object and the disk is obtained. Therefore, the method avoids the problem that the data object is randomly mapped to any disk in the node by setting the disk resource domain strategy, and realizes the purpose of carrying out the destination mapping on the data object so that the data object can only be mapped to the disk in the corresponding disk resource domain. The flexibility of resource allocation is improved, and the storage performance of the distributed storage system is fully exerted.
In addition, the application further provides a data storage device, a device and a readable storage medium, and the technical effects of the data storage device and the device correspond to those of the method, and are not repeated here.
Drawings
For a clearer description of embodiments of the present application or of the prior art, the drawings that are used in the description of the embodiments or of the prior art will be briefly described, it being apparent that the drawings in the description that follow are only some embodiments of the present application, and that other drawings may be obtained from these drawings by a person of ordinary skill in the art without inventive effort.
FIG. 1 is a schematic diagram of an object distribution situation of a conventional hash ring based on a consistent hash algorithm provided in the present application;
FIG. 2 is a flowchart illustrating an implementation of an embodiment of a data storage method provided in the present application;
FIG. 3 is a schematic diagram illustrating the partitioning of disk resources provided in the present application;
FIG. 4 is a schematic diagram of a hash ring applying disk resource domain policies provided herein;
FIG. 5 is a schematic diagram of an object distribution of a hash ring applying a disk resource domain policy provided in the present application;
FIG. 6 is a detailed flowchart of S103 in a first embodiment of a data storage method provided in the present application;
FIG. 7 is a flowchart illustrating a second implementation of a data storage method according to the present application;
FIG. 8 is a functional block diagram of an embodiment of a data storage device provided herein.
Detailed Description
In order to provide a better understanding of the present application, those skilled in the art will now make further details of the present application with reference to the drawings and detailed description. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
As described above, unlike other distributed storage designs, shaepdog has no metadata information, i.e., no record of location information stored by an object. The shaepdog calculates the mapping relation between the object and the storage position through a consistent hash algorithm, and the mapping process can be defined as follows by a formula: object storage location=hash (object name). Then, the position of the hash value with uniqueness calculated according to the object name on the hash ring is searched, so that the storage position of the object is determined, and the storage position is specifically shown in fig. 1.
In fig. 1, a hash ring is schematically illustrated, where triangles represent objects and circles represent virtual nodes. In the consistent hashing algorithm, whether the virtual node corresponding to the physical node or the virtual disk corresponding to the physical disk is referred to as a virtual node in the hash ring, a disk hashing process is described below as an example.
As can be seen from fig. 1, the range of [0,2 n) forms a hash ring, assuming that 3 physical disks exist on a certain physical node, each physical disk corresponds to 4 virtual nodes according to the consistent hash rule, and the names of the virtual nodes are defined by a physical disk number and a virtual node number, that is, each virtual node corresponding to the physical disk 1 is marked as vnode1.1, vnode 1.2, vnode 1.3 and vnode 1.4, and each virtual node is randomly and uniformly distributed on different positions of the hash ring. Assuming that there are 8 objects to be stored, respectively denoted as object 1, object 2, …, and object 8, a hash value is calculated according to the object name, and then the position of the object on the hash ring is determined according to the size of the hash value. According to the consistent hashing algorithm, objects are randomly allocated on different physical disks. The final allocation result is as in fig. 1: object 1 is allocated on physical disk 1, object 5 is allocated on physical disk 3, and the allocation of other objects is not described one by one.
Because there are multiple physical nodes in the shaepdog, and there are multiple physical disks on each physical node, in practical application, two layers of hash are needed to be performed to calculate the position information of an object: the first layer of hash is a hash ring composed of all physical nodes in the cluster, and the object is calculated to be distributed on which physical node in the cluster through the hash; after determining the physical node to which the object maps, all physical disks of the physical node are formed into a hash ring, and the object is calculated on which physical disk according to the hash value of the object. That is, the first layer hash calculates node position information of the object, and the second layer hash calculates disk position information of the object; after two-layer hash computation, the position information of an object is determined.
However, the drawbacks of the two-layer hash mapping described above are: the object map cannot be organized according to certain rules. For example: how to implement the object 1 to be distributed only on the physical node1 and the physical node2, and how to implement the object 1 to be distributed only on the physical node2 and the physical node3, and further how to implement the object 1 to be distributed only on the physical disk 1 of the physical node1, the physical disk 2 of the physical node1, the physical disk 3 of the physical node2, and the physical disk 4 of the physical node2, which is a conventional two-layer hash map of a distributed storage system, cannot solve this problem.
In view of the above problems, the present application provides a data storage method, apparatus, device, and readable storage medium, which avoid the problem that data objects are randomly mapped to any disk in a node by setting a disk resource domain policy, and implement destination mapping on the data objects, so that the data objects can only be mapped to the disk in the corresponding disk resource domain. The flexibility of resource allocation is improved, and the storage performance of the distributed storage system is fully exerted.
An embodiment of a data storage method provided in the present application is described below, with reference to fig. 2, where the embodiment includes:
s201, determining a data object to be stored;
s202, determining a target node to which the data object is mapped, and acquiring a disk resource domain policy of the target node, wherein the disk resource of the target node is divided into more than two disk resource domains, and the disk resource domain policy comprises a corresponding relation between the data object and the disk resource domain and also comprises a corresponding relation between a disk and the disk resource domain;
s203, determining the mapping relation between the data object and the disk by utilizing a consistent hash algorithm on a target disk resource domain corresponding to the data object;
s204, storing the data object according to the mapping relation between the data object and the disk.
In this embodiment, a concept of a domain of resources (domain) is defined on the basis of not changing hash mapping, and a domain of disk resources is taken as an example, where a set of resources of a disk is defined. In this embodiment, the internal disk of the node is divided into more than two disk resource domains, and the disk resource domain policy is used to describe specific disk resource division conditions, that is, describe to which disk resource domain each disk is specifically divided, that is, the corresponding relationship between the disk and the disk resource domain; in addition, the disk resource domain policy is also used for describing a custom mapping rule, that is, a directional mapping policy of the definition object and the disk resource domain, that is, the corresponding relationship between the data object and the disk resource domain.
In summary, the disk resource domain is a set of disk resources, and the disk resource domain policy describes the partition condition of the disk resource domain and the object-to-disk resource domain mapping policy. In order to better illustrate the concepts of the disk resource domain and the disk resource domain policy, the following description is given by taking a specific application as an example:
let the disk resources shown in FIG. 1 be divided into two disk resource domains, labeled domain-1 and domain-2, respectively. Assuming that the division result is shown in fig. 3, virtual nodes corresponding to white circles in fig. 3, that is, vnode1.1, vnode2.2, vnode3.2, and vnode3.3 are all divided into domain-1, and virtual nodes corresponding to black circles in fig. 3 are divided into domain-2. Assume that the naming convention of the virtual node is set to: the physical disk number + the number of the disk resource domain + the number of the virtual node in the disk resource domain, then the naming result is shown in fig. 3, e.g. vnode3.3 in fig. 1 is named vnode 3.1.2 in fig. 3.
The hash rings are respectively constructed according to the two disk resource domains, and in fact, the hash rings in fig. 1 are split into two hash rings according to the division result of the disk resource domains, as shown in fig. 4, that is, virtual nodes of domain-1 form a hash ring 1, and virtual nodes of domain-2 form a hash ring 2.
The corresponding relation between the objects and the disk resource domain is set in the disk resource domain policy, and for 8 objects shown in fig. 1, as shown in fig. 3, the object 1, the object 3, the object 4 and the object 5 all correspond to domain-1, and the other objects correspond to domain-2. Then, according to the consistent hashing algorithm, each object is distributed on a corresponding hash ring according to the hash value size of the object name, as shown in fig. 5.
By comparing fig. 1 and fig. 5, it can be seen that the mapping rule of the object and the mapped virtual node position do not change at all, but only the name of the virtual node. Therefore, only the disk resource domain needs to be defined, the hash ring shown in fig. 1 can be split into more than two, and the consistent hash distribution strategy of the object on the two hash rings is not changed.
Therefore, the disk resource domain can divide the disk resources, and virtual nodes of each disk resource domain obtained by dividing form a complete hash ring. From this perspective, the disk resource domain is a collection of node resources, and different combinations of nodes can be realized by defining different disk resource domain policies.
Specifically, the step S103, namely, a process of determining, by using a consistent hash algorithm, a mapping relationship between the data object and the disk on the target disk resource domain corresponding to the data object, specifically includes the following steps, as shown in fig. 6:
s601, constructing a hash ring according to a target disk resource domain corresponding to the data object;
s602, calculating a hash value of the name of the data object by using a consistent hash algorithm;
s603, determining the position of the data object in the hash ring according to the size of the hash value;
s604, determining the mapping relation between the data object and the disk according to the position of the data object in the hash ring.
In the data storage method provided by the embodiment, the disk resources of the node are divided into more than two disk resource domains in the disk resource domain policy, and the corresponding relation between the data object and the disk resource domain is set in the disk resource domain policy. Therefore, when mapping between the data object and the disk is performed, the target disk resource domain corresponding to the data object is determined first, then the hash algorithm is utilized to determine which disk in the target disk resource domain the data object is specifically mapped to, and finally the mapping relation between the object and the disk is obtained. Therefore, the method avoids the problem that the data object is randomly mapped to any disk in the node by setting the disk resource domain strategy, and realizes the purpose of mapping the data object so that the data object can only be mapped to a specific disk, namely the disk in the disk resource domain corresponding to the data object. The flexibility of resource allocation is improved, and the storage performance of the distributed storage system is fully exerted.
The second embodiment of the data storage method provided by the application is implemented based on the first embodiment, and is expanded to a certain extent based on the first embodiment.
Specifically, the first embodiment only describes that the disk resource domain policy is adopted in the disk mapping process, and on the basis of the first embodiment, the node resource domain policy is also adopted in the node mapping process. Referring to fig. 7, the second embodiment specifically includes:
s701, determining a data object to be stored;
s702, acquiring a node resource domain policy of a current cluster, wherein the node resource of the current cluster is divided into more than two node resource domains, and the node resource domain policy comprises a corresponding relation between a data object and the node resource domain and also comprises a corresponding relation between a node and the node resource domain;
s703, determining a mapping relation between the data object and the node by utilizing a consistent hash algorithm on a target node resource domain corresponding to the data object to obtain a target node to which the data object is mapped;
s704, acquiring a disk resource domain policy of the target node, wherein the disk resource of the target node is divided into more than two disk resource domains, and the disk resource domain policy comprises a corresponding relation between a data object and the disk resource domain and also comprises a corresponding relation between a disk and the disk resource domain;
s705, determining the mapping relation between the data object and the disk by utilizing a consistent hash algorithm on a target disk resource domain corresponding to the data object;
s706, storing the data object according to the mapping relation between the data object and the disk.
In the embodiment, a node resource domain policy and a disk resource domain policy are respectively defined in two-layer hash mapping of a distributed storage system, wherein the node resource domain policy contains partition information of node resources, and the disk resource domain policy contains partition information of disk resources. And determining the node position information of the object in the node resource domain through one-layer hash mapping, and determining the disk position information of the object in the disk resource domain of the node through two-layer hash. Thus, a directed mapping policy from node to disk may be implemented by defining a node resource domain policy and a disk resource domain policy in a configuration file, similar to the rule policy in ceph.
As a preferred embodiment, different disk resource domains may be partitioned according to the characteristics of the disk. Specifically, before the obtaining the disk resource domain policy of the target node, the method further includes: setting a disk resource domain policy of a target node, dividing a high-performance disk and a low-performance disk into different disk resource domains, and storing the disk resource domain policy in a configuration file. If the high-performance storage medium is divided into a disk resource domain, and the low-performance storage medium is divided into a disk resource domain, the function of layered storage can be realized by means of the strategy.
As a preferred embodiment, before the acquiring the node resource domain policy of the current cluster, the method further includes: setting node resource domain policies of the current cluster, dividing nodes in different fault domains into the same node resource domain, and storing the node resource domain policies in a configuration file.
For example, assuming that node1, node2 and node3 are on the same rack, in order to solve the problem that node1, node2 and node3 are down at the same time due to the power failure of the rack, the data copies stored on these nodes fail at the same time, node1, node2 and node3 are typically classified into the same fault domain. In this embodiment, the node1, the node2 and the node3 are defined in different node resource domains by the custom rule, or the nodes in different fault domains are divided into the same node resource domain, and each node resource domain shares its own hash ring, so that the situation that the object copy exists on the 3 nodes at the same time is avoided, so that the definition of the node resource domain policy can realize the function of the fault domain.
It can be seen that, according to the data storage method provided in this embodiment, on the basis of not changing the two-layer hash mapping, the nodes and the disks on the nodes are subjected to resource division and integration according to a certain rule, so as to form a node resource domain and a disk resource domain; the self-defined node resource domain policy and the disk resource domain policy are allowed, and the mapping from the object to the specific node and the characteristic disk is realized according to the self-defined policies, so that the physical resources in the distributed storage system can be used more flexibly, and the functions of fault domain, layered storage and the like can be realized.
A data storage device according to an embodiment of the present application is described below, and a data storage device described below and a data storage method described above may be referred to correspondingly.
As shown in fig. 8, the data storage device of the present embodiment includes:
object determination module 801: for determining a data object to be stored;
policy acquisition module 802: the method comprises the steps of determining a target node to which the data object is mapped, and obtaining a disk resource domain policy of the target node, wherein the disk resource of the target node is divided into more than two disk resource domains, and the disk resource domain policy comprises a corresponding relation between the data object and the disk resource domain and also comprises a corresponding relation between a disk and the disk resource domain;
the mapping relation determining module 803: the mapping relation between the data object and the disk is determined by utilizing a consistent hash algorithm on a target disk resource domain corresponding to the data object;
storage module 804: and the data object is stored according to the mapping relation between the data object and the disk.
The data storage device of this embodiment is used to implement the foregoing data storage method, so the specific implementation in this device may be found in the foregoing example portions of the data storage method, for example, the object determining module 801, the policy obtaining module 802, the mapping relation determining module 803, and the storage module 804, which are respectively used to implement steps S201, S202, S203, and S204 in the foregoing data storage method. Therefore, the detailed description will be omitted herein with reference to the accompanying drawings, which illustrate examples of the respective parts.
In addition, since the data storage device of the present embodiment is used to implement the foregoing data storage method, the functions thereof correspond to those of the foregoing method, and will not be described herein.
In addition, the present application also provides a data storage device, including:
a memory: for storing a computer program;
a processor: for executing the computer program to implement the steps of the data storage method as described above.
Finally, the present application provides a readable storage medium having stored thereon a computer program for implementing the steps of the data storage method as described above when executed by a processor.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The foregoing has outlined rather broadly the more detailed description of the present application and the principles and embodiments of the present application have been presented in terms of specific examples, which are provided herein to assist in the understanding of the method and core concepts of the present application; meanwhile, as those skilled in the art will have modifications in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.
Claims (8)
1. A method of data storage, comprising:
determining a data object to be stored;
determining a target node to which the data object is mapped, and acquiring a disk resource domain policy of the target node, wherein the disk resource of the target node is divided into more than two disk resource domains, and the disk resource domain policy comprises a corresponding relation between the data object and the disk resource domain and also comprises a corresponding relation between a disk and the disk resource domain;
determining the mapping relation between the data object and the disk by utilizing a consistent hash algorithm on a target disk resource domain corresponding to the data object;
storing the data object according to the mapping relation between the data object and the disk;
the determining the target node to which the data object is mapped includes:
acquiring a node resource domain policy of a current cluster, wherein node resources of the current cluster are divided into more than two node resource domains, and the node resource domain policy comprises a corresponding relation between a data object and the node resource domains and also comprises a corresponding relation between a node and the node resource domains;
and determining the mapping relation between the data object and the node by utilizing a consistent hash algorithm on a target node resource domain corresponding to the data object to obtain a target node mapped by the data object.
2. The method of claim 1, wherein determining, on the target disk resource domain corresponding to the data object, a mapping relationship between the data object and a disk using a consistent hashing algorithm comprises:
constructing a hash ring according to the target disk resource domain corresponding to the data object;
calculating a hash value of the name of the data object by using a consistent hash algorithm;
determining the position of the data object in the hash ring according to the size of the hash value;
and determining the mapping relation between the data object and the disk according to the position of the data object in the hash ring.
3. The method of claim 2, further comprising, prior to said obtaining the disk resource domain policy of the target node:
and setting a disk resource domain policy of the target node, and dividing the high-performance disk and the low-performance disk into different disk resource domains.
4. The method of claim 3, wherein storing the data object according to the mapping relationship between the data object and the disk comprises:
and determining storage position information of the data object according to the mapping relation between the data object and the disk, and storing the data object according to the storage position information, wherein the storage position information comprises a disk resource domain number, a disk number and a virtual node number.
5. The method of claim 1, further comprising, prior to the obtaining the node resource domain policy for the current cluster:
setting node resource domain strategies of the current cluster, and dividing nodes in different fault domains into the same node resource domain.
6. A data storage device, comprising:
an object determination module: for determining a data object to be stored;
the strategy acquisition module is used for: the method comprises the steps of determining a target node to which the data object is mapped, and obtaining a disk resource domain policy of the target node, wherein the disk resource of the target node is divided into more than two disk resource domains, and the disk resource domain policy comprises a corresponding relation between the data object and the disk resource domain and also comprises a corresponding relation between a disk and the disk resource domain;
the mapping relation determining module: the mapping relation between the data object and the disk is determined by utilizing a consistent hash algorithm on a target disk resource domain corresponding to the data object;
and a storage module: the data object storage module is used for storing the data object according to the mapping relation between the data object and the disk;
the policy obtaining module is specifically configured to obtain a node resource domain policy of a current cluster, where node resources of the current cluster are divided into more than two node resource domains, and the node resource domain policy includes a correspondence between a data object and a node resource domain, and further includes a correspondence between a node and a node resource domain; and determining the mapping relation between the data object and the node by utilizing a consistent hash algorithm on a target node resource domain corresponding to the data object to obtain a target node mapped by the data object.
7. A data storage device, comprising:
a memory: for storing a computer program;
a processor: for executing said computer program to implement the steps of the data storage method of any one of claims 1-5.
8. A readable storage medium, characterized in that the readable storage medium has stored thereon a computer program for implementing the steps of the data storage method according to any of claims 1-5 when being executed by a processor.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010567658.6A CN111756828B (en) | 2020-06-19 | 2020-06-19 | Data storage method, device and equipment |
| PCT/CN2021/076920 WO2021253853A1 (en) | 2020-06-19 | 2021-02-19 | Data storage method, device and apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010567658.6A CN111756828B (en) | 2020-06-19 | 2020-06-19 | Data storage method, device and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111756828A CN111756828A (en) | 2020-10-09 |
| CN111756828B true CN111756828B (en) | 2023-07-14 |
Family
ID=72675828
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010567658.6A Active CN111756828B (en) | 2020-06-19 | 2020-06-19 | Data storage method, device and equipment |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111756828B (en) |
| WO (1) | WO2021253853A1 (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111756828B (en) * | 2020-06-19 | 2023-07-14 | 广东浪潮大数据研究有限公司 | Data storage method, device and equipment |
| CN112199176B (en) * | 2020-10-16 | 2023-01-17 | 济南浪潮数据技术有限公司 | Service processing method, device and related equipment |
| CN112230861B (en) * | 2020-10-26 | 2022-09-13 | 金钱猫科技股份有限公司 | Data storage method and terminal based on consistent hash algorithm |
| CN114879907B (en) * | 2022-04-28 | 2025-09-26 | 济南浪潮数据技术有限公司 | Method, device, equipment and storage medium for determining data distribution |
| CN114860166B (en) * | 2022-04-30 | 2025-08-19 | 济南浪潮数据技术有限公司 | Method, device and medium for localizing volume |
| CN116204137B (en) * | 2023-05-04 | 2023-08-04 | 苏州浪潮智能科技有限公司 | Distributed storage system, control method, device and equipment based on DPU |
| CN117113395B (en) * | 2023-08-17 | 2024-07-16 | 深圳众投互联信息技术有限公司 | Intelligent storage platform and method based on mobile phone number blacklist data |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103136114A (en) * | 2011-11-30 | 2013-06-05 | 华为技术有限公司 | Storage method and storage device |
| CN103929500A (en) * | 2014-05-06 | 2014-07-16 | 刘跃 | Method for data fragmentation of distributed storage system |
| CN110489059A (en) * | 2019-07-11 | 2019-11-22 | 平安科技（深圳）有限公司 | The method, apparatus and computer equipment of data cluster storage |
| WO2020083106A1 (en) * | 2018-10-25 | 2020-04-30 | 华为技术有限公司 | Node expansion method in storage system and storage system |
Family Cites Families (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8046561B1 (en) * | 2006-12-22 | 2011-10-25 | Emc Corporation | Methods and apparatus for selecting a storage zone for a content unit |
| CN102880428B (en) * | 2012-08-20 | 2015-09-09 | 华为技术有限公司 | The creation method of distributed Redundant Array of Independent Disks (RAID) and device |
| CN103645859B (en) * | 2013-11-19 | 2016-04-13 | 华中科技大学 | A kind of magnetic disk array buffer storage method of virtual SSD and SSD isomery mirror image |
| CN103905540A (en) * | 2014-03-25 | 2014-07-02 | 浪潮电子信息产业股份有限公司 | Object storage data distribution mechanism based on two-sage Hash |
| CN104102709A (en) * | 2014-07-14 | 2014-10-15 | 浪潮(北京)电子信息产业有限公司 | Disk management method and database management system |
| US9851906B2 (en) * | 2015-06-16 | 2017-12-26 | Vmware, Inc. | Virtual machine data placement in a virtualized computing environment |
| CN104965677B (en) * | 2015-06-26 | 2018-04-13 | 北京百度网讯科技有限公司 | Storage system |
| US10114716B2 (en) * | 2015-11-20 | 2018-10-30 | International Business Machines Corporation | Virtual failure domains for storage systems |
| CN106055706B (en) * | 2016-06-23 | 2019-08-06 | 杭州迪普科技股份有限公司 | A kind of cache resources storage method and device |
| CN106201355B (en) * | 2016-07-12 | 2018-12-11 | 腾讯科技（深圳）有限公司 | Data processing method and device and storage system |
| US10693728B2 (en) * | 2017-02-27 | 2020-06-23 | Dell Products L.P. | Storage isolation domains for converged infrastructure information handling systems |
| CN107832017B (en) * | 2017-11-14 | 2021-02-12 | 中国石油天然气集团有限公司 | Method and device for improving storage IO performance of seismic data |
| CN112352216B (en) * | 2018-06-30 | 2022-06-14 | 华为技术有限公司 | Data storage method and data storage device |
| CN110096227B (en) * | 2019-03-28 | 2023-04-18 | 北京奇艺世纪科技有限公司 | Data storage method, data processing device, electronic equipment and computer readable medium |
| CN110058822B (en) * | 2019-04-26 | 2022-06-24 | 北京计算机技术及应用研究所 | Transverse expansion method for disk array |
| CN110083312B (en) * | 2019-04-28 | 2021-01-15 | 联想(北京)有限公司 | Disk capacity expansion method and device and computer equipment |
| CN110347675A (en) * | 2019-06-05 | 2019-10-18 | 阿里巴巴集团控股有限公司 | A kind of date storage method and device |
| CN111756828B (en) * | 2020-06-19 | 2023-07-14 | 广东浪潮大数据研究有限公司 | Data storage method, device and equipment |
-
2020
- 2020-06-19 CN CN202010567658.6A patent/CN111756828B/en active Active
-
2021
- 2021-02-19 WO PCT/CN2021/076920 patent/WO2021253853A1/en not_active Ceased
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103136114A (en) * | 2011-11-30 | 2013-06-05 | 华为技术有限公司 | Storage method and storage device |
| CN103929500A (en) * | 2014-05-06 | 2014-07-16 | 刘跃 | Method for data fragmentation of distributed storage system |
| WO2020083106A1 (en) * | 2018-10-25 | 2020-04-30 | 华为技术有限公司 | Node expansion method in storage system and storage system |
| CN110489059A (en) * | 2019-07-11 | 2019-11-22 | 平安科技（深圳）有限公司 | The method, apparatus and computer equipment of data cluster storage |
Non-Patent Citations (2)
| Title |
|---|
| A new reliability growth model with dual-time domain — A hard disk drive perspective;Feng-Bin Sun等;2015 Annual Reliability and Maintainability Symposium (RAMS);全文 * |
| 一种双均衡的集群存储资源映射方法;章宏灿;薛巍;;清华大学学报(自然科学版)网络.预览(10);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111756828A (en) | 2020-10-09 |
| WO2021253853A1 (en) | 2021-12-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111756828B (en) | Data storage method, device and equipment | |
| CN101771715B (en) | Method, device and system for establishing distribution type network | |
| US20130263151A1 (en) | Consistent Hashing Table for Workload Distribution | |
| US8676951B2 (en) | Traffic reduction method for distributed key-value store | |
| EP2875653B1 (en) | Method for generating a dataset structure for location-based services | |
| US20080201335A1 (en) | Method and Apparatus for Storing Data in a Peer to Peer Network | |
| CN117008818A (en) | Data processing method, apparatus, computer device, and computer readable storage medium | |
| CN115878046B (en) | Data processing method, system, device, storage medium and electronic equipment | |
| CN111552694A (en) | Self-adaptive geographic space grid indexing method | |
| US9667499B2 (en) | Sparsification of pairwise cost information | |
| CN111159193A (en) | Multi-layered consistent hash ring and its application in creating distributed database | |
| CN112884123B (en) | Neural network optimization method and device, electronic equipment and readable storage medium | |
| Guerrero et al. | Optimization policy for file replica placement in fog domains | |
| US11507313B2 (en) | Datafall: a policy-driven algorithm for decentralized placement and reorganization of replicated data | |
| CN114879907B (en) | Method, device, equipment and storage medium for determining data distribution | |
| CN113934377A (en) | Metadata cluster deployment method, apparatus, device and readable storage medium | |
| Azimi et al. | Peer sampling gossip-based distributed clustering algorithm for unstructured P2P networks | |
| US20200036784A1 (en) | Peer-to-peer network for internet of things resource allocation operation | |
| CN117671392B (en) | Grid division method, device, equipment and medium for homeland space data | |
| Lai et al. | A scalable multi-attribute hybrid overlay for range queries on the cloud | |
| CN108304555A (en) | Distributed maps data processing method | |
| US7840598B2 (en) | Method for distributing resources to network nodes in a decentralized data network | |
| Daskos et al. | PePeR: A distributed range addressing space for peer-to-peer systems | |
| Confais et al. | A tree-based approach to locate object replicas in a fog storage infrastructure | |
| CN111884940B (en) | Interest matching method, apparatus, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |