KR20100073154A - Method for data processing and asymmetric clustered distributed file system using the same - Google Patents

Abstract

The present invention relates to a method for distributing access to data in which a centralized access is made from clients in an asymmetric cluster distributed file system, by dividing a data chunk determined as hot data in a data server and replicating the data chunk to another data server. In addition to distributing access requests for hot data from clients while minimizing the cost, it also makes it easy to reduce and expand hot data chunks while efficiently managing distributed hot data chunks.

Description

Metadata server, data server processing method and asymmetric clustered distributed file system using the same

The present invention relates to a method for distributing access to data in which intensive access is made from clients in an asymmetric cluster distributed file system.

The present invention is derived from the research conducted as part of the IT growth engine technology development project of the Ministry of Knowledge Economy and the Ministry of Information and Communication Research and Development. [Task management number: 2007-S-016-02] ].

Recently, when a video service such as UCC is provided as an Internet service, access to specific videos from a plurality of clients is frequently or temporarily congested at a moment.

Like this popular video, data showing a significant amount of access is called hot data. Failure to provide services for hot data effectively can lead to delays or suspensions of services and impact on services for other data.

In order to efficiently provide service for hot data, when the frequency of access to hot data requested from a client increases, the hot data can be efficiently distributed to multiple servers while ensuring that the service for the hot data is not delayed. How is it required?

The asymmetric cluster distributed file system is a system that stores and manages metadata of files and actual data separately. The metadata is managed in a metadata server, and the actual data is distributed and stored in multiple data servers. The metadata managed by the metadata server includes information about the data server where the actual data is stored.

These metadata servers and data servers are distributed over a network. Thus, the client separates the file's metadata from the path to access the data. In other words, to access a file, a client first accesses the file's metadata in the metadata server to obtain information about the data servers where the actual data is stored. Then input and output to the actual data is done through the data servers.

In an asymmetric cluster distributed file system, file data is divided into data chunks of fixed size and distributed and stored in data servers.

Also, in case of server or network failure, in order to solve the problem of not being able to input / output data, make a duplicate of the chunk and store it in other data servers. It is common.

By keeping replicas on multiple data servers, you can distribute the access load from users.

However, most clients accessing a video file tend to focus on the famous or characteristic parts of the video file. Therefore, access to a specific file data chunk may increase rapidly, and a problem that needs to efficiently provide a service for a specific data chunk, that is, a hot data chunk, has been raised.

Currently, asymmetric cluster distributed file system uses a method of distributing access to hot data by replicating the data back to multiple data servers when an event for hot data occurs.

However, replicating hot data to other data servers can be a burden on the data server serving the hot data. In addition, in general, in an asymmetric cluster distributed file system, the size of metadata maintained by the metadata server can be reduced while reducing the cost of accessing the metadata server each time for input and output of file data. The cost of replicating the data chunks themselves can also be a huge burden, because file data is divided and stored in much larger units than blocks in a regular file system.

Considering this, the video data that is expected to be popular is copied to several data servers in advance. However, since the prediction is not always satisfied, it may cause a waste of storage space.

Thus, what is needed is a way to quickly distribute access to hot data chunks.

In the present invention, it is possible to efficiently distribute access to hot data in an asymmetric cluster distributed file system.

In the present invention, by splitting the hot data, it is possible to effectively distribute the access to the hot data.

According to the present invention, access to hot data requested from a client can be distributed among data blocks divided into hot data.

In the present invention, by combining the data blocks that have been divided, it is possible to efficiently manage data when the demand for hot data is reduced.

According to the present invention, the number of data blocks can be reduced while combining hot data blocks that are divided when access to hot data is reduced.

In the present invention, the hot data can be replicated faster by the hot data block unit instead of the hot data as a whole.

In the present invention, the recovery of the hot data can be performed by using the hot data block.

In the present invention, the metadata for the data file; Metadata for each data chunk divided in the data file; And it provides a metadata server of the asymmetric cluster distributed file system including a data storage for storing metadata for the data chunk block divided in the data chunk.

In the metadata server of the asymmetric cluster distributed file system according to the present invention, the metadata for the data file includes a chunk identifier for identifying the data chunk, and the metadata for the data chunk identifies the data file. A file identifier; And a data chunk block identifier for identifying the data chunk block, and the metadata for the data chunk block may include a data chunk identifier for identifying the data chunk.

In the metadata server of the asymmetric cluster distributed file system according to the present invention, the data storage unit further stores metadata of a child data chunk block divided from the data chunk block, and stores the metadata for the data chunk block. The metadata further includes a child data chunk block identifier identifying the child data chunk block, and the metadata for the child data chunk block is a parent identifying the data chunk block. (Iii) may include a data chunkblock identifier.

The present invention also includes dividing and storing a data chunk of a data file into a plurality of data chunk blocks; Receiving a data access request from a client; Retrieving a cumulative number of access requests for the data chunk block that is the target of the access request; If the cumulative number of access requests is equal to or less than a maximum reference value, returning target data of the access request; If the cumulative number of access requests is greater than or equal to a maximum reference value, replicating the data chunk block to another data server; And returning an identifier for the replicated data chunk block to a client, the data processing method of the data server in the asymmetric cluster distributed file system.

In the data processing method of the data server in the asymmetric cluster distributed file system according to the present invention, after the step of replicating the data chunk block, further comprising the step of transmitting information about the data chunk block to a metadata server; It may be.

In the data processing method of the data server in the asymmetric cluster distributed file system according to the present invention, the method may further include dividing and storing the replicated data chunk block into a plurality of child data chunk blocks.

In addition, in the data processing method of the data server in the asymmetric cluster distributed file system according to the present invention, the data chunk block that is the target of the data access request is replicated to another data server before the step of retrieving the cumulative number of times of the access request. Retrieving whether there is an identifier of the data chunk block that has been replicated, and if there is an identifier of the data chunk block that has been replicated to another data server, returning the identifier of the data chunk block to the client and replicating the data chunk block to another data server. If there is an identifier of the data chunk block, the cumulative number retrieval step of the access request may be executed.

Further, in the data processing method of the data server in the asymmetric cluster distributed file system according to the present invention, when the cumulative number of data access requests is less than or equal to the maximum reference value, the data server reduces the cumulative number of data access requests. When the cumulative number of the data access requests decreases below a minimum reference value, the divided data chunk blocks may be combined into data chunks.

The present invention also includes a data server for dividing and storing data chunks of a data file into a plurality of data chunk blocks; A metadata server for storing metadata of the data file, metadata of the data chunk and metadata of the data chunk block; And an asymmetric cluster distributed file system comprising a client.

In the asymmetric cluster distributed file system according to the present invention, in response to a data access request of the client, the data server requests an access request when a cumulative number of access requests for a data chunk block targeted for the access request is equal to or less than a maximum reference value. If the cumulative number of the access request is greater than or equal to the maximum reference value, the data chunk block is replicated to another data server, an identifier for the replicated data chunk block is returned to a client, and the metadata is returned. The server receives the replication information of the data chunk block from the data server or from the data server that has replicated the data chunk block from the data server, and receives metadata information about the replicated data chunk block and the data chunk and data. Chunks It may generate an identifier between.

Further, in the asymmetric cluster distributed file system according to the present invention, when the cumulative number of access requests is less than or equal to the maximum reference value, the data server reduces the cumulative number of access requests, and the cumulative number of access requests is less than or equal to the minimum reference value. In the case of decreasing to, the divided data chunk block may be combined into a data chunk.

In addition, in the asymmetric cluster distributed file system according to the present invention, the data server that has replicated the data chunk block from the data server divides the replicated data chunk block into several data chunk blocks and stores the same data chunk block. It may be replicated to the data server, and the replication information of the data chunk block may be transmitted to the metadata server.

According to the present invention, by dividing and replicating data chunks that are intensively requested from clients of an asymmetric cluster distributed file system, it is possible to effectively distribute access from clients while minimizing the load and cost for data replication.

In addition, according to the present invention, not only the hot data but also the entire hot data block is made, so that not only data replication but also distribution of access load can be performed more quickly.

In addition, according to the present invention, data can be efficiently managed when the demand for hot data is reduced by combining the divided data blocks.

The asymmetric cluster distributed file system according to the present invention is a system in which a metadata server for storing and managing metadata and a data server for storing and managing file data are connected to a network, and the metadata server stores file data. It selects a data server to store file data as well as maintain information about the data servers it is performing and handle metadata access requests from clients.

In addition, the data servers of the asymmetric cluster distributed file system according to the present invention divide and store file data into file data chunks of a predetermined size, process a data access request from a client, and hotly request an intensive access request from a client. When data occurs, it distributes hot data and distributes it to other data servers to distribute access requests from clients.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<Summary of Asymmetric Cluster Distributed File System>

In the asymmetric cluster distributed file system according to the present invention, when a client receives metadata about a file to be accessed from a metadata server and requests file data from the data server, the data server receives the target data by an access request from the client. It is determined whether the access request is a certain amount of hot data or hot data exceeding a certain criterion. If it is determined as hot data, the access request is divided into data blocks, replicated to another data server, and the client's access request is copied. Distributed to data servers with blocks. Then, when accessor requests for hot data are reduced, the blocks of hot data are combined to improve the efficiency of data storage management.

1 schematically illustrates the configuration of an asymmetric cluster distributed file system according to the present invention.

As shown in FIG. 1, an asymmetric cluster distributed file system includes a client 101, a metadata server 102, and a data server 103.

The client 101 accesses file metadata from the metadata server 102 and inputs and outputs file data from the data server 103 while executing a client application.

The metadata server 101 stores and manages metadata about all files in the file system and manages state information for all data servers 103.

In the asymmetric cluster distributed file system, file data is divided into chunk units of a predetermined size and distributed to the data servers 103. The metadata server 101 selects a data server on which the file data chunk is to be generated, and also selects data servers to store the replica. The file data chunks are distinguished by unique identifiers, and the identifier information for the generated file data chunks is maintained in the metadata server 101 as metadata information of the file.

The data server 103 stores and manages file data chunks. The data server 103 periodically reports its status information to the metadata server 102.

The client 101, metadata server 102, and data server 103 are interconnected over a network.

<Data Server-Split Replication of Data>

FIG. 2 schematically illustrates partitioning and distributing hot data into hot data blocks in a data server of an asymmetric cluster distributed file system according to the present invention.

File data chunks are stored in duplicates on multiple data servers for purposes of data loss due to server or network failures, or for load balancing of data servers.

Typically, data chunks are managed to maintain three or more copies, including replicas. That is, data chunks are stored in three separate data servers 201, 202, and 203.

When the data server 201 receives an access request for file data from the client, the data server 201 determines whether the data corresponds to hot data.

More specifically, referring to the drawings, in the present invention, the data server 201 divides and stores the file data chunk 211 into data chunk blocks 211a, 211b, and 211c. Herein, the data chunk block refers to a unit in which data of the data chunk is further divided, and is distinguished from a data block called storage space.

If there is an access request, the data server 201 calculates the number of access requests in units of data chunk blocks. When the number of accumulated access requests for the data chunk blocks exceeds the preset reference cumulative number, the corresponding data block is determined as a hot data chunk block and replicated to another data server.

For example, when the cumulative number of access requests for the data chunk block 211a exceeds a predetermined reference cumulative number, only the chunk block 211a is replicated to another data server 204. As a result, as the number of data servers having corresponding data chunk blocks increases, the load of access requests to the data server 201 may be distributed.

In this case, the data server 204 in which the data chunk block 211a is duplicated may divide the duplicated data chunk block 213 into data chunk blocks 213a, 213b, and 213c.

In addition, in general, when the number of replicas held in the case of duplication is set in advance, data duplication is continuously performed to other data servers accordingly. For example, if all three replicas are preset, the data server 204 receiving the data chunk blocks is replicated in turn to another data server (data server-61 to data server-62 to data server-63). This allows three copies of the hot data chunk block to be maintained.

The procedure for detecting and duplicating hot data chunk blocks is also performed at other data servers 202 and 203.

In the case of replication, either the data server having the original data block, the data server receiving the replication, or both data servers transmit the information to the metadata server.

Requests to access the replicated hot data chunk blocks may continue to occur, again becoming hot data. In this case, the data server may distribute the access request by splitting and replicating the corresponding data chunk block again. Alternatively, the data server may divide and store the replicated data chunk block in several blocks.

For example, the data server 205 stores the replicated data chunk block 213 in three data blocks 213a, 213b, and 213c, and accumulates the number of access requests for each new child data chunk block. You can do it. As a result, if the cumulative number of access requests for the specific child data chunk block 213a again exceeds the reference cumulative number, the data chunk block 213a is copied to another data server 208.

As described above, when the divided data chunk blocks are divided into data chunk blocks and distributed replicated / stored, information for identifying the parent data chunk block and the child data chunk blocks is provided to the metadata server. Send together.

Of course, once a distributed distributed replication of a hot data chunk block from the data server 201 to another data server 204 is made, if the access request for the data chunk block continues to be above a certain criterion, the data server 201 The data chunk block may be replicated to another data server 205 again.

If a request for access to a hot data chunk block does not exceed the threshold, it means that the demand for that data chunk block is decreasing, so that data chunk blocks that have been replicated to multiple data servers (205, 207, etc.) The storage space may be reclaimed by gradually reducing the hot data chunk blocks by combining to the data server 209 or the like.

Metadata Servers and Data Servers-Information Management on Data

3 schematically illustrates metadata stored / managed in a metadata server of an asymmetric cluster distributed file system according to the present invention.

Referring to FIG. 3, metadata 301 for a data file holds not only various metadata information for the file, but also identifier information for identifying data chunks of file data.

Specifically, the metadata 302 for the file data chunks maintains an identifier for identifying the duplicate data chunks, a data server address that stores the chunks, and information about the data chunk blocks.

The metadata 303 for the data chunk block includes an identifier for the data chunk from which the data chunk block is divided, an identifier of the replica data chunk block, a data server address storing the data chunk block, and a child data chunk block. Maintain their identifiers.

The metadata 304 for the lowest data chunk block holds the identifier of the parent data chunk block, the identifier of the replica data chunk block, the data server address that stores the data chunk block, and so forth.

These metadata are connected to each other in a hat relationship, that is, a partition relationship.

In addition, the data chunk blocks are not only always divided and stored, but also may be stored in combination. In this case, the identifier of each data chunk block in the metadata 302 for file data chunks combines two or more data chunk blocks to indicate metadata 305 for one data chunk block that is storing data. do.

In addition, in order to visit the data server with the child data chunk block 304, the metadata 302 of the data chunk is retrieved from the metadata 301 of the data file via an identifier in the data chunk, and the data After searching the metadata 303 of the chunk block again through the identifier for the data chunk block in the metadata 302 of the chunk, the data 304 is not found again. In the metadata 301 of the file, all the information may be managed immediately.

4 schematically illustrates a data structure stored in a data server of an asymmetric cluster distributed file system according to the present invention for managing information about file data chunks and chunk blocks.

As shown in FIG. 4, the information 401 for the data chunks stored in the data server includes a data chunk identifier, an identifier of the data chunk block, and the accumulated number of access requests for the data chunk block. The metadata 402 for the data chunk block and the metadata 403 for the child data chunk block also include the identifier for the chunk block, the number of access requests, and the like, similarly to the metadata for the chunk block.

Through this, the data server may determine whether hot data is generated, and retrieve the partition relationship between the data chunk and the data chunk block by accessing the metadata of the metadata server.

Hot data generated by client's access request

FIG. 5 is a flowchart illustrating processing when hot data occurs when a client attempts to access data in an asymmetric cluster distributed file system according to the present invention.

The client first determines whether metadata for the data chunk or data chunk block to be accessed is already cached (step S501).

If it is cached, a request is made to read the corresponding data to the data server through this metadata (step S504).

If the metadata is not cached, the metadata for the data chunk or data chunk block to be accessed from the metadata server is requested (step S502).

The received metadata is reflected in the cache (S503).

The client determines a data server having a data chunk to read through the metadata and makes a read request for the data chunk to the data server where the data chunk is stored (step S504).

The client receives data of the corresponding data chunk from the data server making a request to read data, and if the data chunk is a hot data chunk, receives the identifier for the data chunk block that is replicated to another data server (step S505). ).

When the identifier of the data chunk block is received from the data server (YES in step S506), the client requests the metadata of the data chunk block again from the metadata server through the identifier of the received data chunk block (step S502). ).

If the data chunk block identifier is not received from the data server (NO in step S506), since the corresponding data is received, the response to the data read request is terminated.

Data processing on the data server

6 is a flowchart schematically illustrating a process of detecting a hot data chunk and duplicating a hot data chunk block in a data server of an asymmetric cluster distributed file system according to the present invention.

The client makes a read request to the data server for the data chunk to be accessed (step S601).

In contrast, the data server increases the accumulated number of access requests for the data chunk block corresponding to the read request from the client (step S602).

The data server determines whether the cumulative number of times of this access request exceeds the preset reference cumulative number (step S603).

If the cumulative number of access requests for the data chunk block does not exceed the reference cumulative number (NO in step S603), the data server decreases the cumulative number of access requests for the data chunk block (step S606). The requested data is returned to the client (step S607).

If the cumulative number of access requests for the data chunk block exceeds the reference cumulative number (YES in step S603), since the data chunk block corresponds to hot data, the data block of another data server to which the data chunk block is to be replicated. Request the allocation to the metadata server (step S606).

When the data block is allocated, the data server replicates the data chunk block in which the cumulative number of access requests exceeds the reference cumulative number in the allocated data block (step S607).

When the replication of the data chunk block is completed, either or both of the replicated data server or the replicated data server transmits the information about the data chunk to the metadata server (step S608). At this time, the replicated data server replicates to another data server by a predetermined number so that a predetermined number of copies can be maintained in the asymmetric cluster distributed file system.

The data server distributes the access request by generating and returning an identifier for identifying the chunk block that has been replicated to the client (step S609).

In contrast, the data server may first check whether an identifier has already been generated for the target data chunk block of the read request (step S602). That is, if an identifier for the data chunk block already exists, the corresponding data is determined to be a hot data chunk, and means that the data chunk block related to the data chunk is replicated and distributed to another data server. In addition, since the load of access requests to the data server is greatly increased, in order to distribute the load, an identifier for the replicated data chunk block is transmitted to the client.

Although a specific implementation method of the present invention has been described above with reference to the drawings, it is intended to be easily understood by those skilled in the art and is not intended to limit the technical scope of the present invention. The above description with reference to the drawings may be sufficiently modified or modified within the scope of the technical idea of the present invention.

1 schematically illustrates the configuration of an asymmetric cluster distributed file system according to the present invention.

FIG. 2 schematically illustrates partitioning and distributing hot data into hot data blocks in a data server of an asymmetric cluster distributed file system according to the present invention.

3 schematically illustrates metadata stored / managed by a metadata server of an asymmetric cluster distributed file system according to the present invention.

4 schematically illustrates a data structure stored in a data server of an asymmetric cluster distributed file system according to the present invention for managing information about file data chunks and chunk blocks.

FIG. 5 is a flowchart illustrating processing when hot data occurs when a client attempts to access data in an asymmetric cluster distributed file system according to the present invention.

6 is a flowchart schematically illustrating a process of detecting a hot data chunk and duplicating a hot data chunk block in a data server of an asymmetric cluster distributed file system according to the present invention.

Claims (12)

Metadata for the data file; Metadata for each data chunk divided in the data file; And It includes a data storage for storing metadata for the data chunk block divided from the data chunk, Metadata server in an asymmetric cluster distributed file system. The method of claim 1, Metadata for the data file, A chunk identifier identifying the data chunk, Metadata for the data chunk, A file identifier identifying the data file; And A data chunk block identifier identifying the data chunk block, Metadata for the data chunk block, A data chunk identifier identifying the data chunk, Metadata server in an asymmetric cluster distributed file system. The method of claim 2, The data storage unit, Additionally storing metadata about a child data chunk block partitioned from the data chunk block, Metadata for the data chunk block, Further comprising a child data chunk block identifier identifying the child data chunk block, Metadata for the child data chunk block, A parent data chunkblock identifier identifying the data chunkblock; Metadata server in an asymmetric cluster distributed file system. Receiving a data access request from a client; Retrieving a cumulative number of access requests for the data chunk block that is the target of the access request; If the cumulative number of access requests is equal to or less than a maximum reference value, returning target data of the access request; If the cumulative number of access requests is greater than or equal to a maximum reference value, replicating the data chunk block to another data server; And Generating an identifier for the replicated data chunk block and returning it to a client, A method for processing data in a data server in an asymmetric cluster distributed file system. The method of claim 4, wherein After executing the copying step of the data chunk block, Transmitting information on the data chunk block to a metadata server; A method for processing data in a data server in an asymmetric cluster distributed file system. The method of claim 4, wherein The method may further include dividing and storing the replicated data chunk block into a plurality of child data chunk blocks. A method for processing data in a data server in an asymmetric cluster distributed file system. The method of claim 4, wherein Before the cumulative number retrieval step of the access request, Searching for the data chunk block targeted for the data access request, whether there is an identifier of the data chunk block replicated to another data server, If there is an identifier of the data chunk block replicated to another data server, the identifier of the data chunk block is returned to the client, If there is an identifier of the data chunk block replicated in another data server, executing the step of retrieving the cumulative number of access requests, A method for processing data in a data server in an asymmetric cluster distributed file system. The method of claim 4, wherein The data server, If the cumulative number of data access requests is less than or equal to a reference cumulative number, the cumulative number of data access requests is decreased, When the cumulative number of the data access request is equal to or less than a reference cumulative number of times during a reference time, the divided data chunk blocks are combined into data chunks. A method for processing data in a data server in an asymmetric cluster distributed file system. A data server for dividing and storing data chunks of the data file into a plurality of data chunk blocks; A metadata server for storing metadata of the data file, metadata of the data chunk and metadata of the data chunk block; And Clients Asymmetric cluster distributed file system containing. 10. The method of claim 9, The data server, In response to the data access request of the client, if the cumulative number of access requests for the data chunk block targeted for the access request is less than or equal to a reference cumulative number, the target data of the access request is returned. If the cumulative number of access requests is greater than or equal to a reference cumulative number, the data chunk block is replicated to another data server, and an identifier for the replicated data chunk block is returned to a client. The metadata server, Receiving the replication information of the data chunk block from the data server or from the data server that has replicated the data chunk block from the data server, Generating metadata information about the replicated data chunk block and an identifier between the data chunk and the data chunk block, Asymmetric Cluster Distributed File System. The method of claim 10, The data server, If the cumulative number of access requests is less than or equal to a reference cumulative number, the cumulative number of access requests is decreased. When the cumulative number of access requests is equal to or less than a reference cumulative number of times during a reference time, the divided data chunk blocks are combined into data chunks. Asymmetric Cluster Distributed File System. The method of claim 10, The data server receiving the data chunk block from the data server, The replicated data chunk block, Split into multiple data chunk blocks and store, replicate as many data servers as the set number Transmitting the replication information of the data chunk block to a metadata server, Asymmetric Cluster Distributed File System.

Images