CN111352587A - A data packing method and device - Google Patents

Abstract

The invention provides a data packing method and a device, wherein the method comprises the following steps: obtaining a source file to be packaged; dividing the acquired source file into data blocks; calculating data blocks with repeated FP fingerprint deletion by adopting a Hash algorithm, and reserving the data blocks with the same FP fingerprint; creating a logic file of a physical file corresponding to the data block and corresponding the physical file to the logic file; and forming the FP fingerprints into a metadata sequence and packaging the source file to be packaged into a data packet with a set format. The method can be divided into a file level and a data block level according to the granularity of eliminating duplication, the duplication elimination granularity of the data block level is smaller, obviously, the data block level duplication elimination method can provide higher data duplication elimination rate, and therefore, the novel data packaging technology adopts a data block level duplication elimination mode. The utilization efficiency of the storage product can be effectively improved, and the energy consumption is reduced.

Description

A data packing method and device

technical field

The invention relates to the technical field of storage data management, in particular to a data packaging method and device.

Background technique

With the development of storage technology, people's requirements for storage capacity are getting higher and higher, and at the same time, the requirements for the utilization of storage space are also higher and higher. Effectively improving the utilization of storage can make storage products quickly occupy the market high ground. At present, there are many common data compression software in the storage market, such as Tar, winrar, winzip, etc. Although these compression software can effectively improve the utilization rate of storage products, the utilization rate of storage products still needs to be improved.

SUMMARY OF THE INVENTION

Aiming at the problem of low utilization rate of the existing compression software, the present invention provides a data packaging method and device.

The technical scheme of the present invention is:

On the one hand, the technical solution of the present invention provides a data packaging method, comprising the following steps:

Get the source file to be packaged;

Divide the obtained source file into data blocks;

Use hash algorithm to calculate FP fingerprints for data blocks to remove duplicate data and retain data blocks with the same FP fingerprints;

Create a logical file of the physical file corresponding to the data block and correspond the physical file to the logical file;

The FP fingerprint is formed into a metadata sequence and the source file to be packaged is packaged into a data package with a set format.

Further, the formatted data packet is composed of three parts: file header, unique data block set and logical file metadata.

Further, the file header is a structure that defines the size of the data block, the number of unique data blocks, the size of the data block ID, the number of files in the package, and the location information of the metadata in the package;

Immediately after the file header are stored all unique data blocks; after the data blocks, the logical representation metadata of the file in the packet is stored.

Further, the logical representation metadata is composed of multiple entities, one of which represents a file.

Further, the length of the file name, the number of data blocks, the size of the ID of the data block and the size of the last data block are recorded in the entity header of the logical file.

Further, file name data is stored after the entity header of the logical file, and after the file name data, a group of unique data block numbers are stored, wherein the numbers correspond one-to-one with the data blocks in the unique data block set. The data packaging method of the present application can be divided into file-level and data-block-level according to the granularity of deduplication. For deduplication at the data-block-level, the deduplication granularity is smaller. Obviously, using the data-block-level deduplication method can provide higher Therefore, this new data packing technology adopts the block-level deduplication method. It can effectively improve the utilization efficiency of storage products and reduce energy consumption.

Further, the method also includes:

When reading data, first read the logical file, and then retrieve the corresponding data block from the storage system according to the FP sequence to restore the physical file copy.

On the other hand, the technical solution of the present invention also provides a data packaging device, including a source file acquisition module, a segmentation module, a calculation processing module, a logic file creation module, and a data packet generation module;

The source file acquisition module is used to acquire the source files to be packaged;

The segmentation module is used to segment the acquired source file into data blocks;

The calculation processing module is used to calculate the FP fingerprint for the data block by using the hash algorithm, delete the duplicate data and retain the data block with the same FP fingerprint;

The logical file creation module is used to create the logical file of the physical file corresponding to the data block and correspond the physical file to the logical file;

The data packet generation module is used for composing the FP fingerprint into a metadata sequence and packaging the source file to be packaged into a data packet of a set format.

Further, the device further includes a physical file restoration module, which is used to read the logical file, and then retrieve the corresponding data block from the storage system according to the FP sequence to restore the physical file copy.

It can be seen from the above technical solutions that the present invention has the following advantages: the data packaging method of the present application can greatly reduce the amount of data, and eliminate redundant data by collecting duplicate data. With this data packaging method, the efficiency of the storage system can be improved, the cost can be effectively saved, and the occupancy rate of the network bandwidth during the data transmission process can be reduced. At the same time, it is also a green storage technology, which can effectively reduce energy consumption. The data packaging method of the present application can be divided into a file level and a data block level according to the granularity of deduplication, and the deduplication granularity of the data block level is smaller, and can reach between 4-24KB. Obviously, using the data block-level deduplication method can provide a higher data deduplication rate, which can effectively improve the utilization efficiency of storage products and reduce energy consumption.

In addition, the present invention has reliable design principle and simple structure, and has a very wide application prospect.

It can be seen that, compared with the prior art, the present invention has outstanding substantive features and significant progress, and the beneficial effects of its implementation are also obvious.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. In other words, other drawings can also be obtained based on these drawings without creative labor.

FIG. 1 is a schematic flowchart of a method according to an embodiment of the present invention.

Detailed ways

In order to make those skilled in the art better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described The embodiments are only some of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Example 1

As shown in Figure 1, the technical solution of the present invention provides a data packaging method, comprising the following steps:

S11: Obtain the source file to be packaged;

S12: dividing the acquired source file into data blocks; in this embodiment, dividing the source file into 8K-sized data blocks;

S13: Calculate FP fingerprints for data blocks using a hash algorithm, remove duplicate data, and retain data blocks with the same FP fingerprint; FingerPrint, FP fingerprint;

It should be noted that the hash algorithm maps a binary value of any length to a shorter fixed-length binary value, and this small binary value is called a hash value. A hash value is a unique and extremely compact numerical representation of a piece of data. If you hash a segment of plaintext and change even one letter of the segment, subsequent hashes will yield different values. It is computationally impossible to find two different inputs that hash to the same value, so hashing the data can verify the integrity of the data; hashing algorithms are used to generate some data fragment (such as a message or a session) item) of the hash value of the algorithm. With a good hashing algorithm, changes made in the input data can change all the bits in the resulting hash value. Turn the file into fixed-length or variable-length data blocks, use MD5/SHA1 and other Hash hashes, delete duplicate data and retain data blocks with the same FP fingerprint;

S14: Create a logical file of the physical file corresponding to the data block and correspond the physical file to the logical file;

S15: compose the FP fingerprint into a metadata sequence and package the source file to be packaged into a data package of a set format. It should be noted that the formatted data packet consists of three parts: a file header, a unique data block set and logical file metadata. The file header is a structure that defines the size of the data block, the number of unique data blocks, the size of the data block ID, the number of files in the package, and the location information of the metadata in the package;

Immediately after the file header are stored all unique data blocks; after the data blocks, the logical representation metadata of the file in the packet is stored. The logical representation metadata is composed of multiple entities, one of which represents a file. The length of the file name, the number of data blocks, the size of the data block ID and the size of the last data block are recorded in the entity header of the logical file. File name data is stored after the entity header of the logical file, and after the file name data, a group of unique data block numbers are stored, wherein the numbers correspond one-to-one with the data blocks in the unique data block set. The data packaging method of the present application can be divided into file-level and data-block-level according to the granularity of deduplication. For deduplication at the data-block-level, the deduplication granularity is smaller. Obviously, using the data-block-level deduplication method can provide higher Therefore, this new data packing technology adopts the block-level deduplication method. It can effectively improve the utilization efficiency of storage products and reduce energy consumption. The method further includes: when reading data, first reading the logical file, and then taking out the corresponding data block from the storage system according to the FP sequence, and restoring the copy of the physical file.

Embodiment 2

The technical solution of the present invention also provides a data packaging device, comprising a source file acquisition module, a segmentation module, a calculation processing module, a logic file creation module, and a data packet generation module;

The source file acquisition module is used to acquire the source files to be packaged;

The segmentation module is used to segment the acquired source file into data blocks;

The calculation processing module is used to calculate the FP fingerprint for the data block by using the hash algorithm, delete the duplicate data and retain the data block with the same FP fingerprint;

The logical file creation module is used to create the logical file of the physical file corresponding to the data block and correspond the physical file to the logical file;

The data packet generation module is used for composing the FP fingerprint into a metadata sequence and packaging the source file to be packaged into a data packet of a set format.

The device also includes a physical file restoration module, which is used to read the logical file, and then retrieve the corresponding data block from the storage system according to the FP sequence to restore the physical file copy.

The resulting packet consists of three parts, the file header, the set of unique data blocks, and the logical file metadata. The file header is a structure that defines meta information such as the size of the data block, the number of unique data blocks, the size of the data block ID, the number of files in the package, and the location of the metadata in the package. Immediately after the file header, all unique data blocks are stored, and the size and number are indicated by the meta information in the file header. Behind the data block is the logical representation metadata of the file in the data package, consisting of multiple entities, one of which represents a file. When unpacking, extract the data blocks one by one according to the metadata of the file, and restore the original physical file. Information such as the length of the file name, the number of data blocks, the size of the data block ID, and the size of the last data block are recorded in the entity header of the logical file. This is followed by the filename data, the length is defined in the entity header. After the file name data, the numbers of a group of unique data blocks are stored, and the numbers correspond one-to-one with the data blocks in the unique data block set. Finally, the last data block of the file is stored. Since the size of this data block is usually smaller than the normal data block, the repetition probability is very small, so it is stored separately.

Although the present invention has been described in detail in conjunction with the preferred embodiments with reference to the accompanying drawings, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, those of ordinary skill in the art can make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions should all fall within the scope of the present invention/any Those skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should all be included within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (9)

1. A method of data packaging, comprising the steps of:

obtaining a source file to be packaged;

dividing the acquired source file into data blocks;

calculating data blocks with repeated FP fingerprint deletion by adopting a Hash algorithm, and reserving the data blocks with the same FP fingerprint;

creating a logic file of a physical file corresponding to the data block and corresponding the physical file to the logic file;

and forming the FP fingerprints into a metadata sequence and packaging the source file to be packaged into a data packet with a set format.

2. A data packing method according to claim 1, wherein the formatted data packet is composed of three parts: a file header, a set of unique data blocks, and logical file metadata.

3. A data packing method according to claim 2, wherein the file header is a structure body defining a data block size, a number of unique data blocks, a data block ID size, a number of files in the packet, and location information of metadata in the packet;

all unique data blocks are stored immediately after the file header; following the data blocks, logical representation metadata for the files in the data package is stored.

4. A data packaging method according to claim 3, wherein said logical representation metadata is composed of a plurality of entities, wherein one entity represents one file.

5. A data packing method according to claim 4, wherein file name length, number of data blocks, data block ID size and last data block size information are recorded in the physical header of the logical file.

6. A data packing method according to claim 5, characterized in that the file name data is stored after the physical header of the logical file, and the file name data is followed by the numbers of a group of unique data blocks, wherein the numbers correspond to the data blocks in the unique data block set one by one.

7. A data packing method according to claim 1, further comprising:

when data is read, the logic file is read firstly, then the corresponding data block is taken out from the storage system according to the FP sequence, and the physical file copy is restored.

8. A data packaging device is characterized by comprising a source file acquisition module, a segmentation module, a calculation processing module, a logic file creation module and a data packet generation module;

the source file acquisition module is used for acquiring a source file to be packaged;

the segmentation module is used for segmenting the acquired source file into data blocks;

the calculation processing module is used for calculating data blocks for which the FP fingerprints are deleted repeatedly by adopting a Hash algorithm and reserving the data blocks with the same FP fingerprints;

the logical file creating module is used for creating a logical file of the physical file corresponding to the data block and corresponding the physical file to the logical file;

and the data packet generating module is used for forming the FP fingerprints into a metadata sequence and packaging the source files to be packaged into data packets with set formats.

9. The data packaging apparatus of claim 8, further comprising a physical file restoring module, wherein the physical file restoring module is configured to read the logical file, and then retrieve the corresponding data block from the storage system according to the FP sequence to restore the physical file copy.

Images