+

WO2018121430A1 - Procédé, appareil, support, dispositif de stockage et d'indexation de fichiers, et procédé de lecture de fichiers - Google Patents

Procédé, appareil, support, dispositif de stockage et d'indexation de fichiers, et procédé de lecture de fichiers Download PDF

Info

Publication number
WO2018121430A1
WO2018121430A1 PCT/CN2017/117967 CN2017117967W WO2018121430A1 WO 2018121430 A1 WO2018121430 A1 WO 2018121430A1 CN 2017117967 W CN2017117967 W CN 2017117967W WO 2018121430 A1 WO2018121430 A1 WO 2018121430A1
Authority
WO
WIPO (PCT)
Prior art keywords
file
key value
index
offset
bytes
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2017/117967
Other languages
English (en)
Chinese (zh)
Inventor
陈闯
张炎泼
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guizhou Baishancloud Technology Co Ltd
Original Assignee
Guizhou Baishancloud Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guizhou Baishancloud Technology Co Ltd filed Critical Guizhou Baishancloud Technology Co Ltd
Publication of WO2018121430A1 publication Critical patent/WO2018121430A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/13—File access structures, e.g. distributed indices

Definitions

  • Embodiments of the present invention relate to, but are not limited to, the field of file storage and indexing, and in particular, to a file storage and indexing method, device, medium, device, and method for reading a file.
  • Some well-known Internet companies in the industry have proposed solutions for a large number of small files.
  • the famous social networking site Facebook has stored more than 60 billion images and has launched the Haystack system to customize and optimize the storage of large numbers of images.
  • Other small file processing schemes include Taobao's TFS, etc.
  • the core idea of these systems is to append small files to a data file, and at the same time generate an index file to locate the location of the small file through the index file.
  • Haystack's data file part Haystack's data file, which encapsulates each small file into a file containing the key value, size, data, etc. of the file. All small files are appended to the data file in the order in which they were written.
  • Haystack's index file stores the key value of each file pin, as well as the offset, size and other information of the file pin in the data file. The program loads the index into memory when it starts, and locates the offset and size in the data file by looking up the index in memory.
  • Read request index Load the index file into memory, locate the index, and locate the offset and size of the file to be read.
  • Facebook's Haystack feature is to load the full key value of the file into memory for file location.
  • Facebook's full 8-byte key value can be fully loaded into memory, but there are two problems in the real world:
  • the storage server memory will not be too large, generally 32G to 64G;
  • the key value corresponding to the small file is difficult to control.
  • MD5 or SHA1 of the file content is selected as the key value of the file.
  • a storage server has 12 4T disks and the memory is about 32GB.
  • the server now needs to store about 4K avatars, thumbnails and other files, about 1 billion.
  • the key value of the file uses MD5, plus the offset and size fields, and the index information corresponding to an average small file occupies 28 bytes. In this case, the index occupies nearly 30GB of memory and the disk occupies only 4TB. Memory consumption is nearly 100%, and disk consumption is only 8%.
  • the indexing scheme adopted by the Haystack system consumes a large amount of memory resources, and the memory resources limit the utilization of disk resources. Therefore, in order to obtain a larger utilization of disk resources, an excessive increase in memory resources is required.
  • the embodiments of the present invention provide a file storage and indexing method, device, medium, device, and method for reading a file, so as to at least solve the problem that the indexing scheme adopted by the Haystack system consumes a large amount of memory resources.
  • an index in the index file uses a first N bytes of an actual key value of each file as a key value, and each index points to the data file
  • the key value corresponding to the offset is an offset of the first file in the one or more files pointed by the key value
  • the size value corresponding to the key value is the key value pointing
  • N is a positive integer.
  • the offset and size fields in the index file are aligned by 512 bytes.
  • the generating an index file for indexing each file in the data file further includes:
  • the index of the index file is hierarchically stored according to a key value prefix, wherein a key value of an index stored in a layer corresponding to the key value prefix is a short key value truncating the key value prefix, wherein the key
  • the value prefix has a byte length less than N.
  • the offset of the index of the index file is an intra-layer offset of the offset of the index, and the number of bytes of the intra-layer offset is determined according to the layered maximum layer address space. of.
  • the method further includes mapping all of the files in the data file to a Bloom filter such that when the file in the data file is read, the Bron filter is quickly searched to determine that the file is to be read. Whether the file may exist.
  • the computer readable storage medium provided by the embodiment of the present invention stores a computer program, and when the program is executed by the processor, the steps of the foregoing method are implemented.
  • a computer device provided by an embodiment of the present invention includes a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor implements the steps of the foregoing method when the program is executed.
  • a data file storage module configured to store a data file, wherein the data file is obtained by storing each file in alphabetical order according to an actual key value of the file;
  • An index file generating module configured to generate an index file for indexing each file in the data file, wherein an index in the index file uses a first N bytes of an actual key value of each file as a key value, and each index Pointing to one or more files in the data file, the offset corresponding to the key value is an offset of a first file in one or more files pointed by the key value, and a size corresponding to the key value
  • the value is the size of the first file in one or more files pointed to by the key value, and N is a positive integer.
  • the above device also has the following features:
  • the index file generating module is further configured to hierarchically store an index of the index file according to a key value prefix, wherein a key value of an index stored in a layer corresponding to the key value prefix is a prefix of the key value Short key value, wherein the key value prefix has a byte length less than N.
  • the above device also has the following features:
  • the offset of the index of the index file is an intra-layer offset of the offset of the index, and the number of bytes of the intra-layer offset is determined according to the layered maximum layer address space. of.
  • the above device also has the following features:
  • the device also includes:
  • mapping module configured to map all the files in the data file into the Bloom filter, so that when the file in the data file is read, by searching the Bloom filter to determine whether the file to be read is That may exist.
  • the method for reading a file in a file storage and indexing device includes:
  • the file is read when it matches a file whose key value is consistent with the actual key value.
  • the index corresponding to the first N bytes of the actual key value in the index file according to the first N bytes of the actual key value of the file to be read includes:
  • each file is stored in alphabetical order according to the actual key value of the file to obtain a data file; an index file for indexing each file in the data file is generated, wherein the index in the index file uses the actual key of each file.
  • the first N bytes of the value are used as key values, and each index points to one or more files in the data file, and the offset corresponding to the key value is the offset of the first file in one or more files pointed to by the key value.
  • the size corresponding to the key value is the size of the first file in one or more files pointed to by the key value, which solves the problem that the index scheme adopted by the Haystack system consumes a large amount of memory resources, and reduces the memory resources of the index system. Consumption.
  • FIG. 1 is a flow chart of a file storage and indexing method in accordance with an embodiment of the present invention
  • FIG. 2 is a structural block diagram of a file storage and indexing apparatus according to an embodiment of the present invention
  • FIG. 3 is a flowchart of a method of reading a file in a file storage and indexing device according to an embodiment of the present invention
  • FIG. 4 is a schematic diagram of a file storage and index structure in accordance with a preferred embodiment of the present invention.
  • FIG. 5 is a flow chart of a method of reading a file in accordance with a preferred embodiment of the present invention.
  • FIG. 6, FIG. 7, and FIG. 8 are schematic diagrams of index hierarchy according to a preferred embodiment of the present invention.
  • FIGS. 9 and 10 are diagrams showing a comparison of memory consumption of an indexing scheme in accordance with a preferred embodiment of the present invention.
  • FIG. 1 is a flowchart of a file storage and indexing method according to an embodiment of the present invention. As shown in Figure 1, the process includes the following steps:
  • Step S101 storing each file in alphabetical order according to the actual key value of the file, to obtain a data file
  • Step S102 generating an index file for indexing each file in the data file, wherein the index in the index file uses the first N bytes of the actual key value of each file as a key value, and each index points to one or more of the data files.
  • File, the offset value corresponding to the key value is the offset value of the first file in one or more files pointed to by the key value
  • the size value corresponding to the key value is the first one or more files pointed to by the key value.
  • the size of the file, N is a positive integer.
  • the size of the index file is reduced; at the same time, such an index no longer points to a file, but Point to the same one or more files of the first N bytes of the actual key value; in order to be able to locate the location of the file according to the offset in the index, store the file in the alphabetical order of the actual key value to the data file when the file is stored.
  • One or more files in which the first N bytes of the actual key value are the same are stored in one continuous position, and an offset is used to indicate their storage location.
  • the Haystack system of the related art will occupy less memory resources, which solves the problem that the index scheme adopted by the Haystack system consumes a large amount of memory resources.
  • the problem is that the consumption of memory resources by the indexing system is reduced.
  • the index can no longer directly index to a certain file according to the index, but will index to a continuous file set; when it is necessary to accurately read a certain file, According to the actual key value of this file, it is possible to read the desired file by matching the files one by one in the file collection.
  • the size of the file is 1024 bytes by multiplying 2 by 512 bytes; the previous need to save is 1024, now only need to save 2, save at least one byte;
  • the number of bytes required for the offset and size fields can be calculated based on the actual size of the entire data file, thereby further reducing the number of bytes occupied by the index.
  • the key value stored in the index file still has a possible row of key value prefixes. Therefore, it is also considered to layer the index in the index file according to the key value prefix.
  • the storage wherein the key value of the index stored in the layer corresponding to the key value prefix is a short key value of the truncated key value prefix, and the byte length of the key value prefix is less than N. In the case where the number of indexes in the hierarchy is larger, the number of bytes occupied by the layered index file will be smaller than the original index file.
  • the offset of the index within each layer can be further optimized to reduce the number of bytes.
  • the offset of the index of the index file is an intra-layer offset of the offset of the index, and the number of bytes of the offset in the layer is determined according to the layered maximum layer address space. . Since the maximum layer address space must be smaller than the size of the entire data file, the number of bytes occupied by the intra-layer offset will also be less than the number of bytes occupied by the original offset in the offset range of the entire data file.
  • the Bloom filter is a binary vector data structure that has good spatial and temporal efficiency and is used to detect if an element is a member of a collection. If the test result is yes, the element is not necessarily in the set; but if the test result is no, the element must not be in the set.
  • the advantage of the Bloom filter is that its insertion and query time are constant, and it does not save the element itself, but it has good security.
  • all files in the data file are also mapped into the Bloom filter, so that when the file in the data file is read, it is possible to determine whether the file to be read is possible by quickly searching for the Bloom filter. presence.
  • the value of N is preferably 4.
  • a software product stored in a storage medium (such as ROM/RAM, disk,
  • the optical disc includes a number of instructions for causing a terminal device (which may be a cell phone, a computer, a server, or a network device, etc.) to perform the methods described in various embodiments of the present invention.
  • a file storage and indexing device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and has not been described again.
  • the term “module” may implement a combination of software and/or hardware of a predetermined function.
  • the apparatus described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware, is also possible and contemplated.
  • FIG. 2 is a structural block diagram of a file storage and indexing apparatus according to an embodiment of the present invention. As shown in FIG. 2, the apparatus includes: a data file storage module 21 and an index file generating module 22, wherein
  • a data file storage module 21 configured to store data files, wherein the data files are obtained by storing the files in alphabetical order according to actual key values of the files;
  • the index file generating module 22 is coupled to the data file storage module 21 for generating an index file for indexing each file in the data file, wherein the index in the index file uses the first N bytes of the actual key value of each file as a key. Value, each index points to one or more files in the data file.
  • the offset corresponding to the key value is the offset of the first file in one or more files pointed to by the key value, and the size value corresponding to the key value is the key.
  • the size of the first file in one or more files pointed to by the value, N is a positive integer.
  • the index file generating module is further configured to hierarchically store the index of the index file according to the key value prefix, wherein the key value of the index stored in the layer corresponding to the key value prefix is a short key value of the prefix of the truncated key value, wherein the key value The prefix has a byte length less than N.
  • the offset of the index of the index file is the intra-layer offset of the offset range of the index, and the number of bytes of the intra-layer offset is determined according to the layered maximum layer address space.
  • the file storage and indexing device further includes: a mapping module, configured to map all the files in the data file to the Bloom filter, so that when the file in the data file is read, the Bron filter is searched to determine that the file is to be read. Whether the file may exist.
  • a mapping module configured to map all the files in the data file to the Bloom filter, so that when the file in the data file is read, the Bron filter is searched to determine that the file is to be read. Whether the file may exist.
  • each of the above modules may be implemented by software or hardware.
  • the foregoing may be implemented by, but not limited to, the foregoing modules are all located in the same processor; or, the modules are located in multiple In the processor.
  • the value of N is preferably 4.
  • FIG. 3 is a flow chart of a method of reading a file in a file storage and indexing device according to an embodiment of the present invention, such as As shown in Figure 3, the process includes the following steps:
  • Step S301 querying an index corresponding to the first N bytes of the actual key value in the index file according to the first N bytes of the actual key value of the file to be read;
  • Step S302 according to the actual key value, matching the file in one or more files pointed to by the index corresponding to the first N bytes of the actual key value;
  • Step S303 when the file matching the key value and the actual key value is matched, the file is read.
  • step S301 before the index is queried, whether the file to be read may be determined according to the Bloom filter; if the result of the determination is possible, according to the actual key of the file to be read The first N bytes of the value query the index corresponding to the first N bytes of the actual key value in the index file, otherwise the file is terminated.
  • the value of N is preferably 4.
  • FIG. 4 is a schematic diagram of a file storage and index structure according to a preferred embodiment of the present invention, as shown in FIG. 4, wherein the hierarchical file is stored in the memory. In the middle, the same key-value prefix is divided into one layer. Index files are used to locate small files. The data files are stored on disk, and each file pin is a small file.
  • FIG. 5 is a flow chart of a method of reading a file according to a preferred embodiment of the present invention.
  • FIG. 5 shows a specific location of a small file by matching an index prefix, and then viewing the file by reading the complete key value. Whether the key values match, if not matched, continue to search for the detailed flow of the next file pin.
  • the file storage and indexing scheme provided by the preferred embodiment includes the following steps: Step 1: compressing the prefix optimization, reducing the key value, the offset, and the size occupied space;
  • the index file only stores the first four bytes of the key value, not the full key value
  • the offset and size fields in the index file are saved by 512 bytes, saving 1 byte; and the number of bytes used for the offset and size is calculated according to the actual size of the entire data file.
  • Step 2 The file pins are stored in order, and the location of the small files is located; the file pins in the data files are stored in alphabetical order according to the key values.
  • the index file Due to the key value of the index file, only the first four bytes are saved. If the first four bytes of the small file key value are the same and the file pins are not stored sequentially, the specific positions of all the file pins scattered can not be found according to an offset. For example, the file key value read by the user is 0xabcdefacee, but since the key value in the index file only saves the first four bytes, it can only match the prefix 0xabcdefac, and the offset to be read cannot be located at this time.
  • the above problem is solved by storing the file pins sequentially: for example, the key value of the user reading the file is 0xabcdefacbb, and the prefix is 0xabcdefac, and the offset points to the file pin of 0xabcdefacaa, the first time. Match miss.
  • Step 3 Index layering optimization
  • the index with the same key-value prefix in the index can be divided into one layer.
  • the layering principle is that the number of files in each layer is controlled as much as possible to about 64, and the hierarchical level is selected according to the number of file pins to be stored in the layer.
  • the level of hierarchy can be determined as needed, for example, an example of a hierarchical level is given below:
  • Level 1 Select the first byte of the file pin key value for layering
  • Level 2 Select the first two bytes of the file pin key value for layering
  • the number of bytes used for the key-value prefix used for layering is less than the byte length of the key in the index.
  • the offset before optimization is the address space of the entire data file.
  • the offset of the layer is offset in the entire data file, and the offset of the index under the layer only needs to be offset within the layer in the data file, which can be calculated according to the maximum layer address space. The number of bytes.
  • access to the file is also avoided by the Bloom filter.
  • the Bloom filter In memory, map existing files to Bloom filters, only through a quick search.
  • the time complexity is O(k), where k is the number of bits required for an element.
  • k is the number of bits required for an element.
  • the false positive rate is 1%. If k is increased by 4.8, the false alarm rate will be reduced to 0.1%.
  • the horizontal axis represents the number of files
  • the vertical axis represents the memory size required for the index file
  • the short dashed line represents the memory consumption of the conventional Haystack
  • the long dashed line represents the memory consumption after the prefix compression by the embodiment of the present invention.
  • the horizontal axis represents the number of files
  • the vertical axis represents the memory size required for the index file
  • the short dashed line represents the memory consumption of the conventional Haystack
  • the long dashed line represents the memory consumption after the prefix compression by the embodiment of the present invention
  • the solid line The memory consumption after prefix compression and index stratification is performed by the embodiment of the present invention.
  • the 9G multi-memory consumption before optimization is further reduced to more than 4G, and one-half memory consumption is saved.
  • the overall performance of the small file is significantly improved, and the number of requests per second (RequestPerSecond, referred to as RPS) is more than doubled, and the input/output of the machine (Input/Output, referred to as For IO) usage has nearly doubled.
  • RPS RequestPerSecond
  • For IO input/output
  • the minimum memory unit is optimized, the fragmentation is reduced by 80%.
  • This embodiment provides a storage medium.
  • the above storage medium may be configured to store program code for performing the following steps:
  • Step S101 storing each file in alphabetical order according to the actual key value of the file, to obtain a data file
  • Step S102 generating an index file for indexing each file in the data file, wherein the index in the index file uses the first N bytes of the actual key value of each file as a key value, and each index points to one or more of the data files.
  • File, the offset corresponding to the key value is the offset of the first file in one or more files pointed to by the key value
  • the size corresponding to the key value is the first file in one or more files pointed to by the key value.
  • the size value, N is a positive integer.
  • the foregoing storage medium may include, but is not limited to, a USB flash drive, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, and a removable hard disk.
  • ROM Read-Only Memory
  • RAM Random Access Memory
  • mobile hard disk a hard disk
  • removable hard disk a variety of media that can store program code, such as a disk or an optical disk.
  • Embodiments of the present invention also provide a storage medium.
  • the above storage medium may be configured to store program code for performing the following steps:
  • Step S301 querying an index corresponding to the first N bytes of the actual key value in the index file according to the first N bytes of the actual key value of the file to be read;
  • Step S302 according to the actual key value, matching the file in one or more files pointed to by the index corresponding to the first N bytes of the actual key value;
  • Step S303 when the file matching the key value and the actual key value is matched, the file is read.
  • computer storage medium includes volatile and nonvolatile, implemented in any method or technology for storing information, such as computer readable instructions, data structures, program modules or other data. Sex, removable and non-removable media.
  • Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridge, magnetic tape, magnetic disk storage or other magnetic storage device, or may Any other medium used to store the desired information and that can be accessed by the computer.
  • communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and can include any information delivery media. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

Procédé, appareil, support, dispositif de stockage et d'indexation de fichiers, et procédé de lecture de fichiers, selon lesquels le procédé de stockage et d'indexation de fichiers comprend : le stockage de chaque fichier selon l'ordre alphabétique de valeurs de clés réelles de fichiers, et l'obtention d'un fichier de données ; la création d'un fichier d'index utilisé pour indexer chaque fichier dans le fichier de données, selon lequel un index dans le fichier d'index utilise des premiers N octets d'une valeur clé réelle de chaque fichier en tant que valeur clé, chaque index indique un ou plusieurs fichiers dans le fichier de données, tandis qu'une valeur de compensation correspondant à la valeur clé est une valeur de compensation du premier fichier dans un ou plusieurs fichiers des points de valeur clé, et une valeur de taille correspondant à la valeur clé est une valeur de taille du premier fichier d'un ou de plusieurs fichiers des points de valeur clé. L'invention résout le problème selon lequel la consommation de ressources de mémoire d'une solution d'indexation utilisée par un système Haystack est importante, ce qui permet de réduire la consommation de ressources de mémoire par un système d'indexation.
PCT/CN2017/117967 2016-12-26 2017-12-22 Procédé, appareil, support, dispositif de stockage et d'indexation de fichiers, et procédé de lecture de fichiers Ceased WO2018121430A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201611221215.1A CN106874348B (zh) 2016-12-26 2016-12-26 文件存储和索引方法、装置及读取文件的方法
CN201611221215.1 2016-12-26

Publications (1)

Publication Number Publication Date
WO2018121430A1 true WO2018121430A1 (fr) 2018-07-05

Family

ID=59164487

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/117967 Ceased WO2018121430A1 (fr) 2016-12-26 2017-12-22 Procédé, appareil, support, dispositif de stockage et d'indexation de fichiers, et procédé de lecture de fichiers

Country Status (2)

Country Link
CN (1) CN106874348B (fr)
WO (1) WO2018121430A1 (fr)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110825940A (zh) * 2019-09-24 2020-02-21 武汉智美互联科技有限公司 网络数据包存储和查询方法
CN111639076A (zh) * 2020-05-14 2020-09-08 民生科技有限责任公司 一种跨平台高效键值存储方法
CN111651480A (zh) * 2020-05-11 2020-09-11 腾讯科技（深圳）有限公司 数据查询方法、装置
CN112748866A (zh) * 2019-10-31 2021-05-04 北京沃东天骏信息技术有限公司 一种增量索引数据的处理方法和装置
CN113792020A (zh) * 2021-08-02 2021-12-14 惠州Tcl云创科技有限公司 一种数据处理方法、装置、终端及存储介质
CN114490014A (zh) * 2020-11-13 2022-05-13 深圳前海微众银行股份有限公司 数据处理方法、装置、设备及存储介质
CN115129476A (zh) * 2022-07-25 2022-09-30 济南浪潮数据技术有限公司 一种分片数据的索引加载方法、装置以及介质
CN115827573A (zh) * 2023-02-16 2023-03-21 麒麟软件有限公司 基于Linux的key-value形数据存储和使用方法
CN117271440A (zh) * 2023-11-21 2023-12-22 深圳市云希谷科技有限公司 一种基于freeRTOS文件信息存储方法、读取方法及相关设备
US12399868B1 (en) * 2023-03-20 2025-08-26 Amazon Technologies, Inc. Managed file compaction for distributed storage systems

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106874348B (zh) * 2016-12-26 2020-06-16 贵州白山云科技股份有限公司 文件存储和索引方法、装置及读取文件的方法
CN110209489B (zh) * 2018-02-28 2020-07-31 贵州白山云科技股份有限公司 一种适用于内存页结构的内存管理方法及装置
CN109614411B (zh) * 2018-11-19 2022-03-04 杭州复杂美科技有限公司 数据存储方法、设备和存储介质
CN110502472A (zh) * 2019-08-09 2019-11-26 西藏宁算科技集团有限公司 一种大量小文件的云存储优化方法及其系统
CN113312313B (zh) * 2021-01-29 2023-09-29 淘宝（中国）软件有限公司 数据查询方法、非易失性存储介质及电子设备
CN112765113B (zh) * 2021-01-31 2024-04-09 云知声智能科技股份有限公司 索引压缩方法、装置、计算机可读存储介质及电子设备

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1227413A1 (fr) * 2001-01-25 2002-07-31 Telefonaktiebolaget L M Ericsson (Publ) Procédé de localisation optimale de données indexées et statiques à longueur variable
CN103810246A (zh) * 2013-12-27 2014-05-21 北京天融信软件有限公司 一种索引创建方法和装置以及索引查询方法和装置
CN103914483A (zh) * 2013-01-07 2014-07-09 深圳市腾讯计算机系统有限公司 文件存储方法、装置及文件读取方法、装置
CN104572670A (zh) * 2013-10-15 2015-04-29 方正国际软件(北京)有限公司 一种小文件的存储、查询及删除方法和系统
CN105069048A (zh) * 2015-07-23 2015-11-18 东方网力科技股份有限公司 一种小文件存储方法、查询方法和装置
CN106874348A (zh) * 2016-12-26 2017-06-20 贵州白山云科技有限公司 文件存储和索引方法、装置及读取文件的方法

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8862555B1 (en) * 2011-05-16 2014-10-14 Trend Micro Incorporated Methods and apparatus for generating difference files
CN102779180B (zh) * 2012-06-29 2015-09-09 华为技术有限公司 数据存储系统的操作处理方法,数据存储系统
CN103870492B (zh) * 2012-12-14 2017-08-04 腾讯科技（深圳）有限公司 一种基于键排序的数据存储方法和装置
CN105117417B (zh) * 2015-07-30 2018-04-17 西安交通大学 一种读优化的内存数据库Trie树索引方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1227413A1 (fr) * 2001-01-25 2002-07-31 Telefonaktiebolaget L M Ericsson (Publ) Procédé de localisation optimale de données indexées et statiques à longueur variable
CN103914483A (zh) * 2013-01-07 2014-07-09 深圳市腾讯计算机系统有限公司 文件存储方法、装置及文件读取方法、装置
CN104572670A (zh) * 2013-10-15 2015-04-29 方正国际软件(北京)有限公司 一种小文件的存储、查询及删除方法和系统
CN103810246A (zh) * 2013-12-27 2014-05-21 北京天融信软件有限公司 一种索引创建方法和装置以及索引查询方法和装置
CN105069048A (zh) * 2015-07-23 2015-11-18 东方网力科技股份有限公司 一种小文件存储方法、查询方法和装置
CN106874348A (zh) * 2016-12-26 2017-06-20 贵州白山云科技有限公司 文件存储和索引方法、装置及读取文件的方法

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110825940B (zh) * 2019-09-24 2023-08-22 武汉智美互联科技有限公司 网络数据包存储和查询方法
CN110825940A (zh) * 2019-09-24 2020-02-21 武汉智美互联科技有限公司 网络数据包存储和查询方法
CN112748866A (zh) * 2019-10-31 2021-05-04 北京沃东天骏信息技术有限公司 一种增量索引数据的处理方法和装置
CN111651480A (zh) * 2020-05-11 2020-09-11 腾讯科技（深圳）有限公司 数据查询方法、装置
CN111639076B (zh) * 2020-05-14 2023-12-22 民生科技有限责任公司 一种跨平台高效键值存储方法
CN111639076A (zh) * 2020-05-14 2020-09-08 民生科技有限责任公司 一种跨平台高效键值存储方法
CN114490014A (zh) * 2020-11-13 2022-05-13 深圳前海微众银行股份有限公司 数据处理方法、装置、设备及存储介质
CN113792020A (zh) * 2021-08-02 2021-12-14 惠州Tcl云创科技有限公司 一种数据处理方法、装置、终端及存储介质
CN115129476A (zh) * 2022-07-25 2022-09-30 济南浪潮数据技术有限公司 一种分片数据的索引加载方法、装置以及介质
CN115827573A (zh) * 2023-02-16 2023-03-21 麒麟软件有限公司 基于Linux的key-value形数据存储和使用方法
US12399868B1 (en) * 2023-03-20 2025-08-26 Amazon Technologies, Inc. Managed file compaction for distributed storage systems
CN117271440A (zh) * 2023-11-21 2023-12-22 深圳市云希谷科技有限公司 一种基于freeRTOS文件信息存储方法、读取方法及相关设备
CN117271440B (zh) * 2023-11-21 2024-02-06 深圳市云希谷科技有限公司 一种基于freeRTOS文件信息存储方法、读取方法及相关设备

Also Published As

Publication number Publication date
CN106874348A (zh) 2017-06-20
CN106874348B (zh) 2020-06-16

Similar Documents

Publication Publication Date Title
WO2018121430A1 (fr) Procédé, appareil, support, dispositif de stockage et d'indexation de fichiers, et procédé de lecture de fichiers
US10417202B2 (en) Storage system deduplication
EP2863310B1 (fr) Procédé et appareil de traitement de données, ainsi que dispositif de stockage partagé
US10678654B2 (en) Systems and methods for data backup using data binning and deduplication
CN112328435A (zh) 目标数据备份和恢复的方法、装置、设备及存储介质
CN111177143B (zh) 键值数据存储方法、装置、存储介质与电子设备
CN103581331B (zh) 虚拟机在线迁移方法与系统
CN109446160A (zh) 一种文件读取方法、系统、装置及计算机可读存储介质
US9977598B2 (en) Electronic device and a method for managing memory space thereof
US20150227535A1 (en) Caseless file lookup in a distributed file system
CN105094695B (zh) 一种存储方法和系统
CN103399823A (zh) 业务数据的存储方法、设备和系统
US20150169570A1 (en) Method and device for managing data
US10515055B2 (en) Mapping logical identifiers using multiple identifier spaces
CN103973810A (zh) 基于互联网协议ip盘的数据处理方法和装置
CN110888837A (zh) 对象存储小文件归并方法及装置
CN111831208A (zh) 一种信息处理方法、装置、终端设备及存储介质
WO2017020668A1 (fr) Procédé et appareil de partage de disque physique
CN107391761A (zh) 一种基于重复数据删除技术的数据管理方法及装置
WO2025190209A1 (fr) Stockage de données basé sur un hachage parfait
CN115964002A (zh) 一种电能表终端档案管理方法、装置、设备及介质
CN107423425A (zh) 一种对k/v格式的数据快速存储和查询方法
CN116821102B (zh) 数据迁移方法、装置、计算机设备和存储介质
CN117591031A (zh) 一种对象存储方法及系统
CN113867622B (zh) 一种快照读写速度提升的方法、装置、设备及可读介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17885665

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17885665

Country of ref document: EP

Kind code of ref document: A1

点击 这是indexloc提供的php浏览器服务，不要输入任何密码和下载