CN112487027B

CN112487027B - Efficient data query implementation method based on block chain electronic transaction

Info

Publication number: CN112487027B
Application number: CN202011390893.7A
Authority: CN
Inventors: 戴鸿君
Original assignee: Shandong Inspur Scientific Research Institute Co Ltd
Current assignee: Shandong Inspur Scientific Research Institute Co Ltd
Priority date: 2020-12-02
Filing date: 2020-12-02
Publication date: 2022-08-23
Anticipated expiration: 2040-12-02
Also published as: CN112487027A

Abstract

The invention provides a high-efficiency data query realization method based on block chain electronic transaction, which comprises the following steps: 1) the method comprises the steps that heterogeneous data sources of a plurality of systems are obtained through data monitoring by adopting an ETL technology, the data are loaded into a data warehouse after being processed and stored in a database capable of providing rich query semantics, and then the stored data are divided into cold and hot data for processing; 2) for the thermal data: supporting faster queries by building indexes or multi-node backups; 3) for cold data: and further compressing the data according to the stored database or carrying out few backups to achieve the aim of reducing the storage space. The invention starts the block chain system query optimization scheme from two aspects, firstly starts from storing a database, secondly establishes an efficient index mechanism, and stores the generated data into the database capable of providing rich query semantics through data monitoring, thereby improving the query efficiency, expanding the query function and enhancing the flexibility of data storage.

Description

Efficient data query implementation method based on block chain electronic transaction

Technical Field

The invention relates to a high-efficiency data query realization method based on block chain electronic transaction, belonging to the technical field of big data retrieval.

Background

The block chain is originated from the bitcoin, and the trust problem in the transaction can be effectively solved through the technologies of a distributed account book, a consensus mechanism, an encryption algorithm and the like. In recent years, the development of block chains tends to be hot and the commercial application projects are exploded, but the technology is not applied in a large scale, and only partial test points are provided in the aspects of finance, medical treatment, logistics and the like. There are still many problems in performance, safety, usability, and the technology thereof has not yet developed to maturity.

The block chain of fig. 1 is essentially a decentralized system, which is a chain-like data structure consisting of a plurality of blocks. Each block is divided into a block head and a block body, the block head is mainly used for realizing the hash value of the block linked with the previous block, and the block body mainly comprises a transaction book.

Taking a transaction scenario as an example, the workflow is as follows:

(1) the client initiates a transaction, broadcasts the transaction to other nodes on the network after digital signature and waits for confirmation;

(2) the node in the network confirms and verifies the received transaction information, and after the verification is passed, the data is recorded into a block;

(3) and multiple receiving nodes in the whole network execute a consensus algorithm on the blocks, and the blocks are formally incorporated into a block chain for storage through the consensus algorithm.

Each user initiating a transaction in the transaction signs a randomly hashed digital signature to the previous transaction and the next owner's public key and attaches this signature to the end of the transferred electronic money, which is sent to the next owner, and the payee verifies the signature and can verify the owner of the chain, while each transaction is open, i.e. secured by a common recognition mechanism, in order to avoid double payment by the electronic money.

The blockchain query can be divided into account query, transaction query and contract query according to the query object. Although the block chain can support 3 types of queries, as the data size increases and the business requirements increase, the query disadvantage is gradually revealed, and the following points are mainly reflected.

Firstly, the query efficiency is low

As can be seen from the description of the levelDB, the levelDB is mainly suitable for scenes with more write operations and less read operations. The method comprises the steps of firstly accessing a memory, then accessing a cache, and finally sequentially querying SStables with different levels, wherein when the stored data volume is very large, the querying efficiency is very low. Moreover, with the ever-increasing data volume and the increasingly frequent query demands, it will become a major bottleneck of query performance.

② limited inquiry function

Most block chaining systems still use a levelDB-like key-value database, which is inherently high performing for large numbers of writes, but which only supports key-value based insertions and queries, and does not support lookups for any field in the value. Since support for analytic queries is not considered at the beginning of the design, no index is designed in the middle tier. The system cannot execute relational query like the traditional database, and cannot execute Top-K query, K-NN query and other complex analytical queries.

③ lack of flexibility in data storage

The number of fields of data stored in the current block chain is small, the structure is relatively fixed, and the query processing logic is simple. With the continuous expansion of the blockchain application, the data structure of the application is more complicated, but the current data storage has low storage expansibility for various data.

Disclosure of Invention

The technical task of the invention is to provide a high-efficiency data query implementation method based on block chain electronic transaction aiming at the defects of the prior art.

The technical scheme adopted by the invention for solving the technical problems is as follows:

an efficient data query implementation method based on block chain electronic transaction comprises the following steps:

1) the method comprises the steps that heterogeneous data sources of multiple systems are obtained through data monitoring by adopting an ETL technology, the data are loaded into a data warehouse after being processed and stored in a database capable of providing rich query semantics, and then the stored data are divided into cold and hot data to be processed;

2) for the thermal data: supporting faster queries by building indexes or multi-node backups;

3) for cold data: and further compressing the block chain data according to a stored database or carrying out few backups to achieve the aim of reducing the storage space.

Further, step 2) for the hot data, optimizing query efficiency by adding an additional index into the level DB, taking the level DB as a main key index, establishing auxiliary indexes for different fields in the data storage module, and designing a query layer through a built-in index structure.

Further, the query process in step 2) is divided into two stages:

2.1) sending the query command to a query module, and determining a key (main key) set of results through the auxiliary index;

2.2) using the key value to find out a result value on a primary key index (levelDB) and returning the result value to the client.

Further, the built-in index structure design query layer in the step 2) comprises a block chain read/write API, a level db, a consistency maintenance module, a communication module and other nodes, the block chain read/write API is connected with the query module in the level db, the consistency maintenance module is connected with the other nodes through the communication module, and the consistency maintenance module is further connected with the auxiliary index module and the block chain read/write API.

Further, step 3) for cold data: and copying the block chain data into an external database, and designing a query layer by means of a functional interface provided by the external database.

Further, in step 3), for cold data, copying the block chain data into an external database by adopting an external database method, and designing a query layer by means of a functional interface provided by the external database.

Further, the etherQL system is adopted in the step 3), the data on the block chain is copied into the MongoDB, and then the analysis query operation is executed by using the Mongodb.

Further, the data processing in the step 1) comprises data extraction, data cleaning and data arrangement.

Further, in the step 1), the heterogeneous data sources include data in a block chain system and a decentralized distributed file system, metadata in the heterogeneous data sources are extracted and recorded into a metadata runtime library, the metadata are managed by using the metadata, the metadata are stored by using the metadata runtime library to form an analysis result for the data query module to query, and the metadata runtime library generates record description of the metadata on the data.

Compared with the prior art, the method for realizing efficient data query based on block chain electronic transaction has the advantages that,

the invention starts from two aspects, namely starting from a storage database and establishing an efficient index mechanism, monitors and stores the generated data into a database which can provide rich query semantics through data, then processes the stored data into cold and hot data, supports faster query by establishing indexes or multi-node backup for the hot data, and further compresses the cold data according to the stored database or performs less backup to achieve the purpose of reducing the storage space.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this application, are included to provide a further understanding of the application, and the description of the exemplary embodiments and illustrations of the application are intended to explain the application and are not intended to limit the application.

FIG. 1 is a block diagram of a bit coin system according to the present invention;

FIG. 2 is a schematic diagram of the block chain-based electronic transaction query optimization method of the present invention;

FIG. 3 is a diagram of the structure of the built-in index method of the present invention.

Detailed Description

With reference to fig. 2, the method for implementing efficient data query based on blockchain electronic transaction of the present invention includes the following steps:

1) the method comprises the steps that heterogeneous data sources of a plurality of systems are obtained through data monitoring by adopting an ETL technology, the data are loaded into a data warehouse after being processed and stored in a database capable of providing rich query semantics, and then the stored data are divided into cold and hot data for processing;

For step 1) above:

Further, in the step 1), the heterogeneous data source comprises data in a block chain system and a decentralized distributed file system, metadata in the heterogeneous data source is extracted and recorded into a metadata runtime library, the metadata is managed by using the metadata, the metadata is stored by using the metadata library to form an analysis result for the data query module to query, and the metadata runtime library generates record description of the metadata on the data.

For step 2) above:

Further, the query process in step 2) is divided into two stages:

With reference to fig. 3, further, the query layer designed by the built-in index structure in step 2) includes a block chain read/write API, a level db, a consistency maintenance module, a communication module, and other nodes, where the block chain read/write API is connected to the query module in the level db, the consistency maintenance module is connected to the other nodes through the communication module, and the consistency maintenance module is further connected to the auxiliary index module and the block chain read/write API.

For step 3 above):

Claims

1. The method for realizing efficient data query based on block chain electronic transaction is characterized by comprising the following steps of:

3) for cold data: further compressing the block chain data according to a stored database or carrying out few backups to achieve the purpose of reducing the storage space;

step 2) for the hot data, optimizing query efficiency by adding an additional index into the level DB, establishing auxiliary indexes for different fields in the data storage module by taking the level DB as a main key index, and designing a query layer through a built-in index structure;

step 2) the query process is divided into two stages:

2.2) finding out a result value on a main key index (levelDB) by using a key value, and returning the result value to the client;

step 2) the built-in index structure design query layer comprises a block chain read/write API, a levelDB, a consistency maintenance module, a communication module and other nodes, wherein the block chain read/write API is connected with the query module in the levelDB, the consistency maintenance module is connected with the other nodes through the communication module, and the consistency maintenance module is also connected with an auxiliary index module and the block chain read/write API;

step 3) for cold data: copying the block chain data into an external database, and designing a query layer by means of a functional interface provided by the external database;

step 3) copying the block chain data to an external database by adopting an external database method for cold data, and designing a query layer by means of a functional interface provided by the external database;

and 3) copying the data on the block chain to MongoDB by adopting an etherQL system in the step 3), and then executing analysis query operation by utilizing the MongoDB.

2. The method as claimed in claim 1, wherein the data processing in step 1) includes data extraction, data cleaning, and data sorting.

3. The method for implementing efficient data query based on blockchain electronic transaction as claimed in claim 1, wherein the heterogeneous data sources in step 1) include data in a blockchain system and a decentralized distributed file system, metadata in the heterogeneous data sources is extracted and entered into a metadata runtime library, the metadata is managed by using the metadata, the metadata is stored by using the metadata runtime library to form an analysis result for query by the data query module, and the metadata runtime library generates record description of the metadata on the data.