CN115033179A - A data storage method, device, device and medium - Google Patents

Abstract

The application relates to the technical field of data storage, in particular to a data storage method, a device, equipment and a medium, and the data storage method, the device, the equipment and the medium can store each embedded point data into a distributed database of corresponding first target identification information according to the first target identification information of the distributed database correspondingly stored by each embedded point data in the implementation of the application, on one hand, the distributed database is more suitable for storing a large amount of data compared with a relational database, the problem that the operating speed of a service system is influenced after the embedded point data reaches a certain scale can be avoided, and the embedded point data is stored in a database, so that the storage pressure of each distributed database can be effectively reduced; on the other hand, the method does not need a developer to develop programs, reduces the workload of the developer, and also reduces the requirements on the disk performance and capacity of the business system.

Description

A data storage method, device, device and medium

technical field

The present application relates to the technical field of data storage, and in particular, to a data storage method, apparatus, device, and medium.

Background technique

With the vigorous development of various businesses, business data has also been accumulated. The administrators of various business systems and higher-level supervision departments have proposed various indicators such as the activity, transaction volume, and user behavior analysis of each system for different business systems. Collection and analysis requirements, for example, statistics Zhang San has conducted several transactions A in the business system in the past three years, etc. Specifically, various indicators can be collected and analyzed through buried point data.

In the related art, the buried point data to be analyzed can be stored in a relational database in the business system, and then the business system can perform data analysis based on the stored buried point data to obtain various indicators of the analysis completion. Since the relational database is not suitable for storing a large amount of data, when the amount of buried point data stored in the relational database reaches a certain scale, it will affect the running speed of the business system, and even cause other functions of the business system to be unable to be used normally.

In the related art, the buried point data to be analyzed can also be written into the log, and then the buried point data in the log can be loaded into the big data computing environment, and data analysis can be performed in the big data computing environment to obtain the analysis results. Because of the process of writing buried point data into the log, developers are required to perform program development operations, which increases the workload of developers, and this method has high requirements on the disk performance and capacity of the business system.

SUMMARY OF THE INVENTION

The present application provides a data storage method, device, device and medium, which are used to solve the problem that storing large-scale buried point data based on relational databases in the prior art will affect the operation speed of a business system, and writing buried point data into logs The problem of increasing the workload of developers.

The application provides a data storage method, the method includes:

Read the configuration file, and obtain the buried point data of the to-be-buried point carried in the configuration file and the first target identification information of the distributed database correspondingly stored in the buried point data;

The buried point data is stored in the distributed database of the first target identification information.

Further, the configuration file also carries the second target identification information of the business system to be buried; the storing the buried point data in the distributed database of the first target identification information includes:

According to the corresponding relationship between the second target identification information of the business system to be buried, the identification information of the pre-stored business system and the identification information of the topic in Kafka, the third identification information of the topic in Kafka corresponding to the second identification information is determined. target identification information, and write the buried point data into the topic of the third target identification information in the Kafka;

Identifying the buried point data written in the subject of the third target identification information, and storing the identified buried point data in the distributed database of the first target identification information.

Further, the method also includes:

If the buried point data in the distributed database of the first target identification information is lost, after the identification conditions are met, the buried point data written in the subject of the third target identification information is identified, and the identified The buried point data is stored in the distributed database of the first target identification information.

Further, the buried point data written in the subject of identifying the third target identification information includes:

The buried point data written in the subject based on Flink identifying the third target identification information; or

The buried point data written in the subject of the third target identification information is identified based on Flink SQL.

Further, the configuration file also carries the fourth target identification information of the data table;

The storing of the buried point data in the distributed database of the first target identification information includes:

According to the fourth target identification information, the buried point data is stored in the target data table of the fourth target identification information in the distributed database of the first target identification information.

Further, the method also includes:

reading the configuration file, and determining whether the configuration file carries the information for processing the buried point data;

If so, perform data analysis on the buried point data.

Further, performing data analysis on the buried point data includes:

Perform data analysis on the buried point data based on the Flink filter operator; or

Based on Flink SQL, data analysis is performed on the buried point data.

The present application also provides a data storage device, the device comprising:

an acquisition module, configured to read a configuration file, and acquire the buried point data of the to-be-buried point carried in the configuration file and the first target identification information of the distributed database correspondingly stored in the buried point data;

A storage module, configured to store the buried point data in the distributed database of the first target identification information.

Further, the storage module is specifically configured to, if the configuration file also carries the second target identification information of the business system of the point to be buried, according to the second target identification information of the business system of the point to be buried, the pre-saved information The corresponding relationship between the identification information of the business system and the identification information of the topic in Kafka, determine the third target identification information of the topic in Kafka corresponding to the second identification information, and write the buried point data into all the data in the Kafka. In the subject of the third target identification information; identify the buried point data written in the subject of the third target identification information, and store the identified buried point data in the first target identification information. in a distributed database.

Further, the storage module is also used for if the buried point data in the distributed database of the first target identification information is lost, then after satisfying the identification conditions, identify the data written in the subject of the third target identification information. The buried point data is stored, and the identified buried point data is stored in the distributed database of the first target identification information.

Further, the storage module is specifically used to identify the buried point data written in the subject of the third target identification information based on Flink; or write in the subject of the third target identification information based on Flink SQL identification of the buried point data.

Further, the storage module is specifically configured to store the buried point data in the first target according to the fourth target identification information if the configuration file also carries the fourth target identification information of the data table. in the target data table of the fourth target identification information in the distributed database of identification information.

Further, the device also includes:

a determination module, configured to read the configuration file, and determine whether the configuration file carries information for processing the buried point data;

The analysis module is configured to perform data analysis on the buried point data if the information for processing the buried point data is carried.

Further, the analysis module is specifically configured to perform data analysis on the buried point data based on the Flink filter operator; or perform data analysis on the buried point data based on Flink SQL.

The present application also provides an electronic device, the electronic device includes at least a processor and a memory, and the processor is configured to implement the steps of any of the above data storage methods when executing a computer program stored in the memory.

The present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, implements the steps of any of the above data storage methods.

The present application also provides a computer program product, wherein the computer program product includes: computer program code, when the computer program code runs on a computer, the computer executes the steps of any of the above data storage methods.

In the embodiment of the present application, the configuration file is read, the buried point data of the to-be-buried point carried in the configuration file and the first target identification information of the distributed database corresponding to the buried point data are obtained, and the buried point data is stored in the first target identification information of the distributed database. A distributed database of target identification information. Because in the implementation of the present application, each buried point data can be stored in the distributed database of the corresponding first target identification information according to the first target identification information of the distributed database correspondingly stored in each buried point data, on the one hand, Compared with relational databases, distributed databases are more suitable for storing a large amount of data, and there will be no problem of affecting the operation speed of the business system after the embedded data reaches a certain scale, and each embedded data is stored in separate databases. It can effectively reduce the storage pressure of each distributed database; on the other hand, this method does not require developers to develop programs, reduces the workload of developers, and also reduces the requirements for disk performance and capacity of business systems.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic diagram of a data storage process provided by some embodiments of the present application;

FIG. 2 is a schematic diagram of a process of burying point data burying points provided by some embodiments of the present application;

3 is a schematic diagram of a detailed process of a buried point data buried point provided by some embodiments of the present application;

FIG. 4 is a schematic structural diagram of a data storage device according to some embodiments of the present application;

FIG. 5 is a schematic structural diagram of an electronic device according to some embodiments of the present application.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

When storing the buried point data in the embodiment of the present application, the configuration file is read, and the buried point data of the buried point carried in the configuration file and the first target identification information of the distributed database corresponding to the buried point data are obtained; The buried point data is stored in the distributed database of the first target identification information.

In order to reduce the workload of developers and avoid the problem that storing large-scale buried point data based on a relational database will affect the running speed of a business system, embodiments of the present application provide a data storage method, apparatus, device, and medium.

Example 1:

1 is a schematic diagram of a data storage process provided by some embodiments of this application, and the process includes the following steps:

S101: Read a configuration file, and obtain the buried point data of the to-be-buryed point carried in the configuration file and the first target identification information of the distributed database corresponding to the buried point data.

The data storage method provided by the embodiment of the present application is applied to an electronic device, and the electronic device may be an intelligent terminal, a PC, a server, or the like.

In the embodiment of the present application, the electronic device obtains a configuration file in advance, wherein the configuration file carries the buried point data of the buried point and the first target identification information of the distributed database corresponding to the buried point data. In order to determine Where to store the buried point data, the electronic device reads the configuration file to obtain the buried point data and the first target identification information corresponding to the stored distributed database. Wherein, the first target identification information of the distributed database stored corresponding to different buried point data may be the same or different. Wherein, the distributed database may be Hive, HBase, ES, etc., and there is no specific limitation.

Wherein, the configuration file may include a first subfile for configuring the buried point data, and may also include a second subfile for configuring the identification information of the distributed database where the buried point data is stored, in order to determine which buried point data to use , which distributed database to store in, the electronic device reads the first sub-file, determines the buried point data of the point to be buried, reads the second sub-file, and determines that the buried point data corresponds to the first sub-file stored in the distributed database. a target identification information. Specifically, since the first subfile may contain buried point data of multiple points to be buried, in order to determine where to store the buried point data, the corresponding relationship between the buried point data and its identification information can be stored in the first subfile, The corresponding relationship between the identification information of the buried point data and the identification information of the distributed database is stored in the second sub-file. For each buried point data, based on the corresponding relationship between the buried point data stored in the first subfile and its identification information, determine the target identification information corresponding to the buried point data, and in the second subfile according to the pre-stored buried point data The corresponding relationship between the identification information of the target identification information and the identification information of the distributed database is determined, and the buried point data of the target identification information is determined to correspond to the first target identification information of the distributed database stored.

S102: Store the buried point data in a distributed database of the first target identification information.

After determining the buried point data and the correspondingly stored first target identification information in the distributed database, the buried point data is stored in the distributed database of the first target identification information.

Because in the implementation of the present application, each buried point data can be stored in the distributed database of the corresponding first target identification information according to the first target identification information of the distributed database correspondingly stored in each buried point data, on the one hand, Compared with relational databases, distributed databases are more suitable for storing a large amount of data, and there will be no problem of affecting the operation speed of the business system after the embedded data reaches a certain scale, and each embedded data is stored in separate databases. It can effectively reduce the storage pressure of each distributed database; on the other hand, there is no need to write the buried point data to the log to synchronize the buried point data in the big data computing environment, which reduces the development of developers. It also reduces the disk performance and capacity requirements of the business system.

Example 2:

In order to store the identified buried point data in the distributed database of the first target identification information, on the basis of the above embodiment, in this embodiment of the present application, the configuration file also carries the first information of the business system of the to-be-buried point. 2. Target identification information;

The storing of the buried point data in the distributed database of the first target identification information includes:

According to the corresponding relationship between the second target identification information of the business system to be buried, the identification information of the pre-stored business system and the identification information of the topic in Kafka, the third identification information of the topic in Kafka corresponding to the second identification information is determined. target identification information, and write the buried point data into the topic of the third target identification information in the Kafka;

Identifying the buried point data written in the subject of the third target identification information, and storing the identified buried point data in the distributed database of the first target identification information.

In the embodiment of the present application, in order to realize the decoupling of the electronic device and the distributed database, the electronic device can also write the buried point data into Kafka, and the subsequent electronic device recognizes the buried point data in Kafka, and records the identified buried point data. The point data is stored in the distributed database of the first target identification information, so as to ensure that when the buried point data in the distributed database of the first target identification information is lost, the buried point data written in Kafka can be directly identified, and the data can be rewritten. The buried point data is stored in the distributed database of the first target identification information without re-identifying the buried point data in the configuration file and the first identification information of the distributed database in which the buried point data is stored.

Among them, Kafka is a high-throughput distributed publish-subscribe messaging system. It is an open source stream processing platform developed by the Apache Software Foundation. It provides fast message persistence capabilities, even if the data is Terabyte (TB). ) can also maintain stable performance for a long time, and at the same time has high throughput characteristics, even with very common hardware, Kafka can support millions of messages per second.

In order to collect and analyze various indicators such as the activity, transaction volume, and user behavior analysis of each business system, the distributed database where the buried point data is stored can be the database in the device deployed by the business system.

In order to facilitate subsequent identification, the buried point data corresponding to different business systems can also be written into different topics in Kafka for different business systems. Specifically, the configuration file can also carry the second target of the business system to be buried. identification information, and according to the second target identification information of the business system to be buried, and the correspondence between the pre-saved identification information of the business system and the topic in Kafka, determine the topic in Kafka corresponding to the second target identification information The third target identification information, and the buried point data is written into the topic of the third target identification information in Kafka, wherein the buried point data of the business system corresponding to different identification information is written into the different identification information in Kafka. in the theme.

In order to store the buried point data in the distributed database of the first target identification information, the electronic device identifies the buried point data in Kafka, and stores the buried point data in Kafka in the distributed database of the first target identification information.

In related technologies, in order to collect and analyze various indicators such as system activity, transaction volume, user behavior analysis, etc., change data capture (CDC) tools can be used to store data in relational databases. The data is synchronized to the big data computing environment, the buried point data in the relational database is deleted, and the data is analyzed in the big data computing environment to obtain various indicators of the analysis. Database exception. In this application, since it is not necessary to synchronize the buried point data in the big data computing environment for data analysis, the data analysis can be performed locally on the device deployed in the business system, avoiding the synchronization of the buried point data in the big data computing environment, resulting in business systems are at risk of being hacked.

FIG. 2 is a schematic diagram of a process of burying point data according to some embodiments of the present application, and FIG. 2 is now described.

The electronic device pre-reads the first sub-file of the configuration file, determines the buried point data saved in the first sub-file, reads the second sub-file in the configuration file, and determines that the buried point data corresponds to the first sub-file stored in the distributed database. A target identification information, and then the buried point data is stored in the distributed database corresponding to the first target identification information, and then the equipment deployed in the business system, that is, the device where the distributed database is located. Data cleaning and processing, and output the obtained error-free buried point data to the display page of the business system. The process of performing data cleaning processing on the buried point data is in the prior art, and details are not described here.

In order to realize the decoupling of the electronic device and the distributed database, on the basis of the foregoing embodiments, in the embodiment of the present application, the method further includes:

If the buried point data in the distributed database of the first target identification information is lost, after satisfying the identification conditions, identify the buried point data written in the subject of the third target identification information, and identify the buried point data. The buried point data is stored in the distributed database of the first target identification information.

If the buried point data obtained after reading the configuration file is directly stored in the distributed database of the first target identification information, the stored buried point data may be lost in subsequent distributed databases. When the buried point data is lost, in order to re-store the buried point data to the distributed database, it is necessary to re-read the buried point data of the configuration file and the first target identification information corresponding to the buried point data stored in the distributed database, and then store the buried point data in the distributed database. The point data is stored in the distributed database of the first target identification information, thus causing over-coupling of the electronic device and the distributed database.

In order to decouple the electronic device from the distributed database, since the buried point data has been written into the topic of the third target identification information in Kafka in advance, and the topic of the buried point data written in different business systems has been written Therefore, after the identification conditions are met, the buried point data written in the subject of the third target identification information can be identified, and the identified buried point data can be stored in the distributed database of the first target identification information. , so that the buried point data is stored in the distributed database, avoiding the process of the electronic device re-identifying the buried point data in the configuration file and the first target identification information corresponding to the stored distributed database.

Wherein, the identification condition may be after the buried point data is lost for a preset length of time or an identification instruction is received.

Example 3:

In order to identify the buried point data, on the basis of the above embodiments, in the embodiment of the present application, the buried point data written in the subject of identifying the third target identification information includes:

The buried point data written in the subject based on Flink identifying the third target identification information; or

The buried point data written in the subject of the third target identification information is identified based on Flink SQL.

In the embodiment of the present application, in order to identify the buried point data written into the topic of the third target identification information in Kafka, in a possible implementation, Flink can identify the topic of the third target identification information based on Flink. Buried point data written.

In another possible implementation manner, the embedded point data written in the subject of the third target identification information may also be identified based on Flink SQL.

Example 4:

In order to accurately save the data of each buried point in the corresponding storage location, on the basis of the above embodiments, in the embodiment of the present application, the configuration file also carries the fourth target identification information of the data table;

The storing of the buried point data in the distributed database of the first target identification information includes:

According to the fourth target identification information, the buried point data is stored in the target data table of the fourth target identification information in the distributed database of the first target identification information.

In the embodiment of the present application, each distributed database pre-stores multiple data tables, and the data tables are used to store buried point data. In order to determine which data table in the distributed database to store the buried point data in, the configuration file also carries the fourth target identification information of the data table. Specifically, the data can be carried in the third subfile in the configuration file. table information. Since there may be a situation where multiple buried point data needs to be stored in the same distributed database, in order to determine which buried point data is stored in which data table, the third sub-file can also be pre-saved for each distributed database. The corresponding relationship between the identification information of the buried point data and the identification information of the data table in the distributed data, for each buried point data, based on the corresponding relationship between the buried point data saved in the first subfile and its identification information, determine the buried point data. The target identification information corresponding to the point data, and based on the corresponding relationship between the identification information of the buried point data pre-saved in the second sub-file and the identification information of the distributed database, determine the distributed storage corresponding to the buried point data of the target identification information. The first target identification information of the database, and finally based on the corresponding relationship between the identification information of the pre-saved buried point data and the identification information of the data table in the distributed data for each distributed database in the third subfile, the buried point is determined. The fourth identification information of the data table to be stored corresponding to the data.

After determining the fourth target identification information of the data table carried in the third sub-file, according to the fourth target identification information, store the buried point data in the fourth target identification information in the distributed database of the first target identification information in the target data table.

Example 5:

In order to meet the needs of performing data analysis on the buried point data, on the basis of the foregoing embodiments, in this embodiment of the present application, the method further includes:

reading the configuration file, and determining whether the configuration file carries the information for processing the buried point data;

If so, perform data analysis on the buried point data.

In the embodiment of the present application, some buried point data may also have a data analysis requirement. In order to determine whether each buried point data has a data analysis requirement, the configuration file also carries the buried point data to be analyzed. Analyzed information. Specifically, the fourth sub-file in the configuration file may carry information for processing the buried point data to be analyzed. In order to determine which buried point data to perform data analysis on, the fourth sub-file also pre-stores the corresponding relationship between the identification information of the buried point data to be analyzed and the information to be processed. For each buried point data, based on the first The corresponding relationship between the buried point data saved in the sub-file and its identification information, determine the target identification information corresponding to the buried point data, and analyze the identification information of the buried point data and process it based on the data to be pre-saved in the fourth sub-file. The corresponding relationship of the information, to determine whether there is information for processing the buried point data corresponding to the target identification information. If it does not exist, it is determined that data analysis is not required for the buried point data. If it exists, it is determined that the buried point data needs to be processed. data for data analysis.

When performing data analysis on the buried point data, for each buried point data, if it is determined that the configuration file carries information for processing the buried point data, data analysis is performed on the buried point data.

In order to perform data analysis on the buried point data, on the basis of the above embodiments, in this embodiment of the present application, the data analysis on the buried point data includes:

Perform data analysis on the buried point data based on the Flink filter operator; or

Based on Flink SQL, data analysis is performed on the buried point data.

In order to perform data analysis on the buried point data, in a possible implementation manner, data analysis may be performed on the buried point data based on the Flinkfilter operator.

In another possible implementation, data analysis can also be performed on the buried point data based on Flink SQL.

Among them, Flink SQL is an open-source, distributed data analysis engine based on the Flink platform, providing high-throughput, low-latency, and high-performance data analysis capabilities based on platforms such as Yarn/distributed computing framework (Mesos). It provides data analysis based on SQL-like methods, and provides two data analysis modes of batch and real-time.

FIG. 3 is a schematic diagram of a detailed process of burying point data according to some embodiments of the present application, and FIG. 3 is now described.

The pre-configured configuration file is obtained first, and the electronic device reads the configuration file to determine whether the first subfile in the configuration file exists. If not, it outputs a prompt message that the first subfile is not found. The buried point data to be buried carried in the subfile is written to Kafka, and the buried point data written in Kafka is identified based on Flink.

Then the electronic device determines whether there is a second sub-file in the configuration file, and if not, outputs a prompt message that the second sub-file is not found; The point data corresponds to the first target identification information stored in the distributed database, and for each buried point data, determine the fourth target identification information of the data table corresponding to the buried point data carried in the third subfile in the configuration file. , and store the buried point data in the target data table of the fourth target identification information in the distributed database of the first target identification information. Then determine whether there is a fourth subfile in the configuration file. If so, then for each buried point data, determine whether the fourth subfile carries the information for processing the buried point data. If so, treat it based on Flink SQL. The buried point data subjected to data analysis is subjected to data analysis, and the results obtained after the data analysis are stored.

After receiving the query instruction, the subsequent service system outputs the corresponding result data according to the information to be queried carried in the query instruction.

Example 6:

FIG. 4 is a schematic structural diagram of a data storage device provided by some embodiments of the present application, and the device includes:

The obtaining module 401 is configured to read a configuration file, and obtain the buried point data of the to-be-buried point carried in the configuration file and the first target identification information of the distributed database correspondingly stored in the buried point data;

The storage module 402 is configured to store the buried point data in the distributed database of the first target identification information.

In a possible implementation manner, the storage module 402 is specifically configured to, if the configuration file also carries the second target identification information of the business system of the point to be buried, according to the second target identification information of the business system of the point to be buried The corresponding relationship between the target identification information, the identification information of the pre-stored business system and the identification information of the topic in Kafka, determine the third target identification information of the topic in Kafka corresponding to the second identification information, and write the buried point data. Enter into the topic of the third target identification information in the Kafka; identify the buried point data written in the topic of the third target identification information, and store the identified buried point data in the in the distributed database of the first target identification information.

In a possible implementation manner, the storage module 402 is further configured to identify the third target identification after the identification condition is satisfied if the buried point data in the distributed database of the first target identification information is lost The buried point data written in the subject of the information is stored, and the identified buried point data is stored in the distributed database of the first target identification information.

In a possible implementation manner, the storage module 402 is specifically configured to identify the buried point data written in the subject of the identification information of the third target based on Flink; or identify the third target based on Flink SQL The buried point data written in the subject of the identification information.

In a possible implementation manner, the storage module 402 is specifically configured to, if the configuration file also carries the fourth target identification information of the data table, store the buried point data according to the fourth target identification information Stored in the target data table of the fourth target identification information in the distributed database of the first target identification information.

In a possible implementation, the device further includes:

A determination module 403, configured to read the configuration file, and determine whether the configuration file carries information for processing the buried point data;

The analysis module 404 is configured to perform data analysis on the buried point data if the information for processing the buried point data is carried.

In a possible implementation manner, the analysis module 404 is specifically configured to perform data analysis on the buried point data based on the Flink filter operator; or perform data analysis on the buried point data based on Flink SQL.

Example 7:

On the basis of the foregoing embodiments, some embodiments of the present application further provide an electronic device, as shown in FIG. 5 , including: a processor 501 , a communication interface 502 , a memory 503 and a communication bus 504 , wherein the processor 501 , the communication interface 502 , and the memory 503 complete the communication with each other through the communication bus 504 .

A computer program is stored in the memory 503, and when the program is executed by the processor 501, the processor 501 is caused to perform the following steps:

Read the configuration file, and obtain the buried point data of the to-be-buried point carried in the configuration file and the first target identification information of the distributed database correspondingly stored in the buried point data;

The buried point data is stored in the distributed database of the first target identification information.

Further, the processor 501 is specifically configured to, if the configuration file also carries the second target identification information of the business system of the point to be buried, save in advance according to the second target identification information of the business system of the point to be buried. The corresponding relationship between the identification information of the business system and the identification information of the topic in Kafka, determine the third target identification information of the topic in the Kafka corresponding to the second identification information, and write the buried point data into the Kafka In the subject of the third target identification information; identify the buried point data written in the subject of the third target identification information, and store the identified buried point data in the first target identification information in a distributed database.

Further, the processor 501 is further configured to, if the buried point data in the distributed database of the first target identification information is lost, after satisfying the identification conditions, write in the subject of identifying the third target identification information. and store the identified buried point data in the distributed database of the first target identification information.

Further, the processor 501 is specifically used to identify the buried point data written in the subject of the third target identification information based on Flink; or write in the subject of the third target identification information based on Flink SQL identification The buried point data entered.

Further, the processor 501 is specifically configured to, if the configuration file also carries the fourth target identification information of the data table, store the buried point data in the first target identification information according to the fourth target identification information. in the target data table of the fourth target identification information in the distributed database of target identification information.

Further, the processor 501 is further configured to read the configuration file, and determine whether the configuration file carries information for processing the buried point data; if so, perform data analysis on the buried point data .

Further, the processor 501 is specifically configured to perform data analysis on the buried point data based on the Flink filter operator; or perform data analysis on the buried point data based on Flink SQL.

The communication bus mentioned by the above server may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA) bus or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

The communication interface 502 is used for communication between the above-mentioned electronic device and other devices.

The memory may include random access memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

The above-mentioned processor may be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it may also be a digital instruction processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array, or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

Example 8:

On the basis of the foregoing embodiments, some embodiments of the present application further provide a computer-readable storage medium, where a computer program executable by an electronic device is stored in the computer-readable storage medium. When running on the electronic device, the following steps are implemented when the electronic device is executed:

A computer program is stored in the memory, and when the program is executed by the processor, the processor is caused to perform the following steps:

Read the configuration file, and obtain the buried point data of the to-be-buried point carried in the configuration file and the first target identification information of the distributed database correspondingly stored in the buried point data;

The buried point data is stored in the distributed database of the first target identification information.

Further, the configuration file also carries the second target identification information of the business system to be buried;

The storing of the buried point data in the distributed database of the first target identification information includes:

According to the corresponding relationship between the second target identification information of the business system to be buried, the identification information of the pre-stored business system and the identification information of the topic in Kafka, the third identification information of the topic in Kafka corresponding to the second identification information is determined. target identification information, and write the buried point data into the topic of the third target identification information in the Kafka;

Identifying the buried point data written in the subject of the third target identification information, and storing the identified buried point data in the distributed database of the first target identification information.

Further, the method also includes:

If the buried point data in the distributed database of the first target identification information is lost, after the identification conditions are met, the buried point data written in the subject of the third target identification information is identified, and the identified The buried point data is stored in the distributed database of the first target identification information.

Further, the buried point data written in the subject of identifying the third target identification information includes:

The buried point data written in the subject based on Flink identifying the third target identification information; or

The buried point data written in the subject of the third target identification information is identified based on Flink SQL.

Further, the configuration file also carries the fourth target identification information of the data table;

The storing of the buried point data in the distributed database of the first target identification information includes:

According to the fourth target identification information, the buried point data is stored in the target data table of the fourth target identification information in the distributed database of the first target identification information.

Further, the method also includes:

reading the configuration file, and determining whether the configuration file carries the information for processing the buried point data;

If so, perform data analysis on the buried point data.

Further, performing data analysis on the buried point data includes:

Perform data analysis on the buried point data based on the Flink filter operator; or

Based on Flink SQL, data analysis is performed on the buried point data.

Example 9:

An embodiment of the present application further provides a computer program product, which implements the data storage method described in any of the foregoing method embodiments applied to an electronic device when the computer program product is executed by a computer.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof, and may be implemented in whole or in part in the form of computer program products. The computer program product includes one or more computer instructions, and when the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated.

Because in the implementation of the present application, each buried point data can be stored in the distributed database of the corresponding first target identification information according to the first target identification information of the distributed database correspondingly stored in each buried point data, on the one hand, Compared with relational databases, distributed databases are more suitable for storing a large amount of data, and there will be no problem of affecting the operation speed of the business system after the embedded data reaches a certain scale, and each embedded data is stored in separate databases. It can effectively reduce the storage pressure of each distributed database; on the other hand, there is no need to write the buried point data to the log to synchronize the buried point data in the big data computing environment, which reduces the development of developers. It also reduces the disk performance and capacity requirements of the business system.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims (17)

1. A method of data storage, the method comprising:

reading a configuration file, and acquiring buried point data of a point to be buried carried in the configuration file and first target identification information of a distributed database correspondingly stored in the buried point data;

and storing the buried point data into a distributed database of the first target identification information.

2. The method of claim 1, wherein the configuration file further carries second target identification information of a service system of the to-be-buried point;

the storing the buried point data in the distributed database of the first target identification information comprises:

determining third target identification information of a subject in Kafka corresponding to second identification information according to the corresponding relation between the second target identification information of the service system of the point to be buried, the pre-stored identification information of the service system and the identification information of the subject in Kafka, and writing the buried point data into the subject of the third target identification information in Kafka;

and identifying the buried point data written in the theme of the third target identification information, and storing the identified buried point data into a distributed database of the first target identification information.

3. The method of claim 2, further comprising:

if the buried point data in the distributed database of the first target identification information is lost, identifying the buried point data written in the theme of the third target identification information after the identification condition is met, and storing the identified buried point data in the distributed database of the first target identification information.

4. The method according to claim 2, wherein said identifying said buried point data written in the subject of said third target identification information comprises:

identifying the buried point data written in the subject of the third target identification information based on the Flink; or

And identifying the buried point data written in the theme of the third target identification information based on Flink SQL.

5. The method of claim 2, wherein the configuration file further carries fourth target identification information of a data table;

the storing the buried point data in the distributed database of the first target identification information comprises:

and storing the buried point data into a target data table of the fourth target identification information in a distributed database of the first target identification information according to the fourth target identification information.

6. The method of claim 1, further comprising:

reading the configuration file, and determining whether the configuration file carries information for processing the buried point data;

and if so, performing data analysis on the buried point data.

7. The method of claim 6, wherein the performing data analysis on the buried point data comprises:

performing data analysis on the buried point data based on a Flink filter operator; or

And performing data analysis on the buried point data based on Flink SQL.

8. A data storage device, the device comprising:

the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for reading a configuration file and acquiring buried point data of points to be buried carried in the configuration file and first target identification information of a distributed database, which is correspondingly stored in the buried point data;

and the storage module is used for storing the buried point data into a distributed database of the first target identification information.

9. The apparatus according to claim 8, wherein the storage module is specifically configured to, if the configuration file further carries second target identification information of a service system of a to-be-buried point, determine, according to a correspondence between the second target identification information of the service system of the to-be-buried point, pre-stored identification information of the service system, and identification information of a topic in Kafka, third target identification information of the topic in Kafka corresponding to the second identification information, and write the buried point data into the topic of the third target identification information in Kafka; and identifying the buried point data written in the theme of the third target identification information, and storing the identified buried point data into a distributed database of the first target identification information.

10. The apparatus of claim 9, wherein the storage module is further configured to, if the buried point data in the distributed database of the first target identification information is lost, identify the buried point data written in the subject of the third target identification information after the identification condition is satisfied, and store the identified buried point data in the distributed database of the first target identification information.

11. The apparatus according to claim 9, wherein the storage module is specifically configured to identify the buried point data written in the subject of the third target identification information based on Flink; or identifying the buried point data written in the subject of the third target identification information based on Flink SQL.

12. The apparatus according to claim 9, wherein the storage module is specifically configured to, if the configuration file further carries fourth target identification information of a data table, store the buried point data in a target data table of the fourth target identification information in a distributed database of the first target identification information according to the fourth target identification information.

13. The apparatus of claim 8, further comprising:

the determining module is used for reading the configuration file and determining whether the configuration file carries information for processing the buried point data;

and the analysis module is used for carrying out data analysis on the buried point data if the information for processing the buried point data is carried.

14. The apparatus according to claim 13, wherein the analysis module is specifically configured to perform data analysis on the buried point data based on a Flink filter operator; or performing data analysis on the buried point data based on Flink SQL.

15. An electronic device, characterized in that the electronic device comprises a processor and a memory, the memory being adapted to store program instructions, the processor being adapted to carry out the steps of the data storage method of any of the preceding claims 1-7 when executing a computer program stored in the memory.

16. A computer-readable storage medium, characterized in that it stores a computer program which, when being executed by a processor, carries out the steps of the data storage method according to any one of claims 1 to 7.

17. A computer program product, characterized in that the computer program product comprises: computer program code for causing a computer to perform the steps of the data storage method as claimed in any one of claims 1 to 7 when said computer program code is run on a computer.

Images