CN115033179B - Data storage method, device, equipment and medium - Google Patents
Data storage method, device, equipment and mediumInfo
- Publication number
- CN115033179B CN115033179B CN202210707372.2A CN202210707372A CN115033179B CN 115033179 B CN115033179 B CN 115033179B CN 202210707372 A CN202210707372 A CN 202210707372A CN 115033179 B CN115033179 B CN 115033179B
- Authority
- CN
- China
- Prior art keywords
- identification information
- data
- target identification
- buried
- point data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3034—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a storage system, e.g. DASD based or network based
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to the technical field of data storage, in particular to a data storage method, a device, equipment and a medium, because in the implementation of the application, each embedded point data can be stored into a corresponding distributed database of first target identification information according to the first target identification information of the distributed database corresponding to each embedded point data, on one hand, compared with a relational database, the distributed database is more suitable for storing a large amount of data, the problem that the operation speed of a service system is influenced after the embedded point data reaches a certain scale can not occur, and each embedded point data is stored separately, so that the storage pressure of each distributed database can be effectively reduced; on the other hand, the method does not need a developer to carry out program development, reduces the workload of the developer, and also reduces the requirements on the disk performance and capacity of the service system.
Description
Technical Field
The present application relates to the field of data storage technologies, and in particular, to a data storage method, apparatus, device, and medium.
Background
With the vigorous development of various services, service data is accumulated, and various service system administrators, superior supervision departments and the like provide requirements for collecting and analyzing various indexes such as activity, transaction amount, user behavior analysis and the like of various systems for different service systems, for example, statistics of transaction A and the like which are carried out in the service systems for three years, and particularly, various indexes can be collected and analyzed through buried point data.
In the related technology, the buried data to be analyzed can be stored in a relational database in the service system, and then the service system performs data analysis based on the stored buried data to obtain various indexes after analysis. Because the relational database is not suitable for storing a large amount of data, when the amount of buried point data stored in the relational database reaches a certain scale, the operation speed of the service system is affected, and even other functions of the service system cannot be used normally.
In the related art, buried point data to be analyzed can be written into a log, then the buried point data in the log is loaded into a big data computing environment, and data analysis is carried out in the big data computing environment to obtain various indexes after the analysis is completed.
Disclosure of Invention
The application provides a data storage method, a device, equipment and a medium, which are used for solving the problems that the operation speed of a service system is affected and the workload of a developer is increased by writing buried point data into a log when the buried point data is stored on a large scale based on a relational database in the prior art.
The application provides a data storage method, which comprises the following steps:
Reading a configuration file, and acquiring buried point data of a to-be-buried point carried in the configuration file and first target identification information of a distributed database corresponding to the buried point data;
And storing the buried data into a distributed database of the first target identification information.
Further, the configuration file also carries second target identification information of the service system of the point to be buried, and the storing the buried data into the distributed database of the first target identification information comprises the following steps:
determining third target identification information of a theme in Kafka corresponding to the second identification information according to the second target identification information of the service system of the point to be buried, the corresponding relation between the pre-stored identification information of the service system and the identification information of the theme in Kafka, and writing the buried point data into the theme of the third target identification information in Kafka;
And identifying the buried point data written in the theme of the third target identification information, and storing the identified buried point data into a distributed database of the first target identification information.
Further, the method further comprises:
if the buried point data in the distributed database of the first target identification information is lost, after the identification condition is met, identifying the buried point data written in the theme of the third target identification information, and storing the identified buried point data in the distributed database of the first target identification information.
Further, the identifying the buried data written in the subject of the third target identification information includes:
identifying the buried point data written in the subject of the third target identification information based on the Flink, or
And identifying the buried point data written in the theme of the third target identification information based on the Flink SQL.
Further, the configuration file also carries fourth target identification information of the data table;
the storing the buried data into the distributed database of the first target identification information includes:
And storing the buried data into a target data table of the fourth target identification information in the distributed database of the first target identification information according to the fourth target identification information.
Further, the method further comprises:
Reading the configuration file and determining whether the configuration file carries information for processing the embedded point data or not;
If yes, carrying out data analysis on the buried data.
Further, the performing data analysis on the buried data includes:
based on FLINK FILTER operator, analyzing the buried data, or
And based on the Flink SQL, carrying out data analysis on the buried data.
The application also provides a data storage device, comprising:
the acquisition module is used for reading the configuration file, and acquiring buried point data of the to-be-buried point carried in the configuration file and first target identification information of the distributed database correspondingly stored by the buried point data;
and the storage module is used for storing the buried data into the distributed database of the first target identification information.
Further, the storage module is specifically configured to determine third target identification information of a topic in Kafka corresponding to the second identification information according to a correspondence between the second target identification information of the service system of the point to be buried, pre-stored identification information of the service system and identification information of the topic in Kafka, write the buried point data into the topic of the third target identification information in Kafka, identify the buried point data written into the topic of the third target identification information, and store the identified buried point data into a distributed database of the first target identification information.
Further, the storage module is further configured to identify the buried point data written in the subject of the third target identification information after the identification condition is satisfied if the buried point data in the distributed database of the first target identification information is lost, and store the identified buried point data in the distributed database of the first target identification information.
Further, the storage module is specifically configured to identify the buried point data written in the subject of the third target identification information based on a link, or identify the buried point data written in the subject of the third target identification information based on a link SQL.
Further, the storage module is specifically configured to store, if the configuration file further carries fourth target identification information of a data table, the buried data into a target data table of the fourth target identification information in the distributed database of the first target identification information according to the fourth target identification information.
Further, the apparatus further comprises:
the determining module is used for reading the configuration file and determining whether the configuration file carries information for processing the buried data or not;
And the analysis module is used for carrying information for processing the buried data, and carrying out data analysis on the buried data.
Further, the analysis module is specifically configured to perform data analysis on the buried data based on FLINK FILTER operators, or perform data analysis on the buried data based on flank SQL.
The application also provides an electronic device comprising at least a processor and a memory, the processor being adapted to implement the steps of any of the data storage methods described above when executing a computer program stored in the memory.
The application also provides a computer readable storage medium storing a computer program which when executed by a processor implements the steps of a data storage method as described in any one of the above.
The application also provides a computer program product comprising computer program code to, when run on a computer, cause the computer to perform the steps of any of the data storage methods described above.
In the embodiment of the application, a configuration file is read, buried point data of a point to be buried carried in the configuration file and first target identification information of a distributed database corresponding to the buried point data are obtained, and the buried point data are stored in the distributed database of the first target identification information. In the implementation of the application, each embedded point data can be stored into the corresponding distributed database of the first target identification information according to the first target identification information of the distributed database corresponding to each embedded point data, on one hand, compared with the relational database, the distributed database is more suitable for storing a large amount of data, the problem that the operation speed of a service system is affected after the embedded point data reach a certain scale is avoided, each embedded point data is stored separately, the storage pressure of each distributed database can be effectively reduced, on the other hand, the method does not need a developer to carry out program development, reduces the workload of the developer, and also reduces the requirements on the disk performance and capacity of the service system.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, it will be apparent that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a data storage process according to some embodiments of the present application;
FIG. 2 is a schematic diagram illustrating a process of embedding point data according to some embodiments of the present application;
FIG. 3 is a schematic diagram illustrating a detailed process of embedding point data according to some embodiments of the present application;
FIG. 4 is a schematic diagram of a data storage device according to some embodiments of the present application;
fig. 5 is a schematic structural diagram of an electronic device according to some embodiments of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described in further detail below with reference to the accompanying drawings, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
When the embodiment of the application is used for storing the buried point data, the configuration file is read, the buried point data of the buried point to be stored carried in the configuration file and the first target identification information of the distributed database corresponding to the buried point data are obtained, and the buried point data are stored in the distributed database of the first target identification information.
In order to reduce the workload of developers and avoid the problem that the operation speed of a service system is affected when large-scale buried point data is stored based on a relational database, the embodiment of the application provides a data storage method, a data storage device, data storage equipment and a data storage medium.
Example 1:
FIG. 1 is a schematic diagram of a data storage process according to some embodiments of the present application, the process including the steps of:
s101, reading a configuration file, and obtaining buried point data of points to be buried carried in the configuration file and first target identification information of a distributed database corresponding to the buried point data.
The data storage method provided by the embodiment of the application is applied to the electronic equipment, and the electronic equipment can be an intelligent terminal, a PC or a server and the like.
In the embodiment of the application, the electronic equipment obtains a configuration file in advance, wherein the configuration file carries buried point data of a to-be-buried point and first target identification information of a distributed database corresponding to the buried point data, and in order to determine which buried point data is stored to where, the electronic equipment reads the configuration file to obtain the buried point data and the first target identification information of the distributed database corresponding to the buried point data. The first target identification information of the distributed database corresponding to and stored in different embedded point data may be the same or different. The distributed database may be Hive, HBase, ES, etc., and is not limited in particular.
The configuration file may include a first sub-file for configuring the buried point data, and may further include a second sub-file for configuring identification information of the distributed database to which the buried point data is stored, and in order to determine which buried point data is stored in which distributed database, the electronic device reads the first sub-file, determines the buried point data of the to-be-buried point, reads the second sub-file, and determines the first target identification information of the distributed database to which the buried point data is stored correspondingly. Specifically, since the first subfile may include buried point data of a plurality of to-be-buried points, in order to determine where to store each buried point data, a correspondence between the buried point data and identification information thereof may be stored in the first subfile, and a correspondence between the identification information of the buried point data and identification information of the distributed database may be stored in the second subfile. For each embedded point data, determining target identification information corresponding to the embedded point data based on the corresponding relation between the embedded point data stored in the first sub-file and the identification information thereof, and determining first target identification information of the distributed database corresponding to the embedded point data of the target identification information according to the corresponding relation between the pre-stored identification information of the embedded point data and the identification information of the distributed database in the second sub-file.
S102, storing the buried data into a distributed database of the first target identification information.
After determining the buried data and the first target identification information in the corresponding stored distributed database, the buried data is stored in the distributed database of the first target identification information.
In the implementation of the application, each embedded point data can be stored into the corresponding distributed database of the first target identification information according to the first target identification information of the distributed database corresponding to each embedded point data, on one hand, compared with the relational database, the distributed database is more suitable for storing a large amount of data, the problem that the operation speed of a service system is affected after the embedded point data reach a certain scale is avoided, each embedded point data is stored separately, the storage pressure of each distributed database can be effectively reduced, on the other hand, the embedded point data is not required to be written into a log to synchronize the embedded point data in a big data computing environment, the workload of a developer is reduced, and the requirements on the disk performance and capacity of the service system are also reduced.
Example 2:
In order to store the identified buried point data into the distributed database of the first target identification information, on the basis of the above embodiment, in the embodiment of the present application, the configuration file further carries second target identification information of the service system of the point to be buried;
the storing the buried data into the distributed database of the first target identification information includes:
determining third target identification information of a theme in Kafka corresponding to the second identification information according to the second target identification information of the service system of the point to be buried, the corresponding relation between the pre-stored identification information of the service system and the identification information of the theme in Kafka, and writing the buried point data into the theme of the third target identification information in Kafka;
And identifying the buried point data written in the theme of the third target identification information, and storing the identified buried point data into a distributed database of the first target identification information.
In the embodiment of the application, in order to realize decoupling of the electronic equipment and the distributed database, the electronic equipment can also write the embedded point data into the Kafka, the subsequent electronic equipment identifies the embedded point data in the Kafka and stores the identified embedded point data into the distributed database of the first target identification information, so that when the embedded point data in the subsequent distributed database of the first target identification information is lost, the embedded point data written in the Kafka can be directly identified and stored into the distributed database of the first target identification information again without re-identifying the embedded point data in the configuration file and the first identification information of the distributed database into which the embedded point data is stored.
Wherein Kafka is a high throughput distributed publish-subscribe messaging system, an open source streaming platform developed by the Apache software foundation, which provides fast message persistence capability, maintains long-term stable performance even if data ethernet (TB) is stored, and has high throughput characteristics, and can support millions of messages per second even in very common hardware.
In order to collect and analyze various indexes such as activity, transaction amount, user behavior analysis and the like of each service system, the distributed database in which the buried data is stored can be a database in equipment deployed by the service system.
In order to facilitate the subsequent identification, the embedded point data corresponding to different service systems can be written into different topics of Kafka, and specifically, the configuration file can also carry second target identification information of the service system of the to-be-embedded point, and according to the second target identification information of the service system of the to-be-embedded point and the correspondence between the pre-stored identification information of the service system and the topic (topic) in the Kafka, third target identification information of the topic in the Kafka corresponding to the second target identification information is determined, and the embedded point data is written into the topic of the third target identification information in the Kafka, wherein the embedded point data of the service system corresponding to the different identification information is written into the topic of the different identification information in the Kafka.
In order to store the buried point data into the distributed database of the first target identification information, the electronic device recognizes the buried point data in Kafka and stores the buried point data in Kafka into the distributed database of the first target identification information.
In the related art, in order to collect and analyze various indexes such as activity, transaction amount and user behavior analysis of each system, the embedded point data stored in the relational database can be synchronized into a big data computing environment by means of a change data Capture (CHANGE DATA Capture, CDC) tool, the embedded point data in the relational database is deleted, and data analysis is performed in the big data computing environment, so as to obtain various indexes after analysis, but the method has a large intrusion on a business system, and may cause database abnormality. In the application, the buried data is not required to be synchronized in the big data computing environment for data analysis, so that the data analysis can be performed on the local equipment deployed by the service system, and the risk of intrusion of the service system caused by synchronizing the buried data in the big data computing environment is avoided.
Fig. 2 is a schematic diagram of a process of embedding point data according to some embodiments of the present application, and the description of fig. 2 will be now described.
The electronic equipment reads a first sub-file of the configuration file in advance, determines buried point data stored in the first sub-file, reads a second sub-file in the configuration file, determines first target identification information of the distributed database corresponding to the buried point data, stores the buried point data into the distributed database corresponding to the first target identification information, performs data cleaning processing on the buried point data in equipment deployed in a service system, namely in the equipment where the distributed database is located, and outputs the obtained error-free buried point data to a display page of the service system. The process of performing the data cleaning process on the buried data is the prior art, and will not be described herein.
In order to achieve decoupling of the electronic device and the distributed database, based on the above embodiments, in the embodiments of the present application, the method further includes:
if the buried point data in the distributed database of the first target identification information is lost, after the identification condition is met, identifying the buried point data written in the theme of the third target identification information, and storing the identified buried point data in the distributed database of the first target identification information.
If the embedded data obtained after the configuration file is read is directly stored in the distributed database of the first target identification information, the stored embedded data may be lost in the subsequent distributed database, and when it is determined that the embedded data stored in the distributed database is lost, in order to re-store the embedded data in the distributed database, the embedded data of the configuration file and the embedded data corresponding to the first target identification information stored in the distributed database need to be re-read, and then the embedded data is stored in the distributed database of the first target identification information, so that the electronic device and the distributed database are over-coupled.
In order to decouple the electronic device from the distributed database, since the buried point data is written into the theme of the third target identification information in Kafka in advance and the written theme of the buried point data corresponding to different service systems is known and different, after the identification condition is met, the buried point data written into the theme of the third target identification information can be identified, and the identified buried point data is stored into the distributed database of the first target identification information, so that the buried point data is stored in the distributed database, and the process of re-identifying the buried point data in the configuration file and the first target identification information of the distributed database correspondingly stored by the electronic device is avoided.
The identification condition may be after the buried data is lost for a preset time period or an identification instruction is received.
Example 3:
In order to identify buried point data, on the basis of the above embodiments, in an embodiment of the present application, the identifying the buried point data written in the subject of the third target identification information includes:
identifying the buried point data written in the subject of the third target identification information based on the Flink, or
And identifying the buried point data written in the theme of the third target identification information based on the Flink SQL.
In an embodiment of the present application, in order to identify buried point data written in the subject of the third target identification information in Kafka, in a possible implementation manner, buried point data written in the subject of the third target identification information may be identified based on the link.
In another possible implementation, the buried point data written in the subject of the third target identification information may also be identified based on the flank SQL.
Example 4:
In order to accurately store each embedded point data in a corresponding storage position, in the embodiment of the present application, the configuration file further carries fourth target identification information of the data table on the basis of the above embodiments;
the storing the buried data into the distributed database of the first target identification information includes:
And storing the buried data into a target data table of the fourth target identification information in the distributed database of the first target identification information according to the fourth target identification information.
In the embodiment of the application, a plurality of data tables are pre-stored in each distributed database, and the data tables are used for storing buried point data. In order to determine which data table in the distributed database each embedded data is stored in, the configuration file further carries fourth target identification information of the data table, and specifically, the data table information may be carried in a third sub-file in the configuration file. In order to determine which buried data needs to be stored in the same distributed database and which data table, the third subfile may also store in advance, for each distributed database, the correspondence between the identification information of the buried data and the identification information of the data table in the distributed data, determine, for each buried data, the target identification information corresponding to the buried data based on the correspondence between the buried data stored in the first subfile and the identification information thereof, determine, based on the correspondence between the identification information of the buried data pre-stored in the second subfile and the identification information of the distributed database, the first target identification information of the distributed database to which the buried data corresponding to the target identification information is stored, and finally determine, for each distributed database, the fourth identification information of the data table to be stored corresponding to the buried data based on the correspondence between the identification information of the pre-stored buried data and the identification information of the data table in the distributed data in the third subfile.
After determining fourth target identification information of the data table carried in the third subfile, storing the buried data into a target data table of fourth target identification information in the distributed database of the first target identification information according to the fourth target identification information.
Example 5:
In order to meet the requirement of data analysis on the buried data, on the basis of the above embodiments, in the embodiments of the present application, the method further includes:
Reading the configuration file and determining whether the configuration file carries information for processing the embedded point data or not;
If yes, carrying out data analysis on the buried data.
In the embodiment of the application, some embedded data may have a requirement of data analysis, and in order to determine whether each embedded data has the requirement of data analysis, the configuration file also carries information for data analysis of the embedded data to be subjected to data analysis. Specifically, the fourth subfile in the configuration file may carry information for processing the buried data to be subjected to data analysis. In order to determine which buried data to analyze, the fourth sub-file also stores the correspondence between the identification information of the buried data to be analyzed and the information for processing the same in advance, for each buried data, determining the target identification information corresponding to the buried data based on the correspondence between the buried data stored in the first sub-file and the identification information thereof, determining whether the information for processing the buried data corresponding to the target identification information exists based on the correspondence between the identification information of the buried data to be analyzed and the information for processing the same stored in the fourth sub-file, if not, determining that the data analysis on the buried data is not needed, and if so, determining that the data analysis on the buried data is needed.
When the data analysis is performed on the buried data, if the configuration file carries information for processing the buried data for each buried data, the data analysis is performed on the buried data.
In order to perform data analysis on the buried data, in the embodiments of the present application, the performing data analysis on the buried data includes:
based on FLINK FILTER operator, analyzing the buried data, or
And based on the Flink SQL, carrying out data analysis on the buried data.
In order to perform data analysis on the buried data, in one possible implementation, the buried data may be subjected to data analysis based on FLINK FILTER operators.
In another possible implementation, the buried data may also be analyzed based on the flank SQL.
The Flink SQL is an open-source distributed data analysis engine based on the Flink platform, and provides high-throughput, low-delay and high-performance data analysis capability based on a Yarn/distributed computing framework (Mesos) and other platforms. The method provides a data analysis based on a SQL-like manner, and provides two data analysis modes, namely batch and real-time.
Fig. 3 is a detailed process schematic diagram of a buried point data buried point according to some embodiments of the present application, and will now be described with reference to fig. 3.
Firstly, obtaining a configuration file which is configured in advance, reading the configuration file by the electronic equipment, determining whether a first sub-file in the configuration file exists or not, if not, outputting prompt information of the first sub-file which is not found, if so, writing buried point data of a point to be buried carried in the first sub-file into Kafka, and identifying the buried point data written in the Kafka based on the Flink.
And the electronic equipment determines whether a second sub-file in the configuration file exists, if not, the prompting information of the second sub-file is output, if so, the first target identification information of the distributed database corresponding to the buried point data is determined according to the second sub-file for each buried point data, the fourth target identification information of the data table corresponding to the buried point data carried in the third sub-file in the configuration file is determined for each buried point data, and the buried point data is stored in the target data table of the fourth target identification information in the distributed database of the first target identification information. And determining whether a fourth subfile in the configuration file exists, if so, determining whether information for processing the buried data is carried in the fourth subfile for each buried data, if so, carrying out data analysis on the buried data to be subjected to data analysis based on the Flink SQL, and storing a result obtained after the data analysis.
And after receiving the query instruction, the follow-up service system outputs corresponding result data according to the information to be queried carried in the query instruction.
Example 6:
FIG. 4 is a schematic diagram of a data storage device according to some embodiments of the present application, the device includes:
an obtaining module 401, configured to read a configuration file, and obtain buried point data of a to-be-buried point carried in the configuration file and first target identification information of a distributed database corresponding to the buried point data;
a storage module 402, configured to store the buried data into the distributed database of the first target identification information.
In a possible implementation manner, the storage module 402 is specifically configured to determine, if the configuration file further carries second target identification information of a service system of a point to be buried, third target identification information of a topic in Kafka corresponding to the second identification information according to a correspondence between the second target identification information of the service system of the point to be buried, pre-stored identification information of the service system and identification information of the topic in Kafka, write the buried point data into the topic of the third target identification information in Kafka, identify the buried point data written into the topic of the third target identification information, and store the identified buried point data into a distributed database of the first target identification information.
In a possible implementation manner, the storage module 402 is further configured to identify, if the buried point data in the distributed database of the first target identification information is lost, the buried point data written in the subject of the third target identification information after the identification condition is satisfied, and store the identified buried point data in the distributed database of the first target identification information.
In a possible implementation manner, the storage module 402 is specifically configured to identify, based on the flank, the buried point data written in the subject of the third target identification information, or identify, based on the flank SQL, the buried point data written in the subject of the third target identification information.
In a possible implementation manner, the storage module 402 is specifically configured to store, if the configuration file further carries fourth target identification information of a data table, the buried data into a target data table of the fourth target identification information in the distributed database of the first target identification information according to the fourth target identification information.
In one possible embodiment, the apparatus further comprises:
a determining module 403, configured to read the configuration file, and determine whether the configuration file carries information for processing the embedded point data;
and the analysis module 404 is configured to perform data analysis on the buried data if the information for processing the buried data is carried.
In a possible implementation manner, the analysis module 404 is specifically configured to perform data analysis on the buried data based on FLINK FILTER operators, or perform data analysis on the buried data based on flank SQL.
Example 7:
Based on the above embodiments, some embodiments of the present application further provide an electronic device, as shown in fig. 5, including a processor 501, a communication interface 502, a memory 503, and a communication bus 504, where the processor 501, the communication interface 502, and the memory 503 complete communication with each other through the communication bus 504.
The memory 503 has stored therein a computer program which, when executed by the processor 501, causes the processor 501 to perform the steps of:
Reading a configuration file, and acquiring buried point data of a to-be-buried point carried in the configuration file and first target identification information of a distributed database corresponding to the buried point data;
And storing the buried data into a distributed database of the first target identification information.
Further, the processor 501 is specifically configured to determine, if the configuration file further carries second target identification information of a service system of a point to be buried, third target identification information of a topic in Kafka corresponding to the second identification information according to a correspondence between the second target identification information of the service system of the point to be buried, pre-stored identification information of the service system and identification information of the topic in Kafka, write the buried point data into the topic of the third target identification information in Kafka, identify the buried point data written into the topic of the third target identification information, and store the identified buried point data into a distributed database of the first target identification information.
Further, the processor 501 is further configured to identify the buried point data written in the subject of the third target identification information after the identification condition is satisfied if the buried point data in the distributed database of the first target identification information is lost, and store the identified buried point data in the distributed database of the first target identification information.
Further, the processor 501 is specifically configured to identify the buried point data written in the subject of the third target identification information based on a Flink, or identify the buried point data written in the subject of the third target identification information based on a Flink SQL.
Further, the processor 501 is specifically configured to store, if the configuration file further carries fourth target identification information of a data table, the buried data into a target data table of the fourth target identification information in the distributed database of the first target identification information according to the fourth target identification information.
Further, the processor 501 is further configured to read the configuration file, determine whether the configuration file carries information for processing the embedded data, and if so, perform data analysis on the embedded data.
Further, the processor 501 is specifically configured to perform data analysis on the buried data based on FLINK FILTER operators, or perform data analysis on the buried data based on flank SQL.
The communication bus mentioned by the server may be a peripheral component interconnect standard (PERIPHERAL COMPONENT INTERCONNECT, PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, etc. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.
The communication interface 502 is used for communication between the electronic device and other devices described above.
The Memory may include random access Memory (Random Access Memory, RAM) or may include Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.
The processor may be a general-purpose processor including a central Processing unit (cpu), a network processor (Network Processor, NP), etc., or may be a digital instruction processor (DIGITAL SIGNAL Processing, DSP), an application specific integrated circuit (asic), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.
Example 8:
On the basis of the above embodiments, some embodiments of the present application further provide a computer readable storage medium having stored therein a computer program executable by an electronic device, which when run on the electronic device, causes the electronic device to perform the steps of:
the memory has stored therein a computer program which, when executed by the processor, causes the processor to perform the steps of:
Reading a configuration file, and acquiring buried point data of a to-be-buried point carried in the configuration file and first target identification information of a distributed database corresponding to the buried point data;
And storing the buried data into a distributed database of the first target identification information.
Further, the configuration file also carries second target identification information of the service system of the point to be buried;
the storing the buried data into the distributed database of the first target identification information includes:
determining third target identification information of a theme in Kafka corresponding to the second identification information according to the second target identification information of the service system of the point to be buried, the corresponding relation between the pre-stored identification information of the service system and the identification information of the theme in Kafka, and writing the buried point data into the theme of the third target identification information in Kafka;
And identifying the buried point data written in the theme of the third target identification information, and storing the identified buried point data into a distributed database of the first target identification information.
Further, the method further comprises:
if the buried point data in the distributed database of the first target identification information is lost, after the identification condition is met, identifying the buried point data written in the theme of the third target identification information, and storing the identified buried point data in the distributed database of the first target identification information.
Further, the identifying the buried data written in the subject of the third target identification information includes:
identifying the buried point data written in the subject of the third target identification information based on the Flink, or
And identifying the buried point data written in the theme of the third target identification information based on the Flink SQL.
Further, the configuration file also carries fourth target identification information of the data table;
the storing the buried data into the distributed database of the first target identification information includes:
And storing the buried data into a target data table of the fourth target identification information in the distributed database of the first target identification information according to the fourth target identification information.
Further, the method further comprises:
Reading the configuration file and determining whether the configuration file carries information for processing the embedded point data or not;
If yes, carrying out data analysis on the buried data.
Further, the performing data analysis on the buried data includes:
based on FLINK FILTER operator, analyzing the buried data, or
And based on the Flink SQL, carrying out data analysis on the buried data.
Example 9:
The embodiment of the application also provides a computer program product, which when being executed by a computer, realizes the data storage method of any method embodiment applied to electronic equipment.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof, and may be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions which, when loaded and executed on a computer, produce, in whole or in part, a process or function in accordance with embodiments of the present application.
In the implementation of the application, each embedded point data can be stored into the corresponding distributed database of the first target identification information according to the first target identification information of the distributed database corresponding to each embedded point data, on one hand, compared with the relational database, the distributed database is more suitable for storing a large amount of data, the problem that the operation speed of a service system is affected after the embedded point data reach a certain scale is avoided, each embedded point data is stored separately, the storage pressure of each distributed database can be effectively reduced, on the other hand, the embedded point data is not required to be written into a log to synchronize the embedded point data in a big data computing environment, the workload of a developer is reduced, and the requirements on the disk performance and capacity of the service system are also reduced.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (15)
1. A method of data storage, the method comprising:
Reading a configuration file, and acquiring buried point data of a to-be-buried point carried in the configuration file and first target identification information of a distributed database corresponding to the buried point data;
Storing the buried data into a distributed database of the first target identification information, wherein the distributed database is a database in equipment deployed by a service system;
The configuration file comprises a first subfile for configuring the embedded point data and a second subfile for configuring the identification information of the distributed database to which the embedded point data is stored;
The obtaining buried point data of the to-be-buried point carried in the configuration file and first target identification information of a distributed database corresponding to and stored by the buried point data includes:
Determining target identification information corresponding to the buried point data based on the corresponding relation between the buried point data stored in the first subfile and the identification information of the buried point data, and determining first target identification information of the distributed database corresponding to the buried point data of the target identification information according to the corresponding relation between the pre-stored identification information of the buried point data and the identification information of the distributed database in the second subfile;
the configuration file also carries second target identification information of a service system of the point to be buried;
the storing the buried data into the distributed database of the first target identification information includes:
Determining third target identification information of a theme in Kafka corresponding to the second target identification information according to the second target identification information of the service system of the point to be buried, the corresponding relation between the pre-stored identification information of the service system and the identification information of the theme in Kafka, and writing the buried point data into the theme of the third target identification information in Kafka;
And identifying the buried point data written in the theme of the third target identification information, and storing the identified buried point data into a distributed database of the first target identification information.
2. The method according to claim 1, wherein the method further comprises:
if the buried point data in the distributed database of the first target identification information is lost, after the identification condition is met, identifying the buried point data written in the theme of the third target identification information, and storing the identified buried point data in the distributed database of the first target identification information.
3. The method of claim 1, wherein the identifying the buried data written in the subject of the third target identification information comprises:
identifying the buried point data written in the subject of the third target identification information based on the Flink, or
And identifying the buried point data written in the theme of the third target identification information based on the Flink SQL.
4. The method of claim 1, wherein the configuration file further carries fourth target identification information of a data table;
the storing the buried data into the distributed database of the first target identification information includes:
And storing the buried data into a target data table of the fourth target identification information in the distributed database of the first target identification information according to the fourth target identification information.
5. The method according to claim 1, wherein the method further comprises:
Reading the configuration file and determining whether the configuration file carries information for processing the embedded point data or not;
If yes, carrying out data analysis on the buried data.
6. The method of claim 5, wherein the data analysis of the buried data comprises:
based on FLINK FILTER operator, analyzing the buried data, or
And based on the Flink SQL, carrying out data analysis on the buried data.
7. A data storage device, the device comprising:
the acquisition module is used for reading a configuration file, and acquiring buried point data of a to-be-buried point carried in the configuration file and first target identification information of a distributed database corresponding to the buried point data, wherein the configuration file comprises a first sub-file for configuring the buried point data and a second sub-file for configuring the identification information of the distributed database corresponding to the buried point data;
The storage module is used for storing the buried data into a distributed database of the first target identification information, wherein the distributed database is a database in equipment deployed by a service system;
the acquisition module is specifically configured to determine, for each piece of embedded point data, target identification information corresponding to the embedded point data based on a correspondence between the embedded point data stored in the first subfile and identification information thereof, and determine, in the second subfile, first target identification information of the distributed database to which the embedded point data of the target identification information corresponds is stored according to a correspondence between the identification information of the embedded point data stored in advance and the identification information of the distributed database;
The storage module is specifically configured to determine third target identification information of a topic in Kafka corresponding to the second target identification information according to a corresponding relationship between the second target identification information of the service system of the point to be buried, pre-stored identification information of the service system and identification information of the topic in Kafka, write the buried point data into the topic of the third target identification information in Kafka, identify the buried point data written into the topic of the third target identification information, and store the identified buried point data into a distributed database of the first target identification information.
8. The apparatus of claim 7, wherein the storage module is further configured to identify the buried point data written in the subject of the third target identification information after the identification condition is satisfied if the buried point data in the distributed database of the first target identification information is lost, and store the identified buried point data in the distributed database of the first target identification information.
9. The apparatus according to claim 7, wherein the storage module is specifically configured to identify the buried point data written in the subject of the third target identification information based on a link, or identify the buried point data written in the subject of the third target identification information based on a link SQL.
10. The apparatus of claim 7, wherein the storage module is specifically configured to store the embedded data into a target data table of the fourth target identification information in the distributed database of the first target identification information according to the fourth target identification information if the configuration file further carries the fourth target identification information of the data table.
11. The apparatus of claim 7, wherein the apparatus further comprises:
the determining module is used for reading the configuration file and determining whether the configuration file carries information for processing the buried data or not;
And the analysis module is used for carrying information for processing the buried data, and carrying out data analysis on the buried data.
12. The apparatus of claim 11, wherein the analysis module is specifically configured to perform data analysis on the buried data based on FLINK FILTER operators, or perform data analysis on the buried data based on Flink SQL.
13. An electronic device comprising a processor and a memory for storing program instructions, the processor being adapted to implement the steps of the data storage method of any of the preceding claims 1-6 when executing a computer program stored in the memory.
14. A computer-readable storage medium, characterized in that it stores a computer program which, when executed by a processor, implements the steps of the data storage method according to any of claims 1-6.
15. Computer program product, characterized in that it comprises computer program code means for causing a computer to carry out the steps of the data storage method according to any one of claims 1-6 when said computer program code means are run on the computer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210707372.2A CN115033179B (en) | 2022-06-21 | 2022-06-21 | Data storage method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210707372.2A CN115033179B (en) | 2022-06-21 | 2022-06-21 | Data storage method, device, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115033179A CN115033179A (en) | 2022-09-09 |
| CN115033179B true CN115033179B (en) | 2025-08-26 |
Family
ID=83125179
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210707372.2A Active CN115033179B (en) | 2022-06-21 | 2022-06-21 | Data storage method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115033179B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116319979A (en) * | 2023-02-21 | 2023-06-23 | 深圳市海邻科信息技术有限公司 | Data collection method, device, equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110502591A (en) * | 2019-08-27 | 2019-11-26 | 北京思维造物信息科技股份有限公司 | A kind of data extraction method, device and equipment |
| CN112506960A (en) * | 2020-12-17 | 2021-03-16 | 青岛以萨数据技术有限公司 | Multi-model data storage method and system based on ArangoDB engine |
| CN114186000A (en) * | 2021-12-14 | 2022-03-15 | 上海识装信息科技有限公司 | A Flink-based business data monitoring method, device and storage medium |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108804644B (en) * | 2018-06-05 | 2023-06-30 | 中国平安人寿保险股份有限公司 | Interface log storage method, device, computer equipment and storage medium |
| CN109656981B (en) * | 2018-11-19 | 2023-09-15 | 南京国通智能科技有限公司 | Data statistics method and system |
| CN110570097B (en) * | 2019-08-15 | 2024-09-03 | 中国平安人寿保险股份有限公司 | Business personnel risk identification method and device based on big data and storage medium |
| CN110647512B (en) * | 2019-09-29 | 2022-05-24 | 北京思维造物信息科技股份有限公司 | Data storage and analysis method, device, equipment and readable medium |
| CN112069384A (en) * | 2020-09-04 | 2020-12-11 | 中国平安人寿保险股份有限公司 | Buried point data processing method, server and readable storage medium |
| CN113204464B (en) * | 2021-04-23 | 2023-04-25 | 商盟商务服务有限公司 | Real-time business monitoring method, system, terminal and medium based on business scenarios |
| CN113918651A (en) * | 2021-10-13 | 2022-01-11 | 广东利通科技投资有限公司 | Business data and capital flow processing method, device, equipment and medium |
| CN114257521B (en) * | 2021-12-17 | 2024-05-17 | 北京沃东天骏信息技术有限公司 | Traffic prediction method, traffic prediction device, electronic equipment and storage medium |
| CN114567650B (en) * | 2021-12-29 | 2022-12-27 | 西安天和防务技术股份有限公司 | Data processing method and Internet of things platform system |
| CN114153620B (en) * | 2022-02-08 | 2022-05-24 | 上海柯林布瑞信息技术有限公司 | Optimal allocation method and device for Hudi operating environment resources |
| CN114546780A (en) * | 2022-02-25 | 2022-05-27 | 携程计算机技术（上海）有限公司 | Data monitoring method, device, equipment, system and storage medium |
-
2022
- 2022-06-21 CN CN202210707372.2A patent/CN115033179B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110502591A (en) * | 2019-08-27 | 2019-11-26 | 北京思维造物信息科技股份有限公司 | A kind of data extraction method, device and equipment |
| CN112506960A (en) * | 2020-12-17 | 2021-03-16 | 青岛以萨数据技术有限公司 | Multi-model data storage method and system based on ArangoDB engine |
| CN114186000A (en) * | 2021-12-14 | 2022-03-15 | 上海识装信息科技有限公司 | A Flink-based business data monitoring method, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115033179A (en) | 2022-09-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111209352B (en) | Data processing method and device, electronic equipment and storage medium | |
| US20180365085A1 (en) | Method and apparatus for monitoring client applications | |
| CN113360554B (en) | Method and equipment for extracting, converting and loading ETL (extract transform load) data | |
| CN110515912A (en) | Log processing method, device, computer installation and computer readable storage medium | |
| CN111177237B (en) | Data processing system, method and device | |
| CN110704484A (en) | Method and system for processing mass real-time data stream | |
| CN112087320B (en) | Abnormality positioning method and device, electronic equipment and readable storage medium | |
| CN113220530B (en) | Data quality monitoring method and platform | |
| CN112235262A (en) | Message analysis method and device, electronic equipment and computer readable storage medium | |
| CN112860527A (en) | Fault monitoring method and device of application server | |
| CN115033179B (en) | Data storage method, device, equipment and medium | |
| CN110688383A (en) | Data acquisition method and system | |
| CN111984444A (en) | A kind of abnormal information processing method and device | |
| CN111291127B (en) | Data synchronization method, device, server and storage medium | |
| CN111459931A (en) | Data duplication checking method and data duplication checking device | |
| CN112988776A (en) | Method, device and equipment for updating text parsing rule and readable storage medium | |
| CN112445869A (en) | Data quality inspection method and device | |
| CN110968518A (en) | Analysis method and device for automatic test log file | |
| CN114936113B (en) | Task avalanche recovery method and device, electronic equipment and storage medium | |
| CN113238901B (en) | Multi-device automatic testing method and device, storage medium and computer device | |
| WO2017190469A1 (en) | Data optimisation method and apparatus in big data processing | |
| CN112115046B (en) | Software fault positioning method, device and terminal | |
| CN115794783A (en) | Data deduplication method, device, equipment and medium | |
| CN114077532A (en) | Method and device for detecting SQL statement execution efficiency | |
| CN111831754A (en) | Method, device, system and medium for copying data in database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |