CN111078738B - Data processing method, device, electronic equipment and storage medium - Google Patents
Data processing method, device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111078738B CN111078738B CN201911165369.7A CN201911165369A CN111078738B CN 111078738 B CN111078738 B CN 111078738B CN 201911165369 A CN201911165369 A CN 201911165369A CN 111078738 B CN111078738 B CN 111078738B
- Authority
- CN
- China
- Prior art keywords
- data
- comparison
- detail
- data source
- files
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Physics & Mathematics (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Development Economics (AREA)
- General Business, Economics & Management (AREA)
- Technology Law (AREA)
- Computing Systems (AREA)
- Strategic Management (AREA)
- Marketing (AREA)
- Economics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data processing method, a device, an electronic device and a storage medium, wherein the method comprises the following steps: reading data of each data source, extracting comparison data of each piece of data of each data source, and detail data corresponding to the comparison data, wherein the comparison data are used for representing key fields of each piece of data, the detail data are used for representing the detail information of the key fields represented by the comparison data, and the detail information of the key fields are used for representing detailed fields and storage positions corresponding to the key fields; comparing the comparison data in the data sources, and if different difference comparison data of the key fields exist, determining the detail information of the key fields represented by the difference comparison data according to the detail data corresponding to the difference comparison data; and displaying the detail information of the key field of the difference comparison data characterization. The data processing method can improve the data processing efficiency.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data processing method, a data processing device, an electronic device, and a storage medium.
Background
The internet financial transaction is characterized in that the amount of each order is small and the order quantity is large, so that the data processing system adopts a distributed architecture to improve the data processing efficiency. Compared with a traditional single architecture, the data in the scene is more circulated and interacted. For example, the data processing system of internet financial risk transaction not only comprises service subsystems such as underwriting, wind control and finance, but also comprises middleware such as cache and message queue, each node may cause data inconsistency among the nodes due to program problems, hardware/network faults and other problems, so that problems such as risk out of control and inconsistent business and financial data occur. These problems not only cause financial losses but also introduce significant verification, analysis, and processing effort. Therefore, it is necessary to automatically check the data of each node (data source) periodically.
The prior art checks the data in each data source in the following way: the data stored in each data source is all written into a common comparison library where the difference data is queried and derived using a table join (join) approach. This approach is basically feasible with smaller amounts of data, but as the amount of data stored by each data source increases, the data processing efficiency drops dramatically.
Disclosure of Invention
The application provides a data processing method, a data processing device, electronic equipment and a storage medium, which can improve data processing efficiency.
A first aspect of the present application provides a data processing method comprising:
reading data of each data source, extracting comparison data of each piece of data of each data source, and detail data corresponding to the comparison data, wherein the comparison data are used for representing key fields of each piece of data, the detail data are used for representing the detail information of the key fields represented by the comparison data, and the detail information of the key fields are used for representing detailed fields and storage positions corresponding to the key fields;
comparing the comparison data in the data sources, and if different difference comparison data of the key fields exist, determining the detail information of the key fields represented by the difference comparison data according to the detail data corresponding to the difference comparison data;
and displaying the detail information of the key field of the difference comparison data characterization.
Optionally, the comparison data is a comparison file, the detail data is a detail file, and after extracting the comparison data of each piece of data of each data source and the detail data corresponding to the comparison data, the method further includes:
And writing the comparison file of each piece of data of each data source and the detail file corresponding to the comparison file into a storage database.
Optionally, the writing the comparison file of each piece of data of each data source and the detail file corresponding to the comparison file into the storage database includes:
reading the comparison file of each piece of data in each data source, and caching the read comparison file of the data and the detail file corresponding to the read comparison file of the data into a cache database;
judging whether the comparison files of N data of each data source are cached in the cache database or not, and detail files corresponding to the comparison files of each data source;
if the comparison files of the N data of each data source and the detail files corresponding to the comparison files of each data source are cached in the cache database, writing the comparison files of the N data of each data source and the detail files corresponding to the comparison files of the N data of each data source into the storage database, and clearing the cache database;
if the comparison files of the N data of each data source and the detail files corresponding to the comparison files of each data source are not cached in the cache database, continuing to read the data in each data source until the comparison files of the N data of each data source and the detail files corresponding to the comparison files of the N data of each data source are cached in the cache database.
Optionally, after the comparing the comparison data in each data source, the method further includes:
reading key fields of comparison files of N data of each data source in the storage database, and caching the key fields of the N data of each data source into the cache database;
and if the key fields of the N data of each data source recorded in the cache database are compared, determining a difference key field with a difference in each data source, and determining a comparison file to which the difference key field in each data source belongs as the difference comparison file.
Optionally, the method further comprises:
displaying a data link, wherein the data link is used for representing identification information, difference generation reasons and data processing suggestions of a data source of a key field of the difference comparison data representation.
Optionally, the method further comprises:
and if the processing flow for processing the difference comparison data is determined to be stored according to the detailed information of the key field of the difference comparison data characterization, processing the difference comparison data by adopting the processing flow.
Optionally, the data in each data source is policy data, and the key field is a policy number or a serial number.
A second aspect of the present application provides a data processing apparatus comprising:
the reading module is used for reading the data of each data source;
the processing module is used for extracting comparison data of each piece of data of each data source and detail data corresponding to the comparison data, comparing the comparison data in each data source, and determining detail information of a key field represented by the difference comparison data according to the detail data corresponding to the difference comparison data if different difference comparison data of the key field exist, wherein the comparison data is used for representing the key field of each piece of data, the detail data is used for representing the detail information of the key field represented by the comparison data, and the detail information of the key field is used for representing the detailed field and a storage position corresponding to the key field;
and the display module is used for displaying the detailed information of the key fields of the differential comparison data representation.
Optionally, the comparison data is a comparison file, and the detail data is a detail file.
And the writing module is used for writing the comparison file of each piece of data of each data source and the detail file corresponding to the comparison file into the storage database.
Optionally, the writing module is specifically configured to read a comparison file of each piece of data in each data source, cache the comparison file of the read data and a detail file corresponding to the comparison file of the read data to a cache database, and determine whether the comparison file of N pieces of data of each data source and the detail file corresponding to the comparison file of each data source are cached in the cache database; if yes, writing the comparison files of the N data of each data source and the detail files corresponding to the comparison files of the N data of each data source into the storage database, and emptying the cache database; if not, continuing to read the data in each data source until the comparison file of the N data of each data source and the detail file corresponding to the comparison file of the N data of each data source are cached in the cache database.
Optionally, the reading module is further configured to read key fields of the comparison file of N pieces of data of each data source in the storage database, and cache the key fields of the N pieces of data of each data source to the cache database;
If the key fields of the N pieces of data of each data source recorded in the cache database are compared, determining a difference key field of each data source, wherein the difference key field is different, the processing module is further configured to determine a comparison file to which the difference key field in each data source belongs as the difference comparison file.
Optionally, the display module is further configured to display a data link, where the data link is used to characterize identification information of a data source of a key field of the difference comparison data characterization, a difference generation reason, and a data processing suggestion.
Optionally, the processing module is further configured to, if the processing flow for processing the difference comparison data is determined to be stored according to the detailed information of the key field represented by the difference comparison data, process the difference comparison data by using the processing flow.
Optionally, the reading module is further configured to use the data in each data source as policy data, and the key field is a policy number or a serial number.
A third aspect of the present application provides an electronic apparatus comprising: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executes computer-executable instructions stored in the memory to cause the electronic device to perform the data processing method of the first aspect described above.
A fourth aspect of the present application provides a computer-readable storage medium having stored thereon computer-executable instructions which, when executed by a processor, implement the data processing method of the first aspect described above.
The application provides a data processing method, a device, electronic equipment and a storage medium, wherein the data processing method avoids the processing mode of checking and comparing all data in data sources after being copied in the prior art, and extracts comparison data of each piece of data of each data source and detail data corresponding to the comparison data. Because the comparison data are used for representing the key field of each piece of data, key information of the data can be embodied, and further, the comparison data with the difference are determined according to the difference of the comparison data of each piece of data of each data source. By acquiring the detail information corresponding to the comparison data with the difference according to the detail data corresponding to the comparison data with the difference, the data in each data source with the difference can be quickly acquired, so that the data processing efficiency is improved.
Drawings
FIG. 1 is a flow chart of a prior art data processing method;
FIG. 2 is a schematic diagram of a scenario where the data processing method provided by the present application is applicable;
FIG. 3 is a flow chart of a data processing method according to the present application;
FIG. 4 is a schematic diagram of interface variation provided by the present application;
FIG. 5 is a second flow chart of the data processing method according to the present application;
FIG. 6 is a flow chart of writing files in the prior art;
FIG. 7 is a schematic flow chart of writing files provided in the present application;
FIG. 8 is a flow chart of determining details of key fields of a differential alignment file characterization in the prior art;
FIG. 9 is a schematic flow chart of the detail information of key fields of the determined difference comparison file characterization provided in the present application;
FIG. 10 is a flow chart of a preservation method in the prior art;
FIG. 11 is a flow chart of a preservation method according to the present application;
FIG. 12 is a schematic diagram of a data processing apparatus according to the present application;
fig. 13 is a schematic structural diagram of an electronic device provided by the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions in the embodiments of the present application will be clearly and completely described in the following in conjunction with the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
For a clearer description of the data processing method provided by the present application, a description will be given first of all to a data processing method in the prior art.
In a data processing system for internet financial transactions, assuming that there are N data sources, it is now necessary to process the data in the N data sources. The data source can be a service subsystem of underwriting, wind control, finance and the like, or middleware of caching, message queue and the like. The data in the N data sources should be consistent, but due to the problems of program problems, hardware/network faults and the like, the data among the data sources are inconsistent, so that the problems of risk runaway, inconsistent business data and the like occur. Therefore, the data in the N data sources need to be periodically checked and compared to acquire inconsistent data, so as to further analyze the reasons for the inconsistent data and solve the problem.
Fig. 1 is a flow chart of a data processing method in the prior art. As shown in fig. 1, in the prior art, when checking and comparing data in N data sources, the following procedure needs to be performed:
s101, starting data checking.
S102, circularly writing the data in each data source into a common comparison library.
For example, data in data source 1, data source 2, data source 3, … …, data source N may be cyclically written to a common alignment library. The process of cyclic writing is adopted here, for example, taking the data source 1 as an example, m pieces of data in the data source 1 can be read and written into the common comparison library, and then m pieces of data in the data source 1 are read and written into the common comparison library until the data in the data source 1 is completely written into the common comparison library.
S103, inquiring the difference information in a public comparison library by using a table connection mode.
The specific principle and mode of inquiring the difference information for the multi-table connection mode of the database in the application are not repeated. It should be understood that when querying the difference information in a table connection manner, the data in each data source needs to be compared line by line to obtain the difference information.
And S104, storing the difference information into a storage database.
S105, displaying the difference information and giving an alarm.
In the prior art, after the difference information is obtained, the difference information can be displayed and an alarm is given, and the alarm mode is not repeated in the application.
S106, ending.
In the prior art, when checking and comparing the data in each data source, the data in each data source needs to be written into a common comparison library, and the data in each data source needs to be compared line by line, which requires a great deal of time. This approach is basically feasible with smaller amounts of data, but as the amount of data stored by each data source increases, the data processing efficiency drops dramatically.
The improvement of the prior art is that: and dividing the data which can be stored in each data source into data sets according to a certain rule by adopting multithreading, and then comparing the data sets at the same time. This improved method can effectively improve efficiency, but can put a great load on the database.
In order to solve the problems in the prior art, the application provides a data processing method, which extracts comparison data of each piece of data of each data source and detail data corresponding to the comparison data. Because the comparison data are used for representing the key fields of each piece of data, key information of the data can be embodied, and further, the comparison data with the difference are determined according to the difference of the comparison data of each piece of data of each data source, and compared with the processing mode in the prior art, the comparison time can be shortened. According to the detail data corresponding to the comparison data with the difference, the detail information corresponding to the comparison data with the difference is obtained, so that the data with the difference in each data source can be obtained rapidly, and the aim of improving the data processing efficiency is achieved.
Fig. 2 is a schematic diagram of a scenario where the data processing method provided by the present application is applicable. As shown in fig. 2, the scenario for which the data processing method provided by the present application is applicable includes: at least two data sources and a data processing device.
According to the application, the data sources are different according to different scenes applied by the data processing method. For example, in a data processing system for internet financial transactions, the data source may be a underwriting, wind control, financial, etc. business subsystem, or may be middleware such as a cache, message queue, etc. As in the data processing system of an educational network, the data sources may be data storage servers, backup servers, and the like. Alternatively, the data source may be a server, database, or the like for storing data.
The data processing device can process the data in each data source to obtain data with differences. The data processing device may be an electronic device having a processing function, such as a server or a terminal device. It should be appreciated that the data sources are illustrated in fig. 2 as databases.
The data processing method provided by the application is described below with reference to specific embodiments. Fig. 3 is a schematic flow chart of a data processing method according to the present application. The subject of the execution of the method flow illustrated in fig. 2 may be a data processing device, which may be implemented by any software and/or hardware. As shown in fig. 3, the data processing method provided in this embodiment may include:
S301, reading data of each data source, extracting comparison data of each piece of data of each data source and detail data corresponding to the comparison data, wherein the comparison data is used for representing key fields of each piece of data, the detail data is used for representing detail information of the key fields represented by the comparison data, and the detail information of the key fields is used for representing detailed fields and storage positions corresponding to the key fields.
In this embodiment, the data processing apparatus may extract the comparison data of each piece of data in each data source and the detail data corresponding to the comparison data when reading the data of each data source. It should be appreciated that a plurality of pieces of data are stored in each data source, wherein each piece of data may be data stored to that data source in a time sequence. Illustratively, if a piece of data is stored at time A as in data source 1, then the piece of data is a piece of data.
Wherein the alignment data is used to characterize key fields of each piece of data. Alternatively, the alignment data may be key fields of each piece of data.
One possible implementation manner of the data processing apparatus extracting the comparison data of each piece of data in each data source in this embodiment may be: and extracting comparison data in the data according to a preset extraction template. Optionally, the data in each data source is policy data, and the key field may be a policy number or a serial number. The policy data is data stored according to a fixed format, in this embodiment, a row of a key field (such as a row to which a policy number or a serial number belongs) to be extracted may be preset, and according to the preset extraction template, data corresponding to the preset row is extracted from the data, that is, the comparison data.
Another possible implementation of the data processing apparatus to extract the alignment data for each piece of data in the respective data sources may be: the semantics of the extracted key fields, such as name, insurance amount, insurance policy number and other data in the extracted data, can be preset as the key fields. The alignment data may be extracted for each piece of data based on the semantics in that piece of data.
The detail data are used for representing the detail information of the key fields representing the comparison data, and the detail information of the key fields is used for representing the detailed fields and the storage positions corresponding to the key fields. In this embodiment, when the comparison data of each piece of data is extracted, the detail data corresponding to the comparison data may be extracted at the same time. The detailed information of the key field may include a detailed field and a storage location corresponding to the key field. Wherein, the detailed field corresponding to the key field can be used for characterizing other fields related to the key field and not embodied in the key field. The storage location of the key field may be a storage space or a buffer area in a data source storing the key field, etc.
Illustratively, for example, an insurance order, the key fields therein may be: an insurance policy number and an amount of insurance applied. The detailed field corresponding to the key field may be the application year, beneficiary, etc., and the storage location may be a C folder in a B data disc in the data source 1. It should be understood that the detailed field corresponding to the key field may be preset.
Alternatively, the performing step in S301 in this embodiment may be triggered by the user, may be performed periodically, or may be performed when the data amounts in the data sources are all greater than the data amount threshold.
When the executing step in S301 is user trigger, a trigger control, such as a "compare" control, may be displayed on the display interface of the data processing apparatus. When the user selects the "align" control, either by clicking or other manipulation, the execution step in S301 may be triggered. Fig. 4 is a schematic diagram of interface change provided by the present application. As shown by interface 401 in fig. 4, the display interface of the data processing apparatus has a "compare" control displayed thereon. Alternatively, the comparison period may be set in advance in this embodiment so that the data processing apparatus periodically executes the execution step in S301 described above. Alternatively, the data processing apparatus executes the execution step in S301 when detecting that the data amount in each data source is greater than the data amount threshold.
Optionally, in this embodiment, if the data amount in each data source is larger, the data in each data source may be partitioned according to time sequence, so as to obtain multiple data areas in each data source, extract comparison data and detail data of the data in each data area, and compare the comparison data with the detail data,
Exemplary, the data after partitioning the data in data source 1 and data source 2 is shown in the following table one:
list one
| Data source 1 | Data source 2 | |
| Data area 1 | Data A, data B | Data A, data B' |
| Data area 2 | Data C, data D | Data C ', data D' |
| Data area 3 | Data E, data F | Data E'Data F |
As shown in the above table, in this embodiment, the comparison data and the detail data of the data area 1 of the data source 1 and the data source 2 can be extracted.
Optionally, in this embodiment, each piece of comparison data corresponds to a data identifier, such as a "policy number" or a "line number to which the data belongs", so that the data processing apparatus may compare, according to the data identifier, the comparison data of the data having the same data identifier, so as to improve the comparison efficiency.
S302, comparing the comparison data in each data source, and if different difference comparison data of the key fields exist, determining the detail information of the key fields represented by the difference comparison data according to the detail data corresponding to the difference comparison data.
In this embodiment, the comparison is performed according to the extracted comparison data in each data source, if there is different difference comparison data of the key fields, it is determined that the data in each data source corresponding to the comparison data is inconsistent, and in this embodiment, the detail information of the key field represented by the difference comparison data is determined according to the detail data corresponding to the difference comparison data. It should be understood that if the differential alignment data is one in which there is a difference in the respective data sources.
Illustratively, as shown in the above table one, when the data B in the data source 1 and the data B 'in the data source 2 are not consistent when the data B in the data source 1 and the data B' in the data region 1 are compared, the detailed data corresponding to the data (data B and data B ') of the difference ratio may be determined that the detailed information of the difference ratio data B is "the application year is 5 years and the beneficiary is X, the storage location is M folder in the data source 1", and the detailed information of the difference ratio data B' is "the application year is 6 years and the beneficiary is X, and the storage location is N folder in the data source 2".
S303, displaying the detail information of the key field of the difference comparison data characterization.
In this embodiment, after the data processing apparatus obtains the detailed information of the key field of the difference comparison data characterization, the detailed information may be displayed to remind the user of the data inconsistency between the data sources.
Illustratively, as shown in interface 402 in FIG. 4, details of "the application year is 5 years and the beneficiary is X, the storage location is M folder in data source 1", "the application year is 6 years and the beneficiary is X, and the storage location is N folder in data source 2" are displayed on the interface.
Correspondingly, based on the detail information, the user can determine inconsistent data in each data source, and the storage location, i.e. can determine an effective solution according to the detail information.
Optionally, a data link may also be displayed in this embodiment. Wherein the data link is used for characterizing identification information of a data source of key fields of the difference comparison data characterization, a difference generation reason and a data processing suggestion. The identification information of the data source of the key field is used to uniquely identify the data source to which the key field belongs, such as data source 1 or data source 2. The reason for the difference may be "network abnormality at the time of packet loss or transmission". The processing recommendation may be, for example, "repair a certain module in the X data source to prevent packet loss". It is to be understood that the cause of the difference generation and the processing advice in the present embodiment may be set corresponding to the line.
In one possible implementation, if the processing flow in which the difference comparison data is stored is determined according to the detailed information of the key field represented by the difference comparison data, the difference comparison data is processed by using the processing flow.
For example, if the data processing apparatus determines that the reason of the difference is "packet loss occurred in a certain module in the repair data source 1" according to the detailed information of the key field of the difference comparison data representation, and if the processing flow of the certain module in the repair data source 1 is stored in the data processing apparatus, the certain module in the data source 1 is repaired by adopting the processing flow, so as to avoid the packet loss of the certain module in the data source 1 again. Alternatively, the process flow is "copy data B in data source 1 into data source 2, and delete data B'" in data source 2 to keep data source 1 consistent with data in data source 2.
The data processing method provided by the embodiment comprises the following steps: reading data of each data source, and extracting comparison data of each piece of data of each data source and detail data corresponding to the comparison data; comparing the comparison data in each data source, and if different difference comparison data of the key fields exist, determining the detail information of the key fields represented by the difference comparison data according to the detail data corresponding to the difference comparison data; and displaying the detail information of the key fields of the differential comparison data characterization. The method avoids the processing mode of checking and comparing the data in each data source after all copying in the prior art, and extracts the comparison data of each piece of data of each data source and the detail data corresponding to the comparison data. Because the comparison data are used for representing the key fields of each piece of data, key information of the data can be embodied, and further, the comparison data with the difference are determined according to the difference of the comparison data of each piece of data of each data source. By acquiring the detail information corresponding to the comparison data with the difference according to the detail data corresponding to the comparison data with the difference, the data in each data source with the difference can be quickly acquired, so that the data processing efficiency is improved.
On the basis of the above embodiment, the data processing method provided by the present application is further described below with reference to fig. 5. Fig. 5 is a second flow chart of the data processing method provided by the present application. As shown in fig. 5, the data processing method provided in this embodiment may include:
s501, reading data of each data source, and extracting a comparison file of each piece of data of each data source and a detail file corresponding to the comparison file.
In this embodiment, the comparison data and the detail data are stored in the form of files. Specifically, the comparison data is a comparison file, and the detail data is a detail file.
S502, writing the comparison file of each piece of data of each data source and the detail file corresponding to the comparison file into a storage database.
For a clearer description of the manner of writing a file in the data processing method in this embodiment, first, a description will be given of a specific manner in which the manner of writing a file in the prior art is applied to the scene of writing an alignment file and a detail file in the present application. FIG. 6 is a flow chart of writing files in the prior art. As shown in fig. 6, the file in the prior art may include:
s601, reading data in each data source.
S602, writing the data in each data source into a storage database in the form of a writing file.
In the prior art, a read stream mode is utilized to circularly read data of a database and then write files. Wherein, after each reading of several pieces of data and writing of a file, S603 may be executed:
and judging whether data exist in each data source. If there is no data in each data source, S604 is executed, and if there is data in each data source, S601 is executed.
The write file ends 604.
In the prior art, in order to reduce the times of opening and closing the file, the file handle object is maintained in the whole process. Testing to derive 250 tens of thousands of data from a data source in this manner takes 125 seconds. Further, the analysis process finds that each time a file is written, a block occurs, and a user state-core state switch is caused, so that each time data is read, a file is written, how many times the switch is caused. Such costs can be particularly expensive when the amount of data is large.
Therefore, in this embodiment, an improved manner is provided, that is, the data in each data source to be read is buffered first, and the file is written once after being read for a certain number of times (for example, 100), so that the switching between the user state and the core state is greatly reduced, and the file writing efficiency is improved. Specifically, fig. 7 is a schematic flow chart of writing files provided in the present application. As shown in fig. 7, the step S502 may include:
S5021, reading the comparison file of each piece of data in each data source, and caching the read comparison file of the data and the detail file corresponding to the read comparison file of the data into a cache database.
In this embodiment, the comparison file of the read data in each data source and the detail file corresponding to the comparison file of the read data may be cached in the cache database, and then the cached comparison file and the detail file in the cache database are written into the storage database, so that the "user state-kernel state" switching may be reduced.
S5022, judging whether the comparison files of N data of each data source and detail files corresponding to the comparison files of each data source are cached in a cache database; if yes, executing S5023, if not, returning to executing S5021 until the comparison file of the N pieces of data of each data source and the detail file corresponding to the comparison file of the N pieces of data of each data source are cached in the cache database.
In order to realize the aim of comparing data in batches to further reduce the time of data processing, in this embodiment, after the comparison file and the detail file are cached, it may be determined whether the comparison file of N data of each data source and the detail file corresponding to the comparison file of each data source are cached in the cache database.
S5023, writing the comparison files of the N data of each data source and the detail files corresponding to the comparison files of the N data of each data source into a storage database, and clearing the cache database.
In this embodiment, after the comparison files of N pieces of data in each data source and the detail files corresponding to the comparison files have been stored in the cache database, the comparison files of N pieces of data in each data source and the detail files corresponding to the comparison files of N pieces of data in each data source may be written into the storage database, and the cache database is emptied to perform the caching of the comparison files of N pieces of data and the corresponding detail files of the next batch.
S503, comparing the comparison files in the data sources, and determining a difference comparison file.
Corresponding to the above S5021-S5023, in this embodiment, specifically, the key fields of the comparison file of the N pieces of data of each data source in the storage database may be read, the key fields of the N pieces of data of each data source are cached in the cache database, and if the key fields of the N pieces of data of each data source recorded in the cache database are compared, the difference key field of the difference between each data source is determined, then the comparison file to which the difference key field in each data source belongs is determined as the difference comparison file.
S504, determining the detail information of the key field represented by the difference comparison file according to the detail file corresponding to the difference comparison file.
In order to more clearly describe how to determine the detail information of the key field of the differential alignment file representation in the data processing method in this embodiment, a specific manner in the scenario of the write alignment file and the detail file applied to the present application in the manner of the prior art is described first. Fig. 8 is a schematic flow chart of determining detail information of key fields of a differential alignment file characterization in the prior art. As shown in fig. 8, the method may include:
s801, key fields of the difference comparison file in each data source are read.
S802, searching a difference detail file corresponding to the key field of the difference comparison file from the detail file.
S803, the difference detail file is written to the storage database.
In the prior art, the key fields of the difference comparison file are circularly read, and the difference comparison file is circularly read. After each reading of the key fields of the several difference comparison files and writing of the difference detail file, S804 may be executed: and judging whether the unread difference is compared with the file.
If there is no unread difference comparison file, step S805 is executed, and if there is an unread difference comparison file, step S801 is executed again.
S805, end.
This approach requires multiple readings of the difference detail file, which is inefficient. The method is improved in this embodiment, in which the detail files are read mainly in the form of a stream, and if the comparison file containing the difference key field is included, the detail files corresponding to the comparison file are stored. This requires only one pass of the detail file to be read.
Specifically, fig. 9 is a schematic flow chart of detail information of key fields of determining difference comparison file characterization provided in the present application. As shown in fig. 9, S504 may include:
s5041, the key fields of the difference comparison file in each data source are read.
S5042, the key fields of the read difference comparison file are cached in a cache database.
S5043, judging whether the key fields of the unread difference comparison file exist. If not, S5044 is executed, if yes, S5041 is returned to be executed until all the key fields of the unread differential comparison file have been completely cached in the cache database.
S5044, reading the detail file, and judging whether the detail file is a difference detail file corresponding to the key field of the difference comparison file; if yes, S5045 is executed, and if no, S5046 is executed.
S5045, writing the difference detail file to a storage database.
S5046, judging whether unread detail files exist; if yes, the process returns to S5044, and if no, S5047 is executed.
S5047, ends.
S505, saving the detail information of the key field of the difference comparison file characterization.
After determining the detail information of the key fields of the differential comparison file characterization, the detail information of the key fields of the differential comparison file characterization is usually stored in a storage database, and then the total amount and the display detail are counted. In general, when the data volume of inconsistent data is small, the detail information and the storage mode of key fields of the differential comparison file characterization in the prior art work well. Fig. 10 is a flow chart of a preservation method in the prior art. As shown in fig. 10, the saving method may include:
s1001, reading detail information of key fields of the difference comparison file characterization.
S1002, storing the detail information of the key fields of the read difference comparison file characterization into a storage database.
S1003, judging whether the unread detail information of the key field of the differential comparison file characterization exists; if yes, the above S1001 is executed again until all the detail information of the key fields of the differential comparison file characterization is read, and if not, S1004 is executed.
S1004, end.
However, when the difference is larger than the data size of the detail information of the key field of the file characterization, the storage method can cause a very large load on the storage database. When a large number of pieces of detail information of key fields of the differential comparison file characterization are considered, only sample data are needed for problem analysis, and all the data are not needed, so that the mode in the prior art is improved in the embodiment, mainly, only a preset number of pieces of data before the detail information of the key fields of the differential comparison file characterization are stored in a storage database, and other pieces of detail information of the key fields of the differential comparison file characterization form a file to be stored. When the data is displayed, the preset number of pieces of data detail information can be displayed, and other detail information provides file downloading.
Specifically, fig. 11 is a flow chart of the preservation method provided by the present application. As shown in fig. 11, the step S505 may include:
s5051, reading detail information of key fields of the preset number of difference comparison file characterization in the cache database.
S5052, writing the detail information of the key fields represented by the preset number of difference comparison files into a storage database, and storing the detail information of the key fields represented by the residual difference comparison files into the storage database in the form of written files.
S506, displaying the detail information of the key fields of the differential comparison file characterization.
It should be understood that the implementation manner in S501 and S506 in this embodiment may refer to the relevant descriptions in S301 and S303 in the foregoing embodiments, which are not described herein.
The data processing method provided in the embodiment can be applied to the scene such as settlement comparison and accumulated risk guarantee comparison:
exemplary, settlement alignment: and comparing whether the order-out data, the underwriting data and the financial data are consistent. If the settlement is inconsistent, the settlement is influenced, on the one hand, the timeliness of the settlement is influenced, and huge reconciliation work is caused, so that time and labor are consumed; on the other hand, affects the company's image. By using the data processing mode in the application, besides daily account checking, the method can also support checking all data of the last month at the beginning of the month, find inconsistent data, process in time, greatly reduce account checking work and promote the timeliness of settlement.
Accumulated risk balance comparison: the comparison is made whether the underwriting data is consistent with the data of the accumulated risk library, if the underwriting data is inconsistent with the data of the accumulated risk library, some malicious underwriting behaviors can be missed, and serious financial risks appear. And its timeliness requirements are very high. By using the data processing mode in the application, unsynchronized data can be found in second level, and an alarm is given, so that the wind control is supported forcefully.
In this embodiment, in order to achieve the purpose of comparing data in batches to further reduce the time of data processing, after the comparison file and the detail file are cached, it may be determined whether the comparison file of N pieces of data of each data source and the detail file corresponding to the comparison file of each data source are cached in the cache database; and the method can also reduce 'user mode-kernel mode' switching by adopting a cache mode when writing the comparison file of N data of each data source and the detail file corresponding to the comparison file of each data source. In addition, in this embodiment, the detail file is read in a stream form, and if the comparison file containing the difference key field, the detail file corresponding to the comparison file is stored. Thus, only one pass of detail file is needed to be read; in addition, in the embodiment, when a large number of difference comparison file characterization key field detail information is considered, only sample data is needed for problem analysis, and all data are not needed, so that the data storage efficiency is improved.
Fig. 12 is a schematic structural diagram of a data processing apparatus according to the present application. As shown in fig. 12, the data processing apparatus 1200 includes: a reading module 1201, a processing module 1202, a display module 1203 and a writing module 1204.
A reading module 1201 is configured to read data of each data source.
The processing module 1202 is configured to extract comparison data of each piece of data of each data source, and detail data corresponding to the comparison data, compare the comparison data in each data source, determine detail information of a key field represented by the comparison data according to the detail data corresponding to the comparison data if there is different comparison data of the key field, where the comparison data is used to represent the key field of each piece of data, the detail data is used to represent the detail information of the key field represented by the comparison data, and the detail information of the key field is used to represent a detailed field and a storage location corresponding to the key field;
and the display module 1203 is used for displaying the detailed information of the key fields of the differential comparison data characterization.
Optionally, the comparison data is a comparison file, and the detail data is a detail file.
And the writing module 1204 is used for writing the comparison file of each piece of data of each data source and the detail file corresponding to the comparison file into the storage database.
Optionally, the writing module 1204 is specifically configured to read the comparison file of each piece of data in each data source, cache the comparison file of the read data and the detail file corresponding to the comparison file of the read data into the cache database, and determine whether the comparison file of N pieces of data of each data source and the detail file corresponding to the comparison file of each data source are cached in the cache database; if yes, writing the comparison files of the N data of each data source and the detail files corresponding to the comparison files of the N data of each data source into a storage database, and clearing the cache database; if not, continuing to read the data in each data source until the comparison file of the N data of each data source and the detail file corresponding to the comparison file of the N data of each data source are cached in the cache database.
Optionally, the reading module 1201 is further configured to read key fields of the comparison file of N pieces of data of each data source in the storage database, and cache the key fields of the N pieces of data of each data source to the cache database;
if the key fields of the N pieces of data of each data source recorded in the cache database are compared, and a difference key field of the difference exists in each data source is determined, the processing module 1202 is further configured to determine a comparison file to which the difference key field in each data source belongs as a difference comparison file.
Optionally, the display module 1203 is further configured to display a data link, where the data link is used to characterize identification information of a data source of a key field of the difference comparison data characterization, a reason for the difference generation, and a data processing suggestion.
Optionally, the processing module 1202 is further configured to, if the processing flow in which the difference comparison data is stored is determined according to the detailed information of the key field of the difference comparison data representation, process the difference comparison data by using the processing flow.
Optionally, the reading module 1201 is further configured to use the policy data as the data in each data source, and the key field is a policy number or a serial number.
The principle and technical effects of the data processing apparatus provided in this embodiment are similar to those of the implementation of the data processing method described above, and are not described herein.
Fig. 13 is a schematic structural diagram of an electronic device provided by the present application. As shown in fig. 13, the electronic device 1300 includes: a memory 1301, and at least one processor 1302.
Memory 1301 for storing program instructions.
The processor 1302 is configured to implement the data processing method in this embodiment when the program instructions are executed, and the specific implementation principle can be seen from the above embodiment, which is not described herein again.
The electronic device 1300 may also include an input/output interface 1303.
The input/output interface 1303 may include a separate output interface and an input interface, or may be an integrated interface that integrates input and output. The output interface is used for outputting data, the input interface is used for acquiring input data, the output data is the generic name output in the method embodiment, and the input data is the generic name input in the method embodiment.
The present application also provides a readable storage medium having stored therein execution instructions which, when executed by at least one processor of an electronic device, when executed by the processor, implement the data processing method in the above embodiment.
The present application also provides a program product comprising execution instructions stored in a readable storage medium. The at least one processor of the electronic device may read the execution instructions from the readable storage medium, the execution instructions being executed by the at least one processor to cause the electronic device to implement the data processing methods provided by the various embodiments described above.
In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, and for example, the division of the modules is merely a logical function division, and there may be additional divisions when actually implemented, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or modules, which may be in electrical, mechanical, or other forms.
The modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, i.e., may be located in one place, or may be distributed over a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist alone physically, or two or more modules may be integrated into one module. The integrated modules may be implemented in hardware or in hardware plus software functional modules.
The integrated modules, which are implemented in the form of software functional modules, may be stored in a computer readable storage medium. The software functional module is stored in a storage medium, and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (english: processor) to perform some of the steps of the methods according to the embodiments of the application. And the aforementioned storage medium includes: u disk, mobile hard disk, read-Only Memory (ROM), random access Memory (Random Access Memory, RAM), magnetic disk or optical disk, etc.
In the above embodiments of the data processing apparatus or the electronic device, it should be understood that the processor may be a central processing unit (in english: central Processing Unit, abbreviated as CPU), or may be other general purpose processors, digital signal processors (in english: digital Signal Processor, abbreviated as DSP), application specific integrated circuits (in english: application Specific Integrated Circuit, abbreviated as ASIC), or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present application may be embodied directly in a hardware processor for execution, or in a combination of hardware and software modules in a processor for execution.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the application.
Claims (7)
1. A method of data processing, comprising:
partitioning the data of each data source according to a time sequence, reading a plurality of data areas in each data source, and extracting comparison data of each piece of data in each data area and detail data corresponding to the comparison data, wherein the comparison data is used for representing key fields of each piece of data, the detail data is used for representing detail information of the key fields represented by the comparison data, and the detail information of the key fields is used for representing detailed fields and storage positions corresponding to the key fields;
comparing the comparison data in each data area, and if different difference comparison data of the key fields exist, determining the detail information of the key fields represented by the difference comparison data according to the detail data corresponding to the difference comparison data;
Displaying the detail information of the key field of the difference comparison data representation;
the comparison data is a comparison file, the detail data is a detail file, and after the comparison data of each piece of data in each data area and the detail data corresponding to the comparison data are extracted, the method further comprises the steps of:
writing a comparison file of each piece of data of each data source and a detail file corresponding to the comparison file into a storage database;
the writing the comparison file of each piece of data of each data source and the detail file corresponding to the comparison file into a storage database comprises the following steps:
reading the comparison file of each piece of data in each data source, and caching the read comparison file of the data and the detail file corresponding to the read comparison file of the data into a cache database;
judging whether the comparison files of N data of each data source are cached in the cache database or not, and detail files corresponding to the comparison files of each data source;
if the comparison files of the N data of each data source and the detail files corresponding to the comparison files of each data source are cached in the cache database, writing the comparison files of the N data of each data source and the detail files corresponding to the comparison files of the N data of each data source into the storage database, and clearing the cache database;
If the comparison files of the N data of each data source and the detail files corresponding to the comparison files of each data source are not cached in the cache database, continuing to read the data in each data source until the comparison files of the N data of each data source and the detail files corresponding to the comparison files of the N data of each data source are cached in the cache database;
after the comparison of the comparison data in each data source, the method further comprises the following steps:
reading key fields of comparison files of N data of each data source in the storage database, and caching the key fields of the N data of each data source into the cache database;
and if the key fields of the N data of each data source recorded in the cache database are compared, determining a difference key field with a difference in each data source, and determining a comparison file to which the difference key field in each data source belongs as the difference comparison file.
2. The method according to claim 1, wherein the method further comprises:
displaying a data link, wherein the data link is used for representing identification information, difference generation reasons and data processing suggestions of a data source of a key field of the difference comparison data representation.
3. The method according to claim 2, wherein the method further comprises:
and if the processing flow for processing the difference comparison data is determined to be stored according to the detailed information of the key field of the difference comparison data characterization, processing the difference comparison data by adopting the processing flow.
4. The method of claim 1, wherein the data in each of the data sources is policy data and the key field is a policy number or a serial number.
5. A data processing apparatus, comprising:
the reading module is used for carrying out partition processing on the data of each data source according to the time sequence and reading a plurality of data areas in each data source;
the processing module is used for extracting comparison data of each piece of data of each data area and detail data corresponding to the comparison data, comparing the comparison data in each data area, and determining detail information of a key field represented by the difference comparison data according to the detail data corresponding to the difference comparison data if different difference comparison data of the key field exist, wherein the comparison data are used for representing the key field of each piece of data, the detail data are used for representing the detail information of the key field represented by the comparison data, and the detail information of the key field is used for representing the detailed field and a storage position corresponding to the key field;
The display module is used for displaying detail information of key fields of the difference comparison data representation;
the data comparison module is used for writing the data comparison files of the data sources and the detail files corresponding to the comparison files into a storage database;
the writing module is specifically configured to read a comparison file of each piece of data in each data source, and cache the read comparison file of the data and a detail file corresponding to the read comparison file of the data into a cache database;
judging whether the comparison files of N data of each data source are cached in the cache database or not, and detail files corresponding to the comparison files of each data source;
if the comparison files of the N data of each data source and the detail files corresponding to the comparison files of each data source are cached in the cache database, writing the comparison files of the N data of each data source and the detail files corresponding to the comparison files of the N data of each data source into the storage database, and clearing the cache database;
If the comparison files of the N data of each data source and the detail files corresponding to the comparison files of each data source are not cached in the cache database, continuing to read the data in each data source until the comparison files of the N data of each data source and the detail files corresponding to the comparison files of the N data of each data source are cached in the cache database;
the reading module is further configured to read key fields of the comparison file of the N pieces of data of each data source in the storage database, and cache the key fields of the N pieces of data of each data source to the cache database;
and if the key fields of the N data of each data source recorded in the cache database are compared, determining a difference key field with a difference in each data source, and determining a comparison file to which the difference key field in each data source belongs as the difference comparison file.
6. An electronic device, comprising: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing computer-executable instructions stored in the memory to cause the electronic device to perform the method of any one of claims 1-4.
7. A computer readable storage medium having stored thereon computer executable instructions which, when executed by a processor, implement the method of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911165369.7A CN111078738B (en) | 2019-11-25 | 2019-11-25 | Data processing method, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911165369.7A CN111078738B (en) | 2019-11-25 | 2019-11-25 | Data processing method, device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111078738A CN111078738A (en) | 2020-04-28 |
| CN111078738B true CN111078738B (en) | 2023-08-15 |
Family
ID=70311511
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911165369.7A Active CN111078738B (en) | 2019-11-25 | 2019-11-25 | Data processing method, device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111078738B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112905653B (en) * | 2021-03-26 | 2025-09-05 | 掌阅科技股份有限公司 | Data comparison method, computing device and computer storage medium |
| CN113902033A (en) * | 2021-10-28 | 2022-01-07 | 中国建设银行股份有限公司 | Configurable keyword-based data error identification method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105930325A (en) * | 2015-11-19 | 2016-09-07 | 中国银联股份有限公司 | Reverse analysis method and device for file report comparative difference |
| CN107833637A (en) * | 2017-06-19 | 2018-03-23 | 平安医疗健康管理股份有限公司 | Medicine regular record update method, device, computer equipment and medium |
| EP3301649A1 (en) * | 2017-09-07 | 2018-04-04 | Siemens Healthcare GmbH | Method for processing medical image data and image processing system for medical image data |
| CN110427473A (en) * | 2019-08-02 | 2019-11-08 | 泰康保险集团股份有限公司 | Data processing method, device, equipment and storage medium |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8200679B2 (en) * | 2008-12-04 | 2012-06-12 | Oracle International Corporation | Estimating cardinalities of XML table constructs within queries |
| US8762407B2 (en) * | 2012-04-17 | 2014-06-24 | Renmin University Of China | Concurrent OLAP-oriented database query processing method |
| CN104376047B (en) * | 2014-10-28 | 2017-06-30 | 浪潮电子信息产业股份有限公司 | A kind of big table join methods based on HBase |
| CN107193813B (en) * | 2016-03-14 | 2021-05-14 | 阿里巴巴集团控股有限公司 | Data table connection mode processing method and device |
| CN107346317A (en) * | 2016-05-06 | 2017-11-14 | 北京神州泰岳软件股份有限公司 | A kind of data query method and apparatus |
| AU2017320475B2 (en) * | 2016-09-02 | 2022-02-10 | FutureVault Inc. | Automated document filing and processing methods and systems |
| CN106339500A (en) * | 2016-09-09 | 2017-01-18 | 浪潮软件股份有限公司 | Different-place database comparison tool and method |
| CN110069571A (en) * | 2019-03-18 | 2019-07-30 | 平安普惠企业管理有限公司 | A kind of automated data control methods and device, electronic equipment |
-
2019
- 2019-11-25 CN CN201911165369.7A patent/CN111078738B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105930325A (en) * | 2015-11-19 | 2016-09-07 | 中国银联股份有限公司 | Reverse analysis method and device for file report comparative difference |
| CN107833637A (en) * | 2017-06-19 | 2018-03-23 | 平安医疗健康管理股份有限公司 | Medicine regular record update method, device, computer equipment and medium |
| EP3301649A1 (en) * | 2017-09-07 | 2018-04-04 | Siemens Healthcare GmbH | Method for processing medical image data and image processing system for medical image data |
| CN110427473A (en) * | 2019-08-02 | 2019-11-08 | 泰康保险集团股份有限公司 | Data processing method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111078738A (en) | 2020-04-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108255925B (en) | Method and terminal for displaying data table structure change condition | |
| CN109815746B (en) | Data tamper-proofing method and system based on block chain technology | |
| US10387389B2 (en) | Data de-duplication | |
| US10528534B2 (en) | Method and system for deduplicating data | |
| CN112434015B (en) | Data storage method and device, electronic equipment and medium | |
| US8972338B2 (en) | Sampling transactions from multi-level log file records | |
| CN111078738B (en) | Data processing method, device, electronic equipment and storage medium | |
| US20190384825A1 (en) | Method and device for data protection and computer readable storage medium | |
| CN113312259A (en) | Interface testing method and device | |
| CN116910079B (en) | A Flink method, system, device, and storage medium for implementing delayed association of CDC data dimension tables | |
| CN113342681B (en) | Regression testing method and device | |
| CN118012956A (en) | Method, device, equipment and storage medium for transmitting data between databases | |
| CN113419964B (en) | Test case generation method and device, computer equipment and storage medium | |
| US20090259659A1 (en) | Identifying entities of interest | |
| CN114564477B (en) | Data storage method and device, electronic equipment and storage medium | |
| US20200104391A1 (en) | Ensuring integrity of records in a not only structured query language database | |
| CN107315806B (en) | Embedded storage method and device based on file system | |
| US10956369B1 (en) | Data aggregations in a distributed environment | |
| CN115328876A (en) | Method and device for recording page operation log | |
| US8650209B1 (en) | System, method, and computer program for determining most of the non duplicate records in high performance environments in an economical and fault-tolerant manner | |
| CN114490663B (en) | Data processing method and device | |
| CN111459949A (en) | Data processing method, device and equipment for database and index updating method | |
| CN116703184B (en) | Data processing method, data processing device, electronic equipment and readable storage medium | |
| US20230162836A1 (en) | Data ducts for processing of medical data | |
| CN114238582B (en) | Method, device, equipment, and storage medium for determining suspected related parties |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |