CN118467549B - A visual data connection method and device based on data model - Google Patents

Abstract

The application relates to a visualized data leading method and device based on a data model. The method comprises the following steps: providing a description method of the leading data model, describing the characteristics of the complex data structure from multiple dimensions, and realizing comprehensive and complete metadata recording and management of complex business data objects; based on the design of the leading data model, a matched visual modeling method is provided, and the rapid and accurate construction of the leading data model is supported; the comprehensive lead configuration template is designed, and automatic generation and execution of lead codes based on template configuration information can be supported. The application constructs a set of generalized, extensible and independent data management system leading data model to adapt to complex heterogeneous leading data structures, ensures the integrity of data description on the premise of controlling the complexity of the model, and then designs a dynamic, visual and concise man-machine interaction method based on the data model description method to guide a user to rapidly and conveniently finish data leading work.

Description

A visual data connection method and device based on data model

Technical Field

The present application relates to the field of data processing, and in particular to a method and device for visual data connection based on a data model.

Background Art

The existing major data import software tools mainly read the data in the data table and related data files from the source database, convert the table structure and field format, and then store them in the target database.

However, in actual applications, it is generally necessary to organically organize several data tables (entities) to form a data model to support a complete description of business objects.

When using ordinary data connection methods to connect data based on complex data models, there are difficulties such as complex human-computer interaction operations in data connection configuration, high requirements on user technical capabilities, and difficulty in fully and completely describing the relationships between data tables or data entities.

Summary of the invention

Based on this, it is necessary to provide a visual data connection method and device based on a data model to address the above technical problems, and to summarize a general and universal connection data model description method for complex and heterogeneous external data structures to assist users in building connection data models and improve the integrity and convenience of data connection configuration.

A visual data connection method based on a data model, the method comprising:

Obtain and complete the description information of all data tables on the data source side, and mark the table type of each data table; table types include: entity table, relationship table, and dictionary table;

For the business object to be linked, select an entity table containing the identification information of the business object from the data table on the data source end as the central table of the linked data model, select the entity table and dictionary table related to the central table that can describe the attributes of the business object and add them to the linked data model, and configure the relationship information between the data tables in the linked data model;

The data table description information in the access data model, the description information of the data fields in the data table, and the relationship information between the data tables are stored in a pre-built multi-dimensional metadata description template to obtain a metadata description model corresponding to the business object; wherein the multi-dimensional metadata description template specifically includes: a data table metadata table, a data field metadata table, and a data table relationship metadata table; the data table metadata table is used to describe the metadata information of all data tables in the data model corresponding to the business data object; one business data object corresponds to one data model; the data field metadata table is used to describe the metadata information of all data fields contained in each data table in the data model; the data table relationship metadata table is used to describe the metadata information of the relationship between all data tables in the data model;

Design the connection template in combination with the metadata description model configuration, and complete the data connection based on the connection template.

A visual data connection device based on a data model, the device comprising:

The description information acquisition module is used to obtain and improve the description information of all data tables on the data source side, and mark the table type of each data table; the table types include: entity table, relationship table and dictionary table;

The connection data model construction module is used to select an entity table containing identification information of the business object from the data table at the data source end as the central table of the connection data model for the business object to be connected, select entity tables and dictionary tables related to the central table that can describe the attributes of the business object to add to the connection data model, and configure the relationship information between the data tables in the connection data model;

The description template filling module is used to store the data table description information in the access data model, the description information of the data fields in the data table, and the relationship information between the data tables into the pre-built multi-dimensional metadata description template to obtain the metadata description model corresponding to the business object; wherein the multi-dimensional metadata description template specifically includes: a data table metadata table, a data field metadata table, and a data table relationship metadata table; the data table metadata table is used to describe the metadata information of all data tables in the data model corresponding to the business data object; one business data object corresponds to one data model; the data field metadata table is used to describe the metadata information of all data fields contained in each data table in the data model; the data table relationship metadata table is used to describe the metadata information of the relationship between all data tables in the data model;

The induction task execution module is used to design the induction template in combination with the metadata description model configuration and complete the data induction based on the induction template.

A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

Obtain and complete the description information of all data tables on the data source side, and mark the table type of each data table; table types include: entity table, relationship table, and dictionary table;

For the business object to be linked, select an entity table containing the identification information of the business object from the data table on the data source end as the central table of the linked data model, select the entity table and dictionary table related to the central table that can describe the attributes of the business object and add them to the linked data model, and configure the relationship information between the data tables in the linked data model;

The data table description information in the access data model, the description information of the data fields in the data table, and the relationship information between the data tables are stored in a pre-built multi-dimensional metadata description template to obtain a metadata description model corresponding to the business object; wherein the multi-dimensional metadata description template specifically includes: a data table metadata table, a data field metadata table, and a data table relationship metadata table; the data table metadata table is used to describe the metadata information of all data tables in the data model corresponding to the business data object; one business data object corresponds to one data model; the data field metadata table is used to describe the metadata information of all data fields contained in each data table in the data model; the data table relationship metadata table is used to describe the metadata information of the relationship between all data tables in the data model;

Design the connection template in combination with the metadata description model configuration, and complete the data connection based on the connection template.

A visual data connection method and device based on a data model, including: proposing a connection data model description method to describe the characteristics of complex data structures from multiple dimensions, and realizing comprehensive and complete metadata recording and management of complex business data objects; based on the design of the connection data model, proposing a matching visual modeling method to support the rapid and accurate construction of the connection data model; designing a comprehensive connection configuration template to support the automatic generation and execution of connection codes based on template configuration information. The present invention constructs a set of universal, extensible, and data management system-independent connection data models to adapt to complex and heterogeneous connection data structures, and ensures the integrity of data description under the premise of controlling the complexity of the model. Then, based on this data model description method, a dynamic, intuitive, and concise human-computer interaction method is designed to guide users to quickly and conveniently complete data connection work.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a flow chart of a method for visualizing data connection based on a data model in one embodiment;

FIG2 is a schematic diagram of a data connection operation process specification in one embodiment;

FIG3 is a schematic diagram of selecting a center table and dragging it to the center of the interface in one embodiment;

FIG4 is a schematic diagram of dragging related dictionary tables and entity tables onto an interface and configuring the relationship between them and the central table in one embodiment;

FIG. 5 is a diagram showing the internal structure of a computer device in one embodiment.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

In one embodiment, as shown in FIG1 , a visual data connection method based on a data model is provided, comprising the following steps:

Step 102: Obtain and complete the description information of all data tables at the data source end, and mark the table type of each data table.

Get the description information of all data tables from the data source, including table names, table comments, etc., which can be displayed in a list to obtain a data source table description information list. In the data source table description information list, manually configure the type of each table (mainly including entity tables, relationship tables, and dictionary tables), and supplement the table description information (such as the Chinese name of the table, table comments, etc.).

All data tables here refer to all data tables that can be obtained from the data source. For example, if the data source is a database of an information management system, when you have permission to access the database connection, you can obtain the description information of all data tables in the database (subject to database permissions).

Step 104, for the business object to be linked, select an entity table containing the identification information of the business object from the data table on the data source end as the central table of the linked data model, select entity tables and dictionary tables related to the central table that can describe the attributes of the business object and add them to the linked data model, and configure the relationship information between the data tables in the linked data model.

In actual database applications, a business object is often not only described by one entity table, but may include multiple components and need to be described by multiple related entity tables and other types of data tables. For example, to describe the properties of a certain model of aircraft, you need a basic information table of the aircraft + a technical parameter table of the sensors equipped on the aircraft + a technical parameter table of the engines equipped on the aircraft + a large number of dictionary tables (aircraft type, country, etc.).

Step 106, storing the data table description information in the reference data model, the description information of the data fields in the data table, and the relationship information between the data tables into a pre-built multi-dimensional metadata description template to obtain a metadata description model corresponding to the business object.

Among them, the multi-dimensional metadata description template specifically includes: data table metadata table, data field metadata table and data table relationship metadata table. When importing data, the data information of one or more business objects should be obtained on demand, rather than simply migrating all the content in the data table. This requires a complete and comprehensive description of the relevant data tables, table relationships and data elements in the table corresponding to the business objects to ensure the integrity and consistency of the imported data. The specific metadata description is recorded in three metadata tables: data table metadata table, data field metadata table and data table relationship metadata table. The three tables are explained as follows:

The data table metadata table is used to describe the metadata information of all data tables in the data model corresponding to the business data object. One business data object corresponds to one data model.

The data field metadata table is used to describe the metadata information of all data fields contained in each data table in the data model.

The data table relationship metadata table is used to describe the metadata information of the relationship between all data tables in the data model.

The multi-dimensional metadata description model of "table information + (in the table) data field information + table relationship information" is adopted to achieve comprehensive and complete metadata recording and management of complex business data objects. It can comprehensively and completely describe the data structure to be introduced and ensure the integrity and consistency of the introduced data in business semantics.

Step 108, design an initiation template in combination with the metadata description model configuration, and complete data initiation based on the initiation template.

The import template refers to the collection of all relevant configuration information involved in the data import process. Various configuration information can be reused, and the user is guided through the visual interface to select or add and modify various configuration information to form a complete import template. During the import process, data processing or calculation can be performed based on the import template information to support the completion of data import.

The above-mentioned visual data connection method based on data model can be well applied to scenarios of accurately connecting required data content to business data objects, and has strong versatility and scalability. It can realize automatic generation and execution of connection code on the premise of detailed description of connection data model and connection template.

It should be understood that, although the various steps in the flowchart of FIG. 1 are shown in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in FIG. 1 may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

In one embodiment, in a multi-dimensional metadata description template:

Each row in the data table metadata table records: the identifier, name, description, and table type of a specific data table; table types include: entity table, relationship table, and dictionary table;

Each row in the data field metadata table records: the identifier, name, data table, description, data type, length, precision, dimension, access control level, and format constraints (regular expression specifications can be used) of a specific data field;

Each row in the data table relationship metadata table records: the relationship between two specific data tables, including the identifiers of the two data tables, the identifiers of the data fields that reference each other in the two data tables, and the relationship type between the two data tables; relationship types include: one-to-many, many-to-one, one-to-one, and many-to-many, etc.

In order to be compatible with unstructured data, each row in the data table metadata table can record the description information of a structured data table in a database, or the overall name and identifier of a group of related files, including unstructured data files such as videos and pictures, and semi-structured data files such as XML/EXCEL. This group of related files is regarded as a data table, in which each file is an entity with a unique identifier (which can be an access path).

In one embodiment, an entity table containing identification information of a business object is selected from a data table at a data source end as a central table of a connection data model, an entity table and a dictionary table related to the central table that can describe attributes of the business object are selected and added to the connection data model, and relationship information between the data tables in the connection data model is configured, including:

Select an entity table containing business object identification information from the data table at the data source end as the central table of the connected data model, and display it on the visualization interface as the first icon. Drag the central table to the interface, and use a circular pattern as a mark.

Select the entity table that can describe the attributes of the business object related to the central table as the other entity table to be connected to the data model, and display it on the visual interface with the second icon. On the visual interface, connect the existing entity tables consisting of the central table and other entity tables, and mark the corresponding associated fields between the two entity tables on the line; if the two entity tables are associated through a relationship table, the corresponding relationship table and the associated fields between the two entity tables and the relationship table need to be marked on the line. Drag other entity tables to the interface, and you can use an elliptical pattern as a mark. The existing entity table here refers to the central table (essentially also an entity table) that has been dragged to the graphical interface with the first icon + multiple other entity tables displayed with the second icon. The connection here includes the connection between the central table and other entity tables, as well as the connection between other entity tables.

Select the dictionary table that can describe the attributes of the business object related to the central table as the dictionary table for the data model, and display it on the visualization interface with the third icon. Drag the existing entity table and the dictionary table on the visualization interface, and mark the corresponding associated fields between the entity table and the dictionary table on the line. Drag the dictionary table to the interface and use a diamond pattern as a mark. The entity table and the dictionary table are connected, but there is no need to connect the dictionary tables because the content of the dictionary table is only the value range of the enumeration field.

Click the icon corresponding to any table on the diagram to display the definition information of each data field in the table, including field name, field comment, field type, whether it is empty, field length, etc. This information may be obtained directly from the data source and can be further supplemented and improved. After completing the graphical data model configuration, save the access data model. The description information of the access data model configured on the visual interface is saved in the three metadata tables defined in the access data model description method.

Each data model has only one central table. Based on all the table information on the same data source, multiple data models can be configured. For example, if the data source is a database of a course scheduling system, then a data model centered on student information can be configured, a data model centered on course information can be configured, or a data model centered on department information can be configured. The configuration steps of each data model can be carried out in sequence according to the above steps, that is, by performing the operations in the above steps, a new data model can be created based on all the table spaces on the data source. In summary, based on the design of the interface data model, a supporting visual modeling method is proposed to support interface operators to quickly and accurately build the interface data model on a graphical interface by dragging, clicking, and configuring. Data tables are represented by icons of different shapes, and the relationship between data tables is represented by lines. Right-click menus are provided on icons and lines to support editing and configuration of data tables and their relationships. The main operations include configuration of data table types, determination of the model central table, drag-and-drop layout of data tables included in the model, format configuration of each data field in each data table in the model, and configuration of the relationship between each data table in the model.

In one embodiment, a design of an induction template is combined with the metadata description model configuration, and data induction is completed based on the induction template, including:

Create an induction task and associate it with an induction template; the induction template includes data source information, data destination information, metadata description model information, induction filtering rules, induction data conversion rules, induction data verification rules and induction data quality evaluation rules.

Automatically parse the boot template to execute the boot task.

The connection template (including connection data model) formed by guided visual configuration contains a large amount of configuration information such as data source configuration information, data destination configuration information, connection data model information, connection filtering rules, connection data conversion rules, connection data verification rules, connection data quality evaluation rules, connection performance parameter configuration information, etc. In the actual data connection execution process, the connection template configuration information can be automatically converted into executable code and executed to realize automatic data connection based on the connection template.

In one embodiment, a connection template refers to a collection of all relevant configuration information involved in the data connection process, including data source information, data destination information, connection data model information, connection filtering rules, connection data conversion rules, connection data verification rules, connection data quality evaluation rules, connection performance parameter configuration information, etc.

Data source information includes: data source type, data source access address, data source name, and data source description. When data is connected, the source end may be in the form of a database, data file, data interface, etc.

The data destination information includes the data destination type, data destination access address, data destination name, data destination description, etc. When data is connected, the destination is generally a local database or data warehouse.

Imported data model information: The imported data model refers to a data set description that combines multiple data entities (tables) at the source end to support information representation and operation, based on application needs, including descriptions of the constituent entities (tables), relationships between entities (tables), structures within entities (tables), formats and descriptions of data elements (fields) within entities (tables), etc. The imported data model information is the result of configuration according to step 104.

The induction filtering rules are built on the central table of the induction data model and are associated with the induction tasks. By limiting the filtering conditions of each data field on the central table, the data content that needs to be inducted is filtered out from the central table, and the required data content is extracted from the related tables through the relationship information between the data tables in the metadata description model information;

The data conversion rules for the connection configure and record the corresponding data field format conversion rules of the source and destination, and support the automatic conversion of the source data format to the destination data format. The data structure of the connection data source and destination may be different, which may be caused by the different models of the databases or data files at both ends. For example, if the format of a field on the source is number, the destination needs to convert it to decimal. If the format of a field on the source is string (from Excel), the destination needs to convert it to numeric format.

The imported data verification rules are used to verify the standardization, integrity, and consistency of the data after the data is imported locally. After the data is imported locally, it is necessary to organize the data to find and solve the problems in the data, mainly focusing on the standardization, integrity, and consistency of the data. Based on the table relationships, business rules in the tables, and table relationships in the selected data model, the verification rules can be further improved and refined to support automatic verification of data to find problems.

The imported data quality evaluation rules are used to comprehensively evaluate the quality of the imported data to decide whether to save the data. After the data is imported locally, the data quality can be evaluated to quantitatively assess the overall quality level of the data. When the data quality level is poor, the specific reasons are analyzed and the data is reorganized or re-imported according to the reasons to improve the data quality. The quality inspection rules are the basis for quality evaluation, and their configuration mainly includes the scoring rules for various types of questions and their weight settings.

The boot performance parameter information is used to configure the boot performance parameters when executing the boot task, including: whether it is multi-threaded, and the size of the data cache occupied, etc.

It can be understood that this method is highly versatile and can be applied to the induction applications of various complex heterogeneous data. By flexibly configuring the induction configuration template information including the induction data model, induction data source, induction filtering conditions, etc., the reusability and scalability of the induction configuration can be improved.

In summary, the present invention constructs a universal, extensible, and data management system-independent access data model to adapt to complex and heterogeneous access data structures, while ensuring the integrity of data descriptions while controlling model complexity. Based on this data model description method, a dynamic, intuitive, and concise human-computer interaction method is designed to guide users to complete data access tasks quickly and conveniently.

In one embodiment, the data verification rules mainly include field verification, entity verification, and relationship verification;

Field validation includes non-emptiness validation, value range validation, and format validation, etc.

Entity verification mainly includes entity data element integrity verification and entity data duplication verification;

Relationship verification mainly includes consistency verification of relationships between entities (including dictionaries), that is, consistency verification of relationships between entity tables, and consistency verification of relationships between entity tables and dictionary tables.

In one embodiment, the configuration of the data quality evaluation rules includes: scoring rules and weight settings for various types of quality issues;

Quality issues mainly include:

1) Problems such as missed data and incorrect data conversion recorded during the connection process monitoring;

2) When compiling data, problems such as incorrect data format, incorrect data value precision, and non-standard data content are discovered using the data verification rules;

3) Issues discovered and recorded after manually reviewing the data structure and content.

In order to improve the planning and standardization of external data connection operations with complex data structures and ensure data integrity and consistency, a general data connection process specification is proposed based on the aforementioned data model and connection template design. It mainly includes three steps: connection task creation and template configuration, connection task execution and process monitoring, and connection data compilation and quality assessment, see Figure 2.

1) Create a new task and configure a template

First, create an induction task. If there is an available induction template, associate it with the induction task; if there is no reusable induction template, you need to create a new induction template.

In the connection template configuration, it is necessary to configure the data source information, connection data model information, connection filtering condition information, connection conversion rules, etc. of the source and destination ends respectively. The entire template configuration and each configuration step can be quickly configured by searching and adjusting the configuration information saved in the selection system.

In addition, you can also configure execution parameters for the induction task, including timed execution configuration, execution performance parameter settings, etc.

2) Task execution and process monitoring

After completing the induction task configuration, you can execute the induction task.

When executing the connection task, according to the carefully described connection template, the code can be generated and executed by automatically parsing the template to complete the data connection process.

During the execution of the induction task, activities such as induction template verification, induction template parsing, and induction code execution will generate some process information, which needs to be recorded in the induction task log.

In order to monitor the execution progress of the induction task, a graphical interactive interface can be provided to show the user the induction progress, data source information, storage location information, elapsed time, remaining time and other information (optional).

3) Data compilation and quality assessment

After executing the induction task and saving the external data to the local database, the data needs to be further compiled and quality assessed, and then the data that meets the quality requirements is retained.

First, you can set data validation rules to automatically validate the imported data and find problems in the data. These rules can be mainly divided into two categories: field validation rules and data table validation rules.

Then, based on the problems found, you can set data processing rules and perform some operations such as batch filling and replacement of data.

Quality problems in the data can also be corrected by manually reviewing and compiling the data, where data quality problems can be automatically discovered by the verification rules and prompted to the user.

Finally, we review and summarize the data quality issues obtained and recorded from multiple channels, including obtaining quality issues that occur during the connection process from the connection monitoring log files, obtaining and recording data quality issues discovered based on verification rules during compilation, and manually recording quality issues during compilation, etc. We conduct a comprehensive evaluation of the quality of the connection data to decide whether to save the data.

In summary, based on the data model and the design of the interface template, a general data interface process specification is proposed, which mainly includes interface task creation, interface template configuration (reuse), interface task execution and process monitoring, interface data compilation and quality assessment, etc. It can support the planning and normalization of external data with complex data structures, standardize the interface operation, and ensure data integrity and consistency.

Taking the aircraft platform and technical parameter data model in a certain system as an example, the application mode and function of the method of the present invention are explained.

1) Analysis of the data structure to be introduced

It is necessary to access the platform and technical parameter data of certain aircraft models in a database. The relevant data tables include entity tables such as aircraft model information table, aircraft sensor technical parameter table, aircraft engine technical parameter table, and aircraft type, country, etc. Since there is a many-to-many relationship between aircraft model and sensor model, aircraft model and engine model, it is necessary to add a special aircraft-sensor relationship table and aircraft-engine relationship table to describe this relationship.

The above mentioned data tables and their data element analysis are shown in Table 1.

Table 1 Aircraft platform and technical parameter data structure composition

Note: The data tables and data elements actually used to describe the aircraft platform and technical parameters are relatively complex, and only some of the contents are listed in the above table as examples.

2) Metadata structure design for storing data model information

The information of the linked data model is recorded in three metadata tables, namely, the data table metadata table, the data field metadata table, and the data table relationship metadata table, as shown in Table 2, Table 3, and Table 4 below.

Table 2 Data table metadata table

Table 3 Data field metadata table

Table 4 Data table relationship metadata table

3) Visualized data model construction process

The data model is constructed on the visualization interface. Figures 3 and 4 show schematic diagrams of the visualization configuration process.

In Figure 3, a list of related data tables is listed on the left side of the visualization interface and marked as entity tables, relationship tables, dictionary tables, etc. The aircraft basic information table is dragged as the central table to the interface and displayed as a circular icon; right-click on the icon to configure the relationship between the table and other data tables.

In Figure 4, the relationship between the central table and the other two data tables has been configured, where the oval icon represents an entity table (sensor technical parameter table), the diamond icon represents a dictionary table (aircraft type table), and the line between the icons represents the association relationship between the two tables.

The name, type and other table description information of the data table represented by each icon will be stored in the data table metadata table and automatically assigned a unique identifier; the field name, format and other information of the data field in the data table represented by each icon will be stored in the data field metadata table and automatically assigned a unique identifier; the association relationship information on the connection line, including the unique identifiers of the data tables on both sides and the unique identifiers of the associated data fields, will be stored in the table relationship metadata table.

4) Data integration based on integration process specifications

According to the data connection process specification in the invention content, the entire data connection process is completed.

In one embodiment, a visual data connection device based on a data model is provided, comprising: a description information acquisition module, a connection data model construction module, a description template filling module and a connection task execution module, wherein:

The description information acquisition module is used to obtain and improve the description information of all data tables on the data source side, and mark the table type of each data table; the table types include: entity table, relationship table and dictionary table;

The connection data model construction module is used to select an entity table containing identification information of the business object from the data table at the data source end as the central table of the connection data model for the business object to be connected, select entity tables and dictionary tables related to the central table that can describe the attributes of the business object to add to the connection data model, and configure the relationship information between the data tables in the connection data model;

The description template filling module is used to store the data table description information in the reference data model, the description information of the data fields in the data table, and the relationship information between the data tables into the pre-built multi-dimensional metadata description template to obtain the metadata description model corresponding to the business object; wherein the multi-dimensional metadata description template specifically includes: a data table metadata table, a data field metadata table, and a data table relationship metadata table; the data table metadata table is used to describe the metadata information of all data tables in the data model corresponding to the business data object; one business data object corresponds to one data model; the data field metadata table is used to describe the metadata information of all data fields contained in each data table in the data model; the data table relationship metadata table is used to describe the metadata information of the relationship between all data tables in the data model;

The induction task execution module is used to design the induction template in combination with the metadata description model configuration and complete the data induction based on the induction template.

For the specific definition of the visual data connection device based on the data model, please refer to the definition of the visual data connection method based on the data model above, which will not be repeated here. Each module in the above-mentioned visual data connection device based on the data model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG5. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a visual data connection method based on a data model is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

Those skilled in the art will understand that the structure shown in FIG. 5 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims (8)

1. A method for interfacing visual data based on a data model, the method comprising:

acquiring and perfecting description information of all data tables of a data source end, and marking table types of each data table; the table types include: an entity table, a relationship table, and a dictionary table;

Selecting an entity table containing service object identification information from a data table of a data source end as a center table of a guiding data model, selecting an entity table and a dictionary table which are related to the center table and can describe service object attributes, adding the guiding data model, and configuring relation information among all data tables in the guiding data model;

Storing the description information of the data tables in the leading data model, the description information of the data fields in the data tables and the relation information among the data tables into a multi-dimensional metadata description template constructed in advance to obtain a metadata description model corresponding to the business object; the multidimensional metadata description template specifically comprises: a data table metadata table, a data field metadata table, and a data table relationship metadata table; the data table metadata table is used for describing metadata information of all data tables in the data model corresponding to the service data object; a business data object corresponds to a data model; the data field metadata table is used for describing metadata information of all data fields contained in each data table in the data model; the data table relation metadata table is used for describing metadata information of relations among all data tables in the data model;

Configuring and designing a guiding template by combining the metadata description model, and completing data guiding based on the guiding template;

Selecting an entity table containing service object identification information from data tables of a data source end as a center table of the leading data model, selecting entity tables and dictionary tables which are related to the center table and can describe service object attributes, adding the leading data model, and configuring relation information among all the data tables in the leading data model, wherein the method comprises the following steps:

selecting an entity table containing service object identification information from a data table of a data source end as a center table for leading to a data model, and displaying the entity table on a visual interface by a first icon;

Selecting entity tables which are related to the center table and can describe the attribute of the business object, taking the entity tables as other entity tables of the leading data model, displaying the entity tables on a visual interface by a second icon, connecting lines between the center table and the existing entity tables formed by the other entity tables on the visual interface, and marking associated fields between the two corresponding entity tables on the lines; if the two entity tables are associated through the relation table, the corresponding relation table needs to be marked on line, and the two entity tables are respectively associated with the associated fields of the relation table;

and selecting a dictionary table which is related to the center table and can describe the attribute of the business object, taking the dictionary table as a dictionary table of the leading data model, displaying the dictionary table by a third icon, connecting the existing entity table and the dictionary table on a visual interface, and marking the associated field between the corresponding entity table and the dictionary table on the line.

2. The method of claim 1, wherein in the multi-dimensional metadata description template:

Each row of the data table metadata table is recorded with: identification, name, description, and table type of a particular data table; the table types include: an entity table, a relationship table, and a dictionary table;

each row of the data field metadata table is recorded with: the identification, name, data table, description, data type, length, precision, dimension, access control level and format constraint information of a specific data field;

Each row of the data table relation metadata table is recorded with: the relation between two specific data tables specifically comprises identifiers of the two data tables, identifiers of data fields which are mutually referenced in the two data tables and relation types between the two data tables; the relationship types include: one-to-many, many-to-one, one-to-one, and many-to-many.

3. The method of claim 1, wherein designing an referencing template in connection with the metadata description model configuration, completing data referencing based on the referencing template, comprises:

Creating a lead task and associating the lead task with a lead template; the leading template comprises data source information, data destination information, metadata description model information, leading and filtering rules, leading and receiving data conversion rules, leading and receiving data verification rules and leading and receiving data quality evaluation rules;

and automatically analyzing the lead template to execute the lead task.

4. A method according to claim 3, wherein in the docking template:

the data source information includes: data source type, data source access address, data source name, and data source description;

The data destination information comprises a data destination type, a data destination access address, a data destination name and a data destination description;

The guiding and filtering rule filters the data content to be guided from the center table by limiting the filtering condition of each data field on the center table, and extracts the required data content from the related table by the relation information among the data tables in the metadata description model information;

the leading data conversion rule configures and records the corresponding data field format conversion rules of the source terminal and the destination terminal, and supports the automatic conversion from the source terminal data format to the destination terminal data format;

the leading data checking rule is used for checking the normalization, the integrity and the consistency of the data after leading the data to the local;

the quality evaluation rule of the leading data is used for comprehensively evaluating the quality of the leading data so as to determine whether to store the data.

5. The method of claim 4, wherein the splice data verification rules include field verification, entity verification, relationship verification;

The field verification comprises non-empty verification, value field verification and format verification;

the entity verification comprises entity data element integrity verification and entity data repeatability verification;

The relationship check includes a consistency check of the relationship between the entity tables and the dictionary tables.

6. The method of claim 4, wherein the configuration of the splice data quality assessment rules comprises: scoring rules and weight settings for various quality problems;

the quality problems include:

1) The problem that missing data recorded during monitoring of the leading process and data conversion are incorrect;

2) When the data is reorganized, the problems of incorrect data format, incorrect data value precision and irregular data content are found by utilizing the leading data verification rule;

3) Problems are found and recorded after manual review of the data structure and content.

7. The method of claim 1, wherein the indexing template further comprises: the splicing performance parameter information is used for configuring splicing performance parameters when the splicing task is executed, and comprises the following steps: whether multithreading, and the size of the occupied data cache.

8. A data model-based visual data interfacing apparatus, the apparatus comprising:

The description information acquisition module is used for acquiring and perfecting the description information of all the data tables of the data source end and marking the table type of each data table; the table types include: an entity table, a relationship table, and a dictionary table;

The leading data model construction module is used for selecting an entity table containing the identification information of the service object from the data table of the data source end as a center table of the leading data model facing the service object to be led, selecting entity tables and dictionary tables which are related to the center table and can describe the attribute of the business object, adding the entity tables and the dictionary tables into the leading data model, and configuring the relation information among the data tables in the leading data model;

The description template filling module is used for storing the description information of the data table in the leading data model, the description information of the data field in the data table and the relation information among the data tables into a pre-constructed multi-dimensional metadata description template to obtain a metadata description model corresponding to the business object; the multidimensional metadata description template specifically comprises the following components: a data table metadata table, a data field metadata table, and a data table relationship metadata table; the data table metadata table is used for describing metadata information of all data tables in the data model corresponding to the service data object; a business data object corresponds to a data model; the data field metadata table is used for describing metadata information of all data fields contained in each data table in the data model; the data table relation metadata table is used for describing metadata information of relations among all data tables in the data model;

The lead task execution module is used for configuring and designing a lead template by combining the metadata description model, and completing data lead based on the lead template;

The leading data model construction module is also used for selecting an entity table containing service object identification information from the data tables of the data source end as a center table of the leading data model, and displaying the entity table on the visual interface by a first icon; selecting entity tables which are related to the center table and can describe the attribute of the business object, taking the entity tables as other entity tables of the leading data model, displaying the entity tables on a visual interface by a second icon, connecting lines between the center table and the existing entity tables formed by the other entity tables on the visual interface, and marking associated fields between the two corresponding entity tables on the lines; if the two entity tables are associated through the relation table, the corresponding relation table needs to be marked on line, and the two entity tables are respectively associated with the associated fields of the relation table; and selecting a dictionary table which is related to the center table and can describe the attribute of the business object, taking the dictionary table as a dictionary table of the leading data model, displaying the dictionary table by a third icon, connecting the existing entity table and the dictionary table on a visual interface, and marking the associated field between the corresponding entity table and the dictionary table on the line.