RU2813110C1 - Method and system for automatic visualization of large data arrays using search and isolation of their construction patterns - Google Patents

Abstract

FIELD: data processing.

SUBSTANCE: method and system for automatic visualization of large data arrays. Technical result is achieved by implementing a method for automatically visualizing large data arrays by searching and highlighting their construction patterns, in which: datasets are obtained and approximated, highlighting a simplified representation of the source data and obtaining digital masks that store their construction patterns; based on the presented datasets, a visualization is built and metafiles with the corresponding visualization parameters are generated; metafiles are associated with previously obtained digital masks; when a new dataset is received, it is approximated and a digital mask is obtained, which is compared with previously generated digital masks and, if a match is found, the corresponding metafiles with visualization parameters are linked; form a visualization of a new dataset using the metafiles linked in the previous step.

EFFECT: expanding the scope of technical means for automatically visualizing large amounts of data by searching and highlighting their construction patterns, increasing the accuracy of visualization of the results of analyzing large data arrays.

14 cl, 9 dwg

Description

TECHNICAL FIELD

[1] This technical solution, in general, relates to the field of computer technology, namely to a method for automatically visualizing large amounts of data by searching and highlighting their construction patterns.

BACKGROUND OF THE ART

[2] Currently, information systems that offer the user the ability to visualize when analyzing data, i.e. The functionality for creating graphs that describe data contains a limited set of graph types, for example: pie chart, time graph, pivot table, funnel, etc. Next, the user independently selects the most informative type of chart from his point of view. This work requires high qualifications, the result is always subjective, and the work can take a long time.

[3] At the same time, in existing approaches to visualizing the results of data analysis in well-known BI systems, the graph, as a rule, is configured for a combination of filters, which limits the possibilities of automating the visualization process.

[4] A technical solution is known from the prior art, described in RU 2386172 “EDITING TEXT OF ARBITRARY GRAPHICS THROUGH A HIERARCHICAL LIST”, 06/29/2005, MICROSOFT CORPORATION (US): “a method of graphics software that automatically creates graphic content when the user enters or modifies the hierarchical list of content data and selects one of a variety of selections of graphical definitions of graphical elements. In addition, the present invention includes a user interface with three different views. The first display presents the user with one or more galleries, including a gallery with multiple graphic definitions. The second display is the content input area. And the final display is a canvas to paint on.” A tool for selecting the type and setting up a chart by a person is described; there is no talk about automation.

[5] A technical solution is known from the prior art, described in RU 2488159 “RANKING VISUALIZATION TYPES BASED ON SUITABILITY FOR VISUALIZATION OF A DATA SET”, 03.30.2009, MICROSOFT CORPORATION (US): “Visualization metadata is generated for each of many different visualization types, and Visualization metadata contains data describing one or more attributes of a visualization type, specific axes, and a specific data sequence for each specific visualization type. Data set metadata is generated for the data set, wherein the data set metadata contains data describing one or more attributes of said data set. Suitability scores are computed for each of the plurality of different visualization types based on the visualization metadata and the dataset metadata, ranking the plurality of different visualization types based on the computed suitability scores. It is determined whether each of the calculated fitness scores exceeds a threshold. Display visual representations of each of the plurality of different types of visualizations on the user interface when the calculated fitness score exceeds a threshold. "Accept a selection of one of the displayed visual representations and present the data set using visualization types corresponding to the selected visual representation."

At the same time, this solution analyzes graphs from a limited set and analyzes their metadata, which limits the possibilities of automation.

[6] The Python AutoViz library is also known from the prior art: it performs visualization in the form of diagrams from a flat text file. The library receives a file (Excel, CSV, txt or json) as input, and AutoViz visualizes it with charts and graphs. It does not have its own interactive graphical interface. Interaction with the library occurs programmatically, through input parameters.

[7] The disadvantages of these solutions are the specific conditions for automating the process of visualizing the results of analyzing large amounts of data, or the lack of automation; in addition, existing solutions do not visualize accurately enough, with significant distortions.

DISCLOSURE OF INVENTION

[8] This technical solution is aimed at eliminating the disadvantages inherent in existing solutions known from the prior art.

[9] The technical problem solved in this technical solution is to provide the ability to automatically visualize large amounts of data by searching and highlighting their construction patterns.

[10] The main technical result that appears when solving the above problem is the expansion of the arsenal of technical means by providing the ability to automatically visualize large amounts of data by searching and highlighting their construction patterns. An additional technical result is to increase the accuracy of visualization of the results of analyzing large amounts of data.

[11] The specified technical results are achieved thanks to the implementation of a method for automatically visualizing large amounts of data by searching and highlighting their construction patterns, implemented using a processor and a data storage device, including the following steps:

• receive datasets and approximate them, highlighting a simplified representation of the source data and obtaining digital masks that store the patterns of their construction;

• based on the presented datasets, a visualization is built and metafiles with the corresponding visualization parameters are generated;

• metafiles generated at the previous step are associated with previously obtained digital masks;

• when receiving a new dataset for visualization, it is approximated and a digital mask is obtained;

• the resulting digital mask is compared with previously generated digital masks and, if a match is found, the corresponding metafiles with visualization parameters are linked;

• create a visualization of a new dataset using the metafiles linked in the previous step.

[12] In one of the particular examples of the method, metafiles store visualization parameters excluding peak values.

[13] In another particular example of the method, metafiles store rendering parameters with transposition.

[14] In another particular example of the method, metafiles store visualization parameters with the addition of a trend with characteristics of the intensity of change.

[15] In another particular example of the method, metafiles store visualization parameters with a combination of periods.

[16] In another particular example of the method, metafiles store visualization parameters with selection of the scale type along the axes.

[17] In another particular example of the method, metafiles store visualization parameters with the installation of minimum and maximum restrictions on the axes.

[18] In addition, the stated technical results are achieved through a system for automatically visualizing large amounts of data by searching and highlighting their construction patterns, containing:

• at least one data processing device;

• at least one data storage device;

• at least one program, wherein the one or more programs are stored on one or more storage devices and executed on one or more processing devices, the one or more programs performing the following steps:

• receive datasets and approximate them, highlighting a simplified representation of the source data and obtaining digital masks that store the patterns of their construction;

• based on the presented datasets, a visualization is built and metafiles with the corresponding visualization parameters are generated;

• metafiles generated at the previous step are associated with previously obtained digital masks;

• when receiving a new dataset for visualization, it is approximated and a digital mask is obtained;

• the resulting digital mask is compared with previously generated digital masks and, if a match is found, the corresponding metafiles with visualization parameters are linked;

• create a visualization of a new dataset using the metafiles linked in the previous step.

[19] In one of the particular examples of the implementation of the system, metafiles store visualization parameters with the exception of peak values.

[20] In another particular embodiment of the system, metafiles store transposed rendering parameters.

[21] In another particular example of the implementation of the system, metafiles store visualization parameters with the addition of a trend with characteristics of the intensity of change.

[22] In another particular example of the implementation of the system, metafiles store visualization parameters with a combination of periods.

[23] In another particular example of the implementation of the system, metafiles store visualization parameters with selection of the scale type along the axes.

[24] In another particular example of the implementation of the system, metafiles store visualization parameters with the installation of minimum and maximum restrictions on the axes.

BRIEF DESCRIPTION OF THE DRAWINGS

[25] The features and advantages of the present technical solution will become apparent from the following detailed description and the accompanying drawings, in which:

Fig. 1 illustrates a block diagram of the claimed method.

Fig. Figure 2 illustrates an example of a simplified graph representation.

Fig. Figure 3 illustrates an example of constructing a digital mask based on a dataset.

Fig. Figure 4 illustrates an example of dataset approximation.

Fig. Figure 5 illustrates an example of a simplified dataset mask.

Fig. 6 illustrates examples of a graph with icons showing the intensity of change (growth or fall) of the indicator.

Fig. 7 illustrates the monthly history of some indicator over 2 years.

Fig. 8 illustrates an example of fusion imaging.

Fig. 9 illustrates a system for implementing the claimed method.

IMPLEMENTATION OF THE INVENTION

[26] The terms and concepts necessary for the implementation of this technical solution will be described below.

[27] A data set or dataset is a collection of data. In the case of tabular data, a data set corresponds to one or more database tables, where each table column corresponds to a different variable, and each row corresponds to a record in the data set. Datasets store values for each variable, such as the height and weight of an object, for each member of the dataset. Datasets can also consist of a collection of documents or files.

[28] The claimed technical solution can be implemented, for example, by a system, machine-readable medium, server, etc. In this technical solution, a system means, inter alia, a computer system, a computer (electronic computer), CNC (computer numerical control), PLC (programmable logic controller), computerized control systems and any other devices capable of executing a given, clearly defined sequence operations (actions, instructions).

[29] A command processing device means an electronic unit or an integrated circuit (microprocessor) that executes machine instructions (programs).

[30] An instruction processing device reads and executes machine instructions (programs) from one or more storage devices, such as devices such as random access memory (RAM) and/or read only memory (ROM). ROM can be, including, but not limited to, hard drives (HDD), flash memory, solid-state drives (SSD), optical storage media (CD, DVD, BD, MD, etc.), etc.

[31] Program - a sequence of instructions intended for execution by a computer control device or a command processing device.

[32] The term "instructions" as used in this application may refer generally to software instructions or software commands that are written in a given programming language to perform a specific function, such as, for example, retrieving and processing data, generating a user profile , reception and transmission of signals, analysis of received data, user identification, etc. Instructions can be implemented in a variety of ways, including, for example, object-oriented methods. For example, instructions can be implemented using the programming language C++, Java, Python, various libraries (for example, Microsoft Foundation Classes), etc. Instructions that carry out the processes described in this solution can be transmitted over both wired and wireless data channels, for example, Wi-Fi, Bluetooth, USB, WLAN, LAN, etc.

[33] The presented method for automatic visualization of large amounts of data by searching and highlighting their construction patterns (Fig. 1 shows a diagram of the method) solves the problem of providing the ability to automatically visualize large amounts of data by searching and highlighting their construction patterns and increasing the accuracy of visualization of analysis results large amounts of data by sequentially performing the following steps:

• receive datasets and approximate them, highlighting a simplified representation of the source data and obtaining digital masks that store the patterns of their construction;

• based on the presented datasets, a visualization is built and metafiles with the corresponding visualization parameters are generated;

• metafiles generated at the previous step are associated with previously obtained digital masks;

• when receiving a new dataset for visualization, it is approximated and a digital mask is obtained;

• the resulting digital mask is compared with previously generated digital masks and, if a match is found, the corresponding metafiles with visualization parameters are linked;

• create a visualization of a new dataset using the metafiles linked in the previous step.

[34] At the first stage of the method, the dataset is “simplified” and this simplified representation of the graph is the setting key (in the pair “setting key - setting value”). Simplification occurs on the basis of data approximation techniques and algorithms. The choice of a specific method for approximating the original dataset is not important for the described technical solution. Dataset approximation is necessary to obtain a simplified representation of the source data in order to automate the reuse of optimal settings for the graphical representation of data based on such a representation. Automation is achieved due to the fact that the approximated data becomes a digital mask of the original data and has the following properties:

• built on the basis of initial data;

• reflect the characteristic features of the source data;

• have a small size for easy storage and connection with visualization settings.

[35] The digital mask obtained in this way will be the configuration key. At the same time, direct application of convolution functions (hash functions) to the original data, without approximation, will give a unique fingerprint of this particular data and will not allow comparison of datasets in order to identify similarity.

[36] In the described solution, the required configuration key, on the one hand, is a direct image of the approximated dataset, stores information about its characteristic features, and on the other hand, is not a strict identifier of the original dataset. Finding similarity using the identifiers obtained in this way allows you to recognize data that has a similar value distribution pattern, the presence of periodic cyclicity, and other characteristic features, and automatically apply previously selected optimal settings for visualization. For example, a simplified representation of the graph in FIG. 2 corresponds to a digital mask:

This value approximately reflects the composition of the source data, and it is quite compact, which allows it to be used as a key to build a connection with a set of visualization settings.

For example, in FIG. 2, the X and Y scales are divided into N parts and a “pixel” representation of the graph is built - if the graph falls into a square, then the value of the square becomes equal to 1. Thus, in this example, the dataset is “simplified” to a two-dimensional N * N array and from there simplified representation, it is easy and quick to calculate the hash value. In the example in FIG. Figure 2 illustrates a coarsened (pixelated) representation of the graph of the sine function. Or, for example, a derivative is taken from the graph, the value of which can always be positive. Or, for example, you can take the minimum and maximum scale values from the graph. Initially, when there is no (or not found) configuration metadata for the analyzed dataset, visualization occurs based on the original data “as is” without applying any settings. Next, having become familiar with this initial graph, the user can use the system to search for a more optimal (convenient for analysis) display of the data graph, including:

- exclusion of peak values;

- transposition;

- supplementing the trend with characteristics of the intensity of change;

- combination of periods;

- selection of scale type along the axes;

- setting min/max limits on axes.

The visualization settings obtained at this stage are saved, and the search key for their reuse becomes the digital mask of the original dataset:

[digital mask] => [set of settings]

For example: see the table below (link Mask -> Settings)

Let's consider an example of constructing a digital mask based on the dataset presented in Fig. 3.

We approximate this dataset with a step of 4 along the X and Y axes (in Fig. 4).

Thus, we will obtain a simplified mask of the original dataset that meets our requirements - close to the original one, but simpler: shown in Fig. 5.

If the size of the original dataset is large enough that one iteration of approximation with the selected step produces more than 1000 points, the operation should be repeated until a mask of less than 1000 points is obtained.

[37] Obtaining digital masks of datasets is based on well-known data approximation techniques and algorithms; it is possible to use ready-made libraries that implement the necessary approximation functionality. The search and communication functionality is implemented at the database level in the SQL language.

[38] As an additional example of practical application, we present the exclusion of points with peak values from the analysis. In manual mode, point exclusion is implemented through user interaction with the points/nodes of the graph. In automatic mode, using the described technical solution, points with significantly different values from the main array of values are identified. Then the order of the dimension of the difference is established: 10 times, 100 times, etc. If such points/nodes are found, they are excluded (or proposed for exclusion) from the graph visualization.

[39] To identify points with peak values, the average value of the indicator under study is first calculated within the calendar boundaries presented for analysis. Then the order of deviation of values from the average for the data set as a whole is assessed. The evaluation occurs sequentially from higher to lower order. If values are found that are 1000 times greater/less than the average, they are excluded (or suggested for exclusion) during visualization. If there is no such order of deviations, a search is made for a value with a lower order of 100 times/10 times, etc. In this case, the user has the opportunity to control the deviation order step or stop the algorithm at an arbitrary step.

[40] Within the framework of this task, to assess deviations, it is permissible to use ready-made libraries that implement common statistical techniques. The novelty of the application is achieved through the use of this tool already during visualization with the ability to manually configure the process by the user to find the most suitable way to graphically represent data.

[41] As another additional example of practical application, we present the identification and designation of a graph of characteristics of the intensity of change on a trend.

Sections of the graph are identified that have:

• constant value of the derivative (section with a uniform change in the indicator);

• monotonic change of the first-order derivative and the second-order derivative is unchanged (section of monotonic acceleration/deceleration of the change in the indicator);

• monotonic change of the second-order derivative and the third-order derivative is unchanged (a section of intensive acceleration/deceleration of changes in the indicator). Then, in accordance with the identified areas, pictograms are placed on the graph, illustrating the intensity of change (growth or fall) of the indicator (in Fig. 6).

[42] As an additional example of practical application, we present the combination of segments of periodic (calendar) trends to identify the cyclical nature of changes (or deviations of cyclicality). For example, in FIG. Figure 7 consistently shows the monthly history of a certain indicator for 2 years.

Using the described technical solution, the presence of a complete monthly history for a 2-year period is automatically detected and a combined visualization option is offered, see Fig. 8.

The combined visualization will thus make it possible to clearly assess the dynamics of the indicator in a period-to-period comparison.

[43] In general (see Fig. 9), a system for automatically visualizing large amounts of data by searching and highlighting their construction patterns (900) contains one or more processors (901) united by a common information exchange bus, memory devices such as RAM (902) and ROM (903) and input/output interfaces (904).

[44] The processor (901) (or multiple processors, multi-core processor, etc.) may be selected from a variety of devices commonly used today, such as from manufacturers such as: Intel™, AMD™, Apple™, Samsung Exynos ™, MediaTEK™, Qualcomm Snapdragon™, etc. The processor or one of the processors used in the system (900) must also include a graphics processor, for example an NVIDIA GPU with a CUDA-compatible programming model or Graphcore, the type of which is also suitable for carrying out the method in whole or in part, and can also be used for training and application of machine learning models in various information systems.

[45] RAM (902) is a random access memory and is designed to store machine-readable instructions executable by the processor (901) to perform the necessary logical data processing operations. RAM (902) typically contains executable operating system instructions and associated software components (applications, program modules, etc.). In this case, the available memory capacity of the graphics card or graphics processor can act as RAM (902).

[46] A ROM (903) is one or more permanent storage devices, such as a hard disk drive (HDD), a solid state drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media ( CD-R/RW, DVD-R/RW, BlueRay Disc, MD), etc.

[47] To organize the operation of device components (900) and organize the operation of external connected devices, various types of I/O interfaces (904) are used. The choice of appropriate interfaces depends on the specific design of the computing device, which can be, but is not limited to: PCI, AGP, PS/2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS/Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[48] To ensure user interaction with the device (900), various means (905) of I/O information are used, for example, keyboard, display (monitor), touch display, touch pad, joystick, mouse, light pen, stylus, touch panel, trackball, speakers, microphone, augmented reality tools, optical sensors, tablet, light indicators, projector, camera, biometric identification tools (retina scanner, fingerprint scanner, voice recognition module), etc.

[49] The network communication means (906) provides data transmission via an internal or external computer network, for example, Intranet, Internet, LAN, etc. One or more means (906) can be used, but not limited to: an Ethernet card, GSM modem, GPRS modem, LTE modem, 5G modem, satellite communication module, NFC module, Bluetooth and/or BLE module, Wi-Fi module, etc.

[50] The specific selection of device elements (900) for implementing various software and hardware architectural solutions can vary while maintaining the required functionality provided. In particular, such an implementation can be performed using electronic components used to create digital integrated circuits. Without limitation, microcircuits can be used, the logic of which is determined during manufacture, or programmable logic integrated circuits (FPGAs), the logic of which is specified by programming. For programming, programmers and debugging environments are used to set the desired structure of a digital device in the form of a circuit diagram or program in special hardware description languages: Verilog, VHDL, AHDL, etc. An alternative to FPGAs are: programmable logic controllers (PLCs), basic matrix crystals ( BMK) requiring a factory production process for programming; ASICs are specialized custom large-scale integrated circuits (LSIs), which are significantly more expensive for small-scale and single-piece production. Thus, implementation can be achieved by standard means based on the classical principles of implementing the fundamentals of computer technology.

[51] The submitted application materials disclose preferred examples of implementation of a technical solution and should not be interpreted as limiting other, particular examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field of technology.

Claims (29)

1. A method for automatically visualizing large amounts of data by searching and highlighting their construction patterns, implemented using a processor and a data storage device, including the following steps: receive datasets and approximate them, highlighting a simplified representation of the source data and obtaining digital masks that store patterns of their construction; Based on the presented datasets, a visualization is built and metafiles with the corresponding visualization parameters are generated; the metafiles generated in the previous step are associated with previously obtained digital masks; when receiving a new dataset for visualization, it is approximated and a digital mask is obtained; the resulting digital mask is compared with previously generated digital masks and, if a match is found, the corresponding metafiles with visualization parameters are linked; form a visualization of a new dataset using the metafiles linked in the previous step. 2. The method according to claim 1, characterized in that the metafiles store visualization parameters excluding peak values. 3. The method according to claim 1, characterized in that the metafiles store visualization parameters with transposition. 4. The method according to claim 1, characterized in that the metafiles store visualization parameters with the addition of the trend with characteristics of the intensity of change. 5. The method according to claim 1, characterized in that the metafiles store visualization parameters with a combination of periods. 6. The method according to claim 1, characterized in that the metafiles store visualization parameters with selection of the type of scale along the axes. 7. The method according to claim 1, characterized in that the metafiles store visualization parameters with the installation of minimum and maximum restrictions on the axes. 8. A system for automatic visualization of large data sets by searching and highlighting their construction patterns, containing: at least one data processing device; at least one data storage device; at least one program, wherein the one or more programs are stored on one or more storage devices and executed on one or more processing devices, wherein the one or more programs perform the following steps: receive datasets and approximate them, highlighting a simplified representation of the source data and obtaining digital masks that store patterns of their construction; Based on the presented datasets, a visualization is built and metafiles with the corresponding visualization parameters are generated; the metafiles generated in the previous step are associated with previously obtained digital masks; when receiving a new dataset for visualization, it is approximated and a digital mask is obtained; the resulting digital mask is compared with previously generated digital masks and, if a match is found, the corresponding metafiles with visualization parameters are linked; form a visualization of a new dataset using the metafiles linked in the previous step. 9. The system according to claim 1, characterized in that the metafiles store visualization parameters excluding peak values. 10. The system according to claim 1, characterized in that the metafiles store visualization parameters with transposition. 11. The system according to claim 1, characterized in that the metafiles store visualization parameters with the addition of the trend with characteristics of the intensity of change. 12. The system according to claim 1, characterized in that the metafiles store visualization parameters with a combination of periods. 13. The system according to claim 1, characterized in that the metafiles store visualization parameters with selection of the type of scale along the axes. 14. The system according to claim 1, characterized in that the metafiles store visualization parameters with the installation of minimum and maximum restrictions on the axes.