CN114154962B

CN114154962B - Batch monitoring method, device and equipment

Info

Publication number: CN114154962B
Application number: CN202111486700.2A
Authority: CN
Inventors: 赖海滨; 高伟钦; 翁世清
Original assignee: China Construction Bank Corp
Current assignee: China Construction Bank Corp
Priority date: 2021-12-07
Filing date: 2021-12-07
Publication date: 2024-12-24
Anticipated expiration: 2041-12-07
Also published as: CN114154962A

Abstract

The embodiment of the application provides a batch processing monitoring method, a device and equipment. The method comprises the steps of obtaining a job of an application plan operation according to configuration information of the job, generating a job operation schedule, obtaining key operation indexes of the job through operation history data of the job, generating a job operation history index table, obtaining the completed job through the job operation schedule to calculate and display the progress of application completion, and combining the job operation history index table to display abnormal conditions and drill information of the abnormal job to obtain an abnormal job list of the application. The technical scheme of the application has the capability of global monitoring, can control the overall processing progress in real time through overall batch processing monitoring of the dimension organization of the scene, improves the operation and maintenance efficiency, can perform operation and maintenance at fixed points through the abnormal operation list of each application in the scene, and can lock the operation range needing to be monitored in advance through the operation schedule without waiting for the completion of instantiation of the operation flow, thereby improving the accuracy of monitoring.

Description

Batch processing monitoring method, device and equipment

Technical Field

The present application relates to the field of computer technologies, and in particular, to a batch processing monitoring method, apparatus, device, and processor.

Background

The traditional batch processing scheduling monitoring faces the following challenges that the daily operation workload is increased along with the increase of the business, the relation between the operation and the operation is also more and more mixed, the batch running monitoring efficiency is low, the importance of different processing links is different from the global monitoring point of view, the important links cannot be monitored in a targeted manner, the real-time monitoring difficulty is high, the processing progress of each application cannot be displayed in real time due to the complexity of the batch processing business, and the abnormal condition existing in the current batch processing and the 'critical path' affecting the final target application are obtained.

Therefore, the job scheduling system has the following problems in the application based on the traditional batch processing scheduling monitoring, namely the provided monitoring basically has fine granularity of job flow or job and the like, is not displayed from a higher dimension, and partial products can display the dependency relationship and processing condition among all jobs on one canvas, so that the defect that the user experience is poor, and the actual business meaning is often disordered and difficult to embody is caused.

Disclosure of Invention

The embodiment of the application aims to provide a batch processing monitoring method, a device, equipment, a storage medium and a processor.

Therefore, a batch processing monitoring method is established, wherein the batch processing monitoring method is displayed in a scene dimension, can display the processing progress and the abnormal operation list of each application specific business date/batch in the scene in real time, and supports drill down to obtain operation examples with finer granularity, and can solve at least one of the following technical problems:

(1) The traditional scheduling software mainly starts from the dimension of a job flow, and the abstract dimension is insufficient, so that many jobs are always displayed on one canvas on monitoring display, which is similar to a spider web, and under the condition of large workload, the real-time refreshing cannot be realized due to the fact that the calculated amount is too large;

(2) The traditional scheduling software is generally only provided for monitoring on an instantiated job flow/job instance, does not well utilize historical operation information, such as the average operation time of a job, the normal starting time of the job and the like, does not combine with data in a planned state for monitoring, can only enable a user to perceive until the job has failed to operate, and cannot combine the historical data more effectively for real-time monitoring;

(3) Traditional dispatch software generally only provides one-level monitoring, lacks the function of boring down, can't better satisfy the operation and maintenance demand.

In order to achieve the above purpose, a first aspect of the present application provides a batch processing monitoring method, which includes obtaining a job scheduled to run by an application according to configuration information of the job, generating a job running schedule, obtaining key running indexes of the job through running history data of the job, generating a job running history index table, obtaining a job which has been run to completion through the job running schedule to calculate and display progress of completion of the application, displaying abnormal conditions and drilling information of the abnormal job in combination with the job running history index table, and obtaining an abnormal job list of the application.

The method comprises the steps of identifying end node applications of a scene, wherein one scene corresponds to one end node application, determining the dependency relationship between the end node applications and the jobs of the corresponding scene, and gradually pushing and calculating each node application forwards according to the dependency relationship between the jobs to obtain an AOE network of the scene, so that an actual job list is generated.

Further, identifying end node applications of the scenes comprises classifying each scene according to configuration information of the job, and identifying target applications associated with each scene as end node applications.

Further, obtaining the key operation index of the job through the operation history data of the job includes taking the median of the operation history data as the key operation index of the job.

Further, the key operation indexes of the job comprise execution time of the job, a start operation time node of the job and an end operation time node of the job.

Further, the method includes regularly refreshing the abnormal job list, and keeping the refreshed data in a cache so as to directly read the refreshed data.

Further, the job running schedule is used for obtaining the running completed jobs to calculate and display the completion progress of each application, and the method comprises the steps of polling the current completion situation of the jobs according to the number of the jobs contained in each application in a scene, wherein the real-time progress of each application = the number of the successfully executed jobs/the total number of the jobs contained in the application in the scene, and representing the overall progress of the scene by the progress of the application of the end node.

Further, displaying abnormal conditions and drilling information of abnormal operations, and obtaining an abnormal operation list of each application, wherein the abnormal operation list comprises the steps of obtaining a path with the maximum path length from an AOE network of a scene, marking the path as a critical path, and marking the abnormal conditions according to the critical path, wherein the abnormal conditions comprise marking operation failure, operation overtime and operation delay starting.

The method of the application monitors the global batch processing condition by the dimension of the scene, and automatically calculates and constructs the whole AOE graph by identifying the end point application and the end point application job set of the scene, thereby greatly reducing the configuration quantity of users.

The application provides a batch processing monitoring device which comprises a first module, a second module, a third module and a fourth module, wherein the first module is used for acquiring a job of planning operation of an application according to configuration information of the job and generating a job operation schedule, the second module is used for acquiring key operation indexes of the job through operation history data of the job and generating a job operation history index table, the third module is used for acquiring the completed job through the job operation schedule to calculate and display the progress of completion of the application, and the fourth module is used for combining the job operation history index table, displaying abnormal conditions and drilling information of the abnormal job and acquiring an abnormal job list of the application.

In the embodiment of the application, the first module is configured to classify each scene according to the configuration information of the job, identify the target application associated with each scene as an end node application, identify the end node application of the scene, wherein one scene corresponds to one end node application, determine the dependency relationship between the end node application and the job of the corresponding scene, and gradually and forwards push out each node application according to the dependency relationship between the jobs to obtain the AOE network of the scene, thereby generating the actual job list.

Further, the second module is configured to poll the current completion of the jobs according to the number of jobs contained in each application in the scene, wherein the real-time progress of each application = the number of successfully executed jobs/the total number of jobs contained in the application in the scene, and to characterize the overall progress of the scene by the progress of the end node application.

Further, the fourth module is configured to obtain abnormal condition classification, calculate the proportion of abnormal operations according to the abnormal condition classification, and conduct distinguishing display according to the proportion of the abnormal operations.

The technical effect of the batch processing monitoring device is the same as that of the batch processing monitoring method.

Another aspect of the application provides a processor configured to perform the batch monitoring method described above.

Another aspect of the application provides a batch monitoring apparatus comprising a processor and a memory, the processor configured to perform the batch monitoring method.

A fifth aspect of the application provides a computer program product comprising a computer program which, when executed by a processor, implements the batch monitoring method.

A machine-readable storage medium having instructions stored thereon that, when executed by a processor, cause the processor to be configured to perform the batch monitoring method.

By the technical scheme, the overall processing progress is controlled in real time through overall batch processing monitoring of the dimension organization of the scene, the operation and maintenance efficiency is improved, operation and maintenance personnel can perform operation and maintenance on the operations at fixed points through the abnormal operation list of each application in the scene, and the labor investment of the operation and maintenance personnel is reduced.

According to the technical scheme, the overall batch processing condition is monitored by the dimension of the scene, each scene represents batch processing logic with business meaning, and the whole AOE graph is automatically calculated and constructed by identifying the end point application and the end point application job set of the scene, so that the configuration quantity of a user is greatly reduced.

The scene is characterized by utilizing AOE, so that the relation of each application in the scene is conveniently displayed, the key path is calculated, the current operation condition of the job is judged to be normal or not by combining the historical data, the job set which needs to be operated by the specific service date/batch of the application is calculated by combining the job operation schedule, the monitoring range is locked, and the job flow is not required to wait for the completion of the instantiation.

Additional features and advantages of embodiments of the application will be set forth in the detailed description which follows.

Drawings

The accompanying drawings are included to provide a further understanding of embodiments of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain, without limitation, the embodiments of the application. In the drawings:

FIG. 1 schematically illustrates a prior art Control-M system architecture block diagram;

FIG. 2 schematically illustrates a flow diagram of a method of batch monitoring in accordance with an embodiment of the application;

FIG. 3A schematically illustrates a job execution schedule generation flow diagram according to an embodiment of the present application;

FIG. 3B schematically illustrates a job execution history index table generation flow chart according to an embodiment of the present application;

FIG. 3C schematically illustrates an abnormal job list generation flow diagram in accordance with an embodiment of the present application;

FIG. 4 schematically illustrates a scene configuration flow diagram according to an embodiment of the application;

FIG. 5 schematically illustrates an exemplary diagram of a scene AOE according to an embodiment of the application;

FIG. 6 schematically illustrates a block diagram of a batch processing monitoring apparatus according to an embodiment of the present application;

fig. 7 schematically shows an internal structural view of a computer device according to an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it should be understood that the detailed description described herein is merely for illustrating and explaining the embodiments of the present application, and is not intended to limit the embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

It should be noted that, if directional indications (such as up, down, left, right, front, and rear are referred to in the embodiments of the present application), the directional indications are merely used to explain the relative positional relationship, movement conditions, and the like between the components in a specific posture (as shown in the drawings), and if the specific posture is changed, the directional indications are correspondingly changed.

In addition, if there is a description of "first", "second", etc. in the embodiments of the present application, the description of "first", "second", etc. is for descriptive purposes only and is not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions of the embodiments may be combined with each other, but it is necessary to base that the technical solutions can be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, the combination of the technical solutions should be considered to be absent and not within the scope of protection claimed in the present application.

Fig. 1 schematically shows a prior art Control-M system architecture block diagram.

Referring to fig. 1, the technical scheme of the control-M system realizes the separation of three functions of management, scheduling and job execution through a three-layer architecture. The Control-M/EM of the first layer is a management module responsible for defining/uploading, running monitoring and the like of the job, the Control-M/Server of the second layer is a core of whole scheduling and is responsible for instantiation of a job flow, scheduling of the job, resource allocation, running management of the job and the like, the Control-M/Agent of the third layer is a job running node and is responsible for running the job issued by the Server, storing a job running state and synchronizing the job running state with the Control-M/Server. Wherein, the monitoring function is mainly completed by Control-M/EM. The principle is that the Control-M/EM obtains the dependency relationship between the jobs from the data of the Control-M/EM DB and obtains the running condition of the job instance from the data of the Control-M/Server to the data of the Control-M/Server DB in real time, and the user carries out real-time monitoring by specifying the target job flow/job set to be monitored.

The method has the defects that a user needs to select all operation flows/operations to be monitored, the user is required to be familiar with the monitored all-link processing process, the relation of the selected operations is displayed in one canvas, the dimension is thin, the better monitoring cannot be performed from the global angle, and the overall processing progress of the target operation monitored by the user cannot be reflected.

For the dispatching system (not shown in fig. 1) based on Airflow in the prior art, the technical scheme mainly monitors from the angle of the operation flow (namely the DAG), provides a tree structure, a graph structure, a Gantt chart and other modes for visual display, and can monitor the operation flow better. The tree-shaped and graph structure can conveniently display the running state of each operation instance in the operation flow, and the Gantt chart mainly analyzes the bottleneck point of the operation flow by analyzing the running start-stop time of each operation constituting the operation flow. The essence is also monitoring from a workflow point of view.

The scheduling system based on Airflow provides monitoring of the dimension of the workflow, lacks monitoring of higher dimension, namely the scene dimension monitoring of the application introduced below is difficult to meet the requirement of global batch processing monitoring, the monitoring is only to show the running state of the instantiated operation, and the workflow which is not yet instantiated cannot be shown.

For the DolphinScheduler-based scheduling system (not shown in fig. 1) in the prior art, the technical scheme is similar to the implementation mode of the Airflow-based scheduling system, and is also used for monitoring from the viewpoint of a workflow (same workflow), so that a tree graph is provided for showing the types and task states of task nodes, and furthermore, dolphinScheduler is also provided with a Gantt chart for completing the analysis capability of workflow bottlenecks. The technical disadvantages are the same as above.

The batch processing monitoring method provided by the application can be applied to an Internet application environment, for example. The Internet application environment comprises a terminal, a server and a network. The terminal communicates with the server via a network. The terminal may be, but not limited to, various personal computers, notebook computers, smartphones, tablet computers and portable wearable devices, and the server may be implemented by a separate server or a server cluster formed by a plurality of servers.

FIG. 2 schematically illustrates a flow diagram of a method of batch monitoring in accordance with an embodiment of the application. As shown in fig. 2, in an embodiment of the present application, a method for batch processing monitoring is provided, and this embodiment is mainly applied to the above terminal (or server) for illustration, and includes the following steps:

s202, acquiring the job of the application plan operation according to the configuration information of the job, and generating a job operation schedule.

According to an embodiment, the application is a plurality of applications, with several jobs under each application. An application may be understood simply as an item (project) under which multiple job streams may be included, with multiple jobs under each job stream. Each job belongs to a certain application.

Firstly, according to the input condition and the output condition of the initial configuration job of a user, the dependency relationship between the jobs is found out. When the input condition of the job is satisfied, the job is started.

And then converting into the dependency relationship between applications according to the dependency relationship between the jobs.

S204, acquiring key operation indexes of the job through operation history data of the job, and generating a job operation history index table. Specifically, the key operation indexes of the job comprise the execution time of the job, the start operation time node of the job and the end operation time node of the job.

S206, acquiring the completed job through the job operation schedule to calculate and display the progress of application completion.

S208, combining the operation history index table, displaying the abnormal condition and drilling the information of the abnormal operation, and obtaining an abnormal operation list of the application.

The application organizes the whole (global) batch processing monitoring through the dimension of the scene and controls the whole processing progress in real time. The operation and maintenance personnel can perform operation and maintenance on the operations at fixed points through the abnormal operation list of each application in the scene, and the labor investment of the operation and maintenance personnel is reduced.

FIG. 3A schematically illustrates a job execution schedule generation flow chart according to an embodiment of the present application.

Referring to fig. 3A, a detailed description is provided in connection with the method flow diagram of the present application of fig. 2.

According to the embodiment, the method of the technical scheme of the application integrally comprises three flows, namely a job operation schedule generation flow, a job operation history index refreshing flow and a scene AOE operation information refreshing flow (namely a main flow). The main function of the job operation schedule is to acquire a job list that each application needs to operate every day according to configuration information (frequency) of the job. The job configuration information used herein is mainly the execution frequency of the job, such as daily, weekly, monthly, the beginning of month, the end of month, etc., or a specific time point, such as 09:00 per day.

In S301, database scheduling configuration information is acquired. Since the real-time operation information of the scene needs to be calculated every day after the scene configuration is effective, the actual operation list (operation set) of the scene, which needs to be operated, of the current service date or batch can be obtained by firstly obtaining the operation set of the current service date or batch of the application of the terminal node (terminal point) and then calculating the operation set step by step according to the dependency relationship among the operations.

According to an embodiment, the job set is found according to the dependency relationship of the destination job, and belongs to all the jobs under the application. It will be appreciated that an endpoint job may be successfully executed and may be concerned with the upstream job on which it depends, where the job set is the job that needs to be concerned, which may also be referred to as the accent job.

According to the embodiment, the business date is understood to be the date of a batch of jobs, and is generally equal to the execution date of the jobs, and there are inconsistent situations.

In the present embodiment, the job execution schedule is regularly refreshed at S303. The preferred job operation schedule is refreshed once a day, or the job operation schedule can be refreshed periodically according to actual conditions.

According to an embodiment, the job execution schedule is set to be automatically loaded daily newday. Operation authority, selection scheduling date, job name, job type, command execution user, job permission scheduling date, such as daily scheduling, abnormal logic processing for job, scene information configuration.

The operation range to be monitored is locked in advance through the schedule, and the operation flow is not required to be waited for to complete the instantiation, so that the monitoring accuracy is greatly improved

Fig. 3B schematically illustrates a job operation history index table generation flow chart according to an embodiment of the present application.

Referring to fig. 3B, according to an embodiment, the job operation history index table functions to acquire key operation indexes of a job through history operation data of the job, and may also be understood as important operation indexes.

According to the embodiment, at S311, job flow information of successful execution is acquired, such as execution duration of the job, start operation time of the job (time node), end operation time of the job (time node).

At S313, the job run length, start time node, end time node are refreshed periodically according to the median method. Specifically, the method of calculation employs median acquisition of history data. The median here refers to the median of the job history operation time period. Some jobs may run daily, each day having a run length, and a stored procedure (Oracle) counts each day the median run length of all jobs.

At S315, the job operation history index table is refreshed.

Fig. 3C schematically shows an abnormal job list generation flow chart according to an embodiment of the present application.

Referring to fig. 3C, according to an embodiment, the scene AOE refreshing main flow may be refreshed every 5 minutes at S321, and at S323, the process proceeds to obtain the currently configured scene information, and at S235, all job sets that need to be executed for applying the service date or batch are obtained according to the job running schedule.

According to the embodiment, at S237, the number of jobs that each application has run successfully, the refresh progress information, and the jobs that have run completed are acquired to calculate the progress of the application.

At S239, by judging the job running pipeline information, the abnormal job list of each application is obtained in combination with the job running history index table, and the information of the abnormal job is further supported. The data refreshed by the scene AOE refreshing main flow are all kept in the Redis cache, and the Redis data are directly read from the page, so that the situation that the page cannot be refreshed due to excessive calculation amount is avoided.

Fig. 4 schematically shows a scene configuration flow diagram according to an embodiment of the application.

Referring to fig. 4, a scenario represents a batch business process logic, such as administrative reporting, data warehouse processing, etc., where the scenario itself is business-meaningful. The scene configuration process is as follows:

in S401, the scenes are classified according to the configuration information of the job. In particular, the scenario is determined according to the batch business logic of interest.

In S403, the target application associated with each scenario is identified as an end node application, specifically, the end point application that is the target application associated with the scenario is identified, which may also be understood as an end node application, and the end node application of each scenario is identified, where one scenario corresponds to one end node application.

At S405, a dependency relationship between the terminated node application and the job of the corresponding scenario is determined.

According to an embodiment, it is determined that the endpoint applies a set of jobs associated with the scenario, a business date/lot rule of interest (the rule will calculate the date, lot to be monitored based on the current natural date, such as T (1) for the day after the current system date, T (-1) for the day before the current system date, lot 4 digits make up, covering 1-9999, such as B (1) for lot 1), and a calculation level, which is primarily to determine the maximum level of automatic calculation.

And S407, gradually pushing forward each node application according to the dependency relationship among the jobs to obtain an AOE network of the scene, thereby generating the actual job list.

Because the scene is significant, one scene also represents an important batch processing logic, batch processing business can be better understood from the business angle through the scene combing process, so that the monitoring is more targeted, in the scene configuration process, a user only needs to determine the terminal application after identifying the scene, and the system automatically calculates and constructs the AOE graph of the scene, so that the configuration complexity is reduced, and the system is more friendly to the user.

According to the embodiment, the upstream application is automatically calculated by gradually pushing up according to the dependency relationship between the jobs in the system and the application to which the jobs belong, so that the AOE graph of the scene is calculated, and the AOE graph represents the processing link of the scene. After the scene configuration is finished, an actual job list can be obtained.

Fig. 5 schematically shows an exemplary diagram of a scene AOE according to an embodiment of the present application.

Referring to fig. 5, a typical scene AOE graphic is shown in fig. 5. The data which can be displayed by the method comprises the dependency relationship of the scene, the real-time processing progress and the abnormal situation display (namely whether the current progress is required to be reflected or not at a normal level). The following describes the implementation respectively.

The method comprises the steps of obtaining the completed jobs through the actual job list to calculate and display the completion progress of each application, and comprises the steps of polling the current completion situation of the jobs according to the number of the jobs contained in each application in a scene, wherein the real-time progress of each application = the number of the successfully executed jobs/the total number of the jobs contained in the application in the scene, and representing the overall progress of the scene by the progress of the application of an end node.

According to an embodiment, in the dependency relationship in fig. 5, by converting the dependency relationship between jobs into the dependency relationship between applications, an application to which an upstream job belongs will have an edge directed to a downstream application. For example, the origin application A1 is an upstream application of the application A2 and the application A3. For the side of application A1, the corresponding job sets upstream and downstream are determined.

According to an embodiment, the real-time processing schedule periodically polls the current completion of the jobs by a background thread according to the set of jobs contained by each application in the scenario of fig. 5. Specifically, the progress of the start application A1 is 100%, and the progress of the application A2 is shown as 30%. The calculation method is that the real-time progress of the application = the number of successfully executed jobs/the total number of jobs contained in the application in the present scenario.

According to an embodiment, the overall progress of the scene is characterized by the progress of the end-node application (end-point application), e.g. application A6 progress is 30%.

According to an embodiment, the abnormal conditions include three conditions, namely job operation failure, job operation timeout and job delay start. Displaying abnormal conditions and drilling operation instance information, and acquiring an abnormal operation list of each application, wherein the abnormal operation list comprises the steps of acquiring a path with the maximum path length from an AOE network of a scene, marking the path as a critical path, and marking the abnormal conditions according to the critical path, wherein the abnormal conditions comprise marking operation failure, operation overtime and operation delay starting.

According to the embodiment, the job operation failure can be acquired in real time according to the operation flow of the job, the job operation overtime and the job delay starting condition need to be acquired by combining historical operation data, the background calculates historical operation indexes (execution duration and starting time nodes) of each job through a fixed thread, and then whether the execution overtime and the delay starting condition exists in the job operation condition is compared in real time. In summary, the proportion of abnormal jobs= (number of failed jobs to run + number of overtime jobs to run + number of jobs to delay start)/the total number of jobs contained in the application in the present scenario.

According to an embodiment, the severity of the abnormal situation may be displayed differently by different colors. For example, 80-100% of the operational anomalies correspond to dark red, 50-80% to orange, 20-50% to pale yellow, 10-20% to blue, 0-10% to green.

According to an embodiment, when an application in a scene is drilled down, the state of each job in the set of jobs under the application is presented. Referring to Table 1, further jumps may be made to job instance pages by job name. In this way, the detailed processing of each job under the current application can be presented in more detail.

Application name	Job flow name	Job name	Service date	Batch of	Anomaly identification
						Application 1	Flowa	Job1	20210101	0001	Execution failure
Application 1	Flowb	Job2	20210101	0001	Delayed start
						Application 1	Flowc	Job3	20210101	0001	Normal state

Table 1 scenario application drill-down information

According to an embodiment, in the AOE graph described above, the path with the largest path length (sum of the durations of the individual activities on that path) from the start point to the end point is the critical path. For example, referring to FIG. 5, where A1-A2 pass through A4 to the endpoint application A6 as the critical path, the key path is calculated with the emphasis on how to define the weights of the edges. Two strategies are provided in the present application to define the weights of the edges. The longest running time value in the job set of the upstream application associated with the edge represents the weight value of the edge, for example, the upstream application A4 is A2 and A3, then the decision of the running time of the A2 application with 30% of the progress through the first strategy results in the critical paths A1 to A2 passing through A4 to reach the end point application A6, wherein the running time of each job can be represented by the median of the running time of the past 30 dates.

According to an embodiment, policy two is that the latest finishing job time in the job set of the upstream application associated with the edge represents (the finishing time of the job also takes the median of the finishing time of the job by 30 dates), and for the convenience of calculation, the latest finishing time needs to be converted into an integer value based on a reference value of a certain time to represent the weight value of the edge. After the weights of the edges are determined, a critical path can be calculated, and the path with the largest sum of the edge weights applied from the starting point to the end point is determined to be the critical path.

The application monitors the overall processing progress in real time through the overall batch processing of the dimension organization of the scene, improves the operation and maintenance efficiency, enables operation and maintenance personnel to operate and maintain the operations at fixed points through the abnormal operation list of each application in the scene, and reduces the manpower input of the operation and maintenance personnel.

Fig. 6 schematically shows a block diagram of a batch processing monitoring apparatus according to an embodiment of the application.

In one embodiment, as shown in fig. 6, a batch processing monitoring apparatus 600 is provided, including a first module 601, a second module 603, a third module 605, and a fourth module 607, wherein:

the first module 601 is configured to obtain a job that is scheduled to run by an application according to configuration information of the job, and generate a job running schedule.

The second module 603 is configured to obtain key operation indexes of the job through operation history data of the job, and generate a job operation history index table.

The third module 605 is configured to obtain, through the job running schedule, a job that has been run to completion to calculate and display the progress of the application completion.

The fourth module 607 is configured to combine the job operation history index table, show the abnormal situation, drill the information of the abnormal job, and obtain the abnormal job list of the application.

According to an embodiment, the first module 601 is configured to classify each scene according to the configuration information of the job, identify a target application associated with each scene as an end node application, identify end node applications of the scenes, wherein one scene corresponds to one end node application, determine a dependency relationship between the end node application and the job of the corresponding scene, and gradually push forward each node application according to the dependency relationship between the jobs to obtain an AOE network of the scene, thereby generating the actual job list.

According to an embodiment, the second module 603 is configured to poll the current completion of the job according to the number of jobs contained in each application in the scenario, wherein the real-time progress of each application = number of jobs that have been successfully executed/total number of jobs contained in the application in the scenario, and to characterize the overall progress of the scenario in terms of the progress of the end-node application.

According to an embodiment, the fourth module 607 is configured to obtain an abnormal situation classification, calculate a proportion of abnormal jobs from the abnormal situation classification, and perform a differentiated display according to the proportion of abnormal jobs.

The batch processing monitoring device comprises a processor and a memory, wherein the first module 601, the second module 603, the third module 605 and the fourth module 607 are all stored in the memory as program units, and the processor executes the program modules stored in the memory to realize corresponding functions.

The batch processing monitoring device has the beneficial effects as the batch processing monitoring method, and is not repeated herein.

The processor includes a kernel, and the kernel fetches the corresponding program unit from the memory. The kernel can be provided with one or more than one, and the batch processing monitoring method is realized by adjusting kernel parameters.

The memory may include volatile memory, random Access Memory (RAM), and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), among other forms in computer readable media, the memory including at least one memory chip.

The embodiment of the application provides a storage medium, on which a program is stored, which when executed by a processor, implements the batch monitoring method described above.

The embodiment of the application provides a processor which is used for running a program, wherein the batch processing monitoring method is executed when the program runs.

In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 7. The computer device includes a processor a01, a network interface a02, a memory (not shown) and a database (not shown) connected by a system bus. Wherein the processor a01 of the computer device is adapted to provide computing and control capabilities. The memory of the computer device includes internal memory a03 and nonvolatile storage medium a04. The nonvolatile storage medium a04 stores an operating system B01, a computer program B02, and a database (not shown in the figure). The internal memory a03 provides an environment for the operation of the operating system B01 and the computer program B02 in the nonvolatile storage medium a04. The database of the computer device is used to store batch monitoring data. The network interface a02 of the computer device is used for communication with an external terminal through a network connection. The computer program B02, when executed by the processor a01, implements a batch monitoring method.

It will be appreciated by those skilled in the art that the structure shown in FIG. 7 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

In one embodiment, the batch processing monitoring apparatus provided by the present application may be implemented in the form of a computer program that is executable on a computer device as shown in FIG. 7. The memory of the computer apparatus may store various program modules constituting the batch processing monitoring apparatus, such as the first module, the second module, and the third and fourth modules shown in fig. 6. The computer program of each program module causes a processor to execute the steps of the batch processing monitoring method of each embodiment of the present application described in the present specification.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

Computer-readable media include both permanent and non-permanent, removable and non-removable media, and information storage may be implemented by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.

The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the application are to be included in the scope of the claims of the present application.

Claims

1. A batch process monitoring method, the method comprising:

acquiring the operation planning of the application according to the configuration information of the operation, and generating an operation planning table of the operation, comprising:

identifying end node applications of scenes, wherein one scene corresponds to one end node application;

Determining a dependency relationship between the end node application and the jobs of the corresponding scene;

the application of each node is gradually and forwards pushed out according to the dependency relationship among the jobs to obtain the AOE network of the scene, thereby generating an actual job list,

Identifying end node applications of scenes, including classifying each scene according to configuration information of the job; identifying the target application associated with each scene as an end node application;

acquiring key operation indexes of the job through operation history data of the job, and generating a job operation history index table;

acquiring the completed operation through the operation schedule to calculate and display the progress of the completion of the application;

And combining the operation history index table, displaying abnormal conditions and drilling information of abnormal operation, and acquiring an abnormal operation list of the application, wherein the method comprises the following steps:

acquiring a path with the maximum path length from an AOE network of a scene, and marking the path as a critical path;

marking the abnormal condition according to the critical path, wherein the abnormal condition comprises marking operation failure, operation overtime and operation delay starting.

2. The method of claim 1, wherein obtaining a key operation index for a job from operation history data for the job comprises using a median of the operation history data as the key operation index for the job.

3. The method of claim 1 or 2, wherein the key operation indicators of the job include:

The execution time of the job, the start running time node of the job and the end running time node of the job.

4. The method according to claim 1, characterized in that the method further comprises:

and regularly refreshing the abnormal job list, and keeping the refreshed data in a cache so as to directly read the refreshed data.

5. The method of claim 1, wherein retrieving, via the job execution schedule, jobs that have been executed to calculate and present progress of completion of each application comprises:

According to the number of the jobs contained in each application in the scene, polling the current completion condition of the jobs, wherein the real-time progress of each application = the number of the successfully executed jobs/the total number of the jobs contained in the application in the scene;

The overall progress of the scene is characterized by the progress of the end-node application.

6. A batch processing monitoring apparatus, the apparatus comprising:

The first module is used for acquiring the operation of the application according to the configuration information of the operation and generating an operation schedule of the operation;

the second module is used for obtaining key operation indexes of the job through operation history data of the job and generating a job operation history index table;

a third module, configured to obtain, through the job running schedule, a job that has been run to completion to calculate and display a progress of application completion;

a fourth module for combining the operation history index table, displaying abnormal conditions and drilling the information of the abnormal operation, obtaining an abnormal operation list of the application,

The first module is configured to:

Classifying each scene according to the configuration information of the job;

identifying the target application associated with each scene as an end node application;

The fourth module is configured to:

7. The apparatus of claim 6, wherein the second module is configured to:

And characterizing the overall progress of the scene by the progress of the end node application.

8. The apparatus of claim 7, wherein the fourth module is further configured to:

Acquiring abnormal condition classification;

Calculating the proportion of abnormal operation through the classification of the abnormal conditions;

and according to the proportion of the abnormal operation, distinguishing and displaying.

9. A batch monitoring device comprising a processor and a memory, wherein the processor is configured to perform the method of any of claims 1-5.

10. A processor configured to perform the batch monitoring method of any one of claims 1 to 5.

11. A machine-readable storage medium having instructions stored thereon, which when executed by a processor cause the processor to be configured to perform the method of any of claims 1 to 5.

12. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the method according to any one of claims 1 to 5.