CN116193165A - A Distributed Cluster Scheduling Method and System for Efficient Video Protocol Conversion - Google Patents

Abstract

The invention relates to distributed cluster scheduling technology, and discloses a distributed cluster scheduling method and system for efficient video protocol conversion, which provides video protocol conversion nodes for registration and video protocol conversion node status maintenance, and performs routing scheduling according to video protocol conversion requests; Distribution of routing tasks, according to different routing strategies and rules for routing task distribution; fault migration, migration of video protocol conversion fault nodes; video stream interruption recovery, when the input and output video stream interruption alarm occurs, through The cache route restarts the task according to the original route to restore the video stream interruption. The invention supports horizontal node expansion without being limited by the capacity of a single node server, and increases the maximum number of concurrency of the platform; supports multiple distributed polling strategies, and can be configured and adjusted according to different business scenarios. Combined with the actual business scenario of video surveillance networking, on-demand dynamic routing is performed based on video protocol conversion requirements, which fits the business and is easy to adjust.

Description

A Distributed Cluster Scheduling Method and System for Efficient Video Protocol Conversion

technical field

The invention relates to a distributed cluster scheduling technology, in particular to a distributed cluster scheduling method and system for high-efficiency video protocol conversion.

Background technique

In the network aggregation platform system, it is clearly defined to support large-scale concurrent video access and protocol conversion output as much as possible. Limited by the bandwidth of the network port and the video conversion processing capability of a single node, the number of access channels of the platform is limited.

Such as prior art, CN201811610264.3, distributed task scheduling method, device and system; CN202110604591.3, a distributed task scheduling system based on ZOOKEEPER; distributed cluster scheduling in the prior art is generally strongly bound to ZOOKEEPER It is not convenient to expand other node registration mechanisms, and the workload of rectification is very large.

Contents of the invention

The present invention aims at the problem that the distributed cluster scheduling in the prior art is strongly bound to ZOOKEEPER; it is not convenient to expand the workload of other node registration mechanism rectification, and provides a high-efficiency video protocol conversion distributed cluster scheduling methods and systems.

In order to solve the above technical problems, the present invention is solved through the following technical solutions:

An efficient video protocol conversion distributed cluster scheduling method, the method comprising:

Video protocol conversion node management, providing video protocol conversion node registration and video protocol conversion node status maintenance;

Routing scheduling, according to the video protocol conversion request, routing scheduling;

Distribution of routing tasks, according to different routing strategies and rules for routing task distribution;

Fault migration, the migration of video protocol conversion failure nodes;

Recovery of video stream disconnection. When a disconnection alarm occurs for the input and output video streams, the task is restarted through the cached route according to the original route to recover the video stream disconnection.

Preferably, the video protocol conversion node management is implemented through zookeeper+dubbo registration.

Preferably, the video protocol conversion node management also provides routing scheduling query; and different nodes support different video protocol conversion types.

Preferably, the routing scheduling includes,

Process video protocol conversion requests, according to the conversion type of the request protocol, combined with whether to transcode or perceive compression, distinguish between transparent transmission routing scheduling or transcoding routing scheduling, dynamically balance to each type of node in the cluster according to the routing strategy, and save the current data in the redis cache Routing node information to ensure that the corresponding node can be quickly found for the same request.

Preferably, the routing policy includes transparent transmission routing and transcoding routing, and the transparent transmission routing configures all accessible video nodes; the transcoding routing configuration supports transcoding nodes.

Preferably, the routing rules calculate the routing nodes according to the capability value scheduling algorithm;

Set the benchmark capability value S _b according to the standard model of the platform, combined with the video protocol conversion task type T _b , the current capability value B _b of each node to be routed, the calculation formula is as follows:

IP _b ＝Max{S _b -B _b }-T _b

Among them, IP _b is the route scheduling result node. .

Preferably, failover includes,

Automatic detection of faulty nodes. For existing cached routing tasks, when rerouting, it is found that the original route cannot be connected, and the failure time point is set, and regular detection is performed; The cached route of the current node is rerouted to the new node. If the node is offline and becomes online within the set time, the route will continue to be maintained, and the current node’s cached data at the time of failure will be cleared, and it will be re-recorded when it occurs again.

In order to solve the above technical problems, the present invention also provides a high-efficiency video protocol conversion distributed cluster scheduling system, which is implemented by the above-mentioned high-efficiency video protocol conversion distributed cluster scheduling method, including a video protocol conversion node management module , routing scheduling module, routing policy and rule module, fault migration module and power failure recovery module;

The video protocol conversion node management module is used for the registration and state maintenance of the video protocol conversion node;

The routing scheduling module schedules tasks according to the conversion type of the requested video protocol;

Routing strategy and rule module, assign tasks according to different routing strategies and rules;

The failure migration module is used for the migration of video protocol conversion failure nodes;

The cut-off recovery module is used to restart the task according to the original route through the cache route when a cut-off alarm occurs for the input and output video streams.

Preferably, the video protocol conversion node management module performs node registration through zookeeper.

The present invention has remarkable technical effect owing to adopted above technical scheme:

The method of the present invention performs video protocol conversion and supports horizontal dynamic expansion. Therefore, for large-scale video protocol conversion requirements, this technology can effectively enhance the processing capacity of the system and support dynamic migration of node failures. When a node failure occurs, the scheduled The video protocol conversion task of the node is migrated to other nodes, which can effectively enhance the system stability. It supports the server configuration capabilities of each node in the cluster, dynamically balances video protocol conversion tasks, and makes full use of cluster resources to maximize resource utilization.

The invention supports horizontal node expansion, is not limited by the capacity of a single node server, and increases the maximum concurrent number of platforms;

The present invention supports multiple distributed polling strategies, and configuration adjustments can be made according to different business scenarios;

The present invention supports national standard, rtsp, rtmp access routing at the same time, supports rtmp, srt active streaming access routing; supports transparent transmission and transcoding task routing;

The invention supports a task retry mechanism. For example, after a service receives an alarm of no source, it can try to resume the task through routing scheduling again. After the service receives an alarm that no one is watching, it can close the task through routing scheduling. It can be switched more flexibly, and the overall scheduling code adjustment workload is very small.

Description of drawings

Fig. 1 is a system diagram of the present invention;

Fig. 2 is a flow chart of the present invention;

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.

Example 1

An efficient video protocol conversion distributed cluster scheduling method, the method comprising:

Video protocol conversion node management, providing video protocol conversion node registration and video protocol conversion node status maintenance;

Routing scheduling, according to the video protocol conversion request, routing scheduling;

Distribution of routing tasks, according to different routing strategies and rules for routing task distribution;

Fault migration, the migration of video protocol conversion failure nodes;

Recovery of video stream disconnection. When a disconnection alarm occurs for the input and output video streams, the task is restarted through the cached route according to the original route to recover the video stream disconnection.

Video protocol conversion node management is implemented through zookeeper+dubbo registration.

The video protocol conversion node management also provides routing scheduling query; and different nodes support different video protocol conversion types.

Routing scheduling includes,

Process video protocol conversion requests, according to the conversion type of the request protocol, combined with whether to transcode or perceive compression, distinguish between transparent transmission routing scheduling or transcoding routing scheduling, dynamically balance to each type of node in the cluster according to the routing strategy, and save the current data in the redis cache Routing node information to ensure that the corresponding node can be quickly found for the same request.

The routing strategy includes transparent transmission routing and transcoding routing. Transparent transmission routing configures all accessible video nodes; transcoding routing configuration supports transcoding nodes.

Routing rules calculate routing nodes according to the ability value scheduling algorithm;

Set the benchmark capability value S _b according to the standard model of the platform, combined with the video protocol conversion task type T _b , the current capability value B _b of each node to be routed, the calculation formula is as follows:

IP _b ＝Max{S _b -B _b }-T _b

Among them, IP _b is the route scheduling result node. , the remaining capacity value after this routing is completed, the business can judge based on this. If the result is less than 0, it is not enough to carry the current task, and the cluster needs to be expanded. If the result is greater than 0, the node returns the result directly according to the scheduling result.

Failover includes,

Automatic detection of faulty nodes. For existing cached routing tasks, when rerouting, it is found that the original route cannot be connected, and the failure time point is set, and regular detection is performed; The cached route of the current node is rerouted to the new node. If the node is offline and becomes online within the set time, the route will continue to be maintained, and the current node’s cached data at the time of failure will be cleared, and it will be re-recorded when it occurs again.

Example 2

On the basis of Embodiment 1, this embodiment is a distributed cluster scheduling system for high-efficiency video protocol conversion, which is implemented by the above-mentioned high-efficiency video protocol conversion distributed cluster scheduling method, including a video protocol conversion node management module , routing scheduling module, routing policy and rule module, fault migration module and power failure recovery module;

The video protocol conversion node management module is used for the registration and state maintenance of the video protocol conversion node;

The routing scheduling module schedules tasks according to the conversion type of the requested video protocol;

Routing strategy and rule module, assign tasks according to different routing strategies and rules;

The failure migration module is used for the migration of video protocol conversion failure nodes;

The cut-off recovery module is used to restart the task according to the original route through the cache route when a cut-off alarm occurs for the input and output video streams.

The video protocol conversion node management module performs node registration through zookeeper.

Example 3

On the basis of the above-mentioned embodiments, the aggregation platform products are based on the pan-security field, and are committed to the aggregation and integration of perceptual data, extending from the traditional data management model to the cloud service model of deep data application, such as preview access service, video playback, electronic Maps, authority control, event center, perception compression service, video processing service, etc., effectively solve the problems of difficult data access, integration, application, and long development cycle of the business platform, and reduce the cost of business construction. Promote the integration of big data and use more than 20 years of technology accumulation in the video field to provide complete solutions for users in industries such as public security, transportation, finance, education, fire protection, communities, iron towers, gas stations, hydropower stations, maritime affairs, and the military.

Among them, video protocol conversion distributed cluster scheduling technology carries video-related cluster scheduling such as overall video access, compression transcoding, video recording, and remote video playback;

It can be seen from Figure 2 that after the cluster is deployed, identify the type of each node, transparent transmission, H264 transcoding, H265 transcoding; platform access devices, such as GB28181, camera RTSP/RTMP, SDK access, Arclink, etc.;

The platform initiates device video stream transparent transmission access, generates routing keys, and initiates different task requests according to the access type, such as GB28181 initiates RTP receiving stream, RTSP/RTMP direct streaming, ArcLink's SRT receiving stream, etc.;

Routing scheduling executes the configured routing policy according to the request parameters such as routing key (the default routing algorithm is based on the capability value), and first checks whether there is an assigned node in the redis cache. If so, keep the original route; The node queries the capability value of each node according to the routing strategy, and finds the IP of the node with the largest remaining capability value. If there is no remaining capability value in the whole, an exception of insufficient capability is thrown. After the allocation is completed, the routing key and the corresponding routing node IP are stored in the redis cache, and the request is routed to the routing allocation node;

After obtaining the assigned routing IP during the routing request process, check the online status of the current routing IP. If the current node is temporarily offline, mark the offline time of the IP in redis. The key of the node deletes the routing record in the cache. When the platform initiates the routing task again, it will be redistributed. If the offline node is not online at this time, the newly initiated routing task will be assigned to other nodes. Join the node to be routed and continue to support the distribution of routing tasks;

After the routing is processed, the platform generates an external access flow address based on the routing IP and routing key, combined with the configured network mapping ip and port of the platform;

When an alarm occurs on the connected stream, the cluster node stream media service calls back the corresponding alarm to the local service, pushes it to the cluster MQ message queue, and the platform subscribes to MQ. When people watch the alarm, close the task according to the original route;

When the platform initiates a transcoding request for a certain video channel, it obtains the rtsp address of the source video according to the routing key corresponding to the video source, and reads the encoding type of the video source. The platform integrates the video source address, source encoding type, and transcoding output encoding type for routing. According to the source code type and the transcode output code type, the route scheduling judges that the required transcoding node type is H264 or H265. According to the required transcoding node type, query the available routing nodes in the transcoding routing node, and query each In terms of node capability value, find the IP of the node with the most remaining capability value. If the remaining capability value is at most 0, an exception of insufficient capability will be thrown. After the distribution is completed, the routing key and the corresponding routing node IP are stored in the redis cache, and the request is forwarded to the routing distribution node according to the routing; for the transparently transmitted video or transcoding video tasks that have been accessed, the platform initiates recording, forwarding, recording and other requests , keep scheduling according to the original route. When the connected video triggers alarms such as no source alarm, no one watching alarm, and sending timeout, the original route will also be maintained to complete the scheduling recovery or shut down.

Claims (9)

1. A high-efficiency video protocol conversion distributed cluster scheduling method comprises the following steps:

the video protocol conversion node is managed, and the video protocol conversion node is provided for registration and video protocol conversion node state maintenance;

routing scheduling, namely performing routing scheduling according to the video protocol conversion request;

distributing routing tasks according to different routing strategies and rules;

fault migration, namely, migration of a video protocol conversion fault node;

and (3) the interruption recovery of the video stream, and when the interruption alarm occurs to the input and output video stream, the interruption recovery of the video stream is carried out according to the original route restarting task through the cache route.

2. The method for efficient video protocol conversion distributed cluster scheduling according to claim 1, wherein video protocol conversion node management is implemented through zookeeper+dubbo registration.

3. The method of claim 1, wherein the video protocol conversion node management further provides a routing scheduling query; and different nodes support different video protocol conversion types.

4. The method of claim 1, wherein the routing schedule comprises,

processing a video protocol conversion request, distinguishing transparent transmission routing scheduling or transcoding routing scheduling according to the request protocol conversion type and combining whether transcoding and perceived compression, dynamically balancing to each type of node in a cluster according to a routing strategy, and simultaneously storing the current routing node information in a redis cache so as to ensure that corresponding nodes are quickly found when the same request is received.

5. The method for efficient video protocol conversion distributed cluster scheduling according to claim 1, wherein the routing policy includes a transparent route and a transcoding route, and the transparent route configures all accessible video nodes; the transcoding routing configuration supports transcoding nodes.

6. The method for efficient video protocol conversion distributed cluster scheduling according to claim 1, wherein the routing rule performs calculation of routing nodes according to a capability value scheduling algorithm;

setting a reference capability value S according to a standard model of a platform _b Task type T is converted in combination with video protocol _b The current capability value B of each node to be routed _b The calculation formula is as follows:

IP _b ＝Max{S _b -B _b }-T _b

wherein, IP _b And scheduling the result node for the route.

7. The method of claim 1, wherein the fault migration comprises,

automatically detecting a fault node, finding that an original route cannot be connected when an existing cache route task is routed again, setting a fault time point, and periodically detecting; if the set time is not exceeded, the node is judged to be faulty, the cache route of the original node is rerouted to the new node, if the set time is not exceeded, the node is offline to be online, the route is continuously maintained, the fault time point cache data of the current node is cleaned, and the fault time point cache data is recorded again.

8. The system is characterized by comprising a video protocol conversion node management module, a routing scheduling module, a routing strategy and rule module, a fault migration module and a cut-off recovery module;

the video protocol conversion node management module is used for registering and maintaining the state of the video protocol conversion node;

the route scheduling module is used for scheduling tasks according to the request video protocol conversion type;

the routing strategy and rule module is used for distributing tasks according to different routing strategies and rules;

the fault migration module is used for migrating the video protocol conversion fault nodes;

and the interruption recovery module is used for restarting the task according to the original route through the cache route when interruption alarm occurs to the input and output video stream.

9. The system of claim 8, wherein the video protocol conversion node management module performs node registration via a zookeeper.

Images