CN116033199B

CN116033199B - Multi-device audio and video synchronization method, device, electronic device and storage medium

Info

Publication number: CN116033199B
Application number: CN202211731219.XA
Authority: CN
Inventors: 药青; 邹仕洪; 张炯明
Original assignee: Yuanxin Information Technology Group Co ltd
Current assignee: Yuanxin Information Technology Group Co ltd
Priority date: 2022-12-30
Filing date: 2022-12-30
Publication date: 2025-08-22
Anticipated expiration: 2042-12-30
Also published as: CN116033199A

Abstract

The embodiment of the application provides a multi-device audio and video synchronization method, a device, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the steps of dividing an audio and video into a plurality of time periods, obtaining played frame numbers of sampling moments of different devices in the time periods, determining the device with the least played frame number at the last sampling moment in the devices as first device, determining second device based on the played frame numbers of the devices at the sampling moments, and adjusting the playing progress of the second device to be synchronous with the playing progress of the first device in the second time period. By dividing the audio and video into a plurality of time periods and adjusting the playing progress of the equipment at the end of each time period, the user cannot perceive the playing difference of the audio and video of different equipment physiologically, so that the effect of audio and video synchronization is achieved. The scheme of the application adjusts according to the playing progress of each device in each time period, and is not influenced by the performance difference of the hardware and the software among different devices and the interference of external factors.

Description

Multi-device audio and video synchronization method and device, electronic device and storage medium

Technical Field

The application relates to the technical field of computers, in particular to a method and a device for realizing audio and video synchronization of multiple devices, electronic equipment and a storage medium.

Background

The current audio and video applications are becoming more and more widespread, and it is very common to play audio and video on electronic devices. And more service scenes require that a plurality of devices can synchronously play a section of same audio and video at the same time, namely, the audio and video of the plurality of devices are synchronous.

In the prior art, frame synchronization is adopted, namely, a plurality of different devices are required to play the same video frames and the same audio frames at the same time so as to realize the effect of audio and video synchronization.

In a real-world scenario, however, it is very difficult for different devices to achieve complete synchronization of audio and video frames. The different devices have different processing speeds on each frame of the same audio and video, so that the different devices cannot play the same audio frame and video frame at the same moment, namely, cannot completely synchronize the frames, and in the running process of the devices, the devices are interfered by a plurality of external factors or intrinsic factors of the devices, for example, a task with higher priority is temporarily processed by one device, so that the resources for playing the audio and video are occupied, and the devices cannot completely synchronize the frames.

Disclosure of Invention

The application aims to at least solve one of the technical defects, and the technical scheme provided by the embodiment of the application is as follows:

in a first aspect, an embodiment of the present application provides a method for implementing audio and video synchronization of multiple devices, including:

dividing the audio and video into at least two preset time periods in sequence;

acquiring played frame numbers of at least one preset sampling moment of different devices in a first preset time period;

Determining the device with the least played frame number at the last preset sampling time in each device as a first device, and determining the playing condition as a second device with advanced playing based on the played frame number of each device at each preset sampling time;

and for each second device, adjusting the playing progress of the second device in a second preset time period.

In an optional embodiment of the present application, the determining, based on the number of played frames of each device at each preset sampling time, the second device with a play advance as the play condition includes:

For each preset sampling time, acquiring a median corresponding to the played frame number of each device at the preset sampling time, and acquiring the size relation between the played frame number and the median of each device;

and determining a second device based on the size relation between the played frame number of each device at each preset sampling time and the corresponding median.

In an alternative embodiment of the present application, determining the second device based on the size relationship between the played frame number of each device at each preset sampling time and the corresponding median includes:

For each preset sampling moment, marking the equipment with the played frame number smaller than the median corresponding to the preset sampling moment by using a first mark, and marking the equipment with the played frame number larger than the median corresponding to the preset sampling moment by using a second mark;

for each device, performing symbol rank sum test on the first marked quantity and the second marked quantity of the device in a first preset time period to obtain a test result corresponding to the device;

and determining the second device by the device with the second mark number exceeding the first mark number by a preset multiple as a result of the detection.

In an alternative embodiment of the present application, for each second device, adjusting the playing progress of the second device to be synchronous with the playing progress of the first device in a second preset period of time includes:

According to the played frame number of the first device at the last preset sampling time in the first preset time period, the playing speed of the first device is obtained, and according to the played frame number of the second device at the last preset sampling time in the first preset time period, the playing speed of the second device is obtained;

Acquiring a first preset frame number and a second preset frame number according to the playing speed of the first equipment, the playing speed of the second equipment and the preset repeated frame number, wherein the ratio of the first preset frame number to the playing speed of the first equipment is equal to the ratio of the second preset frame number to the playing speed of the second equipment, and the second preset frame number is equal to the sum of the repeated frame number and the first preset frame number;

and repeatedly playing the audio and video with the preset repeated frame number when the second equipment plays the audio and video with the second preset frame number after the second preset time period starts, wherein the audio and video with the preset repeated frame number is the audio and video with the last preset repeated frame number of the audio and video with the second preset frame number.

In an alternative embodiment of the present application, the method may further comprise:

For each first preset time period after the second preset time period, according to the played frame number of each device at each preset sampling time in the second preset time period before the first preset time period, acquiring a first variance of the corresponding played frame number, and according to the played frame number of each device at each preset sampling time in the third preset time period after the first preset time period, acquiring a second variance of the corresponding played frame number;

Comparing the first variance corresponding to the preset sampling time with the second variance corresponding to the preset sampling time for each preset sampling time, and obtaining the proportion of the preset sampling time, in which the first variance is larger than the second variance, in the preset sampling time;

if the proportion is smaller than the preset proportion value, third equipment with the playing condition of the third preset time period being playing lag and the playing progress of the third preset time period being adjusted is obtained, and the adjustment of the playing progress of the third equipment is canceled after the first preset time period is started;

If the ratio is not smaller than the preset ratio value, a fourth device with the least played frame number at the last preset sampling time in a third preset time period and a fifth device with the playing condition of playing in advance in the third preset time period are obtained, and the playing progress of the fifth device is adjusted after the first preset time period is started.

In an optional embodiment of the present application, a third device for obtaining a playback lag in a third preset time period and having an adjusted playback progress in the third preset time period is provided, where the third device includes:

And determining a third device from the devices with the playing progress adjusted in the third preset time period based on the size relation between the played frame number of each device at each preset sampling time and the corresponding median.

In an alternative embodiment of the present application, determining, based on a size relationship between a played frame number of each device at each preset sampling time and a corresponding median, a third device from devices whose play progress is adjusted in a third preset time period includes:

For each device, performing symbol rank sum test on the first marked quantity and the second marked quantity of the device in a third preset time period to obtain a test result corresponding to the device;

and determining a third device from the devices with the first mark number exceeding the second mark number of the preset multiple as a result of the inspection in the devices with the playing progress adjusted in the third preset time period.

In a second aspect, an embodiment of the present application provides an audio/video synchronization apparatus for a plurality of devices, including:

the preset time period setting module is used for dividing the audio and video into at least two preset time periods in sequence;

The playing frame number sampling module is used for acquiring the played frame number of at least one preset sampling moment of different equipment in a first preset time period;

the device determining module is required to be adjusted and is used for determining the device with the least played frame number at the last preset sampling moment in each device as a first device, and determining the playing condition as a second device with advanced playing based on the played frame number of each device at each preset sampling moment;

and the play progress adjusting module is used for adjusting the play progress of the second equipment in a second preset time period for each second equipment.

In an alternative embodiment of the present application, the device to be adjusted determination module is specifically configured to:

In an alternative embodiment of the present application, the required adjustment device determination module is further configured to:

In an alternative embodiment of the present application, the play progress adjustment module is specifically configured to:

In an alternative embodiment of the present application, the play progress adjustment module is further configured to:

In an alternative embodiment of the present application, the required adjustment device determination module may be further configured to:

In a third aspect, an embodiment of the present application provides an electronic device including a memory, a processor, and a computer program stored on the memory;

the processor executes a computer program to implement the method provided in the first aspect embodiment or any alternative embodiment of the first aspect.

In a fourth aspect, embodiments of the present application provide a computer readable storage medium having a computer program stored thereon, which when executed by a processor implements the method provided in the embodiment of the first aspect or any of the alternative embodiments of the first aspect.

The technical scheme provided by the embodiment of the application has the beneficial effects that:

By dividing the audio and video into a plurality of preset time periods, the playing progress of the equipment with the too high playing speed is adjusted at the end of each preset time period, and the playing speed of each equipment is enabled to be approximately the same after the next preset time period is started as much as possible, so that users cannot physiologically perceive the playing difference of the audio and video of different equipment, and the effect of audio and video synchronization is achieved. The scheme of the application is adjusted according to the playing progress of each device in each preset time period, and is not influenced by the performance difference of the hardware and the software among different devices and the interference of external factors.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that are required to be used in the description of the embodiments of the present application will be briefly described below.

Fig. 1 is a flow chart of a method for implementing audio/video synchronization of multiple devices according to an embodiment of the present application;

FIG. 2 is a flow chart of a second device determination method in an example of an embodiment of the application;

FIG. 3 is a flowchart illustrating a method for adjusting a play progress in an example of an embodiment of the present application;

fig. 4 is a block diagram of a multi-device audio/video synchronization apparatus according to an embodiment of the present application;

Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

Embodiments of the present application are described below with reference to the drawings in the present application. It should be understood that the embodiments described below with reference to the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application, and the technical solutions of the embodiments of the present application are not limited.

As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and "comprising," when used in this specification, specify the presence of stated features, information, data, steps, operations, elements, and/or components, but do not preclude the presence or addition of other features, information, data, steps, operations, elements, components, and/or groups thereof, all of which may be included in the present specification. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. The term "and/or" as used herein indicates that at least one of the items defined by the term, e.g., "a and/or B" may be implemented as "a", or as "B", or as "a and B".

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail with reference to the accompanying drawings.

The technical solutions of the embodiments of the present application and technical effects produced by the technical solutions of the present application are described below by describing several exemplary embodiments. It should be noted that the following embodiments may be referred to, or combined with each other, and the description will not be repeated for the same terms, similar features, similar implementation steps, and the like in different embodiments.

Fig. 1 is a flow chart of a method for implementing audio and video synchronization of multiple devices according to an embodiment of the present application, where an execution body of the method may be an audio and video playing control end or an audio and video playing device. In general, the control relationship between the audio and video playing control end and the audio and video playing device is weaker, and the audio and video playing control end generally only has functions of providing audio and video materials to be played (audio and video for short when ambiguity is not caused) for the audio and video playing device, sending audio and video playing instructions, collecting or receiving audio and video playing condition feedback of the audio and video playing device, and the like, and does not have the right to interfere the audio and video playing device to execute other tasks. And different audio and video playing devices have no control relation, and even can not directly mutually transmit data (of course, the data can be indirectly transmitted through an audio and video playing control terminal). The audio/video playing control end may be a server, and the audio/video playing device may be a terminal device such as a mobile phone or an intelligent television. As shown in fig. 1, the method may include:

Step S101, sequentially dividing the audio and video into at least two preset time periods.

The preset time period is a partial segment of the audio and video and is used for dividing a target audio and video into at least two segments.

Specifically, after receiving an audio and video to be synchronized, the audio and video playing control unit sequentially divides the audio and video into at least two preset time periods according to the length of the audio and video, i.e. divides the audio and video into at least two segments.

It should be noted that, the duration of the other preset time periods is the same except for the last preset time period.

Step S102, the played frame number of at least one preset sampling time of different devices in a first preset time period is obtained.

The first preset time period is a first preset time period including the audio and video playing start time in the steps, the preset sampling time is a sampling time point preset on each preset time period, each time the audio and video is played to the sampling time point, an audio and video playing control end or audio and video playing equipment (hereinafter referred to as equipment) can perform sampling counting on played frame numbers once, and the played frame numbers refer to total frame numbers of audio and video playing from the starting time of the preset time period to the preset sampling time.

Specifically, after the audio and video starts playing, counting the number of played frames of the audio and video in each device from the starting time of the first preset time period (i.e. the time when the audio and video starts playing), and recording the number of played frames of each device from the starting time of the first preset time period to the preset sampling time when each preset sampling time is reached, until the audio and video is played to the ending time of the first preset time period, stopping counting the number of played frames of each device.

It should be noted that, the number of preset sampling moments set by each device in each preset time period and the corresponding position (i.e. the time difference relative to the starting moment of the preset time period) are the same, and one preset sampling moment is set at the ending moment of the preset time period for recording the total number of frames played by each device in the preset time period, and the preset sampling moment is not set in the last preset time period.

For example, assume that the first preset time period is 00:00 to 02:00, and preset sampling moments are set to be 00:30, 01:00, 01:30, and 02:00, respectively. When the audio and video starts playing (i.e. 00:00 time), the number of played frames of each device starts to be counted, when the audio and video starts playing to 00:30 time, the number of played frames (such as 2000 frames) of each device from 00:00 time to 00:30 time is recorded, when the audio and video starts playing to 01:00 time, the number of played frames (such as 4000 frames) of each device from 00:00 time to 01:00 time is recorded, when the audio and video starts playing to 01:30 time, the number of played frames (such as 6000 frames) of each device from 00:00 time to 02:00 time is recorded, and when the audio and video starts playing to 02:00 time, the number of played frames (such as 8000 frames, i.e. the total number of frames played for the first preset time period of the device) of each device is recorded, and the counting is stopped.

Step S103, the device with the least played frame number at the last preset sampling time in the devices is determined to be the first device, and the second device with the playing condition being the playing advance is determined based on the played frame number of the devices at the preset sampling time.

The last preset sampling time is the ending time of the preset time period, the first device is the device with the minimum number of played frames in the first preset time period, and the second device is the device with the playing condition of playing in the first preset time period being in advance.

Specifically, after the played frame number of all the devices in the first preset time period is obtained, the device with the smallest played frame number in the first preset time period is used as the first device, then the playing condition of each device is determined according to the played frame number of all the devices at all the preset sampling moments in the first preset time period, and all the playing conditions are used as the second device. The play condition may include play advance, play lag and play normal.

If there are a plurality of devices having the smallest number of played frames and the same number of played frames, these devices may be all the first devices.

Step S104, for each second device, adjusting the playing progress of the second device in a second preset time period.

The second preset time period is a first preset time period after the first preset time period from the audio/video playing start time, and the playing progress can be represented by the total number of frames of the audio/video played in the preset time period.

Specifically, after the first preset time period is finished, the first device and the second device in the first preset time period are obtained, and at the beginning time of the second preset time period, the playing progress of the second device is adjusted according to the playing progress of the first device, so that the playing progress of the second device and the playing progress of the first device are synchronized as much as possible after the beginning of the second preset time period.

According to the scheme provided by the application, the audio and video are divided into a plurality of preset time periods, and the playing progress of the equipment with the excessively high playing speed is adjusted at the end of each preset time period, so that the playing speed of each equipment is close to the same as much as possible after the next preset time period starts, and the user cannot perceive the playing difference of the audio and video of different equipment physiologically, so that the effect of audio and video synchronization is achieved. The scheme of the application is adjusted according to the playing progress of each device in each preset time period, and is not influenced by the performance difference of the hardware and the software among different devices and the interference of external factors.

The median is calculated by sequencing played frame numbers of all devices at all preset sampling moments from small to large, and each preset sampling moment has a corresponding median.

Specifically, as shown in fig. 2, for each preset sampling moment, the played frame numbers of the devices at the preset sampling moment are sorted from small to large, the median corresponding to the preset sampling moment is obtained by calculation after sorting, then the played frame numbers of the devices at the preset sampling moment are compared with the median, and finally the second device is determined according to the comparison result of the devices at all the preset sampling moments. The median of the N numbers is generally defined by ordering the N numbers from large to small or from small to large, with the k+1th number being the median if n=2k+1, and the average of the k and k+1th numbers being the median if n=2k.

The first mark indicates that the played frame number of the device at the preset sampling time is smaller than the median corresponding to the preset sampling time, the second mark indicates that the played frame number of the device at the preset sampling time is larger than the median corresponding to the preset sampling time, and in actual operation, the first mark may be "-", and the second mark may be "+". A common symbol rank and test method substitutes the number m of first marks and the number n of second marks of each device in the preset time period into a formula χ ²＝(|n-m|-1)²/(n+m), calculates χ ², and then compares χ ² with a preset control value.

Specifically, as shown in fig. 2, for each preset sampling time, the device whose number of played frames at the preset sampling time is smaller than the median corresponding to the preset sampling time is marked with a first mark, the device whose number of played frames at the preset sampling time is greater than the median corresponding to the preset sampling time is marked with a second mark, the number m of the first mark and the number n of the second mark of each device at the preset sampling time are counted, m and n are substituted into a formula χ ²＝(|n-m|-1)²/(n+m) to calculate a value χ ², and then the value χ ² is compared with a corresponding preset reference value in a symbol rank and a test threshold table. If the value of χ ² is larger than the preset comparison value and n is larger than m, the playing condition of the device in the preset time period can be considered to be playing in advance, if the value of χ ² is larger than the preset comparison value and m is larger than n, the playing condition of the device in the preset time period can be considered to be playing lag, and if the value of χ ² is smaller than or equal to the preset comparison value, the playing condition of the device in the preset time period can be considered to be normal.

And taking the device with the playing condition of playing in advance as the second device. For example, in the first preset time period, 10 preset sampling moments are set, the respective median of 10 preset sampling moments is calculated according to the played frame number of each device at each preset sampling moment, then the played frame number of each preset sampling moment of the device a in the preset time period is compared with the corresponding median, 2 first marks and 8 second marks are marked, and the value χ ² =4 is calculated and is greater than the corresponding preset comparison value in the symbol rank and the check threshold table, so that the device a can be regarded as a device playing in advance in the first preset time period, and the device a is regarded as a second device.

In an optional embodiment of the present application, for each second device, adjusting the playing progress of the second device in a second preset period of time includes:

The playing speed is calculated by dividing the number of played frames of the device at the last preset sampling time of the preset time period by the total duration of the preset time period. The preset repetition frame number is a frame number which is required to be repeatedly played compared with the first device after the second device after being adjusted in advance plays a certain frame number in the next preset time period. The first preset frame number and the second preset frame number are used for indicating that in the same time in the first preset time period, the first device plays audio and video of the first preset frame number, and the second device plays audio and video of the second preset frame number.

Specifically, as shown in fig. 3, the playing speed of the first device in the first preset time period is calculated according to the number of played frames of the first device at the last preset sampling time in the first preset time period divided by the total duration of the preset time period, the playing speed of the second device is calculated by the same method, and then the first preset number of frames and the second preset number of frames are calculated according to the condition that the ratio of the first preset number of frames to the playing speed of the first device is equal to the ratio of the second preset number of frames to the playing speed of the second device. And after the second preset time period starts, repeatedly playing the audio and video with the preset repeated frame number in the audio and video with the second preset frame number every time the second equipment plays the audio and video with the second preset frame number, wherein the audio and video with the preset repeated frame number in the audio and video with the second preset frame number is used for waiting for the playing progress of the first equipment. The audio/video of the preset repetition frame number in the audio/video of the second preset frame number is generally located at the last position of the audio/video of the second preset frame number, and may be the last frame or the last number of frames of the audio/video of the second preset frame number. In order to ensure the synchronization effect, the preset number of repetition frames should not be excessive. This allows the number of frames played by the first device and the second device to be as equal as possible during the same time in the second period.

It should be noted that, at the preset sampling time of the second preset time period, for the audio and video with the preset repetition frame number, the count of the played frame number counted by any frame is 1, and the count of the played frame number is not increased due to the preset repetition frame number.

For example, assume that the duration of the first preset time period is 180 seconds and the preset number of repetition frames is 1 frame. When the preset time period is over, the played frame number of the second device A is 5400 frames, the played frame number of the first device B is 5220 frames, the playing speed of the second device A is 30 frames per second, the playing speed of the first device B is 29 frames per second, and an equation is established according to the dataM (i.e., the first preset number of frames) is calculated to be 29, and M+1 (i.e., the second preset number of frames) is calculated to be 30. In the first preset time period, the first device B only plays 29 frames of audio and video when the second device a plays 30 frames of audio and video. And after the second preset time period starts, when the second device A plays 30 frames of audio and video, repeatedly playing the last 1 frame of audio and video in 1 frame of the audio and video, and waiting for the first device B to play the 30 th frame of the audio and video so as to ensure that the playing progress of the device A and the playing progress of the device B are synchronous as much as possible. If M calculated from the equation is not an integer, the integer closest to the result of the equation calculation is used as M.

The first preset time period is each preset time period after a second preset time period from the audio/video start time, the second preset time period is the preset time period before the first preset time period, the third preset time period is the preset time period after the first preset time period, the first variance is the variance of the played frame number of each device at each preset sampling time in the second preset time period, the second variance is the variance of the played frame number of each device at each preset sampling time in the third preset time period, the preset proportion value is used for indicating the relation between all the first variances and all the second variances, the third device is the device with the playing lag in the third preset time period and the playing progress adjusted in the third preset time period, the fourth device is the device with the least played frame number in the third preset time period, and the fifth device is the device with the playing advance in the third preset time period.

Specifically, for each first preset time period, firstly acquiring the played frame number of each device at each preset sampling time in a second preset time period and a third preset time period corresponding to the first preset time period, taking the played frame number of each device at each preset sampling time as a group of data, respectively calculating the variance of each group of data to obtain the variance of each preset sampling time, wherein the variance of each preset sampling time in the second preset time period is a first variance, the variance of each preset sampling time in the third preset time period is a second variance, then comparing the first variance with the second variance with the same position of the corresponding preset sampling time, recording the number that the first variance is larger than the second variance, and then calculating the proportion of the number that the first variance is larger than the second variance to the number of preset sampling times in the preset time period.

Specifically, if the ratio value is smaller than the preset ratio value, it is possible that the adjustment of the playing progress of the device in the third preset time period is unreasonable, because the adjustment is to repeatedly play the audio and video with a certain number of frames, that is, the playing progress of the device is slowed down, at this time, it is determined that the playing condition is the third device with lag playing and adjusted in the third preset time period, after the first preset time period starts, the possible unreasonable adjustment of the third device is canceled, so that the audio and video with a certain number of frames are not repeatedly played, if the ratio value is not smaller than the preset ratio value, it is possible that the adjustment of the playing progress of the device in the third preset time period is reasonable, then the same adjustment mode is adopted to continuously adjust the device to be adjusted after the first preset time period, that is, it is to determine the fourth device with the least playing frame number of frames played in the third preset time period and the fifth device with advance playing condition in the third preset time period, and repeatedly play the audio and video with a certain number of frames is ensured that the fourth device and the fifth device are synchronously played after each playing the audio and video with a certain number of frames in the first preset time period.

If the third device is found to be not in the third device, the third device is not found to be in the third device, and the possible devices in the third preset time period are considered to be interfered by external factors or accidental factors in the devices, and at this time, the same operation as the operation meeting the condition that the ratio value is smaller than the preset ratio value is adopted, namely, the fourth device and the fifth device in the third preset time period are determined, and the playing progress of the fifth device is adjusted after the first preset time period is started.

It should be noted that, if no fifth device satisfying the condition is found in the process of acquiring the fifth device, no operation is performed after the first preset time period starts.

For example, the number of preset sampling moments in each preset time period is 10, the preset proportion value is 60%, when the third preset time period starts, the played frame number of each device in the first two preset time periods (i.e. the first preset time period and the second preset time period) is obtained, then the variance of each preset sampling moment is calculated according to the played frame number of each device in each preset sampling moment, the first variance is the variance of each preset sampling moment in the first preset time period (i.e. the second preset time period), the second variance is the variance of each preset sampling moment in the second preset time period (i.e. the third preset time period), then each first variance is compared with the second variance of the corresponding position (for example, the variance of the fourth preset sampling moment in the first preset time period is compared with the variance of the fourth preset sampling moment in the second preset time period), the number of the first variance is 8, the number of the total preset sampling moment is 80%, the second variance is the variance of each preset sampling moment in the second preset time period (i.e. the third preset time period), the audio frame is determined to be the audio frame is repeatedly played after the audio frame is repeatedly played in the second time period 1, and the audio frame is not repeatedly played in the preset time period 1, and the audio frame is repeatedly played in the audio frame is not being set to be the audio frame is being played in the preset time 1.

Specifically, the process of calculating the median of the third preset time period and the process of obtaining the size relationship between the played frame number and the median of each device are the same as the process of calculating the median of the first preset time period and the process of obtaining the size relationship between the played frame number and the median of each device described above, and are not repeated here. Unlike the foregoing, the second device obtained in the foregoing is a device in which the playing condition is playing ahead, and the third device obtained here is a device in which the playing condition is playing behind and the playing progress is adjusted in the preset time period.

Specifically, the marking process of the device in the third preset time period is the same as the marking process of the device in the first preset time period, the symbol rank and the checking method are also the same as the foregoing, and a detailed description is omitted herein.

It should be noted that, the foregoing is that the device with the playing condition of playing in advance is used as the second device, and here, the device with the playing condition of playing in lag and with the playing progress adjusted in the third preset time period is used as the third device.

Fig. 4 is a block diagram of a multi-device audio/video synchronization apparatus according to an embodiment of the present application, and as shown in fig. 4, the multi-device audio/video synchronization apparatus 400 may include a preset time period setting module 401, a play frame number sampling module 402, a device to be adjusted determining module 403, and a play progress adjusting module 404, where,

The preset time period setting module 401 is configured to divide the audio and video into at least two preset time periods in sequence;

The playing frame number sampling module 402 is configured to obtain played frame numbers of at least one preset sampling time in a first preset time period of different devices;

The device to be adjusted determining module 403 is configured to determine, as a first device, a device with a minimum number of played frames at a last preset sampling time in each device, and determine, based on the number of played frames of each device at each preset sampling time, a second device with an advanced playing condition;

The play progress adjustment module 404 is configured to adjust, for each second device, a play progress of the second device within a second preset period of time.

Referring now to fig. 5, there is shown a schematic diagram of an electronic device (e.g., a terminal device or server performing the method of fig. 1) 500 suitable for use in implementing an embodiment of the present application. The electronic device in the embodiment of the present application may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a car-mounted terminal (e.g., car navigation terminal), a wearable device, etc., and a fixed terminal such as a digital TV, a desktop computer, etc. The electronic device shown in fig. 5 is only an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present application.

The electronic device comprises a memory for storing a program for executing the method according to the above-described respective method embodiments, and a processor configured to execute the program stored in the memory. Herein, the processor may be referred to as a processing device 501, which is described below, and the memory may include at least one of a Read Only Memory (ROM) 502, a Random Access Memory (RAM) 503, and a storage device 508, which are described below, specifically as follows:

As shown in fig. 5, the electronic device 500 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 501, which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 502 or a program loaded from a storage means 508 into a Random Access Memory (RAM) 503. In the RAM503, various programs and data required for the operation of the electronic apparatus 500 are also stored. The processing device 501, the ROM 502, and the RAM503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.

In general, devices may be connected to I/O interface 505 including input devices 506, including for example, touch screens, touch pads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc., output devices 507, including for example, liquid Crystal Displays (LCDs), speakers, vibrators, etc., storage devices 508, including for example, magnetic tape, hard disk, etc., and communication devices 509. The communication means 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. While fig. 5 shows an electronic device having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead.

In particular, according to embodiments of the present application, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a non-transitory computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication means 509, or from the storage means 508, or from the ROM 502. The above-described functions defined in the method of the embodiment of the present application are performed when the computer program is executed by the processing means 501.

The computer readable storage medium of the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of a computer-readable storage medium may include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present application, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to electrical wiring, fiber optic cable, RF (radio frequency), and the like, or any suitable combination of the foregoing.

In some embodiments, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

The computer readable medium may be included in the electronic device or may exist alone without being incorporated into the electronic device.

The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to:

The method comprises the steps of dividing an audio and video into at least two preset time periods in sequence, obtaining played frame numbers of at least one preset sampling time of different devices in a first preset time period, determining the device with the smallest played frame number in the last preset sampling time of each device as a first device, determining second devices with advanced playing conditions based on the played frame numbers of each device in each preset sampling time, and adjusting the playing progress of the second devices in the second preset time period for each second device.

Computer program code for carrying out operations of the present application may be written in one or more programming languages, including, but not limited to, an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules or units involved in the embodiments of the present application may be implemented in software or in hardware. Where the name of the module or unit does not constitute a limitation of the unit itself in some cases, for example, the first constraint acquisition module may also be described as "a module that acquires the first constraint".

The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic that may be used include Field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems-on-a-chip (SOCs), complex Programmable Logic Devices (CPLDs), and the like.

In the context of the present application, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited in order and may be performed in other orders, unless explicitly stated herein. Moreover, at least some of the steps in the flowcharts of the figures may include a plurality of sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order of their execution not necessarily being sequential, but may be performed in turn or alternately with other steps or at least a portion of the other steps or stages.

The foregoing is only a partial embodiment of the present invention, and it should be noted that it will be apparent to those skilled in the art that modifications and adaptations can be made without departing from the principles of the present invention, and such modifications and adaptations are intended to be comprehended within the scope of the present invention.

Claims

1. A multi-device audio and video synchronization method, comprising:

dividing the audio and video into at least two preset time periods in sequence;

for each second device, adjusting the playing progress of the second device in a second preset time period;

The method further comprises the steps of:

For each first preset time period after the second preset time period, according to the played frame number of each device at each preset sampling time in the second preset time period before the first preset time period, acquiring a first variance of the corresponding played frame number, and according to the played frame number of each device at each preset sampling time in the third preset time period after the first preset time period before the first preset time period, acquiring a second variance of the corresponding played frame number;

If the ratio is smaller than a preset ratio value, acquiring third equipment with the playing condition of the third preset time period being playing lag and the playing progress of the third preset time period being adjusted, and canceling the adjustment of the playing progress of the third equipment after the first preset time period starts;

If the ratio is not smaller than a preset ratio value, a fourth device with the least played frame number at the last preset sampling time in the third preset time period and a fifth device with the playing condition of playing in advance in the third preset time period are obtained, and the playing progress of the fifth device is adjusted after the first preset time period begins.

2. The method as set forth in claim 1, wherein the determining, based on the number of played frames of each device at each preset sampling time, the second device whose play condition is a play advance includes:

For each preset sampling time, acquiring a median corresponding to the played frame number of each device at the preset sampling time, and acquiring the size relation between the played frame number of each device and the median;

and determining the second equipment based on the size relation between the played frame number of each equipment at each preset sampling time and the corresponding median.

3. The method according to claim 2, wherein determining the second device based on the size relationship between the played frame number of each device at each preset sampling time and the corresponding median comprises:

For each device, performing symbol rank sum test on the first marked quantity and the second marked quantity of the device in the first preset time period to obtain a test result corresponding to the device;

And determining the second device by the device with the second mark number exceeding the first mark number by a preset multiple as a test result.

4. The method of claim 1, wherein for each second device, adjusting the playing progress of the second device in a second preset period of time includes:

acquiring a first preset frame number and a second preset frame number according to the playing speed of the first equipment, the playing speed of the second equipment and the preset repetition frame number, wherein the ratio of the first preset frame number to the playing speed of the first equipment is equal to the ratio of the second preset frame number to the playing speed of the second equipment, and the second preset frame number is equal to the sum of the repetition frame number and the first preset frame number;

5. The method according to claim 1, wherein the obtaining a third device that plays with a lag in playback during the third predetermined period and has an adjusted progress in playback during the third predetermined period comprises:

And determining the third device from the devices with the playing progress adjusted in the third preset time period based on the size relation between the played frame number of each device at each preset sampling time and the corresponding median.

6. The method according to claim 5, wherein determining the third device from the devices whose play progress is adjusted in the third preset time period based on the size relationship between the number of played frames of each device at each preset sampling time and the corresponding median, comprises:

for each device, performing symbol rank sum test on the first marked quantity and the second marked quantity of the device in the third preset time period to obtain a test result corresponding to the device;

And determining the third device from the devices with the first mark number exceeding the second mark number of the preset multiple as the check result in the devices with the play progress adjusted in the third preset time period.

7. An audio and video synchronization device for a plurality of devices, comprising:

the playing progress adjusting module is used for adjusting the playing progress of each second device in a second preset time period;

the play progress adjustment module is further configured to:

8. An electronic device comprising a memory, a processor and a computer program stored on the memory, characterized in that the processor executes the computer program to carry out the steps of the method according to any one of claims 1-6.

9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any of claims 1-6.