WO2018192414A1 - Qoe determining method and apparatus, storage medium, and processor - Google Patents

Abstract

Provided are a Quality of Service (QoE) determining method and apparatus, a storage medium, and a processor. The QoE determining method comprises: determining a key performance indicator (KPI) for video playback of an Internet Protocol Television or OTT; determining a mean opinion score (MOS) value of user experience at a predetermined moment according to the KPI; determining, by using a plurality of MOS values calculated within a predetermined time period, a time sequence of the MOS values; and determining the QoE according to the time sequence of the MOS values.

Description

Method, device, storage medium and processor for determining QoE Technical field

The present disclosure relates to the field of communications, and in particular, to a method, an apparatus, a storage medium, and a processor for determining a quality of user experience QoE.

Background technique

When telecom operators provide users with Internet Protocol Television (IPTV) or provide various over-the-top (OT) video services to users through the Internet, they are often concerned about quality of service (Quality of Service). , referred to as QoS, quality of service, the most direct reflection is the quality of experience (QoE), QoE can be understood as user experience or user perception, that is, the service performance provided by the end user to the network Subjective feelings.

Summary of the invention

The following is an overview of the topics detailed in this document. This Summary is not intended to limit the scope of the claims.

Embodiments of the present disclosure provide a method, an apparatus, a storage medium, and a processor for determining a user experience quality QoE.

According to an embodiment of the present disclosure, a method for determining a user experience quality QoE is provided, including: determining a key performance indicator KPI of a video broadcast of an Internet Protocol Television IPTV or OTT; and determining an average opinion of a user experience at a predetermined time according to the KPI. The MOS value is scored; a time series of the MOS values is determined using a plurality of the MOS values calculated within a predetermined time period; and a user experience quality QoE is determined according to a time series of the MOS values.

In an exemplary embodiment, determining the KPI of the IPTV or the video playback of the OTT includes: calculating the KPI of the IPTV or the OTT according to a first predetermined period; wherein the KPI includes at least the following One of: a user identification ID, a time point at which the KPI is sampled, a first play buffer delay of the video, and a sequence of time durations of the video.

In an exemplary embodiment, determining the MOS value of the user experience at a predetermined time according to the KPI comprises: calculating the MOS value of the user experience at the predetermined time according to a polynomial mode, wherein the polynomial mode is At least one of the following factors is determined: a first play buffering delay of the video, a total length of the video of the video, and a magnitude of a change in the video.

In an exemplary embodiment, determining a time series of the MOS values by using the plurality of the MOS values calculated within the predetermined time period includes determining, by using a plurality of the MOS values calculated within the predetermined time period At least one of an average MOS value, a minimum MOS value, and a maximum MOS value in each sub-period within a predetermined time period; determined according to at least one of an average MOS value, a minimum MOS value, and a maximum MOS value in each of the sub-periods The time series of the MOS values.

In an exemplary embodiment, in a case where the average MOS value and the minimum MOS value in each of the sub-periods within the predetermined time period are determined, determining the user experience quality QoE according to the time series of the MOS values includes: determining by the first a time series corpus consisting of a time series and a second time series, wherein the first time series is a time series of MOS values consisting of average MOS values in the respective sub-periods, and the second time series is A time series of MOS values composed of minimum MOS values in each sub-period; a pre-established time series model is trained using the time-series corpus; and a user experience quality QoE is determined using the trained time series model.

In an exemplary embodiment, after determining the QoE according to a time series of the MOS values, the method further comprises: sorting the determined QoEs according to a quality tolerance, wherein the quality tolerance is It is calculated according to the first time series and the second time series; some users are selected as the QoE difference users according to the sorting result; and the QoE difference users are operated and processed.

According to another embodiment of the present disclosure, a device for determining a user experience quality QoE is further provided, including: a first determining module, configured to determine a key performance indicator KPI of video playback of an Internet Protocol TV IPTV or OTT; and a second determining module And determining, according to the KPI, an average opinion score MOS value of the user experience at a predetermined time; the third determining module is configured to determine a time series of the MOS value by using the plurality of the MOS values calculated within a predetermined time period; The determining module is configured to determine a user experience quality QoE according to a time series of the MOS values.

In an exemplary embodiment, the first determining module includes: a first calculating unit, configured to calculate the KPI of the IPTV or the OTT according to a first predetermined period; wherein the KPI includes at least one of the following : a user identification ID, a time point at which the KPI is sampled, a first play buffering delay of the video, and a sequence of time durations of the video.

In an exemplary embodiment, the second determining module includes: a second calculating unit configured to calculate the MOS value of the user experience at the predetermined time according to a polynomial mode, wherein the polynomial mode is based on at least a factor of One determines: the first play buffer delay of the video, the total duration of the video, and the magnitude of the change in the video.

In an exemplary embodiment, the third determining module includes: a first determining unit configured to determine an average MOS in each of the sub-periods within the predetermined time period by using the plurality of the MOS values calculated within the predetermined time period At least one of a value, a minimum MOS value, and a maximum MOS value; the second determining unit configured to determine the MOS according to at least one of an average MOS value, a minimum MOS value, and a maximum MOS value in the respective sub-periods The time series of values.

In an exemplary embodiment, in a case where the average MOS value and the minimum MOS value in each of the sub-periods within the predetermined time period are determined, the fourth determining module includes: a third determining unit configured to determine by the first a time series corpus consisting of a time series and a second time series, wherein the first time series is a time series of MOS values consisting of average MOS values in the respective sub-periods, and the second time series is a time series of MOS values composed of minimum MOS values in each sub-period; a training unit configured to train a pre-established time series model using the time series corpus; and a fourth determining unit configured to utilize the trained time series model Determine user experience quality QoE.

In an exemplary embodiment, the apparatus further includes: a sorting module, configured to sort the determined QoE according to a quality tolerance after determining the QoE according to a time series of the MOS values, where The quality tolerance is calculated according to the first time series and the second time series; the selection module is configured to select some users as QoE difference users according to the sorting result; and the processing module is set to the QoE Poor users perform operation and maintenance processing.

According to another embodiment of the present disclosure, there is also provided a storage medium comprising a stored program, wherein the program is executed to perform the method of any of the above.

According to another embodiment of the present disclosure, there is further provided a processor for running a program, wherein the program is executed to perform the method of any of the above.

Through the present disclosure, since the set top box determines the key performance indicator KPI of the video broadcast of the Internet Protocol Television IPTV or OTT; determines the average opinion score MOS value of the user experience at the predetermined time according to the KPI; and utilizes the plurality of MOS values calculated within the predetermined time period Determining a time series of MOS values; determining user experience quality QoE based on a time series of MOS values. Therefore, the quality user can be selected according to the determined user experience quality QoE. Therefore, the problem that the accuracy of the user who detects the quality difference is not high can be solved, and the effect of improving the accuracy of the user of the detection quality is achieved.

Other aspects will be apparent upon reading and understanding the drawings and detailed description.

DRAWINGS

1 is a block diagram showing a hardware structure of a mobile terminal for determining a user experience quality QoE according to an embodiment of the present disclosure;

2 is a flow chart in accordance with an embodiment of the present disclosure;

3 is a flowchart of time-series-based quality difference user detection in the IPTV or OTT in this embodiment;

4 is a flow chart for calculating a MOS value in the embodiment;

FIG. 5 is a flowchart of training IPTV/OTT quality difference user detection time series model in the embodiment; FIG.

6 is a topological structural diagram of a time series model in the embodiment;

Figure 7 is a diagram showing the internal structure of the LSTM element in this embodiment;

FIG. 8 is a structural block diagram of a determining apparatus of a user experience quality QoE according to an embodiment of the present disclosure.

detailed description

At present, a user who detects a poor experience (quality difference) in a time period scores the MOS value according to the average opinion of the user experience, and determines whether the user MOS value is lower than a certain threshold value within a time period, if a low MOS value occurs. If the ratio exceeds a certain threshold, for example, 10%, it is considered that the current user is a poor user in the current time period. The accuracy of detecting the user of the poor quality is not high because it ignores the difference in user experience at different moments, and does not consider The up-and-down relationship of the experience, that is, the time series relationship, is to treat the experience at all times as equal and indistinguishable.

The present disclosure will be described in detail below with reference to the drawings in conjunction with the embodiments.

The terms "first", "second" and the like in the specification and claims of the present disclosure and the above-mentioned figures are used to distinguish similar objects, and are not necessarily used to describe a particular order or order.

The method provided by the embodiments of the present disclosure may be performed in a mobile terminal, a computer terminal, or the like. Taking a mobile terminal as an example, FIG. 1 is a hardware structural block diagram of a mobile terminal for determining a user experience quality QoE according to an embodiment of the present disclosure. As shown in FIG. 1, mobile terminal 10 may include one or more (only one shown in FIG. 1) processor 102 (processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA. A memory 104 for storing data, and a transmission device 106 for communication functions. It will be understood by those skilled in the art that the structure shown in FIG. 1 is merely illustrative and does not limit the structure of the above electronic device. For example, the mobile terminal 10 may also include more or fewer components than those shown in FIG. 1, or have a different configuration than that shown in FIG.

The memory 104 can be used to store software programs and modules of the application software, such as program instructions/modules corresponding to the determination method of the user experience quality QoE in the embodiment of the present disclosure, and the processor 102 runs the software programs and modules stored in the memory 104, Thereby performing various functional applications and data processing, that is, implementing the above method. Memory 104 may include high speed random access memory, and may also include non-volatile memory such as one or more magnetic storage devices, flash memory, or other non-volatile solid state memory. In some examples, memory 104 may further include memory remotely located relative to processor 102, which may be connected to mobile terminal 10 over a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

Transmission device 106 is for receiving or transmitting data via a network. The above-described network specific example may include a wireless network provided by a communication provider of the mobile terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC) that can be connected to other network devices through a base station to communicate with the Internet. In one example, the transmission device 106 can be a Radio Frequency (RF) module for communicating with the Internet wirelessly.

In this embodiment, a method for determining user experience quality QoE is provided. FIG. 2 is a flowchart according to an embodiment of the present disclosure. As shown in FIG. 2, the process includes the following steps:

Step S202, determining a key performance indicator KPI of the video broadcast of the Internet Protocol Television IPTV or OTT;

Step S204, determining an average opinion score MOS value of the user experience at a predetermined time according to the KPI;

Step S206, determining a time series of the MOS values by using a plurality of the MOS values calculated within a predetermined time period;

Step S208, determining a user experience quality QoE according to the time series of the MOS values.

Through the above steps, after the set top box determines the key performance indicator KPI of the Internet protocol television IPTV or OTT video playback; determines the average opinion score MOS value of the user experience at the predetermined time according to the KPI; and utilizes multiple MOS values calculated within the predetermined time period Determining a time series of MOS values; determining user experience quality QoE based on a time series of MOS values. Therefore, the quality user can be selected according to the determined user experience quality QoE. Therefore, the problem that the accuracy of the user who detects the quality difference is not high can be solved, and the effect of improving the accuracy of the user of the detection quality is achieved.

In an exemplary embodiment, the execution body of the above steps may be a terminal (eg, a set top box) or the like, but is not limited thereto.

At present, the method of quantitative evaluation of QoE usually adopts the Mean Opinion Score (MOS) scoring model. Some MOS models select the indicators such as the Carton, the number of times, and the first delay in the KPI index of the video playback. The MOS value is determined by a simple threshold threshold. The MOS value of this scheme is not high, and the video card is not deeply considered. The effect of the length of time and the number of video jams on the user's viewing experience. Some MOS models select indicators such as packet loss rate, delay, jitter, etc. in network transmission, and predictive analysis through machine learning models. One of the major problems of this scheme is that training corpus is difficult to obtain because of different video sources and video colors. Complexity and complexity of change, even under the same packet loss rate, delay, and jitter indicators, can bring different experiences to users, which can easily lead to the failure of the prediction model.

In an exemplary embodiment, determining the KPI of the IPTV or the video playback of the OTT includes: calculating the foregoing IPTV or the KPI of the OTT according to a first predetermined period; wherein the KPI includes at least one of the following: a user identifier ID, The time point at which the KPI is sampled, the first play buffer delay of the above video, and the sequence of the duration of the video. In this embodiment, the first predetermined period may be a period of time, such as 10 seconds.

In an exemplary embodiment, determining, according to the KPI, the foregoing MOS value of the user experience at a predetermined time comprises: calculating the MOS value of the user experience by using the predetermined time according to a polynomial mode, wherein the polynomial mode is determined according to at least one of the following factors: : The first play buffer delay of the above video, the total duration of the above video, and the magnitude of the change of the above video. In this embodiment, by comprehensively considering the user identification ID, the time point of sampling the KPI, the first play buffer delay of the video, the sequence of the video's duration, and other factors affecting the user's viewing experience, the IPTV or the OTT is calculated by the above factors. The KPI value solves the problem that there is no in-depth consideration of the effect of the video card duration and the number of video jams on the user's viewing experience. It effectively improves the accuracy of calculating MOS values and improves the accuracy of polynomial mode calculations.

In an exemplary embodiment, determining a time series of the MOS values by using the plurality of MOS values calculated within the predetermined period of time includes determining, by using a plurality of MOS values calculated within a predetermined time period, in each of the sub-periods within the predetermined period of time At least one of an average MOS value, a minimum MOS value, and a maximum MOS value; determining a time series of the MOS values according to at least one of an average MOS value, a minimum MOS value, and a maximum MOS value in each of the sub-periods described above. In the present embodiment, the predetermined period of time may refer to one day or other specified period of time. By calculating the average MOS value, the minimum MOS value and the maximum MOS value in the time series, the problem that the current corpus is difficult to obtain is solved.

In an exemplary embodiment, in a case where the average MOS value and the minimum MOS value in each of the sub-periods within the predetermined period of time are determined, determining the user experience quality QoE according to the time series of the MOS values includes: determining the first time series And a time series corpus composed of a second time series, wherein the first time series is a time series of MOS values composed of average MOS values in the respective sub-periods, and the second time series is determined by each of the sub-periods A time series of MOS values composed of minimum MOS values; training a pre-established time series model using the above time series corpus; and determining a user experience quality QoE using the trained time series model. In the present embodiment, the scheme of the time series model established by using the average MOS value and the minimum MOS value enriches the training corpus of the established time series model, and effectively improves the accuracy of the established time series model.

In an exemplary embodiment, after determining the QoE according to the time series of the MOS values, the method further includes: sorting the determined QoE according to the quality tolerance, wherein the quality tolerance is according to the first The time series and the second time series are calculated; according to the sorting result, some users are selected as users with poor QoE; and the users with poor QoE are operated and processed. In this embodiment, the mass tolerance is calculated according to each average MOS value in the first time series and each minimum MOS value in the second sequence. The specific calculation formula is detailed in the specific embodiment.

The present disclosure is described in detail below in conjunction with specific embodiments:

Specific example 1:

The embodiment of the present disclosure provides a method for detecting an IPTV or OTT quality difference user (corresponding to the method for determining the user experience quality QoE in the foregoing), which enables the IPTV and OTT operators to accurately monitor the video viewing experience of the user, which is a potential Provide early warning of fault problems, so as to timely repair problems such as program source, CDN network, transmission network or playback terminal, and improve user satisfaction.

The detection method of the IPTV or OTT quality difference user mainly includes the following steps:

(1) Step 1: The set-top box probe calculates the Key Performance Indicators (KPI) indicators of the IPTV or OTT video playback according to a certain period (for example, 10 seconds) (corresponding to the first period in the above). The basic KPI indicators include a user identification ID, a sampling time point (corresponding to the video sampling time point in the above), a first slow delay (corresponding to the first play buffer delay of the video in the above), and a carton duration sequence (corresponding to the above) The video has a long duration sequence) and so on.

(2) Step 2: Calculate the MOS value of the user experience at the current time according to the basic KPI indicator. The MOS value calculation is a polynomial model. The polynomial model considers the first latency, the freezeTime (corresponding to the total length of the video in the above), and the freezeTime fluctuation (corresponding to the video in the above). Carton variation range) three impact factors. The respective models of the three impact factors are described in detail below.

a) first latency (in ms)

Note x ₁ = latency / (1000 * playDuration), (playDuration indicates the playing time, in seconds, when playDuration < =0, MOS value is not calculated, the same below), then 0 ≤ x ₁ ≤ 1. Let y ₁ be the MOS value of the corresponding part, then

y ₁ = a*x ₁ +b,

Among them, a and b are function parameters.

For y ₁ constraints:

Note: The latency here is the first play buffer delay, referred to as the first slow delay. For continuous playback, it is not the first play record, and the latency value should be 0.

b) total length of the carton (freezeTime, in ms)

Note x ₂ = freezeTime / (1000 * playDuration), then 0 ≤ x ₂ ≤ 1; x ₃ = freezeTime. Here, for the case where the sampling period is 10 s, for a sampling period other than 10 s, it is mapped to within 10 s. Let y ₂ be the MOS value of the corresponding part, then

y ₂ =c/(d+exp(-e*x ₂ -f*x ₃ ))

Among them, c, d, e, and f are function parameters, and exp is an exponential function. Consider the relative and absolute quantities of the total duration of the Carton, where x ₂ represents the relative amount of carton and x ₃ represents the absolute amount of carton.

For y ₂ constraints:

c) Carton change range

Let Q denote the stuck time series, then

among them

Indicates the duration of the ith monitoring point. For example, in the 10-second adoption period, the duration of each second is 1, 0, 0, 0, 0, 0, 0, 0, 0, 0.5, then Q = {1, 0, 0 , 0,0,0,0,0,0,0.5}.

Note that σ is the standard deviation of the time series Q, then

Where μ is the expectation of the time series Q, then 0 ≤ σ ≤ 0.5. Let y ₃ be the MOS value of the corresponding part, then

y ₃ = 2σ

Combining the above three partial mos values, the formula of the overall objective MOS (oMOS) is obtained:

y=y ₂ -h·y ₃ -i·(5-y ₁ )

Where h and i are function parameters, and the weights of y ₃ and y ₁ are controlled.

For y constraints:

In addition, the MOS model also introduces direct constraints on the indicator. When the indicator satisfies the constraint, the MOS value can be directly derived. Let j and m denote the minimum lower limit and the maximum upper limit threshold of x ₁ , respectively, and k and n represent the minimum lower limit and the maximum upper limit threshold of x ₂ , respectively. Then the final oMOS value can be given directly (skip the three local MOS formulas above):

(3) Step 3: During a certain period (for example, 5 minutes), the set-top box probe will report three MOS values of avgmos, minmos, and maxmos, which are simple mathematical statistics of the MOS value of the previous step. Avgmos represents the average MOS value, minmos represents the minimum MOS value, and maxmos represents the maximum MOS value. For example, if the sampling period of the probe is 10 seconds and the data is reported every 5 minutes, then there are 30 MOS values, which are assumed to be {4, 5, 5, 5, 5, 5, 5, 5, 5 1,1,5,2,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5}, then avgmos is 4.6,minmos Is 1, maxmos is 5.

(4) Step 4: In a period of time, such as 1 day, the time series of avgmos and minmos are obtained. This time period is set according to actual needs. The avgmos and minmos indicators are used to discard the maxmos indicator because the avgmos and minmos indicators are basically able to determine the current user experience is not good, without the need to add maxmos, the model can be simplified. Of course, if you keep maxmos, it is ok to use avgmos, minmos, and maxmos as the next input, but the model will be slightly more complicated and the accuracy will not improve much. The time series style of avgmos is this: {4.6, 4, 3.5, 4.1, 5, 4.2,...}; the time series style of minmos is similar to avgmos, like this: {2,1,1,1,2,2.5 ,...}.

(5) Step 5: According to the time series corpus composed of the avgmos and minmos binary groups, the time series model (such as LSTM, GRU) in the deep learning that has been trained is used for prediction to determine whether it is a quality user. The topology of the time series model is such that the input layer consists of two common neurons, the hidden layer consists of several circulating neurons (LSTM elements, GRU elements), and the output layer consists of two common neurons. The two neurons of the input layer receive the two variable values of avgmos and minmos, respectively.

When training a time series model, you must first obtain training corpus. The training corpus is obtained mainly through two steps. Firstly, the time series corpus composed of the avgmos and minmos binary groups is obtained through the process similar to the above-mentioned time series model prediction process, and then the corpus is manually classified, if it is the quality difference. Set to class 1, if not, set to class 0. The training corpus style after classification is as follows, where the colon indicates the class to which the binary time series belongs; the colon indicates the binary time series, the binary elements are separated by spaces, and the binary elements are separated by commas, commas The front represents the avgmos element, and the comma represents the minmos element. This training data format is only a reference format, and the implementer can set his own data format as long as it is easy to identify.

0:5,5 5,5 5,5 5,5

0:5,4 5,4.3 4.8,3.5 5,5 5,4.2 4.9,4.6 5,5

0:4.9,4 5,5 5,5 4.7,3.6 4.8,3.9 5,5 5,5 4.7,4

1:1,1 1,1 1,1 1,1 1,1

1:1.5,1 2,1 2.6,1.1 2.5,1.5 2,1.3

1:2,1 2.7,1 1.3,1 2,1.6 3,1.8 5,1 2.8,1.5 3.6,1 1.5,1 2.5,1.8

...

(6) Step 6: According to the selected quality difference users, sort by weight tolerance in descending order. To calculate the tolerance for tolerance, first, calculate the probability contribution rate of avgmos and minmos in a single record. Let x ₁ denote avgmos for a single record, x ₂ denote minmos for a single record, and g(x) denote a contribution probability of a single record with MOS (avgmos or minmos) as an independent variable (referred to as MOS quality difference contribution rate), then

Among them, α ₁ and α ₂ are parameters of MOS mass difference contribution rate, and there are α ₁ ∈[1,3), α ₂ ∈(3,5], which respectively represent the lower limit cutoff threshold and upper limit cutoff of MOS mass difference contribution rate. Threshold. The lower limit cutoff threshold means that when MOS is smaller than α ₁ , the MOS mass difference contribution rate is directly set to 1; the upper limit cutoff threshold means that when MOS is greater than α ₂ , the mos quality difference contribution rate is directly 0. According to g(x) For the formula, we know that g(x) ∈ [0, 1].

Here, the quality tolerance corresponding to a single user avgmos and minmos can be calculated. Assume that there are m play records in a period of time (such as one day).

Indicates the tolerance of the avgmos corresponding to the quality difference,

Indicates the tolerance of mass tolerance corresponding to minmos, t _i (i=1...m) indicates the playing time of the ith playback record, then there is

Let f(x) denote the tolerance of the quality difference user x, then

Where w ₁ represents the weight of the asymmetry tolerance corresponding to avgmos, and w ₂ represents the weight of the tolerance tolerance corresponding to minmos. According to experience, w ₁ can be set to 1.0, and w ₂ can be set to 0.25.

Finally, according to the obtained user's quality tolerance f(x), the order is sorted in descending order of f(x). According to the actual situation, the user with the highest ranking is intercepted and taken as the final user of the quality difference, and presented to the operation and maintenance personnel.

Specific example 2

FIG. 3 is a flowchart of time-series-based quality difference user detection in an IPTV or OTT according to an embodiment of the present disclosure. As shown in FIG. 3, the embodiment provides a method for detecting an IPTV/OTT quality difference user, including the following steps:

(1) Step 302: The set top box probe calculates the basic KPI indicator of the IPTV/OTT video playing according to a certain period (for example, 10 seconds). These basic KPI indicators include user ID, sampling time point, first slow delay, and carton duration sequence.

(2) Step 304: Calculate the MOS value of the user experience at the current time according to the basic KPI indicator. 4 is a flow chart for calculating a MOS value in the embodiment. As shown in FIG. 4, the following steps are included:

Step 402: Extract two indexes of the first slow delay and the long duration sequence from the basic KPI indicator of the IPTV video playing;

Step 404: Establish a linear model for the first slow delay; obtain a total stagnation time from the Carton duration sequence, and establish a sigmoid model; and establish a Karton transform amplitude model for the Carton time series. Using these three models, respectively obtain the MOS values of the corresponding parts, which are recorded as y1, y2, y3;

Step 406: The MOS value of the corresponding portion of the total length of the Carton is the main influence factor, and a certain proportion of y1 and y3 are respectively subtracted, thereby obtaining the final MOS value.

The MOS value calculation is a polynomial model. The polynomial model considers three factors: the first delay, the freezeTime, and the freezeTime fluctuation. The following is a detailed description of the respective models of these three impact factors:

a) first latency (in ms)

Note x ₁ = latency / (1000 * playDuration), (playDuration indicates the playing time, in seconds, when playDuration < =0, MOS value is not calculated, the same below), then 0 ≤ x ₁ ≤ 1. Let y ₁ be the MOS value of the corresponding part, then

y ₁ = a*x ₁ +b,

Among them, a and b are function parameters.

For y ₁ constraints:

Note: The latency here is the first play buffer delay, referred to as the first slow delay. For continuous playback, it is not the first play record, and the latency value should be 0.

b) total length of the carton (freezeTime, in ms)

Note x ₂ = freezeTime / (1000 * playDuration), then 0 ≤ x ₂ ≤ 1; x ₃ = freezeTime, where the sampling period is 10 s, for a sampling period other than 10 s, map it to 10 s Inside. Let y ₂ be the MOS value of the corresponding part, then

y ₂ =c/(d+exp(-e*x ₂ -f*x ₃ ))

Among them, c, d, e, and f are function parameters, and exp is an exponential function. Consider the relative and absolute quantities of the total duration of the Carton, where x ₂ represents the relative amount of carton and x ₃ represents the absolute amount of carton.

For y ₂ constraints:

c) Carton change range

Let Q denote the stuck time series, then

among them

Indicates the duration of the ith monitoring point. For example, in the 10-second adoption period, the duration of each second is 1, 0, 0, 0, 0, 0, 0, 0, 0, 0.5, then Q = {1, 0, 0 , 0,0,0,0,0,0,0.5}.

Note that σ is the standard deviation of the time series Q, then

Where μ is the expectation of the time series Q, then 0 ≤ σ ≤ 0.5. Let y ₃ be the MOS value of the corresponding part, then

y ₃ = 2σ

Combining the above three partial mos values, the formula of the overall objective MOS (oMOS) is obtained:

y=y ₂ -h·y ₃ -i·(5-y ₁ )

Where h and i are function parameters, and the weights of y ₃ and y ₁ are controlled.

For y constraints:

In addition, the MOS model also introduces direct constraints on the indicator. When the indicator satisfies the constraint, the MOS value can be directly derived. Let j and m denote the minimum lower limit and the maximum upper limit threshold of x ₁ , respectively, and k and n represent the minimum lower limit and the maximum upper limit threshold of x ₂ , respectively. Then the final oMOS value can be given directly (skip the three local MOS formulas above):

The MOS value of the user experience can be obtained by the above model. Through many experiments, the reference experience values of the above parameters a, b, c, d, e, f, h, i, j, k, m, n in IPTV live broadcast, IPTV on demand, OTT live broadcast, OTT on-demand are obtained:

In the IPTV live broadcast:

a=-6, b=5

c=1.22610377404164,d=-0.782831270518380,e=-0.645419518877440,f=-0.0258933009008712

h=0.45, i=1

j=0, k=0, m=0.7, n=0.9

In IPTV on demand:

a=-6, b=5.5

c=1.22610377404164,d=-0.782831270518380,e=-0.645419518877440,f=-0.0258933009008712

h=0.4, i=1

j=0, k=0, m=0.8, n=0.9

In the OTT live broadcast:

The parameter values are on demand with IPTV.

In OTT on demand:

The parameter values are on demand with IPTV.

The parameters a, b, c, d, e, and f are obtained through model fitting training; the parameters h, i, j, k, m, and n are obtained through manual experience. According to the video subjective MOS evaluation standard, the training data of the corresponding part of the first slow delay and the total duration of the Karton is obtained manually. The video subjective MOS evaluation standard is shown in Table 1.

Table 1

Score	MOS scoring standard
5	Excellent (video playback is very smooth, can not be perceived as Caton)
4	Good (can sense that the video is slightly stuck, but acceptable)
3	Qualified (can clearly perceive the video to have a card, but can bear)
2	Inferior (video card is serious, barely acceptable)
1	Oops (video stuck is very serious, totally unacceptable)

The total length of the Caton corresponds to the parameters of the partial MOS model, which is obtained by fitting the least squares algorithm. The training data is shown in Table 2:

Table 2

(3) Step 306: In a certain period (for example, 5 minutes), the set top box probe reports three MOS values of avgmos, minmos, and maxmos, and these MOS values are simple mathematical statistics of the MOS values of the previous step. Avgmos represents the average MOS value, minmos represents the minimum MOS value, and maxmos represents the maximum MOS value. For example, if the sampling period of the probe is 10 seconds and the data is reported every 5 minutes, then there are 30 MOS values, which are assumed to be {4, 5, 5, 5, 5, 5, 5, 5, 5 1,1,5,2,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5}, then avgmos is 4.6,minmos Is 1, maxmos is 5.

(4) Step 308: In a period of time, such as 1 day, a time series of avgmos and minmos is obtained. This time period is set according to actual needs. The avgmos and minmos indicators are used to discard the maxmos indicator because the avgmos and minmos indicators are basically able to determine the current user experience is not good, without the need to add maxmos, the model can be simplified. Of course, if you keep maxmos, it is ok to use avgmos, minmos, and maxmos as the next step, but the model will be slightly more complicated and the accuracy will not improve much. The time series style of avgmos is this: {4.6, 4, 3.5, 4.1, 5, 4.2,...}; the time series style of minmos is similar to avgmos, like this: {2,1,1,1,2,2.5 ,...}.

(5) Step 310: According to the time series corpus composed of the avgmos and minmos binary groups, using the time series model that has been trained in deep learning (such as Long Short-term Memory (LSTM), global positioning) The system receiving unit (GPS Receiving Unit (GRU)) performs prediction to determine whether it is a quality user. Take the LSTM time series model as an example to introduce how to train a time series model. FIG. 5 is a flowchart of training the IPTV/OTT quality difference user detection time series model in the embodiment, as shown in FIG. 5:

Step 502: First, obtain a time series corpus composed of avgmos and minmos binary groups by a process similar to the time series model prediction process above, and classify the corpora by manual, and if the quality difference is set to class 1, if It is not set to class 0. The training corpus style after classification is as follows, where the colon indicates the class to which the binary time series belongs; the colon indicates the binary time series, the binary elements are separated by spaces, and the binary elements are separated by commas, commas The front represents the avgmos element, and the comma represents the minmos element.

0:5,5 5,5 5,5 5,5

0:5,4 5,4.3 4.8,3.5 5,5 5,4.2 4.9,4.6 5,5

0:4.9,4 5,5 5,5 4.7,3.6 4.8,3.9 5,5 5,5 4.7,4

1:1,1 1,1 1,1 1,1 1,1

1:1.5,1 2,1 2.6,1.1 2.5,1.5 2,1.3

1:2,1 2.7,1 1.3,1 2,1.6 3,1.8 5,1 2.8,1.5 3.6,1 1.5,1 2.5,1.8

...

Step 504: According to the time series corpus composed of the avgmos and minmos binary groups, the time series model (such as LSTM, GRU) in the deep learning is used to train, and the trained time series is obtained.

FIG. 6 is a topological structural diagram of the time series model in the present embodiment. As shown in FIG. 6, only one hidden layer is used in the middle of the LSTM time series model. The topology of the LSTM time series model is such that the input layer consists of two common neurons, the hidden layer consists of ten LSTM elements, and the output layer consists of two ordinary neurons.

7 is an internal structure diagram of an LSTM element in this embodiment. As shown in FIG. 7, the structure of the LSTM element is such that it includes a new input x _t , an output h _t , an input gate i _t , a forgotten gate f _t , and an output. The gate o _t , the input gate i _t , the forget gate f _t , and the output gate o _t are used to control the value of each step output so that the error remains unchanged in the neuron transfer. The LSTM element is a special case of the cyclic neural network. The new input and each gate will use the previous output h _t-1 as part of this input, so the new input x _t , the input gate i _t , the forget gate f _t , the output The input of the gate o _t is composed of a [x _t , h _t-1 ] binary group. In this embodiment, x _t is a two-dimensional vector composed of avgmos and minmos.

The LSTM element new input [x _t , h _t-1 ] obtains a candidate value C _{t of the} memory element via the activation function σ _C , and its formula is:

C _t =σ _C (Wc[x _t ,h _t-1 ]+b _C )

Where Wc represents the connection weight and b _C represents an activation threshold of the activation function.

The input gate is used to adjust the size of the candidate value C _t , and the output of the input gate is:

C _t =σ _i (W _i [x _t ,h _t-1 ]+b _i )

Wherein W _i represents the connection weight, b _i denotes a threshold of activation of the activation function. The candidate value C _t is adjusted by the input gate, and its value is: C _t · i _t .

The forgotten gate is used to control the memory state of the LSTM element S _t-1 , and the output of the forgotten gate is:

f _t =σ _f (W _f [x _t ,h _t-1 ]+b _f )

Where W _f represents the connection weight and b _f represents an activation threshold of the activation function. The memory state S _t-1 is adjusted by the input gate, and its value is: f _t · S _t-1 .

At this time, the state S _t at time _t is obtained by weighting the previous time state S _t-1 and the candidate value of the state update:

S _t =f _t ·S _t-1 +C _t ·i _t

As output gate o _t S _t a weight state of the final output, the size of the control output of a state S _t. The formula for the output gate o _t is:

o _t =σ _o (W _o [x _t ,h _t-1 ]+b _o )

The output of the final LSTM element is:

σ _C , σ _i , σ _f , σ _o , and σ _S are all activation functions. Usually, the three functions σ _i , σ _f , and σ _o are set to the sigmoid function, and the two functions σ _C and σ _S are set to Tanh function (hyperbolic tangent function). In this embodiment, the three activation functions σ _i , σ _f , and σ _o on the hidden layer LSTM element adopt a tanh function; the activation functions of two common neurons in the output layer adopt a softmax function.

During the training process, the weights are updated using the Nesterov method, and the gradients are stochastic gradient descent; the training learning rate is set to 0.025.

(6) Step 312: Sort the quality tolerances in descending order according to the selected quality difference users. To calculate the tolerance for tolerance, first, calculate the probability contribution rate of avgmos and minmos in a single record. Let x ₁ denote avgmos for a single record, x ₂ denote minmos for a single record, and g(x) denote a contribution probability of a single record with MOS (avgmos or minmos) as an independent variable (referred to as MOS quality difference contribution rate), then

Where α ₁ and α ₂ are parameters of MOS mass difference contribution rate, and α ₁ ∈[1,3), α ₂ ∈(3,5], respectively represent the lower cutoff threshold and upper limit cutoff threshold of mos quality difference contribution rate. The lower limit cutoff threshold means that when mos is smaller than α ₁ , the mos quality difference contribution rate is directly set to 1; the upper limit cutoff threshold means that when mos is greater than α ₂ , the mos quality difference contribution rate is directly 0. According to g(x) The formula is g(x)∈[0,1].

Next, the quality tolerance corresponding to the individual users avgmos and minmos is calculated. Assume that there are m play records in a period of time (such as one day).

Indicates the tolerance of the avgmos corresponding to the quality difference,

Indicates the tolerance of mass tolerance corresponding to minmos, t _i (i=1...m) indicates the playing time of the ith playback record, then there is

Let f(x) denote the tolerance of the quality difference user x, then

Where w ₁ represents the weight of the asymmetry tolerance corresponding to avgmos, and w ₂ represents the weight of the tolerance tolerance corresponding to minmos. According to experience, w _{1 is} set to 1.0 and w _{2 is} set to 0.25.

Finally, according to the obtained user's quality tolerance f(x), the order is sorted in descending order of f(x). According to the actual situation, the user with the highest ranking is intercepted and taken as the final user of the quality difference, and presented to the operation and maintenance personnel.

The time series-based IPTV or OTT quality difference user detection method provided by the present disclosure realizes an accurate evaluation of the user experience, and presents the evaluation value of the user experience in a time series form, and uses the time series model in the deep learning through the time sequence. Accurately determine whether the current user is a poor user in a certain period of time, that is, a poorly perceived user, which improves the accuracy of the IPTV and OTT operators detecting the quality difference user, and finds the problem of the program source, server, transmission network or playback terminal in time. Provide early warning to help IPTV and OTT operators better maintain users.

A more accurate user experience MOS model scoring method provided in this embodiment is based on the MOS scoring time series of the user experience, and uses the time series model in the deep learning to accurately determine whether the current user is a quality difference within a certain period of time. user.

Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases, the former is The usual implementation. Based on this understanding, the solution of the present disclosure may be embodied in the form of a software product stored in a storage medium (such as a ROM/RAM, a magnetic disk, an optical disk), and includes a plurality of instructions for making one The terminal device (which may be a cell phone, computer, server, or network device, etc.) performs the methods described in various embodiments of the present disclosure.

In the embodiment, a device for determining the user experience quality QoE is provided. The device is used to implement the foregoing embodiments and exemplary embodiments, and details are not described herein. As used below, the term "module" may implement a combination of software and/or hardware of a predetermined function. Although the devices described in the following embodiments are typically implemented in software, hardware, or a combination of software and hardware, is also possible and contemplated.

FIG. 8 is a structural block diagram of a determining apparatus for user experience quality QoE according to an embodiment of the present disclosure. As shown in FIG. 8, the apparatus includes: a first determining module 802, a second determining module 804, a third determining module 806, and a fourth The module 808 is determined, and the device is described in detail below:

The first determining module 802 is configured to determine a key performance indicator KPI of the video broadcast of the Internet Protocol Television IPTV or OTT;

The second determining module 804 is connected to the first determining module 802, and is configured to determine an average opinion score MOS value of the user experience at a predetermined time according to the KPI;

The third determining module 806 is connected to the second determining module 804, and configured to determine a time series of the MOS values by using the plurality of MOS values calculated within a predetermined time period;

The fourth determining module 808 is connected to the third determining module 806, and is configured to determine the user experience quality QoE according to the time series of the MOS values.

In an exemplary embodiment, the first determining module 802 includes: a first calculating unit, configured to calculate the foregoing IPTV or the KPI of the OTT according to a first predetermined period; wherein the KPI includes at least one of the following: a user identifier ID The time point at which the KPI is sampled, the first play buffer delay of the video, and the sequence of the duration of the video.

In an exemplary embodiment, the second determining module 804 includes: a second calculating unit configured to calculate the MOS value of the user experience in the predetermined time according to the polynomial mode, wherein the polynomial mode is determined according to at least one of the following factors: : The first play buffer delay of the above video, the total duration of the above video, and the magnitude of the change of the above video.

In an exemplary embodiment, the third determining module 806 includes: a first determining unit configured to determine an average MOS value and a minimum MOS value in each of the sub-periods within the predetermined time period by using a plurality of MOS values calculated within a predetermined time period And at least one of a maximum MOS value; the second determining unit being configured to determine a time series of the MOS value according to at least one of an average MOS value, a minimum MOS value, and a maximum MOS value in each of the sub-periods described above.

In an exemplary embodiment, in a case where the average MOS value and the minimum MOS value in each of the sub-periods within the predetermined period of time are determined, the fourth determining module includes: a third determining unit configured to determine by the first time series And a time series corpus composed of a second time series, wherein the first time series is a time series of MOS values composed of average MOS values in the respective sub-periods, and the second time series is determined by each of the sub-periods a time series of MOS values composed of minimum MOS values; a training unit configured to train the pre-established time series model using the time series corpus; and a fourth determining unit configured to determine a user experience quality QoE using the trained time series model.

In an exemplary embodiment, the apparatus further includes: a sorting module, configured to sort the determined QoE according to a quality tolerance after determining the QoE according to a time series of the MOS values, wherein the quality tolerance is According to the first time sequence and the second time sequence, the selection module is configured to select a part of the user as a QoE difference user according to the sorting result, and the processing module is configured to perform operation and maintenance processing on the user with the poor QoE.

According to another embodiment of the present disclosure, there is further provided a storage medium, the storage medium comprising a stored program, wherein the program is executed to perform the method described above.

According to another embodiment of the present disclosure, there is further provided a processor, wherein the processor is configured to execute a program, wherein the program is executed to perform the method described above.

The above modules may be implemented by software or hardware. For the latter, the foregoing may be implemented by, but not limited to, the above modules are all located in the same processor; or, the above modules are respectively located in different combinations. In the processor.

In the present embodiment, the above storage medium may be arranged to store program code for performing the above steps.

In this embodiment, the foregoing storage medium may include, but is not limited to, a USB flash drive, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, and a magnetic memory. A variety of media that can store program code, such as a disc or a disc.

Embodiments of the present disclosure also provide a processor for running a program, wherein the program executes the steps of any of the above methods when executed.

For specific examples in this embodiment, reference may be made to the examples described in the foregoing embodiments and exemplary embodiments, and details are not described herein again.

Those of ordinary skill in the art will appreciate that all or some of the steps, systems, and functional blocks/units of the methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be composed of several physical The components work together. Some or all of the components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer readable medium, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and nonvolatile, implemented in any method or technology for storing information, such as computer readable instructions, data structures, program modules or other data. Sex, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridge, magnetic tape, magnetic disk storage or other magnetic storage device, or may Any other medium used to store the desired information and that can be accessed by the computer. Moreover, it is well known to those skilled in the art that communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and can include any information delivery media. .

The above description is only exemplary embodiments of the present disclosure, and is not intended to limit the disclosure, and various changes and modifications may be made to the present disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the scope of the present disclosure are intended to be included within the scope of the present disclosure.

Industrial applicability

Through the present disclosure, since the set top box determines the key performance indicator KPI of the video broadcast of the Internet Protocol Television IPTV or OTT; determines the average opinion score MOS value of the user experience at the predetermined time according to the KPI; and utilizes the plurality of MOS values calculated within the predetermined time period Determining a time series of MOS values; determining user experience quality QoE based on a time series of MOS values. Therefore, the quality user can be selected according to the determined user experience quality QoE. Therefore, the problem that the accuracy of the user who detects the quality difference is not high can be solved, and the effect of improving the accuracy of the user of the detection quality is achieved.

Claims (14)

1. A method for determining user experience quality QoE, comprising:

   Determining the Internet Protocol Television IPTV or providing the user with key performance indicators KPI for video playback of various application services OTT via the Internet;

   Determining, according to the KPI, an average opinion score MOS value of a user experience at a predetermined time;

   Determining a time series of the MOS values using a plurality of the MOS values calculated within a predetermined time period;

   The user experience quality QoE is determined according to the time series of the MOS values.
2. The method of claim 1, wherein determining the KPI of the IPTV or the video playback of the OTT comprises:

   Calculating the KPI of the IPTV or the OTT according to a first predetermined period; wherein the KPI includes at least one of: a user identifier ID, a time point at which the KPI is sampled, and a first play buffer delay of the video The sequence of the duration of the video.
3. The method of claim 1, wherein determining the MOS value of the user experience at a predetermined time according to the KPI comprises:

   Calculating the MOS value of the user experience at the predetermined time according to a polynomial mode, wherein the polynomial mode is determined according to at least one of: a first play buffer delay of the video, a total of the video Duration, the magnitude of the change in the video.
4. The method of claim 1, wherein determining a time series of the MOS value using a plurality of the MOS values calculated within the predetermined time period comprises:

   Determining at least one of an average MOS value, a minimum MOS value, and a maximum MOS value in each of the sub-periods within the predetermined time period using a plurality of the MOS values calculated within the predetermined time period;

   A time series of the MOS values is determined based on at least one of an average MOS value, a minimum MOS value, and a maximum MOS value in each of the sub-periods.
5. The method according to claim 4, wherein, in the case where the average MOS value and the minimum MOS value in each of the sub-periods within the predetermined time period are determined, determining the user experience quality QoE according to the time series of the MOS values comprises:

   Determining a time series corpus consisting of a first time series and a second time series, wherein the first time series is a time series of MOS values consisting of average MOS values in each of the sub-periods, the second time The sequence is a time series of MOS values consisting of minimum MOS values in each of the sub-periods;

   Training the pre-established time series model with the time series corpus;

   The user experience quality QoE is determined using the trained time series model.
6. The method of claim 5, wherein after determining the QoE based on a time series of the MOS values, the method further comprises:

   Sorting the determined QoE according to the tolerance of the quality, wherein the quality tolerance is calculated according to the first time series and the second time series;

   According to the sorting result, some users are selected as users with poor QoE;

   The operation and maintenance process is performed on the user with poor QoE.
7. A device for determining quality of user experience QoE, comprising:

   a first determining module, configured to determine an Internet Protocol Television IPTV or provide a user with a key performance indicator KPI for video playback of various application services OTT via the Internet;

   a second determining module, configured to determine an average opinion score MOS value of the user experience at a predetermined time according to the KPI;

   a third determining module, configured to determine a time series of the MOS value by using the plurality of the MOS values calculated within a predetermined time period;

   The fourth determining module is configured to determine a user experience quality QoE according to a time series of the MOS values.
8. The apparatus of claim 7, wherein the first determining module comprises:

   a first calculating unit, configured to calculate the KPI of the IPTV or the OTT according to a first predetermined period; wherein the KPI includes at least one of: a user identifier ID, a time point at which the KPI is sampled, and the The first play buffer delay of the video, the sequence of the duration of the video.
9. The apparatus of claim 7, wherein the second determining module comprises:

   a second calculating unit, configured to calculate the MOS value of the user experience at the predetermined time according to a polynomial mode, wherein the polynomial mode is determined according to at least one of: a first play buffer delay of the video, The total duration of the video, the magnitude of the change in the video.
10. The apparatus of claim 7, wherein the third determining module comprises:

    a first determining unit, configured to determine at least one of an average MOS value, a minimum MOS value, and a maximum MOS value in each of the sub-periods within the predetermined time period by using the plurality of the MOS values calculated within the predetermined time period;

    The second determining unit is configured to determine a time series of the MOS value according to at least one of an average MOS value, a minimum MOS value, and a maximum MOS value in each of the sub-periods.
11. The apparatus according to claim 10, wherein, in the case where the average MOS value and the minimum MOS value in each of the sub-periods within the predetermined time period are determined, the fourth determining module comprises:

    a third determining unit, configured to determine a time series corpus consisting of the first time series and the second time series, wherein the first time series is a time of a MOS value composed of average MOS values in each of the sub-periods a sequence, the second time series being a time series of MOS values consisting of minimum MOS values in each of the sub-periods;

    a training unit configured to train the pre-established time series model using the time series corpus;

    The fourth determining unit is configured to determine the user experience quality QoE by using the trained time series model.
12. The apparatus of claim 11 wherein said apparatus further comprises:

    a sorting module, configured to sort the determined QoE according to a quality tolerance after determining the QoE according to a time series of the MOS values, wherein the quality tolerance is according to the first time series Calculated with the second time series;

    Select a module, and set to select some users as QoE poor users according to the sorting result;

    The processing module is configured to perform operation and maintenance processing on the user with poor QoE.
13. A storage medium, wherein the storage medium comprises a stored program, the program being executed to perform the method of any one of claims 1 to 6.
14. A processor, wherein the processor is operative to execute a program, the program running to perform the method of any one of claims 1 to 6.

Images