CN107872716B - A streaming data analysis method, device and electronic equipment with post-packaging - Google Patents

Abstract

The embodiment of the present invention discloses a streaming data analysis method, device and electronic equipment with post-package, which relate to data analysis technology and can improve the efficiency of streaming analysis. The post-encapsulated streaming data analysis method includes: determining that the data in the received file is post-encapsulated streaming data; analyzing the information of the first threshold byte before the post-encapsulated streaming data, Obtain atomic structure information; extract the information of the second threshold byte in the atomic structure information to obtain type structure information, the first threshold is twice the second threshold; after determining that the type structure information is identified as an index structure Afterwards, the requested data is parsed according to the index structure. The present invention is suitable for parsing the streaming data of MP4 post-encapsulation.

Description

A streaming data analysis method, device and electronic equipment with post-packaging

technical field

The present invention relates to data parsing technology, in particular to a stream data parsing method, device and electronic equipment with post-packaging.

Background technique

MP4 is a set of compression coding standards for audio and video information, formulated by the Moving Picture Experts Group (MPEG, Moving Picture Experts Group) under the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC). Compared with media files encapsulated by other encapsulation formats, such as RMVB, AVI, FLV, WMV, MKV and other media files, MP4 files are more practical and popular than media files encapsulated by ISO/IEC 14496-12 standard , and thus has been widely used.

In order to be applicable in different scenarios, the encapsulation formats of MP4 files include: MP4-pre-encapsulation, MP4-post-encapsulation, MP4-H5 encapsulation and MP4-DASH encapsulation. Among them, MP4-post-encapsulation is mainly used in file data scenarios, less used in streaming data scenarios, MP4-H5 encapsulation and MP4-DASH encapsulation are mainly used in streaming data scenarios, and MP4-front encapsulation can be applied to file Data scenarios can also be used in streaming data scenarios. For the file data scenario, when the client parses the encapsulated MP4 file (data), it can obtain the file handle in the encapsulated MP4 file (data), use the file handle to search the index, and realize the positioning and analysis of the data according to the index , and start playing from the positioned data. For streaming data scenarios, due to the index of MP4-pre-encapsulation, MP4-H5 encapsulation and MP4-DASH encapsulation (specifically the MOOV or MOOFBox structure, this structure specifies the specific codec parameters, frame length, frame offset, frame Timestamp, frame position and other important information) are located in front of the load data. When performing stream parsing, by obtaining the index and parsing the subsequent load data according to the index, there will be no timing logic problems in parsing; but for MP4-post Encapsulation, since the index is located behind the load data, when parsing on the client side, the target MP4 file (data) needs to be downloaded and stored in the cache, and parsed in the cache, but this method requires the entire target MP4 file Parsing can only be done after the download is complete, which takes a long time to parse, and the stream parsing efficiency is low; further, if the target file is larger, the cache space required will be larger, and a large amount of memory resources will be consumed; moreover, when streaming When stream parsing, since the index is located behind the load data, there may be timing logic problems in parsing, resulting in stream parsing failure.

Contents of the invention

In view of this, the embodiments of the present invention provide a post-encapsulation stream data analysis method, device, and electronic equipment, which can improve the efficiency of stream analysis, so as to solve the problems in the existing post-package stream data analysis method. The parsing can only be performed after the entire target MP4 file is downloaded, which results in long parsing time and low streaming parsing efficiency.

In the first aspect, the embodiment of the present invention provides a post-encapsulation streaming data parsing method, including:

Determine that the data in the received file is streaming data with post-encapsulation;

Analyzing the information of the first threshold byte before the post-encapsulated streaming data to obtain atomic structure information;

Extracting the information of the last second threshold byte in the atomic structure information to obtain the type structure information, the first threshold is twice the second threshold;

If the type structure information identification is not an index structure, calculate the index offset according to the information of the first second threshold byte in the atomic structure information; request data from the index offset, and perform the parsing and the post-processing A step of setting the information of the first threshold byte of the encapsulated streaming data;

After determining that the type structure information is identified as an index structure, the requested data is parsed according to the index structure;

Wherein, determining that the data in the received file is streaming data using post-encapsulation includes:

Parse the file type field of the received file. If the file type field is identified as MP4-post-encapsulation and applied to streaming data scenarios, the data in the received file is streaming data using post-encapsulation.

With reference to the first aspect, in a second implementation manner of the first aspect, the first threshold is 8, and the second threshold is 4.

With reference to the first aspect, in the third implementation manner of the first aspect, after determining that the type structure information is identified as an index structure, parsing the requested data according to the index structure includes:

If the type structure information is identified as an index structure, parse the index structure to obtain an index;

Request data, and analyze the requested data according to the index.

With reference to the third implementation manner of the first aspect, in the fourth implementation manner of the first aspect, the calculating the index offset based on the information of the first second threshold byte in the atomic structure information includes:

Obtain the stored reserved index offset;

adding the length corresponding to the information of the first second threshold byte to the stored reserved index offset to obtain the index offset;

Utilizing the index offset to update the stored reserved index offset.

With reference to the fourth implementation manner of the first aspect, in the fifth implementation manner of the first aspect, if the type structure information is identified as an index structure, the method further includes:

Clear the stored reserved index offset.

With reference to the third implementation manner of the first aspect, in the sixth implementation manner of the first aspect, the requesting data, parsing the requested data according to the index includes:

Obtain the playback positioning time point;

Request data from the playback location point in time and parse the playback.

With reference to the sixth implementation manner of the first aspect, in the seventh implementation manner of the first aspect, the requesting data at the playback location time point includes:

Accumulate the frame timestamp in the video track timestamp-frame correspondence table structure box in the client;

From the accumulated frame time stamps, obtain the frame number corresponding to the frame time stamp closest to the playback positioning time point;

Use the block offset information table box of the video track to query the video frame offset corresponding to the frame number;

Request data at the video frame offset from the server.

With reference to the seventh implementation manner of the first aspect, in an eighth implementation manner of the first aspect, the video frame offset is an I-frame video frame offset.

With reference to the sixth implementation manner of the first aspect, in the ninth implementation manner of the first aspect, the requesting data at the playback location time point includes:

According to the frame duration and frame naming rules, calculate the frame number corresponding to the playback positioning time point;

Use the block offset information table box of the video track to query the video frame offset corresponding to the frame number;

Request data at the video frame offset from the server.

With reference to the sixth implementation manner of the first aspect, in the tenth implementation manner of the first aspect, the parsing and playing includes:

The analysis layer parses the requested data, if it is the first frame, and the first frame is a B frame or a P frame, discarding the B frame or P frame, if the first frame is an I frame, transparently transmits it to the playback layer for playback;

If it is not the first frame, transparently transmit the analyzed B frame or P frame or I frame to the playback layer for playback.

In the second aspect, an embodiment of the present invention provides a post-encapsulation streaming data analysis device, including: a post-encapsulation determination module, an atomic structure analysis module, a type structure extraction module, a judgment unit, an index offset calculation unit, a data request unit and data parsing module, in which,

A post-encapsulation determination module, configured to determine that the data in the received file is streaming data with post-encapsulation;

The atomic structure analysis module is used to analyze the information of the first threshold byte of the post-encapsulated streaming data to obtain the atomic structure information;

The type structure extraction module is used to extract the information of the second threshold byte in the atomic structure information to obtain the type structure information, and the first threshold is twice the second threshold;

A judging unit, configured to judge whether the type structure information identifier is an index structure, if yes, notify the data parsing module, and if not, notify the index offset calculation unit;

An index offset calculation unit, configured to calculate an index offset according to the information of the first second threshold byte in the atomic structure information;

A data request unit, configured to request data from the index offset and output to the atomic structure analysis module;

A data parsing module, configured to parse the requested data according to the index structure after determining that the type structure information is identified as an index structure;

Wherein, determining that the data in the received file is streaming data using post-encapsulation includes:

Parse the file type field of the received file. If the file type field is identified as MP4-post-encapsulation and applied to streaming data scenarios, the data in the received file is streaming data using post-encapsulation.

With reference to the second aspect, in a second implementation manner of the second aspect, the first threshold is 8, and the second threshold is 4.

With reference to the second aspect, in a third implementation manner of the second aspect, the data parsing module includes: an index acquisition unit and a data parsing unit, wherein,

an index acquiring unit, configured to parse the index structure to obtain an index;

The data parsing unit is configured to request data, and parse the requested data according to the index.

With reference to the third implementation manner of the second aspect, in the fourth implementation manner of the second aspect, the index offset calculation unit includes: a reserved index offset acquisition subunit, an index offset calculation subunit, and an update subunit unit, where

The reserved index offset acquisition subunit is used to acquire the stored reserved index offset;

An index offset calculation subunit, configured to add the length corresponding to the information of the first second threshold byte to the stored reserved index offset to obtain an index offset;

An updating subunit, configured to use the index offset to update the stored reserved index offset.

With reference to the fourth implementation manner of the second aspect, in the fifth implementation manner of the second aspect, the index offset calculation unit further includes:

The clearing subunit is configured to clear the stored reserved index offset after receiving the notification output from the judging unit to the index acquiring unit.

With reference to the third implementation manner of the second aspect, in the sixth implementation manner of the second aspect, the data parsing unit includes: a playback positioning time point acquisition subunit and an analysis playback subunit, wherein,

The playback positioning time point acquisition subunit is used to obtain the playback positioning time point;

The parsing and playing subunit is used for requesting the data of the positioning time point from the playing and parsing the playing.

With reference to the sixth implementation manner of the second aspect, in the seventh implementation manner of the second aspect, the parsing and playing subunit includes: an accumulator, a frame number acquirer, a queryer, a data requester, and an analytic player, in,

The accumulator is used to accumulate the frame timestamps in the video track timestamp-frame correspondence table structure box in the client;

A frame number acquirer, configured to acquire, from the accumulated frame time stamps, the frame number corresponding to the frame time stamp closest to the playback positioning time point;

A query device, configured to use the block offset information table box of the video track to query the video frame offset corresponding to the frame number;

A data requester, configured to request data from a server at an offset from the video frame;

The parsing player is used to parse the requested data and play it.

With reference to the seventh implementation manner of the second aspect, in an eighth implementation manner of the second aspect, the video frame offset is an I-frame video frame offset.

With reference to the sixth implementation manner of the second aspect, in the ninth implementation manner of the second aspect, the analysis and playback subunit includes: a frame number calculator, a video frame offset device, a data requester, and an analysis player, in,

A frame number calculator, configured to calculate the frame number corresponding to the playback positioning time point according to the frame duration and frame naming rules;

A video frame offset device, configured to use the block offset information table box of the video track to query the video frame offset corresponding to the frame number;

A data requester, configured to request data from a server at an offset from the video frame;

The parsing player is used to parse the requested data and play it.

With reference to the sixth implementation manner of the second aspect, in the tenth implementation manner of the second aspect, the parsing and playing includes:

The analysis layer parses the requested data, if it is the first frame, and the first frame is a B frame or a P frame, discarding the B frame or P frame, if the first frame is an I frame, transparently transmits it to the playback layer for playback;

If it is not the first frame, transparently transmit the analyzed B frame or P frame or I frame to the playback layer for playback.

In a third aspect, an embodiment of the present invention provides an electronic device, the electronic device includes: a housing, a processor, a memory, a circuit board, and a power supply circuit, wherein the circuit board is placed inside the space enclosed by the housing, and the processor and the memory are arranged on the circuit board; the power supply circuit is used to supply power to each circuit or device of the above-mentioned electronic equipment; the memory is used to store the executable program code; the processor runs and can read the executable program code stored in the memory Executing a program corresponding to the program code is used to execute any one of the aforementioned post-encapsulation streaming data analysis methods.

The embodiment of the present invention provides a post-encapsulation stream data analysis method, device, and electronic equipment, by determining that the data in the received file is stream data using post-encapsulation; parsing the post-encapsulation stream data The information of the first threshold byte before the data is obtained to obtain the atomic structure information; the information of the second threshold byte in the atomic structure information is extracted to obtain the type structure information, and the first threshold is twice the second threshold; After determining that the type structure information is identified as an index structure, parsing the requested data according to the index structure can improve the efficiency of streaming parsing, so as to solve the problem that in the existing post-encapsulation streaming data parsing method, the The entire target MP4 file can only be parsed after it has been downloaded, resulting in a long time for parsing and low stream parsing efficiency.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

FIG. 1 is a schematic flow chart of a streaming data parsing method for post-encapsulation according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of an existing playback operation;

Fig. 3 is a schematic diagram of the playback operation of the present embodiment;

FIG. 4 is a schematic flow chart of a post-encapsulation streaming data analysis method according to Embodiment 2 of the present invention;

FIG. 5 is a schematic flow chart of a post-encapsulation streaming data analysis method according to Embodiment 3 of the present invention;

FIG. 6 is a schematic structural diagram of a post-package streaming data analysis device according to Embodiment 4 of the present invention;

FIG. 7 is a schematic structural diagram of an embodiment of the electronic device of the present invention.

Detailed ways

Embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings.

It should be clear that the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

Fig. 1 is a schematic flow chart of a post-encapsulated stream data parsing method according to an embodiment of the present invention. As shown in Fig. 1, the method of this embodiment may include:

Step 101, determining that the data in the received file is streaming data with post-encapsulation;

In this embodiment, as an optional embodiment, determining that the data in the received file is streaming data using post-encapsulation includes:

Parse the file type field of the received file. If the file type field is identified as MP4-post-encapsulation and applied to streaming data scenarios, the data in the received file is streaming data using post-encapsulation.

In the present embodiment, for the MP4 file, according to the provisions of the ISO/IEC 14496-12 standard, the file or data in the MP4 encapsulation format is used as the initial structure with the file type box (FTYP Box, File Type Box) structure, and the file type box includes There is a file type (FTYP, File Type) field, which is located at the beginning of the file and used to identify the MP4 format.

In this embodiment, if the parsed file type field identifier is not MP4-post-encapsulation and is applied to a streaming data scenario, processing is performed according to the existing technical process.

Step 102, analyzing the information of the first threshold byte of the post-encapsulated streaming data to obtain atomic structure information;

In this embodiment, as an optional embodiment, the first threshold is 8.

In this embodiment, the media file that adopts MP4-post-packaging is a structure of MDAT (data)+MOOV (index), wherein, MOOV is the index, which is a codec parameter, frame timestamp, frame offset, The index structure of frame length, frame position and other information, MDAT (Media Data) is the encapsulation structure of multimedia data, that is, payload data (bare data).

According to the provisions and definitions of the ISO/IEC 14496-12 standard, no matter what kind of encapsulation structure, it will follow the description of the basic atomic structure, that is, in the first 8 bytes of stream data (frame), use the length structure and type structure The atomic structure description method, wherein, the first 4 bytes are the length structure, which identifies the payload data (the length of the MDAT data), and the last 4 bytes are the type structure, which identifies the type of the streaming data (frame). Therefore, in this embodiment, by acquiring the atomic structure information of the streaming data (frame), the atomic structure information can be used to determine whether it is an index.

Step 103, extracting the information of the last second threshold byte in the atomic structure information to obtain the type structure information, the first threshold is twice the second threshold;

In this embodiment, as an optional embodiment, the first threshold is 8, and the second threshold is 4.

Step 104, after determining that the type structure information is identified as an index structure, parse the requested data according to the index structure.

In this embodiment, the index is the basis and key for streaming data parsing. Since the MP4-post-encapsulated index is behind the load data, the existing technology needs to receive all the streaming data before it can be stored in the cache. Parsing, resulting in a long time required for parsing and low efficiency of streaming parsing. Therefore, if the index can be obtained quickly, the efficiency of streaming parsing will be directly improved.

In this embodiment, the atomic structure information is obtained by analyzing the first 8 bytes of the data, and the type structure information of the last 4 bytes in the atomic structure information is used to judge whether the current structure is the required index (MOOV Box) structure, If the type structure information indicates that the current structure is not an index structure, you can notify the server to skip the current atomic structure that is not an index structure according to the length type indicated by the first 4 bytes, that is, use the type structure of the last 4 bytes in the atomic structure information to determine the Whether the type structure is an index structure, if not, skip the structure according to the length information (load data length) indicated by the length structure in the first 4 sections, so that the server skips the streaming data of the length information until it jumps to the type structure as an index structure, there is no need to transmit the length data identified by the length type, so there is no need to parse the length data, which effectively reduces the time required for parsing and improves the efficiency of streaming parsing.

In this embodiment, as an optional embodiment, it includes:

A11, if the type structure information identifier is not an index structure, calculate the index offset according to the information of the first second threshold byte in the atomic structure information;

In this embodiment, the information of the first second threshold byte is the length structure in the atomic structure information, which indicates the length of the payload data.

In this embodiment, as an optional embodiment, calculating the index offset according to the information of the first second threshold byte in the atomic structure information includes:

B11, obtain the stored reserved index offset;

In this embodiment, if no reserved index offset is stored in the preset storage area, the stored reserved index offset is set to zero.

B12, adding the length corresponding to the information of the first second threshold byte to the stored reserved index offset to obtain an index offset;

In this embodiment, if the current structure is not a MOOV Box, calculate the index offset: use the parsed data length plus the length of the current atomic structure to obtain the data length to be offset and send it to the server. The server will start sending data from the offset position corresponding to the requested offset data length, and the client will continue to analyze after obtaining the new data, and then judge whether the current structure is a MOOV Box, and so on, until the file parsing ends or the file is found MOOV Box. For example, suppose the 4-byte field type structure is parsed for the first time. If it is determined that it is not an index structure, then according to the length information of the first 4 bytes, the server is notified to skip the data of this length information and then send it. The first 8 bytes come to judge, if the field type structure of the last 4 bytes is not an index structure, calculate the data offset of this time, that is, the sum of the data length skipped last time and the data length that needs to be skipped this time .

B13. Utilize the index offset to update the stored reserved index offset.

A12, requesting data from the index offset, and performing the step of parsing the information of the first threshold byte before the post-encapsulated streaming data;

In this embodiment, the server is notified to skip the streaming data with the payload data length indicated by the length structure, that is, the streaming data is transmitted from the calculated index offset, and the streaming data before the index offset is not transmitted .

After determining that the type structure information is identified as an index structure, parsing the requested data according to the index structure includes:

A13, if the type structure information is identified as an index structure, parse the index structure to obtain an index;

In this embodiment, the index (MOOV Box) includes information such as codec parameters, frame length, frame offset, frame time stamp, and frame position. After obtaining the index, the analysis of streaming data can be realized, that is, when the client receives the MOOV Box, it can perform streaming analysis.

In this embodiment, as an optional embodiment, if the type structure information is identified as an index structure, the method further includes:

Clear the stored reserved index offset.

A14, requesting data, parsing the requested data according to the index.

In this embodiment, as an optional embodiment, requesting data, parsing the requested data according to the index includes:

C11, obtain the playback positioning time point;

In this embodiment, the user may select a playback positioning time point during playback, or may not select a playback positioning time point, and play from a default time point.

In this embodiment, as an optional embodiment, obtaining the playback positioning time point includes:

Obtain the position point of the positioning operation performed by the user on the player progress bar of the client;

According to the total playback time, the total length of the progress bar and the current positioning operation position point, calculate the playback positioning time point.

In this embodiment, the calculated playback positioning time point is a positioning timestamp.

C12, requesting the data of the positioning time point from the playback and parsing the playback.

In this embodiment, as an optional embodiment, requesting data from the playback positioning time point includes:

D11, accumulating the frame timestamp in the video track timestamp-frame correspondence table structure box (STTS Box, Time toSample Box) in the client;

D12, from the accumulated frame time stamps, obtain the frame number corresponding to the frame time stamp closest to the playback positioning time point;

In this embodiment, the frame timestamps in the STTS Box structure field are accumulated one by one according to the order of the linked list until the sum of the frame timestamps reaches the playback positioning time point, and the frame number corresponding to the last frame timestamp participating in the accumulation is obtained.

In this embodiment, for the first frame of the multimedia data, the media data starting with the I frame is valid data (the B/P frame cannot independently present a complete picture, which will cause a blurred screen phenomenon), therefore, in this embodiment In, as an optional embodiment, the frame number is the frame number of the I frame. That is, from the accumulated frame time stamps, the frame number of the I frame corresponding to the time stamp of the I frame closest to the playback positioning time point is obtained. Specifically, the STSS Box, Sync Sample Box, that is, the I-frame frame numbers in all I-frame frame number recording tables are converted into specific timestamps through the STTS Box structure, and the converted timestamps are sequentially compared With the playback positioning time point, select a timestamp closest to the playback positioning time point (that is, the closest I frame, so the maximum error is a GOP duration, the minimum is 0, and the average is half a GOP duration), and the time Stamp and restore the I frame number, and find the offset corresponding to the I frame number of the video frame in the STCO Box.

D13, utilizing the block offset information table box (STCO Box, Chunk Offset Box) of the video track to query the video frame offset corresponding to the frame number;

In this embodiment, as an optional embodiment, the video frame offset is an I-frame video frame offset.

In this embodiment, in order to synchronize the audio and video, the audio also needs to find the corresponding data, directly search the audio STCO Box, obtain the first offset greater than the video frame offset, and obtain the audio offset.

D14, requesting the server for data offset from the video frame.

In this embodiment, STCO Box is an offset information table of each sample (frame) relative to the file header.

In this embodiment, the user performs a positioning operation on the player progress bar of the client, and a positioning timestamp can be calculated proportionally according to the total playback time, the total length of the progress bar, and the position of the current positioning point, and the timestamp is accumulated on the client side. Video track (the calculation of time stamp only refers to the video frame, and the calculation of other track frames must be synchronized with the current video frame) the frame time stamp in the STTS Box, get the frame number closest to the positioning time stamp, and then pass the STCO Box of the video track , search for the offset corresponding to the frame number (the offset is an accurate video frame offset), and finally send the offset value to the server, and the server returns the data at the requested offset to the client, and the client follows up with The pre-obtained index is used to parse and play raw data.

In this embodiment, as another optional embodiment, requesting data from the playback positioning time point includes:

E11. Calculate the frame number corresponding to the playback positioning time point according to the frame duration and frame naming rules;

E12, using the block offset information table box (STCO Box, Chunk Offset Box) of the video track to query the video frame offset corresponding to the frame number;

E13, request the data at the offset from the video frame from the server.

In this embodiment, the audio STCO Box is searched, and the first offset greater than the video frame offset is obtained to obtain the audio offset.

In this embodiment, as an optional embodiment, parsing and playing includes:

The analysis layer parses the requested data, if it is the first frame, and the first frame is a B frame or a P frame, discarding the B frame or P frame, if the first frame is an I frame, transparently transmits it to the playback layer for playback;

If it is not the first frame, transparently transmit the analyzed B frame or P frame or I frame to the playback layer for playback.

In the present embodiment, if the above-mentioned frame number is an I frame frame number, then parse the requested data, the first frame is an I frame, which can be transparently transmitted to the playback layer; if the frame number is not an I frame frame number, then parse the requested data For data, the first frame can be an I frame, a B frame, or a P frame. Since in the first frame, the B frame or the P frame cannot independently present a complete picture, which will cause a blurred screen phenomenon, only the media data starting with the I frame is valid data. In this embodiment, by adding the latest I-frame positioning mechanism in the positioning process, that is, positioning the frame number of the I-frame; Playback delay of GOP duration.

Fig. 2 is a schematic diagram of the existing playback operation. Referring to Figure 2, taking the playback positioning time point T2 as an example, in the playback operation, the underlying analysis layer will sequentially input the accurate frame (T1 time point) data corresponding to the playback positioning time point into the control layer, and the control layer will effectively Data filtering, that is, all invalid frame data before the I frame is lost. The data shown in the figure shows the data between T1 and T4, and the data that actually starts to be played is the I frame at the time point of T4. In this way, not only the analysis (request, sending, receiving, analysis, filtering, etc.) delay of invalid data between T1 and T4 is generated, but also the total playing time is reduced by (T4-T1) time.

FIG. 3 is a schematic diagram of the playback operation of this embodiment. Referring to FIG. 3 , taking the playback positioning time point T2 as an example, in this embodiment, the underlying analysis layer uses the positioning operation of the nearest I frame positioning mechanism to place the I frame time point T0 closest to the playback positioning time point T1 and subsequent The data is directly transparently transmitted to the playback layer, and the control layer does not need to filter valid data, which completely avoids the delay of sending, receiving and analyzing invalid data, and the total playback time is 1 GOP longer than the traditional positioning solution (T4-T1+T1-T0=T4-T0), which can enable the client to play valid data in the shortest time, can effectively avoid the phenomenon of blurred screen, reduce the logic load of the upper layer control, and the average error is half a group of pictures (GOP , Group OfPicture) duration, wherein, GOP refers to an I frame and other frames decoded by the I frame behind it.

Fig. 4 is a schematic flow chart of the post-encapsulated streaming data parsing method of Embodiment 2 of the present invention. As shown in Fig. 4, the method of this embodiment may include:

Step 401, the server sends data to the client;

In this embodiment, the data sent by the server is MP4-post-encapsulated streaming data (file).

Step 402, the client locates the index position, and requests the data of the index offset (offset1) from the server;

Step 403, the server sends data from the index offset, and the client parses and obtains the index;

Step 404, the client performs a positioning operation, calculates the video frame offset corresponding to the positioning operation, and requests the data of the video frame offset (offset2) from the server;

Step 405, the server sends data from the video frame offset, and the client parses the data.

Fig. 5 is a schematic flow chart of a post-encapsulated streaming data parsing method according to Embodiment 3 of the present invention. As shown in Fig. 5, the method of this embodiment may include:

Step 501, analyzing the file type of the received data;

In this embodiment, the file type (FTYP, File Type) is located at the beginning of the file and is used to identify the MP4 format. If it is MP4-post-encapsulation, step 502 is performed.

Step 502, analyzing the non-MOOV length;

In this embodiment, the non-MOOV length is the atomic structure information. If the type information in the atomic structure information indicates that it is not an index structure, step 503 is performed.

Step 503, calculating index offset;

In this embodiment, the index offset is calculated according to the length structure information in the atomic structure information and the stored reserved index offset.

Step 504, request the data at the index offset;

Step 505, judge whether it is a MOOV Box, if not, return to step 502, if yes, execute step 506;

Step 506, analyzing the index;

Step 507, whether positioning is required, if yes, execute step 508, if not, execute step 512;

In this embodiment, after the positioning operation is performed, the data requested by the client to the server is valid data, that is, payload data.

Step 508, process according to the preset offset conversion mechanism;

In this embodiment, the offset conversion mechanism is the nearest I frame positioning mechanism.

Step 509, request data;

In this embodiment, the data is requested based on the video frame offset obtained by positioning.

Step 510, analyzing data;

In this embodiment, parsing the data refers to decapsulating or decapsulating the encapsulated media data to make it into bare stream data.

Step 511, judge whether it is the last frame of data, if yes, end the process, if not, return to step 507;

Step 512, acquire data sequentially, and execute step 510.

The post-encapsulated streaming data analysis method of this embodiment determines that the data in the received file is post-encapsulated streaming data; and analyzes the information of the first threshold byte before the post-encapsulated streaming data , to obtain the atomic structure information; extract the information of the second threshold byte in the atomic structure information to obtain the type structure information, and the first threshold is twice the second threshold; after determining that the type structure information is identified as an index After the structure, the requested data is parsed according to the index structure. In this way, the type structure information can be judged through the atomic structure information, and the index can be effectively obtained without downloading the entire target MP4 file, and the MP4 file can be parsed, the time required for parsing is short, and the streaming parsing efficiency is high; at the same time, caching is required The data is small, and the memory resource consumption is low; moreover, by quickly obtaining the index, it is possible to avoid temporal logic problems in parsing, thereby improving the success rate of streaming parsing. Specifically, by proposing a fast, accurate and effective flow parsing and positioning mechanism, it has the following beneficial technical effects:

1. Get the index quickly. Among them, obtaining the index includes: determining the offset position of the index and obtaining the complete index, wherein,

Determining the offset position of the index, the media file of the MP4-post-package is a structure of MDAT (data)+MOOV (index), wherein, MOOV is the index, and MDAT (Media Data) is multimedia data, i.e. load data (naked data) encapsulation structure. Through the length field in the MDAT field, the length of the payload data is obtained, and then the server is notified to skip the data information of this length until it jumps to MOOV. When the client receives the MOOV, it can perform streaming analysis, so that it can be quickly determined. The offset position of the index.

After determining the offset position of the index, since the index is much smaller than the load data, the complete index can be obtained quickly, so that the user needs to wait for a short response time for stream parsing.

2. Accurate. The time stamp (or frame number)-offset conversion mechanism is used in the positioning operation to provide users with two positioning options of frame number and time stamp. The client requests the precise data offset position, which can avoid redundant data parsing. Additional time-consuming and waiting;

3. Effective. The latest I-frame positioning mechanism is adopted in the positioning operation, so that the client can play valid data in the shortest time, that is, the data starting with the I frame, which can effectively avoid the phenomenon of blurred screen and reduce the burden of upper-layer control logic.

Fig. 6 is a schematic structural diagram of a streaming data analysis device with post-packaging in Embodiment 4 of the present invention. As shown in Fig. 6, the device in this embodiment may include: a post-packaging determination module 61, an atomic structure analysis module 62, a type structure Extraction module 63 and data parsing module 64, wherein,

Post-encapsulation determination module 61, configured to determine that the data in the received file is streaming data using post-encapsulation;

In this embodiment, as an optional embodiment, determining that the data in the received file is streaming data using post-encapsulation includes:

Parse the file type field of the received file. If the file type field is identified as MP4-post-encapsulation and applied to streaming data scenarios, the data in the received file is streaming data using post-encapsulation.

In this embodiment, if the parsed file type field identifier is not MP4-post-encapsulation and is applied to a streaming data scenario, processing is performed according to the existing technical process.

The atomic structure analysis module 62 is used to analyze the information of the first threshold byte before the post-encapsulated streaming data to obtain the atomic structure information;

The type structure extraction module 63 is used to extract the information of the second threshold byte in the atomic structure information to obtain the type structure information, and the first threshold is twice the second threshold;

In this embodiment, as an optional embodiment, the first threshold is 8, and the second threshold is 4.

In this embodiment, the first 4 bytes are a length structure, which identifies the payload data (the length of the MDAT data), and the last 4 bytes are a type structure, which identifies the type of the streaming data (frame). Therefore, in this embodiment, by acquiring the atomic structure information of the streaming data (frame), the atomic structure information can be used to determine whether it is an index.

The data parsing module 64 is configured to parse the requested data according to the index structure after determining that the type structure information is identified as an index structure.

In this embodiment, the atomic structure information is obtained by analyzing the first 8 bytes of the data, and the type structure information of the last 4 bytes in the atomic structure information is used to judge whether the current structure is the required index (MOOV Box) structure, If the type structure information indicates that the current structure is not an index structure, the server may be notified to skip the current atomic structure that is not an index structure according to the length type indicated by the first 4 bytes.

In this embodiment, as an optional embodiment, the device further includes: a judgment unit, an index offset calculation unit, and a data request unit, wherein,

A judging unit, configured to judge whether the type structure information identifier is an index structure, if yes, notify the index acquisition unit, if not, notify the index offset calculation unit;

An index offset calculation unit, configured to calculate an index offset according to the information of the first second threshold byte in the atomic structure information;

In this embodiment, as an optional embodiment, the index offset calculation unit includes: a reserved index offset acquisition subunit, an index offset calculation subunit, and an update subunit, wherein,

The reserved index offset acquisition subunit is used to acquire the stored reserved index offset;

In this embodiment, if no reserved index offset is stored in the preset storage area, the stored reserved index offset is set to zero.

An index offset calculation subunit, configured to add the length corresponding to the information of the first second threshold byte to the stored reserved index offset to obtain an index offset;

An updating subunit, configured to use the index offset to update the stored reserved index offset.

As another optional embodiment, the index offset calculation unit further includes:

The clearing subunit is configured to clear the stored reserved index offset after receiving the notification output from the judging unit to the index acquiring unit.

The data request unit is used to request data from the index offset and output to the atomic structure analysis module.

As an optional embodiment, the data parsing module 64 includes:

an index acquiring unit, configured to parse the index structure to obtain an index;

The data parsing unit is configured to request data, and parse the requested data according to the index.

In this embodiment, as an optional embodiment, the data parsing unit includes: a playback positioning time point acquisition subunit and a parsing playback subunit, wherein,

The playback positioning time point acquisition subunit is used to obtain the playback positioning time point;

In this embodiment, the user performs a positioning operation on the player progress bar of the client, and can calculate a positioning timestamp proportionally according to the total playback time, the total length of the progress bar, and the position of the current positioning point, and the positioning timestamp is the playback positioning time point.

The parsing and playing subunit is used for requesting the data of the positioning time point from the playing and parsing the playing.

In this embodiment, as an optional embodiment, the parsing and playing subunit includes: an accumulator, a frame number acquirer, a query device, a data requester, and a parsing player, wherein,

The accumulator is used to accumulate the frame timestamps in the video track timestamp-frame correspondence table structure box in the client;

A frame number acquirer, configured to acquire, from the accumulated frame time stamps, the frame number corresponding to the frame time stamp closest to the playback positioning time point;

In this embodiment, as an optional embodiment, the video frame offset is an I-frame video frame offset.

In this embodiment, in order to synchronize the audio and video, the audio also needs to find the corresponding data, directly search the audio STCO Box, obtain the first offset greater than the video frame offset, and obtain the audio offset.

A query device, configured to use the block offset information table box of the video track to query the video frame offset corresponding to the frame number;

A data requester, configured to request data from a server at an offset from the video frame;

The parsing player is used to parse the requested data and play it.

In this embodiment, as another optional embodiment, the parsing and playing subunit includes: a frame number calculator, a video frame shifter, a data requester, and a parsing player, wherein,

A frame number calculator, configured to calculate the frame number corresponding to the playback positioning time point according to the frame duration and frame naming rules;

A video frame offset device, configured to use the block offset information table box of the video track to query the video frame offset corresponding to the frame number;

A data requester, configured to request data from a server at an offset from the video frame;

The parsing player is used to parse the requested data and play it.

In this embodiment, as an optional embodiment, parsing and playing includes:

The analysis layer parses the requested data, if it is the first frame, and the first frame is a B frame or a P frame, discarding the B frame or P frame, if the first frame is an I frame, transparently transmits it to the playback layer for playback;

If it is not the first frame, transparently transmit the analyzed B frame or P frame or I frame to the playback layer for playback.

In the present embodiment, if the above-mentioned frame number is an I frame frame number, then parse the requested data, the first frame is an I frame, which can be transparently transmitted to the playback layer; if the frame number is not an I frame frame number, then parse the requested data For data, the first frame can be an I frame, a B frame, or a P frame. Since in the first frame, the B frame or the P frame cannot independently present a complete picture, which will cause a blurred screen phenomenon, only the media data starting with the I frame is valid data. In this embodiment, by adding the latest I-frame positioning mechanism in the positioning process, that is, positioning the frame number of the I-frame; Playback delay of GOP duration.

The device in this embodiment can be used to execute the technical solutions of the method embodiments shown in FIG. 1 to FIG. 5 , and its implementation principles and technical effects are similar, and will not be repeated here.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. any such actual relationship or order exists between them. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

Each embodiment in this specification is described in a related manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments.

In particular, as for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for relevant parts, please refer to part of the description of the method embodiment.

The logic and/or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium, For use with instruction execution systems, devices, or devices (such as computer-based systems, systems including processors, or other systems that can fetch instructions from instruction execution systems, devices, or devices and execute instructions), or in conjunction with these instruction execution systems, devices or equipment for use. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use in or in conjunction with an instruction execution system, device or device. More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connection with one or more wires (electronic device), portable computer disk case (magnetic device), random access memory (RAM), Read Only Memory (ROM), Erasable and Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program can be printed, since the program can be read, for example, by optically scanning the paper or other medium, followed by editing, interpretation or other suitable processing if necessary. processing to obtain the program electronically and store it in computer memory.

It should be understood that various parts of the present invention can be realized by hardware, software, firmware or their combination.

In the embodiments described above, various steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques known in the art: Discrete logic circuits, ASICs with suitable combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

An embodiment of the present invention further provides an electronic device, the electronic device including the device described in any one of the foregoing embodiments.

FIG. 7 is a schematic structural diagram of an embodiment of the electronic device of the present invention, which can realize the flow of the embodiment shown in FIGS. 1-6 of the present invention. As shown in FIG. 73. A circuit board 74 and a power supply circuit 75, wherein the circuit board 74 is placed inside the space enclosed by the casing 71, and the processor 72 and the memory 73 are arranged on the circuit board 74; the power supply circuit 75 is used for the above-mentioned electronic equipment Each circuit or device is powered; the memory 73 is used to store the executable program code; the processor 72 runs the program corresponding to the executable program code by reading the executable program code stored in the memory 73, and is used to execute any of the foregoing embodiments The post-encapsulation streaming data parsing method described above.

For the specific execution process of the above steps by the processor 72 and the further steps executed by the processor 72 by running executable program codes, refer to the description of the embodiment shown in FIGS. 1-6 of the present invention, and details are not repeated here.

This electronic device exists in many forms, including but not limited to:

(1) Mobile communication equipment: This type of equipment is characterized by mobile communication functions, and its main goal is to provide voice and data communication. Such terminals include: smart phones (such as iPhone), multimedia phones, feature phones, and low-end phones.

(2) Ultra-mobile personal computer equipment: This type of equipment belongs to the category of personal computers, with computing and processing functions, and generally also has the characteristics of mobile Internet access. Such terminals include: PDA, MID and UMPC equipment, such as iPad.

(3) Portable entertainment equipment: This type of equipment can display and play multimedia content. Such devices include: audio and video players (such as iPod), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

(4) Server: A device that provides computing services. The composition of a server includes processors, hard disks, memory, and system buses. , Reliability, security, scalability, manageability and other aspects have high requirements.

(5) Other electronic devices with data interaction functions.

Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. During execution, one or a combination of the steps of the method embodiments is included.

For the convenience of description, the above devices are described by dividing their functions into various units/modules and describing them separately. Of course, when implementing the present invention, the functions of each unit/module can be implemented in one or more pieces of software and/or hardware.

It can be seen from the above description of the implementation manners that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products can be stored in storage media, such as ROM/RAM, disk , CD, etc., including several instructions to make a computer device (which may be a personal computer, server, or network device, etc.) execute the methods described in various embodiments or some parts of the embodiments of the present invention.

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention. All should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (21)

1. A post-encapsulation streaming data parsing method is characterized by comprising the following steps:

determining the data in the received file as streaming data adopting post encapsulation;

Analyzing the information of the first threshold byte of the post-packaged streaming data to obtain atomic structure information;

extracting information of a second threshold byte in the atomic structure information to obtain type structure information, wherein the first threshold is twice of the second threshold;

if the type structure information identifier is not an index structure, calculating index offset according to the information of the first second threshold byte in the atomic structure information; requesting data from the index offset and performing the step of parsing a first threshold byte of information of the post-encapsulated streaming data;

after determining that the type structure information identifier is an index structure, analyzing the requested data according to the index structure;

Wherein the determining that the data in the received file is streaming data adopting post encapsulation comprises:

and analyzing the file type field of the received file, and if the file type field is identified as MP 4-post encapsulation and is applied to a streaming data scene, the data in the received file is streaming data adopting the post encapsulation.

2. The post-packaged streaming data parsing method of claim 1, wherein the first threshold is 8 and the second threshold is 4.

3. the post-encapsulation streaming data parsing method according to claim 1, wherein after determining that the type structure information is identified as an index structure, parsing the requested data according to the index structure comprises:

If the type structure information identifier is an index structure, analyzing the index structure to obtain an index;

And requesting data, and analyzing the requested data according to the index.

4. the post-packaged streaming data parsing method according to claim 3, wherein the calculating an index offset according to the information of the first second threshold byte in the atomic structure information comprises:

Acquiring a stored reserved index offset;

adding the length corresponding to the information of the first second threshold byte and the stored reserved index offset to obtain index offset;

updating the stored reserved index offset with the index offset.

5. the post-packaged streaming data parsing method of claim 4, wherein if the type structure information is identified as an index structure, the method further comprises:

clearing the stored reservation index offset.

6. the post-encapsulation streaming data parsing method of claim 3, wherein the requesting data, parsing the requested data according to the index comprises:

acquiring a play positioning time point;

Requesting to locate the data of the time point from the playing and analyzing the playing.

7. The post-packaged streaming data parsing method according to claim 6, wherein the requesting data for locating a time point from the playing comprises:

Accumulating the video track time stamp in the client and the frame time stamp in the frame corresponding table structure box;

acquiring a frame number corresponding to the frame timestamp closest to the playing positioning time point from the accumulated frame timestamps;

Inquiring video frame offset corresponding to the frame number by using a block offset information table box of the video track;

Requesting data from a server at an offset from the video frame.

8. The post-packaged streaming data parsing method of claim 7, wherein the video frame offset is an I-frame video frame offset.

9. the post-packaged streaming data parsing method according to claim 6, wherein the requesting data for locating a time point from the playing comprises:

calculating the frame number corresponding to the playing positioning time point according to the frame duration and the frame naming rule;

Inquiring video frame offset corresponding to the frame number by using a block offset information table box of the video track;

requesting data from a server at an offset from the video frame.

10. The post-packaged streaming data parsing method according to claim 6, wherein the parsing and playing comprises:

Analyzing the requested data by the analysis layer, if the requested data is a first frame and the first frame is a B frame or a P frame, discarding the B frame or the P frame, and if the first frame is an I frame, transmitting the first frame to the play layer for playing;

And if the frame is not the first frame, transmitting the B frame or the P frame or the I frame obtained by analysis to a playing layer for playing.

11. A post-packaged streaming data parser, comprising: a post-encapsulation determining module, an atomic structure analyzing module, a type structure extracting module, a judging unit, an index offset calculating unit, a data requesting unit and a data analyzing module, wherein,

The post encapsulation determining module is used for determining that the data in the received file is streaming data adopting post encapsulation;

The atomic structure analysis module is used for analyzing the information of the first threshold byte in front of the post-packaged streaming data to obtain atomic structure information;

the type structure extraction module is used for extracting information of a second threshold byte in the atomic structure information to obtain type structure information, wherein the first threshold is twice of the second threshold;

The judging unit is used for judging whether the type structure information identifier is an index structure, if so, the data analysis module is informed, and if not, the index offset calculation unit is informed;

the index offset calculation unit is used for calculating the index offset according to the information of the first second threshold byte in the atomic structure information;

The data request unit is used for requesting data from the index offset and outputting the data to the atomic structure analysis module;

the data analysis module is used for analyzing the requested data according to the index structure after determining that the type structure information identifier is the index structure;

wherein the determining that the data in the received file is streaming data adopting post encapsulation comprises:

and analyzing the file type field of the received file, and if the file type field is identified as MP 4-post encapsulation and is applied to a streaming data scene, the data in the received file is streaming data adopting the post encapsulation.

12. The post-packaged streaming data parser of claim 11, wherein the first threshold is 8 and the second threshold is 4.

13. the post-packaged streaming data parser of claim 11, wherein the data parser module comprises: an index acquisition unit and a data parsing unit, wherein,

the index acquisition unit is used for analyzing the index structure to obtain an index;

And the data analysis unit is used for requesting data and analyzing the requested data according to the index.

14. the post-packaged streaming data parser of claim 11, wherein the index offset calculator comprises: a reserved index offset acquisition subunit, an index offset calculation subunit, and an update subunit, wherein,

a reserved index offset obtaining subunit, configured to obtain a stored reserved index offset;

The index offset calculation subunit is configured to add the length corresponding to the information of the first second threshold byte to the stored reserved index offset to obtain an index offset;

an updating subunit, configured to update the stored reserved index offset with the index offset.

15. the post-packaged streaming data parser of claim 14, wherein the index offset calculator further comprises:

and the emptying subunit is used for emptying the stored reserved index offset after receiving the notification output to the index acquisition unit by the judgment unit.

16. The post-packaged streaming data parser of claim 13, wherein the data parser unit comprises: a playing positioning time point obtaining subunit and an analyzing playing subunit, wherein,

a play positioning time point obtaining subunit, configured to obtain a play positioning time point;

and the analysis playing subunit is used for requesting the data of the playing positioning time point and analyzing and playing the data.

17. The post-packaged streaming data parser of claim 16, wherein the parsing play sub-unit comprises: accumulator, frame number obtainer, querier, data requester and parsing player, wherein,

the accumulator is used for accumulating the video track timestamp in the client-frame timestamp in the frame corresponding table structure box;

A frame number acquirer for acquiring a frame number corresponding to a frame timestamp closest to the play positioning time point from the accumulated frame timestamps;

the querier is used for querying the video frame offset corresponding to the frame number by utilizing a block offset information table box of the video track;

a data requestor to request data offset from the video frame from a server;

and the analysis player is used for analyzing the requested data and then playing the data.

18. the post-packaged streaming data parser of claim 17, wherein the video frame offset is an I-frame video frame offset.

19. the post-packaged streaming data parser of claim 16, wherein the parsing play sub-unit comprises: a frame number calculator, a video frame shifter, a data requestor, and a parsing player, wherein,

The frame number calculator is used for calculating the frame number corresponding to the playing positioning time point according to the frame duration and the frame naming rule;

The video frame shifter is used for inquiring the video frame shift corresponding to the frame number by utilizing a block shift information table box of the video track;

a data requestor to request data offset from the video frame from a server;

And the analysis player is used for analyzing the requested data and then playing the data.

20. the post-packaged streaming data parser of claim 16, wherein said parsing comprises:

Analyzing the requested data by the analysis layer, if the requested data is a first frame and the first frame is a B frame or a P frame, discarding the B frame or the P frame, and if the first frame is an I frame, transmitting the first frame to the play layer for playing;

And if the frame is not the first frame, transmitting the B frame or the P frame or the I frame obtained by analysis to a playing layer for playing.

21. An electronic device, characterized in that the electronic device comprises: the device comprises a shell, a processor, a memory, a circuit board and a power circuit, wherein the circuit board is arranged in a space enclosed by the shell, and the processor and the memory are arranged on the circuit board; a power supply circuit for supplying power to each circuit or device of the electronic apparatus; the memory is used for storing executable program codes; the processor executes a program corresponding to the executable program code by reading the executable program code stored in the memory, and is used for executing the post-packaged streaming data parsing method of any one of claims 1 to 10.