RU2366103C2 - Storage of advanced video coding (avc) set of parametres in avc file format - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: invention relates to video coding, and particularly, to the advanced video coding (AVC) format for storage of AVC set of parametres in AVC file format. The said result is achieved due to that, use of information of a set of parametres, of the corresponding AVC file involves stages on which: parametre information is embedded, so as to provide for streaming video data, into a file also storing video data such that, parametre information is located in a part of the file separate from video data, so as to allow streaming parametre information independently from video data.

EFFECT: possibility of streaming multimedia from a client to a server, using real time streaming protocol (RTSP) of sets of AVC parametres independently from multimedia using the method.

22 cl, 8 dwg, 1 tbl

Description

FIELD OF THE INVENTION

The present invention, in General, relates to a file format enhanced video coding (AVC), and more specifically, to storing sets of AVC parameters in the file format AVC.

State of the art

Enhanced Video Coding (AVC) file format provides support for streaming multimedia data over a network as well as local playback. The most common streaming servers use the Real-time Streaming Protocol (RTSP) together with the Session Description Protocol (SDP) as the control protocol for setting up and terminating multimedia streaming sessions between clients and servers. Additionally, the real-time streaming protocol (RTP) is used as the data transfer protocol for multimedia information between servers and clients.

The AVC codec provides a means to separate specific information from a video stream that is relevant to more than one video frame. It is understood that this special information must be sent via a reliable transport protocol (such as, for example, transmission control protocol, TCP) in addition to sending via an unreliable transport protocol (such as, for example, user datagram protocol, UDP) embedded in a multimedia stream. This special information, combined together, forms a set of parameters. The set of AVC parameters may include, for example, such information: frame or image size; display window size; macroblock distribution table, etc.

One or many sets of parameters may exist for any given AVC video stream. The encoder and decoder must keep the same list of parameter sets. Each slot header in the AVC video stream includes a code that indicates a set of parameters to be used in decoding. Based on this code, the decoder can then determine which set of parameters to use for proper decoding. Since the parameter sets are not related to the main video stream, it is possible and desirable to send them from the multimedia streaming server to the multimedia streaming client in an out-of-band way.

The following is a description of a typical media streaming transaction scenario. When a media streaming client requests media streaming from a server, an initial negotiation typically takes place before streaming. The initial negotiation sequence entails a client sending a request to the server to obtain information about a particular stream. The server then sends a response to the client with the minimum required details required to receive and decode the stream. This initial negotiation is typically accomplished using a network protocol, such as real-time streaming protocol (RTSP). RTSP can be used to send parameter sets from a multimedia streaming server to a multimedia streaming client. However, due to the nature of the streaming server, in order for the streaming server to send media-dependent data to the client, the set of parameters must be stored in a file so that the streaming server does not know or worry about what type of media file it is.

Therefore, it will be necessary and very advantageous to have a method for enabling the RTSP protocol of a streaming media server to transmit parameter sets in a manner independent of multimedia to the client.

SUMMARY OF THE INVENTION

The above problems, as well as other related problems of the prior art, are solved by the present invention by storing AVC parameter sets in the AVC file format. The present invention provides an extension to the AVC file format for storing advanced metadata to improve streaming services. The extension entails inserting the AVC parameter sets into the metadata of the Hint Track records of the AVC file format while maintaining the ability to provide backward compatibility with existing streaming servers.

According to an aspect of the present invention, there is provided a method of using parameter set information corresponding to an advanced video coding (AVC) file. The method includes the step of embedding the parameter set information in the recording of the directions of the AVC file.

According to another aspect of the present invention, there is provided an apparatus for using parameter set information corresponding to an advanced video coding (AVC) file. The device includes an embedder for embedding parameter set information in a recording of directions of an AVC file.

According to another aspect of the present invention, there is provided a method of using parameter set information corresponding to an advanced video coding (AVC) file. The method includes the steps of embedding the information of the parameter set in the recording of the directions of the AVC file and transmitting the information of the parameter sets during out-of-band and multimedia-independent transmission.

These and other aspects, features and advantages of the present invention should become more apparent from the following detailed description of preferred embodiments that should be considered in connection with the accompanying drawings.

Brief Description of the Drawings

FIG. 1 is a diagram illustrating an advanced video coding (AVC) file 100 to which the present invention can be applied, according to an illustrative embodiment of the present invention;

FIG. 2 is a diagram illustrating an advanced video coding (AVC) video encoder 200 to which the present invention can be applied, according to an illustrative embodiment of the present invention;

FIG. 3 is a diagram illustrating exemplary metadata 300 output from the AVC encoder 200 of FIG. 2, according to an illustrative embodiment of the present invention;

FIG. 4 is a diagram illustrating session description protocol (SDP) recording description information 400 according to an illustrative embodiment of the present invention;

FIG. 5 is a high-level block diagram illustrating an apparatus 500 for inserting parameter sets in an instruction record of an AVC file 599 according to an illustrative embodiment of the present invention;

FIG. 6 is a high level block diagram illustrating the generation of metadata tables and writing metadata tables to an AVC file according to an illustrative embodiment of the present invention;

FIG. 7 is a high level block diagram illustrating the recording of multimedia data in an AVC file according to an illustrative embodiment of the present invention;

FIG. 8 is a high level block diagram illustrating the recording of an instruction record in an AVC file according to an illustrative embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention is directed to an extension for the AVC file format, a well-known digital streaming media storage format, for the purpose of storing extended metadata to improve streaming services. The extension entails inserting AVC parameter sets into the Hint Track metadata of the AVC file format while maintaining the ability to provide backward compatibility with existing streaming servers. The parameter sets are embedded in the recording of the AVC file format instructions so that the RTSP protocol of the multimedia streaming server has the ability to transmit the parameter sets in a manner independent of multimedia to the client.

It should be appreciated that the present invention generally relates to an advanced video coding (AVC) file format (International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) 14496, part 15), jointly developed by the International Telecommunication Union (ITU) and Motion Picture Expert Group (MPEG). Nevertheless, in view of the techniques of the present invention proposed herein, those skilled in the art should consider this and other file formats to which the present invention may be applied without departing from the spirit and scope of the present invention.

It should be understood that the present invention can be implemented in various forms of hardware, software, firmware, special purpose processors, or a combination thereof. Preferably, the present invention is implemented as a combination of hardware and software. Moreover, the software is preferably implemented as an application program materially implemented on a program storage device. An application can be loaded into and executed by a machine containing any appropriate architecture. Preferably, the machine is implemented on a computing platform having hardware such as one or more central processing units (CPUs), random access memory (RAM), and input-output interfaces. The computing platform also includes the operating system and micro-command code. The various processes and functions described in this document can be either part of the micro-command code, or part of an application program (or a combination of the above) that are executed by the operating system. In addition, various other peripheral devices can be connected to a computing platform, such as an additional data storage device and a printing device.

Additionally, it should be understood that since some of the constituent system components and process steps depicted in the accompanying drawings are preferably implemented in software, the actual relationships between system components (or process steps) may differ depending on the method by which the present invention is programmed. Based on the techniques presented herein, those skilled in the art are able to consider these and similar implementations or configurations of the present invention.

The AVC file format provides support for streaming multimedia data over a network as well as local playback. An AVC file that supports multimedia streaming includes additional information about data items in the stream. Additional information is included in additional file entries known as Hint Track entries. Recording instructions provides streaming instructions to the server, such as, for example, packet formatting information to ensure proper delivery and synchronization information to enable proper transmission synchronization. In addition, the guidance record may also provide further information, such as details about the type of media that the client requests, which can then be sent to the client during the initial interaction between the client and the server.

The most common streaming servers use Real-time Streaming Protocol (RTSP) together with Session Description Protocol (SDP) as the control protocol for setting up and breaking streaming sessions between clients and servers. Additionally, the real-time streaming protocol (RTP) is used as the data transfer protocol for multimedia information between servers and clients. It will be appreciated that the present invention is directed to streaming servers that use RTSP as the control protocol for establishing a session. Of course, those of ordinary skill in the art should certainly appreciate that the present invention can also be modified and implemented with respect to streaming servers that use a protocol other than RTSP as a control protocol, without departing from the spirit and scope of the present invention. .

RTSP is a text-based protocol in which messages include a header and information execution. The title is usually specified by the standard, while parts of the information execution may depend on the applications. In the case of an RTSP-based streaming media server, the server by default inserts the Session Description Protocol (SDP) information included in the AVC file into the RTSP message during the initial negotiation of the media streaming with the client. Therefore, by embedding the AVC parameter sets in the information, the SDP server is able to transmit the multimedia stream parameter sets to the client without knowledge of the multimedia format. As a result, parameter sets are transmitted before any multimedia, and in addition they are transmitted using a reliable transport protocol, because RTSP uses the Transmission Control Protocol (TCP) by default.

FIG. 1 is a diagram illustrating an advanced video coding (AVC) file 100 to which the present invention can be applied, according to an illustrative embodiment of the present invention. The AVC file 100 includes AVC video information 160, audio recording information 165, video direction recording information 170, and audio direction recording information 175. The AVC file 100 further includes audio recordings 110, video recordings 120, and direction recordings 130. Guidance records 130 may be used for streaming purposes according to the present invention.

Records 130 directions in the AVC file 100 include special instructions for the streaming server to help in the formation of packets. This information is provided in such a way that the server is able to package multimedia data in a manner suitable for streaming using a specific network transport. In addition to including specific streaming instructions, other data can be embedded in the recording of instructions that the server should immediately send to the client (for example, in accordance with the present invention, other data may include parameter sets).

The following describes the process of saving the AVC parameter sets to recording the directions of the AVC file. FIG. 2 is a diagram illustrating an advanced video coding (AVC) video encoder 200 to which the present invention can be applied, according to an illustrative embodiment of the present invention. First, the AVC video encoder 200 must have at least one output interface for delivering compressed video data and corresponding metadata to the second module for storage purposes. In addition, in FIG. 2, a transmitter 250 is included for transmitting parameter sets during out-of-band transmission. Sets of parameters are transmitted embedded in the content of the SDP AVC file.

FIG. 3 is a diagram illustrating exemplary metadata 300 output from the AVC encoder 200 of FIG. 2, according to an illustrative embodiment of the present invention. The metadata 300 that is delivered from the AVC video encoder 200 may include information such as size 210 of the compressed video frame, type 215 of the video frame (e.g., I-frame, P-frame, and B-frame), synchronization information 220 (to for example, frame rate), other attributes 225 and new sets of 230 parameters. Of particular interest in this application are parameter sets 230, since parameter sets 230 are stored in the AVC file as part of the video stream, as well as in the guidance record that refers to the video stream. Thus, for sets of 230 parameters that must be stored in the guidance records, the data received via the output interface of the AVC video encoder must be properly managed.

When the parameter sets 230 are part of the metadata 300 output from the AVC video encoder, they are copied and stored in the repository for later reference when an instruction record is created. This makes it possible to create a recording of instructions to be simpler and more optimized, since the multimedia recording does not have to be analyzed from beginning to end in order to find sets of parameters.

FIG. 4 is a diagram illustrating Session Description Protocol (SDP) recording description information 400 according to an illustrative embodiment of the present invention. The AVC parameter sets 230 are stored in the SDP record description information 400 included in the direction record. Saving sets of 230 parameters in the recording of instructions provides a means for transporting sets of 230 parameters using reliable network transport during the initialization of the streaming session. Equally important, backward compatibility with current streaming servers is provided, since no additional file extension is required. SDP recording description information 400 further includes recording duration information 410, recording description information 420 and other attributes 430. Table 1 illustrates the storage of AVC parameter sets in session description protocol (SDP) recording description information according to an illustrative embodiment of the present invention.

Table 1 v = 0 s = AVC Streaming Media Program u = http: // e = admin @ a = control: * a = range: npt = 0 - 60.96 m = video 0 RTP / AVP 96 a = rtpmap: 96 AVC / 90,000 a = paramset: "data: video / jvt; base64, MIME encoded Parameter Sets" a = control: trackID = 2 m = audio 0 RTP / AVP 14 a = rtpmap: 14 MPA / 22050 a = control: trackID = 6

Parameter set information can be encoded using the Internet Multipurpose Email Extensions (MIME) before embedding instruction records in the SDP content.

FIG. 5 is a high-level block diagram illustrating an apparatus 500 for inserting parameter sets in an instruction record of an AVC file 599 according to an illustrative embodiment of the present invention. The device 500 includes a media and metadata manager 510, a metadata analyzer 520, an insert module 530, a multimedia recording module 540, a metadata recording module 550, an instruction recording module 560, and base file recording module 570.

Video data 501 and metadata 502 are received by the media and metadata manager 510. The media and metadata manager 510 ensures that the file writing process is performed properly and that the file is not written in an inconsistent manner. In most cases, metadata 502 should be located first in the file, followed by media records and guidance records. Therefore, the media and metadata manager 510 is provided to control the recording process.

There are three main recording modules (540, 550, 560) that are coupled to the basic file recording module 570. It may seem that each of these modules sends data to be written to the base file at the same time, but in fact they send the data to be written in a specific order (for example, the order described above, i.e. metadata, multimedia data and data directions). Next, these recording modules are described further with reference to FIG. 6-8.

FIG. 6 is a high level block diagram illustrating the generation of metadata tables and writing metadata tables to an AVC file according to an illustrative embodiment of the present invention. FIG. 7 is a high level block diagram illustrating the recording of multimedia data in an AVC file according to an illustrative embodiment of the present invention. FIG. 8 is a high level block diagram illustrating the recording of an instruction record in an AVC file according to an illustrative embodiment of the present invention. As indicated above, writing the AVC instruction record to the AVC file entails extracting the parameter sets from the metadata and storing them in the SDP instruction record information.

Referring to FIG. 6 and 8, metadata is entered into the media and metadata manager 510. As you know, metadata is a set of data structures for storing multimedia-dependent information, such as, but not limited to, timestamps, frame sizes, parameter sets, etc. Metadata is used, for example, in order to maintain synchronization of the respective multimedia with additional data for purposes, for example, decoding, encoding, display, access and transmission.

The media and metadata manager 510 extracts the metadata from the multimedia and analyzes the metadata for the sets of parameters (see FIG. 8). Thus, if the parameter sets are present in the metadata, the metadata analyzer 520 retrieves the parameter sets and transfers them to the SDP data structure of the guidance record. Otherwise, if a parameter set is not found, the metadata analyzer 520 converts the metadata into standardized data structures, which provides a means for writing information to a file. The metadata recording unit 550 uses the application programming interface (API) of the base recording unit 570 to write metadata to the AVC file 599.

In FIG. 7, the media and metadata manager 510 retrieves the media and the total media size. Then, the data is transmitted directly to the multimedia recording module 540. The media writer also uses the API of the base writer 570 to write media samples to the AVC file 599.

Although illustrative embodiments have been described with reference to the accompanying drawings, it should be understood that the present invention is not limited solely to these embodiments and that various changes and modifications can be made by those skilled in the art without departing from the spirit and scope of the invention. It is intended to include all of these changes and modifications within the scope of the invention as defined by the appended claims.

Claims (22)

1. A method for providing streaming video information, comprising the step of: embedding parameter information for streaming video information into a file also storing video information such that the parameter information is located in a part of the file separately from the video information to enable streaming of parameter information regardless, but along with video information. 2. The method according to claim 1, in which the step of embedding the parameter also comprises the step of embedding information about the parameters in the recording instructions of the mentioned file. 3. The method according to claim 2, in which the step of embedding the parameter also comprises the step of embedding the parameter information in the content of the session description protocol '(SDP) recording instructions. 4. The method according to claim 3, in which the step of embedding the parameter also comprises encoding the parameter information by means of multipurpose Internet e-mail extensions (MIME) before embedding instructions into the SDP payload. 5. The method of claim 1, further comprising transmitting parameter information during out-of-band transmission. 6. The method according to claim 5, in which the transmission step also comprises transmitting information of the parameter set with execution of the transmission control protocol (TCP). 7. The method according to claim 5, in which the transmission step also comprises transmitting information of the parameter set with real-time streaming protocol execution. 8. The method according to claim 1, comprising the step of transmitting parameter information during transmission independent of multimedia. 9. The method according to claim 1, in which the transmitting step comprises the step of transmitting parameter information before transmitting any multimedia corresponding to it. 10. The method according to claim 1, comprising the step of retrieving parameter information from metadata corresponding to at least one media stream. 11. A device for streaming video information, comprising: an embedding device for embedding parameter information in a file, also storing video information so that the parameter information is located in a part of the file separately from the video information to enable streaming of parameter information regardless, but along with video information. 12. The device according to claim 11, in which the embedment device embeds the information of a set of parameters in a record of file directions. 13. The device according to item 12, in which the embedder embeds the information of a set of parameters in the content of the session description protocol (SDP) recording instructions. 14. The device of claim 12, wherein the parameter information embedded in the SDP payload through the embedder is encoded using Internet Multipurpose Email Extensions (MIME). 15. The device according to claim 11, which also contains a transmitting device for transmitting parameter information during out-of-band transmission. 16. The device according to clause 15, in which the transmitting device transmits information of a set of parameters with the execution of the transmission control protocol. 17. The device according to clause 15, in which the transmitting device transmits information of a set of parameters with the execution of the streaming protocol in real time. 18. The device according to claim 11, which also contains a transmitting device for transmitting information about the parameters during transmission, independent of multimedia. 19. The device according to claim 11, in which the transmitting device transmits information about the parameters before transmitting any multimedia corresponding to it. 20. The device according to claim 11, which also includes an extraction device for extracting the parameter from the metadata corresponding to at least one multimedia stream. 21. A method for providing streaming video information in a file, along with information about parameters located in a part of the file separately from video information, by processing a file containing video information and information about the parameters, the method comprising the step of: separating the parameter information in the aforementioned parts of a file separate from video information to enable processing of video information during streaming. 22. A device for streaming video information, comprising: a separation device for separating parameter information from a file that also stores video information so that the parameter information is located in a part of the file separately from the video information, for providing video information processing during streaming along with parameter information .

Images