RU2378790C1 - Scalability techniques based on content information - Google Patents

Abstract

FIELD: physics; video technology.

SUBSTANCE: invention relates to devices for re-encoding video data for real time streaming, and particularly to re-encoding video data for real time streaming in a mobile broadcast application. Proposed is a device for using content information to encode multimedia data, which includes a content classification module, which is configured to classify content multimedia data and provide content classification data, and an encoder which is configured to encode multimedia data in a first data group and a second data group based on content classification, wherein the first data group contains a coefficient, and the second group of data contains a first differential refinement associated with the coefficient of the first group of data.

EFFECT: design of a transcoder which provides for highly efficient processing and compression of multimedia data, which uses information defined from the said multimedia, and is scalable and error-tolerant for use in several multimedia data applications.

49 cl, 45 dwg

Description

Advantage Statement 35 U.S.C. §119

This Patent Application claims priority (a) of Provisional Patent Application No. 60/721416, entitled “A VIDEO TRANSCODER FOR REAL-TIME STREAMING AND MOBILE BROADCAST APPLICATIONS”, registered on September 27, 2005, (b) of Provisional Patent Application No. 60/789377, entitled "A VIDEO TRANSCODER FOR REAL-TIME STREAMING AND MOBILE BROADCAST APPLICATIONS", registered April 4, 2006, (c) Provisional application number 60/727643, entitled "METHOD AND APPARATUS FOR SPATIO-TEMPORAL DEINTERLACING AIDEDAS FOREDELED FOREDED MOTION FILTER VIDEO ", registered on October 17, 2005, (d) of Provisional Application No. 60/727644, entitled" M ETHOD AND APPARATUS FOR SHOT DETECTION IN VIDEO STREAMING ", registered on October 17, 2005, (e) Provisional application number 60/727640, entitled" A METHOD AND APPARATUS FOR USING AN ADAPTIVE GOP STRUCTURE IN VIDEO STREAMING ", registered on October 17, 2005, (f) Provisional application number 60/730145, entitled "INVERSE TELECINE ALGORITHM BASED ON STATE MACHINE", registered on October 24, 2005, and (g) Provisional application number 60/789048, entitled "SPATIO-TEMPORAL DEINTERLACING AIDED BY MOTION COMPENSATION FOR FIELD -BASED MULTIMEDIA DATA ", registered April 3, 2006. All seven of these provisional patent applications are assigned to its assignee and are hereby incorporated by reference in their entirety.

Link to pending patent applications

This patent application is related to (U.S. Patent Application No. 11/373577, entitled "CONTENT CLASSIFICATION FOR MULTIMEDIA PROCESSING", registered March 10, 2006, assigned to the assignee of this application and thus expressly incorporated herein by reference.

State of the art

FIELD OF THE INVENTION

The present application is directed to devices and methods for transcoding video data for streaming in real time, and more specifically to transcoding video data for streaming in real time in a mobile broadcast application.

State of the art

Efficient video compression is used in many multimedia applications, such as video streaming and video telephony, due to limited bandwidth resources and the variability of available bandwidth. Specific video encoding standards, such as MPEG-4 (ISO / IEC), H.264 (ITU), or similar video encoding, provide highly efficient encoding that is best suited for applications such as wireless broadcast. Some multimedia data, such as digital television presentations, are typically encoded according to other standards, such as MPEG-2. Accordingly, transcoders are used to transcode or convert multimedia data encoded according to one standard (for example, MPEG-2) to another standard (for example, H.264) before wireless broadcasting.

Enhanced speed-optimized codecs can provide benefits in error tolerance, error recovery, and scalability. Moreover, the application of information derived from the multimedia data itself may also provide additional coding enhancements, including error tolerance, error recovery, and scalability. Accordingly, there is a need for a transcoder that provides highly efficient processing and compression of multimedia data, which uses information determined from the multimedia data itself, and is scalable and error resistant for use in a variety of multimedia data applications, including mobile broadcast streaming multimedia information .

SUMMARY OF THE INVENTION

Each of the inventive devices and methods based on the content of the transcoding described and illustrated has several aspects, none of which is solely responsible for the required attributes. Without limiting the scope of this invention, its most prominent features will be briefly discussed below. After considering this discussion, and more precisely, after reading the section entitled "Detailed Description of the Invention", it will become clear how the features of this content-driven transcoding provide improvements to devices and methods for processing multimedia data.

The inventive aspects described herein relate to the use of content information for various methods of encoding multimedia data and in various modules and components of an encoder, for example, an encoder used in a transcoder. The transcoder can control the transcoding of multimedia data using content information. Content information may be received from another source, for example, metadata that is received along with the video. The transcoder may be configured to generate content information using a variety of different processing operations. In some aspects, the transcoder generates a classification of the contents of the multimedia data, which is then used in one or more encoding processes. In some aspects, the content-driven transcoder may determine spatial and temporal information content of the multimedia data and use the content information for encoding with the same quality based on the content over the channels and based on the classification of bit compression / assignment.

In some aspects, content information (eg, metadata, content metrics, and / or content classification) of the multimedia data is obtained and computed, and then provided to transcoder components for use in encoding the multimedia data. For example, a preprocessor can use certain content information to detect scene changes, convert from video format to movie (IVTC), eliminate interlacing, compensate for motion and suppress noise (for example, two-dimensional wavelet transform) and spatio-temporal noise attenuation, for example, removing image defects, suppressing reverb, unlocking and / or reducing noise. In some aspects, the preprocessor can also use content information to downsample spatial resolution, for example, to determine the corresponding “protected” areas and “moving object control” zones in downsampling from standard resolution (SD) to a quarter of the video graphics logic matrix (QVGA).

In some aspects, the encoder includes a content classification module that is configured to calculate content information. The encoder can use content classification to control the bit rate (for example, bit assignment) when determining the quantization parameters for each MB, for motion estimation, for example, performing color motion estimation (ME), performing motion vector prediction (MV), scalability when providing a baseline and improvement level, as well as error tolerance through the use of content classification so as to influence prediction hierarchy and error tolerance patterns, including for example, processes of adaptive internal updating, edge alignment and provision of redundant I-frame data at the improvement level. In some aspects, the transcoder uses content classification in conjunction with a data multiplexer to maintain optimal quality of multimedia data in the channels. In some aspects, the encoder may use content classification information to force I frames to periodically appear in the encoded data to allow fast channel switching. These implementations may also use I-blocks, which may be required in the encoded data for error tolerance, so that random access switching and error tolerance (based on, for example, content classification) can be effectively combined through a prediction hierarchy to increase coding efficiency with increasing margin of error stability.

In one aspect, a method for processing multimedia data comprises classifying the contents of the multimedia data and encoding the multimedia data in the first data group and in the second data group based on the content classification, the first data group containing a coefficient and the second data group containing a first differential refinement associated with the coefficient first group of data. Encoding may include determining a bit rate based on the classification of the contents of the multimedia data and encoding the multimedia data based on the bit rate. The content classification may include determining the complexity of the multimedia data, and wherein the selected multimedia data is encoded based on the complexity of the multimedia data. Complexity may include temporal complexity or spatial complexity or temporal complexity and spatial complexity. Encoding may include encoding multimedia data so as to enable decoding of only the first data group or the first data group and the second data group into one combined data group. The first differential refinement may indicate the difference between the selected video frame and the frame data resulting from decoding the first group of data. The first group of data may be the base level, and the second group of data may be the level of improvement. In addition, the method may include selecting a coefficient from one of the residual error coefficient of the initial base level and the residual error coefficient of the initial improvement level and calculating the first differential refinement based on the coefficient and residual error coefficient of the initial improvement level. The encoding may further comprise encoding macroblock header information and motion vector information in a first data group. The encoding may further comprise quantizing a first data group with a first step size and quantizing a second data group with a second step size, wherein the first step size and the second step size are correlated by a scaling factor. The encoding may further include determining a first quantization parameter having a first quantization step size for use in encoding the first data group, and determining a second quantization parameter having a second quantization step size for use in encoding the second data group, the first and second parameters quantizations are determined based on the content information of the selected frame data, and wherein said first quantization step size is coarser than said second dimension the quantization step measures. In another aspect, encoding includes encoding a first data group using I-frames and P-frames, or any combination thereof, and encoding a second data group using I-frames, P-frames and B-frames, or any combination thereof.

In another aspect, the multimedia data encoding device includes means for classifying the contents of the multimedia data, means for encoding the multimedia data in the first data group and in the second data group based on the content classification, the first data group containing a coefficient and the second data group containing a first differential refinement, associated with the coefficient of the first data group. The encoding means may comprise means for determining a bit rate based on the classification of the contents of the multimedia data and encoding the multimedia data based on the bit rate. The content classification means may include means for determining the complexity of the multimedia data, and wherein the selected multimedia data is encoded based on the complexity of the multimedia data, the complexity comprising temporal complexity or spatial complexity or temporal complexity and spatial complexity. The encoding means may comprise means for decoding only the first data group or the first data group and the second data group into one combined data group.

In another aspect, an apparatus includes a media content classification module configured to classify media content and provide content classification data, and an encoder configured to encode media in the first data group and in the second data group based on the content classification moreover, the first group of data contains a coefficient, and the second group of data contains a first differential refinement associated with the coefficient by the first data group. The encoder may include a bit rate component configured to determine a bit assignment based on the classification of the contents, wherein the encoding component is further configured to encode the selected multimedia data using the bit assignment.

In another aspect, a computer-readable medium comprises instructions that, when executed, instruct the machine to classify the contents of multimedia data and encode multimedia data in a first data group and a second data group based on the content classification, the first data group containing a coefficient and the second data group containing a first differential refinement associated with the coefficient of the first data group.

In another aspect, the processor is configured to classify the contents of the multimedia data and encode the multimedia data in the first data group and in the second data group based on the content classification, wherein the first data group contains a coefficient and the second data group contains a first differential refinement associated with the coefficient first group of data.

Brief Description of the Drawings

1A is a block diagram of a multimedia broadcast system including a transcoder for transcoding between different video formats.

1B is a block diagram of an encoder configured to encode multimedia data and provide an encoded first data group and an encoded second data group.

1C is a block diagram of a processor configured to encode multimedia data.

FIG. 2 is a block diagram of an example transcoder of the system of FIG. 1.

FIG. 3 is a flowchart illustrating the operation of an analyzer used in the transcoder of FIG. 2.

FIG. 4 is a flowchart illustrating the operation of the decoder used in the transcoder of FIG. 2.

FIG. 5 is a system timing diagram illustrating a flow of operations performed by the transcoder of FIG. 2.

6 is a flowchart illustrating the sequence of operations and functions of the preprocessor that can be used in the transcoder of FIG. 2.

FIG. 7 is a block diagram of an exemplary two-pass encoder that can be used in the transcoder of FIG. 2.

FIG. 8 illustrates an example of a classification diagram that illustrates one aspect of how to associate texture and motion values with content classification.

FIG. 9 is a flowchart illustrating an example content classification operation, for example, for use in the encoder of FIG. 7.

FIG. 10 is a flowchart illustrating a speed control operation, for example, for use in the encoder of FIG. 7.

11 is a flowchart illustrating the operation of an exemplary motion estimation unit, for example, for use in the encoder of FIG. 7.

12 is a flowchart illustrating the operation of an exemplary function of an encoder for determining a mode, for example, for use in the encoder of FIG. 7.

FIG. 13 is a flowchart illustrating an example scalability implementation operation for use in the encoder of FIG. 7.

Fig. 14 is a flowchart illustrating an example operation of performing a data stream with distortion depending on speed, as occurs, for example, in the encoder of Fig. 7.

15 is a graph illustrating the relationship between coding complexity, assigned bits, and human visual quality.

16 is a graph illustrating a non-linear scene detection circuit.

17A is a flowchart illustrating the processing of multimedia data that is received, received, or otherwise available.

Figv is a block diagram of a multimedia encoding system.

FIG. 18 is a diagram illustrating a process for eliminating interlaced scanning using motion estimation / compensation.

Fig. 19 is a block diagram of a multimedia communication system.

20 is a diagram illustrating the organization of a video bitstream in an enhancement layer and a base layer.

21 is a diagram illustrating the alignment of a series of macroblocks with the boundaries of video frames.

FIG. 22 is a block diagram illustrating a prediction hierarchy.

23 is a flowchart illustrating a method of encoding multimedia data based on content information.

24 is a flowchart illustrating a method of encoding multimedia data so as to combine data boundaries based on the level of content information.

25 is a graph illustrating a protected area of moving objects and a protected area of titles of a data frame.

26 is a graph illustrating a protected area of moving objects of a data frame.

27 is a flowchart illustrating a process for encoding multimedia data using adaptive internal updating based on the multimedia content information.

28 is a flowchart illustrating a process for encoding multimedia data using redundant I-frames based on multimedia content information.

29 illustrates motion compensation vectors between the current frame and the previous frame MV _P and the current frame and the next frame MV _N.

30 is a flowchart illustrating frame detection.

31 is a flowchart illustrating encoding of a base layer and an enhancement layer.

32 is a schematic diagram illustrating coding of a macroblock.

33 is a schematic diagram illustrating modules for encoding a base layer and an enhancement layer.

Fig. 34 illustrates an example of a process in a coefficient selection block of a base level and an enhancement level.

Fig illustrates another example of a process in the block selection of the coefficients of the base level and level of improvement.

Fig illustrates another example of a process in the block selection of the coefficients of the base level and level of improvement.

37 is a flowchart illustrating encoding of multimedia data based on content information.

Fig. 38 is a diagram illustrating possible system solutions in a conversion process from a video format to a film.

Fig. 39 illustrates boundaries in a macroblock that are to be filtered out by an unlocking process.

40 is a diagram illustrating a process of spatio-temporal elimination of interlacing (deinterlacing).

Fig. 41 illustrates an example of one-dimensional monophasic resampling.

42 is a flowchart illustrating an example of an adaptive GOP structure in video streaming.

It should be noted that similar reference numbers mean similar details across all representations in the drawings accordingly.

DETAILED DESCRIPTION OF THE INVENTION

The following detailed description is directed to certain aspects described in this application. However, the invention can be practiced in a large number of different ways. Reference in this detailed description to “one aspect” or “aspect” means that a particular feature, design, or characteristic described in connection with an aspect is included in at least one aspect. The occurrence of the phrase “in one aspect”, “according to one aspect” or “in some aspects” in different places of this detailed description does not necessarily all refer to the same aspect, nor are they separate or alternative aspects mutually exclusive of other aspects. Moreover, various features are described that can be demonstrated by some aspects, but not by others. Similarly, various requirements are described, which may be requirements for some aspects, but not for other aspects.

The following description includes details in order to provide a thorough understanding of the examples. However, those skilled in the art should understand that examples can be used in practice even if all the details of the process or device in the example or aspect are not explained or illustrated in this document. For example, block diagrams may show electrical components that do not illustrate all electrical connections or all electrical components, so as not to distract from the understanding of examples by unnecessary details. In other cases, such components, other structures and techniques may be shown in detail to further illustrate examples.

The present device relates to devices and methods for controlling encoding and transcoding using information content of encoded multimedia data. “Content information” or “content” (multimedia data) are broad terms referring to information related to the content of multimedia data, and may include, for example, metadata, metrics computed from multimedia data, and related information, associatively Associated with one or more metrics, such as content classification. Content information may be provided to the encoder or determined by the encoder, depending on the particular application. Content information can be used for many aspects of coding multimedia data, including detecting scene changes, temporal processing, spatio-temporal noise reduction, downsampling, determining bit rates for quantization, scaling error tolerance, maintaining optimal multimedia quality in broadcast channels and fast channel switching. Using one or more of these aspects, the transcoder can control the processing of multimedia data and generate encoded multimedia data associated with the content. The descriptions and drawings herein that describe transcoding aspects can also be applied to coding aspects and decoding aspects.

Transcoder devices and methods are associated with transcoding from one format to another and are described in detail herein as being related to transcoding MPEG-2 video into an enhanced scalable H.264 format for wireless transmission to mobile devices illustrating some aspects. However, the description of the transcoding of MPEG-2 video to H.264 is not intended to limit the scope of the invention, but is merely an example of some aspects of the invention. The disclosed devices and methods provide a highly efficient architecture that supports error-resistant coding with random access and leveling capabilities and can also be applied to transcoding and / or encoding of video formats other than MPEG-2 and H.264.

“Multimedia data” or simply “multimedia” as used herein is an extensive term that includes video data (which may include audio data), audio data, or both video data and audio data. "Video data" or "video" as used herein is a broad term referring to frame-based or field-based data that includes one or more images or related image sequences containing text, image information and / or audio data, and can be used to mean multimedia data (for example, the terms can be used interchangeably), unless otherwise indicated.

The following describes examples of various components of the transcoder and examples of processes that can use content information to encode multimedia data.

1A is a flowchart illustrating a data stream of some aspects of a multimedia broadcast system 100. At system 100, the media provider 106 transmits the encoded media 104 to the transcoder 200. The encoded media 104 is received by the transcoder 200, which processes the media 104 into raw raw media at 110. The processing at block 110 decodes and parses the encoded media data 104 and further processes the multimedia data to prepare them for encoding into another format. The decoded multimedia data is provided at block 112, where the multimedia data is encoded into a predetermined multimedia format or standard. After the multimedia data has been encoded, in step 114 it is prepared for transmission, for example, by means of a wireless broadcast system (for example, a cellular telephone broadcast network or through another communication network). In some aspects, the received multimedia data 104 is encoded according to the MPEG-2 standard. After the transcoded multimedia data 104 is decoded, the transcoder 200 encodes the multimedia data into the H.264 standard.

FIG. 1B is a block diagram of a transcoder 130 that may be configured to perform processing in steps 110 and 112 of FIG. 1A. Transcoder 130 may be configured to receive multimedia data, decode and parse multimedia data into packetized elementary streams (for example, subtitles, audio, metadata, raw video, CC data and presentation timestamps), encode them to the desired format and provide encoded data for further processing or transmission. The transcoder 130 may be configured to provide encoded data in two or more data groups, for example, an encoded first data group and an encoded second data group, which is referred to as multi-level encoding. In some examples of aspects, different groups of data (or levels) in a layered coding scheme can be encoded at different quality levels and formatted so that the data encoded in the first group of data is of lower quality (for example, they provide a lower level of visual quality when displayed), than the data encoded in the second data group.

FIG. 1C is a block diagram of a processor 140 that may be configured to transcode multimedia data, and may be configured to perform part or all of the processing illustrated in steps 110 and 112 of FIG. 1A. The processor 140 may include modules 124a ... n, perform one or more of the transcoding processes described herein, including decoding, parsing, preprocessing, and encoding, and use the content information for processing. The processor 140 also includes internal storage device 122 and may be configured to communicate with external storage device 120 directly or indirectly through another device. The processor 140 also includes a communication module 126 configured to exchange data with one or more devices external to the processor 140, including receiving multimedia data and providing encoded data, such as data encoded in the first data group, and data encoded in a second data group. In some examples of aspects, different groups of data (or levels) in a layered coding scheme can be encoded at different quality levels and formatted so that the data encoded in the first group of data is of lower quality (for example, they provide a lower level of visual quality when displayed), than the data encoded in the second data group.

The transcoder 130 or its components of the preprocessor 140 (configured for transcoding), as well as the processes contained therein, may be implemented by hardware, software, firmware, middleware, microcode, or a combination of the above. For example, the analyzer, decoder, preprocessor or encoder may be stand-alone components included as hardware, firmware, middleware in a component of another device, or may be implemented in microcode or software that is executed in a processor, or as a combination of the foregoing. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform motion compensation, frame classification, and coding processes can be stored on a computer-readable medium, such as a storage medium. A code segment can represent a procedure, function, subprogram, program, standard procedure, nested procedure, module, software package, class, or any combination of instructions, data structures, or program statements. A code segment may be associated with another code segment or a hardware circuit by transmitting and / or receiving information, data, arguments, parameters, or memory contents.

Illustrative Transcoder Architecture Example

FIG. 2 illustrates a block diagram of an example transcoder that can be used for a transcoder 200 illustrated in the multimedia broadcast system 100 of FIG. 1. The transcoder 200 comprises an analyzer / decoder 202, a preprocessor 226, an encoder 228, and a clock level 240, further described below. The transcoder 200 is configured to use the information content of the multimedia data 104 for one or more aspects of the transcoding process, as described herein. Content information can be obtained from a source external to transcoder 200, through multimedia metadata, or computed by transcoder, for example, by preprocessor 226 or encoder 228. The components shown in FIG. 2 illustrate components that can be included in a transcoder that uses content information for one or more transcoding processes. In a specific implementation, one or more components of transcoder 200 may be omitted or additional components may be included. Additionally, portions of the transcoder and transcoding processes are described in order to enable those skilled in the art to practice the invention, even if all details of the process or device may not be described herein.

FIG. 5 illustrates a timing diagram as a graphic illustration of the temporal relationship of the operation of various components and / or processes of transcoder 200. As shown in FIG. 5, encoded video stream 104 (encoded multimedia data), for example, MPEG-2 video is first received at arbitrary zero point in time (0) through the analyzer 205 (figure 2). Next, the video stream is disassembled 501, demultiplexed 502, and decoded 503, for example, by an analyzer 205 in combination with a decoder 214. As illustrated, these processes can be performed in parallel with a small time offset to provide streaming output of the processing data to the preprocessor 226 (Fig. 2) . At time T ₁ 504, after the preprocessor 226 has received enough data from decoder 214 to start outputting the processing results, the remaining processing steps become sequential in nature, with encoding 505 of the first pass, encoding 506 of the second pass and re-encoding 507 performed sequentially after preprocessing until re-coding is completed at time point T _f 508.

The transcoder 200 described herein may be configured to transcode various multimedia data, and many of the processes apply to the type of multimedia data that is transcoded. Although some of the examples provided herein relate to transcoding MPEG-2 data into H.264 data, these examples are not intended to limit the invention to these data. The encoding aspects described below can be applied to transcoding any relevant multimedia data standard to another corresponding multimedia data standard.

Analyzer / decoder

Referring again to FIG. 2, the analyzer / decoder 202 receives the multimedia data 104. The analyzer / decoder 202 includes a transport stream analyzer (“analyzer”) 205, which receives the multimedia data 104 and parses the data into an elementary stream (ES) of the video 206, ES 208 audio, presentation time stamps (PTS) 210, and other data such as subtitles 212. The ES carries one type of data (video or audio) from a single video or audio encoder. For example, an ES video contains video data for a data sequence including a sequence header and all sub-sequences. A packetized elementary stream, or PES, consists of one ES, which is composed of packets, each of which typically begins with an added packet header. A PES stream contains only one data type from a single source, for example, from a single video or audio encoder. PES packets have a variable length that does not match the fixed length of the transport packets, and can be much longer than the transport packet. When transport packets are generated from the PES stream, the PES header can be placed at the beginning of the transport packet operational data, immediately after the transport packet header. The remaining content of the PES packet fills the operational data of the serial transport packets until the PES packet is fully utilized. The final transport packet can be filled to a fixed length, for example, by padding with bytes, for example, bytes = 0xFF (all units).

Analyzer 205 transmits video ES 206 to decoder 214, which is part of analyzer / decoder 202, shown here. In other configurations, analyzer 205 and decoder 214 are separate components. PTS 210 are sent to transcoder PTS shaper 215, which can generate separate presentation timestamps specific to transcoder 200 for use in composing data to be sent from transcoder 200 to a broadcast system. The transcoder PTS 215 may be configured to provide data to the synchronization layer 240 of the transcoder 200 to coordinate the synchronization of the broadcast data.

FIG. 3 illustrates a flowchart of one example of a process 300 that analyzer 205 can follow when parsing the various packetized elementary streams described above. The process 300 begins at block 302 when multimedia data 104 is received from the content provider 106 (FIG. 1). The process 300 proceeds to step 304, where the analyzer 205 is initialized. The initialization can be activated by an independently generated data acquisition command 306. For example, a process that is independent of analyzer 205 and is based on an externally received TV schedule and channel setting information can generate a data acquisition command 306. Additionally, real-time transport stream (TS) buffer descriptors 308 can be entered to aid in initialization and for basic processing.

As illustrated in step 304, initialization may include a command syntax check request, execution of the first pass PSI / PSIP / SI processing (program-specific information / program and system information protocol / system information), execution of processing specifically associated with either the collection command data, or by checking the consistency of PSI / PSIP / SI, assigning PES buffers for each PES and setting the time distribution (for example, to combine it with the required time to start collecting data). PES buffers contain parsed ES data and transmit all parsed ES data to respective audio decoder 216, test decoder 220, decoder 214, or PTS transformer 215.

After initialization, the process 300 proceeds to block 310 for the main processing of the received multimedia data 104. The processing in block 310 may include filtering of target packet identifiers (PIDs), continuous monitoring and processing of PSI / PSIP / SI, and a time allocation process (for example, to achieve the required duration of data collection) so that incoming multimedia data is transmitted to the corresponding PES buffers. As a result of the processing of multimedia data at step 310, a software descriptor and a read PES buffer indicator are formed, which interact with decoder 214 (FIG. 2) as described below.

After step 310, process 300 proceeds to step 314, which completes parsing operations, generating a timer interrupt, and freeing the PES buffers after using them. It should be noted that PES buffers are available for all relevant elementary program streams mentioned in the descriptor, for example, audio, video and subtitle streams.

Referring again to FIG. 2, the analyzer 205 sends the audio ES 208 to the audio decoder 216 to match the transcoder implementation and provides encoded text 216 to the synchronization layer 240 and decodes the audio information. Subtitle information 212 is delivered to a text decoder 220. Inter-frame encoded subtitle (CC) data 218 from decoder 214 is also provided to a text encoder 220 that encodes subtitle information 212 and CC data 218 in a format implemented by transcoder 200.

The analyzer / decoder 202 also includes a decoder 214 that receives the ES 206 video. Decoder 214 may generate metadata associated with the video data, decodes the encoded packetized elementary video stream into raw video 224 (eg, in standard definition format), and processes the encoded subtitle video data between frames into an ES video stream.

FIG. 4 illustrates a flowchart showing one example of a decoding process 400 that can be performed by a decoder 214. Process 400 begins by inputting elementary video stream data 206 at step 402. Process 400 proceeds to step 404, at which the decoder is initialized. Initialization may include a number of tasks, including detecting a video sequence header (VSH), performing a first pass VSH, processing a video sequence (VS) and display extension VS (including video format, composite colors and matrix coefficients) and assigning data buffers so that it is appropriate buffering the decoded image, associated metadata, and inter-frame encoded subtitle (CC) data. Additionally, the read 40 information of the PES video buffer provided by the analyzer 205 is input (for example, which may be generated by the process 300 in step 310 of FIG. 3).

After initialization in step 404, the process 400 proceeds to step 408, in which the main processing of the ES video is performed by the decoder 214. The main processing includes requesting the “read” information of the video PES buffer or “interface” for the availability of new data, decoding the ES video, restoration and preservation of pixel data during synchronization of image boundaries, metadata of video and audio-video generation, and storage at image boundaries and storage of CC data at image boundaries. Step 410 of the results of the main processing 408 includes generating sequence descriptors, decoded image buffer descriptors, metadata buffer descriptors, and CC data buffer descriptors.

After the main processing 408, the process 400 proceeds to step 412, where it performs the completion process. The completion process may include determining completion conditions, including the absence of new data appearing within a specific time above a predetermined threshold, detecting a final sequence code and / or detecting an explicit completion signal. The completion process may further include freeing the decoded image, associated metadata, and CC data buffers after use by the preprocessor, which is described below. The process 400 ends at block 414, where it can transition to the idle state of the ES video, which should be received as input.

Preprocessor

FIG. 2 and FIG. 6 in more detail illustrate an exemplary aspect of a preprocessor 226 that can use content information for one or more preprocessing operations. Preprocessor 226 receives metadata 222 and decoded "raw" video data 224 from analyzer / decoder 202. Preprocessor 226 is configured to perform certain types of processing for video data 224 and metadata 222 and provide processed multimedia (for example, reference level frames, level reference frames improvements, bandwidth information, content information) and video to encoder 228. Such preliminary processing of multimedia data allows to increase visual clarity, suppression of overlays Crater and data compression efficiency. In general, preprocessor 226 receives the video sequences provided by decoder 214 in analyzer / decoder 202, and converts the video sequences to progressive video sequences for further processing (eg, encoding) by encoder 228. In some aspects, preprocessor 226 may be configured for a variety of operations, in including conversion operations from video format to film, eliminating interlacing, filtering (for example, removing image defects, suppressing reverb radios, unlocking and reducing noise), resizing (for example, downsampling spatial resolution from standard resolution to a quarter of the logical matrix of video graphics (QVGA)) and generating a GOP structure (for example, calculating the formation of a complexity table, detecting scene changes and detecting shading / flash).

Preprocessor 226 may use metadata from a decoder to affect one or more preprocessing operations. Metadata may include information related to, describing or classifying the contents of the multimedia data ("content information"); in particular, metadata may include classification of content. In some aspects, metadata does not include content information required for encoding operations. In these cases, the preprocessor 226 may be configured to determine the content information and use the content information for preprocessing operations and / or provide the content information to other components of the transcoder 200, for example, the decoder 228. In some aspects, the preprocessor 226 may use this content information in order to act on the partitioning of the GOP, determine the appropriate type of filtering and / or determine the encoding parameters that are transmitted to the encoder.

6 shows an illustrative example of various processing steps that may be included in preprocessor 226, and illustrates processing that may be performed by preprocessor 226. In this example, preprocessor 226 receives metadata and video 222, 224 and provides output 614 containing ( processed) metadata and video, to encoder 228. In a typical embodiment, there are three types of video that can be received. Firstly, the received video can be a progressive video where elimination of interlacing is not required. Secondly, the video data may be video converted from a movie to video format, interlaced video converted from movie sequences to 24 frames per second, in this case video. Thirdly, the video may be interlaced video, not converted from film to video format. Preprocessor 226 may process these types of videos as described below.

At step 601, the preprocessor 226 determines whether the received video data 222, 224 is a progressive video. In some cases, this can be determined from the metadata if the metadata contains this information, or by processing the video data itself. For example, the process of converting from a video format to a film, described below, allows you to determine whether the received video 222 is a progressive video. If so, the process proceeds to step 607 where filtering operations (e.g., noise reduction) are performed for the video to reduce noise, such as white Gaussian noise. If the video data 222, 224 is not progressive video, at block 601, the process proceeds to block 604 to the phase detection module 604.

The phase detection module 604 distinguishes between video that was originally converted from a movie to video format and that started in a standard broadcast format. If a decision has been made that the video has been converted from the film to video format (the solution path is “YES” coming out of the phase detection module 604), the video converted from the film to the video format is returned to the original format when the video is converted back to video format 606. Redundant frames are identified and eliminated, and fields extracted from the same video frame are intertwined into a full image. Since the sequence of reconstructed motion pictures is photographically recorded at regular intervals of 1/24 second, the motion estimation process performed in the GOP sectioning module 612 or decoder 228 is more accurate when using images converted from a video format to a film than data converted from a film to a video format, which have an irregular time base.

In one aspect, the phase detection module 604 makes specific decisions after receiving the video frame. These solutions include: (i) is the current video from the conversion output from the movie to video format, and the 3: 2 advance phase is one of the five phases P ₀ , P ₁ , P ₂ , P ₃ and P ₄ shown in Fig.38, or not; and (ii) the video is formed as a traditional NTSC. This solution is referred to as phase P ₅ . These decisions are displayed as outputs of the phase detection module 604 shown in FIG. The path from the phase detection module 604 marked “YES” activates the conversion 606 from the video format to the film, indicating that it is provided with the correct advancement phase per frame step so that it can sort the fields that are formed from one photographic image, and combine them. The path from the phase detection module 604 marked “NO” likewise activates the interlace scanning elimination module 605 to divide the visible NTSC frame into fields for optimal processing. The phase detection module 604 can continuously analyze video frames since various types of video can be received at any time. As an example, a video conforming to the NTSC standard can be inserted into the video as an advertisement. After converting from the video format to the film, the resulting progressive video is sent to the noise reduction module (filter) 607, which can be used to reduce white Gaussian noise.

When the traditional NTSC video is recognized (path "NO" from the phase detection module 601), it is transmitted to the interlace scanning elimination unit 605 for compression. The interlacing elimination unit 605 converts the interlaced fields to a progressive video, and noise reduction operations can then be performed for the progressive video. One illustrative example of interlacing elimination processing is described below.

Traditional analog video devices, such as televisions, render video interlaced, i.e. these devices transmit scan lines with even numbers (even field) and scan lines with odd numbers (odd field). In terms of signal sampling, this is equivalent to spatio-temporal subsampling in a pattern described by:

[one]

where Θ means the image of the original frame, F means the interlaced field, and (x, y, n) represents the horizontal, vertical and temporary position of the pixel, respectively.

Without loss of generality, we can assume that n = 0 is an even field in this description, so that the above equation 1 is simplified as follows:

[2]

Since thinning is not performed in the horizontal direction, the downsampling pattern can be illustrated in the next ny coordinate.

The purpose of the interlaced scan elimination module is to convert the interlaced video (field sequence) to non-interlaced progressive frames (frame sequence). In other words, interpolate even and odd fields so as to “restore” or form full-frame images. This can be represented by equation 3:

[3]

where F _i represents the interlacing elimination results for missing pixels.

FIG. 40 is a block diagram illustrating certain aspects of an interlacing elimination unit 605 that utilizes Wmed filtering and motion estimation in order to form a progressive frame from interlaced multimedia data. The upper part of FIG. 40 shows a traffic intensity table 4002, which can be generated using information from the current field, two previous fields (PP field and P field) and two subsequent fields (Next field and Next Next field). A motion intensity table 4002 categorizes or partitions the current frame into two or more different levels of motion and can be generated by spatio-temporal filtering, described in more detail below. In some aspects, the traffic intensity table 4002 is formed to identify static regions, slow motion regions, and fast motion regions, as described in relation to equations 4-8 below. A spatio-temporal filter, for example, a Wmed filter 4004 filters interlaced multimedia data using criteria based on a motion table and generates a spatio-temporal prepared frame with interlaced eliminated. In some aspects, the Wmed filtering process entails a horizontal neighborhood of [-1, 1], a vertical neighborhood of [-3, 3] and a temporary neighborhood of five subsequent fields, which are represented by five fields (PP field, P field, field Current, Next field, Next Next field), illustrated in FIG. 40, where Z ^-1 represents the delay of one field. Regarding the Current field, the Next field and the P field are non-parity fields, and the PP field and the Next Next field are parity fields. "Neighborhood" used for spatio-temporal filtering means the spatial and temporal placement of fields and pixels actually used during the filtering operation, and can be illustrated as an "aperture", as shown, for example, in FIGS. 6 and 7.

The interlacing elimination module 605 may also include a noise reduction module 4006 (noise reduction filter) configured to filter out a spatio-temporal prepared interlaced frame formed by the Wmed filter 4004. Decreasing the noise of a spatio-temporal prepared frame with eliminated interlacing makes the subsequent motion search process more accurate, especially if the original sequence of interlaced multi The single data is dirty with white noise. It can also remove, at least in part, sampling interference between even and odd lines in a Wmed image. Noise reduction module 4006 may be implemented as a plurality of filters, including a noise reduction module based on a wavelet compression and a Wiener wavelet filter. The noise reduction module can be used to remove noise from the Wmed frame option before it is further processed using motion compensation information, and can remove the noise that is present in the Wmed frame and preserve the presence of the signal regardless of the frequency content signal. Various types of noise reduction filters can be used, including wavelet filters. Wavelets are a class of functions used to localize a given signal both in the spatial domain and in the scaling domain. The fundamental idea for wavelets is to analyze the signal at different scales or resolutions so that minor changes in the wavelet representation produce corresponding small changes in the original signal.

Wiener wavelet compression or Wiener wavelet filter can also be used as a noise reduction module. Wavelet compression consists of a wavelet transform of a noise signal, followed by compression of small wavelet coefficients to zero (or a lower value) while leaving large coefficients unchanged. Finally, the inverse transform is performed in order to obtain an estimated signal.

Noise reduction filtering improves the accuracy of motion compensation in noisy environments. Noise reduction with wavelet compression can entail compression in the wavelet transform domain and typically includes three stages: linear direct wavelet transform, nonlinear noise reduction with compression, and linear inverse wavelet transform. The Wiener filter is an MSE-optimal linear filter that can be used to enhance images that are degraded by additive noise and blur. These filters are generally known in the art and are described, for example, in the work “Ideal spatial adaptation by wavelet shrinkage” mentioned above and in the work of the authors SP Ghael, AM Sayeed and RG Baraniuk, “Improvement Wavelet denoising via empirical Wiener filtering ", Proceedings of SPIE , Volume 3169, pp. 389-399, San Diego, July 1997, which are hereby incorporated by reference in their entirety.

In some aspects, the noise reduction filter is based on the aspect of the biorthogonal cubic B-spline wavelet filter (4, 2). One such filter can be specified by performing forward and reverse transforms:

(прямое преобразование)

(direct conversion) [four]

and

(обратное преобразование)

(inverse transformation) [5]

The use of a noise reduction filter improves the accuracy of motion compensation in noisy environments. Implementations of such filters are further described in "Ideal spatial adaptation by wavelet shrinkage," by DL Donoho and IM Johnstone, Biometrika, Volume 8, pp. 425-455, 1994, which are clearly incorporated herein by reference.

The bottom of FIG. 40 illustrates an aspect for determining motion information (e.g., motion vector options, motion estimation, motion compensation) of interlaced multimedia data. In particular, FIG. 40 illustrates a motion estimation and motion compensation scheme that is used to form a preliminary progressive frame after motion compensation from a selected frame, and then combined with a preliminary Wmed frame to form a “final” progressive frame, shown as current frame 4014 with interlaced eliminated. In some aspects, options (or estimates) of the motion vectors (MV) of the interlaced multimedia data are provided to the interlacing elimination unit from the external motion estimation modules and are used to provide a starting point for the bidirectional motion estimation and compensation module (ME / MC) 4018. In some aspects, the MV-option selection module 4022 uses the previously defined MVs for neighboring blocks for the MV-variants of the processed blocks, for example, the MVs of the previous processed blocks, for example, the blocks in the previous Adre 4020 deinterlacing. Motion compensation can be performed bi-directionally based on the previous frame 70 with interlaced scanned and the next (for example, future) Wmed frame 4008. The current Wmed frame 4010 and the current frame 4016 after motion compensation (MC) are combined or combined by combining module 4012. The resulting current interlaced frame 4014, now a progressive frame, is provided back to the ME / MC 4018 to be used as the previous interlaced frame 4020 and is also transferred outside the interlaced elimination unit 605 for further processing.

It is possible to distinguish inter-field prediction prediction schemes containing interfield interpolation from intra-field interpolation using the Wmed + MC interlace interruption correction scheme. In other words, spatio-temporal Wmed filtering can be used primarily for the purpose of in-field interpolation, while the inter-field interpolation can be performed during motion compensation. This reduces the peak signal-to-noise ratio of the Wmed result, but the visual quality after the motion compensation is applied is more satisfactory since defective pixels from inaccurate in-field prediction mode solutions are removed from the Wmed filtering process.

After appropriately processing the conversion from the video format to the film or eliminating interlacing at step 608, the progressive video is processed to suppress sampling noise and resampling (e.g., resizing). In some aspects of resampling, a phase resampling module is implemented for resizing images. In one downsampling example, the ratio between the original and the resized image may be p / q , where p and q are relatively prime numbers. The total number of phases is p. The cutoff frequency of the polyphase filter is, in some aspects, 0.6 for a coefficient of change of size of approximately 0.5. The cutoff frequency does not exactly match the resizing factor to enhance the high frequency response of the resized sequence. This inevitably provides certain sampling interference. However, it is well known that the human eye prefers sharp, but with little interference, images to blurry and interference-free images.

Fig. 41 illustrates an example of polyphase resampling showing phases if the size change ratio is 3/4 . The cutoff frequency illustrated in FIG. 41 is also 3/4. The original pixels are illustrated in the above figure with vertical axes. A sinusoidal function is also drawn centered around the axes to represent the filter waveform. Since the cutoff frequency is selected to be the same as the resampling ratio, the zeros of the sinusoidal function overlap the position of the pixels after the resizing illustrated in FIG. 41 by intersections. To find the pixel value after resizing, the fraction can be summed from the original pixels, as shown in the following equation:

[6]

where f _c is the cutoff frequency. The above one-dimensional polyphase filter can be applied to the horizontal direction and the vertical direction.

Another aspect of resampling (resizing) allows for off-screen scanning. In an NTSC television signal, the image has 486 scan lines, and in digital video, it can have 720 pixels for each scan line. However, not all of the full image is visible on the TV due to discrepancies between the size and screen format. The part of the image that is not visible is called the off-screen scan.

To help broadcast stations put useful information in an area visible to as many TVs as possible. The Society of Film and Television Engineers (SMPTE) has set specific frame sizes for moving objects, called the protected area of moving objects and the protected area of titles. See SMPTE RP 27.3-1989 Guidelines " Specifications for Safe Action and Safe Title Areas Test Pattern for Television Systems ". The protected area of moving objects is defined by SMPTE as the area in which all significant moving objects should be located. A protected title area is defined as an area in which all useful information can be enclosed to provide visibility on most home television sets.

For example, referring to FIG. 25, the protected area 2510 of moving objects occupies 90% of the screen in the center, giving a 5% border around. The protected area of 2505 titles occupies the center of 80% of the screen, giving 10% of the border. Referring now to FIG. 26, since the protected area of the titles is too small to add additional content to the image, some stations place text in the protected area of moving objects that is inside the white rectangular window 2615.

Typically, black borders can be seen in the scan outside the screen. For example, in FIG. 26, black borders are displayed on the upper side 2620 and the lower side 2625 of the image. These black borders can be removed in the scan outside the screen, because the H.264 video uses the extension of borders in the estimation of motion. Extended black borders allow you to increase the remainder. Conservatively, the border can be trimmed by 2% and then resized. Filters for resizing can be formed accordingly. Clipping is performed in order to remove the scan outside the screen to polyphase downsampling.

Referring again to FIG. 6, the progressive video then proceeds to step 610, where the unlock and suppress reverb operations are performed. Two types of interference, “blocking” and “reverb,” typically occur in video compression applications. Image noise occurs because compression algorithms divide each frame into blocks (for example, 8x8 blocks). Each block is restored with some small errors, and errors at the edges of the block often contrast with errors at the edges of neighboring blocks, making the boundaries of the blocks visible. In contrast, reverb noise appears as distortion at the edges of image elements. Reverb noise occurs because the encoder discards too much information when quantizing high-frequency DCT coefficients. In some illustrative examples, releasing and suppressing reverbs can use low-pass filters with FIR (finite impulse response) to mask this visible interference.

In one example of release processing, a release filter can be applied to all edges of 4x4 blocks of a frame, except for edges at the frame boundary and all edges for which the release filter process is disabled. This filtering process should be performed on the basis of macroblocks after completion of the compilation process, and all macroblocks in the frame are processed in the order of increasing addresses of the macroblocks. For each macroblock, the vertical edges are filtered first, from left to right, and then the horizontal edges are filtered from top to bottom. The release filter process of the luminance component is performed at four edges of 16 samples, and the release filter process for each color component is performed at two edges of 8 samples for horizontal direction and for vertical direction, as shown in Fig. 39. The sample values above and to the left of the current macroblock, which may have already been modified by the release process operation for previous macroblocks, should be used as input to the release filter process for the current macroblock and can be further modified by filtering the current macroblock. Sample values modified by filtering vertical edges can be used as input to filter horizontal edges for the same macroblock. The release process can be activated separately for the luminance and chrominance components.

In an example of a reverb suppression processing, a two-dimensional filter can be adaptively applied so as to smooth out areas near the edges. Edge pixels are slightly filtered or not filtered to avoid blurring.

GOP Partition Module

After the reverb is released and suppressed, the progressive video is processed by the GOP section module 612. GOP placement can include frame change detection, generation of complexity tables (e.g., temporary, spatial bandwidth tables), and adaptive GOP partitioning. All of the foregoing is described below.

A. Scene change detection

Frame detection refers to determining when a frame in an image group (GOP) provides data that indicates that a scene change has occurred. In general, in a GOP, frames may not have a significant shift for any two or three (or more) adjacent frames, or there may be slow shifts or fast shifts. Of course, these classifications of scene changes, if necessary, can be further broken down to a higher level of change depending on the application.

Detecting frame or scene changes is important for efficient video encoding. In a typical embodiment, when the GOP does not change significantly, the I-frame at the beginning of the GOP is followed by a series of predictive frames, it allows to encode the video to a large extent so that subsequent decoding and display of the video is visually acceptable. However, when a scene changes, abruptly or slowly, additional I-frames and less predictive coding (P-frames and B-frames) may be required to provide subsequently decoded, visually acceptable results.

Systems and methods for detecting frames and coding that enhance the performance of existing coding systems are described below. These aspects can be implemented in the GOP sectioning module 612 of the preprocessor 226 (FIG. 7) or included in an encoder device that can operate with or without a preprocessor. These aspects use statistics (or indicators), which include statistical comparisons between adjacent frames of video data, to determine if a sharp change of scene occurs, the scene changes slowly or there are camera flashes in the scene, which can especially complicate video encoding. Statistics can be obtained from the preprocessor and then sent to the encoding device, or it can be generated in the encoding device (for example, by means of a processor configured to perform motion compensation). The resulting statistics helps in deciding whether to detect scene changes. In a system that transcodes, there is often an appropriate preprocessor or configurable processor. If the preprocessor performs interlaced resolution using motion compensation, motion compensation statistics are available and ready to use. In such systems, the frame detection algorithm may slightly increase the complexity of the system.

The illustrative example of the frame detection module described in this document should only use statistics from the previous frame, the current frame, and the next frame, and therefore has a very small delay. The frame detection module distinguishes several different types of frame events, including a sharp change of scenes, editing transition and other slow change of scenes, as well as a camera flash. By defining various types of frame events with different strategies in the encoder, coding efficiency and visual quality are improved.

Scene change detection can be used for any video coding system so that it intelligently saves bits by inserting an I-frame at a fixed interval. In some aspects, content information obtained by a preprocessor (eg, either contained in metadata or computed by preprocessor 226) can be used to detect scene changes. For example, depending on the content information, thresholds and other criteria described below may be dynamically adjusted for various types of video content.

Video encoding is usually performed for a structured image group (GOP). A GOP usually starts with an internally encoded frame (I-frame), followed by a sequence of P (predictive) or B (bidirectional) frames. In a typical embodiment, the I-frame can store all the data required to display the frame, the B-frame is based on the data in the previous and subsequent frames (for example, containing only data modified from the previous frame or different from the data in the next frame) , and the P-frame contains data that has been changed from the previous frame. In most cases, P-frames and B-frames in encoded video are inserted between I-frames. In terms of size (for example, the number of bits used to encode a frame), I-frames are typically much larger than P-frames, which, in turn, are larger than B-frames. For efficient encoding, transmission and decoding processing, the GOP length must be large enough to reduce the actual loss of large I-frames, and small enough to withstand the mismatch between the encoder and decoder or the degradation of channel quality. In addition, macroblocks (MBs) in P frames may be internally encoded for the same reason.

Scene change detection can be used by the video encoder to determine the corresponding GOP length and insert I-frames based on the GOP length instead of inserting the often unnecessary I-frame with a fixed interval. In a practical video streaming system, the quality of the communication channel is usually degraded by bit errors or packet loss. Where I-frames or I-MBs are placed can significantly affect decoded video quality and viewing comfort. One coding scheme is to use internally encoded frames for images or parts of images, which are significantly different than previous images or parts of images shared. Typically, these zones cannot be effectively and rationally predicted in motion estimation, and coding can be performed more efficiently if these zones are excluded from interframe coding techniques (for example, coding using B-frames and P-frames). In the context of channel quality degradation, these zones are likely to experience error propagation, which can be reduced or eliminated (or practically eliminated) by means of intraframe coding.

Parts of a GOP video can be classified into two or more categories, with each zone having different in-frame coding criteria, which may depend on the particular implementation. As an example, a video can be classified into three categories: abrupt scene change, editing transition and other slow scene changes, as well as camera flashes.

Sudden scene changes include frames that are significantly different from the previous frame, usually caused by camera operation. Since the contents of these frames are different from the contents of the previous frame, the scene change frames must be encoded as I-frames.

Flick transition and other slow scene changes include slow scene switching, usually caused by the computational processing of the camera shot. Gradually mixing two different scenes may look more pleasing to the human eye, but it poses a daunting task in coding a video. Motion compensation cannot effectively reduce the bit rate of these frames, and more internal MBs can be updated for these frames.

Camera flashes, or camera flash events, occur when the contents of a frame include camera flashes. These flashes have a relatively short duration (for example, one frame) and are very bright, so that the pixels in the frame rendering the flashes show a very high brightness relative to the corresponding area in the adjacent frame. Camera flashes shift image brightness suddenly and quickly. Typically, the duration of the camera flash is less than the duration of the temporary masking of the human visual system (HVS), which is typically set to 44 ms. Human eyes are not sensitive to the quality of these short bursts of brightness, and therefore they can be encoded more roughly. Since flash frames cannot be efficiently processed using motion compensation and they are a poor prediction option for future frames, rough coding of these frames does not reduce the coding efficiency of future frames. Scenes classified as flashes should not be used to predict other frames due to “artificial” high brightness, and other frames cannot be used effectively to predict these frames, for the same reason. Once identified, these frames can be retrieved since they may require a relatively high amount of processing. One option is to remove camera flashes and encode the DC factor in their place; this solution is simple, computationally fast and saves a lot of bits.

When any of the above frame categories are detected, a frame event is declared. Frame detection is not only useful for improving coding efficiency, it can also help in identifying search and indexing video content. One illustrative aspect of the scene detection process is described below. In this example, the frame detection process first calculates information, or metrics, for the selected frame processed for frame detection. Indicators may include information from two-dimensional processing of estimation and motion compensation for video and other brightness-based indicators.

In order to perform two-dimensional motion estimation / compensation, the video sequence must be pre-processed using the bidirectional motion compensation module, which maps each 8x8 block of the current frame to the blocks in two frames closest to neighboring frames, one in the past and one in the future. The motion compensation unit generates motion vectors and differential indicators for each block. Fig. 29 is an illustration that shows an example of matching the pixels of the current frame C with the past frame P and the future (or next) frame N and shows the motion vectors for the matched pixels (past motion vector MV _P and future motion vector MV _N ). A general description of the formation of bidirectional motion vectors and associated coding, in General, is given below with reference to Fig.32.

After determining the bi-directional motion information (for example, motion information that identifies the MBs (most matching) in the respective neighboring frames, additional metrics can be generated (for example, by the motion compensation module in the GOP sectioning module 612 or other appropriate component) through various comparisons the current frame with the next frame and the previous frame.The motion compensation module can generate a differential metric for each block. Lemma may be the sum of squared differences (SSD) or sum of absolute difference (SAD). Without loss of generality, here used as an example SAD.

For each frame, the SAD ratio, also referred to as the "contrast ratio", is calculated as follows:

[6]

where SAD _P and SAD _N are the sum of the absolute differences of the direct and inverse difference index, respectively. It should be noted that the denominator contains a small positive number ε in order to prevent division by zero errors. The numerator also contains ε to compensate for the influence of unity in the denominator. For example, if the previous frame, the current frame, and the next frame are identical, the motion search should yield SAD _P = SAD _N = 0. In this case, the above shapers compute γ = 1 instead of 0 or infinity.

A luminance histogram can be calculated for each frame. In a typical embodiment, multimedia images have a depth of brightness (for example, the number of "resolution elements") of eight bits. The luminance depth used to calculate the luminance histogram according to some aspects may be set to 16 to obtain a histogram. In other aspects, the depth of brightness may be set equal to the corresponding number, which may depend on the type of data being processed, available processing power, or other predetermined criteria. In some aspects, the depth of luminance may be set dynamically based on a calculated or received metric, for example, data content.

The following equation illustrates one example of calculating the difference of luminance histograms (lambdas):

[7]

where N _Pi is the number of blocks in the i-th resolution element for the previous frame, N _Ci is the number of blocks in the i-th resolution element for the current frame, and N is the total number of blocks in the frame. If the difference between the brightness histograms of the previous and the current frame is completely dissimilar (or disjoint), then λ = 2 .

Using this information, the frame difference metric (D) is calculated as follows:

[8]

и

where A is a constant selected through an application,

and

The selected (current frame) is classified as a frame of a sharp change of scenes if the difference indicator of the frames satisfies the criterion shown in equation 9:

[9]

where A is a constant selected through an application, and T ₁ is a threshold.

In one example that the simulation shows, setting A = 1 and T ₁ = 5 provides good detection performance. If the current frame is a scene change frame, then γc should be large and γp should be small. The ratio γ _c / γ _p can be used instead of only γ _c so that the indicator is normalized to the level of activity.

It should be noted that the above criterion uses the difference of the histograms of lambda brightness (λ) in a nonlinear way. Fig. 16 illustrates that λ · (2λ + 1) is a convex function. When λ is small (for example, near zero), this is just a predistortion. The more λ becomes, the more pre-emphasis is provided through the function. Using this pre-emphasis for any λ greater than 1.4, a sharp change of scene is detected if the threshold T _{1 is} set to 5.

The current frame is defined as a fader transition or a slow scene change if the scene intensity indicator D satisfies the criterion shown in equation 5:

T ₂  ≤ D <T _one [10]

for a certain number of continuous frames, where T ₁ is the same threshold as used above, and T ₂ is a different threshold value.

A flash event usually causes the luminance histogram to shift to the brighter side. In the camera of this illustrative aspect, luminance histogram statistics are used to determine if the current frame of the camera flash. The frame detection process can determine whether the brightness of the current frame is greater than the brightness of the previous frame by a certain threshold T ₃ and the brightness of the current frame is greater than the brightness of the next frame by threshold T ₃ , as shown in equations 11 and 12:

[eleven]

[12]

If the above criterion is not satisfied, the current frame is not classified as containing camera flashes. If the criterion is met, the frame detection process determines whether the inverse difference indicator SAD _P and the direct difference indicator SAD _N are larger than a certain threshold T ₄ , as illustrated in the equations below:

SAD _P ≥ T _four [13] SAD _N ≥ T _four [fourteen]

- это средняя яркость текущего кадра,

- это средняя яркость предыдущего кадра,

- это средняя яркость следующего кадра, а SAD_P и SAD_N - это прямой и обратный разностные показатели, ассоциативно связанные с текущим кадром.Where

is the average brightness of the current frame,

is the average brightness of the previous frame,

is the average brightness of the next frame, and SAD _P and SAD _N are the forward and backward difference metrics associated with the current frame.

The frame detection process determines camera flash events by first determining whether the brightness of the current frame is greater than the brightness of the previous frame and the brightness of the next frame. If not, the frame is not a camera flash event; but if so, it could be. The frame detection process in this case can determine whether the inverse difference indicator of the threshold T _{3 is} greater and whether the forward difference indicator of the threshold T _{4 is} greater; if both of these conditions are met, the frame detection process classifies the current frame as having camera flashes. If the criterion is not satisfied, the frame is not even classified as any type of frame, or it may be assigned a default classification that identifies the encoding that must be performed for the frame (for example, delete a frame, encode as an I-frame).

Some exemplary values of T ₁ , T ₂ , T _3, and T ₄ are shown above. Typically, these thresholds are selected by testing a particular implementation of frame detection. In some aspects, one or more of the threshold values T ₁ , T ₂ , T _3, and T ₄ are predefined, and these values are included in the frame classifier in the encoding device. In some aspects, one or more threshold values T ₁ , T ₂ , T _3, and T ₄ may be set during processing (eg, dynamically) based on the use of information (eg, metadata) provided in the frame classifier, or based on information calculated by the frame classifier itself.

The encoding of the video using the frame detection information is typically performed in an encoder, but is described here for completeness of disclosure of frame detection. Referring to FIG. 30, the encoding process 301 may use the frame detection information to encode video based on the detected frames in a sequence of frames. Process 301 proceeds to step 303 and checks whether the current frame is classified as a scene change. If so, in step 305, the current frame may be encoded as an I-frame, and the GOP boundary may be determined. If not, process 301 proceeds to step 307; if the current frame is classified as part of a slowly changing scene, at step 309, the current frame and other frames in the slowly changing scene can be encoded as predictive frames (for example, a P-frame or B-frame). The process 301 then proceeds to step 311, in which it checks whether the current frame is classified as a flash scene containing camera flashes. If yes, in step 313, the frame can be identified for special processing, for example, to remove or encode the DC coefficient for the frame; if not, the classification of the current frame is not performed, and the current frame can be encoded in accordance with other criteria, encoded as an I-frame or deleted.

In the above aspect, the magnitude of the difference between the frame to be compressed and two adjacent frames is indicated by the frame difference indicator D. If a significant amount of one-sided brightness change is detected, this indicates the effect of a fader transition in the frame. The shorter the transition time, the greater the gain can be achieved by using B frames. In some aspects, a modified frame difference metric is used as shown in the equation below:

[fifteen]

where d _P = | Y _C -Y _P | and d _N = | Y _C -Y _P | - this is the difference of the brightness component between the current frame and the previous frame and the difference of the brightness component between the current frame and the next frame, respectively, Δ represents a constant that can be determined in the usual way, since it can be implementation dependent, and α is a weighting variable with a value from 0 to 1.

B. Bandwidth table generation

Preprocessor 226 (FIG. 6) may also be configured to form a bandwidth table that can be used to encode multimedia data. In some aspects, content classification module 712 in encoder 228 (FIG. 7) instead generates a bandwidth table.

The visual quality of human V can be a function of the coding complexity of C and the assigned bits of B (also referred to as bandwidth). 15 is a graph illustrating this relationship. It should be noted that the coding complexity index C considers spatial and temporal frequencies from the point of view of the user's vision. For distortions that are more sensitive to human eyes, the value of complexity is correspondingly higher. In a typical embodiment, it can be assumed that V monotonically decreases in C and monotonically increases in B.

In order to achieve constant visual quality, the bandwidth (B _i ) is assigned to the i-th object (frame or MB), which must be encoded, which satisfies the criteria expressed in the following two equations:

B _i = B ( C _i , V ) [16]

[17]

In the two equations above, C _i is the coding complexity of the ith object, B is the total available bandwidth, and V is the achieved visual quality for the object. The visual quality of a person is difficult to formulate as an equation. Therefore, the above set of equations is not precisely defined. Nevertheless, if it is assumed that the three-dimensional model is continuous in all variables, the bandwidth ratio ( B _i / B ) can be interpreted as unchanged with the neighborhood of the pair (C, V). The bandwidth ratio β _i is set in the equation shown below:

β _i = B _i / B [eighteen]

The assignment of bits in this case can be specified as expressed in the following equations:

для (C _i ,V)

δ(C ₀ ,V ₀ ) β _i = β (C _i )

for ( C _i , V )

δ ( C ₀ , V ₀ ) [19]

where δ indicates "neighborhood".

The coding complexity is affected by the sensitivity of the human visual system, both spatial and temporal. The gyro visual model of a person is an example of a model that can be used to specify spatial complexity. This model takes into account local spatial frequency and ambient lighting. The resulting metric is called D _csat . At the pre-processing point in the process, whether the image should be internally encoded or externally encoded is unknown, and bandwidth ratios for both cases are formed. Bits are assigned according to the ratio between β _{INTRA of} various video objects. For internally encoded images, the bandwidth ratio is expressed in the following equation:

β _INTRA = β _0INTRA  log ₁₀ (1 + α  _INTRA Y ² D _csat ) [twenty]

In the above equation, Y is the component of the average brightness of the macroblock,

α _INTRA is the weight coefficient for the square of brightness and the term D _csat following it,

. Например, значение α_INTRA=4 достигает хорошего зрительного качества. Информация содержимого (к примеру, классификация содержимого) может быть использована для того, чтобы присваивать α_INTRA значение, которое соответствует требуемому уровню хорошего зрительного качества для конкретного содержимого видео. В одном примере, если видеосодержимое содержит широковещательную передачу новостей с "говорящей головой", уровень зрительного качества может быть задан ниже, поскольку изображение информации или отображаемая часть экрана может считаться меньшей важности, чем аудиочасть, и меньше бит может быть выделено для того, чтобы кодировать данные. В другом примере, если видеосодержимое содержит спортивный сюжет, информация содержимого может быть использована так, чтобы присваивать α_1NTRA значение, которое соответствует более высокому уровню зрительного качества, поскольку отображаемые изображения могут быть более важны зрителю, и, соответственно, большее число битов выделяется для того, чтобы кодировать данные.β _INTRA is the normalization coefficient to ensure that

. For example, α _INTRA = 4 achieves good visual quality. Content information (for example, content classification) can be used to assign α _{INTRA a} value that corresponds to the required level of good visual quality for a particular video content. In one example, if the video content contains broadcasting news with a talking head, the level of visual quality can be set lower, since the image of the information or the displayed part of the screen can be considered less important than the audio part, and fewer bits can be allocated in order to encode data. In another example, if the video content contains a sports plot, the content information can be used to assign an α _1NTRA value that corresponds to a higher level of visual quality, since the displayed images may be more important to the viewer, and accordingly, a larger number of bits are allocated to to encode data.

To understand this relationship, it should be noted that the bandwidth is assigned in a logarithmic manner in accordance with the coding complexity. The member of the brightness square Y reflects the fact that coefficients with a large modulus use more bits in order to encode. To represent the logarithm of getting negative values, the unit is added to the term in brackets. Logarithms with other bases can also be used.

The time complexity is determined by measuring the difference indicator of frames, which measures the difference between two consecutive frames taking into account the magnitude of the movement (for example, motion vectors) along with the difference indicator of frames, such as the sum of the absolute differences (SAD).

The assignment of bits for internally encoded images may take into account both spatial and temporal complexity. This is expressed below:

[21]

In the above equation, MV _P and MV _N are the forward and reverse motion vectors for the current MB (see FIG. 29). It should be noted that Y ² in the internally coded bandwidth formula is replaced by the sum of squared differences (SSD). To understand the role || MV _P + MV _N || ² in the above equation, we note the following characteristics of the human visual system: areas subject to smooth predictable movement (small || MV _P + MV _N || ² ) attract attention and can be tracked by the eye and, in a typical embodiment, cannot tolerate more distortions than stationary areas. However, areas subject to fast or unpredictable movement (large || MV _P + MV _N || ² ) cannot be tracked and can admit significant quantization. The experiments showed that α _INTER = 1, γ = 0.001 achieves good visual quality.

C. Adaptive GOP Partitioning

In another illustrative example of processing that can be performed by preprocessor 226, the GOP sectioning module 612 of FIG. 6 can also adaptively change the structure of a group of images encoded together and is described with reference to an example using MPEG2. Some older video compression standards (such as MPEG2) do not require the GOP to have a regular structure, although it can be imposed. The MPEG2 sequence always starts with an I-frame, i.e. frame encoded without reference to previous images. The MPEG2 GOP format is usually pre-set in the encoder by fixing diversity in the GOP P- or prediction images that follow the I-frame. P frames are images that are partially predicted from previous I or P images. Frames between the initial I-frame and subsequent P-frames are encoded as B-frames. A B-frame (B means bidirectional) can use the previous and next I- or P-images separately or simultaneously as a link. The number of bits required to encode an I-frame is on average greater than the number of bits required to encode a P-frame; likewise, the number of bits required to encode a P-frame is on average greater than the number of bits required for a B-frame. A skipped frame, if used, should not require bits for its presentation.

The concept underlying the use of P- and B-frames, and in newer compression algorithms, frame skipping in order to reduce the data rate required to present the video is the elimination of temporary redundancy. When temporary redundancy is high, i.e. there is a slight change from image to image, the use of P-, B- or skipped images effectively represents the video stream, since I- or P-images decoded earlier are used hereinafter as links in order to decode other P- or B-images.

Adaptive GOP partitioning is based on the adaptive use of this concept. Differences between frames are quantized, and the decision to represent the image by I, P, B or a skipped frame is automatically taken after the corresponding tests for the quantized differences are performed. An adaptive structure has advantages not available in a fixed GOP structure. A fixed structure should ignore the possibility that small changes have occurred in the content; an adaptive procedure should represent much more B-frames that must be inserted between each I- and P- or two P-frames, thereby reducing the number of bits required to adaptively represent a sequence of frames. On the contrary, when the change in the video content is significant, the efficiency of the P-frames is significantly reduced, since the difference between the predicted and reference frames is too large. Under these conditions, matching objects may fall out of the motion search areas, or the similarity between matching objects is reduced due to distortion caused by changes in camera angle. At this point, P-frames or I- and its adjacent P-frame should be selected so that they are closer to each other, and fewer B-frames should be inserted. A fixed GOP may not perform this adjustment.

In the system disclosed here, these conditions are automatically measured. The GOP structure is flexible and tailored to accommodate these changes in content. The system evaluates the difference indicator of frames, which can be considered as a measure of the distance between frames, with the same additive methods of distance. In the concept, under the condition of frames F ₁ , F ₂ and F ₃ having an interframe distance d ₁₂ and d ₂₃ , the distance between F ₁ and F ₃ is taken to be at least d ₁₂ + d ₂₃ . Frame assignments are made on the basis of this distance-like indicator.

The GOP partitioning module works by assigning image types to frames as they are received. The image type indicates the prediction method that may be required when encoding each block.

I-images are encoded without reference to other images. Because they are autonomous, they provide access points in the data stream where decoding can begin. Encoding type I is assigned to a frame if the distance to the previous frame exceeds the scene change threshold.

P images can use previous I or P images for motion compensation prediction. They use blocks in previous fields or frames that can be offset from the predicted block as the basis for encoding. After the reference frame is subtracted from the block in question, the residual block is encoded, typically using a discrete cosine transform, to eliminate spatial redundancy. Encoding types P are assigned to a frame if the distance between it and the last frame assigned to be a P-frame exceeds a second threshold, which is typically less than the first.

B frame images can use the previous and next P or I images to compensate for motion, as described below. The block in the B-image can be predicted directly, back, or bi-directionally; or it can be internally encoded without reference to other frames. In H.264, a reference block can be a linear combination of a maximum of 32 blocks out of a maximum of so many frames. If a frame cannot be assigned to be I- or P-type, it is assigned to be B-type if the distance from it to the intermediate predecessor is greater than the third threshold, which is typically less than the second threshold.

If a frame cannot be assigned to become an encoded B-frame, it is assigned a frame skip state. This frame may be skipped because it is practically a copy of the previous frame.

The evaluation of the metric, which quantifies the distance between adjacent frames in the display order, is the first part of this processing that is performed. This indicator is the distance that refers to the above; with it, each frame can be evaluated for the proper type. Thus, the spacing between I and adjacent P or two consecutive P frames can be variable. The distance calculation starts by processing video frames with a block-based motion compensation module, the block being the basic unit of video compression, usually consisting of 16x16 pixels, although other block sizes are possible, for example, 8x4, 4x4 and 8x16. For frames that make up two fields with interlaced eliminated, motion compensation can be performed based on the field, and the search for support blocks is performed in the fields rather than in the frames. For a block in the first field of the current frame, a direct reference block is found in the fields of the frame that follows it; likewise, an inverse reference frame is detected in fields that immediately precede the current field. Current blocks are collected in a compressed field. The process continues with the second field of the frame. Two compensated fields are combined so as to form a direct and inverse compensated frame.

For frames created by converting 606 from a video format to a movie, the search for reference blocks is performed only on the basis of frames, since only the restored movie frames are present. Two reference blocks and two differences, forward and reverse, are also found, leading also to the forward and reverse compensated frames. So, the motion compensation module generates motion vectors and difference indicators for each block; but the block is part of the NTSC field in the case of output of the processed interlaced scanning module 605 and is part of the movie frame if the conversion output from the video format to the film is processed. Note that the differences in the indicator are estimated between the block in the field or frame under consideration and the block that coincides with it, either in the previous field or frame, or in the field or frame immediately following it, depending on whether the direct or reverse difference. Only luminance values fall into this calculation.

Thus, the motion compensation step forms two sets of differences.

They are between blocks of current brightness values and brightness values taken from frames that are directly in front and directly behind the current frame in time. The absolute value of each direct and each inverse difference is determined for each pixel, and each is separately summed over the entire frame. Both fields are included in two summations when NTSC fields with interlaced scans that contain a frame are processed. Thus, SAD _P and SAD _N , the summed absolute values of the forward and reverse differences, are found.

For each frame, the SAD ratio is calculated using the relationship:

[22]

where SAD _P and SAD _N are the summed absolute values of the direct and inverse differences, respectively. A small positive number is added to the numerator ε to prevent a division by zero error. A similar term ε is added to the denominator, decreasing sensitivity γ when SAD _P or SAD _{N is} close to zero.

In an alternative aspect, the difference may be SSD, the sum of the squared differences, and SAD, the sum of the absolute differences, or SATD, in which blocks of pixel values are converted by applying a discrete cosine transform to them before the differences in the block elements are taken. Amounts are estimated in the area of active video, although a smaller area can be used in other aspects.

A histogram of the brightness of each frame, as received (without motion compensation), is also calculated. The histogram works for the DC coefficient, i.e. coefficient (0,0) in the matrix of 16x16 coefficients, which is the result of applying a two-dimensional discrete cosine transform to a block of brightness values, if available. Equivalently, the average value of 256 brightness values in a 16x16 block can be used on a histogram. For images whose brightness depth is eight bits, the number of image elements is set to 16. The following indicator estimates the difference between the histograms:

[23]

In the above, N _Pi is the number of blocks from the previous frame in the i-th resolution element, N _Ci is the number of blocks from the current frame that belongs to the i-th resolution element, and N is the total number of blocks in the frame.

These intermediate results are collected in order to form a difference indicator of the current frame as:

[24]

where γ _C is the SAD ratio based on the current frame, and γ _P is the SAD ratio based on the previous frame. If the scene has smooth motion and its histogram of the brightness components remains almost unchanged, then D ~ 1. If the current frame shows a sharp change of scenes, then γ _C should be of great importance, and γ _P should be of small value. The ratio γ _c / γ _{p is} used instead of only γ _c so that the indicator is normalized to the level of context activity.

Fig. 42 illustrates a process for assigning compression types to frames. D, the current frame difference specified in Equation 19, is the basis for decisions made regarding frame assignments. As decision block 4202 indicates, if the frame in question is the first in the sequence, the decision path marked “YES” goes to block 4206, thereby declaring that the frame is an I-frame. The accumulated frame differences are set to zero in step 4208, and the process returns (in step 4210) to the initial step. If the frame in question is not the first frame in the sequence, the path marked “NO” goes from step 4202 where the decision is made, and in test step 4204, the current frame difference is tested against the scene change threshold. If the current frame difference exceeds this threshold, the decision path marked “YES” goes from step 4206, again leading to the assignment of an I-frame.

If the current frame difference is less than the scene change threshold, the NO path goes to step 4212, where the current frame difference is added to the accumulated frame difference. Continuing to go through the flowchart at decision block 4214, the accumulated frame difference is compared with a threshold t, which is generally less than a scene change threshold. If the accumulated frame difference exceeds t, control proceeds to step 4216, and the frame is assigned to be a P-frame; the accumulated frame difference is then reset to zero in step 4218. If the accumulated frame difference is less than t, control proceeds from step 4214 to step 4220. Here, the accumulated frame difference is compared with x, which is less than t. If the current frame difference is less than t, the frame is assigned as skipped in step 4222, and then the process returns; if the current frame difference is greater than t, the frame is assigned to be a B-frame, at 4226.

Coder

Referring again to FIG. 2, transcoder 200 includes an encoder 228 that receives processed metadata and raw video from preprocessor 226. Metadata may include any information originally received in source video 104 and any information calculated by preprocessor 226. Encoder 228 includes a first-pass encoder 230, a second-pass encoder 232, and a repeat encoder 234. Encoder 228 also receives input from transcoder control 231, which can provide information (e.g., metadata, device information error tolerance, content information, encoded bit rate information, base level and improvement level balance information and quantization information) from second pass encoder 232 to first pass encoder 230, repeat encoder 234, and preprocessor 226. Encoder 228 encodes the received video from using content information received from the preprocessor 226 and / or content information that is generated by the encoder 228 itself, for example, by the content classification module 712 (FIG. 7).

FIG. 7 illustrates a block diagram of functional modules that may be included in an exemplary two-pass encoder that may be used for the encoder 228 illustrated in FIG. 7, although FIG. 7 and the description herein do not necessarily allow all functionality that may be included in the encoder. Accordingly, certain aspects of the functional modules are described further after the following description of the coding of the base layer and the enhancement layer.

Basic and Enhancement Coding

Encoder 228 may be a scalable SNR encoder that can encode raw video and metadata from preprocessor 226 to a first group of encoded data, also referred to herein as a base layer, and one or more additional groups of encoded data, also referred to in this document as layers improvements. The encoding algorithm generates base level and enhancement level coefficients that, when decoded, can be combined in the decoder when both levels are available for decoding. When both layers are unavailable, basic layer coding allows them to be decoded as one layer.

One aspect of such a layered coding process is described in reference to FIG. At 321, an I-frame is encoded with fully intra-coded macroblocks (intra-coded MB). In H.264, intra-coded MBs in I frames are encoded with fully exploited spatial prediction, which provides a significant amount of coding gain. There are two sub-modes: internal 4x4 and internal 16x16. If the base layer should take advantage of the coding enhancement provided by spatial prediction, then the base layer must be encoded and decoded before encoding and decoding the enhancement layer. Two-pass encoding and decoding of I-frames is used. At the basic level, the quantization parameter of the basic level QP _b provides the transform coefficients with a rough quantization step size. The pixel-by-pixel difference between the original frame and the reconstructed base-level frame is encoded at the enhancement level. The enhancement level uses the quantization parameter QP _e , which provides a more accurate quantization step size. An encoding means, such as the encoder 228 of FIG. 2, may perform encoding at 321.

At 323, the encoder encodes the base layer data and enhancement layer data for the P and / or B frames in the processed GOP. An encoding means, such as encoder 228, may perform encoding in step 323. In step 325, the encoding process checks to see if there are more P or B frames to encode. An encoding means, such as a scalable SNR encoder 228, may perform step 325. If there are still P or B frames, step 323 is repeated until all frames in the GOP are complete encoding. P and B frames are composed of externally encoded macroblocks (externally encoded MB), although there may be internally encoded MB in P and B frames, as described below.

In order for the decoder to distinguish between data of a base level and an enhancement level, encoder 228 encodes overhead information, step 327. Types of overhead information include, for example, data identifying the number of levels, data identifying the level as a base level, data identifying a level as an improvement level , data identifying the relationship between the levels (for example, level 2 is the improvement level for basic level 1 or level 3 is the improvement level for improvement level 2), or data identifying the level of Shade as the ultimate level of improvement in the line of enhancement layers. Service information may be contained in the headers associated with the data of the basic and / or level of improvement to which it relates, or may be contained in separate data messages. An encoding means, such as the encoder 228 of FIG. 2, may perform the process in step 327.

In order to have single-level coding, the coefficients of two levels can be combined before inverse quantization. Therefore, the coefficients of the two levels must be generated interactively, otherwise it can introduce a significant amount of overhead information. One reason for increasing overhead is that base level coding and enhancement level coding may use various temporary references. An algorithm is required in order to generate the coefficients of the base level and the level of improvement, which can be combined in the decoder before dequantization, when both levels are available. At the same time, the algorithm should provide an acceptable base-level video when the enhancement level is not available, or the decoder decides to decode the enhancement level for reasons such as, for example, energy saving. The details of an illustrative example of such a process are further described below in the context of a brief explanation of standard predictive coding immediately below.

P frames (or externally coded sections) may use temporal redundancy between the area in the current image and the most matching prediction area in the reference image. The location of the most matching prediction zone in the reference image can be encoded in a motion vector. The difference between the current zone and the most matching prediction reference zone is known as residual error (or prediction error).

32 is an illustration of an example P-frame compilation process, for example, in MPEG-4. Process 331 is a more detailed illustration of an exemplary process that may be performed at block 323 of FIG. 31. Process 331 includes a current image 333 composed of 5x5 macroblocks, with the number of macroblocks in this example being arbitrary. The macroblock is composed of a maximum of 16x16 pixels. Pixels can be specified as an 8-bit value of the luminance component (Y) and two 8-bit values of the color component (Cr and Cb). In MPEG, the Y, Cr, and Cb components can be saved in a 4: 2: 0 format, with the Cr and Cb components downsampled by 2 in the X and Y directions. Therefore, each macroblock must consist of 256 Y components, 64 Cr components and 64 components of Cb. The macroblock 335 of the current image 333 is predicted from the reference image 337 at a point in time different from the current image 333. In the reference image 337, a search is performed to find the best match macroblock 339 that is closest in relation to the values of Y, Cr and Cb to the current encoded macroblock 335. The placement of the best-matched macroblock 339 in the reference image 337 is encoded in a motion vector 341. The reference image 337 may be an I-frame or a P-frame, which the decoder must have restored before compiling the current image 333. The best matching macroblock 339 is subtracted from the current macroblock 335 (the difference for each of the components Y, Cr and Cb is calculated), leading to residual error 343. Residual error 343 is encoded using a two-dimensional discrete cosine transform (DCT) 345 and then quantized 347. Quantization 347 can be performed in order to provide spatial compression, for example, by means of Ia smaller number of bits the high-frequency components, while assigning more bits low frequency components. The quantized residual error coefficients 343 along with the identification information of the motion vector 341 and the reference image 333 are encoded information representing the current macroblock 335. The encoded information may be stored in a memory for future use or processed, for example, for error correction or image enhancement purposes, or transmitted over the network 349.

The encoded quantized residual error coefficients 343 along with the encoded motion vector 341 can be used to reconstruct the current macroblock 335 in the encoder for use as part of a reference frame for subsequent motion estimation and compensation. The encoder can simulate decoder procedures for a given P-frame recovery. Modeling in the decoder leads to the fact that the encoder and decoder work with the same reference image. The recovery process performed either in the encoder for subsequent external encoding or in the decoder is presented here. P-frame recovery can be started after the reference frame (or part of the image or frame that is indicated by reference) is restored. The encoded quantized coefficients are dequantized 351, and then the two-dimensional inverse DCT, or IDCT, 353 is performed, resulting in a decoded or reconstructed residual error 355. The encoded motion vector 341 is decoded and used to find the already reconstructed most matching macroblock 357 in the already reconstructed reference image 337. The restored residual error 355 is then added to the restored most matching macroblock 357 to form the restored macroblock 359. The restored m the acroblock 359 may be stored in a storage device, displayed independently or in an image with other restored macroblocks, or processed further to enhance images.

B-frames (or any section encoded with bidirectional prediction) may use temporal redundancy between the area in the current image and the most matching prediction zone in the previous zone, the most matching prediction zone in the subsequent zone. The next most matching prediction zone and the previous most matching prediction zone are combined to form a bidirectional forecast zone. The difference between the current image area and the most matching prediction reference area is known as residual error (or prediction error). The locations of the most coincident prediction zone in the subsequent reference image of the most coincident prediction zone in the previous reference image can be encoded in two motion vectors.

FIG. 33 illustrates an example encoder process for encoding coefficients of a base layer and enhancement layer that can be performed by encoder 228. The base layer and enhancement layer are encoded to provide a scaled SNR bit stream. FIG. 33 illustrates an example for encoding residual error coefficients between MBs, as, for example, should be performed in step 323 of FIG. 31. However, similar methods can be used to also encode coefficients within MB. An encoding means, such as the encoder component 228 of FIG. 2, may perform the process illustrated in FIG. 33 and step 323 in FIG. 32. The source (which should be encoded) video data 406 (video data contains information of luminance and color components in this example) is input into the loop 302 of the most matching macroblock of the base layer and loop 365 of the most matching macroblock of the enhancement level. The purpose of both loops 363 and 365 is to minimize the residual error that is calculated in adders 367 and 369, respectively. Circuits 363 and 365 may be implemented in parallel, as shown, or sequentially. Loops 363 and 365 include logic to search buffers 371 and 373, respectively, which contain reference frames to identify the most matching macroblock, which minimizes residual error between the most matching macroblock and the original data 361 (buffers 371 and 373 may be one and the same buffer). The residual errors of loop 363 and loop 365 are different because the base layer loop 363 generally uses a coarser quantization step size (higher QP value) than the enhancement layer loop 365. Transformation blocks 375 and 377 convert the residual errors of each loop.

The converted coefficients are then parsed into coefficients of the base level and the level of improvement in the selection module 379. The parsing of the selection module 379 may take several forms, as described below. One common feature of parsing techniques is that the improvement level coefficient, C ' _enh , is calculated so that it is a differential refinement to the C' _base level coefficient . The calculation of the improvement level, in order to be an refinement to the base level, enables the decoder to decode the base level coefficient independently and have an acceptable image representation, or combine the coefficients of the base level and improvement level and have an updated image representation. The coefficients selected by selection module 379 are then quantized by quantizers 381 and 383. The quantized coefficients C ' _base and C' _enh (calculated by quantizers 381 and 383, respectively) can be stored in memory or transmitted over the network to a decoder.

To correlate macroblock recovery in the decoder, the dequantizer 385 decantages the residual error coefficients of the base level. The dequantized residual error coefficients are inverted 387 and added 389 to the most matching macroblock found in buffer 371, resulting in a restored macroblock that matches what needs to be restored in the decoder. Quantizer 383, dequantizer 391, inverter 393, adder 397, and buffer 373 perform similar calculations on the enhancement level circuit 365 to those performed on the base level circuit 363. In addition, adder 393 is used to combine the inverse quantized coefficients of the enhancement level and the base level used in reconstructing the enhancement level. The quantizer and dequantizer of the enhancement level generally use a more accurate quantizer step size (lower QP) than the base level.

FIGS. 34, 35, and 36 illustrate examples of processes of a base level coefficient and enhancement level coefficient selection module that can be used in the selection module 379 of FIG. 33. A selector, such as the encoder 228 of FIG. 2, can perform the processes illustrated in FIGS. 34, 35, and 35. Using FIG. 34 as an example, the transformed coefficients are disassembled into the coefficients of the base and improvement level, as shown in the following equations:

[25]

C ' _enh = C _enh -Q _b ^-1 (Q _b (C' _base )) [26]

where the min function can be either a mathematical minimum or the minimum value of two arguments. Equation 25 is illustrated as step 401, and equation 26 is illustrated as adder 510 in FIG. 34. In equation 26, Q _b means a base level quantizer 381, and Qb ^-1 means a base level quantizer 385. Equation 26 converts the coefficient of the improvement level into a differential refinement of the coefficient of the base level calculated using equation 25.

Fig. 35 is an illustration of yet another example of a base level coefficient and enhancement layer coefficient selection module 379. In this example, the equation (.) Contained in step 405 represents the following:

[27]

The adder 407 calculates the coefficient of the level of improvement, as shown in the following two equations:

C ' _enh = C _enh -Q _b ^-1 (Q _b (C' _base )) [28]

where C _base is given by equation 27.

Fig. 36 is an illustration of yet another example of a base level coefficient and enhancement layer coefficient selection module 379. In this example, the base level coefficient remains unchanged, and the improvement level is the difference between the quantized / dequantized base level coefficient and the original coefficient of the improvement level.

In addition to the residual error rates of the base layer and the enhancement layer, the decoder needs information that identifies how MBs are encoded. An encoding means, such as the encoder component 228 of FIG. 2, may encode overhead information, which may include a table of internally encoded and externally encoded parts, such as, for example, an MB table in which macroblocks (or submacroblocks) are identified as internally encoded or externally encoded (also identifying what type of external encoding, including, for example, forward, reverse, or bidirectional), and which externally encoded parts of the frame (s) are referenced. In an exemplary aspect, the MB table and base level coefficients are encoded at the base level, and enhancement level coefficients are encoded at the enhancement level.

P-frames and B-frames may contain intra-coded MB as well as inter-MB. It is common for hybrid video encoders to use bit-rate-distortion (RD) optimization to decide to encode specific macroblocks in P- or B-frames as internally encoded MBs. In order to have one-level coding in which the intra-coded MBs are independent of the inter-MB enhancement level, all adjacent inter-MBs are not used for spatial prediction of the intra-coded base layer MBs. In order to keep the computational complexity unchanged for decoding the enhancement layer, refinement at the enhancement layer may be omitted for internally encoded MBs in the P- or B-frame of the base layer.

Internally encoded MBs in P- or B-frames require much more bits than between MBs. For this reason, internally encoded MBs in P- or B-frames can only be encoded with basic layer quality at a higher QP. This leads to a certain deterioration in video quality, but this degradation should be imperceptible if it is eliminated in a subsequent frame using the inter-MB coefficients in the base level and the improvement level, as described above. Two reasons make this deterioration invisible. The first is a sign of the human visual system (HVS), and the second is that inter-MBs improve internal MBs. With objects that change position from the first frame to the second frame, some pixels in the first frame are not visible in the second frame (information that should be hidden), and some pixels in the second frame are visible for the first time (open information). The eyes of a person are insensitive to open and hidden visual information. Thus, for undisclosed information, even if it is encoded with lower quality, the eyes may not report the difference. If the same information remains in the next P-frame, there is a high probability that the next P-frame at the improvement level can improve it, since the improvement level has a lower QP.

Another standard technique that introduces intra-coded MBs into P- or B-frames is known as internal update. In this case, some MBs are encoded as internally coded MBs, even if standard R-D optimization

should dictate that they must be externally encoded MB. These internally encoded MBs contained in the base layer can be encoded using QP _b or QP _e . If QP _{e is} used for the baseline, then refinement is not required at the enhancement level. If QP _{b is} used for the baseline, then refinement may be required; otherwise, quality degradation will be noticeable at the improvement level. Since external coding is more efficient than internal coding with respect to coding efficiency, these refinements at the enhancement level are internally encoded. Thus, base level coefficients are not used for the improvement level. Consequently, quality improves at the level of improvement without introducing new operations.

B-frames are typically used at the enhancement level due to the high compression quality they offer. However, B frames may not have reference internally encoded P frame MBs. If the pixels of the B-frame must be encoded with the quality of the enhancement level, this may require too many bits due to the lower quality of the internally encoded MB of the P-frame, as described above. By taking advantage of the quality benefits of HVS, as described above, MB B-frames can be encoded with lower quality by reference to lower quality internally encoded MB P-frames.

Another limiting case of internally encoded MBs in P- or B-frames is when all MBs in a P- or B-frame are internally encoded due to the presence of scene changes in the encoded video. In this case, the entire frame can be encoded with the quality of the basic level and without clarification at the level of improvement. If the scene change occurs in a B-frame and provided that B-frames are encoded only at the enhancement level, then the B-frame can be encoded with the quality of the basic level or simply discarded. If the scene change occurs in a P-frame, changes may not be required, but the P-frame may be dropped or encoded with basic quality. Scalable level coding is further described in the U.S. Patent Application, pending [Attorney No. 050078], entitled "Scalable Video Coding With Two Layer Encoding And Single Layer Decoding" and owned by its assignee, which is incorporated in its entirety by reference in this document .

Part of the first pass of the encoder

FIG. 7 shows an illustrative example of the encoder 228 of FIG. 2. The steps shown illustrate various processing in an encoder that may be included in encoder 228. In this example, encoder 228 includes a first pass part 706 above a dividing line 704 and a second pass part 706 (including the functionality of a second pass encoder 232 and a repeat encoder 234 in FIG. .2) under line 704.

Encoder 228 receives metadata and raw video from preprocessor 226. Metadata may include any metadata received or calculated by preprocessor 226, including metadata associated with the video content information. The first pass portion 702 of the encoder 228 illustrates exemplary processes that may be included in the first pass encoding 702, which is described below with respect to functionality. Specialists in the art should know that this functionality can be implemented in various forms (for example, hardware, software, firmware, or a combination of the above).

7 illustrates an adaptive internal update (AIR) module. The AIR module 710 provides input to an I-frame instance module 708, which creates an I-frame instance based on metadata. The first pass portion 702 may also include a content classification module 712 configured to receive metadata and video and determine content information associated with the video. Content information may be provided to a bit rate control bit assignment module 714, which also receives metadata and video. The control bit assignment unit 714 determines the bit assignment control information and provides it to the mode determination unit 715. Content and video information may be provided to an internal model (distortion) module 716, which provides internal encoding distortion information to a mode determination module 715 and a transmission rate-distortion ratio scalability module 718 for a base and enhancement level. Video and metadata are provided to motion (distortion) estimation module 720, which provides external coding distortion information to transmission rate-distortion scalability module 718 for a base layer and an enhancement layer. The transmission rate-distortion ratio scalability module 718 for the base level and enhancement level determines the transmission rate-distortion ratio scalability information using the distortion estimates from the motion estimation module 720 and the internal model distortion module 716, which is provided to the mode determination module 715. The mode determination unit 715 also receives input from the macroblock / MB series sequencing unit 722. The macroblock / MB series sequencing module 722 receives input from the error tolerance module 740 (shown in second pass part 706) and provides information on combining the independently encoded video parts (macroblock series) with access unit boundaries for error tolerance to the mode determination module 715 . The mode determination module 715 determines the encoding mode information based on the input data and provides an “optimal” encoding mode to the second pass part 706. A further illustrative explanation of some examples of such coding in part 702 of the first pass is given below.

As indicated above, content classification module 712 receives metadata and raw video provided by preprocessor 226. In some examples, preprocessor 226 calculates content information from multimedia data and provides content information to content classification module 712 (e.g., metadata) that can use content information in order to determine the classification of content for multimedia data. In some other aspects, the content classification module 712 is configured to determine various content information from the multimedia data, and can also be configured to determine the classification of the content.

The content classification module 712 may be configured to determine a different classification of content for a video having different types of content. Different classification of content can lead to different parameters used in aspects of encoding multimedia data, for example, determining the bit rate (for example, bit assignment) for determining quantization parameters, motion estimation, scalability, error tolerance, maintaining optimal quality of multimedia data in channels , as well as for fast channel switching schemes (for example, periodic activation of I-frames to provide fast channel switching). According to one example, encoder 228 is configured to determine an optimization of the bit rate-distortion (R-D) ratio and bit rate assignment based on content classification. The definition of content classification enables multimedia data to be compressible for a given quality level according to the required bit rate based on the classification of the content. In addition, by classifying the contents of the multimedia data (for example, determining the classification of the contents based on the human visual system), the resulting perceptual quality of the transmitted multimedia data on the display of the receiving device depends on the video content.

As an example of the procedure that a content classification module 712 goes through to classify content, FIG. 9 illustrates a process 900 illustrating an example process by which a content classification module 712 can operate. As shown, the process 900 begins at input block 902, where the content classification module 712 receives raw multimedia data and metadata. Next, the process 900 proceeds to step 904, where the content classification module 712 determines spatial information and temporal information of the multimedia data. In some aspects, spatial and temporal information is determined by spatial and temporal masking (e.g., filtering). Spatial and temporal information can be determined based on metadata, which includes scene change data and smoothing motion vectors (MV). Process 900 then proceeds to step 912, which performs estimates of spatial complexity, temporal complexity, and sensitivity. The process 900 then proceeds to step 916, in which the content of the multimedia data is classified based on the results of the determined spatial, temporal and sensitivity data, at steps 904 and 912. In addition, at step 916, a specific bit rate-distortion (RD) curve can be selected and / or RD curve data can be updated. The process 900 then proceeds to an output step 918, the output may include a complexity-distortion table or a value indicating spatial or temporal activity (eg, content classification) and / or selected R-D curves. Referring again to FIG. 7, a content classification module 712 provides output to a bit rate control bit assigning module 714, an internal model (distortion) module 716, and also to an I-frame instance creating module 708 described above.

Content Information

The content classification module 712 may be configured to calculate a variety of content information from multimedia data, including a variety of content related metrics, including spatial complexity, time complexity, contrast ratio values, standard deviations, and frame difference metrics, further described below.

The content classification module 712 may be configured to determine the spatial complexity and time complexity of the multimedia data, as well as associatively associate a texture value with spatial complexity and a motion value with time complexity. The content classification module 712 receives pre-processed content information associated with the contents of the multimedia data encoded from the preprocessor 226, or alternatively, the preprocessor 226 may be configured to calculate the content information. As described above, the content information may include, for example, one or more D _csat values, contrast coefficient values, motion vectors (MV), and the sum of the absolute differences (SAD).

In general, multimedia data includes one or more sequences of images, or frames. Each frame can be divided into blocks of pixels for processing. Spatial complexity is an extensive term that, in general, describes a measure of the level of spatial detail in a frame. Scenes with predominantly flat, or unchanging, or slightly changing regions of the luminance and color component have low spatial complexity. Spatial complexity is associated with the texture of the video data. Spatial complexity, in this aspect, is based on a human visual sensitivity index called D _csat , which is calculated for each block as a function of local spatial frequency and ambient lighting. Those skilled in the art will know how to use patterns of spatial frequency and lighting characteristics and the contrast of visual images to take advantage of the human visual system. A number of sensitivity indicators are known to take advantage of the promising limitations of the human visual system and can be used with the method described in this document.

Temporal complexity is an extensive term that is used to generally describe a measure of the level of movement in multimedia data referenced between frames in a sequence of frames. Scenes (for example, sequences of frames of video data) with little or no movement have low temporal complexity. The time complexity can be calculated for each macroblock and can be based on the value of D _csat , motion vectors and the sum of the absolute pixel differences between one frame and another frame (for example, a reference frame).

The frame difference indicator provides a measure of the difference between two consecutive frames taking into account the motion (for example, a motion vector or MV)

along with the residual energy, presented as the sum of the absolute difference (SAD) between the predicted parameter and the current macroblock. The frame difference also provides a measure of bidirectional and unidirectional prediction efficiency.

One example of a frame difference indicator based on motion information received from a preprocessor potentially performing motion compensation interlacing with motion compensation is as follows. The interlacing elimination module performs a bi-directional motion estimation, and thus a bi-directional motion vector, and SAD information is available. The frame difference represented by SAD_MV for each macroblock can be extracted as follows:

SAD_MV = log ₁₀  [SAD * exp (-min (1, MV))] [29]

where MV = Square_root (MV _x ² + MV _y ² ), SAD = min (SAD _N , SAD _P ), where SAD _N is SAD calculated from the inverse reference frame, and SAD _P is SAD calculated from the direct reference frame .

Another approach for estimating the frame difference is described above with reference to equations 6-8. The SAD ratio (or contrast ratio) γ can be calculated as described above in Equation 6. The brightness histogram of each frame can also be determined, and the histogram difference λ is calculated using Equation 7. The frame difference indicator D can be calculated as shown in equation 8.

In one illustrative example, the contrast coefficient and the difference indicator of frames are used as follows in order to obtain a classification of video content that allows reliable prediction of features in a given video sequence. Although described herein as being executed in encoder 228, preprocessor 226 can also be configured to determine the classification of content (or other content information) and transmit the classification of content to encoder 228 via metadata. The process described in the example below classifies the contents into eight possible classes, similar to the classification obtained from analysis based on R-D curves. The classification process displays a value in the range from 0 to 1 for each superframe based on the complexity of the scene and the number of scene change events in that superframe. The content classification module in the preprocessor can perform the following steps (1) to (5) for each superframe to obtain a content classification index from the frame contrast and frame difference values.

1. Calculation of the average contrast of frames and deviations of the contrast of frames from the contrast values of macroblocks.

2. Normalization of the values of the contrast of frames and the difference of frames using the values obtained from the simulation, ie 40 and 5, respectively.

3. The calculation of the classification indicator of the content using, for example, a generalized equation:

CCMetric = CCW1 * I_Frame_Contrast_Mean + CCW2 * Frame_Difference_Mean-CCW3 * I_Contrast_Deviation ^ 2 * exp (CCW4 * Frame_Difference_Deviation 2) [thirty]

where CCW1, CCW2, CCW3 and CCW4 are weights. In this example, the values are chosen to be 0.2 for CCW1, 0.9 for CCW2, 0.1 for CCW3, and -0.00009 for CCW4.

4. Determining the number of scene changes in a superframe. In general, a superframe means a group of images or frames that can be displayed in a specific time period. Typically, the time period is one second. In some aspects, a superframe contains 30 frames (for a video, 30 frames per second). In other aspects, a superframe contains 24 frames (for video, 24 frames per second). Depending on the number of scene changes, one of the following cases is executed.

(a) No scene change. When there is no scene change in the superframe, the indicator completely depends only on the values of the frame difference, as shown in the following equation:

CCMetric = (CCW2 + (CCW1 / 2) * Frame_Difference_Mean- (CCW3- (CCW1 / 2)) * 1 * exp (-CCW4 * Frame_Difference_Deviation ^ 2) [31]

(b) One scene change. When there is a frame with one scene change observed in a superframe, the default equation should be used to calculate the metric, as shown below:

CCMetric = CCWl * I_Frame_Contrast_Mean + CCW2 * Frame_Difference_Mean-CCW3 * I_Contrast_Deviation ^ 2 *
exp (CCW4 * Frame_Difference_Deviation ^ 2) [32]

(a) Two scene changes. When it is observed that there are at most 2 scene changes in a given superframe, the last superframe is assigned a larger weight coefficient than the first, since the first in any case must be updated by the second quickly, as shown in the following equation:

CCMetric = 0.1 * I_Frame_Contrast_Mean1 + CCW1 * I_Frame_Contrast_Mean2 + (CCW2-0.1)
* Frame_Difference Mean-CCW3 * I_Contrast_Deviationl ^ 2 * I_Contrast_Deviation2 2 * exp (CCW4 * Frame_Difference_Deviation ^ 2) [33]

(d) Three or more scene changes. If more than 3 I-frames (say, N) are observed in a given superframe, the last I-frame is assigned a higher weight coefficient, and all other I-frames are assigned a weight coefficient of 0.05, as shown in the following equation:

CCMetric = 0.05 * I_Frame_Contrast_Mean _{(1 ... N-1)} + CCW1 * I_Frame_Contrast_Mean (N) + (CCW2- (0.05 * (N-1))) * Frame_Difference_Mean-CCW3 * I_Contrast_Deviation (N) ^ 2 * I_Contrast_Deviation _{(1 ... N-1)} ^ 2 * exp (CCW4 * Frame_Difference_Deviation ^ 2) [34]

5. Correction can be used for the indicator in the case of scenes with slight movement, when the average frame difference is less than 0.05. Offset (CCOFFSET) 0.33 must be added to CCMetric.

The content classification module 712 uses the D _{csat value} , motion vectors, and the sum of the absolute differences to determine a value indicating spatial complexity for the macroblock (or a specified amount of video data). The time complexity is determined by measuring the indicator of the difference in the frames (the difference between two consecutive frames, taking into account the volume of motion, with the motion vectors and the sum of the absolute differences between the frames).

In some aspects, the content classification module 712 may be configured to form a bandwidth table. For example, bandwidth table generation may be performed by content classification module 712 if preprocessor 226 does not generate bandwidth table.

Defining texture and motion values

For each macroblock in multimedia data, a content classification module 712 associatively associates a texture value with spatial complexity and a motion value with time complexity. The texture value refers to the glow values of the multimedia data, while a low texture value indicates small changes in the glow values of neighboring data pixels, and a high texture value indicates significant changes in the glow values of neighboring data pixels. After the texture and motion values are calculated, the content classification module 712 determines the classification of the content by reviewing the motion and texture information. The content classification module 712 associates the texture for the classified video data with a relative texture value, for example, a “low” texture, a “medium” texture, or a “high” texture, which generally indicates the complexity of the macroblock brightness values. In addition, the content classification module 712 associatively associates the motion value calculated for the classified video data with a relative motion value, for example, “low” motion, “medium” motion, or “high” motion, which generally indicates the volume of motion of the macroblocks. In alternative aspects, more or fewer categories for movement and texture may be used. Further, the content classification indicator is determined taking into account the associated values of texture and movement.

FIG. 8 illustrates an example of a classification diagram that illustrates how texture and motion values are associated with content classification. Those skilled in the art are familiar with a variety of ways to implement such a classification diagram, for example, a lookup table or a database. A classification chart is generated based on predefined estimates of the content of the video data. To determine the classification of the video data, the texture value “low”, “medium” or “high” (on the x-axis) is cross-referenced with the motion value “low”, “medium” or “high” (on the y-axis). The content classification indicated in the intersecting rectangle is assigned by the video data. For example, the value of the texture is “high” and the value of the motion is “average” leads to a classification of seven (7). FIG. 8 illustrates various combinations of relative texture and motion values that are associated with eight different classifications of content in this example. In some other aspects, more or fewer classifications may be used. A further description of an illustrative aspect of the classification of content is disclosed in U.S. Patent Application No. 11/373577, entitled "Content Classification For Multimedia Processing", filed March 10, 2006, assigned to the assignee of this application and thus explicitly contained herein link.

Bit Rate Control Assignment

As described herein, the classification of multimedia content can be used in coding algorithms in order to effectively improve bit control while maintaining a constant perceived video quality. For example, a classification metric can be used in algorithms to detect scene changes, control the assignment of the bit rate for encoding, and up-transform the frame rate (FRUC). Compressor / decompressor systems (codec) and digital signal processing algorithms are typically used when exchanging video data and can be configured to save bandwidth, but there is a trade-off between quality and bandwidth savings. The best codecs offer the most bandwidth savings while creating the least degradation in video quality.

In one illustrative example, the bit rate control bit assigning unit 714 uses content classification to determine the bit rate (for example, the number of bits assigned to encode multimedia data), and stores the bit rate in a memory for use by other processes and components of the encoder 228. The bit rate determined from the classification of video data helps to save bandwidth when providing multimedia data with a consistent level of quality. In one aspect, a different bit rate may be associated with each of eight different classifications of content, and then this bit rate is used to encode multimedia data. The net effect is that although different classifications of multimedia data are assigned a different number of bits for encoding, the perceived quality is similar or consistent when viewed on a display.

In general, multimedia data with a higher classification of content indicates a higher level of motion and / or texture, and is assigned more bits during encoding. Lower-classified multimedia data (indicating less texture and movement) is assigned fewer bits. For multimedia data with a specific classification of bits, the bit rate can be determined based on the selected target perceived quality level for viewing multimedia data. Determining the quality of multimedia data can be determined by viewing and organizing people multimedia data. In some alternative aspects, multimedia quality assessments can be performed by automated test systems using, for example, signal to noise ratio algorithms. In one aspect, a set of standard quality levels (e.g., five) and the corresponding bit rate required to achieve each specific quality level are predetermined for the multimedia data of each content classification. To define a set of quality levels, the multimedia data of a specific content classification can be evaluated by generating an average expert judgment (MOS), which provides a numerical indicator of the visually perceived quality of the multimedia data when it is encoded using a specific bit rate. MOS can be expressed as a single number in the range from 1 to 5, where 1 is the lowest perceived quality and 5 is the highest perceived quality. In other aspects, a MOS may have more than five or less than five quality levels, and various descriptions of each quality level may be used.

Determining the quality of multimedia data can be determined by viewing and organizing people multimedia data. In some alternative aspects, multimedia quality assessments can be performed by automated test systems using, for example, signal to noise ratio algorithms. In one aspect, a set of standard quality levels (e.g., five) and the corresponding bit rate required to achieve each specific quality level are predetermined for the multimedia data of each content classification.

Knowing the relationship between the level of visually perceived quality and the bit rate of the multimedia data of a certain classification of content can be determined by calling the target (for example, the desired) level of quality. The target quality level used to determine the bit rate may be pre-selected, selected by the user, selected by an automatic process or a semi-automatic process that requires input from the user or from another process, or selected dynamically by a device or coding system based on predefined criteria. The target quality level may be selected, for example, based on the type of encoding application or the type of client device that receives multimedia data.

In the illustrated example in Fig. 7, the bit rate control bit assignment module 714 receives both data from the content classification module 712 and metadata directly from the preprocessor 226. The bit rate control bit assignment module 714 is located in the first pass portion of the encoder 228, and the fine-tuning module 738 speed control is located in part 706 of the second pass. This aspect of the two-pass bit rate control is configured such that the first pass (bit rate control bit assignment module 714) performs context-adaptive bit assignment with look-ahead per superframe (for example, with a target of a long-term average bit rate of 256 kbit / s) and limits the peak speed, and the second pass (module 738 fine-tuning the control of the transmission speed) updates the results of the first pass for two-level scalability and adapts the speed. Speed control works at four levels: (1) GOP level - controls the distribution of bits of I-, P-, B- and F-frames so as to make them heterogeneous within the GOP; (2) superframe level - controls hard limits for the maximum superframe size; (3) frame level - controls the requirements for bits according to the spatial and temporal complexity of the frames of multimedia data, which is based on content information (for example, content classification); and (4) macroblock level - controls the assignment of bits of macroblocks based on spatial and temporal complexity tables, which are based on content information (e.g., content classification).

An exemplary flow chart of a transmission rate control module 714 is illustrated in FIG. 10. As shown in FIG. 10, the process 1000 begins at input step 1002. The baud rate control module 714 receives various input data, not all of which are necessarily illustrated in FIG. For example, the input information may include metadata from preprocessor 226, the target bit rate, the size of the encoder buffer (or, equivalently, the maximum delay time for controlling the bit rate), the initial delay of the bit rate control, and frame rate information. Additional input information may include input at the group of image (GOP) level, including, for example, the maximum superframe size, the length and distribution of P / B frames in the GOP (including scene change information), the required layout of the base layer and layer improvements, distortion index depending on complexity for images in the GOP for the next 30 frames. Other input information includes input data at the image level, including a table of distortions depending on complexity for the current image (received from the content classification module 712), quantization parameters (QP), and bit break for the last 30 frames (getting into the transmission window variable duration). Finally, the input information at the macroblock (MB) level includes, for example, the average absolute difference (MAD) of the co-located macroblocks (MB) in the reference image and the coded block pattern (CBP) of the macroblocks after quantization (whether skipped or not) .

After entering the data in step 1002, the process 1000 proceeds to step 1004 to initialize the encoding of the bitstream. At the same time, buffer initialization 1006 is performed. The GOP is then initialized, as shown in step 1008, with the assignment of 1010 GOP bits being accepted as part of the initialization. After GOP initialization, the flow proceeds to block 1012, where a series of macroblocks is initialized. This initialization includes updating the header bits, as shown by step 1014. After the initialization in steps 1004, 1008, and 1012 is completed, speed control (RC) for the base unit or macroblock (MB) is performed as shown by step 1016 As part of the determination of the macroblock speed control in step 1016, input is received via interfaces in encoder 228. This input may include assigning 1018 macroblock (MB) bits, updating quadratic model parameters 1020, and updating e parameters 1022 of the median absolute deviation from the median (MAD, stable variance estimate). Then, the process 1000 proceeds to step 1024 to execute operations after encoding one image 1024. This procedure includes receiving an update of the buffer parameters, as shown by step 1026. After this, the process 1000 proceeds to step 1028, where the transmission rate control module 714 outputs the QP quantization parameters for each macroblock MB to be used by the mode determination module 715, as shown in FIG. 7.

Motion estimation

Motion estimation module 720 receives input from metadata and raw video from preprocessor 226 and provides output, which may include block size, motion vector distortion metrics, and reference frame identifiers, to mode determination module 715. 11 illustrates an example operation of motion estimation module 720. As shown, process 1100 begins with input 1102. At the frame level, module 720 receives input of a reference frame identifier and motion vectors. At the macroblock level, input 1102 includes input pixels and pixels of reference frames. Process 1100 proceeds to step 1104, in which a color motion estimation (ME) and motion vector prediction are performed. To perform this process, various input data are received, including the MPEG-2 motion vectors and the motion component vectors of the luminance component MV 1106, smoothing 1108 motion vectors and non-causal motion vectors 1110. Then, the process 1100 proceeds to step 1112, in which an algorithm or methodology for searching motion vectors, for example, hexagonal or rhomboid search methods, is executed. The input to the process in step 1112 may include the sum of the absolute differences (SAD), the sum of the squares of the differences (SSD) and / or other indicators, as shown by step 1114. After the search for motion vectors is completed, the process 1100 proceeds to step 1116 completion on which completion processing is performed. After this, the process 100 ends at step 1118 output, which provides the output block size, motion vector (MV), distortion indicators and identifiers of reference frames.

R-D scalability for baseline and enhancement level

FIG. 13 illustrates an example flowchart of a scalability process 1300 that can be performed by an R-D scalability module 718. The process 1300 starts at an initial step 1302 and proceeds to step 1304, where the R-D scalability module 718 receives input from the motion estimation module 720 and performs a motion estimation. The motion estimation is based on the input of reference frames of the base layer, reference frames of the enhancement layer and the source frame to be encoded as indicated by step 1306. This information can be calculated by the GOP partitioning module 612 and transmitted to the RD scalability module 718 by for example, metadata. Process 1300 proceeds to block 1308 to determine the scalability information of the base layer data and the enhancement layer. The base level encoding is then performed as shown in step 1310, followed by the enhancement level encoding in step 1312. The enhancement level encoding can use the results of the base level coding for inter-layer prediction as input, as illustrated by step l314, thus, in time it is executed after encoding the base level. This is further described in the U.S. Patent Application, pending [Law No. 050078], entitled "Scalable Video Coding With Two Layer Encoding And Single Layer Decoding". After the encoding is completed, the process 1300 ends at step 1316.

Sorting macroblock / macroblock series

The first pass portion 702 also includes a macroblock / macroblock series sequencing module 722, which receives input from the error tolerance module 740 in the second pass portion and provides macroblock series alignment information to the mode determination module 715. A series of macroblocks are portions of independently decoded (entropy-encoded) encoded video data. Access Units (AUs) are coded video frames, each of which contains a set of NAL units, always containing exactly one primary encoded image. In addition to the primary encoded image, the access unit may also contain one or more excessively encoded images or other NAL units that do not contain a series of macroblocks or data segments of a series of macroblocks in an encoded image. Decoding an access unit always results in a decoded image.

Frames can be time division multiplexed blocks of physical layer packets (called TDM capsules) that provide the greatest time diversity. A superframe corresponds to one unit of time (for example, 1 second) and contains four frames. Combining the boundaries of a series of macroblocks and AUs with the boundaries of frames in the time domain leads to the most efficient separation and localization of damaged data. During deep attenuation, most of the adjacent data in the TDM capsule is affected by errors. Due to temporary spacing, the remaining TDM capsules are highly likely to be intact. Intact data can be used to recover and hide lost data from the affected TDM capsule. Similar logic applies to frequency domain multiplexing (FDM), where frequency diversity is achieved by dividing in frequency subcarriers that modulate data symbols. In addition, similar logic applies to spatial (through separation of transmitting and receiving devices in antennas) and other forms of diversity, often used in wireless networks.

In order to combine a series of macroblocks and AUs with frames, the creation of code blocks of an external code (FEC) and the encapsulation of the MAC layer must also be combined. 20 illustrates the organization of encoded video data or a video bitstream in a series of macroblocks and AUs. The encoded video may be composed in one or more bit streams, for example, in a base layer bit stream and an enhancement layer bit stream, where multi-level video coding is used.

The video bitstream contains the AU, as illustrated in FIG. 20 by frame 1 ′ 2005, frame 3 ′ 2010, and frame M ′ 2015. AUs contain a series of data macroblocks, as illustrated by a series of macroblocks 1 2020, a series of macroblocks 2 2025, and a series of macroblocks N 2030. Each beginning of a series of macroblocks is identified by means of an initial code and is provided in a network adaptation. In general, I-frames or internally encoded AUs are large, followed by P-frames or directly predicted frames, followed by B-frames. AU coding on several macroblock series imposes significant additional costs with respect to the coded bit rate, since spatial prediction in the macroblock series is limited, and macroblock headers also incur additional costs. Since the boundaries of the macroblock series are points of resynchronization, the restriction of adjacent physical layer packets to the macroblock series manages errors, because when the PLP is damaged, the error is limited to the macroblock series in the PLP, whereas if the PLP contained several macroblock series or parts of several macroblock series, the error should affect all macroblock series or parts of macroblock series in PLP.

Since I-frames are typically large, for example, of the order of tens of kbit / s, the additional overhead due to several series of macroblocks is not a significant part of the total size of the I-frame or the total bit rate. In addition, the presence of a larger number of macroblock series in an internally encoded AU provides more optimal and more frequent resynchronization and more efficient masking of spatial errors. In addition, I-frames carry the most important information in the video bitstream, since P- and B-frames are predicted from I-frames. I-frames also act as random access points for channel detection.

Referring now to FIG. 21, a careful combination of I-frames with frame boundaries and also a series of macroblocks with I AU with frame borders provides the most efficient error management, error protection (since if one series of macroblocks that belonged to frame 1 2105 is lost, series macroblocks that belong to frame 2110 have a high probability of being intact, since frame 2110 has a significant time separation from frame 1 2105, error recovery can be performed by re-synchronization and masks errors.

Since P-frames typically have a size of the order of several kbps, combining a series of macroblocks of a P-frame and an integer number of P-frames with frame boundaries provides error tolerance without undesired performance loss (for similar reasons as for I-frames) . Temporary error concealment can be used in such aspects. Alternatively, dispersing consecutive P frames so that they arrive in different frames provides added temporal diversity in P frames, which may be due to the fact that temporal masking is based on motion vectors and data from previously reconstructed I or P frames. B-frames can range from extremely small (hundreds of bits) to moderately large (several thousand bits). Therefore, combining an integer number of B frames with frame boundaries is required in order to achieve error tolerance without undesired loss of efficiency.

Mode Definition Module

12 illustrates some examples of the operation of the mode determination module 715. As shown, process 1200 begins at input step 1202. In one illustrative example, various information input to the mode determination module 715 includes a macroblock series type, Intra 4x4cost mode, Intra 16xl6cost mode, IntraUV 8x8cost mode, IntraY 16x16 mode, IntraUV mode, motion vector data (MVD), quantization parameters ( QP), SpPredMB4x4Y, SpPredMB16xl6Y, SpPredMB8x8U, SpPredMB8x8V, transmission-distortion flag, raw YMB pixels, raw UMB pixels, and raw VMB pixels. Next, process 1200 proceeds to encoding initialization step 1204, which may be initiated by an input signal or an interface directing the encoder initialization, as indicated by step 1206. Initialization may include setting allowed modes (including skipping, direction), setting mode weighting factors (if necessary, by default, all weights are equal for all modes) and the setting of buffers. After initialization, process 1200 proceeds to step 1208, where basic processing for determining the mode is performed, including: calculating macroblock (MB) mode costs for each allowed mode, weighting the costs of each MB macroblock mode

using the weight coefficient and the choice of the minimum cost mode of the MB mode. The input involved in these operations includes motion estimation (eg, MVD and predictions) and spatial prediction (eg, internal costs and predictions), as shown by steps 1210 and 1212. Coding interacts with mode determination module 715 entropy at step 1214, which, among other things, increases the compression ratio. Process 1200 proceeds to step 1216, where the buffers are updated to transmit information to part 706 of the second pass of the encoder. Finally, process 1200 proceeds to step 1218 where the “optimal” coding mode may be communicated to part 706 of the second pass of the encoder.

Part of the second pass of the encoder

Referring again to FIG. 7, a second pass portion 706 of encoder 228 includes a second pass encoder module 232 for performing a second encoding pass. The second pass encoder 232 receives the output from the mode determination module 715. The second pass encoder 232 includes an MC / transform quantization module 726 and a zigzag encoder (ZZ) / entropy encoder 728. The results of the second pass encoder 232 are output to a scaling module 730 and a bit packet packing module 731, which outputs the encoded base and enhancement level for transmission by transcoder 200 via a synchronization layer 240 (illustrated in FIG. 2). As shown in FIG. 2, it should be noted that the baseline and enhancement level from the second pass encoder 232 and the re-encoder 234 are assembled by the synchronization layer 240 into packetized PES 242 including the base level and enhancement levels, data PES 244 (for example, CC and other text data) and PES 246 audio. It should be noted that the audio decoder 236 receives the decoded audio information 218 and, in turn, encodes the information and outputs the information 238 to the clock level 240.

Re encoder

Referring again to FIG. 7, the second pass portion of the encoder 706 also includes a repeat encoder 234, which corresponds to a repeat encoder 234 in FIG. 2. The repeat encoder 234 also receives the output of the first pass part 702, and includes the MC / quantization transform 726 and ZZ / entropy encoding 728 parts. Additionally, the scalability module 730 outputs to the re-encoder 234. The re-encoder 234 outputs the resulting base and enhancement level from the re-encoding to the packet packet packing module 731 for transmission to the synchronization module (for example, the synchronization level 240 shown in FIG. 2). The example encoder 228 of FIG. 7 also includes a bit rate control fine-tuning module 738 that provides feedback on packing the bitstream into an MC / quantization module 234 with conversion in second pass encoder 232 and a ZZ / entropy re-encoding module 736 encoder 234, to help fine-tune the encoding of the second pass (for example, to increase compression efficiency).

Error tolerance module

An example of encoder 228, illustrated in FIG. 7, also includes an error tolerance module 740 in a second pass portion 706. The error tolerance module 740 communicates with the bitstream packing module 731 and the macroblock series / MB sequencer 722. The error tolerance module 740 receives metadata from the preprocessor 228 and selects an error tolerance scheme, for example, combining a series of macroblocks and access units with frame boundaries, a prediction hierarchy, and adaptive internal update. The selection of the error tolerance scheme may be based on information received in the metadata, or from information transmitted to the error tolerance module, from the bitstream packing module 731 and the macroblock / MB series sequencing module 722. The error tolerance module 740 provides information to the macroblock / macroblock (MB) series sequencer in the first pass part 702 to implement the selected error tolerance processes. Video transmission in fault-tolerant environments can use error-resistant strategies and algorithms that can lead to the presentation of clearer and less error-filled data to the viewer. The following description of error tolerance can be applied to any single or combination of an existing or future use case, transport and physical layer and other technologies. Effective error tolerance algorithms integrate understanding of the properties of error susceptibility and error protection capabilities for OSI layers in combination with the required properties of a communication system such as low latency and high throughput. Error tolerance processing may be based on the information content of the multimedia data, for example, the classification of the content of the multimedia data. One of the main advantages is the recoverability after attenuation and multipath channel errors. The error tolerance approaches described below relate, in particular, to the processes that may be contained in the encoder 228 (for example, in particular, the error tolerance module 740 and the macroblock series / MB sequencer 722), and may be expanded, in general, before sharing data in fault-tolerant environments.

Error tolerance

For a prediction-based hybrid compression system, internally encoded frames are independently encoded without any temporal prediction. Externally coded frames can be predicted in time from previous frames (P-frames) and future frames (B-frames). An optimal predictor can be identified by a search process in a reference frame (one or more), and a distortion measurement, such as SAD, is used to determine the best match. The predicted coded region of the current frame may be a block of variable length and shape (16 × 16, 32 × 32, 8 × 4, etc.) or a group of pixels identified as an object by, for example, segmentation.

The temporal prediction typically stretches to many frames (for example, ten or tens of frames) and ends when the frame is encoded as an I-frame, and the GOP is typically specified by the frequency of I-frames. For maximum coding efficiency, the GOP is a scene, for example, the boundaries of the GOP are aligned with the boundaries of the scene, and scene change frames are encoded as I-frames. With little movement, the sequences contain a relatively static background, and the movement is generally limited to an object in the foreground. Examples of the contents of such low-motion sequences include news and weather forecasting programs, where more than 30% of the most viewed content is of this nature. In sequences with little movement, most of the zones are externally encoded, and the predicted frames are referenced back to the I-frame through intermediate predicted frames.

Referring to FIG. 22, an internally coded block 2205 in an I-frame is a predictor for an externally coded block 2210 in a coded PI frame (or AU). In this example, the block data zone is the stationary part of the background. Through consistent temporal prediction, the sensitivity of the internally encoded block 2205 to errors increases because it is a “good” predictor, which also implies that its “importance” is higher. Additionally, the internally coded block 2205, by virtue of this temporal prediction chain, called the prediction chain, remains longer on the display (during the scene in the example in the drawing).

The prediction hierarchy is defined as a block tree created on the basis of this level of importance, or a measure of persistence with a parent at the top (internally encoded block 2205) and children at the bottom. Note that the externally encoded block 2215 on P1 is at the second level of the hierarchy, etc. Leaves are blocks that complete the prediction chain.

A prediction hierarchy can be created for video sequences regardless of the type of content (for example, also music and sports, and not just news) and is applicable to the compression of prediction-based video (and data) in general (this applies to all inventions described in this application) . Once the prediction hierarchy is established, error tolerance algorithms such as adaptive internal updating, described below, can be applied more efficiently. The importance measurement can be based on the recoverability of a given block after errors, for example, through masking operations and the use of adaptive internal updates in order to increase the error resistance of the encoded bit stream. The importance measurement score may be based on the number of times a block is used as a predictor, also referred to as a measure of persistence. A retention metric is also used to improve coding efficiency by stopping the propagation of prediction errors. Retention also increases bit assignment for blocks of higher importance.

Adaptive Internal Update

Adaptive internal updating is used in the error tolerance technique, which can be based on the content information of multimedia data. During the internal upgrade process, some MBs are coded internally, even if the standard RD optimization should dictate that they must be externally coded MBs. AIR uses motion-weighted internal updating to introduce internally coded MBs into P- or B-frames. These internally encoded MBs contained in the base layer can be encoded using QP _b or QP _e . If QP _{e is} used for the base level, refinement at the enhancement level is not required. If QP _{b is} used for the baseline, then refinement may be required; otherwise, at the improvement level, degradation will be noticeable. Since external coding is more efficient than internal coding with respect to coding efficiency, these refinements at the enhancement level are internally encoded. Thus, baseline coefficients are not used for the improvement level, and the quality is improved at the improvement level without introducing new operations.

In some aspects, adaptive internal updating may be based on multimedia content information (eg, content classification) instead of or in addition to the basis of weighted movement. For example, if the content classification is relatively high (for example, scenes having high spatial and temporal complexity), adaptive internal updating may introduce a relatively larger number of internally encoded MBs in P- or B-frames. Alternatively, if the content classification is relatively low (indicating a less dynamic scene with low spatial and / or temporal complexity), then adaptive intra-update may introduce fewer internally encoded MBs in P- and B-frames. These indicators and methods for improving error tolerance can be applied not only in the context of wireless multimedia communications, but also to data compression and multimedia processing, in general (for example, when rendering graphics).

Channel Switch Frame

The Channel Switching Frame (CSF) defined herein is an extensive term for describing a random access frame inserted at an appropriate location in a broadcast stream to quickly detect a channel and thereby quickly change channels between streams during broadcast multiplex transmission. Channel switching frames also increase error tolerance because they provide redundant data that can be used if the primary frame is transmitted in error. I-frames or a progressive I-frame, such as a progressive decoder update frame, in H.264 typically acts as a random access point. However, frequent I-frames (or short GOPs, shorter than scene lengths) lead to a significant reduction in compression efficiency. Because internally coded blocks can be used for error tolerance, random access and error tolerance can be effectively combined through prediction efficiency to increase coding efficiency while increasing error tolerance.

The improvement of random access switching and error tolerance can be achieved in collaboration and can be based on content information, such as classification of content. For sequences with little movement, the prediction chains are long, and a significant portion of the information required to reconstruct a superframe or scene is contained in the I-frame that was at the beginning of the scene. Channel errors are often pulsating, and when attenuation occurs and both FEC and channel coding fail, there is a serious residual error that masking fails. This is especially serious for sequences with little movement (and therefore low bit rate), because the amount of encoded data is not large enough to provide good temporal diversity in the video bitstream, and since these are sequences with a high compression ratio, which each does bit suitable for recovery. Sequences with greater movement are more error-resistant due to the nature of the content - more new information in each frame increases the number of internally encoded blocks that are independently decoded and more error-resistant in nature. Adaptive internal updating based on the prediction hierarchy provides high performance for high motion sequences, and performance improvements are negligible for small motion sequences. Therefore, a channel switching frame containing most of the I-frame is a good source of diversity for sequences with little movement. When an error occurs in a superframe, decoding in a sequential frame begins with a CSF, which recovers the lost information due to prediction, and error tolerance is achieved.

In the case of sequences with a large movement, such as sequences having a relatively high classification of contents (for example, 6-8), the CSF may consist of blocks that are contained in the SF, i.e. those that are good predictors. All other CSF zones should not be encoded, since they are blocks that have short prediction chains, which implies that they terminate with internal blocks. Therefore, the CSF still serves to perform recovery of the lost information by predicting when an error occurs. CSFs for sequences with small motion are the same as the size of I-frames, and they can be encoded at a lower bit rate due to stronger quantization, while CSFs for sequences with large movement are much smaller than the corresponding I-frames.

Error tolerance based on the prediction hierarchy can work well with scalability and allows for highly efficient multi-level coding. Scalability in order to support hierarchical modulation in physical layer technologies may require partitioning of video bitstream data with specific bandwidth coefficients. These may not always be ideal coefficients for optimal scalability (for example, with a little extra overhead). In some aspects, bi-level scalability with a 1: 1 bandwidth ratio is used. Partitioning a video bitstream into two layers of the same size may not be as effective for sequences with little movement. For sequences with little movement, the base level, containing all header and metadata information, is greater than the improvement level. However, since CSFs for sequences with little movement are larger, they fit exactly into the remaining bandwidth of the enhancement level.

High-motion sequences have enough residual information so that 1: 1 data partitioning can be performed with the least overhead information. Additionally, the channel switching frame for such sequences is much smaller for sequences with greater movement. Therefore, error tolerance based on the prediction hierarchy can also work well with scalability for high-motion sequences. The extension of the concepts described above to clips with moderate motion is possible based on the descriptions of these algorithms, and the proposed concepts are applied to video encoding in general.

Multiplexer

In some aspects of the encoder, a multiplexer may be used to encode multiple multimedia streams generated by the encoder and used to prepare encoded bits for broadcast transmission. For example, in the illustrative aspect of the encoder 228 shown in FIG. 2, the clock level 240 comprises a multiplexer. The multiplexer may be implemented to provide control of the bit rate assignment. The estimated complexity can be provided to the multiplexer, which can then assign the available bandwidth for the set of multiplexed video channels according to the coding complexity assumed for these video channels, which in this case allows the quality of a particular channel to remain relatively constant, even if the bandwidth for the set of multiplexed video streams is relatively constant . This enables the channel within the set of channels to have a variable bit rate and relatively constant visual quality instead of a relatively constant bit rate and variable visual quality.

FIG. 18 is a block diagram illustrating a coding system for multiple multimedia streams or channels 1802. The multimedia streams 1802 are encoded by respective encoders 1804 that support communication with a multiplexer (MUX) 1806, which, in turn, supports communication with the medium 1808 transmission. For example, multimedia streams 1802 may correspond to various content channels, such as news channels, sports channels, movie channels, and the like. Encoders 1804 encode multimedia streams 1802 into the encoding format specified for the system. Although described in the context of encoding video streams, the principles and advantages of the disclosed techniques are generally applicable to multimedia streams including, for example, audio streams. Encoded multimedia streams are provided to a multiplexer 1806, which combines various encoded multimedia streams and sends the combined stream to transmission medium 1808 for transmission.

A transmission medium 1808 can correspond to a variety of media, such as, but not limited to, digital satellite communications channels, for example, DirecTV®, digital cable, wired and wireless Internet communications channels, optical networks, cellular telephone networks, and the like. Transmission medium 1808 may include, for example, modulation to radio frequency (RF). Typically due to spectral limitations, etc. the transmission medium has a limited bandwidth, and data from the multiplexer 1806 to the transmission medium is supported with a relatively constant bit rate (CBR).

In conventional systems, the use of a constant bit rate (CBR) when output from the 1806 multiplexer may require that the encoded media or video streams that enter the 1806 multiplexer also be CBR. As described in the prior art, the use of CBR in coding video content can lead to variable visual quality, which is typically undesirable.

In the illustrated system, two or more encoders 1804 transmit the estimated coding complexity of the input. One or more encoders 1804 may receive adaptive bit rate control from multiplexer 1806 in response. This allows the encoder 1804, which is expected to encode a relatively complex video, to receive a higher bit rate or a higher bandwidth (more bits per frame) for these frames by a video capable of a quasi-variable bit rate. This allows the multimedia stream 1802 to be encoded with consistent visual quality. The additional bandwidth that is used by a particular encoder 1804 encoding a relatively complex video comes from bits that would otherwise be used to encode other video streams 1804 if the encoders were implemented to operate at constant bit rates. This supports the output of the 1806 multiplexer at a constant bit rate (CBR).

Although a separate multimedia stream 1802 may be relatively "pulsating", i.e. vary the bandwidth used, the cumulative sum of several multimedia streams may be less pulsating. The bit rate from channels that encode a less complex video can be re-allocated, for example, by a multiplexer 1806, to channels that encode a relatively complex video, and this improves the visual quality of the combined video streams as a whole.

Encoders 1804 provide, in multiplexer 1806, an indicator of the complexity of the set of video frames that must be encoded and multiplexed together. The output of multiplexer 1806 should provide an output that is not higher than the bit rate specified for the transmission medium 1808. Difficulty indicators can be based on the classification of content, as described above, to provide a selected level of quality. A multiplexer 1006 analyzes the complexity indicators and provides the assigned number of bandwidth bits to various encoders 1004, and encoders 1804 use this information to encode the video frames in the set. This makes it possible for a set of video frames to individually have a variable bit rate and at the same time achieve a constant bit rate as a group.

Content classification can also be used to provide quality-based compression of multimedia, in general, for any generalized compression module. Content classification and methods and devices described herein can be used in quality-based and / or content-based multimedia processing of any multimedia data. One example is its use in multimedia compression, in general, for any generalized compression module. Another example is to decompress or decode in any decompression unit or decoder or post processor, for example, interpolation, resampling, enhancement, reconstruction, and presentation operations.

Referring now to FIG. 19, a typical video communication system includes a video compression system consisting of a video encoder and a video decoder connected via a communication network. Wireless networks are one class of fault tolerant networks where the communication channel provides log-normal attenuation or correction and multipath attenuation in mobile scenarios in addition to channel losses. To counter channel errors and provide reliable communications for application layer data, the RF modulator includes direct error correction, including interleaver modules, and channel coding, such as convolutional or turbo coding.

Video coding reduces redundancy in the original video and increases the amount of information carried in each bit of encoded video data. This increases the impact on quality when even a small portion of the encoded video is lost. The spatial and temporal coding inherent in video compression systems exacerbates losses and causes errors to propagate, resulting in visible interference in the restored video. Error tolerance algorithms in the video encoder and error recovery algorithms in the video decoder increase the resiliency of the video compression system.

In a typical embodiment, the video compression system is aware of the core network. Nevertheless, in fault-tolerant networks, the integration or combination of error protection algorithms in the application layer with FEC and channel coding in the channel / physical layer is very desirable and provides the greatest efficiency for increasing system error performance.

FIG. 14 illustrates one example of a distortion data stream depending on a transmission rate that occurs in encoder 228 to encode frames. The process starts 1400 from the beginning 1402 and proceeds to decision block 1404, where it receives input 1410 of the scene change detection module from preprocessor 226 (for example, by means of metadata), and error-resistant input 1406 is obtained. If the information indicates that the selected frame is an I-frame, the process internally encodes the frame. If the information indicates that the selected frame is P-

or a B-frame, a process uses internal coding and (internal) motion estimation coding in order to encode the frame.

After the affirmative mode arises for the modes of step 1404, process 1400 proceeds to preparation step 1414, where the speed R is set equal to R = Rqual, the desired target quality based on the R-D curves. This setting is received from data block 1416 containing R-D curves. Process 1400 then proceeds to step 1418, where the bit rate control (Qpi) bits are assigned based on image / video action information (e.g., content classification) from the content classification process in step 1420.

The bit rate control bit allocation step 1418 is used, in turn, for motion estimation in step 1422. The motion estimate 1422 may also receive metadata input from preprocessor 1412, smoothing motion vectors (MPEG-2 + history) from step 1424 and several reference frames ( causal + non-causal macroblocks MB) from step 1426. The process 1400 then proceeds to step 1428, in which the calculation of the internally encoded modes is determined to assign bit rate control (Qpi) bits. Then, the process 1400 proceeds to step 1430, in which the mode and quantization parameters are determined. The determination of the mode in step 1430 is based on the assessment of the input of step 1422, the input of error tolerance 1406 and the scalability of the RD, which is determined in step 1432. After the mode is determined, the process proceeds to step 1432. It should be noted that the sequence of operations from step 1430 to step 1432 is performed when data is transmitted from the first pass to the second pass portion of the encoder.

At step 1432, the conversion and quantization is performed by the second pass of the encoder 228. The conversion / quantization process is corrected or fine-tuned as indicated by step 1444. This conversion / quantization process may be affected by the fine-tuning module for controlling the transmission rate (Fig. 7) . Process 1400 then proceeds to step 1434 for zigzag sorting and entropy coding to provide an encoded base layer. Zigzag sorting prepares data in an efficient format for encoding. Entropy encoding is a compression technique that uses sequences of bit codes to represent sets of possible characters. The result of step 1432 transform / quantize the level of improvement is also sent to the adder 1436, which subtracts the base level and sends the result to the encoder 1438 ZZ / entropy for the level of improvement, as previously described with reference to Fig.31-36. Additionally, note that the improvement level is sent back (see line 1440 true speed update) to update the classification 1420 of the true speed contents and the operation of determining the long and short-term history of bit rates used by controlling the bit rate.

17A is a flowchart illustrating the processing of multimedia data that is received, received, or otherwise available. Process 1700 begins at block 1702, where it classifies the contents of the multimedia data. In one illustrative aspect, content classification can be performed by means of a classification means, for example, content classification module 712 in FIG. 7. Process 1700 proceeds to step 1704, where it encodes multimedia data in a first data group and a second data group based on content classification. This encoding is performed such that the first data group contains a coefficient, and the second data group contains a first differential refinement associated with the coefficient of the first data group. This can be accomplished by the encoding means described herein, for example, encoder 228 in FIG. 7.

FIG. 17B is a block diagram of a multimedia encoding system 1710 that can perform the process illustrated in FIG. 17A. In some aspects, the multimedia encoding system 1710 may be a transcoder, for example, a transcoder 200. In other aspects, the encoding system 1710 may comprise a transcoder portion. The multimedia coding system 1710 includes means for classifying the contents of the multimedia data, a module for classifying the contents of the multimedia data 1712. The means for classifying the content may be, for example, a classification module in a preprocessor (eg, preprocessor 226) or an encoder (eg, encoder 228). The encoding system 1710 also includes media encoding means, a media encoding module 1714, which may be configured to encode multimedia data in a first data group and in a second data group based on the content classification, and this encoding is performed such that the first data group contains a coefficient, and the second data group contains a first differential refinement associated with the coefficient of the first data group. Other components of the transcoder, such as those described herein, may also be included in coding system 1710.

23, 24, 27, and 28 are flowcharts illustrating examples of encoding multimedia data that implement the aspects described herein. 23 is a flowchart illustrating a process 2300 for encoding multimedia data based on content information. At step 2305, the process 2300 receives the encoded multimedia data, and at step 2310, the process 2300 decodes the multimedia data. At step 2315, process 2300 determines content information associated with the decoded multimedia data. At 2320, process 2300 encodes multimedia data based on content information.

24 is a flowchart illustrating a process 2400 for a method of encoding multimedia data so as to combine data boundaries based on the level of content information. At step 2405, process 2400 obtains content information associated with multimedia data, which may be accomplished, for example, by preprocessor 226 or content classification module 712 shown in FIG. 7. At 2410, process 2400 encodes multimedia data so as to align data boundaries based on content information. For example, the boundaries of a series of macroblocks and the boundaries of access units are aligned with the boundaries of frames based on the classification of the contents of the encoded multimedia data. The encoded data is then available for subsequent processing and / or transmission to a mobile device, and process 2400 ends.

27 is a flowchart illustrating a process 2700 for encoding data using adaptive internal updating based on content information. When process 2700 begins, multimedia data has been received. At step 2705, process 2700 obtains multimedia data content information. Obtaining content information can be performed, for example, by preprocessor 226 or content classification module 712, as described above. Process 2700 proceeds to step 2710, where it encodes multimedia data using an error tolerance scheme based on adaptive internal update, wherein the error tolerance scheme based on adaptive internal update is based on content information. The functionality of step 2710 may be implemented by encoder 228. The encoded data becomes available for subsequent processing and transmission, and process 2700 then terminates.

28 is a flowchart illustrating a process for encoding multimedia data using redundant I-frames based on multimedia content information. When process 2800 begins, multimedia data is available for processing. At 2805, process 2800 obtains multimedia data content information. As described above, this can be accomplished, for example, by preprocessor 226 and / or encoder 228. At 2810, process 2800 encodes multimedia data so as to insert one or more additional I-frames into encoded data based on the content information. This can be done by encoder 228, as described above in connection with the error tolerance scheme, inserting I-frames into the base layer or improvement level depending on the error tolerance scheme used. After block 2810, the encoded data is available for subsequent processing and / or transmission to a mobile device.

It should be noted that the methods described herein can be implemented for a variety of hardware, processors, and communication systems known to those skilled in the art. For example, a general requirement for a client to work as described in this document is that the client has a display to display content and information, a processor to control the operation of the client, and a memory device for storing data and programs related to customer work. In one aspect, the client is a cell phone. In another aspect, the client is a handheld computer having communications. In yet another aspect, the client is a personal computer having communication means. In addition, hardware such as a GPS receiving device may be contained in a client in order to implement various aspects. The various illustrative logical nodes, logic blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or implemented using a general purpose processor, digital signal processor (DSP), custom programmable integrated circuit (ASIC) matrix LSI (FPGA) or other programmable logic device, discrete logic element or transistor logic, discrete hardware components, or any combination thereof, etc. designed to perform the functions described in this document. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors together with a DSP core, or any other similar configuration.

The various illustrative logical nodes, logic blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or implemented using a general purpose processor, digital signal processor (DSP), custom programmable integrated circuit (ASIC) matrix LSI (FPGA) or other programmable logic device, discrete logic element or transistor logic, discrete hardware components, or any combination thereof, etc. designed to perform the functions described in this document. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors together with a DSP core, or any other similar configuration.

The disclosed methods and devices provide transcoding of video data encoded in one format to video data encoded in another format, the encoding being based on the contents of the video data and the encoding is error resistant. The methods or algorithms described in connection with the examples disclosed herein may be implemented directly in hardware, in a software module executed by a processor, in firmware, or in a combination of two or more of these elements. A software module may reside in RAM memory, flash memory, ROM memory, EEPROM memory, registers, on a hard disk, a removable disk, a CD, or any other form of storage medium known in the art. An exemplary storage medium is connected to the processor so that the processor can read information and write information to the storage medium. Alternatively, the storage medium may be integrated in the processor. The processor and storage medium may reside in an ASIC. ASIC can reside in a user terminal. Alternatively, the processor and the storage medium may reside as discrete components in a user terminal.

The examples described above are merely illustrative, and those skilled in the art can now perform various applications and departures from the above examples without deviating from the concepts of the invention disclosed herein. Various modifications to these examples may be apparent to those skilled in the art, and the general principles described herein can be applied to other examples, such as an instant messaging service or any general wireless data exchange application, without departing from the spirit and the scope of the new aspects described in this document. Thus, the scope of the invention is not intended to be limited by the examples shown in this document, but should satisfy the widest scope, consistent with the principles and new features disclosed in this document. The word “exemplary” is used herein solely to mean “serving as an example, individual case, or illustration”. Any example described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other examples. Accordingly, the new features described herein should be defined solely by means of the scope of the claims below.

Claims (49)

1. A method for encoding multimedia data, comprising the steps of: classifying the contents of multimedia data; sectioning multimedia data into a first data group and a second data group based on the classification of the contents, wherein the partitioning is determined in accordance with a bandwidth coefficient; and encode multimedia data in the first data group and in the second data group based on sectioning, wherein the amount of information in the first data group with respect to the amount of information in the second data group corresponds to a bandwidth coefficient. 2. The method according to claim 1, wherein the first data group contains a coefficient, and the second data group contains a first differential refinement associated with the coefficient of the first data group. 3. The method according to claim 1, wherein said encoding comprises the step of determining a bit rate based on the classification of the contents of the multimedia data and encoding the multimedia data based on the bit rate. 4. The method according to claim 1, in which the classification of the content comprises the stage of determining the complexity of the multimedia data. 5. The method according to claim 4, in which the complexity contains temporal complexity or spatial complexity. 6. The method according to claim 4, in which the complexity contains temporal complexity and spatial complexity. 7. The method according to claim 1, wherein the encoding comprises encoding the multimedia data so as to enable decoding of only the first data group or the first data group and the second data group into one combined data group. 8. The method according to claim 1, in which the first differential refinement indicates the difference between the selected video frame and frame data resulting from decoding the first group of data. 9. The method according to claim 1, in which the first group of data is the base level, and the second group of data is the level of improvement. 10. The method according to claim 9, further comprising stages in which: a coefficient is selected from one of the residual error coefficient of the initial base level and the residual error coefficient of the initial improvement level; and calculating a first differential refinement based on said coefficient and the residual error coefficient of the initial improvement level. 11. The method according to claim 1, in which the encoding further comprises the step of encoding the macroblock header information and the motion vector information in the first data group. 12. The method according to claim 1, wherein the encoding further comprises quantizing a first data group with a first step size and quantizing a second data group with a second step size, wherein the first step size and the second step size are correlated by a scaling factor. 13. The method according to claim 1, in which the encoding further comprises determining a first quantization parameter having a first quantization step size for use in encoding the first data group, and determining a second quantization parameter having a second quantization step size for use with encoding a second group of data, wherein the first and second quantization parameters are determined based on the content information of the selected frame data, and wherein said first quantization step size is coarser than that of the second quantization step size mentioned. 14. The method according to claim 1, in which the encoding comprises the step of encoding the first group of data using I-frames and P-frames or any combination thereof and encoding the second group of data using I-frames, P-frames and B- frames or any combination thereof. 15. A device for encoding multimedia data, comprising: means for classifying the contents of multimedia data; means for partitioning multimedia data into a first data group and a second data group based on the classification of the contents, wherein the partitioning is determined in accordance with a bandwidth coefficient; means for encoding multimedia data in the first data group and in the second data group based on sectioning, wherein the amount of information in the first data group with respect to the amount of information in the second data group corresponds to a bandwidth coefficient. 16. The device according to clause 15, in which the first group of data contains a coefficient, and the second group of data contains a first differential refinement associated with the coefficient of the first data group. 17. The apparatus of claim 15, wherein said encoding means comprises means for determining a bit rate based on the classification of the contents of the multimedia data and encoding the multimedia data based on the bit rate. 18. The device according to clause 15, in which said means for classifying the content contains means for determining the complexity of the multimedia data, while the selected multimedia data is encoded based on the complexity of the multimedia data. 19. The device according to p, in which the complexity contains time complexity or spatial complexity. 20. The device according to p, in which the complexity contains time complexity and spatial complexity. 21. The device according to clause 15, in which said encoding means comprises means for encoding multimedia data so as to decode only the first data group or the first data group and the second data group into one combined data group. 22. The device according to clause 16, in which the first differential refinement indicates the difference between the selected video frame and the frame data resulting from the decoding of the first group of data. 23. The device according to clause 16, in which the first group of data is the base level, and the second group of data is the level of improvement. 24. The apparatus of claim 15, wherein the encoding means comprises means for encoding macroblock header information and motion vector information in a first data group. 25. The apparatus of claim 15, wherein said encoding means further comprises means for quantizing a first data group with a first step size and quantizing a second data group with a second step size, wherein the first step size and the second step size are correlated by a scaling factor. 26. The apparatus of claim 15, wherein said encoding means comprises means for determining a first quantization parameter having a first quantization step size for use in encoding a first data group and determining a second quantization parameter having a second quantization step size for use in encoding a second data groups, wherein the first and second quantization parameters are determined based on the content information of the selected frame data, and wherein said first quantization step size is more than coarser than said second quantization step size. 27. The device according to clause 15, in which said encoding means comprises: means for encoding the first group of data using I-frames and P-frames; and means for encoding a second group of data using I-frames, P-frames and B-frames. 28. The device according to item 23, wherein said encoding means comprises: means for selecting a coefficient from one of the residual error coefficient of the initial base level and the residual error coefficient of the initial improvement level; and means for calculating a first differential refinement based on said coefficient and residual error coefficient of the initial improvement level. 29. An apparatus for encoding multimedia data, comprising: a content classification module configured to classify the contents of multimedia data and provide content classification data; a partitioning module that partitions the multimedia data into a first data group and a second data group based on the classification of the contents, wherein the partitioning is determined in accordance with a bandwidth coefficient; and an encoder configured to encode multimedia data in the first data group and in the second data group based on sectioning, wherein the amount of information in the first data group with respect to the amount of information in the second data group corresponds to a bandwidth coefficient. 30. The device according to clause 29, in which the first group of data contains a coefficient, and the second group of data contains a first differential refinement associated with the coefficient of the first data group. 31. The device according to clause 29, in which the encoder contains a component of the bit rate configured to determine the purpose of the bits based on the classification of the content, while the encoding component is additionally configured to encode the selected multimedia data using the assignment of bits. 32. The device according to clause 29, in which the classification of the content includes determining the complexity of the multimedia data, while the selected multimedia data is encoded based on the complexity of the multimedia data. 33. The device according to p, in which the complexity contains temporal complexity or spatial complexity. 34. The device according to p, in which the complexity contains temporal complexity and spatial complexity. 35. The device according to clause 29, in which the encoding comprises encoding multimedia data so as to provide decoding of only the first data group or the first data group and the second data group into one combined data group. 36. The device according to clause 30, in which the first differential refinement indicates the difference between the selected video frame and the frame data resulting from the decoding of the first group of data. 37. The device according to clause 29, in which the first group of data is the basic level, and the second group of data is the level of improvement. 38. A machine-readable medium containing instructions that, when executed, instruct a machine: to classify the contents of multimedia data; sectioning multimedia data into a first data group and a second data group based on the classification of the contents, wherein the partitioning is determined in accordance with a bandwidth coefficient; and encode multimedia data in the first data group and in the second data group based on sectioning, wherein the amount of information in the first data group with respect to the amount of information in the second data group corresponds to a bandwidth coefficient. 39. The computer readable medium of claim 38, wherein the first data group contains a coefficient and the second data group contains a first differential refinement associated with the coefficient of the first data group. 40. The computer-readable medium of claim 38, wherein the instructions to encode comprise instructions to determine the assignment of bits based on the classification of the contents, wherein the encoding component is further configured to encode the selected multimedia data using the assignment of bits. 41. The computer-readable medium of claim 38, wherein the classification of the content comprises determining the complexity of the multimedia data, wherein the selected multimedia data is encoded based on the complexity of the multimedia data. 42. Machine-readable instructions according to paragraph 41, in which the complexity contains temporal complexity or spatial complexity. 43. Machine-readable instructions according to paragraph 41, in which the complexity contains time complexity and spatial complexity. 44. A processor configured to: classify the contents of multimedia data; sectioning multimedia data into a first data group and a second data group based on the classification of the contents, wherein the partitioning is determined in accordance with a bandwidth coefficient; and encode multimedia data in the first data group and in the second data group based on sectioning, wherein the amount of information in the first data group with respect to the amount of information in the second data group corresponds to a bandwidth coefficient. 45. The processor according to item 44 in which the first group of data contains a coefficient, and the second group of data contains a first differential refinement associated with the coefficient of the first data group. 46. The processor according to item 44, wherein the processor is further configured to determine the assignment of bits based on the classification of the contents, while the encoding component is further configured to encode the selected multimedia data using the assignment of bits. 47. The processor according to item 44, wherein the processor is further configured to determine the complexity of the multimedia data, and the classification of the content is based on the complexity of the multimedia data. 48. The processor of claim 47, wherein the complexity comprises temporal complexity or spatial complexity. 49. The processor of claim 47, wherein the complexity comprises temporal complexity and spatial complexity.

Images