BR112017020635B1 - METHOD FOR DECODING VIDEO DATA, METHOD AND APPARATUS FOR ENCODING VIDEO DATA, AND COMPUTER READABLE MEMORY - Google Patents

Abstract

DETERMINING MOTION INFORMATION DERIVATION MODE IN VIDEO-TO-CODE CONVERSION. In one example, a method for decoding video data includes selecting a motion information derivation mode from a plurality of motion information derivation modes to determine motion information for a current block, wherein each motion information derivation mode of the plurality comprises performing a motion search for a first set of reference data that corresponds to a second set of reference data outside the current block, and wherein the motion information indicates motion of the current block relative to the reference video data. The method also includes determining motion information for the current block using the selected motion information derivation mode. The method also includes decoding the current block using the determined motion information and without decoding syntax elements representative of the motion information.

Description

[001] This application claims the benefit of Provisional Application No. U.S. 62/139,572 filed on March 27, 2015, and Provisional Application No. U.S. 62/182,367 filed on June 19, 2015, the entire contents of which are incorporated herein by reference.

FIELD OF TECHNIQUE

[002] This disclosure relates to video code conversion.

BACKGROUND

[003] Digital video capabilities can be incorporated into a wide variety of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, electronic gaming devices, electronic game consoles, satellite or cellular radio telephones, so-called "smart phones", video teleconferencing devices, video streaming devices, and the like. Digital video devices implement video coding techniques such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4, Part 10, Advanced Video Coding (AVC), the High Efficiency Video Coding (HEVC) standard, and extensions of such standards. Video devices can transmit, receive, encode, decode, and/or store digital video information more efficiently by implementing such video code conversion techniques.

[004] Video coding techniques include spatial (intra-picture) prediction and/or temporal (inter-picture) prediction to reduce or remove inherent redundancy in video sequences. For block-based video coding, a video slice (e.g., a video frame or a portion of a video frame) may be partitioned into video blocks, which for some techniques may also be referred to as tree blocks, coding units (CUs), and/or coding nodes. Video blocks in an intra-coded (I) slice of a picture are coded using spatial prediction with respect to reference samples in nearby blocks in the same picture. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction with respect to reference samples in nearby blocks in the same picture or temporal prediction with respect to reference samples in other reference pictures. Pictures may be referred to as frames and reference pictures may be referred to as reference frames.

[005] Spatial or temporal prediction results in a predictive block for a block to be coded. Residual data represents pixel differences between the original block to be coded and the predictive block. An inter-coded block is coded according to a motion vector pointing to a block of reference samples forming the predictive block, and wherein the residual data indicates the difference between the coded block and the predictive block. An intra-coded block is coded according to an intra-coded mode and the residual data. For further compression, the residual data may be transformed from the pixel domain to a transform domain, resulting in residual transform coefficients that can then be quantized. The quantized transform coefficients initially arranged in a two-dimensional array may be scanned to produce a one-dimensional transform coefficient vector, and entropy coding may be applied to achieve further compression.

SUMMARY

[006] The techniques of this disclosure relate to deriving motion information in video code conversion. For example, a video code converter (a video encoder or video decoder) may generate motion information that is not included in the bit stream to convert video data into code that is included in the bit stream. By deriving the motion information in the encoding or decoding cycle, bit savings may be achieved relative to techniques that include motion information in the bit stream such as traditional interprediction techniques.

[007] In one example, a method for decoding video data includes selecting a motion information derivation mode from a plurality of motion information derivation modes to determine motion information for a current block, wherein each motion information derivation mode of the plurality comprises performing a motion search for a first set of reference data that corresponds to a second set of reference data outside of the current block, and wherein the motion information indicates motion of the current block relative to the reference video data, determining motion information for the current block using the selected motion information derivation mode, and decoding the current block using the determined motion information and without decoding syntax elements representative of the motion information.

[008] In another example, a method for encoding video data includes selecting a motion information derivation mode from a plurality of motion information derivation modes to determine motion information for a current block, wherein each motion information derivation mode of the plurality comprises performing a motion search for a first set of reference data that corresponds to a second set of reference data outside of the current block, and wherein the motion information indicates motion of the current block relative to the reference video data, and determining motion information for the current block using the selected motion information derivation mode, and encoding the current block using the determined motion information and without encoding syntax elements representative of the motion information.

[009] In another example, a device for converting video data to code includes a memory configured to store a current block of video data, and one or more processors configured to select a motion information derivation mode from a plurality of motion information derivation modes to determine motion information for the current block, wherein each motion information derivation mode of the plurality comprises performing a motion search for a first set of reference data that corresponds to a second set of reference data outside of the current block, and wherein the motion information indicates motion of the current block relative to the reference video data, determining the motion information for the current block using the selected motion information derivation mode, and converting to code the current block using the determined motion information and without coding syntax elements representative of the motion information.

[0010] In another example, an apparatus for converting video data to code includes means for selecting a motion information derivation mode from a plurality of motion information derivation modes to determine motion information for a current block, wherein each motion information derivation mode of the plurality comprises performing a motion search for a first set of reference data that corresponds to a second set of reference data outside of the current block, and wherein the motion information indicates motion of the current block relative to the reference video data, means for determining motion information for the current block using the selected motion information derivation mode, and means for converting to code the current block using the determined motion information and without decoding syntax elements representative of the motion information.

[0011] In another example, a non-transitory computer-readable medium has instructions stored therein that, when executed, cause one or more processors to select a motion information derivation mode from a plurality of motion information derivation modes to determine motion information for a current block, wherein each motion information derivation mode of the plurality comprises performing a motion search for a first set of reference data that corresponds to a second set of reference data outside of the current block, and wherein the motion information indicates motion of the current block relative to the reference video data, determining motion information for the current block using the selected motion information derivation mode, and encoding the current block using the determined motion information and without decoding syntax elements representative of the motion information.

[0012] In another example, a method for processing video data includes determining a candidate motion vector for deriving motion information from a current block of video data, wherein the motion information indicates the motion of the current block relative to reference video data, and determining a derived motion vector for the current block based on the determined candidate motion vector, wherein determining the derived motion vector comprises performing a motion search for a first set of reference data that corresponds to a second set of reference data outside of the current block.

[0013] In another example, a device for processing video data includes a memory configured to store a current block of video data, and one or more processors configured to determine a candidate motion vector for deriving motion information from a current block of video data, wherein the motion information indicates motion of the current block relative to reference video data, and determining a derived motion vector for the current block based on the determined candidate motion vector, wherein determining the derived motion vector comprises performing a motion search for a first set of reference data that corresponds to a second set of reference data outside of the current block.

[0014] In another example, an apparatus for processing video data includes means for determining a candidate motion vector for deriving motion information from a current block of video data, wherein the motion information indicates motion of the current block relative to reference video data, and means for determining a derived motion vector for the current block based on the determined candidate motion vector, wherein the means for determining the derived motion vector comprises means for performing a motion search for a first set of reference data that corresponds to a second set of reference data outside of the current block.

[0015] In another example, a non-transitory computer-readable medium has instructions stored therein that, when executed, cause one or more processors to determine a candidate motion vector for deriving motion information from a current block of video data, wherein the motion information indicates motion of the current block relative to reference video data, and determine a derived motion vector for the current block based on the determined candidate motion vector, wherein, to determine the derived motion vector, the instructions cause the one or more processors to perform a motion search for a first set of reference data that corresponds to a second set of reference data outside of the current block.

[0016] In another example, a method for processing video data includes dividing a current block of video data into a plurality of sub-blocks to derive motion information of the current block, wherein the motion information indicates motion of the current block relative to reference video data, separately for each respective sub-block among the plurality of sub-blocks, wherein the motion information comprises performing a motion search for a first set of reference data that corresponds to a second set of reference data out of each respective sub-block, and decoding the plurality of sub-blocks based on the derived motion information and without decoding syntax elements representative of the motion information.

[0017] In another example, a method for processing video data includes dividing a current block of video data into a plurality of sub-blocks to derive motion information of the current block, wherein the motion information indicates motion of the current block relative to reference video data, separately for each respective sub-block among the plurality of sub-blocks, wherein the motion information comprises performing a motion search for a first set of reference data that corresponds to a second set of reference data out of each respective sub-block, and encoding the plurality of sub-blocks based on the derived motion information and without encoding syntax elements representative of the motion information.

[0018] In another example, a device for processing video data includes a memory configured to store a current block of video data, and one or more processors configured to divide a current block of video data into a plurality of sub-blocks for deriving motion information of the current block, wherein the motion information indicates motion of the current block relative to reference video data, separately for each respective sub-block among the plurality of sub-blocks, wherein the motion information comprises performing a motion search for a first set of reference data that corresponds to a second set of reference data out of each respective sub-block and coding the plurality of sub-blocks based on the derived motion information and without coding the syntax elements representative of the motion information.

[0019] In another example, a non-transitory computer-readable medium has instructions stored therein that, when executed, cause the one or more processors to divide a current block of video data into a plurality of sub-blocks to derive motion information of the current block, wherein the motion information indicates motion of the current block relative to reference video data, separately for each respective sub-block among the plurality of sub-blocks, wherein the motion information comprises performing a motion search for a first set of reference data that corresponds to a second set of reference data out of each respective sub-block, and decoding the plurality of sub-blocks based on the derived motion information and without decoding syntax elements representative of the motion information.

[0020] In another example, a device for processing video data includes a memory configured to store a current picture, and one or more processors configured to obtain a coded bit stream containing a plurality of encoded pictures, interpolate one or more reference pictures that are not included in the coded bit stream, and decode video data of a current picture of the coded bit stream based on the one or more interpolated reference pictures.

[0021] Details of one or more examples of the disclosure are set forth in the accompanying drawings and in the description below. Other features, objects and advantages will be apparent from the description, drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a block diagram illustrating an exemplary video decoding and encoding system that may implement techniques of this disclosure.

[0023] Figure 2 is a block diagram illustrating an example video encoder that may implement techniques of this disclosure.

[0024] Figure 3 is a block diagram illustrating an example video decoder that may implement techniques of this disclosure.

[0025] Figures 4A and 4B are conceptual diagrams illustrating exemplary spatial neighboring motion vector candidates for a blending mode and an advanced motion vector prediction (AMVP) mode.

[0026] Figures 5A and 5B are conceptual diagrams illustrating an exemplary temporal motion vector predictor (TMVP) candidate and motion vector scaling.

[0027] Figure 6 is a conceptual diagram illustrating an example of one-way motion estimation (ME) in frame rate upconversion (FRUC).

[0028] Figure 7 is a conceptual diagram illustrating an example of bilateral motion estimation (ME) in FRUC.

[0029] Figure 8 is a conceptual diagram illustrating an example of decoder-side motion vector derivation (DMVD) based on template matching.

[0030] Figure 9 is a conceptual diagram illustrating an example of mirror-based bidirectional motion vector derivation in DMVD.

[0031] Figure 10 is a conceptual diagram illustrating motion vector derivation based on extended bilateral correspondence.

[0032] Figure 11 is a flowchart illustrating an example of decoding a prediction unit (PU) using DMVD.

[0033] Figure 12 is a flowchart illustrating an exemplary process for determining a mode of deriving motion information for encoding a block of video data.

[0034] Figure 13 is a flowchart illustrating an exemplary process for deriving a motion vector for encoding a block of video data.

[0035] Figure 14 is a flowchart illustrating an exemplary process for deriving motion information for sub-blocks of a video data block.

DETAILED DESCRIPTION

[0036] The techniques of this disclosure relate to deriving decoder-side motion information, block partitioning, and/or data interpolation in block-based video codecs. The techniques can be applied to any of the existing video codecs, such as High Efficiency Video Coding (HEVC), or be an effective video codec tool for any future video codec standards.

[0037] Video encoder devices implement video compression techniques to efficiently encode and decode video data. Video compression techniques may include applying spatial prediction (e.g., intraframe prediction), temporal prediction (e.g., interframe prediction), and/or other prediction techniques to reduce or remove inherent redundancy in video sequences. A video encoder typically partitions each picture of an original video sequence into rectangular regions referred to as video blocks or encoder units (described in greater detail below). These video blocks may be encoded using a particular prediction mode.

[0038] For inter-prediction modes, a video encoder typically searches for a block similar to one being encoded in a frame at another temporal location, referred to as a reference frame. The video encoder may restrict the search to a given spatial offset of the block to be encoded. A best match may be located using a two-dimensional (2D) motion vector that includes a horizontal offset component and a vertical offset component. For an intra-prediction mode, a video encoder may form the predicted block using spatial prediction techniques based on data from previously encoded neighboring blocks within the same image.

[0039] The video encoder may determine a prediction error, i.e., the difference between pixel values in the block being encoded and in the predicted block (also referred to as residual). The video encoder may also apply a transform to the prediction error, such as a distinct cosine transform (DCT), to generate transform coefficients. After the transformation, the video encoder may quantize the transform coefficients. The quantized transform coefficients and motion vectors may be represented using syntax elements and, together with control information, form a coded representation of a video sequence. In some cases, the video encoder may entropy-code syntax elements, thereby further reducing the number of bits required for their representation.

[0040] A video decoder may, using the syntax elements and control information discussed above, construct predictive data (e.g., a predictive block) for decoding a current frame. For example, the video decoder may add the predicted block and the compressed prediction error. The video decoder may determine the compressed prediction error by weighting the transform basis functions using the quantized coefficients. The difference between the reconstructed frame and the original frame is called the reconstruction error.

[0041] In some cases, a video decoder or post-processing device may interpolate images based on one or more reference images. These interpolated images are not included in an encoded bitstream. The video decoder or post-processing device may interpolate images to upconvert an original frame rate of an encoded bitstream. This process may be referred to as frame rate upconversion (FRUC). Alternatively, the video decoder or post-processing device may interpolate images to insert one or more images that were skipped by a video encoder to encode a video sequence at a reduced frame rate. In either case, the video decoder or post-processing device interpolates frames that are not included in an encoded bitstream that was received by the video decoder. The video decoder or post-processing device may interpolate the images using any of several interpolation techniques, for example, using motion-compensated frame interpolation, frame repetition, or frame averaging.

[0042] Although certain techniques for interpolating images have been used for up-conversion purposes, these techniques have not been widely used during conversion to video code, e.g., to convert video data into code that is included in a coded bit stream. For example, techniques for interpolating images can be relatively time-consuming and/or require a relatively large amount of processing power. Consequently, these techniques have typically not been performed in a loop when decoding video data.

[0043] In accordance with aspects of this disclosure, a video codec (e.g., a video encoder or a video decoder) may derive motion information when it encodes a bit stream. For example, the video codec may generate motion information that is not included in the bit stream to encode video data that is included in the bit stream. By deriving motion information in the encoding or decoding cycle, bit savings may be achieved relative to techniques that include motion information in the bit stream (such as the aforementioned interprediction techniques).

[0044] In accordance with some aspects of this disclosure, a video codec may utilize a plurality of motion information derivation techniques during codec conversion. In these examples, the video codec may determine a motion information derivation mode to determine which motion information derivation techniques to use when determining motion information for a current block. In general, using a motion information derivation mode to derive motion information may include performing a motion search for a first set of reference data that corresponds to a second set of reference data outside of the current block. For example, using the motion information derivation mode (e.g., a bilateral matching technique, a template matching technique, or other technique as described in greater detail below), the video codec may select a candidate motion vector from a list of motion vector candidates. The video encoder may select the candidate motion vector based on the candidate motion vector identifying reference data in a reference picture that relatively closely matches data in the current picture (which may be referred to as determining a “best match” of reference data).

[0045] In some cases, the video codec may use the selected motion vector candidate to identify a search window in a reference picture. The video codec may refine the candidate motion vector based on reference data in the search window that relatively closely matches corresponding data in the current picture. That is, the video codec may derive new motion information for the current block based on motion between reference data in the search window that closely matches data in the current picture. The video codec may then perform motion compensation for the current block using the derived motion information. In this way, the video codec may derive motion information for a current block without motion information being signaled in an encoded bit stream.

[0046] In accordance with aspects of this disclosure, in some examples, the video codec may divide a block of video data into sub-blocks when deriving motion information. For example, the video codec may separately derive motion information for each sub-block of a larger block. In some cases, the video codec may initially determine motion information for the block and use the derived motion information as candidate motion information for each of the sub-blocks. The video codec may then further refine the derived motion information for each of the sub-blocks, for example, using a motion information derivation mode (e.g., a bilateral matching technique, a template matching technique, or other technique as described in greater detail below).

[0047] The techniques of this disclosure may also include techniques for interpolating images. In some cases, any combination of the above techniques may be used to interpolate an image that is not included in the bitstream, e.g., similar to frame rate upconversion. However, rather than simply adding the interpolated image to the video sequence, a video decoder may use the interpolated frame during conversion to code. For example, the video decoder may decode data from a current image based on at least a portion of the interpolated image. In some cases, the video decoder may set the interpolated image equal to the current image. For example, the video decoder may decode syntax data for the current image that is included in the bitstream (e.g., slice header data and the like), interpolate the image, and set the interpolated image as the current image. In other cases, the video decoder may interpolate the image and decode data for the current image relative to the interpolated image. In this case, the video decoder can add the interpolated image to a reference image memory for prediction purposes.

[0048] Accordingly, certain techniques described herein referring to FRUC may, in some examples, be used to determine motion information (e.g., in a decoder-side motion information derivation process). In other examples, techniques described herein referring to FRUC may be used to interpolate video data, e.g., for reference to convert video data to code, or to output.

[0049] Figure 1 is a block diagram illustrating an exemplary video decoding and encoding system 10 that may utilize techniques to derive motion information, perform block partitioning, and/or interpolate video data. As shown in Figure 1, system 10 includes a source device 12 that provides encoded video data to be decoded at a later time by a destination device 14. In particular, source device 12 provides the video data to destination device 14 via a computer-readable medium 16. Source device 12 and destination device 14 may comprise any of a wide range of devices, including desktop computers, notebook computers (i.e., laptop computers), tablet computers, set-top boxes, telephone devices such as so-called "smart" phones, so-called "smart" pad devices, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, or the like. In some cases, source device 12 and destination device 14 may be equipped for wireless communication.

[0050] The destination device 14 may receive the encoded video data to be decoded via the computer-readable media 16. The computer-readable media 16 may comprise any type of media or device capable of moving the encoded video data from the source device 12 to the destination device 14. In one example, the computer-readable media 16 may comprise a communication medium to enable a source device to transmit the encoded video data directly to a destination device 14 in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the destination device 14. The communication media may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication media may be part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication media may include routers, switches, base stations, or any other equipment that may be useful in facilitating communication from the source device 12 to the destination device 14.

[0051] In some examples, the encoded data may be output from the output interface 22 to a storage device. Similarly, the encoded data may be accessed from the storage device via the input interface. The storage device may include any of a variety of locally accessed or distributed data storage media such as a hard drive, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other digital storage media suitable for storing encoded video data. In a further example, the storage device may correspond to a file server or other intermediate storage device that may store the encoded video generated by the source device 12. The destination device 14 may access stored video data from the storage device via streaming or downloading. The file server may be any type of server capable of storing encoded video data and transmitting that encoded video data to the destination device 14. Exemplary file servers include a web server (e.g., for a website), an FTP server, network attached storage (NAS) devices, or a local disk drive. The destination device 14 may access the encoded video data through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both that is suitable for accessing encoded video data stored on a file server. The transmission of encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.

[0052] The techniques of this disclosure are not necessarily limited to wireless applications or definitions. The techniques may be applied to video code conversion to support any of a variety of multimedia applications, such as over-the-air television broadcasts, cable television broadcasts, satellite television broadcasts, Internet video streaming such as dynamic adaptive streaming over HTTP (DASH), digital video that is encoded onto a data storage medium, decoding digital video stored on a data storage medium, or other applications. In some examples, system 10 may be configured to support unidirectional or bidirectional video transmission to support applications such as video streaming, video playback, video broadcasting, and/or video telephony.

[0053] In the example of Figure 1, the source device 12 includes a video source 18, a video encoder 20, and an output interface 22. The destination device 14 includes an input interface 28, a video decoder 30, and a display device 32. In accordance with this disclosure, the video encoder 20 of the source device 12 may be configured to apply techniques for deriving motion information, performing block partitioning, and/or interpolating video data. In other examples, a source device and a destination device may include other components or arrangements. For example, the source device 12 may receive video data from an external video source 18, such as an external camera. Likewise, the destination device 14 may interface with an external display device, rather than including an integrated display device.

[0054] The illustrated system 10 of Figure 1 is merely an example. The techniques for deriving motion information, performing block partitioning, and/or interpolating video data may be performed by any digital video encoding and/or decoding device. While, in general, the techniques of this disclosure are performed by a video encoding device, the techniques may also be performed by a video encoder/decoder typically referred to as a "CODEC." In addition, the techniques of this disclosure may also be performed by a video preprocessor. Source device 12 and destination device 14 are merely examples of such encoder devices wherein source device 12 generates encodered video data for transmission to destination device 14. In some examples, devices 12, 14 may operate in a substantially symmetrical manner such that each of devices 12, 14 includes video encoding and decoding components. Accordingly, the system 10 may support unidirectional or bidirectional video transmission between the video devices 12, 14, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0055] The video source 18 of the source device 12 may include a video capture device, such as a video camera, a video file containing previously captured video, and/or a video feed interface for receiving video from a video content provider. As a further alternative, the video source 18 may generate data based on computer graphics as the source video or a combination of live video, archived video, and computer-generated video. In some cases, if the video source 18 is a video camera, the source device 12 and the target device 14 may form so-called camera phones or videophones. However, as mentioned above, the techniques described in this disclosure may be applicable to video code conversion in general and may be applied to wired and/or wireless applications. In each case, the captured, pre-captured, or computer-generated video may be encoded by the video encoder 20. The encoded video information may then be output by the output interface 22 onto computer-readable media 16.

[0056] Computer-readable media may include transient media, such as a wired network transmission or wireless broadcast, or storage media (i.e., non-transitory storage media), such as a hard drive, flash drive, compact disc, digital video disc, Blu-ray Disc, or other computer-readable media. In some examples, a network server (not shown) may receive encoded video data from source device 12 and provide the encoded video data to destination device 14, e.g., via network transmission. Similarly, a computing device of a media production facility, such as a disc labeling facility, may receive encoded video data from source device 12 and produce a disc containing the encoded video data. Thus, computer-readable media may be understood to include one or more computer-readable media of various forms, in various examples.

[0057] The network interface 28 of the target device 14 receives information from the computer-readable media 16. The information from the computer-readable media 16 may include syntax information defined by the video encoder 20, which is also used by the video decoder 30, which includes syntax elements that describe characteristics and/or processing of blocks and other units converted to code, e.g., GOPs. The display device 32 displays the decoded video data to a user and may comprise any of a variety of display devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0058] Although not shown in Figure 1, in some aspects, the video encoder 20 and video decoder 30 may each be integrated with an audio encoder and decoder, and may include appropriate MUX-DEMUX units, or other hardware and software, to handle the encoding of both audio and video into a common data stream or separate data streams. If applicable, the MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol, or other protocols such as User Datagram Protocol (UDP).

[0059] Video encoder 20 and video decoder 30 may be implemented as any of a variety of suitable encoder circuitry, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combinations thereof. Where the techniques have been implemented partially in software, a device may store instructions for the software on a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Each of video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, one of which may be integrated as part of a combined encoder/decoder (CODEC) in a respective device.

[0060] This disclosure may generally refer to video encoder 20 "signaling" certain information to another device, such as video decoder 30. The term "signaling" may generally refer to the communication of syntax elements and/or other data used to decode the compressed video data. Such communication may occur in real time or near real time. Alternatively, such communication may occur over a period of time, as may occur when storing syntax elements on a computer-readable storage medium in a bit stream encoded at the time of encoding, which may then be retrieved by a decoding device at any time after being stored on such medium.

[0061] Video encoder 20 and video decoder 30 may operate in accordance with a video coding standard. Exemplary video coding standards developed by the Joint Collaboration Team on Video Coding (JCT-VC) as well as the Joint Collaboration Team on 3D Video Extension Development (JCT-3V) of the ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Moving Picture Experts Group (MPEG) include High Efficiency Video Coding (HEVC) or ITU-T H.265, including its range extension, multi-view extension (MV-HEVC), and scalable extension (SHVC). The finalized HEVC standards document is published as “ITU-T H.265, SERIES H: AUDIOVISUAL AND MULTIMEDIA SYSTEMS Infrastructure of audiovisual services - Coding of moving video - High efficiency video coding”, Telecommunication Standardization Sector of International Telecommunication Union (ITU), April 2013. Alternatively, video encoder 20 and video decoder 30 may operate in accordance with other industry or proprietary standards, such as ISO/IEC MPEG-4 Visual and ITU-T H.264 (also known as ISO/IEC MPEG-4 AVC), including its Scalable Video Coding (SVC) extension and Multiview Video Coding (MVC) extension. However, the techniques of this disclosure are not limited to any particular coding standard. For example, the techniques of this disclosure may be used with a variety of other non-proprietary and proprietary video coding techniques or subsequent standards, such as ITU-T H.266.

[0062] As noted above, for interprediction modes, the video encoder 20 may search for a block similar to the one being encoded (a “current block”) in a picture from another temporal location, referred to as a reference picture. The information used to identify the reference picture may be referred to as motion information. For example, for each block, a set of motion information may be available. A set of motion information contains motion information for forward and backward prediction directions. In the present context, the forward and backward prediction directions are two prediction directions of a bidirectional prediction mode, and the terms “forward” and “backward” do not necessarily have a geometric meaning, rather they correspond to the reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1) of a current picture. When only one reference image list is available for an image or slice, only RefPicList0 is available and the motion information for each block of a slice is always forward.

[0063] In some cases, a motion vector together with its reference index is used in decoding processes, this motion vector with the associated reference index is denoted as a uni-predictive set of motion information.

[0064] For each prediction direction, the motion information must contain a reference index and a motion vector. In some cases, for simplicity, a motion vector itself may be referred to in a way that is assumed to have an associated reference index. A reference index is used to identify a reference picture in the current reference picture list (RefPicList0 or RefPicList1). A motion vector has a horizontal and a vertical component.

[0065] Picture Order Count (POC) is widely used in video encoder standards to identify a display order of a picture. While there are cases where two pictures within an encoded video sequence may have the same POC value, this is typically not the case within an encoded video sequence. When multiple encoded video sequences are present in a bitstream, pictures with the same POC value may be close to each other in decoding order. The POC values of the pictures are typically used for reference picture list construction, reference picture set derivation as in HEVC, and motion vector scaling.

[0066] In H.264/AVC, each inter macroblock (MB) can be partitioned into four different ways which include: one 16 x 16 MB partition; two 16 x 8 MB partitions; two 8 x 16 MB partitions; and four 8 x 8 MB partitions. Different partitions within a MB can have different reference index values for each direction (RefPicList0 or RefPicList1). When a MB is not partitioned into four 8 x 8 MB partitions, there is only one motion vector for each MB partition in each direction.

[0067] When a MB is partitioned into four 8 x 8 MB partitions, each 8 x 8 MB partition may be further partitioned into sub-blocks, each of which may have a different motion vector in each direction. There are four different ways to obtain sub-blocks from an 8 x 8 MB partition which include: one 8 x 8 sub-block; two 8 x 4 sub-blocks; two 4 x 8 sub-blocks; and four 4 x 4 sub-blocks. Each sub-block may have a different motion vector in each direction. Therefore, the motion vector is present at a level equal to or greater than the sub-block.

[0068] In AVC, temporal direct mode could be enabled at MB or MB partition level for skip or direct mode on B slices. For each MB partition, the motion vectors of the block co-located with the current MB partition in the RefPicList1[0] of the current block are used to derive the motion vectors. Each motion vector in the co-located block is scaled based on POC distances. In AVC, a direct mode can also predict the motion information from the spatial neighbors, which can be referred to as spatial direct mode.

[0069] In HEVC, to generate a coded representation of an image, the video encoder 20 may generate a set of code tree units (CTUs). Each of the CTUs may comprise a code tree block (CTB) of luma samples, two corresponding CTBs of chroma samples, and syntax structures used to code-convert the samples from the CTBs. In monochrome images or images that have three separate color planes, a CTU may comprise a single CTB block and syntax structures used to code-convert the samples from the code tree block.

[0070] A CTB block may be an NxN block of samples. The size of a CTB may be in the ranges of 16 x 16 to 64 x 64 in the HEVC main profile (although technically 8 x 8 CTB sizes may be supported). A CTB (CU) could be the same size as a CTB and as small as 8 x 8. Each CTU is code-converted with a mode. A CTU may also be referred to as a "tree block" or a "largest code-conversion unit" (LCU). HEVC CTUs may be broadly analogous to macroblocks in other standards such as H.264/AVC. However, a CTU is not necessarily limited to a particular size and may include one or more CTBs (CUs). A slice may include an integer number of CTUs ordered consecutively in a raster scan order.

[0071] In order to generate a code-converted CTU, the video encoder 20 may recursively perform quadratic tree partitioning on the code-converting tree blocks of a CTU to divide the code-converting tree blocks into code-converting blocks, hence the name "code-converting tree units". A code-converting tree block may be an NxN block of samples. A CU may comprise a code-converting block of luma samples and two corresponding code-converting blocks of chroma samples of a picture having a luma sample matrix, a Cb sample matrix, and a Cr sample matrix, and syntax structures used to code-convert the samples of the code-converting blocks. In monochrome pictures or pictures having three separate color planes, a CU may comprise a single code-converting block and syntax structures used to code-convert the samples of the code-converting block.

[0072] The video encoder 20 may partition a code conversion block of a CU into one or more prediction blocks. A prediction block is a rectangular (i.e., square or non-square) block of samples to which the same prediction is applied. A prediction unit (PU) of a CU may comprise a prediction block of luma samples, two corresponding prediction blocks of chroma samples, and syntax structures used to predict the prediction blocks. In monochrome images or in images that have three separate color planes, a PU may comprise a single prediction block and syntax structures used to predict the prediction block. The video encoder 20 may generate luma, Cb, and Cr predictive blocks for luma, Cb, and Cr prediction blocks of each PU of the CU.

[0073] Video encoder 20 may use intra-prediction or inter-prediction to generate the predictive blocks for a PU. If video encoder 20 uses intra-prediction to generate the predictive blocks for a PU, video encoder 20 may generate predictive blocks for the PU based on decoded samples of the picture associated with the PU. If video encoder 20 uses inter-prediction to generate the predictive blocks for a PU, video encoder 20 may generate the predictive blocks for the PU based on decoded samples of one or more pictures other than the picture associated with the PU. When a CU is inter-code converted, the CU may be further partitioned into two or four PUs. When two PUs are present in a CU, the PUs may in some cases be the size of half a rectangle or the size of two rectangles with a size of one-quarter or three-quarters of the CU.

[0074] After video encoder 20 generates predictive luma, Cb, and Cr blocks for one or more PUs of a CU, video encoder 20 may generate a luma residual block for the CU. Each sample in the CU's luma residual block indicates a difference between a luma sample in one of the CU's predictive luma blocks and a corresponding sample in the CU's original luma code conversion block. Additionally, video encoder 20 may generate a Cb residual block for the CU. Each sample in the CU's Cb residual block may indicate a difference between a Cb sample in one of the CU's predictive Cb blocks and a corresponding sample in the CU's original Cb code conversion block. Video encoder 20 may also generate a Cr residual block for the CU. Each sample in the CU Cr residual block may indicate a difference between a Cr sample in one of the CU's predictive Cr blocks and a corresponding sample in the CU's original Cr code conversion block.

[0075] Furthermore, the video encoder 20 may use quadratic tree partitioning to decompose the luma, Cb, and Cr residual blocks of a CU into one or more luma, Cb, and Cr transform blocks. A transform block is a rectangular (e.g., square or non-square) block of samples to which the same transform is applied. A transform unit (TU) of a CU may comprise a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax structures used to transform the transform block samples. Thus, each TU of a CU may be associated with a luma transform block, a Cb transform block, and a Cr transform block. The luma transform block associated with the TU may be a subblock of the luma residual block of the CU. The Cb transform block may be a subblock of the Cb residual block of the CU. The Cr transform block may be a sub-block of the Cr residual block of the CU. In monochrome images or in images that have three separate color planes, a TU may comprise a single transform block and syntax structures used to transform the samples of the transform block.

[0076] Video encoder 20 may apply one or more transforms to a luma transform block of a TU to generate a luma coefficient block for the TU. A coefficient block may be a two-dimensional array of transform coefficients. A transform coefficient may be a scalar quantity. Video encoder 20 may apply one or more transforms to a Cb transform block of a TU to generate a Cb coefficient block for the TU. Video encoder 20 may apply one or more transforms to a Cr transform block of a TU to generate a Cr coefficient block for the TU.

[0077] After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), the video encoder 20 may quantize the coefficient block. In general, quantization refers to a process in which transform coefficients are quantized to potentially reduce the amount of data used to represent the transform coefficients, which provides additional compression. After the video encoder 20 quantizes a coefficient block, the video encoder 20 may entropy encode syntax elements indicating the quantized transform coefficients. For example, the video encoder 20 may perform Context Adaptive Binary Arithmetic Coding (CABAC) on the syntax elements indicating the quantized transform coefficients.

[0078] The video encoder 20 may output a bit stream that includes a sequence of bits that form a representation of encoded images and associated data. The bit stream may comprise a sequence of network abstraction layer (NAL) units. A NAL unit is a syntax structure that contains an indication of the type of data in the NAL unit and bytes that contain that data in the form of a raw byte sequence payload (RBSP) interleaved as necessary with emulation prevention bits. Each of the NAL units includes a NAL unit header and encapsulates an RBSP.

[0079] Different types of NAL units may encapsulate different types of RBSPs. For example, a first type of NAL unit may encapsulate an RBSP for a picture parameter adjustment (PPS), a second type of NAL unit may encapsulate an RBSP for a coded slice, a third type of NAL unit may encapsulate an RBSP for SEI, and so on. NAL units that encapsulate RBSPs for video codec data (as opposed to RBSPs for parameter sets and SEI messages) may be referred to as video codec layer (VCL) NAL units.

[0080] The video decoder 30 may receive a bit stream generated by the video encoder 20. In addition, the video decoder 30 may perform parsing of the bit stream to obtain syntax elements from the bit stream. The video decoder 30 may reconstruct images from the video data based, at least in part, on the syntax elements obtained from the bit stream. The process for reconstructing the video data may generally be reciprocal to the process performed by the video encoder 20. In addition, the video decoder 30 may inversely quantize the coefficients of blocks associated with the TUs of a current CU. The video decoder 30 may perform inverse transforms on the coefficient blocks to reconstruct transform blocks associated with the TUs of the current CU. The video decoder 30 can reconstruct the code conversion blocks of the current CU by adding the samples of the predictive blocks for the PUs of the current CU to the corresponding samples of the transform blocks of the TUs of the current CU. By reconstructing the code conversion blocks for each CU of a picture, the video decoder 30 can reconstruct the picture.

[0081] When a CU is inter-code converted, a set of motion information is present for each PU. In addition, each PU is code converted with a unique inter-prediction mode to derive a set of motion information. In the HEVC standard, there are two inter-prediction modes, called blending (skipping is considered a special case of blending) and advanced motion vector prediction (AMVP) modes respectively for a prediction unit (PU).

[0082] In AMVP or blend mode, a motion vector (MV) candidate list is maintained for multiple motion vector predictors. The motion vector (or vectors), as well as reference indices in blend mode, of the current PU is generated by taking a candidate from the MV candidate list.

[0083] The MV candidate list contains up to five candidates for the blending mode and two candidates for the AMVP mode. A blend candidate may have a set of motion information, e.g., motion vectors corresponding to both reference image lists (list 0 and list 1) and reference indices. If a blend candidate is identified by a blend index, the reference images are used for prediction of the current blocks, and the associated motion vectors are determined. However, under the AMVP mode for each potential prediction direction from list 0 or list 1, a reference index needs to be explicitly flagged, along with an MVP index for the MV candidate list since the AMVP candidate contains only one motion vector. In the AMVP mode, the predicted motion vectors may be further refined.

[0084] As can be seen above, a blend candidate corresponds to a complete motion information set while an AMVP candidate contains only a motion vector for a specific prediction direction and reference index. Candidates for both modes are similarly derived from the same spatial and temporal neighboring blocks, as described with respect to Figures 4 and 5 below.

[0085] In accordance with aspects of this disclosure as described in greater detail below, video encoder 20 and/or video decoder 30 may be configured to perform any combination of the techniques described herein to derive motion information, perform block partitioning, and/or interpolate video data. With respect to deriving motion information, video encoder 20 and/or video decoder 30 may be configured to derive motion information by performing a motion search for a first set of reference data that corresponds to a second set of reference data outside of the current block. The match may be determined based on an amount of similarity between reference data, and may be referred to herein as determining a “match” or “best match.”

[0086] In some examples, video encoder 20 and/or video decoder 30 may initially encode one or more syntax elements that indicate whether the motion derivation process is enabled. In some cases, the one or more syntax elements may be incorporated with another mode, such as the blending mode described above. For example, as described in greater detail with respect to the example of Figure 10, video encoder 20 and/or video decoder 30 encodes one or more syntax elements when performing the blending mode (e.g., a flag, an index into a blend candidate list, or the like) that indicates whether to perform motion derivation.

[0087] In cases where motion derivation is enabled, in accordance with aspects of this disclosure, video encoder 20 and/or video decoder 30 may perform pattern-matched motion vector derivation. For example, video encoder 20 and/or video decoder 30 may encode one or more syntax elements that indicate which motion information derivation mode to apply from a plurality of motion information derivation modes. Video encoder 20 and/or video decoder 30 may encode a flag to distinguish between two motion information derivation modes or an index to distinguish between more than two motion information derivation modes. As described herein, exemplary pattern-matched motion information derivation modes include two-way matching or template matching.

[0088] During the motion derivation process, the video encoder 20 and/or video decoder 30 may derive an initial motion vector for an entire block (e.g., an entire PU) based on the selected motion derivation process. For example, the video encoder 20 and/or video decoder 30 may use motion vectors from a candidate list associated with the blending mode and determine which motion vector from the candidate list results in a best match. That is, the video encoder 20 and/or video decoder 30 may determine which motion vector from the candidate list, when used in the selected motion derivation process, results in a first set of reference data that matches a second set of reference data outside of the current block, e.g., reference data that closely matches data in the current image or another reference image. In general, “best match” may refer to video data that is most similar in terms of pixel differences.

[0089] As an example for purposes of illustration, as described in greater detail with respect to Figure 8, the video encoder 20 and/or video decoder 30 may select a template-matching motion information derivation mode. In this example, the video encoder 20 and/or video decoder 30 may select the candidate motion vector from the vector candidate blending mode that results in a template that most closely matches a template in a reference image. For example, the video encoder 20 and/or video decoder 30 may be configured to perform a motion search for a first set of reference data that corresponds to a second set of reference data outside of the current block, wherein the first set of data comprises the template in the current image and the second set of reference data comprises the template in the reference image. In some cases, as described in greater detail below, video encoder 20 and/or video decoder 30 may additionally or alternatively select a candidate motion vector based on a minimum matching cost.

[0090] In accordance with aspects of this disclosure, after determining the candidate motion vector, the video encoder 20 and/or video decoder 30 may further refine the candidate motion vector to derive motion information for the current block. For example, the video encoder 20 and/or video decoder 30 may perform a search (e.g., a search using the motion information derivation mode) in a predefined area of the reference picture indicated by the candidate motion vector. The video encoder 20 and/or video decoder 30 may derive the motion information for the current block based on the motion vector indicating reference data in the predefined area that has a best match and/or a minimum matching cost to video data of the current picture.

[0091] In some cases, in accordance with aspects of this disclosure, video encoder 20 and/or video decoder 30 may divide the block of video data into sub-blocks when deriving motion information. For example, video encoder 20 and/or video decoder 30 may separately derive motion information for each sub-block of a larger block. In some cases, video encoder 20 and/or video decoder 30 may use the derived motion information for a block (e.g., using the process described above) as candidate motion information for each of the sub-blocks. Video encoder 20 and/or video decoder 30 may then further refine the derived motion information for each of the sub-blocks, for example, using a particular motion information derivation mode (e.g., a bilateral matching technique, a template matching technique, or another technique as described in greater detail below).

[0092] Figure 2 is a block diagram illustrating an example video encoder 20 that may implement techniques for deriving motion information, performing block partitioning, and/or interpolating video data. Video encoder 20 may perform intra- and inter-code conversion of video blocks within video slices. Intra-code conversion relies on spatial prediction to reduce or remove spatial redundancy in the video within a given video frame image. Inter-code conversion relies on temporal prediction to reduce or remove temporal redundancy in the video within adjacent frames or images of a video sequence. Intra-mode (I-mode) may refer to any of several spatially based coding modes. Inter-modes, such as mono-directional prediction (P-mode) or bi-directional prediction (B-mode), may refer to any of several temporally based coding modes.

[0093] As shown in Figure 2, the video encoder 20 receives a current video block within a video frame to be encoded. In the example of Figure 2, video encoder 20 includes video data memory 38, mode selection unit 40, reference picture memory 64, adder 50, transform processing unit 52, quantization unit 54, and entropy coding unit 56. Mode selection unit 40 in turn includes motion compensation unit 44, motion estimation unit 42, intraprediction unit 46, and partition unit 48. For video block reconstruction, video encoder 20 also includes inverse quantization unit 58, inverse transform unit 60, and adder 62. A block removal filter (not shown in Figure 2) may also be included at the filter block boundaries to remove blocking artifacts from the reconstructed video. If desired, the block removal filter would typically filter the output of adder 62. Additional filters (loop or post-loop) may also be used in addition to the block removal filter. These filters are not shown for brevity, but if desired, they may filter the output of adder 50 (like a loop filter).

[0094] During the encoding process, the video encoder 20 receives a video frame or slice to be converted to code. The frame or slice may be divided into multiple video blocks. The video data memory 38 may store the video data to be encoded by the components of the video encoder 20. The video data stored in the video data memory 38 may be obtained, for example, from the video source 18. The reference picture memory 64 may be referred to as a DPB that stores reference video data for use in encoding video data by the video encoder 20, for example, in intra- or inter-code conversion modes. Video data memory 38 and a reference picture memory 64 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), which includes synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Video data memory 38 and reference picture memory 64 may be provided by the same memory device or separate memory devices. In various examples, video data memory 38 may be on-chip with other video encoder components 20, or off-chip with respect to those components.

[0095] Motion estimation unit 42 and motion compensation unit 44 perform inter-predictive coding of the received video block with respect to one or more blocks in one or more reference frames to provide temporal prediction. Intra-prediction unit 46 may alternatively perform intra-predictive coding of the received video block with respect to one or more neighboring blocks in the same frame or slice as the block to be coded to provide spatial prediction. Video encoder 20 may perform multiple coding steps, for example, to select an appropriate coding mode for each block of video data.

[0096] Furthermore, the partition unit 48 may partition blocks of video data into sub-blocks, based on the evaluation of previous partitioning schemes in previous encode steps. For example, the partition unit 48 may initially partition a frame or slice into LCUs, and partition each of the LCUs into sub-CUs based on rate distortion analysis (e.g., rate distortion optimization). The mode selection unit 40 may further produce a quadratic tree data structure indicating the partitioning of an LCU into sub-CUs. The leaf node CUs of the quadratic tree may include one or more PUs and one or more TUs.

[0097] The mode selection unit 40 may select one of the coding modes, intra or inter, for example, based on error results, and provides the resulting intra or intercoded block to the adder 50 to generate residual block data and to the adder 62 to reconstruct the coded block for use as a reference frame. The mode selection unit 40 also provides syntax elements, such as motion vectors, intra-mode indicators, partition information, and other such syntax information, to the entropy coding unit 56.

[0098] The motion estimation unit 42 and the motion compensation unit 44 may be highly integrated, but are illustrated separately for conceptual purposes. Motion estimation, performed by the motion estimation unit 42, is the process of generating motion vectors, which estimate motion for video blocks. A motion vector, for example, may indicate the displacement of a PU of a video block within a current video frame or image relative to a predictive block within a reference frame (or other encoded unit) relative to the current block being encoded within the current frame (or other encoded unit). A predictive block is a block that closely matches the block to be encoded in terms of pixel difference, which may be determined by summation of absolute difference (SAD), summation of squared difference (SSD), or other difference measures.

[0099] In some examples, the video encoder 20 may perform a fast motion search to determine a motion vector of a block. There are many fast motion search methods proposed in the literature, such as Block-Based Maximum Slope Search (BBGDS) as described, for example, in Lurng-Kuo Liu, Ephraim Feig, “A block-based gradient descent search algorithm for block motion estimation in video coding,” IEEE Trans. Circuits Syst. Video Technol., vol. 6, pages 419-422, August 1996, Unrestricted Center-Biased Diamond Search (UCBDS) as described, for example, in Jo Yew Tham, Surendra Ranganath, Maitreya Ranganath, and Ashraf Ali Kassim, “A novel unrestricted center-biased diamond search algorithm for block motion estimation,” IEEE Trans. Circuits Syst. Video Technol., vol. 8, pages 369–377, August 1998, and HEXagon-Based Search (HEBS) as described, for example, in Ce Zhu, Xiao Lin, and Lap-Pui Chau, “Hexagon-Based Search Pattern for Fast Block Motion Estimation,” IEEE Trans. Circuits Syst. Video Technol. , vol. 12, pages 349–355, May 2002. Basically, these techniques involve searching only a certain number of positions within a search window based on predefined search patterns. These techniques typically work well when the motion is small to moderate.

[00100] In some examples, the video encoder 20 may calculate values for sub-integer pixel positions of reference pictures stored in the reference picture memory 64. For example, the video encoder 20 may interpolate values of quarter-pixel positions, one-eighth-pixel positions, or other fractional pixel positions of the reference picture. Therefore, the motion estimation unit 42 may perform a motion search with respect to the full pixel positions and fractional pixel positions and output a motion vector with fractional pixel precision.

[00101] The motion estimation unit 42 calculates a motion vector for a PU of a video block in an intercoded slice by comparing the position of the PU to the position of a predictive block of a reference picture. The reference picture may be selected from a first list of reference pictures (List 0) or a second list of reference pictures (List 1), each of which identifies one or more reference pictures stored in the reference picture memory 64. The motion estimation unit 42 sends the calculated motion vector to the entropy coding unit 56 and the motion compensation unit 44.

[00102] Motion compensation, performed by motion compensation unit 44, may involve obtaining or generating the prediction block based on the motion vector determined by motion estimation unit 42. Again, motion estimation unit 42 and motion compensation unit 44 may be functionally integrated, in some examples. Upon receiving the motion vector for the PU of the current video block, motion compensation unit 44 may locate the predictive block to which the motion vector points in one of the reference picture lists. Adder 50 forms a residual video block by subtracting pixel values of the predictive block from pixel values of the current video block being encoded, forming pixel difference values, as discussed below. In general, motion estimation unit 42 performs motion estimation relative to luma components, and motion compensation unit 44 uses motion vectors calculated based on luma components for both chroma components and luma components. The mode selection unit 40 may also generate syntax elements associated with the video blocks and the video slice for use by the video decoder 30 in decoding the video blocks of the video slice.

[00103] The intra-prediction unit 46 may intra-predict a current block, as an alternative to the intra-prediction performed by the motion estimation unit 42 and the motion compensation unit 44, as described above. In particular, the intra-prediction unit 46 may determine an intra-prediction mode to be used to encode a current block. In some examples, the intra-prediction unit 46 may encode a current block using multiple intra-prediction modes, e.g., during separate encoding steps, and the intra-prediction unit 46 (or mode selection unit 40, in some examples) may select a suitable intra-prediction mode to use from the tested modes.

[00104] For example, the intra-prediction unit 46 may calculate rate distortion values using a rate distortion analysis for the various intra-prediction modes tested and select the intra-prediction mode that has the best rate distortion characteristics from among the modes tested. The rate distortion analysis generally determines an amount of distortion (or error) between an encoded block and an original unencoded block that was encoded to produce the encoded block, as well as a bit rate (i.e., a number of bits) used to produce the encoded block. The intra-prediction unit 46 may calculate ratios from distortions and rates for the various encoded blocks to determine which intra-prediction mode exhibits the best rate distortion value for the block.

[00105] After selecting an intraprediction mode for a block, the intraprediction unit 46 may provide information indicative of the selected intraprediction mode for the block to the entropy coding unit 56. The entropy coding unit 56 may encode information indicating the selected intraprediction mode. The video encoder 20 may include in the transmitted bitstream copicture data, which may include a plurality of intraprediction mode index tables and a plurality of modified intraprediction mode index tables (also referred to as codeword mapping tables), definitions of coding contexts for various blocks and indications of a most likely intraprediction mode, a table of intraprediction mode indexes, and a table of modified intraprediction mode indexes to use for each of the contexts.

[00106] In accordance with aspects of this disclosure as described herein, video encoder 20 may be configured to perform any combination of the techniques described herein to derive motion information, perform block partitioning, and/or interpolate video data. In particular, certain techniques of this disclosure may be performed by derivation unit 49. For example, derivation unit 49 may be configured to determine motion information for a current block and without including data indicating the motion information in the bitstream.

[00107] In some cases, the derivation unit 49 (and/or mode selection unit 40) may determine whether to perform motion derivation for a particular block (e.g., versus traditional intraprediction or interprediction) based on a rate distortion analysis. For example, the derivation unit 49 may determine whether to perform motion derivation in a manner similar to a rate distortion cost selection as is performed for merge candidates in the merge mode. In this example, the derivation unit 49 may check each motion information derivation mode from a plurality of motion information derivation modes (e.g., a bilateral matching mode, a template matching mode, or the like) using a rate distortion cost selection. The derivation unit 49 may further compare the motion information derivation mode that has the minimum cost to other PU modes (e.g., intraprediction modes, a traditional interprediction mode, a pallet code conversion mode, or the like). In cases where the motion derivation mode is the most effective mode with respect to code conversion efficiency, the video encoder 20 may encode one or more syntax elements indicating that motion information is derived (rather than signaled) for the current block. The video encoder 20 may also encode one or more syntax elements to indicate the motion derivation mode from a plurality of motion information derivation modes.

[00108] In other examples, in accordance with aspects of this disclosure, the derivation unit 49 may interpolate video data that is not included in the encoded bitstream for a video sequence. For example, the derivation unit 49 may perform any motion derivation matching techniques to interpolate an image that is not included in the bitstream, e.g., similar to frame rate upconversion. In some cases, the video encoder 20 may use the interpolated image during encoding. For example, the derivation unit 49 may interpolate an image and the video encoder 20 may encode data for a current image relative to the interpolated image. In this example, the video encoder 20 may add the interpolated image to the reference image memory 64 and encode data from other images based on at least a portion of the interpolated image.

[00109] In other examples, the derivation unit 49 may interpolate a picture and the video encoder 20 may set the interpolated picture equal to the current picture. For example, the derivation unit 49 may interpolate the current picture and the video encoder 20 may encode syntax data for the current picture to be included in the bitstream (e.g., slice header data and the like), but may skip encoding the video data for the current picture.

[00110] While the derivation unit 49 may be configured to perform certain derivation and/or interpolation techniques as described herein, it should be understood that one or more other units of the video encoder 20 may also or alternatively be configured to interpolate data. For example, the video encoder 20 may include a variety of other interpolators or filters, for example, to interpolate a pixel at a sub-pixel (sub-pel) location during motion compensation.

[00111] Video encoder 20 forms a residual video block by subtracting the prediction data from mode selection unit 40 from the original video block being encoded. Adder 50 represents the component or components that perform this subtraction operation. Transform processing unit 52 applies a transform, such as a discrete cosine transform (DCT) or one conceptually similar, to the residual block, producing a video block comprising residual transform coefficient values. Transform processing unit 52 may perform other transforms that are conceptually similar to DCT. Wavelet transforms, integer transforms, subband transforms, or other types of transforms could also be used.

[00112] In each case, the transform processing unit 52 applies the transform to the residual block, producing a block of residual transform coefficients. The transform may convert the residual information from a pixel value domain into a transform domain, such as a frequency domain. The transform processing unit 52 may send the resulting transform coefficients to the quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the quantization unit 54 may then perform a scan of the matrix that includes the quantized transform coefficients. Alternatively, the entropy coding unit 56 may perform the scan.

[00113] After quantization, the entropy coding unit 56 entropy-codes the quantized transform coefficients. For example, the entropy coding unit 56 may perform context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy coding (PIPE), or another entropy-code technique. In the case of context-based entropy-code, the context may be based on neighboring blocks. After entropy-code conversion by the entropy coding unit 56, the encoded bitstream may be transmitted to another device (e.g., video decoder 30) or archived for later transmission or retrieval.

[00114] The inverse quantization unit 58 and the inverse transform unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain, e.g., for later use as a reference block. The motion compensation unit 44 may calculate a reference block by adding the residual block to a predictive block of one of the frames of the reference image memory 64. The motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values for use in motion estimation. The adder 62 adds the reconstructed residual block to the motion compensated prediction block produced by the motion compensation unit 44 to produce a reconstructed video block for storage in the reference picture memory 64. The reconstructed video block may be used by the motion estimation unit 42 and the motion compensation unit 44 as a reference block to code interconvert a block into a subsequent video frame.

[00115] Figure 3 is a block diagram illustrating an example video decoder 30 that may implement techniques for deriving motion information, performing block partitioning, and/or interpolating video data. In the example of Figure 3, video decoder 30 includes video data memory 68, an entropy decoding unit 70, motion compensation unit 72, intraprediction unit 74, inverse quantization unit 76, inverse transform unit 78, reference picture memory 82, and adder 80. Video decoder 30 may, in some examples, perform a decoding step generally reciprocal to the encoding step described with respect to video encoder 20 (Figure 2). The motion compensation unit 72 may generate prediction data based on motion vectors received from the entropy decoding unit 70, while the intra-prediction unit 74 may generate prediction data based on the intra-prediction mode indicators received from the entropy decoding unit 70.

[00116] During the decoding process, the video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video slice and associated syntax elements from the video encoder 20. The video data stored in the video data memory 68 may be obtained, for example, from computer-readable media, for example, from a local video source such as a camera, via wireless or wired video data network communication, or by accessing physical data storage media. The video data memory 68 may form a coded picture buffer (CPB) that stores encoded video data from an encoded video bitstream.

[00117] Reference picture memory 82 may be referred to as a DPB that stores reference video data for use in decoding video data by video decoder 30, for example, in intra- or inter-code conversion modes. Video data memory 68 and reference picture memory 82 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), which includes synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Video data memory 68 and reference picture memory 82 may be provided by the same memory device or separate memory devices. In various examples, video data memory 68 may be on-chip with other components of video decoder 30 or off-chip with respect to those components.

[00118] The entropy decoding unit 70 of the video decoder 30 entropy decodes the bit stream to generate quantized coefficients, motion vectors or intraprediction mode indicators, and other syntax elements. The entropy decoding unit 70 forwards the motion vectors and other syntax elements to the motion compensation unit 72. The video decoder 30 may receive the syntax elements at the video slice level and/or at the video block level.

[00119] When the video slice is code-converted as an intra-code-converted slice (I), the intra-prediction unit 74 may generate prediction data for a video block of the current video slice based on a signaled intra-prediction mode and data from previously decoded blocks of the current frame or picture. When the video frame is code-converted as an inter-code-converted slice (i.e., B, P, or GPB), the motion compensation unit 72 produces predictive blocks for a video block of the current video slice based on motion vectors and other syntax elements received from the entropy decoding unit 70. The prediction blocks may be produced from one of the reference pictures within one of the reference picture lists. The video decoder 30 may construct the reference frame lists, List 0 and List 1, using standard construction techniques based on the reference pictures stored in the reference picture memory 82.

[00120] The motion compensation unit 72 determines prediction information for a video block of the current video slice by analyzing the motion vectors and other syntax elements, and uses the prediction information to produce the predictive blocks for the current video block under decoding. For example, the motion compensation unit 72 may use a portion of the received syntax elements to determine a prediction mode (e.g., intra- or inter-prediction) used to encode the video blocks of the video slice, a type of inter-prediction slice (e.g., B-slice, P-slice, or GPB-slice), construction information for one or more of the reference picture lists for the slice, motion vectors for each inter-coded video block of the slice, inter-prediction status for each inter-coded video block of the slice, and other information for decoding the video blocks in the current video slice.

[00121] The motion compensation unit 72 may also perform interpolation based on interpolation filters. The motion compensation unit 72 may use interpolation filters as used by the video encoder 20 during encoding of the video blocks to calculate interpolated values for sub-integer pixels of reference blocks. In this case, the mode compensation unit 72 may determine the interpolation filters used by the video encoder 20 from the received syntax elements and use the interpolation filters to produce prediction blocks.

[00122] In accordance with aspects of this disclosure, video decoder 30 may be configured to perform any combination of the techniques described herein for deriving motion information, performing block partitioning, and/or interpolating video data. In particular, certain techniques of this disclosure may be performed by derivation unit 75. For example, in accordance with aspects of this disclosure, derivation unit 75 may be configured to determine motion information for a current block and without decoding the motion information from an encoded bit stream.

[00123] In some cases, the derivation unit 75 may determine whether to perform motion derivation for a particular block (e.g., versus traditional intraprediction or interprediction). For example, the video decoder 30 may decode one or more syntax elements that indicate that motion information is derived (rather than signaled) for a block being decoded. The video decoder 30 may also decode one or more syntax elements that indicate a motion information derivation mode from a plurality of motion information derivation modes to be used to decode a block. The derivation unit 75 may determine whether to perform motion derivation and the motion information derivation mode for a block based on the decoded syntax. In some examples, as described herein, the syntax may be associated with one or more other modes, such as blend mode, AMVP, or other decoding functions.

[00124] In accordance with other aspects of this disclosure, the derivation unit 75 may interpolate video data that is not included in the encoded bitstream for a video sequence. For example, the derivation unit 75 may perform any motion derivation matching techniques to interpolate a picture that is not included in the parsed bitstream, e.g., similar to frame rate upconversion. In some cases, the video decoder 30 may use the interpolated picture during encoding. For example, the derivation unit 75 may interpolate a picture and the video decoder 30 may decode data for a current picture relative to the interpolated picture. In this example, the video decoder 30 may add the interpolated picture to the reference picture memory 82 and decode data from other pictures based on at least a portion of the interpolated picture.

[00125] In other examples, the derivation unit 75 may interpolate a picture and the video decoder 30 may set the interpolated picture equal to the current picture. For example, the derivation unit 75 may interpolate the current picture and the video decoder 30 may decode syntax elements for the current picture from the encoded bitstream (e.g., slice header data and the like), but may skip decoding the video data for the current picture and instead interpolate the current picture.

[00126] Although the derivation unit 75 may be configured to perform certain interpolation techniques as described herein, it should be understood that one or more other units of the video decoder 30 may also or alternatively be configured to interpolate data. For example, the video decoder 30 may include a variety of other interpolators or filters, for example, to interpolate a pixel at a sub-pixel (sub-pel) location during motion compensation.

[00127] The inverse quantization unit 76 inversely quantizes, e.g., dequantizes, the quantized transform coefficients provided in the bitstream and decoded by the entropy decoding unit 70. The inverse quantization process may include using a quantization parameter calculated by the video decoder 30 for each video block in the video slice to determine a degree of quantization and, similarly, a degree of inverse quantization that should be applied.

[00128] The inverse transform unit 78 applies an inverse transform 78, e.g., a DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients in order to produce residual blocks in the pixel domain.

[00129] After the mode compensation unit 72 generates the prediction block for the current video block based on the motion vectors and other syntax elements, the video decoder 30 forms a decoded video block by summing the residual blocks from the inverse transform unit 78 with the corresponding prediction blocks generated by the mode compensation unit 72. The adder 80 represents the component or components that perform this summation operation. If desired, a block removal filter may also be applied to filter the decoded blocks to remove blockiness artifacts. Other loop filters (both in the encoding loop and after the encoding loop) may also be used to smooth pixel transitions, or otherwise improve video quality. The decoded video blocks in a given frame or reference image are then stored in the image memory 82, which stores reference images used for subsequent motion compensation. The reference picture memory 82 also stores the decoded video for later presentation on a display device, such as the display device 32 of Figure 1.

[00130] Figures 4A and 4B are conceptual diagrams illustrating spatial neighbor candidates in HEVC. In some examples, video encoder 20 and/or video decoder 30 may derive spatial motion vector (MV) candidates from neighbor block 0, neighbor block 1, neighbor block 2, neighbor block 3, or neighbor block 4 for PU0.

[00131] In some cases, the techniques for generating the MV candidates from the blocks differ for blending and AMVP modes. Figure 4A illustrates an example for blending mode. For example, in HEVC, a video encoder (e.g., as video encoder 20 and/or video decoder 30 of FIGS. 1-3) may derive up to four spatial MV candidates. The candidates may be included in a candidate list that has a particular order. In one example, the order for the example of Figure 4A may be neighbor block 0 (A1), neighbor block 1 (B1), neighbor block 2 (B0), neighbor block 3 (A0), and neighbor block 4 (B2).

[00132] Figure 4B illustrates an example for the AMVP mode. For example, in HEVC, the video encoder may divide neighboring blocks into two groups: where the left group includes neighboring block 0 and neighboring block 1, and the above group includes neighboring block 2, neighboring block 3, and neighboring block 4. For each group, the potential motion vector candidate associated with a neighboring block that refers to the same reference picture as that indicated by the flagged reference index (for the block currently being encoded) may have the highest priority to be chosen to form a final candidate of the group. It is possible that none of the neighboring blocks contain a motion vector that points to the same reference picture. Therefore, if such a candidate cannot be found, the video encoder may schedule the first available candidate to form the final candidate, thus temporal distance differences can be compensated.

[00133] In accordance with aspects of this disclosure, motion vector candidates, such as motion vectors associated with neighboring blocks shown in Figures 4A and 4B, can be used to derive a motion vector for a block. For example, the video codec may generate a candidate list that includes motion vector candidates from neighboring blocks shown in Figures 4A and 4B. In this example, the video codec may use one or more of the candidates from the candidate list as an initial motion vector in a motion information derivation process (e.g., bilateral matching, template matching, or the like). The video codec may apply one or more of the motion vector candidates in a motion search of a motion vector derivation process to identify reference data. The video codec may select the candidate from the list that identifies closely matching reference data (e.g., as described with respect to Figures 8-9 below). For example, the video codec may perform motion search for a first set of reference data that corresponds to a second set of reference data outside of the current block. The video codec may, in some cases, further refine the candidate, for example, by performing additional motion search in an area indicated by the selected candidate, to determine a derived motion vector using the motion information derivation process.

[00134] Figures 5A and 5B are conceptual diagrams illustrating temporal motion vector prediction in HEVC. A temporal motion vector predictor (TMVP) candidate, if enabled and available, is added to a MV candidate list after spatial motion vector candidates. In HEVC, the motion vector derivation process for a TMVP candidate is the same for both blending and AMVP modes, however, the target reference index for the TMVP candidate in blending mode is typically set to zero.

[00135] Figure 5A illustrates a primary block location (shown as block “T”) for a TMVP candidate, which is the bottom right block outside of the co-located PU. The location can compensate for bias to the above and left blocks used to generate spatial neighbor candidates. However, if the T block is located outside of the current CTB row or motion information is not available, the block is replaced by a central PU block, as illustrated by the dashed arrows for the T block in Figure 5A.

[00136] Figure 5B illustrates the derivation of a TMVP candidate 84 for a current block 86 of a current picture 88 from a co-located PU 90 of a co-located picture 92, as indicated at the slice level (e.g., in a slice header). Similar to the temporal direct mode in AVC, a motion vector of the TMVP candidate may undergo motion vector scaling, which is performed to compensate for distance differences, e.g., temporal distances between pictures. In connection with motion vector scaling, a video encoder (such as video encoder 20 and/or video decoder 30) may be configured to initially determine that the value of the motion vectors is proportional to the distance of the pictures at presentation time. A motion vector associates two pictures, the reference picture and the picture containing the motion vector (namely, the containment picture). When one motion vector is used to predict another motion vector, the distance of the containment image and the reference image is calculated based on the Picture Order Count (POC) values.

[00137] For a motion vector to be predicted, both the associated containment picture for the motion vector and a reference picture of the motion vector may be different. Therefore, the video encoder may calculate a new distance based on POC values, and the video encoder may scale the motion vector based on these two POC distances. For a spatial neighbor candidate, the containment pictures for the two motion vectors are the same, while the reference pictures are different. In HEVC, motion vector scaling applies both TMVP and AMVP to spatial and temporal neighbor candidates.

[00138] In some examples, a video codec may be configured to determine one or more artificial motion vector candidates. For example, if a motion vector candidate list is not complete, the video codec may generate artificial motion vector candidates and insert the artificial motion vector candidates at the end of the list until the list includes a predetermined number of entries. In blend mode, there are two types of artificial MV candidates that include a combined candidate derived only for B slices and a zero candidate. In some cases, the zero candidate is used only for AMVP if the combined type does not provide enough artificial candidates.

[00139] For each pair of candidates that are already in the candidate list and have required motion information, bidirectional combined motion vector candidates are derived by a combination of the motion vector of the first candidate that refers to an image in list 0 and the motion vector of a second candidate that refers to an image in list 1.

[00140] In accordance with aspects of this disclosure, motion vector candidates, such as the TMVP shown in Figures 5A and 5B, may be used to derive a motion vector for a block. For example, the video codec may generate a candidate list that includes a TMVP determined in accordance with the process described above. In this example, the video codec may use the TMVP as an initial motion vector in a motion information derivation process (e.g., bilateral matching, template matching, or the like). The video codec may apply the TMVP in a motion vector derivation process to identify reference data. The video codec may select the TMVP in cases where the TMVP identifies closely matching reference data (e.g., as described with respect to Figures 8-9 below). The video codec may, in some cases, further refine the TMVP to determine a derived motion vector using the motion information derivation process.

[00141] In some examples, the video codec may prune a candidate list that includes motion vector candidates (such as those described in connection with Figures 4A through 5B). For example, in some cases, candidates from different blocks may be the same, which decreases the efficiency of a merge/AMVP candidate list. The video codec may apply a pruning process to address this issue. The video codec may compare a candidate against others in the current candidate list to avoid inserting an identical candidate. To reduce complexity, the video codec may only apply a limited number of pruning processes rather than comparing each potential candidate to all existing candidates.

[00142] Figure 6 is a conceptual diagram illustrating an example of one-way motion estimation (ME) in frame rate up-conversion (FRUC). In particular, Figure 6 illustrates a current frame 100, a reference frame 102, and an interpolated frame 104. In some cases, a video decoder or post-processing device may interpolate images based on one or more reference images. The video decoder or post-processing device may interpolate images to up-convert an original frame rate of an encoded bitstream. Alternatively, the video decoder or post-processing device may interpolate images to insert one or more images that have been skipped by a video encoder to encode a video sequence at a reduced frame rate. In any case, the video decoder or post-processing device interpolates frames (such as interpolated frame 104) that are not included in an encoded bitstream that was received by the video decoder using images that have been decoded (such as current frame 100 and reference frame 102). The video decoder or post-processing device may interpolate the images using any of a number of interpolation techniques, for example, using motion-compensated frame interpolation, frame repetition, or frame averaging.

[00143] The frame interpolation techniques noted above are typically implemented post-cycle. For example, a video decoder typically receives and decodes an encoded bitstream to generate a reconstructed representation of a video sequence that includes the current frame 100 and the reference frame 102. After the decoding cycle, the video decoder or other post-processing device may interpolate the images to be included with the reconstructed representation that includes the interpolated frame 104. In some cases, the process of interpolating the image may be referred to as frame rate upconversion (FRUC), due to the fact that the resulting sequence of images includes additional (interpolated) images that were not included in the encoded bitstream.

[00144] Consequently, FRUC technology can be used to generate high frame rate videos based on low frame rate videos. FRUC was used in the display industry. Examples include H. Liu, R. Xiong, D. Zhao, S. Ma, W. Gao, “Multiple Hypotheses Bayesian Frame Rate Up-Conversion by Adaptive Fusion of Motion-Compensated Interpolations”, IEEE transactions on circuits and systems for video technology, vol. 22, No. 8, August 2012; W. H. Lee, K. Choi, J. B. Ra, “Frame rate up conversion based on variational image fusion,” IEEE transactions on image processing, vol. 23, No. 1, January 2014; and U. S. Kim, M. H. Sunwoo, “New frame rate up-conversion algorithms with low computational complexity,” IEEE transactions on circuits and systems for video technology, vol. 24, no 3, March 2014.

[00145] FRUC algorithms can be divided into two types. One type of methods interpolates intermediate frames by averaging or simple frame repetition. However, this method provides inadequate results in an image that contains a lot of motion. The other type of method, called motion-compensated FRUC (MC-FRUC), considers object motion when it generates intermediate frames and consists of two steps: motion estimation (ME) and motion-compensated interpolation (MCI). ME generates motion vectors (MVs), which represent object motion using vectors, whereas MCI uses MVs to generate intermediate frames.

[00146] The block matching algorithm (BMA) is widely used for ME in MC-FRUC as it is simple to implement. BMA divides an image into blocks and detects the motion of those blocks, for example, to determine whether the blocks match. Two types of ME are primarily used for BMA: unilateral ME and bilateral ME.

[00147] As shown in Figure 6 , the one-sided ME obtains MVs by seeking the best block match of the reference frame 102 of the current frame 100. Then, the block on the motion path in the interpolated frame may be located such that the MV is achieved. As shown in Figure 6 , three blocks including 106A, 106B, and 106C of the current frame 100, the reference frame 102, and the interpolated frame 104, respectively, are involved in following the motion path. Although the block 106A in the current frame 100 belongs to a code-converted block, the best matching block 106B in the reference frame 102 may not completely belong to a code-converted block, and neither may the block 106C in the interpolated frame 104. Consequently, overlapping regions of the blocks and unfilled regions (holes) may occur in the interpolated frame.

[00148] To handle overlaps, simple FRUC algorithms merely involve averaging and overwriting the overlapping pixels. Furthermore, the holes are covered by the pixel values of a reference or a current frame. However, these algorithms result in blocking and blurring artifacts. Therefore, motion field segmentation, consecutive extrapolation using the distinct Hartley transform, and image retouching are proposed to handle holes and overlaps without increasing the blocking and blurring artifacts.

[00149] In accordance with aspects of this disclosure, a video codec (such as video encoder 20 and/or video decoder 30) may generate interpolated frame 104 in the encoding or decoding cycle using the one-way matching technique shown in Figure 6. For example, the video codec may use picture-level FRUC to interpolate interpolated frame 104 as a predictor of the current picture using the reconstructed pixel array. In some examples, this interpolated picture may be considered a reference picture or reconstruction of the current frame 100. In other examples, the video codec may set the current picture equal to the interpolated picture. This picture may be marked as a discarded picture and/or non-reference picture by syntax elements or decoding processes.

[00150] In some examples, the video codec may interpolate a current picture such that a FRUC mode is the only mode allowed, where the FRUC mode indicates the one-way matching technique shown in Figure 6 or any other motion information derivation or interpolation techniques described herein. Accordingly, instead of a quadratic tree-based CU structure signaling, all blocks may have the same predefined or signaled size. In some cases, only a subset of valid CU codec modes, such as regular skip, regular blend, FRUC mode, and intra mode may be allowed. Alternatively or additionally, a hole-filling mode for FRUC may be allowed in that picture or slice.

[00151] In accordance with aspects of this disclosure, an SEI message may be introduced to indicate which pictures or subsequence of pictures are code-converted by the picture level FRUC. Such pictures may be discarded without impacting the quality of the other pictures. Such an SEI message may indicate (or further indicate) which temporal level (or levels) contains pictures converted to FRUC code, or contains only pictures converted to FRUC code. Such information in an SEI message may also be present as other places in the high-level syntax, such as PPS, SPS, and VPS.

[00152] In some examples, a video encoder may encode a portion of a picture and interpolate the remaining video data. For example, the video encoder may encode a so-called “hint” for decoder-side frame rate upconversion, which may allow feature-rich or intelligent decoders to optionally generate FRUC frames. For example, several key regions (such as rectangle regions) may be signaled as a hint for these FRUC frames. When the hint is received and optionally processed, the FRUC method specified as part of the decoder may be used primarily for the regions that are not key regions, while the key regions may be further processed by means that may not be specified by the decoder, such as hole-filling methods.

[00153] Regarding hole filling, in accordance with aspects of this disclosure, a video codec converter may implement block-based hole filling techniques. For example, one hole filling technique is to use spatially neighboring interblocks to predict the current intrablock in the same CTU. For example, the video codec converter may encode/decode a CTU twice. The first encoding/decoding is as normal. In the second round, only intrablocks are encoded/decoded and overwritten. For an intrablock, all of its spatially neighboring interblocks in the same CTU, which include those to the lower right of the current block, are marked as available for intra prediction. The hole filling method may also be slice, crop, image, or any other level. Another hole filling method may use an image retouching technique. Other hole filling techniques may also be applicable.

[00154] Figure 7 is a conceptual diagram illustrating an example of bilateral motion estimation (ME) in FRUC. In particular, Figure 7 illustrates an interpolated block 108 of an interpolated frame 110 that is interpolated from a current block 112 of a current frame 114 and a reference block 116 of a reference frame 118. As shown in Figure 7, bilateral ME is another solution (in MC-FRUC) that can be used to avoid the problems caused by overlaps and holes shown in Figure 6. Bilateral ME obtains MVs passing through interpolated block 108 by using temporal symmetry between blocks 112 and 116 of current frame 114 and reference frame 118, respectively. As a result, it does not generate overlaps and holes. Since the current block is assumed to be a block that is being processed in a certain order, for example as in the case of video encoding, a sequence of such blocks would cover the entire intermediate picture without overlap. For example, in the case of video encoding, the blocks can be processed in the decoding order.

[00155] In accordance with aspects of this disclosure, the bilateral motion estimation shown in the example of Figure 7 may be leveraged to derive motion information. For example, as described in greater detail with respect to Figure 9 below, a video codec (such as video encoder 20 or video decoder 30) may apply bilateral matching as a motion information derivation mode to derive motion information during codec. In bilateral matching, the video codec may perform a motion search for a first set of reference data in a first reference picture that corresponds to a second set of reference data in a second reference picture.

[00156] In accordance with other aspects of this disclosure, a video codec (such as video encoder 20 and/or video decoder 30) may generate the interpolated frame in the encoding or decoding cycle using the two-way matching technique shown in Figure 7. For example, the video codec may use picture-level FRUC to interpolate the interpolated picture as a predictor of the current picture using the reconstructed pixel array. In some examples, this interpolated picture may be considered as a reference picture or reconstruction of the current picture. In other examples, the video codec may set the current picture equal to the interpolated picture. This picture may be marked as a discarded picture and/or non-reference picture by syntax elements or decoding processes.

[00157] Figure 8 is a conceptual diagram illustrating an example of decoder-side motion vector derivation (DMVD) based on template matching. With advanced video codecs, the bit percentage of motion information in bitstream becomes larger and larger. In some cases, DMVD can reduce the bit cost of motion information. Template matching based DMVD can exhibit an improvement in code conversion efficiency, as described, for example, in S. Kamp, M. Wien, “Decoder-side motion vector derivation for block-based video coding”, IEEE transactions on circuits and systems for video technology, vol. 22, no. 12, December 2012.

[00158] In the example of Figure 8, a current image 120 includes a prediction target 122 (e.g., a block currently being encoded) and a template 124. Reference images 126 include a co-located template 128, a best match 130, and a displacement vector 132. A video encoder (such as video encoder 20 and/or video decoder 30) may use template 124 to search for a best match for prediction target 122 (e.g., rather than using prediction target 122 itself, which is still being encoded). For example, the video encoder may perform a motion search to identify a first set of reference data (e.g., best match 130) that corresponds to a second set of reference data outside of prediction target 122 (e.g., template 124). As noted above, correspondence may be determined based on an amount of similarity between reference data, and may be referred to herein as determining a “match” or “best match.”

[00159] In the example shown, the video codec may identify the co-located template 128 in the reference images 126. The video codec may then search for the best match 130, which includes pixel values that are similar to the template 124. The video codec may determine the displacement vector 132 based on the displacement of the co-located template 128 and the best match 130 in the reference images 126.

[00160] Assuming that the template 124 and the prediction target 122 are from the same object, the motion vector of the template can be used as the motion vector of the prediction target. Accordingly, in the example of Figure 8, the video encoder can apply the displacement vector 132 to the prediction target 122. Since the template matching is conducted at both a video encoder and a video decoder, the motion vector can be derived at the decoder side to avoid the signaling cost.

[00161] In accordance with aspects of this disclosure, the video codec may apply template matching as a mode of deriving motion information. For example, the video codec may apply template matching to derive motion information from a current block by finding a best match between the template 124 of the current picture and corresponding reference data in reference pictures 126. Although the example of Figure 8 illustrates the template 124 as an L-shaped block of video data, it should be understood that other templates may be used. For example, the video codec may use multiple blocks as a template, e.g., one or more blocks positioned to the left of the prediction target 122 and one or more blocks positioned above the prediction target 122.

[00162] In accordance with aspects of this disclosure, the video codec may apply the template matching techniques shown in Figure 8 using one or more motion vectors from a candidate list of motion vectors. For example, the video codec may be configured to determine one or more candidate motion vectors using any combination of the techniques described herein (e.g., blend mode candidates, AMVP candidates, a TMVP, or the like). The video codec may then be configured to apply one or more of the candidate motion vectors to the template 124 to locate the co-located template 128 (in this example, the location of the co-located template 128 is dictated by the one or more candidate motion vectors and is not necessarily strictly co-located). The video codec may be configured to determine which of the candidate motion vectors results in a better match between the template 124 and the co-located template 128.

[00163] In accordance with aspects of this disclosure, the video codec may then be configured to refine the candidate motion vector to derive motion information for the prediction target 122. For example, the video codec may search for a best match for the template 124 in a region of the reference images 126 identified by the candidate motion vector. Upon determining a best match, the video codec may determine an offset between the template 124 and the match based on the determination. The video codec may designate the offset as a derived motion vector for the prediction target 122.

[00164] Figure 9 is a conceptual diagram illustrating an example of bidirectional motion vector derivation in DMVD. Another category of DMVD is mirror-based bidirectional MV derivation, as described, for example, in Y.-J. Chiu, L. Xu, W. Zhang, H. Jiang, “Decoder-side Motion Estimation and Wiener filter for F£EVC”, Visual communications and Image Processing (VCIP), 2013. The concept of bidirectional motion vector derivation in DMVD can be similar to bilateral ME in FRUC. For example, mirror-based MV derivation can be applied for centrosymmetric motion estimation around search centers in fractional sample precision.

[00165] The example of Figure 9 includes the current picture 140 having the current block 142 (where the block is currently encoded), a first candidate motion vector PMV0 identifying a first template block 144 of a first reference picture 146 (ref L0), and a second candidate motion vector PMV1 identifying a second template block 148 of a second reference picture 150. The video encoder may apply dMV as an offset to locate a first reference block 152 in the search window 154 of the first reference picture 146 and to locate a second reference block 156 in the search window 158 of the second reference picture 150.

[00166] For example, the video codec may add dMV to PMV0 and subtract dMV from PMV1 to generate an MV pair, MV0 and MV1. The video codec may check all dMV values within search windows 154 and 158 to determine which dMV value results in the best match between the first reference block 152 (e.g., a first set of reference data) of ref L0 and the second reference block 156 (e.g., a second set of reference data) of ref L1. In some examples, the video codec may determine the best match based on Sum of Absolute Difference (SAD). In other examples, the video codec may use another metric to determine the best match. The size and location of search windows 154 and 158 may be predefined or may be signaled in a bitstream.

[00167] The video encoder may select the MV pair with the minimum SAD as the output of the centrosymmetric motion estimation. Since this technique uses a forward reference (reference at a later temporal position relative to the current frame) and a backward reference (reference at an earlier temporal position relative to the current frame) for SAD matching, it cannot be applied to low-delay P-frame or B-frames where only the latter reference is available.

[00168] In accordance with aspects of this disclosure, the video codec may apply bidirectional motion vector derivation techniques as a mode of deriving motion information. In some examples, the video codec may apply the techniques shown in Figure 9 using one or more motion vectors from a candidate list of motion vectors. For example, the video codec may be configured to determine one or more candidate motion vectors using any combination of the techniques described herein (e.g., blend mode candidates, AMVP candidates, a TMVP, or the like). The video codec may then be configured to apply one or more of the candidate motion vectors such as PMV0 and/or PMV1 to locate the first template block 144 and second template block 148. The video codec may be configured to determine which of the candidate motion vectors results in a better match between the first template block 144 and the second template block 148.

[00169] In accordance with aspects of this disclosure, the video codec may then be configured to refine the candidate motion vector to derive motion information for the current block 142. For example, the video codec may search for a best match by applying a variety of dMV values, in the manner described above. In this manner, the video codec may derive the MV pair MV0 and MV1.

[00170] Figure 11 is a conceptual diagram illustrating motion vector derivation based on extended bilateral correspondence. A potential drawback of mirror-based bidirectional MV derivation (e.g., as shown in Figure 10) is that it does not work when two references of the current picture are both earlier or both later with respect to the current picture. The extended bilateral correspondence techniques described herein can, in some cases, overcome the drawback that all reference pictures of the current picture are on the same side (in display order) as the current picture.

[00171] The example of Figure 11 includes a current picture 160 that includes a current block 162, a first reference picture (Ref0) 164 that includes a first reference block 166, and a second reference picture (Ref1) 168 that includes a second reference block 170. As shown in Figure 11, the first reference picture (Ref0) 164 and the second reference picture (Ref1) 168 are both located before the current picture in the temporal direction. Assuming that the first reference block 166, the second reference block 170, and the current block 162 are along the same motion path, the ratio of MV0 to MV1 should be equal to the ratio of the temporal distance TDO to TD1. In other words, given two references Ref0 and Ref1 with temporal distance TDO and TD1 to the current image, for any MV0 in Ref0, MV1 in Ref1 can be determined by scaling MV0.

[00172] The video codec may select the final MV0 and MV1 pair as the pair that minimizes the matching cost between the paired block pointed to by MV0 and MV1 (e.g., as described above with respect to Figure 10). Theoretically, the current block 162 may be considered as an extrapolated block based on first reference block 166 and second reference block 170. It should be noted that extended bilateral matching also works in the normal bidirectional case where the current picture is temporarily between the two references. In that case, the current block 162 may be considered as an interpolated block based on first reference block 166 and second reference block 170. Furthermore, the bilateral matching techniques described herein do not require a “mirror relationship” between MV0 and MV1, even in the bidirectional case. The bilateral correspondence assumption is that the ratio between MV0 and MV1 is in proportion to the ratio of the temporal distance from Ref0 to the current image to that from Ref1 to the current image.

[00173] Clearly, for reference blocks other than first reference block 166 and second reference block 170, the video codec may derive a different MV pair. In one example, the video decoder may select reference pictures to perform two-way matching according to an order in which the reference pictures appear in a reference picture list. For example, the video codec may select the first reference in reference list 0 as Ref0 and the first reference in reference list 1 as Ref1. The video codec may then search for the MV pair (MV0, MV1). In another example, the video codec selects Ref0 based on an entry in an initial list (e.g., an initial motion vector candidate list). The video codec may then establish Ref1 as a reference picture in the other reference picture list that is temporally closest to the current picture. Consequently, the video encoder can search for the MV pair (MV0, MV1) in Ref0 and Ref1.

[00174] Accordingly, in accordance with aspects of this disclosure, the video codec may apply the extended bidirectional motion derivation techniques illustrated in Figure 11 as a mode of deriving motion information. For example, the video codec may use bilateral matching to derive current block 162 motion information by finding the best match between two blocks (e.g., as first reference block 166 and second reference block 170) along the motion trajectory of the current block in two different reference pictures. Under the continuous motion trajectory assumption, the motion vectors MV0 and MV1 pointing to the two reference blocks, first reference block 166 and second reference block 170, should be proportional to the temporal distances, i.e., TD0 and TD1, between the current picture and the two reference pictures. As a special case, when the current image 160 is temporarily between two reference images (as shown in the example of Figure 10) and the temporal distance from the current image to the two reference images is the same, bilateral matching becomes mirror-based bidirectional MV derivation.

[00175] Figure 11 is a flowchart illustrating an example of decoding a prediction unit (PU) using DMVD. In Y.-J. Chiu, L. Xu, W. Zhang, H. Jiang, “Decoder-side Motion Estimation and Wiener filter for HEVC,” Visual communications and Image Processing (VCIP), 2013, it was further proposed to combine mirror-based bidirectional MV derivation with blending mode in HEVC. In the proposed technique, a flag called pu_dmvd_flag is added to a PU of B slices to indicate whether a DMVD mode is applied to the current PU. Because the DMVD mode does not explicitly convey any MV information in the bitstream, the pu_dmvd_flag syntax element is integrated with the blending mode syntax in HEVC (which uses an index to the data representing a motion vector rather than the motion vector itself).

[00176] In the example of Figure 11, a video decoder (such as video decoder 30) may begin decoding a PU (180). Video decoder 30 may determine whether the mode used to decode the PU is blend mode (182), for example, based on syntax included in a bitstream that includes the PU. If blend mode is not used (the “no” branch of step 182), video decoder 30 may use a regular process for a non-blend PU to decode the PU (184) and terminate process (186).

[00177] If the blend mode is used (the “yes” branch of step 182), the video decoder 30 may determine whether DMVD is used to determine motion information for the PU based on the value of the pu_dmvd_flag syntax element (188). If DMVD is not used (the “no” branch of step 188), the video decoder 30 may use a regular blend mode to decode the PU (190) and terminate the process (186). If DMVD is used (the “yes” branch of step 188), the video decoder 30 may apply a DMVD process to determine motion information for the PU (192) and terminate the process (186).

[00178] In some cases, current video encode techniques may have certain limitations. For example, certain DMVD techniques may be considered a subset of FRUC technology. Although DMVD has been applied in video encode, other effective FRUC techniques have not been implemented in video encode, e.g., in the video encode cycle by a video encoder or video decoder. Furthermore, although different DMVD techniques have been proposed, the interaction and overlap of these techniques is not explored in the decoder. That is, only one mode of DMVD has been used in other video encode systems. How to use a plurality of DMVD techniques to further improve encode efficiency has not been studied.

[00179] As another potential limitation, DMVD can only be applied to relatively large blocks of video data and therefore may not be very effective. Applying these methods to smaller blocks may lead to significant overhead due to signaling cost. In some cases, the search range for traditional DMVD techniques may be relatively small, and only several points are searched, e.g., 16 points. Furthermore, as noted above, mirror-based bidirectional MV derivation cannot be applied in the low-delay B case, due to the fact that two reference pictures with display order before and after a current picture need to be identified, and this is not possible in the low-delay case.

[00180] Another potential limitation may be that, at the decoder, the matching cost of traditional DMVD techniques may only consider distortion. However, the motion vector magnitude has not yet been considered in the matching cost, which may lead to local optimization or inaccurate matching result, e.g., due to noise in the pixel domain. Furthermore, the complexity of traditional DMVD techniques may be relatively high, with respect to both memory bandwidth and computational complexity, especially since interpolation is required for fractional pel motion vectors during search at the decoder.

[00181] The techniques of this disclosure may address one or more of the potential limitations described above. In some examples, the techniques for deriving motion information may be applied individually. Alternatively, any combination of the techniques described herein may be applied together. As described herein, reference index information may generally be considered part of the motion information. In some cases, the motion vector information and reference index information may be referred to as a set of motion information.

[00182] Certain techniques of this disclosure include selecting a motion derivation mode from a plurality of potential motion derivation modes. For example, in accordance with aspects of this disclosure, a video codec converter (e.g., video encoder 20 or video decoder 30) may use two or more motion derivation techniques to provide better prediction of motion information at a video decoder to avoid flagging motion information in the bitstream. The two or more motion derivation techniques may include, but are not limited to, bilateral matching, template matching, and any other matching method. These techniques may generally be referred to as motion information derivation modes or FRUC modes. Accordingly, it should be understood that in some cases a technique referred to as a FRUC mode may instead be used to interpolate motion information into a current block (e.g., rather than interpolating a new block of video data).

[00183] In some examples, when multiple motion derivation methods have been used, rather than having different optimization methods for different derivation methods, the process for finding the best motion for two or more of the motion derivation methods can be aligned, with respect to selecting the starting points to search and how to search around the starting points. For example, the video encoder can construct a list of motion vector candidates, select an initial candidate from the list, and refine the candidate using the same matching and searching techniques. In this example, bidirectional motion derivation and template matching-based motion derivation can be used adaptively on the decoder side.

[00184] In accordance with aspects of this disclosure, additional block-level signaling is introduced to identify which motion derivation method is used to encode the current block. For example, the video codec may encode one or more syntax elements to indicate whether motion information derivation is enabled. The video codec may also encode one or more syntax elements to indicate a particular motion information derivation mode from a plurality of potential modes. In other examples, the motion information derivation technique to be used may not be signaled, but derived at the video decoder, e.g., based on a prediction mode or other information available before the current block is decoded. In still other examples, the video codec may perform multiple motion information derivation modes and determine a weighted average of the predictions from the two or more derivation techniques to encode the current block.

[00185] In one example for purposes of illustration, a video decoder, such as video decoder 30, may first identify whether to apply a motion information interpolation mode (e.g., based on syntax in the bitstream). Video decoder 30 may then identify which motion information derivation mode to apply (e.g., based on syntax in the bitstream). In one example, when multiple motion derivation methods are supported simultaneously, e.g., both extended bilateral matching and template matching modes, an index value may be flagged in the bitstream to indicate which motion derivation method is actually in use for a current block. The index may have three values including, bilateral matching and template matching.

[00186] When using CABAC coding, two bins may be used to represent the index. The two bins may both use spatial neighbors as contexts, or only one of them may use spatial neighbor contexts. Alternatively, one or both bins may use other coded information, such as CU depth, as contexts. Index binarization may be predefined, such as “0” for off, “10” for extended bilateral matching, and “11” for template matching. Alternatively, index binarization may be signaled in a slice header or derived from coded information such as slice type, temporal level, or QP information.

[00187] In accordance with aspects of this disclosure, syntax indicating a motion information derivation mode may be included with another code conversion mode. In one example, a motion information derivation mode may be considered a particular merge candidate, as indicated by a merge index. In this case, the size of the merge candidate list may be increased to accommodate the additional index. The merge candidate index for the motion information derivation mode may be predefined or signaled in the bit stream.

[00188] In some examples, extended bilateral matching and template matching are both supported with blend mode. In these examples, when the blend flag is equal to 1, a new motion information derivation flag is signaled to indicate whether motion information is derived for the current PU. The flag may use the same flag as its spatial neighbor, such as top and left blocks as CABAC code conversion contexts. When this flag is set, a second flag is signaled to indicate which motion information derivation mode (e.g., extended bilateral matching or template matching) is used to derive the block's motion information. When motion derivation is set, even when the current mode is blend mode, no blend index is signaled. Alternatively or additionally, a particular motion information derivation (e.g., such as template matching) is not allowed if the PU is not the first PU of a CU in decoding order. In this case, only the flag needs to be flagged to indicate whether the extended bilateral correspondence by used for the PU or not.

[00189] In another example, the motion information derivation mode may be combined with the AMVP mode, e.g., at the PU level. In one example, additional syntax elements (e.g., an additional flag) may be signaled to indicate the motion information derivation mode. When this mode is enabled, no AMVP index may be signaled in the bit stream. Otherwise, a regular AMVP index may be signaled in the bit stream. In another example, the motion information derivation mode may be considered as a particular AMVP candidate, such as the first AMVP candidate, in an AMVP candidate list. In some cases, when combined with the AMVP mode, a motion vector may not be derived at a sub-PU level.

[00190] According to other aspects, a video codec converter may initially conditionally code a CU level flag (e.g., for an inter-codec CU) to indicate whether all PUs within the current CU use the motion information derivation mode. In one example, the PU level flag is not signaled. In another example, when the CU flag is equal to 0 (i.e., not all PUs are codec converted with the mode), the PU level flag of the first PU is additionally signaled when the second PU does not include the PU level flag.

[00191] In some examples, the motion information derivation mode may be disabled for specific slice types, temporal levels, block types, or block sizes. In one example, motion information derivation is not allowed when the current slice includes only reference pictures whose temporal positions are all before or after that of the current picture. In another example, motion information derivation is not allowed for PUs other than 2Nx2N. When motion information derivation is disabled, no block-level signaling related to motion information derivation is required.

[00192] In accordance with aspects of this disclosure, enabling or disabling the motion information derivation techniques described herein may be controlled by high-level syntax to provide a tradeoff of improved complexity versus code conversion efficiency and/or encoder/decoder flexibility. In one example, a flag may be flagged in an SPS, PPS, slice header, or any other high-level syntax header to indicate the use of a motion information derivation mode. When such a flag indicates that such a code conversion mode is not enabled, the CU/PU level flags may not be code converted in the bitstream.

[00193] In some cases, high-level syntax may additionally or alternatively be used to indicate other motion information derivation parameters. For example, an index of the search algorithm that is to be used to search for the PU-level motion vector may be encoded in a bitstream within an SPS, PPS, or slice header. In some cases, an index of the search algorithm that is to be used to search for the sub-block-level motion vector may be encoded in the bitstream within an SPS, PPS, or slice header. In some examples, to keep computational complexity low on the decoder side, the maximum block/partition match numbers at the PU level, the maximum block/partition match numbers at the PU sub-level, and/or the total match number of both the PU and sub-PU levels may be constrained. In some cases, these maximum numbers may be predefined or signaled in the bitstream.

[00194] The techniques of this disclosure include a variety of techniques for deriving motion information. In some examples, the video encoder may determine an initial list (initial candidate list) of inputs that contain motion vectors, and a best starting point is identified as an input from the initial list. The inputs that contain motion vectors may be motion vectors of spatial neighbors, temporal neighbors, and/or motion vectors derived by other means. Alternatively, the best starting point (or index of the best starting point) may be signaled to the decoder.

[00195] In some examples, the seed list may contain the motion vectors of spatial and/or temporal neighbors. Each entry of the seed list may be a set of uni-predictive motion information, including a motion vector and its reference index. In one example, the seed list may be generated in the same manner as the motion prediction candidate list used in another coding mode, e.g., the same as the merge candidate list. In this case, up to two motion vectors from each merge candidate may be used to generate up to two entries in the seed list. In some cases, the entries in the seed list may be generated from a subset of motion prediction candidates in a list used in another coding mode, e.g., a subset of the merge candidate list.

[00196] In another example, additional motion vectors may be added to the initial list in addition to those in the motion prediction candidate list used in another code conversion mode, e.g., in addition to those in the merge candidate list. Alternatively or additionally, motion vectors of the spatial neighbors of the current block, such as a top block, a left block, a top right block, or another block may be added to the initial list. In some cases, zero motion vectors with different reference image indices may also be added to the list.

[00197] Alternatively or additionally, motion vectors of temporally co-located blocks of the current block (e.g., a TMVP for the current block) and/or motion vectors of temporally co-located blocks to the upper right of the current block in the reference images may be added to the initial list. Prior to adding a particular candidate motion vector to the list, the motion vectors may be scaled or unscaled based on temporal distance.

[00198] According to some aspects, a motion vector may be interpolated and/or extrapolated from a reference picture and may be added to the initial list. For example, prior to coding an interpicture, an interpolated and/or extrapolated motion field of the picture may be generated based on its reference pictures with a technique similar to one-sided ME. The interpolated and/or extrapolated motion field of a block may be used for MV prediction or used as additional initial candidates in a MV search of a motion information derivation mode. Note that the interpolated and/or extrapolated motion field is saved at the unit of 4 x 4 or 8 x 8 or any other predefined/signaled block level, and a PU may contain multiple such blocks such that multiple interpolated and/or extrapolated MVs may be used.

[00199] In one example, the motion field of each of the reference pictures in both reference lists is crossed by the NxN block, where N may be predefined as 4, 8, or 16, or flagged. For each block, if the motion associated with the block passes through an NxN block in the current picture and the block has not been assigned any interpolated motion, the motion of the reference block is scaled to the current picture in the same manner as that of MV scaling in TMVP and the scaled motion is assigned to the block in the current frame. If no scaled MV is assigned to an NxN block, the motion of the block is marked as unavailable in the interpolated motion field. In another example, an NxN block in the current picture may be assigned multiple motion vectors in the interpolated motion field.

[00200] In some cases, a video encoder may remove one or more candidates from a candidate list. Pruning may be applied to remove identical entries from an initial list prior to the best starting point selection process, for example, prior to calculating the matching cost for each candidate in the list.

[00201] In some cases, the first image in each reference list may be used as the reference image, and motion vector candidates are scaled accordingly if necessary. In this case, the reference index of each entry in the initial list may be modified after the motion vector is scaled based on, for example, the POC distance, similar to TMVP. In some cases, the reference index of each entry may be fixed to one or two images, and the associated motion vectors may be scaled to those images.

[00202] In one example, for bilateral matching, a motion vector pair, which is a complete motion information set that contains both motion vectors and their reference indices associated with reference list 0 and list 1, respectively, may be obtained based on each entry of an initial candidate list. A video codec converter may then perform bilateral matching for all MV pairs generated from all entries in the initial list and select the candidate that leads to the minimum matching cost.

[00203] To generate the MV pair, the video codec may select an entry from the initial list as the first motion vector and generate the second motion vector. For example, assume an entry contains the current first motion vector MVA and is associated with a reference index (with POC value POCA) for a first reference picture in reference picture list A (with A being equal to 0 or 1). Assuming that the POC value of the current picture is POCC, the video codec may be configured to find a second reference picture from reference picture list B (with B being equal to 1-A) such that its POC value POCB is equal to (2 X POCC - POCA). If no reference picture in the reference picture list B has POC value equal to (2 X POCC-POC0), the video codec can select the second reference picture by checking all the reference pictures in the list B such that the POCB is not a POCA and the absolute value of POCC-POCB is the minimum. In short, the video codec can select a picture located on the other side of the current picture (in the display order) that has the same POC distance. If it is not available, the video codec can select a picture on the other side that has the smallest distance to the current picture. If all reference pictures are on the same temporal side as the first reference with POCA compared to the current picture, the video encoder can select the reference that is temporally closest to the current picture and has a POC beyond POCA. Under the above assumptions, the video encoder can scale the first MVA motion vector to generate the second motion vector associated with the second reference picture, for example, based on the difference in POC as in TMVP. Alternatively, any techniques based on the bilateral MV assumption can be used to generate the second reference picture and its motion vector.

[00204] According to some aspects, two initial motion vectors may be selected from the initial list. Each of these two motion vectors is selected (with respect to a minimum matching cost) from entries in two subsets of the initial list. Each subset contains motion vectors associated with reference index only to reference image list 0 or only to reference image list 1.

[00205] In accordance with aspects of this disclosure, a video codec converter may be configured to select a candidate from a list of candidates based on a matching cost associated with the motion vector. Further, after selecting a candidate from a list of candidates, the video codec converter may refine the candidate motion vector to derive motion information for a block. For example, the candidate motion vector may be used to indicate a starting point of a reference image, which may then be searched to find a best match based on a matching cost.

[00206] In accordance with aspects of this disclosure, a variety of matching costs may be used, for example, when determining a best match for an initial motion vector candidate and/or when refining the initial motion vector candidate. In one example, when searching for the motion of a block, an MV cost may be included in the matching cost to avoid negative impact by noise. For example, the refinement portion of the motion vector (e.g., difference between the current MV and the search center), denoted as MVR, may be used to calculate the cost. In this example, the cost may be w*(|MVR[0]|+|MVR[1]|), where w is a weighting factor that may be signed or predefined, and MVR[0] and MVR[1] are the two components of MVR. Alternatively, the refined motion vector MV may be used to calculate the cost, for example, as w*(|MV[0]|+|MV[1]|).

[00207] In some examples, when the block is relatively small, e.g., 4 x 4 or smaller, a larger block that covers the block (e.g., the block with extended boundaries) may be used in the matching cost calculation in order to suppress noise. For example, when searching for the best match for a 4 x 4 block, the matching cost may be calculated based on the 8 x 8 block with a center block being the block.

[00208] In some examples, the matching cost may be any type of distance/cost, such as sum of absolute differences (SAD), sum of squared prediction errors (SSE), or sum of absolute transform differences (SATD). To reduce computational complexity, the SAD, SSE, or other cost may be calculated with reduced resolution in the horizontal, vertical, or both directions. For example, for an 8 x 8 block, the SAD may be calculated based on only odd rows. In another example, the matching cost may be calculated based on a selected subset of a block, e.g., only a central region of a may be used.

[00209] In accordance with aspects of this disclosure, the refinement process to the best match (e.g., the candidate selected based on matching cost) may be performed within a predefined or signaled search window rather than always using a small window, e.g., within a 2 x 2 window (in the present context the unit is pixel and fractional motion vectors may be searched within a window) to achieve a low complexity but efficient search. In this example, the range of a search window (e.g., with a size of 16 x 16) may be predefined or signaled in the bitstream.

[00210] The search algorithm for searching may be predefined, such as a full search, a three-step search, a diamond search, a block-based gradient descent (BBGDS) search algorithm as described, for example, in Lurng-Kuo Liu, Ephraim Feig, “A block-based gradient descent search algorithm for block motion estimation in video coding,” IEEE Trans. Circuits Syst. Video Technol., vol. 6, pages 419-422, August 1996, or an unconstrained near-center diamond search (UCBDS) algorithm as described, for example, in Jo Yew Tham, Surendra Ranganath, Maitreya Ranganath, and Ashraf Ali Kassim, “A novel unrestricted center-biased diamond search algorithm for block motion estimation,” IEEE Trans. Circuits Syst. Video Technol., vol. 8, pages 369-377, August 1998.

[00211] In some cases, different search techniques may be used in different cases based on predefined flags or criteria. For example, to search an entire PU, a diamond search may be used. To search a sub-PU, a small diamond search may be used. Additionally or alternatively, early stopping may be applied during the search, for example, when the matching cost is below adaptive or predefined thresholds. When using template matching, a constraint may additionally be applied that the two motion vectors of the two reference lists after refinement will not point to the same block in the same reference image.

[00212] As noted above, this disclosure describes certain optimization techniques for existing DMVD processes. For example, as described above with respect to the example of Figure 10, the techniques include extending bilateral matching from bidirectional prediction to unidirectional prediction. Bilateral matching may also be applied when the temporal positions of two reference images are before or after the temporal position of the current slice (i.e., the POC values are less than or greater than that of the current image). These techniques may be collectively referred to herein as extended bilateral matching.

[00213] Other aspects of this disclosure relate to interpolation techniques. For example, in accordance with aspects of this disclosure, interpolation techniques may be simplified to reduce complexity. As noted above with respect to Figure 2, motion search is typically performed using subpixel precision. Consequently, interpolation is required for non-integer pixel positions. To reduce computational complexity, in accordance with aspects of this disclosure, a video codec may use an interpolation filter with shorter taps compared to normal motion compensation interpolation. In one example, the video codec may use a bilinear interpolation filter during motion search, e.g., when applying initial candidate motion vectors or refining those motion vectors. In another example, the video codec may be configured to initially perform motion search using integer pixel positions, followed by performing motion search at fractional pixel positions with the best integer pixel position match as the starting point.

[00214] Other techniques of this disclosure relate to deriving motion information for sub-blocks. For example, in accordance with aspects of this disclosure, a video codec may divide a block/PU into (non-overlapping) sub-PUs/sub-blocks based on any motion derivation technique. When a PU is divided into multiple sub-PUs of smaller size, the video codec may derive a unique set of motion information for each sub-PU.

[00215] In one example for purposes of illustration, a 32x32 PU may be divided into 16 8x8 sub-PUs. In this example, the video codec may determine different reference indices and/or motion vectors for each of the 8x8 sub-PUs. In other examples, subPUs may have other sizes, e.g., 4x4, 2x2, or 1x1.

[00216] In some cases, the sub-block/sub-PU size may be predefined and fixed independently of the block/PU size. In other examples, a split depth D for PUs may be defined, which controls the number of times a PU may be split according to a quadratic tree structure. In some examples, a minimum sub-PU/sub-block size may be predefined or flagged to indicate the target sub-block/sub-PU size to which the current block/PU should be split. The target size may be the greater of the minimum sub-PU/sub-block size and the size obtained by splitting the current block D times according to a quadratic tree structure.

[00217] In accordance with aspects of this disclosure, a video codec converter may leverage the motion information derived for a PU when deriving motion information for sub-PUs of the PU. For example, the video codec converter may, for each sub-PU, search for its unique motion information by establishing the motion information of the entire block as the search center (starting search point). The video codec converter may then refine the motion for each sub-PU. Alternatively, the search center for each sub-PU may be derived from a list of starting point candidates.

[00218] In another example, a motion vector candidate list may be generated for the sub-PU using any of the techniques described herein. The video codec may then check each candidate in the list after checking the search center (e.g., the starting search point derived from the PU). In accordance with aspects of this disclosure, sub-PUs may be refined using any of the techniques described herein as being applicable to regular blocks. In other examples, refinement may be always or conditionally skipped for sub-PUs after checking the motion vector candidate list for a best match in the manner described above. One possible condition may be whether the best match remains the candidate indicated by the PU's motion information (e.g., the starting search point) after checking the candidate list. If true, the video codec may skip the refinement. Otherwise, the video codec may perform the refinement.

[00219] In some examples, motion vector filtering may be performed for sub-PUs to correct an isolated erroneous motion vector. For example, a video codec converter may use a median filter with the motion vector of the current sub-PU and motion vectors of up to four neighboring sub-PUs. In accordance with aspects of this disclosure, when applying transforms, the video codec converter may consider the entire PU as an entire block such that a transform may cross sub-PU boundaries. In other examples, the video codec converter may apply transforms to each sub-PU such that the transform size is no larger than the sub-PU size. In some cases, when template matching is used, a complete PU may also be further divided into smaller sub-PUs. For sub-PUs whose spatial neighbors are all in the current PU (their templates are not available), the video codec converter may set their motion vectors to the derived motion vectors for the entire PU.

[00220] In some examples, a video codec may encode separate syntax elements that indicate whether a particular block is split. In another example, all 2Nx2N blocks with extended bilateral matching mode are further split into small partitions and no additional flags are signaled. The subblock size may be predefined or signed. Alternatively, the subblock size may be based on the current block size. For example, the subblock size may be derived as the larger of two values. The first value is a predefined or signed minimum subblock size (e.g., such as 4 x 4 or 8 x 8). The second value is a size relative to the current block, such as (S»d)x(S»d) where SxS is the current block size, while d is a predefined or signed value to indicate the quadratic tree depth with which the current block is split into subblocks. In the above examples, it should be understood that the term PU is used interchangeably with the term block and the term sub-PU is used interchangeably with the term sub-block.

[00221] As noted above, the techniques described herein may be used independently or in combination. As an example, a motion information derivation process associated with a motion information derivation mode (e.g., an extended bilateral matching mode, a template matching mode, or any other mode) may include three steps, although the third step may be conditionally performed based on slice type, temporal level, block type, block size, or syntax defined in the bitstream.

[00222] In a first step, a video codec may form an initial candidate list of unipredictive sets of motion information for a current block. The video codec may select the candidate that has the best starting point from the list, for example, based on a matching cost. In a second step, the video codec may refine the selected candidate to derive the motion information for the current block. The refinement may also be based on a matching cost, as described herein. The video codec may then optionally further divide the current block into subblocks as described herein, and refine each motion vector for each subblock. Finally, the video codec may apply motion compensation to the current block using the derived motion information. If division has been performed, the video codec may apply motion compensation on a subblock by subblock basis.

[00223] In one example, the following pseudocode may be used to reconstruct a block using a motion information derivation mode, wherein the motion information is derived by bilateral matching or template matching. When bilateral matching is used, the block is further divided into sub-blocks (note that MV in the present context includes reference information): RECONSTRUCT BLOCK B (WITH SIZE WXH) INTO FRUC Construct initial list if B is bilateral matching { Find the best match (e.g. bilateral matching) in the initial list as the starting point with bilateral matching measurement Refine MV based on the bilateral matching starting point to obtain the motion vector MVB for block B for each sub-block in block B { Use MVB as the starting point, refine MV for each sub-block Do motion compensation for the sub-block with the derived MV information } } else // template matching { Find the best match (e.g. template matching) in the initial list as the starting point with template matching measurement Refine MV based on the template matching starting point }

[00224] Accordingly, in accordance with aspects of this disclosure, video encoder 20 or video decoder 30 may determine a motion information interpolation mode to determine motion information for a current block (a block currently being encoded or decoded). Using the motion information interpolation mode (e.g., two-way matching, template matching, or other technique), video encoder 20 or video decoder 30 may determine the best candidate motion vector from a list of motion vector candidates, e.g., the motion vector that identifies a reference block that closely matches the current block. Video encoder 20 or video decoder 30 may use the candidate motion vector to identify a search window in a reference picture.

[00225] The video encoder 20 or video decoder 30 may refine the candidate motion vector based on a reference block in the search window that closely matches the current block. That is, the video encoder 20 or video decoder 30 may determine a new interpolated motion vector for the current block based on the motion between the reference block in the search window that closely matches the current block and the current block. The video encoder 20 or video decoder 30 may then perform motion compensation for the current block using the interpolated motion vector.

[00226] In some cases, video encoder 20 or video decoder 30 may divide the current block into more than one sub-block for prediction purposes. Furthermore, in other examples, video encoder 20 or video decoder 30 may perform more, fewer, or a different arrangement of techniques for interpolating motion information.

[00227] Accordingly, certain techniques of this disclosure may be generalized as a block-level code conversion tool that leverages certain concepts of FRUC, given the assumption that a current block of a current image may be considered as predicted by reference images in a similar manner as a current image may be considered as interpolated by reference images in FRUC. In one example, only motion-based processes are used for the block-level code conversion tool. In another example, only pixel-based processes are used for the block-level code conversion tool. In another example, either motion-based processes or pixel-based processes are used for a given block. In another example, both pixel-based processes and motion-based processes are used for the block-level code conversion tool. In another example, other syntax may be reused or predicted from the other temporal frames and may be used for the code conversion tool, such as code conversion tree information, SAO, ALF, RQT.

[00228] Figure 12 is a flowchart illustrating an exemplary process for determining a mode of deriving motion information for encoding a block of video data. The exemplary process of Figure 12 is described with respect to a video encoder, which may be configured as a video encoder 20, a video decoder 30, or another processor.

[00229] In the example of Figure 12, the video codec converter may select a motion derivation mode from a plurality of motion derivation modes (200). In general, each of the motion derivation modes may include performing a motion search for a first set of reference data that corresponds to a second set of reference data outside of the current block. For example, with respect to template matching, the video codec converter may perform a motion search to identify a template in a current picture (e.g., a first set of reference data) that corresponds to the template in a reference picture (e.g., a second set of reference data). In another example, with respect to bilateral motion, the video codec converter may perform a motion search to identify a reference block in a first reference picture (e.g., a first set of reference data) that corresponds to a second reference block in a second reference picture (e.g., a second set of reference data). Exemplary motion derivation modes may include, a unilateral motion estimation mode, a bilateral matching mode, a template matching mode, or a mirror-based mode.

[00230] According to some aspects, the video codec converter may select the motion information derivation mode according to one or more syntax elements included in a bit stream. For example, a video decoder may parse and decode the one or more syntax elements from the bit stream and determine the motion information derivation mode based on the syntax. A video encoder may test the plurality of motion information derivation modes, select the mode that has the best RD cost, and encode the one or more syntax elements in the bit stream indicating the selected mode.

[00231] The video codec may determine motion information for the block using the selected motion derivation mode, which may include determining initial motion information using the selected mode (202) and using an optimization process to refine the initial motion information (204). For example, the video codec may construct a motion vector candidate list with candidates for determining initial motion information. The initial motion information may provide a starting point for refining the motion information as described herein.

[00232] The video codec may then encode the block using the determined motion information and without encoding the syntax representing the motion information (206). For example, in cases where the video codec comprises a video decoder, the video decoder may determine a reference block in a reference picture based on the determined motion information, decode residual data for each sub-block from an encoded bitstream, and combine the determined reference block and decoded residual to reconstruct the current block. In cases where the video codec comprises a video encoder, the video encoder may encode residual data for the current block into an encoded bitstream without encode reference indices or motion vectors for the residual data.

[00233] Figure 13 is a flowchart illustrating an exemplary process for deriving a motion vector for encoding a block of video data. The exemplary process of Figure 13 is described with respect to a video encoder, which may be configured as a video encoder 20, a video decoder 30, or another processor.

[00234] The video codec may generate a motion vector candidate list (210). The motion vector candidate list may include one or more motion vector candidates that may be used to derive motion information for the current block. In some examples, motion vector candidates may be determined from spatially neighboring blocks, temporal blocks, or from other locations.

[00235] The video codec may determine a candidate from the list for deriving motion information (212). In some examples, the video codec may perform one or more cost calculations to determine the candidate using a particular motion derivation mode. For example, the video codec may determine a matching cost for a first set of reference data and a second set of reference data, which may include a cost associated with respective motion vectors, as described herein.

[00236] The video codec may then determine the derived motion vector based on the determined candidate (214). For example, the video codec may refine the determined candidate to determine the derived motion vector using the motion derivation mode. The video codec may then encode the block using the derived motion vector (216). For example, in cases where the video codec comprises a video decoder, the video decoder may determine a reference block in a reference picture based on the derived motion vector, decode residual data for each subblock from an encoded bitstream, and combine the determined reference block and decoded residual to reconstruct the current block. In cases where the video codec comprises a video encoder, the video encoder may encode residual data for the current block into an encoded bitstream without encoding the derived motion vector.

[00237] Figure 14 is a flowchart illustrating an exemplary process for deriving motion information for sub-blocks of a block of video data. The exemplary process of Figure 14 is described with respect to a video codec, which may be configured as a video encoder 20, a video decoder 30, or another processor.

[00238] The video codec may derive motion information for a current block (220). In some examples, the video codec may use any combination of motion information derivation techniques described herein to derive the motion information. For example, the video codec may use any of the techniques described herein to perform a motion search for a first set of reference data that corresponds to a second set of reference data outside of the current block (e.g., template matching, bilateral matching, or the like).

[00239] In accordance with aspects of this disclosure, the video codec may also divide the block into a plurality of sub-blocks (222). The video codec may separately derive motion information for respective sub-blocks comprising a motion search for a first set of reference data that corresponds to a second set of reference data out of each respective sub-block (224). Again, the video codec may use any techniques described herein to derive the motion information, such as template matching, bilateral matching, or the like. For example, the video codec may use the derived motion vector as a starting point to derive motion information for each of the sub-blocks, and may further refine the derived motion information using any combination of the motion information derivation techniques described herein.

[00240] The video codec may then code-convert each of the sub-blocks based on derived motion information without code-conversion syntax elements representative of the motion information (226). For example, in cases where the video codec comprises a video decoder, the video decoder may determine a reference block in a reference picture for each sub-block based on the determined motion information, decode residual data for each sub-block from an encoded bitstream, and combine the determined reference block and decoded residual to reconstruct each sub-block. In cases where the video codec comprises a video encoder, the video encoder may encode residual data for each sub-block into an encoded bitstream without code-conversion reference indices or motion vectors for the residual data.

[00241] It should be recognized that, depending on the example, certain actions or events of any of the techniques described herein may be performed in a different sequence, may be added, merged, or left out entirely (e.g., not all of the actions and events described are necessary to practice the techniques). Furthermore, in certain examples, actions or events may be performed concurrently, e.g., through multithreaded processing, interrupt processing, or on multiple processors, rather than sequentially.

[00242] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transmitted as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which correspond to a tangible medium such as data storage media, or communication media, which includes any media that facilitates the transfer of a computer program from one place to another, for example, in accordance with a communication protocol. Thus, computer-readable media may generally correspond to (1) tangible computer-readable storage media that are non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, data structures, and/or code for implementing the techniques described in this disclosure. A computer program product may include computer-readable media.

[00243] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. In addition, any connection is properly referred to as computer-readable media. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are instead intended to refer to non-transitory, tangible storage media. Magnetic disk and optical disk, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray Disc, where magnetic disks generally reproduce data magnetically, while optical disks reproduce data optically with lasers. Combinations of the foregoing shall also fall within the scope of computer-readable media.

[00244] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated into a combined codec. Furthermore, the techniques may be implemented entirely within one or more circuits or logic elements.

[00245] The techniques of this disclosure may be implemented in a wide variety of devices or apparatus, including a handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require performance by different hardware units. Rather, as described above, multiple units may be combined into a codec hardware unit or provided by a collection of interoperable hardware units including one or more processors as described above in combination with suitable software and/or firmware.

[00246] Various examples have been described. These and other examples are within the scope of the following claims.

Claims (11)

1. A method for decoding video data, comprising: selecting a motion information derivation mode from a bilateral matching motion information derivation mode for determining motion information for a current block of a current picture and a template matching motion information derivation mode for determining motion information for a current block of a current picture, the motion information indicating motion of the current block relative to reference video data, wherein the selection of the motion information derivation mode is based on a syntax element contained in a received bit stream, the syntax element indicating the motion information derivation mode that is in use for the current block;; determining an initial list of inputs comprising one or more candidate motion vectors for providing initial motion information relative to a reference picture, wherein the initial list comprises one or more candidate motion vectors for blending mode or one or more candidate motion vectors for advanced motion vector prediction (AMVP) mode; if the bilateral matching mode is selected, selecting from the initial list a first candidate motion vector as initial motion information relative to a first reference image; generating a second candidate motion vector from the first candidate motion vector, the second candidate motion vector identifying a second reference image, the ratio of the first and second motion vectors being equal to the ratio of the temporal distance between the current image and the first reference image and the current image and the second reference image, wherein the first motion vector is selected by performing bilateral matching to determine a motion vector in the initial list that results in the best match between blocks of the first and second reference images; and refining the initial motion information, wherein the refinement comprises performing, using the candidate motion vector as a starting point, a motion search for a first reference block included in the first reference image that corresponds to a second reference block included in a second reference image, wherein the first reference block represents the best match with a reference block of the second reference image, and wherein the search for the best match comprises performing bilateral matching; if the template matching mode is selected, selecting from the initial list a candidate motion vector as initial motion information relative to a reference image, wherein the candidate motion vector is selected by performing template matching to determine a motion vector among the initial list that results in a best match; and refining the initial motion information, wherein the refinement comprises performing, using the candidate motion vector as a starting point, a motion search for a template included in the first reference image that corresponds to a template included in an image containing the current block, wherein the template of the first reference image represents the best match to the template included in the image containing the current block, and wherein the search for the best match comprises performing template matching; determining motion information for the current block using the selected motion information derivation mode by determining a motion vector indicating the best-matching reference data; and decoding the current block using the determined motion information and without decoding syntax elements representative of the motion information. 2. The method of claim 1, further comprising decoding one or more syntax elements indicating whether to perform motion derivation for the current block. 3. The method of claim 1, wherein decoding one or more syntax elements indicating the mode of deriving motion information comprises decoding a flag indicating one of a bilateral matching mode or a template matching mode as the mode of deriving motion information. 4. The method of claim 1, wherein decoding one or more syntax elements indicating the mode of deriving motion information comprises decoding an index indicating the mode of deriving motion information. 5. The method of claim 4, wherein decoding the index comprises context-adaptive entropy decoding of the index. 6. The method of claim 1, wherein selecting the motion information derivation mode comprises selecting a bilateral matching mode, and wherein determining motion information for the current block using the selected motion information derivation mode comprises: performing bilateral matching such that the temporal positions of both reference pictures occur before or after a current picture containing the current block; determining two interpolated motion vectors for the current block based on the reference pictures. 7. The method of claim 1, wherein searching for the best-matching reference data is performed using an interpolation filter that is different from an interpolation filter used to perform motion compensation for the current block; and wherein searching for the best-matching reference data is performed using a bilinear filter to determine pixel values from the first set of reference data or the second set of reference data at a subpixel location. 8. The method of claim 1, wherein decoding the current block comprises: decoding residual data from a coded bit stream, wherein the coded bit stream does not include syntax elements representative of motion information; and combining the decoded residual data and the determined reference block to reconstruct the current block. 9. A method for encoding video data, comprising: selecting a motion information derivation mode from a two-way matching motion information derivation mode to determine motion information for a current block of a current picture and a template matching motion information derivation mode to determine motion information for a current block of a current picture, the motion information indicating motion of the current block relative to reference video data, wherein the selection of the motion information derivation mode is based on a coded syntax element indicating the motion information derivation mode that is in use for the current block; determining an initial list of inputs comprising one or more candidate motion vectors to provide initial motion information relative to a reference picture, wherein the initial list comprises one or more candidate motion vectors for blending mode or one or more candidate motion vectors for advanced motion vector prediction (AMVP) mode; if the two-way matching mode is selected, selecting from the initial list a first candidate motion vector as an initial motion information relative to a first reference picture; generating a second candidate motion vector from the first candidate motion vector, the second candidate motion vector identifying a second reference image, the ratio of the first to the second motion vectors being equal to the ratio of the temporal distance between the current image and the first reference image and the current image and the second reference image, wherein the first motion vector is selected by performing bilateral matching to determine a motion vector in the initial list that results in the best match between blocks of the first and second reference images; and refining the initial motion information, wherein the refinement comprises performing, using the candidate motion vector as a starting point, a motion search for a first reference block included in the first reference image that corresponds to a second reference block included in a second reference image, wherein the first reference block represents the best match with a reference block of the second reference image, and wherein the search for the best match comprises performing bilateral matching; if the template matching mode is selected, selecting from the initial list a candidate motion vector as an initial motion information relative to a reference image, wherein the candidate motion vector is selected by performing template matching to determine a motion vector among the initial list that results in a best match; and refining the initial motion information, wherein the refinement comprises performing, using the candidate motion vector as a starting point, a motion search for a template included in the first reference image that corresponds to a template included in an image containing the current block, wherein the template of the first reference image represents the best match to the template included in the image containing the current block, and wherein the search for the best match comprises performing template matching; determining motion information for the current block using the selected motion information derivation mode by determining a motion vector indicating the best-matching reference data; and encoding the current block using the determined motion vector and without encoding syntax elements representative of the motion vector. 10. An apparatus for encoding video data, comprising: means for selecting a motion information derivation mode from a bilateral matching motion information derivation mode for determining motion information for a current block of a current picture and a template matching motion information derivation mode for determining motion information for a current block of a current picture, the motion information indicating motion of the current block relative to reference video data, wherein selection of the motion information derivation mode is based on a coded syntax element indicating the motion information derivation mode that is in use for the current block; means for determining an initial list of inputs comprising one or more candidate motion vectors for providing initial motion information relative to a reference picture, wherein the initial list comprises one or more candidate motion vectors for the blending mode or one or more candidate motion vectors for the advanced motion vector prediction (AMVP) mode; means for performing the selected motion information derivation mode, the means being configured to: if the bilateral matching mode is selected, select from the initial list a first candidate motion vector as an initial motion information relative to a first reference picture; generate a second candidate motion vector from the first candidate motion vector, the second candidate motion vector identifying a second reference picture, the ratio of the first and second motion vectors being equal to the ratio of the temporal distance between the current picture and the first reference picture and the current picture and the second reference picture, wherein the first motion vector is selected by performing bilateral matching to determine a motion vector in the initial list that results in the best match between the blocks of the first and second reference pictures; and refining the initial motion information, wherein the refinement comprises performing, using the candidate motion vector as a starting point, a motion search for a first reference block included in the first reference image that corresponds to a second reference block included in a second reference image, wherein the first reference block represents the best match with a reference block of the second reference image and wherein the search for the best match comprises performing bilateral matching; if the template matching mode is selected, selecting from the initial list a candidate motion vector as an initial motion information relative to a reference image, wherein the candidate motion vector is selected by performing template matching to determine a motion vector among the initial list that results in a best match; and refining the initial motion information, wherein the refinement comprises performing, using the candidate motion vector as a starting point, a motion search for a template included in the first reference image that corresponds to a template included in an image containing the current block, wherein the template of the first reference image represents the best match to the template included in the image containing the current block, and wherein the search for the best match comprises performing template matching; means for determining motion information for the current block using the selected motion information derivation mode by determining a motion vector indicating the best-matching reference data; and means for encoding the current block using the determined motion vector and without encoding syntax elements representative of the motion vector. 11. Computer readable memory characterized by the fact that it comprises instructions stored therein, the instructions being executed by a computer to perform the method as defined in any one of claims 1 to 9.