CN109076210B

CN109076210B - Video codec method and device

Info

Publication number: CN109076210B
Application number: CN201780026466.0A
Authority: CN
Inventors: 庄子德; 陈庆晔; 孙域晨; 黄毓文; 刘杉; 许晓中
Original assignee: MediaTek Inc
Current assignee: MediaTek Inc
Priority date: 2016-05-28
Filing date: 2017-05-26
Publication date: 2021-04-09
Anticipated expiration: 2037-05-26
Also published as: WO2017206803A1; EP3449630A1; US20190222859A1; TW201804804A; US11089323B2; CA3025490A1; TWI702830B; EP3456043A1; TW201804796A; EP3449630A4; EP3456043A4; CN109076214A; WO2017206804A1; EP3449630B1; US20200322599A1; RU2710667C1; TWI652939B; CN109076210A

Abstract

The present invention provides a method and apparatus for a video encoding system with a current image reference mode enabled. According to one method, if the current image reference mode is selected for luminance and chrominance blocks, the same coding unit structure is used to jointly encode and decode the luminance and chrominance blocks of the current image. Alternatively, if a separate coding unit structure is used to divide the luminance component and the chrominance component into luminance blocks and chrominance blocks, the luminance blocks and the chrominance blocks are encoded or decoded using a coding mode selected from a coding mode group other than the current image reference mode. According to another method, if the current image reference mode is selected for luminance blocks and chrominance blocks, different coding unit structures are used to encode the luminance blocks and chrominance blocks of the current image, respectively. In another method, it is disclosed that in the case where the coding unit is equal to the prediction unit, the reference data is reconstructed for the current image reference mode.

Description

Video coding and decoding method and device

Cross-referencing

The present invention claims priority from united states provisional patent application serial No. 62/342,883, filed on 28/5/2016, the contents of which are incorporated herein by reference in their entirety.

Technical Field

The present invention relates to block partitioning for codec and/or prediction processing in video codec. In particular, the present invention relates to various codec schemes of a codec system for Current Picture Reference (CPR).

Background

The High Efficiency Video Coding (HEVC) standard was developed under the Joint Video project of the ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Moving Picture Experts Group (MPEG) standardization groups, especially in cooperation with what is called the Joint Video Coding Joint collaboration Team (JCT-VC). In HEVC, one slice (slice) is divided into a plurality of Coding Tree Units (CTUs). In the main profile (profile), the minimum size and the maximum size of the CTU are specified by syntax elements in a Sequence Parameter Set (SPS). The allowed CTU size may be 8x8,16x16,32x32, or 64x 64. For each segment, the CTUs within the segment are processed in a sequential scanning (raster scan) order.

The CTU is also partitioned into multiple Coding Units (CUs) to accommodate various local characteristics. A quadtree called a coding tree (coding tree) is used to divide the CTU into CUs. Let the CTU size be MxM, where M is one of 64,32 or 16. The CTUs may be a single CU (i.e., not partitioned) or may be divided into four smaller units of the same size (i.e., each size is M/2xM/2) that correspond to nodes of the coding tree. If a unit is a leaf node of the coding tree, the unit becomes a CU. Otherwise, the quadtree partitioning process may be iterated until the size of the node reaches the minimum allowed CU size specified in the Sequence Parameter Set (SPS). This representation forms a recursive structure specified by the coding tree (also referred to as a partition tree structure) 120 in fig. 1. The partitioning of the CTU110 is shown in fig. 1, where the solid lines represent CU boundaries. The decision to encode a picture region using inter (temporal) or intra (spatial) prediction is made at the CU layer. Since the minimum CU size may be 8x8, the minimum granularity (granularity) to switch between different basic prediction types is 8x 8.

Furthermore, in accordance with HEVC, each CU may be partitioned into one or more Prediction Units (PUs). Together with the CU, the PU serves as a basic representative block of shared prediction information. Within each PU, the same prediction process is applied and the relevant information is sent to the decoder on a PU basis. A CU may be divided into one, two, or four PUs depending on the PU partition type. As shown in fig. 2, HEVC defines eight shapes that decompose a CU into PUs, including partition types 2Nx2N, 2NxN, Nx2N, NxN, 2NxnU, 2NxnD, nLx2N, and nRx 2N. Unlike a CU, a PU can only be partitioned once according to HEVC. The segmentation shown in the second row corresponds to an asymmetric segmentation, wherein the two segments have different sizes.

After the residual block is obtained through the prediction process based on the PU partition type, the prediction residual of the CU may be partitioned into Transform Units (TUs) according to another quadtree structure similar to the coding tree of the CU as shown in fig. 1. The solid line represents CU boundaries and the dashed line represents TU boundaries. A TU is a basic representation block with residuals or transform coefficients for applying integer transform (integer transform) and quantization. For each TU, one integer transform of the same size is applied to the TU to obtain residual coefficients. These coefficients are transmitted to the decoder after TU-based quantization.

The terms Coding Tree Block (CTB), Coding Block (CB), Prediction Block (PB) and Transform Block (TB) are defined to specify a 2-D sample array of one color component associated with the CTU, CU, PU and TU, respectively. Thus, a CTU consists of one luma CTB, two chroma CTBs and associated syntax elements. Similar relationships are valid for CU, PU and TU. Tree splitting is typically applied to both luminance and chrominance simultaneously, although there are exceptions when some minimum size for chrominance is reached.

Alternatively, in JCTVC-P1005 (D.F. Flynn et al, "HEVC Range Extensions Draft 6", Joint photographic Team on Video Coding (JCT-VC) of ITU-T SG 16WP 3and ISO/IEC JTC 1/SC 29/WG 11,16th Meeting: San Jose, US, 9-17 January 2014, Document: JCTVC-P1005), a binary tree splitting structure is proposed. As shown in fig. 3, in the proposed binary tree partitioning structure, a block can be recursively partitioned into two smaller blocks using various binary partitioning types. The most efficient and simplest are the symmetric horizontal segmentation and the symmetric vertical segmentation shown in the first two segmentation types of fig. 3. For a given block of size MxN, a flag is sent to indicate whether the given block is divided into two smaller blocks. If so, another syntax element is issued to indicate which partition type to use. If horizontal segmentation is used, a given block is divided into two blocks of size Mx (N/2). If vertical partitioning is used, a given block is divided into two blocks of size (M/2) xN. The binary tree splitting process may be repeated until the size (width or height) of the split block reaches the minimum block size (width or height) allowed. The minimum block size allowed may be defined in an advanced syntax such as SPS. Since the binary tree has two partition types (i.e., horizontal and vertical), the minimum allowed block width and block height should be noted. Non-horizontal partitioning is implicit when partitioning would result in block heights less than a specified minimum. Non-vertical partitioning is implicit when partitioning would result in block widths less than a specified minimum. Fig. 4 shows an example of a block partition 410 and its corresponding binary tree 420. In each partitioning node (i.e., non-leaf node) of the binary tree, a flag is used to indicate which type of partitioning (horizontal or vertical) is used, where 0 may indicate horizontal partitioning and 1 may indicate vertical partitioning.

The binary tree structure may be used to partition an image region into a number of smaller blocks, e.g., a slice into CTUs, a CTU into CUs, a CU into PUs, or a CU into TUs, etc. A binary tree may be used to partition the CTUs into CUs, where the root node of the binary tree is the CTU and the leaf nodes of the binary tree are the CUs. The leaf nodes may be further processed by prediction and transform coding. For simplicity, there is no further partitioning from CU to PU or from CU to TU, which means that CU equals PU and PU equals TU. Thus, in other words, the leaf nodes of the binary tree are the basic units for prediction and transform coding.

Binary tree structures are more flexible than quad tree structures because more partition shapes can be supported, which is also a source of coding efficiency improvement. However, in order to select the optimal segmentation shape, the encoding complexity will also increase. To balance complexity and coding efficiency, a method of combining quadtree and binary tree structures, also known as quadtree plus binary tree (QTBT) structures, has been disclosed. According to the QTBT structure, blocks are first partitioned by a quadtree structure, and the quadtree partitioning can iterate until the size of the partitioned blocks reaches the minimum allowed quadtree leaf node size. If the leaf quadtree block is not larger than the maximum allowed binary tree root node size, the partitioning may be further partitioned by a binary tree structure, and the binary tree partitioning may iterate until the size (width or height) of the partitioned block reaches the minimum allowed binary tree leaf node size (width or height) or the binary tree depth reaches the allowed maximum binary tree depth. In the QTBT structure, the minimum allowed quadtree leaf node size, the maximum allowed binary tree root node size, the minimum allowed binary tree leaf node width and height, and the maximum allowed binary tree depth may be indicated in an advanced syntax, such as in SPS. Fig. 5 shows an example of the partitioning of block 510 and its corresponding QTBT 520. The solid line represents the quadtree split and the dashed line represents the binary tree split. In each split node (i.e., non-leaf node) of the binary tree, one flag indicates which type of split (horizontal or vertical) is used, 0 may indicate horizontal split, and 1 may indicate vertical split.

The QTBT structure described above may be used to partition an image region (e.g., a slice, CTU, or CU) into a plurality of smaller blocks, such as a slice into CTUs, a CTU into CUs, a CU into PUs, a CU into TUs, and so forth. For example, QTBT may be used to partition a CTU into CUs, where the root node of the QTBT is the CTU, which is partitioned into CUs by the QTBT structure, and the CUs are further processed by prediction and transform coding. For simplicity, there is no further partitioning from CU to PU or CU to TU. This means that CU equals PU and PU equals TU. Thus, in other words, the leaf nodes of the QTBT structure are the basic units of prediction and transformation.

An example of the QTBT structure is shown below. For a CTU of size 128x128, the minimum allowed leaf node size of the quadtree is set to 16x16, the maximum allowed root node size of the binary tree is set to 64x64, the minimum allowed leaf node width and height of the binary tree are both set to 4, and the maximum allowed depth of the binary tree is set to 4. First, the CTU is divided by a quadtree structure, and a leaf quadtree unit may have a size from 16 × 16 (i.e., a minimum allowed quadtree leaf node size) to 128 × 128 (equal to the size of the CTU, without division). If the leaf quadtree element is 128x128, it cannot be further partitioned by the binary tree because the size exceeds the maximum allowed binary tree root node size of 64x 64. Otherwise, the leaf quadtree unit may be further partitioned by a binary tree. The leaf quadtree element is also a root binary tree element, whose binary tree depth is 0. When the binary tree depth reaches 4 (i.e., the maximum allowed binary tree as indicated), no partitioning is implied. When the width of the block of the corresponding binary tree node is equal to 4, non-horizontal partitioning is implied. When the height of the corresponding block of binary tree nodes is equal to 4, non-vertical partitioning is implied. Leaf nodes of the QTBT are further processed by prediction (intra or inter images) and transform coding.

For I-slices, QTBT tree structures typically apply luma/chroma separate coding. For example, the QTBT tree structure applies to the luma and chroma components of an I-slice, respectively, and to the luma and chroma components of a P-slice and a B-slice simultaneously (except for some minimum size to achieve chroma). In other words, in an I-slice, the luma CTB has a block partition of the QTBT structure, and the two chroma CTBs have a block partition of the other QTBT structure. In another example, two chroma CTBs may also have their own block partitioning of the QTBT structure.

For block-based coding, it is always necessary to partition an image into blocks (e.g., CU, PU, and TU) for coding purposes. As is known in the art, prior to applying block segmentation, an image may be segmented into smaller image regions, such as slices, tiles (tiles), CTU lines, or CTUs. The process of dividing an image into blocks for encoding purposes is referred to as dividing the image using a coding unit structure. The method for generating CU, PU and TU by special partitioning adopted by HEVC is an example of a Coding Unit (CU) structure. The QTBT tree structure is another example of a Coding Unit (CU) structure.

Current picture reference

Motion estimation/compensation is a well-known key technique in hybrid video coding that exploits the pixel correlation between neighboring images. In a video sequence, the movement of objects between adjacent frames is small and the movement of objects can be modeled by a two-dimensional translational motion. Thus, the pattern (patterns) corresponding to the object or background in a frame is shifted in position to form a corresponding object in a subsequent frame, or the pattern corresponding to the object or background in a frame is associated with other modes within the current frame. With an estimate of the mobile position (e.g. using block matching techniques), a large part of the pattern can be reproduced without the need to re-encode the pattern. Similarly, block matching and copying are also attempted to allow selection of reference blocks from within the same frame. When applying this concept to video captured by a camera, inefficiencies are observed. In part because text patterns (textual patterns) in spatially adjacent regions may be similar to current coding blocks, but typically have some gradual change in space. Thus, it is difficult for a block to find an exact match within the same image of a video captured by a camera. Therefore, improvement of the coding performance is limited.

However, the spatial correlation between pixels within the same frame is different for the screen content. For typical video with text and graphics, there will usually be a repeating pattern in the same picture. Therefore, it has been observed that intra (picture) block compensation is very effective. New prediction modes, i.e., Intra Block Copy (IBC) modes or so-called Current Picture Reference (CPR), have been introduced to take advantage of this feature for screen content encoding. In the CPR mode, Prediction Units (PUs) are predicted from previously reconstructed blocks within the same frame. In addition, a displacement vector (referred to as a block vector or BV) is used to transmit a relative displacement from the position of the current block to the position of the reference block. The prediction error is then encoded using transform, quantization and entropy coding. An example of CPR compensation is shown in fig. 6, where region 610 corresponds to a picture, slice or picture region to be encoded.

Blocks

620 and 630 correspond to two blocks to be encoded. In this example, each block may find a corresponding block (i.e., 622 and 632, respectively) in a previously encoded region in the current picture. In accordance with the techniques, reference samples correspond to reconstructed samples of a currently decoded picture prior to an in-loop filter operation (SAO) included in HEVC, the in-loop filter operation including a deblocking filter and a Sample Adaptive Offset (SAO) filter.

In JCTVC-M0350(Madhukar Budagavi et al, "AHG 8: Video Coding using Intra-motion compensation", Joint collagen Team on Video Coding (JCT-VC) of ITU-T SG 16WP 3and ISO/IEC JTC 1/SC 29/WG 11,13th Meeting: Incheon, KR, 18-26 Apr.2013, Document: JCTVC-M0350), an early version of CPR was disclosed, which was submitted as a candidate for HEVC Range extension (RExt) development. In JCTVC-M0350, CPR compensation is limited to a small local area, and the search is limited to 1-D block vectors for only 2Nx2N block sizes. Subsequently, a more advanced CPR method was developed during HEVC Screen Content Coding (SCC) standardization.

In order to efficiently transmit a Block Vector (BV), the BV is predictively transmitted using a BV predictor (BVP) in a similar manner to MV encoding. Thus, as shown in fig. 7, a BV difference (BVD) is sent and the BV is reconstructed from BV ═ BVP + BVD, where the reference block 720 is selected to be predicted for the current block 710 (i.e., CU) according to intra block copy (IntraBC). One BVP for the current CU is determined. Methods of deriving Motion Vector Predictors (MVPs) are known in the art. A similar derivation may be applied to BVP derivation.

Turning to fig. 8, fig. 8 shows an example of a constrained reference pixel region for intra block copy (IntraBC) mode (i.e., current image reference, CPR mode). As shown in fig. 8, the reference pixel area must not exceed the picture boundary or overlap the current coding block.

Referring to fig. 9, fig. 9 illustrates an example of a trapezoidal reference data region for wave front parallel processing (WPP) associated with the CPR mode. When WPP is used, the reference pixel area of the CPR mode must not exceed the black line boundary.

When CPR is used, only a part of the current image can be used as a reference image. Some bitstream conformance constraints are applied to adjust the effective MV values of the reference current picture.

First, one of the following two equations must be true:

BV _ x + offsetX + nPbSw + xPbs-xCBs < ═ 0, and (1)

BV_y+offsetY+nPbSh+yPbs–yCbs<＝0。 (2)

Second, the following Wavefront Parallel Processing (WPP) condition must be true:

(xPbs+BV_x+offsetX+nPbSw-1)/CtbSizeY–xCbs/CtbSizeY<＝yCbs/CtbSizeY-(yPbs+BV_y+offsetY+nPbSh-1)/CtbSizeY

(3)

in equations (1) to (3), (BV _ x, BV _ y) are luma block vectors of the current PU (i.e., motion vectors of the CPR); npbssw and npbssh are the width and height of the current PU; (xPBs, yPbs) is the position of the upper left pixel of the current PU relative to the current image; (xCbs, yCbs) is the position of the top left pixel of the current CU relative to the current image; CtbSizeY is the size of the CTU (e.g., corresponding to the height of the CTU). Considering the chroma sampling interpolation for the CPR mode, OffsetX and offsetY are two adjusted offsets in the two-dimensional space:

offsetX＝BVC_x&0x72:0， (4)

offsetY＝BVC_y&0x72:0。 (5)

(BVC _ x, BVC _ y) is a chroma block vector of 1/8 pixel resolution in HEVC.

Third, the reference blocks for CPR must be within the same box/slice boundary.

Affine motion compensation

Affine models can be used to describe 2D block rotations and to deform 2D of a square (rectangle) into a parallelogram. The model can be described as follows:

x’＝a0+a1*x+a2*y，

y’＝b0+b1*x+b2*y。 (6)

in this model, 6 parameters need to be determined. For each pixel (x, y) in the region of interest, a motion vector is determined as the difference between the position of a given pixel (a) and the position of its corresponding pixel in the reference block (a '), i.e. MV ═ a' -a ═ (a0+ (a1-1) × x + a2 ×, b0+ b1 × + (b2-1) ×. Thus, the motion vector of each pixel is position dependent.

According to this model, the above parameters can be solved if the motion vectors for the three different positions are known. This condition is equivalent to 6 parameters being known. Each position with a known motion vector is called a control point. The 6-parameter affine model corresponds to the 3-control-point model.

Some exemplary embodiments of Affine Motion Compensation are presented in the technical literature ("An affinity Motion Compensation frame for high affinity Video Coding", in 2015 IEEE International Symposium on Circuits and Systems (ISCAS),24-27 May 2015, Pages: 525-. In the Li et al technical document, an affine flag is sent for 2Nx2N block partitioning when the current block is encoded in merge mode or AMVP mode. If the flag is true, the derivation of the motion vector for the current block follows an affine model. If this flag is false, the derivation of the motion vector for the current block follows the conventional translation model. When the affine AMVP mode is used, three control points (3 motion vectors) are transmitted. At each control point location, the MV is predictively encoded. The MVDs for these control points are then encoded and transmitted. In the technical literature of Huang et al, predictive coding of MVs at different control point locations and in control points is investigated.

A syntax table for an affine motion compensation implementation is shown in table 1. As shown in table 1, as shown in the notes (1-1) to (1-3) of the merge mode, if at least one merge candidate is affine coding and the partition mode is 2Nx2N (i.e., PartMode ═ PART _2Nx2N), a syntax element use _ affine _ flag is transmitted. As shown in the annotations (1-4) to (1-6) for the B slice, if the current block size is greater than 8x8 (i.e., (log2CbSize >3) and the partition mode is 2Nx2N (i.e., PartMode ═ PART _2Nx2N)), a syntax element use _ affine _ flag is issued. As with comments (1-7) to comments (1-9), if use _ affine _ flag indicates that the affine model is being used (i.e., use _ affine _ flag with a value of 1 is used), the information of the other two control points is sent to the reference list L0, and as with comments (1-10) to comments (1-12), the information of the other two control points is sent for the reference list L1.

TABLE 1

In the present invention, the problem of various aspects of CPR encoding with QTBT structure or luma/chroma separation encoding is addressed.

Disclosure of Invention

The invention provides a video coding method and a video coding device using a quadtree binary tree structure or luminance/chrominance separation coding. According to the proposed embodiments of the present invention, when a Current Picture Reference (CPR) mode is enabled, a luminance component and one or more chrominance components of a current picture are commonly divided into a plurality of luminance blocks and a plurality of color blocks using the same Coding Unit (CU) structure, and if the current picture reference mode is selected for the plurality of luminance blocks and the plurality of chrominance blocks, current picture reference coding is commonly applied to the plurality of luminance blocks and the plurality of chrominance blocks, or the plurality of luminance blocks and the plurality of chrominance blocks are encoded or decoded using a coding mode selected from a group of coding modes other than the current picture reference mode when the luminance component and the one or more chrominance components of the current picture are respectively divided into the plurality of luminance blocks and the plurality of chrominance blocks using separate coding unit structures.

According to another embodiment of the present invention, a luminance component and a chrominance component of a current picture are divided into a plurality of luminance blocks and a plurality of chrominance blocks using separate coding unit structures, respectively. If the current picture reference mode is selected for multiple luma blocks or multiple chroma blocks, respectively, then the current picture reference coding is applied to the multiple luma blocks or the multiple chroma blocks, respectively. For a plurality of luma blocks, the current picture reference means that luma reference data in a reconstructed luma picture of the current picture uses a plurality of first Motion Vectors (MVs). For a plurality of chroma blocks, current picture reference means that chroma reference data in a reconstructed chroma picture of the current picture uses a plurality of second motion vectors. In one embodiment, the plurality of first motion vectors and the plurality of second motion vectors are different. In another embodiment, a third motion vector of the collocated luma block is used to derive a fourth motion vector of the corresponding chroma block, and wherein the third motion vector corresponds to a scaled motion vector of the collocated luma block if the luma component and one or more chroma components have different resolutions. If the current picture reference mode is applied to the corresponding chrominance block and the collocated luminance block is encoded in the current picture reference mode, the scaled motion vector for the third motion vector or the third motion vector of the collocated luminance block is directly used as the fourth motion vector of the corresponding chrominance block. In this case, a flag is used to indicate whether the third motion vector of the collocated luminance block or the scaled motion vector of the third motion vector is directly used as the fourth motion vector of the corresponding chrominance block. The flag is sent or parsed when the corresponding chroma block is encoded in merge mode or when the collocated luma block is encoded in current picture reference mode. If the current picture reference mode is applied to the corresponding chroma block and the collocated luma block is not encoded in the current picture reference mode, the default motion vector is used as the fourth motion vector for the corresponding chroma block. In another embodiment, if the current picture reference mode is applied to the corresponding chroma block and the collocated luma block is encoded in the current picture reference mode, the third motion vector of the collocated luma block or a scaled motion vector of the third motion vector is used as a motion vector predictor added to the merge candidate list or the advanced motion vector prediction candidate list for the corresponding chroma block. The third motion vector of the collocated luma block or a scaled motion vector of the third motion vector is added to a leading candidate position in the merge candidate list or the advanced motion vector prediction candidate list for the corresponding chroma block. If the current picture reference mode is applied to the corresponding chroma block and the collocated luma block is not encoded in the current picture reference mode, the default motion vector is used as a motion vector predictor added to the merge candidate list or the advanced motion vector predictor candidate list for the corresponding chroma block. For example, the default motion vector is selected from the group of motion vectors consisting of (-w, 0), (0, -h), (-w, -h), (-2w, 0), and (0, -2h), where w is the block width and h is the block height. In yet another embodiment, the third motion vector or a scaled motion vector of the third motion vector is used to concatenate the luma block associated with the neighboring chroma blocks of the corresponding chroma block, and the third motion vector or a scaled motion vector of the third motion vector is used as the fourth motion vector of the corresponding chroma block.

According to another aspect of the invention constrained reconstruction data for a current image reference mode is proposed. When the current picture reference mode is applied to the current picture, reference data of the current block of the current picture is constrained to available reconstructed data before predicting the current coding unit including the current block. Accordingly, the block vector of the current block is restricted such that one of the following two equations is true: BV _ x + offsetX + npbssw < 0and BV _ y + offsetY + nPbSh < 0, where (BV _ x, BV _ y) corresponds to a block vector of a luma component of the current block, and (offsetX, offsetY) is two adjusted offsets in view of chroma sample interpolation for the current picture reference mode, and npbssw and nPbSh correspond to the width and height of the current block. The current block may correspond to a current luma block or a current chroma block. The reference data for the current block of the image area is further constrained to the trapezoidal prediction area associated with the wavefront parallel processing, further constraining the block vector of this current block: (xPbs + BV _ x + offset X + nPbSw-1)/CtbSizeY-xPbs/CtbSizeY < ═ yPbs/CtbSizeY- (yPbs + BV _ y + offset Y + nPbSh-1)/CtbSizeY, where (xPbs, yPbs) is the position of the top left pixel of the current block with respect to the current image, and CtbSizeY corresponds to the height of the image region. If the luminance component and the chrominance component of the current region are respectively divided into a plurality of luminance blocks and a plurality of chrominance blocks using separate coding unit structures, respectively, offset x and offset y are 0.

Drawings

Fig. 1 illustrates an example of block partitioning of a coding tree unit into coding units using a quadtree structure.

Fig. 2 illustrates Asymmetric Motion Partitioning (AMP) in accordance with high efficiency video coding, wherein the AMP defines eight shapes for partitioning a CU into PUs.

Fig. 3 illustrates examples of various binary partition types used by a binary tree partition structure, where a block may be recursively partitioned into two smaller blocks using the partition type.

Fig. 4 shows an example of block partitioning and its corresponding binary tree, where in each partitioning node (i.e., non-leaf node) of the binary tree, a flag is used to indicate which partition type (horizontal or vertical) is used, where 0 may indicate horizontal partitioning and 1 may indicate vertical partitioning.

Fig. 5 shows an example of a block partition and its corresponding quadtree plus binary tree structure (QTBT), where the solid lines represent the quadtree partition and the dashed lines represent the binary tree partition.

Fig. 6 shows an example of CPR compensation, where region 610 corresponds to a picture, slice or picture region to be encoded.

Blocks

620 and 630 correspond to two blocks to be encoded.

Fig. 7 illustrates an example of prediction Block Vector (BV) encoding in which a Block Vector Difference (BVD) corresponding to a difference between a current BV and a BV predictor is transmitted.

Fig. 8 shows an example of a constrained reference pixel region for intra block copy (IntraBC) mode (i.e. current image reference, CPR mode).

Fig. 9 illustrates an example of a trapezoidal reference data region for wave front parallel processing (WPP) associated with the CPR mode.

Fig. 10 shows an example of collocated color plane candidate derivation from other color planes in the same frame, where (Y1, U1, V1) and (Y2, U2, V2) are the color planes of two consecutive frames.

Fig. 11 illustrates a flow diagram of an exemplary encoding system with Current Picture Reference (CPR) mode enablement in which luma and chroma blocks of a current picture are jointly encoded using the same coding unit if a CPR mode is selected for the luma and chroma blocks, or are encoded or decoded using a coding mode selected from coding modes other than the CPR mode if the luma and chroma components are respectively partitioned into luma and chroma blocks using separate CU structures, in accordance with an embodiment of the present invention.

Fig. 12 shows a flow diagram of an exemplary encoding system with Current Picture Reference (CPR) mode enabled in accordance with an embodiment of the present invention, wherein if CPR is selected for luma and chroma blocks, luma and chroma blocks of the current picture are encoded separately using different CU structures.

Fig. 13 shows a flow diagram of an exemplary coding system with Current Picture Reference (CPR) mode enabled in accordance with an embodiment of the present invention, wherein if a CPR mode is selected for luma and chroma blocks, the reference data for the current block is constrained to available reconstructed data before predicting the current coding tree unit including the current block.

Detailed Description

The following description is of the best mode for carrying out the invention. This description is made for the purpose of illustrating the general principles of the invention and should not be taken in a limiting sense. The scope of the invention is best determined by the claims.

In video coding based on an original quadtree plus binary tree (QTBT) structure and luma/chroma separate coding, luma and chroma are coded separately for all intra frames (e.g., I-slices). However, in HEVC-SCC, CPR is designed for three color components. The MV of the CPR is used for all three components. In accordance with one aspect of the invention, the CPR design is modified when the luma and chroma are separately encoded. In the present disclosure, various methods of CPR encoding with a luma/chroma separation CU structure are proposed as follows. In the following, various aspects of a CPR mode for video data segmentation with QTBT structure or luma/chroma separation coding are disclosed.

Joint color component coding for CPR mode

When the CPR mode is enabled for the current image, luma and chroma (or R/G/B) separate encoding is disabled. In other words, when the CPR mode is enabled, joint luma and chroma (or R/G/B) coding is used. The use of CPR mode may be indicated by an advanced syntax, e.g., PPS _ curr _ pic _ ref _ enabled _ flag in Picture Parameter Set (PPS) is true. In this case, the luminance and chrominance components will be jointly encoded as in the case of P-or B-slices. In other words, the same Coding Unit (CU) structure is used to partition the luma and chroma components of an image region. Furthermore, the same Motion Vector (MV) or Block Vector (BV) for the luma and chroma components of the image area is used to locate the reference data.

In another embodiment, the CPR mode is disabled if luma and chroma (or R/G/B) separate coding is applied. In other words, the luminance and chrominance components are encoded using an encoding mode selected from an encoding group other than the CPR mode.

In other words, according to the above embodiments, the use of the CPR mode and separate luma and chroma (or R/G/B) encoding do not occur simultaneously.

Separate CPR encoding for luminance and chrominance

According to another embodiment, when luminance and chrominance are coded separately, separate CPR coding is used for the luminance component and the chrominance component. For example, for luma component coding, the MVs of the luma blocks coded in the CPR mode only refer to the luma reconstructed picture. For chroma component encoding, the MV reference for the chroma blocks encoded in the CPR mode is the U/V (or Cb/Cr) reconstructed picture. For example, a luma image region (e.g., a luma slice or coding tree unit, CTU) may be partitioned into luma blocks using a first CU structure, and a chroma image region (e.g., a chroma slice or coding tree unit, CTU) may be partitioned into chroma blocks using a second CU structure. The luma block and the chroma block of the CU/PU may have different MVs if the luma block of the luma picture area has a first MV and the chroma block of the chroma picture area has a second MV. In other words, the first MV and the second MV may be different.

MV prediction for chroma CU coding in CPR mode

In accordance with an embodiment of the present invention, when the luma component and the chroma component are coded separately and separate CPR coding is used for the luma component and the chroma component, either the MV from the luma MV or the scaled luma MV (if the luma component has a different resolution than the chroma component) is used for MV coding of the chroma blocks (e.g., CU/PU). For example, when a chroma CU/PU is encoded using the CPR mode, if the collocated (collocated) luma CU/PU is encoded in the CPR mode, the luma MV or the scaled luma MV may be used directly for the chroma CU/PU or may be used as an MV predictor (MVP) for the chroma CU/PU. Further, luma _ merge _ flag may be issued to indicate whether the luminance MV or the scaled luminance MV is directly used for the current chroma CU/PU. The luma _ merge _ flag may also be conditionally sent. For example, the collocated luma CU/PU is only sent when it is coded in CPR mode. In another example, luma _ merge _ flag is always sent to indicate whether the current chroma CU/PU is coded in CPR mode, and MV is copied from the scaled luma MV. In this example, a default MV may be applied if the collocated luma CU/PU is not coded in CPR mode. The default MV may be (-w, 0), (0, -h), (-w, -h), (-2w, 0), (0, -2h) or any other predefined MV, where w is the CU width or PU width and h is the CU height or PU height.

In another example, luma _ merge _ flag is always sent when merge mode is applied to chroma CU/PU. For example, when the merge _ flag or skip _ flag for the chroma component is true, the luma _ merge _ flag is always transmitted. In this example, a default MV may be applied if the collocated luma CU/PU is not coded in CPR mode. The default MV may be (-w, 0), (0, -h), (-w, -h), (-2w, 0), (0, -2h) or any other predefined MV, where w is the CU width or PU width and h is the CU height or PU height.

In another embodiment, the luminance MV or the scaled luminance MV of the collocated luminance block may be used as the MVP. It may be inserted into the merge candidate list or/and the AMVP candidate list. It may be inserted into the first candidate position (i.e., the leading position) in the list. MV clipping (pruning) may be applied to the following candidates. If the collocated luma CU/PU is not coded in CPR mode, the candidate may be deleted or a default MV may be used as a candidate. The default MV may be (-w, 0), (0, -h), (-w, -h), (-2w, 0), (0, -2h) or any other predefined MV, where w is the CU width or PU width and h is the CU height or PU height.

In another embodiment, the luma MV or the scaled luma MV of the collocated luma block of the current chroma neighboring block may be used as the MVP. It may insert a merge candidate list or/and an AMVP candidate list. If the collocated luma CU/PU is not coded in CPR mode, the candidate may be deleted or a default MV may be used as a candidate.

In another embodiment, the luminance MV or the scaled luminance MV of the collocated luminance block may be used as the temporal collocated MVP.

Juxtaposing color plane candidates

When a block in a color plane is encoded, the decoder may derive its collocated color plane candidate to predict the motion vector of the current block. Collocated color plane candidates are derived from other color planes in the same frame. Fig. 10 shows an example of collocated color plane candidate derivation from other color planes in the same frame, where (Y1, U1, V1) and (Y2, U2, V2) are the color planes of two consecutive frames. When encoding a block in U2, collocated color plane candidates can be derived from Y2. Note that the decoder may also obtain collocated temporal candidates from previous frames encoded in HEVC. The newly derived collocated color plane candidate may be inserted into the candidate list along with other candidates in HEVC. The U/V (or Cb/Cr) color planes in FIG. 10 may be combined. For example, it may be changed to four images corresponding to (Y1, U1+ V1) and (Y2, U2+ V2).

In another embodiment, when encoding chrominance images, the encoded luminance MV field (field) may be used as MV field for temporally collocated images. The original MV coding method in joint color component coding can be applied to chroma image coding.

In another embodiment, when encoding a chrominance image, the encoded luminance MV field is copied to the current reconstructed image. When the CU/PU is decoded, the reconstructed samples and MVs are updated. The original MV coding method in joint color component coding can be applied to chroma image coding.

LM mode for CPR

Since the luminance image and the chrominance image are encoded separately, cross color prediction can be applied to generate a new prediction image for encoding. For example, when decoding a chroma image, a new virtual chroma image for prediction may be generated using the decoded luma image. An LM-like luminance pattern prediction generation method disclosed in HM (HEVC test Model)/JEM (Joint Exploration Model) based software may be used, where prediction _ sample is a source _ sample + b. The parameters a and b may be sent in the SPS/PPS/slice header (SliceHeader). The decoded luminance MV field can be scaled to the MV field of the new virtual chrominance image generated. Bi-directional prediction may be applied. For example, one image may be a currently reconstructed chroma image and the other image may be a virtual chroma image. The trapezoidal prediction region constraint in HEVC-SCC may be mitigated when using virtual chroma images.

CPR reference region constraints for QTBT

When QTBT is used with CUs equal to PU, there is only one prediction block per CU. The constraint variations specified in equations (1) to (3) are as follows:

BV_x+offsetX+nPbSw<＝0， (1’)

BV_y+offsetY+nPbSh<＝0， (2’)

(xPbs+BV_x+offsetX+nPbSw-1)/CtbSizeY–xPbs/CtbSizeY<＝yPbs/CtbSizeY-(yPbs+BV_y+offsetY+nPbSh-1)/CtbSizeY。

(3’)

the above modified equation is also applicable to the case where a Coding Unit (CU) structure is used to generate CUs, where each CU corresponds to one PU. When three color components of an image are separately encoded, respectively, where chroma U/V has an independent block vector, the variables offset x and offset y may be set to 0.

The above disclosed invention may be incorporated in various forms in various video encoding or decoding systems. For example, the present invention may be implemented using hardware-based methods, such as application specific Integrated Circuits (ICs), field programmable logic arrays (FPGAs), Digital Signal Processors (DSPs), Central Processing Units (CPUs), etc. The invention may also be implemented using software code or firmware code executed on a calculator, laptop, or mobile device such as a smart phone. Further, the software code or firmware code may be executed on a hybrid platform, such as a CPU with a dedicated processor (e.g., a video coding engine or a co-processor).

Fig. 11 shows a flow diagram of an exemplary codec system with Current Picture Reference (CPR) mode enablement in accordance with an embodiment of the present invention, wherein a luma block and a chroma block of a current picture are jointly encoded using the same coding unit structure if a CPR mode is selected for the luma block and the chroma block, or are encoded or decoded using a coding mode selected from a group of coding modes other than the CPR mode if a separate CU structure is used to divide the luma component and the chroma component into the luma block and the chroma block, respectively. The steps shown in the flow diagrams, as with other alternative flow diagrams in this application, may be implemented as program code executable on one or more processors (e.g., one or more CPUs) at the encoder side and/or the decoder side. The steps shown in the flowcharts may also be implemented on a hardware basis, such as one or more electronic devices or processors arranged to perform the steps in the flowcharts. According to the method, input data associated with a current image is received in step 1110, wherein the current image region includes a luma component and one or more chroma components. On the encoder side, the input data may correspond to video data to be encoded. On the decoder side, the input data may correspond to compressed video data to be decoded. In step 1120, when a current image reference (CPR) mode is enabled, a luminance component and the one or more chrominance components of the image region are jointly divided into a luminance block and a chrominance block using the same Coding Unit (CU) structure, and if a CPR mode is selected for the luminance block and the chrominance block, CPR coding is commonly applied to the luminance block and the chrominance block, or the luminance block and the chrominance block are encoded or decoded using a coding mode selected from a group of coding modes other than the CPR mode when the luminance component and the one or more chrominance components of the image region are respectively divided into the luminance block and the chrominance block using separate CU structures. If the CPR mode is selected for the luminance block and the chrominance block, the luminance block and the chrominance block are encoded using reference data in the current image. When a picture is partitioned using a Coding Unit (CU) structure, the picture is typically partitioned into slices, blocks, CTU lines, or CTUs, and the Coding Unit (CU) structure is applied to the CTUs to generate CUs, PUs, and TUs for the encoding process. A quad-tree plus binary tree (QTBT) structure may be used as the coding unit structure.

Fig. 12 shows a flow diagram of an exemplary codec system with current picture reference mode enablement in which luma and chroma blocks of a current picture are separately coded using different CU structures if a CPR mode is selected for the luma and chroma blocks in accordance with an embodiment of the present invention. According to the method, input data associated with a current image is received in step 1210, wherein an image region includes a luma component and one or more chroma components. In step 1220, the one luma component and the one or more chroma components of the image region are partitioned into luma blocks and chroma blocks, respectively, using separate CU structures. In step 1230, if a CPR mode is selected for the luma block or the chroma block, current image reference (CPR) encoding is applied to the luma block or the chroma block, respectively. For example, CPR encoding may reference only luma reconstructed reference data with motion vectors of luma blocks for encoding luma components and CPR encoding may reference only chroma reconstructed reference data with motion vectors of chroma blocks for encoding chroma components. That is, for luminance blocks, applying current picture reference means that luminance reference data in a reconstructed luminance picture of the current picture uses first motion vectors; for chroma blocks, applying the current picture reference means that chroma reference data in a reconstructed chroma picture of the current picture uses second motion vectors. In one embodiment, the first motion vector and the second motion vector are different.

FIG. 13 shows a flow diagram of an exemplary codec system with current picture reference mode enablement in accordance with an embodiment of the invention in which reference data for a current block is constrained to available reconstructed data prior to predicting a current coding tree unit that includes the current block if a CPR mode is selected for luma and chroma blocks. In accordance with this embodiment, input data associated with a current image is received in step 1310, where the current image includes a luma component and one or more chroma components. In step 1320, the luma component and the one or more chroma components of the current picture are partitioned into luma and chroma blocks using a Coding Unit (CU) structure, wherein the Coding Unit (CU) is equal to a Prediction Unit (PU). In step 1330, when the Current Picture Reference (CPR) mode is applied to the luma and chroma blocks, the reference data of the current block is constrained to the available reconstructed data before predicting the current coding unit including the current block according to BV _ x + offset x + npbssw < > 0, or BV _ y + offset y + nPbSh < > 0. The current block corresponds to a current luma block or a current chroma block, (BV _ x, BV _ y) corresponds to a current block vector of a luma component of the current block, and (offsetX, offsetY) are two adjusted offsets, and nbbsw and nbbsh correspond to a width and a height of the current block, in view of interpolation of chroma samples for the CPR mode, and wherein if the luma component and the one or more chroma components of the current picture are divided into a luma block and a chroma block, respectively, using a separate coding unit structure, offsetX and offsetY are 0.

The flow diagrams depicted are intended to illustrate examples of exemplary video codecs according to the present invention. One skilled in the art may modify, rearrange, split, or combine steps to practice the invention without departing from the spirit of the invention. In this disclosure, specific syntax and semantics have been used to illustrate examples of implementing embodiments of the present invention. Those skilled in the art can practice the invention with the same syntax and semantics in place of those described without departing from the spirit of the invention.

The previous description is presented to enable any person skilled in the art to practice the invention in the context of a particular application and its requirements. Various modifications to the described embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments. Thus, the present invention is not intended to be limited to the particular embodiments shown and described, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, those skilled in the art will appreciate that the present invention may be practiced.

The embodiments of the present invention described above may be implemented in various hardware, software codes, or a combination of both. For example, an embodiment of the invention may be one or more circuits integrated into a video compression chip or program code integrated into video compression software to perform the processing described herein. Embodiments of the present invention may also be program code to be executed on a Digital Signal Processor (DSP) to perform the processes described herein. The invention may also relate to a number of functions performed by a computer processor, digital signal processor, microprocessor or Field Programmable Gate Array (FPGA). The processors may be configured to perform certain tasks in accordance with the invention by executing machine-readable software code or firmware code that defines certain methods embodied by the invention. Software code or firmware code may be developed in different programming languages and in different formats or styles. Software code may also be compiled for different target platforms. However, different code formats, styles and languages of software code, and other ways of configuring code to perform tasks consistent with the present invention will not depart from the spirit and scope of the present invention.

The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims

1. A video encoding and decoding method for use by a video encoding system and a video decoding system, the method comprising:

receiving input data associated with a current image, wherein the current image comprises a luma component and one or more chroma components;

dividing the luma component and the one or more chroma components of the current picture into a plurality of luma blocks and a plurality of chroma blocks using separate coding unit structures, respectively, wherein a first coding unit structure used for the luma component is different from a second coding unit structure used for the chroma components; and

if a current picture reference mode is selected for the luma blocks or the chroma blocks, respectively, the current picture reference mode is applied to the luma blocks or the chroma blocks, respectively.

2. The video coding and decoding method according to claim 1,

for the luma blocks, applying the current picture reference mode means that luma reference data in a reconstructed luma picture of the current picture uses first motion vectors.

3. The video coding and decoding method according to claim 2,

for the chrominance blocks, applying the current picture reference mode means that chrominance reference data in a reconstructed chrominance picture of the current picture uses second motion vectors.

4. The video coding and decoding method according to claim 3,

the plurality of first motion vectors and the plurality of second motion vectors are different.

5. The video coding and decoding method according to claim 3,

the third motion vector of the collocated luma block is used to derive a fourth motion vector of a corresponding chroma block, and wherein the third motion vector corresponds to a scaled motion vector of the collocated luma block if the luma component and the one or more chroma components have different resolutions.

6. The video coding and decoding method of claim 5, wherein if the current picture reference mode is applied to a corresponding chroma block and the collocated luma block is coded in the current picture reference mode, the scaled motion vector of the third motion vector or the third motion vector for the collocated luma block is directly used as the fourth motion vector of the corresponding chroma block.

7. The video coding and decoding method according to claim 6,

a flag is used to indicate whether the third motion vector of the collocated luma block or the scaled motion vector of the third motion vector is directly used as the fourth motion vector of the corresponding chroma block.

8. The video coding and decoding method according to claim 7,

the flag is transmitted or parsed when the corresponding chroma block is encoded in a merge mode or when the collocated luma block is encoded in the current picture reference mode.

9. The video coding and decoding method according to claim 5,

if the current picture reference mode is applied to the corresponding chroma block and the collocated luma block is not encoded in the current picture reference mode, a default motion vector is used as the fourth motion vector for the corresponding chroma block.

10. The video coding and decoding method according to claim 5,

if the current picture reference mode is applied to the corresponding chroma block and the collocated luma block is encoded in the current picture reference mode, the third motion vector of the collocated luma block or the scaled motion vector of the third motion vector is used as a motion vector predictor added to a merge candidate list or an advanced motion vector prediction candidate list for the corresponding chroma block.

11. The video coding and decoding method according to claim 10,

the third motion vector of the collocated luma block or the scaled motion vector of the third motion vector is added to a leading candidate location in the merge candidate list or the advanced MVP candidate list for the corresponding chroma block.

12. The video coding and decoding method according to claim 5,

if the current picture reference mode is applied to the corresponding chroma block and the collocated luma block is not encoded in the current picture reference mode, a default motion vector is used as a motion vector predictor added to a merge candidate list or an advanced motion vector predictor candidate list for the corresponding chroma block.

13. The video coding and decoding method of claim 12,

the default motion vector is selected from the group of motion vectors consisting of (-w, 0), (0, -h), (-w, -h), (-2w, 0) and (0, -2h), where w is the block width and h is the block height.

14. The video coding and decoding method according to claim 5,

the third motion vector or the scaled motion vector of the third motion vector is used for the collocated luma block associated with neighboring chroma blocks of the corresponding chroma block, and the scaled motion vector of the third motion vector or the third motion vector is used as the fourth motion vector of the corresponding chroma block.

15. A video coding and decoding apparatus for use by a video coding system and a video decoding system, the apparatus comprising one or more electronic circuits or one or more processors configured to:

applying a current picture reference mode to the luma blocks or the chroma blocks if the current picture reference mode is selected for the luma blocks or the chroma blocks, respectively.

16. A video encoding and decoding method for use by a video encoding system and a video decoding system, the method comprising:

dividing the luma component and the one or more chroma components of the current image into a plurality of luma blocks and a plurality of chroma blocks using a plurality of coding unit structures; and

when the current image reference mode is applied to the luma blocks and the chroma blocks, reference data for the current block is constrained to available reconstructed data that is available prior to predicting a current coding unit that includes the current block according to the following formula:

BV _ x + offsetX + nPbSw < (equal to 0), or

BV_y+offsetY+nPbSh<＝0，

Wherein the current block corresponds to a current luma block or a current chroma block, (BV _ x, BV _ y) corresponds to a current block vector of the luma component of the current block, and (offsetX, offsetY) are two adjusted offsets, and nbbsw and nbbsh correspond to a width and a height of the current block, in view of chroma sample interpolation for the current picture reference mode, and wherein the fsoffsetx and the offsetY are 0 if the plurality of luma blocks and the plurality of chroma blocks are respectively divided into a plurality of luma blocks and a plurality of chroma blocks using separate coding unit structures.

17. The video coding and decoding method of claim 16,

the reference data of the current block are further constrained to the trapezoidal prediction region associated with the wavefront parallel processing, further constraining the block vector of the current block:

(xPbs+BV_x+offsetX+nPbSw-1)/CtbSizeY–xPbs/CtbSizeY<＝

yPbs/CtbSizeY-(yPbs+BV_y+offsetY+nPbSh-1)/CtbSizeY，

where (xPs, yPbs) is the position of the top-left pixel of the current block relative to the current picture, and CtbSizeY corresponds to the height of the current coding tree unit.

18. A video coding and decoding apparatus for use by a video coding system and a video decoding system, the apparatus comprising one or more electronic circuits or one or more processors configured to:

receiving input data associated with an image region in a current image, wherein the image region comprises a luma component and one or more chroma components;

dividing the luminance component and the one or more chrominance components of the current region into a plurality of luminance blocks and a plurality of chrominance blocks using a plurality of coding unit structures; and when the current picture reference mode is applied to the picture region, constraining reference data for a current block of the picture region to available reconstructed data available prior to predicting a current coding unit including the current block according to the following formula:

BV _ x + offsetX + nPbSw < (equal to 0), or

BV_y+offsetY+nPbSh<＝0，

Wherein (BV _ x, BV _ y) corresponds to a current block vector of the luma component of the current block, (offsetX, offsetY) are two adjusted offsets, and nPbSw and nPbSh correspond to a width and a height of the current block, in view of chroma sample interpolation for the current picture reference mode, and wherein the offsetX and the offsetY are 0 if the plurality of luma blocks and the plurality of chroma blocks are respectively divided into a plurality of luma blocks and a plurality of chroma blocks using separate coding unit structures.