MXPA06003284A - Video comfort noise addition technique - Google Patents

Abstract

A decoding arrangement (10, 100, 1000) for decoding pictures in an incoming video stream includes a noise generator (16 ) for adding a dither signal containing random noise to the pictures after video decoding, to improve the subjective video quality. The noise generator adds noise to each pixel in an amount correlated to the luminance of at least a portion of the current picture.

Description

VIDEO COMFORT NOISE ADDICTION TECHNIQUE CROSS REFERENCE WITH RELATED REQUESTS This application claims priority in accordance with the 35 U.S.C. 119 (e) of Provisional Patent Application Serial No. 60 / 505,254, filed September 23, 2003, which is hereby incorporated by reference in its entirety.

FIELD OF THE INVENTION The invention relates to a technique for reducing artifacts in connection with the decoding of a coded video stream.

BACKGROUND OF THE INVENTION The decoding of a video stream compressed at a low bit rate tends to cause visible artifacts for the viewer. Structured block and noise patterns are common artifacts that arise when using block-based compression techniques. The human visual system is more sensitive to certain types of artifacts, and therefore, such artifacts are more noticeable and annoying than others. The addition of random noise to the decoded stream can reduce the notoriety of such compression artifacts, but the larger frame-to-frame differences created by adding random noise can produce artifacts that are more noticeable and annoying.

The addition of an oscillation signal can reduce human sensitivity to such artifacts, for example, to hide contour and block artifacts. A prior art has proposed to add a random oscillation noise that is based on the film grain for an image to disguise the effects of the block. The reason for adding such random noise is that the random error is much more permissible than the structure or correlated error. Other prior techniques have proposed adding an oscillation signal to the video stream to hide the compression artifacts. A prior art has proposed to add a random noise oscillation in the video encoding and decoding process in the cycle unlock filter for the ITU / ISO H.264 video coding standard, commonly referred to as the JVT coding standard. The amount of oscillation to be added depends on the position of a pixel with respect to the edge of the block. Another prior art proposes to add random noise after video decoding (ie add noise as a "post-process") to be used as comfort noise. The amount of noise added depends on the quantization parameter and the amount of noise added to spatially neighboring pixels. The term "comfort noise" comes from the use of noise in audio compression to indicate a noise pattern generated at the receiving end to avoid total silence that is uncomfortable for the listener.

Previous techniques for reducing artifacts by adding noise typically reduce spatial artifacts at the risk of creating temporary abnormalities, that is, large frame-by-frame differences. This is why there is a need for a technique for reducing artifacts during the decoding of a coded video stream that overcomes the aforementioned disadvantages.

BRIEF DESCRIPTION OF THE INVENTION Briefly, in accordance with a preferred embodiment of the present principles, a method is provided for reducing artifacts in a video stream during decoding. The method begins by decoding a video stream. After decoding, the noise is added to the video stream by adding the noise to each pixel in an amount correlated with the luminescence of at least one previously decoded image. Thus, in accordance with the present principles, the luminescence correlation helps to determine the additive noise to reduce the large differences frame by frame, a disadvantage of the previous noise additive techniques.

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 illustrates a schematic block diagram of a video decoder according to a first embodiment of the present principles, for reducing artifacts in connection with the decoding of a coded video stream, by adding noise correlated with the luminescence of at least a portion of the current image.

Figure 2 illustrates a schematic block diagram of a video decoder according to a second embodiment of the present invention, for reducing artifacts in connection with the decoding of a coded video stream, by adding the noise correlated with the luminescence of at least a portion of the current image; and Figure 3 illustrates a schematic block diagram of a video decoder in accordance with a third embodiment of the present principles, for reducing artifacts in connection with the decoding of a coded video stream by adding noise correlated with the luminescence of minus a portion of the current image.

DETAILED DESCRIPTION OF THE INVENTION In accordance with the present principles, adding an oscillation signal containing random noise to an already decoded signal, in an amount correlated with the additive noise of pixels in a previous image, serves to improve the subjective quality of the video . Up to here, it has been found that adding noise to a decoded signal improves the quality of the video signal. The visual impact of adding a noise signal to a video sequence, rather than just a single image, becomes a consideration in determining the magnitude of the noise signal. The magnitude of the additive noise signal for a pixel in an image can be correlated with the value of the additive noise signal of the pixels in the previously displayed image, for example, the noise signals are temporally correlated. Alternatively, the temporal correlation may be based on the previously decoded image, rather than the previously displayed image. Based on the foregoing, the added noise signal, with the use of temporal correlation with a correlation factor a, 0 < _a < .1, can be computed as: N (k, x, y) = (1 - o) * N (k-1, x, y) + a * R (k, x, y) (1) The random number R (k, x, y) it can be generated with the use of any type of random number distribution, for example, the normalized, Gaussian, Laplacian distribution. The R (k, x, y) can be attached with a certain interval if necessary. The random number generator can be implemented by means of a look-up table. The R (k, x, y) may also include the spatial correlation, such as that used for example, in the generation of film grain noise. In accordance with the present principles, the addition of noise appears somewhat dependent on the brightness (ie, luminescence) of a block or macroblock, but also that of its adjacent blocks. The darker the block / macroblock, the easier it will be to perceive the noise with a slight variation. On this basis, the amount of additive noise N (k, x, y) can be determined by the relationship: N (k, x, y) = (1 -? (K, x, y)) * N (k-1, x, y) +? (K, x, y) * (1 - (k5 x, y )) * R (k, x, y) (2) With the function? (K, x, y) that represents a correlation factor dependent on the temporal correlation of the current image with the previously displayed or decoded one. The term? (K, x, y) can be computed as: ? (k, x, y) = a - ß * // (D (k, x, y), D (k-1, x, y)), 0 < ß < a < l (3) where f -i () takes the values between 0 and 1 and computes the temporal correlation factor of the pixel (x, y) in the image k with its co-located pixel in the image k-1. The factors a and ß are related to a type of image (image I. P or B) as well as to the quantifier used to encode the current image or block, and can be calculated with the use of a look-up table. Alternatively, the full resolution difference image between images k and k-1 can be used and two images can be considered as correlated (i.e., f? = 1) when the absolute total difference is below a value. It is possible to consider simpler metrics. For example, considerable savings in storage and computation may occur by considering the average of the NxN blocks in place and carrying out noise adaptation at the block level. In this case, I (D (kJx, y), D (k-l, x, y)) the term will be equal to: N tf / 1 (D (k, x! Y), D (kl, x, y)) = (- ™ -? S (?? D (k, x + k, y + k) NXN k = m =? (4) - ?? D (k- + k, y + k)) &? 0WA) Muid where 0 < < ffl < 255. The term f (k, x, y) reflects the spatial information to adjust the intensity of the noise that will be used. In particular, f (k, x, y) will be computed as: f (k, x, y) = 2 (D (k, x, y)) + f3f (, x-bsx, y), D (k, x + bsx, y), D (k, x, y- bsy ), D (k, x, y + bsyj) (4) where f2 () is related to the brightness of the current pixel or the NxN block to which it belongs, while f3 () computes the spatial relationship between the current pixel / block and its adjacent horizontal or vertical at a distance of bsx or i sy. For example, (D (k, x, y)) =? L1Q: 1) (5) / 3 (D (k, x, y)) = (/ 2 (D (k, X > y)) -? ( D (klx + Ny)) = ll (? (D (k, x, y)) - 2 (D (k, x-Ny)) =: r3) (6) il (/ 2 (D (k, x , y)) - 2 (D (k, x> y + N)) = -4) ll ((D (k, x, y)) - / 2 (D (k; x1y-N)) = 5) , _, 0 = ?} < 255, and -255 < ^ 3,? 3,? 4,? 5 < 255. where Figure 1 shows a schematic block diagram of a first embodiment of a video decoder array 10 for adding noise correlated with the luminescence of at least a portion of the current image in a manner compatible with Equation 2 to reduce artifacts. The decoder array 10 includes a decoder 12 for decoding the incoming encoded video stream. The design of the decoder 12 depends on the compression format used to encode the incoming video stream. In a preferred embodiment, the incoming video stream undergoes compression with the use of the well known ITU / ISO H.264 standard. Under such circumstances, the decoder 12 takes the form of an H.264 decoder known in the art. A reference image storage 14 stores the images decoded by the decoder 12 with the use of the decoder in the decoding of future images. The decoder 12 supplies a noise generator 16 with a decoded image and with the bitstream information contained in the decoded image. The output of the bitstream information by the decoder 12 may include a quantization parameter input for the noise generator. The severity of the compression artifacts correlates with the quantization parameter, and the more severe compression artifacts are presented when the high values of the quantization parameter are used. The strength of added comfort noise can be increased as the value of the quantization parameter increases. An adder block 18 adds each decoded image from the decoder 12 with the noise from a noise generator 16. An attacher 20 then appends the resulting signal emitted by the summing block 18 to produce a decoded image for display, which exhibits reduced artifacts. It should be noted that the addition of noise occurs after the storage of the decoded images in the reference image storage 14, since the reference images must remain unchanged in order to properly decode the subsequent incoming images. A noise image storage 17 stores the noise signal N (k.x.y) for the image k for later use by the noise generator 16. The noise generator 16 responds to the reference images stored in the reference image storage 14, which contains information about the previously decoded images. Although not necessary, additional storage can be added when the compute-based block is used for temporal correlation between decoded images. While the generation of noise for each pixel within an image remains possible, in certain cases (for example, for a higher resolution material), the generation of noise of larger size (grain) often proves to be more desirable. For example, when applying a discontinuous cosine transformation of block size NxN in the noise image, and then discard the resulting higher frequencies will produce a larger noise similar to the noise of the film grain. However, this process incurs a relatively high expense and will typically require an unlocking process in order to reduce artifacts that may be generated at the edges of the block. Figure 2 illustrates a block schematic diagram of a second embodiment of an array 100 of the decoder for adding large grain noise correlated with the luminescence of at least a portion of the current image. The array 100 of the decoder of Figure 2 contains elements in common with the array 10 of the decoder of Figure 1. Therefore, like numbers indicate similar elements. Compared with the decoder array 10 of FIG. 1, array 100 of FIG. 2 also includes a reduced image average storage 22 NxN coupled with the reference image storage 14. The image storage 22 typically stores the average values of the luma block NxN. The average luma values stored in the image storage 22 allow the decoder array to generate large grain noise as described. Figure 3 illustrates a schematic block diagram of a third embodiment of the 1000 array of the video decoder for adding large grain noise correlated with the luminescence of at least a portion of the current image. The array 1000 of the decoder of Figure 3 includes many of the same elements as those of the array 100 of the decoder of Figure 2, and similar reference numbers identify similar elements. Compared to the array 100 of the decoder of FIG. 2, the decoder array 1000 of FIG. 2 does not contain an image storage 17, but only an average image storage 22 NxN. An alternative and simpler process would be to generate the noise at a lower resolution than that of the original image (for example, medium horizontal and vertical resolution) and then sample upwards the noise (for example, with the use of a sample replication). The use of a lower resolution or original resolution can be decided based on the resolution of the original images (for example, with the use of the same resolution for the standard definition and a lower definition material, while using a generation of lower resolution noise for high definition material). The lateral parameters can also be transmitted with the bit stream that will allow the decoder to decide the process that should be used. The lateral information can be used for the generation of noise (for example, weight variation of noise). This same process can be applied to the color components. However, to reduce complexity and computation, noise generation can occur based only on one luminescence component (ie, luma) and while the same noise is reused in all color components, after performing a simple scaling and sub-sampling, when necessary. For example, for a 4: 2: 0 material, luma noise is sub-sampled horizontally and vertically by 2 and can be divided by 2 in order to generate chroma noise. Arrays 10 and 100 of the decoder of Figures 1 and 2 represent cases of an infinite impulse response (HR) filter. The IIR filter can be generalized by using more filter intakes. MR filters can generally be approximated by the use of higher order FIR filters with the use of so many t-taps, as desired in accordance with the following relationship: N (k, x, y) = l (l-? (Kj, .x, yy) xN (kt, x, y) (7)? (Kj, x, y)) x? (K ~ i, x , y) x (lf (k ~ i, x, y)) xR (ki, x, y) A measurement of a finite impulse response (F1R) filter can be implemented with the use of the decoder array of Figure 3. The decoder array 1000 only makes use of the previous random R numbers, and when necessary, the values average luma block NxN, better than the previous N noise, in such a FIR filter measure, which reduces the memory bandwidth. It is also possible to use and store only the average values of the luma NxN block of the previous and current images and to reuse the same values and their differences for all the shots. For example, the following system can be used: N (k, x, y) = (l ~? (K, x, y)) x (l ~? (K - i, x, y)) xR (k-2, x, y) + (l- ? (k, x, yy) x? (k -y, x, y) x (lf (kl, x, y)) xR (k- • > (8) +? (k, x, y) x (lf (k, x, y)) xR (k, x, y) although it is also possible to simplify the above by forcing the difference images used in the computation of ^ ~ '' ') for. . and (k, x, y),,? ? • _, -i J be the same as. This will completely avoid the need to store or re-compute the difference image and greatly reduce the memory bandwidth. The foregoing describes a technique for reducing artifacts in connection with the decoding of a decoded video stream by adding noise correlated with the luminescence of at least a portion of the current image.

Claims (15)

1. A method for reducing artifacts in a video stream, characterized in that it comprises the steps of: decoding a video stream; and adding random noise to at least one pixel in an image in the video stream after decoding in an amount correlated with the luminescence information of at least a portion of a current image.

The method according to claim 1, characterized in that it also comprises the step of correlating the noise with the use of a factor dependent on the temporal correlation of the current image with one of the previously displayed or decoded image.

3. The method according to claim 2, characterized in that the correlation factor is established in accordance with one of a color or luma component.

4. The method according to claim 2, characterized in that it further comprises the step of adding noise to a color component of the image in accordance with the luma component.

The method according to claim 2, characterized in that the correlation factor is first established on a basis of pixel image block NxN (where N is an integer) before the additive noise interpolation.

6. The method according to claim 1, characterized in that it further comprises the step of adjusting the noise based on the intensity of an NxN block (where N is an integer) of adjacent pixels.

The method according to claim 1, characterized in that the amount of noise is correlated with the use of an approximation of a finite impulse response (IR) filter.

8. A decoder arrangement for decoding a video stream encoded to produce small artifacts, characterized in that it comprises: a video decoder for decoding an incoming encoded video stream to produce decoded images; a reference image storage for storing at least the previously decoded image for use by the decoder when decoding future images; a noise generator for generating random noise to be added to at least one pixel in the decoded image in an amount correlated with the luminescence information of at least a portion of the current image; a noise image storage for storing the noise information for later use by the noise generator; an adder block to sum the noise generated by the noise generator with a decoded image of the decoder; and an adjuster to append the added noise and the decoded image.

9. The decoder arrangement according to claim 8, characterized in that the noise generator implements an initiation of an infinite impulse response filter.

10. The decoder arrangement according to claim 8, characterized in that the noise generator implements an approximation of an infinite impulse response filter.

The decoder arrangement according to claim 8, characterized in that the noise generator generates noise in accordance with the decoded images and the bitstream information supplied from the decoder.

12. The decoder arrangement according to claim 8, characterized in that the bitstream information comprises a quantization parameter.

13. The decoder arrangement according to claim 8, characterized in that it also includes a second noise image storage for storing the average of the block image of the pixel NxN, where N is an integer, to be used by the generator of noise.

14. A decoder arrangement for decoding a coded video stream to produce small artifacts, characterized in that it comprises: a video decoder for decoding an incoming encoded video stream to produce decoded images; a reference image storage for at least one storage of at least one previously decoded image for use by the decoder in decoding future images; a noise generator for generating noise in accordance with the decoded images and the bitstream information from the decoder to add it to at least one pixel in the decoded image in an amount correlated with the additive noise of at least one pixel in a previous image; an image storage for storing an average pixel block image NxN, where N is an integer to be used by the noise generator; and an adder block to sum the noise generated by the noise generator with a decoded image from the decoder.

15. The decoder arrangement according to claim 14, characterized in that the noise generator implements an initiation of the finite impulse response filter.