WO1999030507A1

WO1999030507A1 - Video signal processing

Info

Publication number: WO1999030507A1
Application number: PCT/GB1998/003693
Authority: WO
Inventors: Martin Weston; William Beningfield Collis
Original assignee: Snell & Wilcox Limited
Priority date: 1997-12-10
Filing date: 1998-12-10
Publication date: 1999-06-17
Also published as: GB9726134D0; AU1496399A; GB2332324A; GB2332324A9

Abstract

A composite video decoder has a separation filter serving to separate luminance and chrominance information. The filter takes weighted sums of pixels and products of triplets of pixels to provide non-linear behaviour without adaption. A method is given for calculating filter coefficients.

Description

VIDEO SIGNAL PROCESSING

This invention relates principally to the decoding of composite video signals into components. The technology of line and field combs for decoding is well known.

Essentially, line or field delays are employed to make available signals that (for a constant colour region of a still picture) have identical colour information but inverted chrominance phase. Adding these signals and dividing by two will generate a luminance signal, whilst a similar subtraction process will generate chrominance. With "real" video signals, there are of course variations in colour from line to line and movements between one field and the next. Decoders have therefore been developed which adapt between different luminance/chrominance separation filters in accordance with detected motion. Filters are also employed which take linear combinations of pixels that are more sophisticated than the simple comb.

Experience has shown that high quality decoding of composite video signals can only be achieved with complex signal processing. This is understandably expensive to produce, both in design and in manufacture. What is more, conventional decoders - even those taken to be of high quality - can still create visually disturbing artefacts.

It is an object of one aspect of the present invention to provide an improved composite video decoder that offers high quality decoding on a wide range of picture material with minimal artefacts.

Accordingly, the present invention consists in one aspect in a composite video decoder having a separation filter serving to separate luminance and chrominance information, the filter taking weighted sums of pixels in a filter aperture; characterised in that the filter incorporates weighted sums of products of pixels.

Preferably, the filter incorporates weighted sums of products of triplets of pixels. Suitably, the filter comprises weighted sums of pixels and products of triplets of pixels.

Advantageously, the separation filter operates on a demodulated signal, an output of the separation filter being remodulated. In a preferred form of the invention, first and second different filters are provided for operation on respective adjacent video fields.

In a further aspect, the present invention consists in a method of decoding a composite video signal to components, comprising training and decoding modes, wherein in a training mode a luminance/chrominance separation filter taking weighted sums of products of pixels in a filter aperture is fed a composite signal at least one constituent component of which is known, and the filter weightings are optimised to minimise an error between said known component and an output of the filter; and wherein in a decoding mode the luminance/chrominance separation filter with optimised parameters operates on a composite signal.

In still a further aspect, the present invention consists in a composite video decoder for use with a film-originating composite video signal having a first video field and a second video field derived from each film frame, the decoder having two separation filters operating to separate luminance and chrominance information, the filters taking weighted sums of pixels in respective different filter apertures; one of the said separation filters operating on said first video fields and the other operating on said second video fields.

The invention will now be described by way of example with reference to the accompanying drawings, in which:-

Figure 1 is a diagram illustrating a four tap, third order polynomial filter;

Figure 2 is a block diagram of a decoder according to this invention; - 3 -

Figure 3 is a diagram illustrating a training process according to this invention;

Figure 4 is a diagram illustrating a 5 field filter aperture; and

Figure 5 is a diagram illustrating an intra-frame aperture.

The currently accepted approach to the complex problem of decoding a generalised composite signal is to make decoders data adaptive, that is to say switching between linear filters in a given set of filters, based on an appropriate function of the data. The major difficulty in designing a data adaptive filter is in choosing the adaptation parameters; that is, knowing when to switch to a different filter.

According to an aspect of the present invention, an alternative approach is taken based on a non-linear filter in which polynomial combinations of pixel values in the aperture are formed. A general polynomial filter of infinite order is capable of modelling a large range of nonlinear functions, but to design a realisable polynomial filter it is necessary to truncate at some arbitrary order. In the example discussed here, for reasons to be discussed, this is chosen to be done at the third order. This produces a relatively smooth non-linear function, as it will contain only functions of straight lines, quadratics and cubics, and as such is capable of modelling mild non-linearities. Such a non-linear filter can remain continuously in service, in the sense that that there is no need for adaption according to the video input. In a traditional linear filter each filter tap is multiplied by a filter weight and the resulting products are summed to give the filter output. In contrast, a polynomial non-linear filter, in addition to the linear terms, includes the sum of filter coefficients multiplied by products of pairs of pixel values, triplets of pixel values, and so on. For example, a four tap polynomial filter will contain four filter coefficients that are multiplied by single pixel values, ten filter coefficients that are multiplied by products of pixel values, twenty filter coefficients that are multiplied by triplets of pixel values, and so on. - A -

In any practical embodiment, the polynomial series must be truncated at some point. A filter truncated at the third order is convenient and there is shown diagrammatically in Figure 1 a four tap third order filter. This is illustrated graphically as the combination of a linear filter 100, a quadratic filter 102 and a cubic filter 103. The linear filter utilises three delay elements 104 to generate four taps from the input signal. Each tap is multiplied by a coefficient in a respective multiplier 105 and a weighted sum generated in summing device 106. In the quadratic filter 102, similar delay elements 114 provide four taps from the input video signal and ten multipliers 115 generate all ten possible products, again weighted by respective coefficients. A sum is formed in summing device 116. The cubic filter has twenty multipliers 125 operating on the taps from delay elements 124 to generate all possible combinations of triplets of taps and a weighted sum is formed in summing device 126. Although the delay elements 104, 114 and 124 have been shown separately in the three filters, one set of delay elements would usually suffice.

It will be understood that in any practical circuit there are very many ways of embodying the described filter. Typically, a single processing element will receive the four taps and with appropriate multipliers, coefficient stores and one or more summing devices, output directly the sum of the linear, quadratic and cubic filters.

Since the filter output will contain linear, quadratic and cubic terms, it is able to model systems which contain both quadratic and cubic non-linear elements. These generate both skewed and symmetric distortions of the probability density function. Higher order models can be used and should give improved results, but the size of the filter and the computation required in its estimation rise exponentially and there are rapidly diminishing returns. For example, the fifth order six tap filter contains over five times as many parameters as the third order six tap filter. A decoder according to this invention will now be described, based on an otherwise conventional decoder structure. Referring to Figure 2, a composite television signal (for example NTSC or PAL) at input terminal 200 is high pass filtered in high pass filter 202. The high frequency signal is then demodulated in demodulator 204, to produce a demodulated chrominance signal that will contain cross colour. This is passed to a four tap third order polynomial filter 206 which performs the function illustrated in Figure 1. The input signal is also passed through a complementary low pass filter 208 to generate a low frequency luminance signal.

The polynomial filter 206 separates the chrominance from the cross colour. The chrominance C is output directly. The cross colour is passed to remodulator 208 and added in adder 210 to the low frequency luminance (which is passed through a suitable compensating delay 212) to give wide band luminance Y.

It will be recognised that other strategies can be adopted, preserving the advantage of the luminance/chrominance separation filter operating on a base band signal. For example, a chrominance output from the separation filter could alternatively be remodulated for subtraction from a full bandwidth composite signal to derive luminance.

Traditionally processing techniques have used some form of linear filter, which takes weighted sums of horizontally, vertically, and/or temporarily displaced pixels of the contaminated chrominance to form an estimate of the pure chrominance. More advanced decoders use adaptive techniques, i.e. they allow different combinations of filter taps to be used in different areas of the picture. The new decoder described here utilises a filter based on the weighted sum of a polynomial expansion of sums of horizontally, vertically and temporally displaced pixels. When designing a linear decoder, the problem is deciding what filter weights and filter taps to use in order to minimise cross colour whilst not introducing artefacts, such as hanging dots. In designing a non-linear decoder using a polynomial filter, the analogous problem exists. A typical high quality linear decoder may contain 9 such filter weights the value of which can be calculated by empirical means. An equivalent cubic, non-linear filter would contain 220 parameters.

To avoid the problem of empirically choosing values for such large numbers of parameters, a further aspect of the present invention provides an automated method of calculating the filter weights.

Referring to Figure 3, a training sequence in the form of a component digital video signal is provided to a conventional encoder 300. The composite output from this encoder provides the input x to the decoder 302 according to the present invention. Attention is focussed on the chrominance output z of the decoder and this is compared in block 304 with the desired chrominance output y, which is simply the chrominance signal of the training sequence input. The output of block 304 is the difference between the actual filter output y and the desired output d and the optimisation process generates optimal filter coefficients for the decoder 302 to minimise the square of this difference.

In practice, the desired signal y is the chrominance signal uncontaminated by cross colour. This is obtained from the encoder by taking the modulated and bandpass filtered chrominance before the luminance is added to it and then demodulating and lowpass filtering. This will result in a chrominance signal with no cross colour. In contrast the input to the decoder contains the chrominance signal which has been encoded with the luminance and so will contain cross colour. The objective is to design a filter that minimises the mean squared error between its output and the desired signal. This should force the output of the decoder towards the pure chrominance signal.

Such a scheme will create a set of filter coefficients that produce the optimum decoder, in the sense of least mean square error, for that particular sequence. There are many error functions that could be minimised to design the optimum filter. The mean squared error is just one example, others include, mean fourth error, just noticeable difference, etc. A major advantage of using the mean squared error is that it generates a quadratic error function that can be solved by a simple closed form solution. Other possible error functions will require some form of iterative solution that will be far more complex and require much more computational power.

The choice of training sequence is obviously of importance to this technique. It should be representative of the material the decoder is likely to see in practice. If the decoder is only being trained by a software simulation, the number of frames that can be processed is limited. A hardware trainer would allow the decoder to be trained on a much larger data set. If the decoder is trained on one particular sequence it may become 'over trained'. That is, its parameters have adapted to that particular picture and may work extremely well with it, but when the decoder is presented with sequences that it has not been trained on, it may fail. It is therefore important to test the decoder on a different sequence to that with which it was originally trained. The way in which the filter coefficients are generated will now be described.

The object is to design an N point digital finite impulse response filter, h, to modify the input, x(n), in such a way as to minimise the mean square error, e(n), between the filter output z(n) and the desired signal, y(n). In the case of decoding,

x(n) is the demodulated, band pass filtered chrominance, contaminated with cross colour

z(n) is the output of the decoder

y(n) is the desired signal, i.e. the pure chrominance signal with no cross colour.

The filter impulse response which minimises the sum of the squared errors of data of length L, is given by the solution of the over-determined (assuming L>N) system of equations

Xh = y

where,

and,

The least squares solution of which is,

h = (X^TX)^"1X^Ty.

Note X^TX and X^TY are usually much smaller than X. Hence, it is much more efficient to compute X^TX and X^TY directly from x(n) and y(n) rather than to form X.

The same method can be extended to obtain an estimate of the N^th order non-linear filter. For the sake of this explanation, a second order polynomial filter which contains only linear and quadratic terms will be considered, although it is conceptually easy to extend the method to any arbitrary order. The extension of this to a more general polynomial model is in principle simply a matter of modifying the data matrix X. Below we show the data matrix for a second order polynomial filter, in which a constant (DC) term has also been included. A symmetric form for the filter has been assumed so this matrix has dimension :

1 χ(L) x(L-l) - x(L-N+l) x(L)² x(L)x(L-l) ••• x(L)x(L-N+l) x(L-l)² • • • x(L-N+l)² 1 x(L-l) x(L-2) ... x(L-N) x(L-l)² x(L-l)x(L-2) •• • x(L-l)x(L-N) x(L-2)² •• • x(L-N)² 1 x(L-2) x(L-3) ... x(L-N-l) x(L-2)² x(L-2)x(L-3) ... x(L-l)x(L-N-l) x(L-3)² ••^■ x(L-N-l)²

x =

1 x(3) x(2) - 0 x(3)² x(3)x(2) - 0 x(2)² -. 0

1 x(2) x(l) - 0 x(2)² x(2)x(l) - 0 x(l)² -. 0

1 x(l) 0 ••• 0 x(l)² 0 0 0 0

The optimal filter, in the least squares sense, can then be estimated by computing (X^TX)^"1X^Ty- The filter will contain three separate components, the DC term, the standard linear coefficients which should be multiplied by single pixel values, and the quadratic coefficients which will be multiplied by products of pixel values.

This approach will produce a decoder that will provide the optimally decoded output for the particular sequence used to calculate the filter coefficients. It is found that if a relatively long sequence is used (that is to say L is very large), with that sequence containing a wide variety of picture material, a decoder is produced with provides results which are superior to the prior art over a wide range of inputs. An alternative to using a single long input sequence, is to use a number of shorter sequences, each exemplifying a different category of picture material, and to combine the respective auto correlation matrices X^TX and the respective cross correlation matrices X^τy, before generating the filter coefficients h according to:

h = (X^TX)^"1X^Ty

Consideration now turns to the choice of aperture. It is in theory possible to choose a very large aperture, i.e. a large number of input taps to the polynomial filter, and allow the optimisation procedure to select whichever inputs are of the most use. Any filter taps that are of no use will be forced towards zero. In practice this is not preferred for two reasons. Firstly, the optimisation procedure is limited by the available computing power. Secondly, if there are too many inputs to the filter, the system may become ill- conditioned and the optimisation will fail. Therefore in practice it is important to choose the correct aperture for the decoder. In the preferred form of the invention, taking into consideration the interlace structure of television sequences, two filters are trained, one for field 1 and the other for field 2. In one approach, it is could be assumed that these two filters should be statistically similar and so their associated correlation matrices could be added together to produce one filter which was the same for field 1 and field 2. Experimentally, it is found that where the test sequence consists of a number of frames, the filters for field 1 and field 2 are substantially different. Such a frame based input would arise for example with a video signal that had been originated as film.

It is accordingly proposed that a decoder for use with film-originating material be provided with different filters for the respective fields associated with a single film frame. With a 50Hz television signal derived from 24 frames per second film, two filters will suffice. With a 60Hz signal derived from film using the 3:2 pull down process, more thyan two filters will be required.

It is envisaged that if the two filters for a 50Hz decoder are generated in a training process, they will tend to take most information from - in one case - the preceding field, and - in the other case - the succeeding field. The combined effect will be to give precedence to picture information from the film frame associated with the current video field. Where filters are designed manually, this bias towards the film frame can be in-built. Although this feature is particularly useful with filters incorporating a weighted sum of products of pixels, it is expected to have wider application.

It will be understood that with "true" video material, a distinction is still drawn between field 1 and field 2 and - in editing and other post production processes - it is conventional to position cuts between video frames rather than between field 1 and field 2. Accordingly, in video material which contains a large number of cuts, there will be an advantage if the filter aperture gives precedence to picture information from the video frame associated with the current video field, and thereby reduces the risk of irrelevant information being taken from across a video cut.

Two particularly preferred apertures are shown in Figures 4 and 5. In each case the filters for both field 1 and field 2 are shown. Figure 4 shows a 5 field aperture with two additional horizontal filter taps. This has been found to perform extremely well although it produces a filter with a large number of parameters. An 11 tap third order polynomial filter will have 363 parameters. The pixels in the outer two fields reduce the cross colour significantly but unlike its linear counterpart the non-linear filter is able to adapt to reduce trailing dots caused by chroma smear. It will be seen that the taps are the same for the two fields although thew coefficients will differ.

Figure 5 shows is a slightly smaller aperture which is confined to a single frame. The taps are different for the two fields, one looking forward, the other backward. This aperture does not have quite such a dramatic effect on cross colour but the reduced number of temporal taps allows a slightly larger horizontal component in order to help with horizontal dots.

Simulations have shown that an optimised decoder according to the present invention can perform substantially better than current state of the art decoders. A polynomial non-linear filter could be implemented using a very large number of multipliers and adders or more simply as a lookup table. Although large, its structure is very simple and repetitive and so it may be easier to implement in silicon than an adaptive architecture.

Claims

1. A composite video decoder having a separation filter serving to separate luminance and chrominance information, the filter taking weighted sums of pixels in a filter aperture; characterised in that the filter incorporates weighted sums of products of pixels.

2. A decoder according to Claim 1 , wherein the filter remains continuously in circuit.

3. A decoder according to Claim 1 or Claim 2, wherein the filter incorporates weighted sums of products of triplets of pixels.

4. A decoder according to Claim 3, wherein the filter comprises weighted sums of pixels and products of triplets of pixels.

5. A decoder according to any one of the preceding claims, wherein the separation filter operates on a demodulated signal, an output of the separation filter being remodulated for combination with a composite signal.

6. A method of decoding a composite video signal to components, comprising training and decoding modes, wherein in a training mode a luminance/chrominance separation filter according to Claim 1 is fed a composite signal at least one constituent component of which is known, and the filter weightings optimised to minimise an error between said known component and an output of the filter and wherein in a decoding mode the luminance/chrominance separation filter with optimised parameters operates on a composite signal.

7. A composite video decoder for use with a film-originating composite video signal having a first video field and a second video field derived from each film frame, the decoder having two separation filters operating to separate luminance and chrominance information, the filters taking weighted sums of pixels in respective different filter apertures; one of the said separation filters operating on said first video fields and the other operating on said second video fields.

8. A composite video decoder according to Claim 7, wherein each separation filter incorporates weighted sums of products of pixels.

9. A decoder according to Claim 7 or Claim 8, wherein the separation filters remain continuously in circuit.

10. A decoder according to any one of Claims 7 to 9, wherein the separation filters incorporate weighted sums of products of triplets of pixels.

11. A decoder according to Claim 10, wherein the separation filters comprise weighted sums of pixels and products of triplets of pixels.

12. A luminance/chrominance separation filter operating on a composite signal x which is derived through encoding of a chrominance signal y, the filter taking weighted sums of products of N pixels in a filter aperture; the coefficients h employed in the weighting being derived according to h - (X^TX) ¹X^Ty where X is the matrix of products of N pixels of the signal x over an arbitrary data length.