US20110205438A1

US20110205438A1 - Method and apparatus for motion estimation and motion compensation in video image data

Info

Publication number: US20110205438A1
Application number: US12/677,523
Authority: US
Inventors: Voker Blume
Original assignee: TRIDENT MICROSYSTEMS (FAR EAST) Ltd
Current assignee: Entropic Communications LLC
Priority date: 2007-09-10
Filing date: 2008-08-05
Publication date: 2011-08-25
Also published as: CN101803361A; WO2009034489A2; WO2009034487A2; EP2188978A2; EP2206342A2; WO2009034486A3; US20100271554A1; CN101836441B; EP2206341A2; CN101803363B; CN101836441A; CN101803362A; WO2009034486A2; WO2009034487A3; EP2188990A2; WO2009034492A3; US20100238355A1; CN101803363A; WO2009034488A2; WO2009034493A2

Abstract

The present invention is related to a method and an apparatus for motion estimation and motion compensation in video image data and especially for line based field rate up-conversion motion estimation and motion compensation of video image data.

Description

FIELD OF THE INVENTION

The present invention is related to a method and an apparatus for motion estimation and motion compensation in video image data and especially for line based field rate up-conversion motion estimation and motion compensation of video image data. The present invention is further related to a computer program product and a data carrier comprising a computer program.

TECHNICAL BACKGROUND OF THE INVENTION

The present invention relates to a motion estimation and compensation device and more particularly to a motion estimation and motion compensation device that estimates motion vectors and performs motion compensated predictions of an interlaced and non interlaced sequence of chrominance sub-sampled and non sub-sampled video frames.
Hereinafter, the present invention and its underlying problem is described with regard to the processing of a video signal for line-based motion estimation and motion compensation within a video processing apparatus such as a microprocessor or microcontroller having line memory devices, whereas, it should be noted, that the present invention is not restricted to this application, but can also be used for other video processing apparatus.
The market introduction of TV-sets based on 100/120 Hz frame rate or even higher required the development of reliable Field/Frame Rate Up-conversion (FRU) techniques to remove artefacts within a picture such as large area flickers and line flickers. Standard FRU methods, which interpolate the missing image fields to be displayed on Displays without performing an estimation and compensation of the motion of moving objects in successive image fields are satisfactory in many applications, especially with regard to a better quality of the image and with regard to the reduction of the above-mentioned artefacts. However, many pictures contain moving objects, like persons, subtitles and the like, which cause so-called motion judders.
This problem is better understood by referring to FIG. 1, wherein the motion trajectory of the moving objects (white squares) in the original image fields (i.e. transmitted and received image fields) is supposed to be straight-lined. If the missing fields/frames result from interpolation by means of the above mentioned standard FRU methods (i.e. without motion estimation and compensation), the motion of the moving object in the interpolated fields (dark grey squares) is not at a position as expected by the observer (dotted squares). Such artefacts are visible and induce a blurring effect especially of fast moving objects. These blurring effects typically reduce the quality of the displayed images significantly.
In order to avoid such blurring effects and to reduce artefacts several methods for motion estimation and motion compensation—or shortly MEMC—are proposed. This MEMC provides the detecting of a moving part or object within the received image fields and then the interpolation of the missing fields according to the estimated motion by incorporating the missing object or part in an estimated field.
FIG. 2 shows schematically the change of the position of a moving object between two successive image fields. Between two successive received image fields/frames, the moving objects will have changed their position, e. g. object MO which is in the previous field/frame T in position A is then in the current field/frame T+1 then in position B. This means, that a motion exists from the previous field/frame T to the current field/frame T+1.
This motion of an object in successive image fields/frames can be represented by a so-called motion vector. The motion vector AB represents the motion of the object MO from position A in the previous field T to position B in the current field/frame T+1. This motion vector AB typically has a horizontal and a vertical vector component. Starting from point A in the previous field T and applying this motion vector AB to the object MO the object MO is then translated in position B in the current field/frame T+1. The missing position I of the object MO in the missing field/frame T+½ that has to be interpolated must be calculated by the interpolation of the previous field T and the current field T+1 taken account of the respective positions A, B of the moving object MO. If the object MO does not change its position between the previous field/frame and the current field/frame, e. g. if A and B are the same, position I in the missing field is obtained by the translation of A with a motion vector |AB|/2. In this manner the missing field T+½ is interpolated with a moving object in the right position with the consequence that blurring effects are effectively avoided.
Theoretically, for each pixel of a field a corresponding motion vector has to be calculated. However, this would increase the number of calculation needed and thus the memory requirements enormously. To reduce this enormous calculation and memory effort there exist basically two different approaches:
The first approach employs a so-called block-based MEMC. This first approach assumes that the dimension of the object in the image is always larger than that of a single pixel. Therefore, the image field is divided into several image blocks. For MEMC only one motion vector is calculated for each block.
The second approach employs a so-called line-based MEMC. In this second approach the algorithm is based on a reduced set of video input data of a single line of a field or a part of this line. The present invention is based on this second MEMC approach.
In present line-based MEMC systems, image data is usually stored in a local buffer or on chip memory, the so-called line memory, to which rather extreme bandwidth requirements are made. Many present MEMC systems, like the implementations described by Gerard de Haan in EP 765 571 B1 and U.S. Pat. No. 6,034,734, apply a cashe memory (e.g. a two-dimensional buffer) to reduce the bandwidth requirements and to store a sub-set of an image. The motion compensation device fetches video image data from this cashe while applying motion vectors. Typically, in MEMC systems this cashe covers the whole search range of the motion vectors. Usually, the cashe consists of a great amount of so-called line memories. This results in a relatively large amount of memory, e.g. 720 pixels wide and 24 lines (with an associated maximum vertical vector range of [−12-+12]. Such a cashe comprising a great amount of single line memories requires a huge memory needed only for MEMC data buffering. As a consequence, the memory portion within the processor covers a relatively sizable chip area.
Commonly used MEMC algorithms compensate the motion in two directions, i.e. the motion in the horizontal direction and as well in the vertical direction. For that operation a memory access should be randomly possible, which requires for an application in hardware sufficient embedded chip memory within the video processor for the different temporal incoming data streams. The size of this embedded chip memory strongly depends on the search range (i.e. search area) for the motion of an object, as already outlined above, where the motion estimation can match similar video patterns in two temporal positions and derive the velocity of the motion in terms of pixels per frame or per field.
However, this matching process does not always work perfectly, since methods to determine the quality of the measured motion vector are required. Therefore, for the internal storage of further temporal incoming video signals additional memory resources are required. This, however, increases the amount of embedded memory even further, which leads to an increase of the chip area since for an integrated circuit it is the chip internal memory which significantly determines the chip area. Consequently, the chip is getting more and more expensive. Especially in the mainstream market segment such as for modern Plasma- and LCD-TVs these additional costs typically form a limiting factor for an MEMC implementation.
Therefore, there is a need for an efficient internal memory included in a motion estimation device to store data from a reference frame.
The present invention is, therefore, based on the object to provide a more efficient use of the chip-internal resources and especially of the chip-internal memory with regard to motion estimation and motion compensation.

SUMMARY OF THE INVENTION

In accordance with the present invention, a method comprising the features of claim 1 and/or an apparatus comprising the features of claim 16 and/or a computer program product comprising the features of claim 22 and/or a data carrier comprising the features of claim 23 is/are provided.
Accordingly, it is provided:
A method for line-based motion estimation and motion compensation in video image data, especially for motion estimated and compensated field rate up-conversion in consecutive frames of a motion picture, comprising the steps of: providing a video signal comprising video image data of a line or part of the line of the picture; buffering the video image data in at least one line memory having the size of one video line or at least of the incoming or actually processing video image data.
An apparatus for line-based motion estimation and motion compensation in video image data, especially for motion estimated and compensated field rate up-conversion in consecutive frames or fields of a motion picture, wherein the apparatus is configured to perform a method according to the present invention.
A computer program product comprising a code, said code being configured to implement a method according to the present invention.
A data carrier comprising a computer program product ac-cording to the present invention.
One basic idea of the present invention is based on the conclusion that for the main stream market the performance and therefore the search range can be limited to the occurrence of the most likely motion direction in natural captured scenes, since most of the motion in natural scenes has only one direction. The present invention describes a method for motion estimation and motion compensation which operates only in one direction and therefore performs the motion estimation and motion compensation operations using a single line buffer memory, the so-called line memory. This offers the possibility to reduce the chip embedded memory to one single line memory for the previous and one single line memory for the current field or frame. This advantageously enables significant silicon area reducing and cost saving implementations.
In a preferred embodiment of the present invention, the MEMC is limited to motion in the horizontal direction only, since most of the motion in natural scenes has this direction.
In video signal processing line memories are often used in other applications which already have access to the previous and current motion portrayal, e.g. like so-called de-interlacer applications or temporal noise reduction applications. In a preferred embodiment these already existing line memories of the video application are now additionally used also for MEMC operations. By using existing line memories of the video signal processing system, no further memory bandwidth has to be added to the memory bus and the memory bandwidth is kept uninfluenced.
Thus, this solution offers the possibility to accomplish the MEMC operations by adding a minimal or in the optimal case no additional memory to the video processing system.
The present invention allows a variable performance of matching of accumulation profiles, where beside a comparison of the whole profile also (shift like) matching of subparts of the accumulation profiles is enabled. The later way represents a kind of fine grinned matching increasing accuracy of the matching process and thus of the derived vertical motion vector.
Additionally, a set of vector candidates derived from the previous accumulation profile can be used in the matching. Thus, the efficiency of performance can be increased, as only a set of vector candidates has to be checked.
A further advantage is that several error reduction methods can be employed to reduce the visibility of artefacts.
Advantages, embodiments and further developments of the present invention can be found in the further subclaims and in the following description, referring to the drawings.
In a preferred embodiment the method comprises the steps of: performing the motion estimation by detecting and analysing the video image data and by deriving a motion vector depending on the detected motion, wherein the motion vector contains only motion data of a motion of an object in one direction and especially in the horizontal direction; using the motion vector for motion compensation to interpolate a picture.
In a preferred embodiment the image data of the previous frame is derived from a first line memory and image data of the current frame is derived from a second line memory.
In a preferred embodiment the first line memory and/or the second line memory are further used in a de-interlacer application and/or a temporal noise reduction application.
In a preferred embodiment for the motion estimation a matching process is employed, wherein the luminance profile and/or chrominance profile is/are used as matching parameter.
In a preferred embodiment the matching process comprises the steps of: providing a first series of pixels within a line of a previous frame or field; providing several adjacent second series of pixels within a line of a current frame or field around a given centre pixel for which the motion shall be determined; successively comparing luminance and/or chrominance values of the second series of pixels with the first series of pixels; assigning a quality degree and/or a failure degree to each second series of pixels; selecting that second series of pixels having the highest quality-degrees and/or the lowest failure degrees to be the most probable second series of pixels; computing the motion vector in the horizontal direction on the basis of the selected second series of pixels.
In a preferred embodiment a SAD-based method and/or an ADRC-based method are used for the comparison of the luminance and/or chrominance values.
In a preferred embodiment pre-selected samples of motion vectors are used for the motion estimation.
In a preferred embodiment a variation of at least one selected sample of the motion vectors is performed in order to set up the process of motion estimation and/or to follow the deviation from the constant motion.
In a preferred embodiment the selection of the selected motion vectors is treated differently for the first line of a frame or a field compared to the other lines of the same frame or field.
In a preferred embodiment a damping value which depends on the selected motion vector is used to control those motion vectors with similar failure values and/or to bring the selection process of the motion vector into a desired direction.
In a preferred embodiment a motion vector histogram is used to provide a motion vector ranking on the basis of most and less used motion vectors in a current frame or field.
In a preferred embodiment for detecting unreliable motion vectors the method comprises the steps of: Comparing a luminance pixel value of a motion vector compensated previous frame or field with the luminance pixel value of a motion vector compensated current frame or field; Selecting the current motion vector as an unreliable vector if the evaluated difference in the comparing step exceeds a given threshold value; Marking the selected unreliable vector and/or replacing the unreliable vector by a reliable vector.
In a preferred embodiment artefacts generated by remaining and not replaced unreliable vectors are covered by an intentional blurring of those regions which comprises this artefact.
In a preferred embodiment an additional vertical motion estimation and motion compensation process is performed independently of the horizontal motion estimation and motion compensation process and/or in combination with the horizontal motion estimation and motion compensation process.
In a preferred embodiment the apparatus comprises a first line memory for storing image data of the previous frame and a second line memory for storing image data of the current frame.
In a preferred embodiment the first line memory and/or the current line memory are configured to be further used in a de-interlacer device and/or a temporal noise reduction device.
In a preferred embodiment the apparatus further comprises: an input terminal for providing a video signal comprising video image data; a buffer for buffering the video image data, wherein the buffer comprises at least one line memory having at least the size of one video line or at least of the incoming or actually processing video image data; a motion estimation and compensation device for performing the line-based motion estimation and motion compensation; an output terminal for providing a motion compensated video output signal.
In a preferred embodiment a motion vector memory, especially a third line memory, is provided to store at least one motion vector. However, the motion vectors may also be stored in external memory devices.
In a preferred embodiment the motion estimation and compensation device further comprises: A motion estimation device which generates a motion vector out of the image data stored in the first and second line memories; A motion compensation device which performs a motion compensation using the image data stored in the first and second line memories and employing the vector data.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention and advantages thereof, reference is now made to the following description taken in conjunction with the accompanying drawings. The invention is explained in more detail below using exemplary embodiments which are schematically specified in the figures of the drawings, in which:

FIG. 1 shows the result of a standard (i.e. non motion compensated) FRU method;

FIG. 2 shows the change of position of a moving object between two consecutive received image fields;

FIGS. 3A, 3B show the motion estimation principle for the line-based motion estimation by means of a current frame and the corresponding previous frame;

FIG. 4 shows a block diagram of a first embodiment of a line-based MEMC system according to the present invention;

FIG. 5 shows an example to illustrate the matching process of the motion estimation;

FIG. 6 shows the basic principle for the provision of a motion vector histogram;

FIG. 7 shows an example of a flow diagram to illustrate the process for the detection of unreliable vectors;

FIG. 8 shows an example of a flow diagram to illustrate the search process for bad vector replacement and error concealment;

FIG. 9 shows by means of three illustrations (A)-(C) examples to illustrate the artefact concealment according to the present invention;

FIG. 10 shows a block diagram illustrating an embodiment of the line-based motion estimation according to the present invention;

FIG. 11 shows a block diagram illustrating an embodiment of the line-based motion compensation according to the present invention using adaptive artefact concealments;

FIG. 12 shows a block diagram of a second embodiment of a line-based MEMC system according to the present invention using the line memories assigned to the de-interlacer device also for the motion estimation device;

FIG. 13 shows a block diagram of a third embodiment of a line-based MEMC system according to the present invention which combines the line-based MEMC with non-vector based-up conversion methods.

In all figures of the drawings elements, features and signals which are the same or at least have the same functionality have been provided with the same reference symbols, descriptions and abbreviations unless explicitly stated otherwise.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE PRESENT INVENTION

In the following description of the present invention first of all a short overview of the motion estimation and motion compensation is presented.
The MEMC method consists mainly of two sections, the motion estimation and the motion compensation method. The motion estimation performs the measurement of the motion and derives the velocity of the displayed regions in pixel per picture (i.e. field or frame). Also the direction of the motion will be indicated by a positive or negative sign. These measured motion information is described in the form of a motion vector. The motion vector is used for the motion compensation to interpolate the picture at the temporal accurate position and to avoid so-called judder effects and/or so-called motion blurring effects.
FIGS. 3A, 3B show the motion estimation principle for the line-based motion estimation by means of a current picture (field or frame) 10 (n) and the corresponding previous picture 11 (n−1). According to the temporal positions the motion vector 12, 13 will be split by its length into two parts, where the first vector part 12 points into the previous picture 11 and the second vector part 13 points into the current picture 10. For the interpolation of a missing picture 14 (n−½) between the current and the previous pictures 10, 11 pixels 15 from both temporal pictures 10, 11 are taken into account for the compensation. In line-based MEMC only the pixels 15 within the same line 16 are used at the same time and the MEMC is performed for a single line 16 of a field or frame only. For this kind of MEMC the pixels 15 of the current picture 10 are compared with the corresponding pixels 15 of the previous picture 11 to estimate and compensate the corresponding pixels 15 of the missing picture 14.
FIG. 4 shows a block diagram of a line-based MEMC system according to the present invention. The MEMC system is denoted by reference number 20. The MEMC system 20 comprises an input terminal 21, a bus 22, two line memories 23, 24, a motion estimation device 25, a motion compensation device 26 and an output terminal 27. It is assumed that the bus 22 is an external bus 22 and especially an external memory bus 22. However, it may also be possible, that the bus 22 is an internal bus 22. At the input side the bus 22 is connected to an external memory 28 device such as a SDRAM, a DDR-RAM, etc. Image data to be displayed in a panel 29 such as a plasma- or LCD-panel or a CRT-screen is stored in this external memory 28. Via the input terminal 21 and the memory bus 22 this image data X1, X1′ is transferred to both line memories 23, 24. According to one embodiment of the present invention only two line memories 23, 24 are needed whereas the first line memory 23 is used for buffering image data X1 of the previous picture and the other line memory 24 is used for storing the image data X1′ of the current picture. However, it should be clear that also a greater number of line memories may be provided.
A line memory 23, 24 as used in the present patent application indicates an embedded memory of a size of one video line of a frame or a field or at least less of the incoming video signal stream or actually processing video signal stream. A field denotes a video image or picture which comprises either odd or even lines. A frame denotes a video image comprising of the complete video information for one picture, i.e. of a field for the odd lines and the corresponding field for the even lines. A line denotes a full horizontal row within a field of one video picture or at least a part of this row.
Both of the line memories 23, 24 are coupled—on their output sides—to the motion estimation device 25 and to the motion compensation device 26. This enables the image data X1, X1′ stored in the line memories 23, 24 to be transferred to the motion estimation device 25 and to the motion compensation device 26, respectively. In FIG. 4 the corresponding data signals to the motion estimation device 25 are denoted by X2, X2′ and the corresponding data signals motion compensation device 26 are denoted by X3, X3′.
The motion estimation device 25 generates a motion vector signal X4 out of the image data X2, X2′ stored in the line memories 23, 24 by employing a matching process. This vector signal X4 is transferred to the motion compensation device 26. The motion compensation device 26 performs a motion compensation using the image data X3, X3′ stored in the line memories 23, 24 and applying the vector data X4 to this image data X3, X3′. At the output terminal 27, the motion compensation device 27 provides a video signal X5 which comprises information for a motion compensated picture. This video signal X5 is transferred via the output terminal 27 to a display 29, such as a LCD-panel 29 or the like.
With regard to FIG. 5, hereinafter the operation of the motion estimation device 25 is described in more detail:
For the motion estimation a matching process is employed to select a corresponding series of pixels 32 which fits best to a given amount of pixels 30. For this selection a given amount of pixels 30 of a line of a current frame around the centre pixel 31 for which the motion shall be determined is taken from a line memory 24 of the current frame 32. Hereinafter this given amount of pixels 30 is denoted to as series of pixels 30. In the present embodiment a series of pixels 30 comprises 9 single pixels 33. It is self-understood that a series can also comprise a greater or a smaller amount of pixels 33.
For the selection the luminance profile of the pixels 33 is used as the matching parameter. Luminance is a photometric measure of the density of luminous intensity in a given direction. It describes the amount of light that passes through or is emitted from a particular area, and falls within a given solid angle. Thus, luminance is the photometric measure of the brightness in a frame of a motion picture. If the luminance is high, the picture is bright and if it is low the picture is dark. Thus, luminance is the black and white part of the picture.
This luminance profile is used to find out that series of nine pixels 34 out of the previous frame 35 which fits best with the series of nine pixels 30 of the current frame 32. In the embodiment of FIG. 5 the luminance profile of the series of nine pixels 30 of the current frame 32 are compared with the luminance profiles of several corresponding series of nine pixels 34 of the previous frame 35. In order to derive the true motion the series of nine pixels 30 will be shifted over the search range in the horizontal direction 36. It is assumed that that series of nine pixels 34 of the previous frame 35 which shows the best luminance profile matching (with the series of nine pixels 30 of the current frame 32) is the correct series of pixels. These series of pixels 30, 34 are then used for the computation of the motion vector.
A typical value for the search range is, e.g. 64 pixels (+31 . . . −32). However, it may also be possible to use less than 64 pixels, however, then the quality of the result of this comparison is increasingly going down. On the other hand it is also possible to use more than 64 pixels. Then the quality of the selection result is going up, however, this needs more computational effort. Therefore, typically a trade-off which provides an optimization between best quality of the selection result and simultaneously a minimum computation effort is employed.
In a preferred embodiment for each selected motion vector 37 a single matching process is performed in the way described above. This matching process is performed by assigning a quality degree and/or a failure degree for each series of pixels 30. Then, a quality-degree and/or a failure degree are assigned to each one of those series of pixels 30 which are undergoing the matching process. Those series of pixels 30 having the highest quality-degrees and/or the lowest failure degrees are selected as most probable series of pixels. These series of pixels 30 are then used for computing the motion vectors for the horizontal motion. Typically, but not necessarily a SAD method (SAD=sum of absolute difference) and/or ADRC (Adaptive Dynamic Range Coding) method is used for the comparison of the luminance and/or chrominance values.
Using Pre-Selected Motion Vector Samples for the Motion Estimation:
Assuming the motion of an object in the scene will be constant from field/frame to field/frame and the object is larger than a series of pixels (e.g. the above mentioned 9 pixels), then the matching process can then be performed more efficiently if a set 38 of pre-selected motion vectors 37—the so-called motion vector samples 37—are checked for a matching of the luminance profile (see FIG. 5). For example, one selected motion vector 37 can be taken from the neighbouring pixel. A second selected motion vector can be taken from the previous line, if the already estimated motion vectors are stored in a vector memory specially designed for the different motion vector samples.
The zero-vector which indicates no motion of the object is typically one of the most used motion vector samples. This zero-vector is used in order to more efficiently detect regions within a picture showing no motion. In principle the amount of pre-selected motion vectors 37 which will taken into account depend strongly on what kind of motion vector quality is desired.
Variation of Selected Motion Vectors:
In order to set up the process of motion estimation and to follow the deviation from the constant motion, a variation of certain pre-selected motion vectors is required for test operation purposes. That means that for pre-selected motion vector samples a certain amount of motion will be added or subtracted. This can be done by applying a variance with different amount of motion speed to these motion vectors. The tested implementation checks between odd pixels and even pixels alternating an update of +/−1 pixels and +/−4 pixels on the previously determined motion vector. The selection of the variance is adjustable and variable as required or as the need arises and depends e.g. on the resolution of the incoming video signal.
For the line-based motion estimation it is very advantageous that the motion vector will converge quickly for the real motion in the scene. Therefore, the selection of the tested motion vectors is treated differently for the first line of a frame or field. For the first line of a frame or field testing is not possible in the normal way since a line above the first line which is needed for testing is not existing. In the first line of each field the selected motion vectors which normally test the motion vectors of the line above are loaded with vector values, which e.g. vary according to a triangle function from pixel to pixel. The triangle function oscillates between an adjustable minimum value and an adjustable maximum value. For that purpose also other regular oscillating functions e.g. a saw tooth function, a sinusoidal function, and the like may be employed for the determination of the motion vector of the first line.
The Matching Process:
In a preferred embodiment the matching process assigns a failure value to each tested motion vector. In another embodiment this value may be also a quality value. It might also be possible to evaluate as well a failure value and a quality value for the matching process. Preferably, the sum of the absolute difference (SAD) is used as the failure value or to at least derive the failure value. Ideally, to find the optimal motion vector a failure value of zero is needed. However, typically the failure value is different from zero. Therefore, the motion vector corresponding with the lowest failure value is then selected as the most probably motion vector representing the motion of an object in the local scene.
Attenuation of the Vector Selection, Vector Damping:
In a preferred embodiment a damping value is used which depends on the vector attenuation of the different motion vectors. This enables to control the motion vectors with equal failure values and/or to furnish the motion vector selection process with a certain direction.
Vector Memory:
The different motion vectors are advantageously stored in a vector memory. These motion vectors can be then—if required—fetched from the vector memory for further processing and/or for the motion estimation of the next pixels.
The motion estimation process forms a recursive process. Therefore, the size of this vector memory mainly depends on the desired quality level of the matching process. In one embodiment, the tested implementation comprises only one line of a vector memory. In this vector memory every second motion vector will be stored alternately, in order that an access of the motion vectors from the measured line above is possible.
Robustness Improvement by Providing a Vector Histogram:
In a preferred embodiment a motion vector histogram is calculated in order to create a highly reliable and homogeneous field of motion vectors. This vector histogram allows a vector majority ranking to derive most and less used motion vectors in an actual scene.
FIG. 6 shows an example to illustrate the basic principle for the provision of a motion vector histogram. FIG. 6 shows a vector histogram generator 40 to provide a motion vector histogram. In the embodiment in FIG. 6 the vector histogram generator 40 comprises a switching device 41, which is controlled by an +1-incrementing device 42. The switching device 41 is controlled on the one hand by a motion vector 43 information and on the other hand by the incrementing device 42 which shift the switching device 41 to the next input terminal of a counting device 45 whensoever the next identical motion vector 43 occurs. The counting device 45 which comprises different counter cells 44 counts the occurrence of each motion vector and increments the counter by +1 for each occurrence of the motion vector. A ranking device 46—which e.g. comprises a simple comparing means—is coupled to the output terminals of the different counter cells 44 of the counting device 45. This ranking device 46 selects the most often used motion vector and applies this motion vector for the estimation determination. The most often used motion vector may be then stored in a motion vector histogram memory 47.
The provision of a motion vector histogram can be done either for the whole frame or field or only for parts of the frame or field. It is very efficient to split the picture into horizontal stripes and return a most often used vector for each stripe. In very a preferred embodiment news ticker information within a picture can be detected in that way very reliable.
Unreliable Vector Detection:
Under certain circumstances the motion estimation will not deliver reliable motion vectors. Especially for vertical movements as well as for occlusions and uncovered areas the matching process sometimes does not provide a reliable access of the current and previous line. Not reliable motion vectors, however, lead to relatively large, undesired differences for the compensation process.
FIG. 7 shows an example of a flow diagram to illustrate the process for the detection of unreliable vectors. In this detection process the luminance pixel value 50 of the vector compensated previous frame or field is compared 52 with the luminance pixel value 51 of the vector compensated current frame or field. If this difference 52 strongly exceeds an adjustable threshold value 53 the current motion vector will be selected as a bad vector 54. Otherwise, it is assumed that this vector is a reliable vector 55. This bad vector 54 will be marked 56 and the MSB-bit of the vector memory will be set e. g. to one. The MSB-bit then indicates whether this vector can be used for further estimation processing or not.
Bad Vector Replacement:
FIG. 8 shows an example of a flow diagram to illustrate the search process for a bad vector replacement and an error concealment. In this search process bad motion vectors are treated in a special way in order to avoid introducing artefacts. In case of the occurrence of a bad vector, a reliable vector will be fetched from the memory e. g. for a motion vector for the line above. Therefore, preferably only those motion vectors are used in this search process which are not marked as unreliable (see above with regard to FIG. 7).
The search process starts e.g. from the most outer border of the search range in order to find a reliable vector of the surrounding area. If the bad vector signal 60 indicates no bad vector 61 then this motion vector is used for the further motion estimation process 62. Otherwise, a search process 63 is started to select a motion vector from previous estimation processes. If this search process 63 indicates a bad vector 64 then this bad vector is replaced by a reliable vector 65 (see also steps 53-55 in FIG. 7). After replacing the bad vector against a reliable vector this reliable vector is then used for the motion compensation process. Otherwise, it is checked if the end of the search range 66 is reached. If the end of the search range 66 is reached the output signal is blurred 67. If the end of the search range 66 is not yet reached the search process returns 68 to the step 63.
Adaptive Error Region Extension and Error Spike Suppression:
The remaining unreliable vectors typically lead to artefacts in the interpolated picture. In one embodiment—as already outlined above with regard to FIG. 8—these artefacts can be covered by an intentional blurring of this artefact comprising regions. In a preferred embodiment these artefact comprising regions—hereinafter also referred to as error regions—are cleaned up from single spikes in order to provide efficient artefact concealment. This process is shown with regard to FIG. 9. FIG. 9 shows by means of three illustrations (A)-(C) examples to illustrate the artefact concealment according to the present invention.
The artefact concealment according to the present invention may be performed on the basis of an error spike suppression and/or an error region extension method. In FIGS. 9(A)-(C) the left diagram side denotes the vector samples before the artefact concealment and the right diagram side denotes the vector samples after the artefact concealment. The reliable motion vectors are denoted by reference number 70 and the unreliable motion vectors which additionally comprise an arrow to illustrate the spike are denoted by reference numbers 71, 72, 73.
Single unreliable vectors 71—hereinafter also referred to as spikes—will be suppressed (FIG. 9(A)). Double spikes 72 will be not suppressed (FIG. 9(A)). Double spikes 73 intersected by a single reliable motion vector 70′ will be combined to form one uniform region of three spikes 74 (FIG. 9(B)). In a preferred embodiment, after cleaning up the spikes 71-73 in that way, the remaining error regions 75 are extended additionally by an amount of bad vectors 76 on both sides of this error region 75 (FIG. 9(B)). Typically two spikes 76 are added on both sides of the error region 75.
This artefact concealment operation ensures a proper behaviour of the blurring filter. In a preferred embodiment, the blurring filter is used only for blurring operations in the horizontal direction.
An error spike denotes the single occurence of one error surrounded by a mostly error-free region.
It is self understood that the above mentioned numerical data is merely illustrative and may be adapted to best provide an optimized blurring effect.
FIG. 10 shows a block diagram illustrating an embodiment of the line-based motion estimation according to the present invention as described above and as implemented in a motion estimation device 25 as shown in FIG. 4.
The motion estimation device 25 comprises a matching device 80, a cost/quality function device 81 and a vector selector device 82, which are arranged in series connection between the input side 83 of the motion estimation device 25 where the image data signals X1, X1′ stored in the both line memories 23, 24 are provided and the output side 84 of the motion estimation device 25 where the motion vector signal X4 in present. In the device elements 80-82 a matching process and a vector selection as described with regard to FIG. 5 is implemented.
The motion estimation device 25 further comprises a vector quality device 85 which is connected on the one hand to the input side 83 and on the other hand to the output side 84. The vector quality device 85 generates a quality signal X6 comprising an information of the vector quality out of the image data signals X1, X1′ and the motion vector signal X4.
The motion estimation device 25 further comprises a vector histogram device 86 and a vector majority device 87 which are arranged in series connection in a feedback path between the output side 84 and the matching device 80. Here, in the device elements 86, 87 a vector histogram is generated to provide a ranking of most and less used vectors in the actual scene as shown and described with regard to FIG. 6. Thus, the elements 86, 87 correspond to the vector histogram generator 40 of FIG. 6.
The motion estimation device 25 may further comprise a further line memory 88 to store the motion vector data X4 and/or the data X6 for the vector quality.
The motion estimation device 25 further comprises a vector sample device 89.
This vector sample device 89 is also arranged in the feedback path and is connected at its input side with the line memory 88, the vector majority device 87 and advantageously with a further device 90. This further device 90 performs a variation of the motion vector samples by using a special signal having a certain magnitude, e.g. a sinusoidal signal, a saw tooth signal or the like. This certain signal is then used for a testing and/or matching process and/or an up-dating process of the first line of a frame or field. However, it might also be possible to randomly up-date different lines of the frame or field. On its output side, the vector sample device 89 is connected at its output side to the matching device 80.
The motion estimation device 25 further comprises a vertical motion estimation device 91. For vertical motions the above described one-dimensional motion estimation algorithm is not able to compensate fully motion in the vertical direction. However, the occurrence of vertical motions can be used to reduce the compensation in same regions of the picture by splitting the picture into different regions to derive vertical motion for each region. In this case the luminance values of the lines in the different region of the same picture will be summed up and stored individually for each line of this picture. This results in an accumulated vertical profile for different regions of the same picture. Then, the whole picture can be divided into smaller regions to derive a vertical motion for each of these regions. This vertical motion estimation process is performed in the vertical motion estimation device 91 which is connected to the input side 83 and which provides at its output side a sector based vertical motion index X7.
Thus, the vertical MEMC as sketched above can be performed independently of horizontal MEMC and also in combination with the horizontal MEMC, wherein the combination can be performed in dependence on a certain situation or the motions present, respectively. Further, such a methodology allows an implementation of vertical MEMC, which does not need large amounts of additional memory capacity to analyze data of consecutive frames being the case in the most methodologies of the prior art.
The motion estimation device 25 further comprises a vector damping device 92. In this damping device 92 a damping value as described above may be used to damp vector samples of the vector sample device 89 and to provide damped vector samples to the vector selector 82.
Hereinafter the motion compensation process which is performed in the motion compensation device 26 of FIG. 4 is described with regard to FIG. 11 in more detail. FIG. 11 shows a block diagram illustrating an embodiment of the line-based motion compensation according to the present invention using adaptive artefact concealments as described above.
The motion compensation device 26 comprises a compensation device 100 which performs the temporal motion interpolation according to the motion vectors X4 estimated by the motion estimation device 25. In a preferred embodiment the compensation device 100 comprises a Median Filter which uses as input data the luminance values of the vector compensated previous line, the vector compensated current and the uncompensated previous line. Additionally, also the chrominance values can be compensated.
Depending on the vector quality a replacement vector indicated as reliable vector will be searched in the local area of the vector memory from the line above. If no reliable vector can be found the adaptive blurring or an other fallback methodology typically tries to cover this artefact.
The motion compensation device 26 further comprises a vertical motion control device 101 which provides a control signal X8 to the compensation device 100 in order to incorporate also information of a vertical motion to the compensation device 100.
The motion compensation device 26 further comprises a bad vector modification device 102. Based on information X4, X6 provided by the motion estimation device 25 the bad vector modification device 102 modifies bad vectors as shown and described above with regard to FIGS. 8 and 9. These information X9 about modified bad vectors is then used—together with the control signal X8—to perform the motion compensation within the compensation device 100. The compensation device 100 then generates at its output side a motion compensated image data signal X10.
The motion compensation device 26 further comprises an adaptive blurring device 103. Based on the motion compensated image data signal X10 and a blurring control signal generated by the bad vector modification device 102 this adaptive blurring device 103 performs an adaptive blurring e.g. such as described with regard to the FIGS. 8 and 9. The adaptive blurring device 103 generates an adaptive blurred image data signal X5′ which might correspond to the image signal X5 of FIG. 4.
FIG. 12 shows a block diagram of a second embodiment of a line-based MEMC system according to the present invention using the line memories assigned to the de-interlacer device also for the motion estimation device.
Unlike the first embodiment in FIG. 4 a de-interlacer device 113 is arranged between the line memories 110, 111, 112 and the motion compensation device 26. The de-interlacer device 113 is typically used to convert a field represented by video data stream into a full frame which is then also represented by another video data stream.
On-chip solutions for video processing which are memory-based have already existing internal line buffers 110-112—the so-called line memories 110-112—which carry video data from the previous and current field or frame. These line buffers 110-112 can be located e. g. within temporal noise reductions or de-interlacing units 113 which operate motion adaptive. With the proposed line-based MEMC these line buffers can be reused additionally for the motion estimation. For that purpose and in order to reduce motion judder artefacts from movie sources, a movie detector which indicates the current interpolated sequence of pull-down mode is used. A line buffer selector transfers the video signal data to the motion estimation device according to the previous and the current video input signal. This technique allows using already existing memory resources also for motion estimation which also prevents additional bandwidth for the temporal up-conversion process. Therefore, the chip area for the motion estimation and the motion compensation can be reduced to a minimum.
The de-interlacer device 113 uses three line memories 110, 111, 112 coupled on their input side to the memory bus 22 and providing at their output side line data. This line data provided by the line memories 110, 111, 112 is processed within the de-interlacer device and then provided to the motion compensation device 26. According to the present invention, these line memories 110, 111, 112 are additionally used also for the motion estimation device 25. For this purpose, the system 20 additionally comprises a selector device 114, where a movie sequence X0 is provided to this selector device 114. This movie sequence X0 may be then stored in an external memory 28 via the memory bus 22 and can be read out from this external memory 28 through the line memories 110, 111, 112. For an IMC operation, this data stored in the line memories 110, 111, 112 of the de-interlacer device 113 can be also used for MEMC. For this purpose the data stored in the line memories 110, 111, 112 is then provided as well to the motion estimation device 25 and the motion compensation 26 device.
FIG. 13 shows a block diagram of a third embodiment of a line-based MEMC system according to the present invention.
Unlike the second embodiment in FIG. 2 this third embodiment combines the line-based MEMC with non-vector based-up conversion methods like MCSFI and GFI. FIG. 24 shows a block diagram for the combination of line based MEMC with non-vector based up-conversion methods like MCSFI and GFI.
GFI (Grey Frame Insertion) denotes an up-conversion method to double the incoming frame rate of a video sequence. MCSFI (Motion Controlled Smooth Frame
Insertion) denotes an up-conversion method which is motion detector controlled. Instead of these up-conversion method also a SFI-method (Smooth Frame Insertion) and/or a DFI-method (Dynamic Frame Insertion) may be applied, whereas SFI stands for an up-conversion method to double the incoming frame rate of a video sequence and DFI stands for an up-conversion method to double the incoming frame rate of a video sequence.
Typically for those up-conversion methods the high contrast regions are detected per unit and excluded from the compensation processing due to the incapability of those methods to compensate within these regions. Therefore a pixel gain detector controls a fader, which normally fades over from the processed video signal to the unprocessed video signal in order to reduce the introduction of artefacts. The present invention of the line-based MEMC is capable to perform the up-conversion also in these high contrast regions. Therefore, the signal of the line-based MEMC is applied to the unprocessed fader input of the non-vector-based up-conversion methods.
While embodiments and applications of this invention have been shown and described above, it should be apparent to those skilled in the art, that many more modifications (than mentioned above) are possible without departing from the inventive concept described herein. The invention, therefore, is not restricted except in the spirit of the appending claims. It is therefore intended that the foregoing detailed description is to be regarded as illustrative rather than limiting and that it is understood that it is the following claims including all equivalents described in these claims that are intended to define the spirit and the scope of this invention. Nor is anything in the foregoing description intended to disavow the scope of the invention as claimed or any equivalents thereof.
It is also noted that the above mentioned embodiments, examples and numerical data should be understood to be only exemplary. That means that additional system arrangements and functional units and operation methods and standards may be implemented within the MEMC-system.

Claims

1. A method for line-based motion estimation and motion compensation in video image data, comprising the steps of:

providing a video signal comprising video image data of a video line or part of the video line of the picture;

buffering the video image data in at least one line memory having the size of one video line or at least of the incoming or actually processing video image data.

2. The method according to claim 1, wherein the method comprises the steps of:

performing the motion estimation by detecting and analysing the video image data and by deriving a motion vector depending on the detected motion, wherein the motion vector contains only motion data for motion of an object in one direction in the horizontal direction;

using the motion vector for motion compensation to interpolate a picture.

3. The method according to claim 2, wherein image data of the previous frame is derived from a first line memory and image data of the current frame is derived from a second line memory.

4. The method according to claim 3, wherein the first line memory and/or the second line memory is/are further used in a de-interlacer application and/or a temporal noise reduction application.

5. The method according to claim 4, wherein for the motion estimation a matching process is employed, wherein the luminance profile and/or chrominance profile is/are used as matching parameter.

6. The method according to claim 5, wherein the matching process comprises the steps of:

providing a first series of pixels within a line of a current frame or field;

providing several second series of pixels within a line of a previous frame or field around a given centre pixel for which the motion shall be determined;

successively comparing luminance and/or chrominance values of the second series with the first series;

assigning a quality degree and/or a failure degree to each second series;

selecting that second series having the highest quality-degrees and/or the lowest failure degrees to be the most probable second series;

computing the motion vector in the horizontal direction on the basis of the selected second series.

7. The method according to the claim 6, wherein a SAD-based method and/or an ADRC-based method are employed for the comparison of the luminance and/or chrominance values.

8. The method according to claim 7, wherein pre-selected samples of motion vectors are used for the motion estimation.

9. The method according to claim 8, wherein a variation of at least one selected motion vector is performed in order to set up the process of motion estimation and/or to follow the deviation from the constant motion.

10. The method according to claim 9, wherein the selection of the selected motion vectors is treated differently for the first line of a frame or a field compared to the other lines of the same frame or field.

11. The method according to claim 10, wherein a damping value which depends on the selected motion vector is used to adapt motion vectors with similar failure values and/or to bring the selection process into a desired direction.

12. The method according to claim 11, wherein a motion vector histogram is used to provide a ranking of the motion vector on the basis of most and less used motion vectors in a current frame or field.

13. The method according to claim 12, wherein for detecting unreliable motion vectors the method comprises the steps of:

Comparing a luminance pixel value of a motion vector compensated previous frame or field with the luminance pixel value of a motion vector compensated current frame or field;

Selecting the current motion vector as an unreliable vector if the evaluated difference in the comparing step exceeds a given threshold value;

Marking the selected unreliable vector and/or replacing the unreliable vector by a reliable vector.

14. The method according to the claim 12, wherein artefacts generated by not replaced unreliable vectors are covered by an intentional blurring of those region which comprises this artefact.

15. The method according to claim 14, wherein an additional vertical motion estimation and motion compensation process is performed independently of the horizontal motion estimation and motion compensation process and/or in combination with the horizontal motion estimation and motion compensation process.

16. An apparatus for line-based motion estimated and compensated field rate up-conversion in consecutive frames or fields of a motion picture, wherein the apparatus comprises:

a processing unit configured to provide a video signal comprising video image data of a video line or part of the video line of a picture; and

a memory unit coupled to the processing unit to buffer the video image data.

17. The apparatus according to claim 16, wherein the apparatus comprises a first line memory for storing image data of the previous frame and a second line memory for storing image data of the current frame.

18. The apparatus according to claim 17, wherein the first line memory and/or the second line memory are configured to be further used in a de-interlacer device and/or a temporal noise reduction device.

19. The apparatus according to at claim 18, wherein the apparatus further comprises:

an input terminal for providing a video signal comprising video image data;

a buffer for buffering the video image data, wherein the buffer comprises at least one line memory having at least the size of one video line or at least of the incoming or actually processing video image data;

a motion estimation and compensation device for performing the line-based motion estimation and motion compensation;

an output terminal for providing a motion compensated video output signal.

20. The apparatus according to claim 19, comprising a third line memory, to store at least one motion vector.

21. The apparatus according to claim 20, wherein the motion estimation and compensation device further comprises:

A motion estimation device which generates a motion vector X4 out of the image data stored in the first and second line memories;

A motion compensation device which performs a motion compensation using the image data stored in the first and second line memories and employing the vector data.

22. A computer program product comprising a computer usable medium having a computer readable program code embodied therein, said computer readable program code being configured to be executed to implement a method for line-based motion estimation and motion compensation in video image data, comprising the steps of:

23. The computer program product of claim 22 wherein the method further comprises: e

performing the motion estimation by detecting and analysing the video image data and by deriving a motion vector depending on the detected motion, wherein the motion vector contains only motion data for motion of an object in a horizontal direction:

using the motion vector for motion compensation to interpolate a picture.