RU2576490C1 - Background hybrid retouch method for 2d to 3d conversion - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: method of retouching a video image background comprises the steps of: obtaining background movement information from a series of frames of a video sequence; for frames with a missing region: transferring pixel data from one or more preceding frames using the background movement information, transferring pixel data from one or more subsequent frames using the background movement information, and merging the said data with partially recovered pixel data in one or more of the said regions on the current frame; selecting a frame with the remaining missing region; performing a spatial retouching procedure on the said frame to recover pixel data of the said missing region; and transferring the recovered pixel data from the said selected frame to all the frames of the sequence where it is possible using the background movement information.

EFFECT: enabling video image background retouching.

9 cl, 5 dwg

Description

FIELD OF THE INVENTION

The invention relates generally to video processing, and more particularly to retouching video data, i.e. evaluating values for a background previously hidden by foreground content.

BACKGROUND

Retouching video data is generally used in various tasks, such as video stabilization, conversion with increasing frame rate, converting two-dimensional (2D) images into three-dimensional (3D) and synthesis of angles. The method according to the invention is aimed at a full assessment of the background pixel values for arbitrary sequences of video data, which are accompanied by information about the foreground objects for each frame of video data.

In general, most prior art video retouching methods use either frame-by-frame retouching, or estimation of camera movement, followed by overlapping and overlapping frames.

US 2006/0257042 A1 describes general methods for improving video data (such as stabilizing and eliminating blur) based on motion retouching. In accordance with the method, it is proposed the distribution of motion vectors to missing areas and the subsequent use of the thus supplemented map of motion vectors for transferring pixel information to missing areas. It is believed that the number of remaining pixels is generally small, and it is suggested that they be filled by blurring. This document deals mainly with video stabilization artifacts, and therefore the missing areas are usually not foreground objects, but areas adjacent to the edges of the frame. Therefore, in this case, spatial retouching is not important and is not considered in this document. Thus, the application of the described method will be limited in the case of removing foreground objects from sequences of video data having background regions that are not visible on any frame.

In US 8243805 B2, the retouching process is based on the addition of a motion vector map. First, the local motion of the pixels is estimated and the space-time gap in the sequence is determined based on the provided masks (from user input or automatically). Further, the missing values of the motion vectors in the gap are transferred in the form of space-time inserts from known data by using a specially implemented similarity measure. As a result, full information about motion vectors for the entire sequence becomes available, which can be further used to propagate pixel values into a space-time gap. However, it is obvious that this approach does not always complement the entire sequence. For example, a scene with an almost static object in front of a static background or scenes with zooming cannot provide enough data to completely fill the area of the object.

US 2013/0182184 A1 presents a different approach to retouching a video image. It is based on the presentation of video from one scene in the form of mosaic images (mosaic images are created from frames combined in a certain coordinate system with a fixed reference frame). The described system fills in the missing parts by combining frames at the coordinates of the same reference frame (mosaic representation) and then retouching the remaining parts of the mosaic using a suitable spatial retouching method. Then the mosaic representation is converted back to a sequence of frames in which all the missing parts are completely filled in the same way. It is expected that the described system will provide a complete complement of the sequence in cases of simple movement. For scenes with complex local background motion, the combination of frames by tracking characteristic points may not be sufficient to describe the motion, which can lead to visible artifacts in the resulting sequence.

US 2013/0128121 A1 describes a device for retouching video data based on tracking of characteristic points. Using two-dimensional (2D) tracking, the trajectories of the rare points of the scene are obtained and some of them are extracted to describe the smooth movement of the scene. This movement is used to predict scene points that are invisible on the target frames and to transfer content from source frames by combining characteristic points and then using transfer technologies while preserving the content and overlay to fill in the missing areas. The proposed method works well only for sequences with certain types of motion, which provide the ability to find a source for each target pixel. This document does not address cases where additional spatial retouching technologies are required to provide retouching of video data with the least number of visible artifacts.

US 2012/0162395 A1 describes the synthesis of angles for three-dimensional (3D) video systems. In the proposed method, the depth values in the missing region are determined (for example, by selecting the smallest value in a certain region around the missing pixel) and then the color values are calculated using the obtained depth values by applying special weighted filtering for all missing pixels and neighboring pixels. The method was developed to fill the opening areas for the synthesis of three-dimensional angles and in general can show poor visual results with large retouched areas. In addition, it does not use temporal information in a video sequence, since it is based only on depth and color information for one frame.

US 2013/0128121 A1 and US 2006/0257042 A1 can be considered as the closest analogues of the invention.

SUMMARY OF THE INVENTION

In view of the above problems, the present invention provides a method for retouching a video image background and a corresponding video processing system configured to implement such a method that overcome at least some of the disadvantages of the prior art video processing systems.

The objective of the invention is to provide a method for processing video data for retouching the background of a video image working with pixel data, pixel-by-pixel fields of motion vectors and indicating missing areas, i.e. areas to be restored in the pixel data for each frame of the sequence of video data, for example, the positions of the foreground objects, all of the mentioned data refer to the same video sequence. A video sequence in the context of this document is a sequence of frames that could be shot with a single continuous camera shot without interruptions and instant transitions.

In one aspect, the invention relates to a method for retouching a video background working with pixel data, the method comprising the steps of:

a. get background motion information from a series of frames of a video sequence,

b. for all frames with at least one missing area:

- distribute pixel data from one or more previous frames using background motion information,

- distributing pixel data from one or more subsequent frames using background motion information, and

- combine the above data with partially restored pixel data in one or more of these areas on the current frame;

c. repeat until all missing areas on all frames have been restored, the stages in which:

- choose a frame with at least one remaining missing area;

- perform a spatial retouching procedure on said frame to recover pixel data of said missing region; and

- distributing the restored pixel data from said selected frame to all frames of the sequence into which it is possible to extend said restored pixel data using background motion information.

In an embodiment, obtaining background motion information may comprise stages in which pixel-by-pixel motion for a full frame for a given frame is evaluated and a spatial motion filling procedure is applied to estimate background motion in areas to be restored.

The spatial motion filling procedure can use an estimate of global motion within missing regions and provide a smooth transition from pixel-by-pixel motion to global motion near the boundaries of the missing region.

The distribution of pixel data using background motion information may comprise the step of temporarily integrating full-frame motion fields between source frames and target frames, and distributing available pixel data from source frames to said missing areas in the target frame.

In another aspect, a video processing system is provided that is capable of retouching the background of a video image while working with pixel data, the system comprising:

at least one processor; and

a memory for storing data related to the processing of video data, as well as computer program instructions;

and when executed by the at least one processor, computer program instructions prompt the processor:

- receive information about the movement of the background from a series of frames of a video sequence;

- for all frames with at least one missing area:

- distribute pixel data from one or more previous frames using background motion information,

- distribute pixel data from one or more subsequent frames using background motion information, and

- combine the mentioned data with partially restored pixel data in one or more of the indicated areas on the current frame;

- select a frame with at least one remaining missing area;

- perform the spatial retouching procedure on said frame to recover pixel data of said missing region; and

- distribute pixel data from said selected frame to all frames of a sequence into which it is possible to distribute said recovered pixel data using background motion information.

When executed by the at least one processor, computer program instructions may additionally cause the at least one processor to evaluate pixel-by-pixel motion for a full frame for a given frame and apply a spatial motion filling procedure to estimate background motion in areas to be restored. Also, computer program instructions may further encourage at least one processor to use an estimate of global motion in missing regions and to provide a smooth transition from pixel-by-pixel motion to global motion near the boundaries of the missing region. In an embodiment, computer program instructions may further encourage at least one processor to temporarily integrate full-frame motion fields between source frames and target frames and distribute available pixel data from source frames to said missing areas in the target frame.

Another aspect of the invention relates to a computer-readable medium on which a computer program is stored, which, when executed by at least one processor, causes the at least one processor to perform a background retouching method of the video image working with pixel data, the computer program comprising:

- a code for obtaining information about the movement of the background from a series of frames of a video sequence,

- a code for distributing pixel data from one or more previous frames using background motion information,

- a code for distributing pixel data from one or more subsequent frames using background motion information, and

- a code for combining said data with partially restored pixel data in one or more of the indicated areas on the current frame;

- code for selecting a frame with at least one remaining missing area;

- a code for performing a spatial retouching procedure on said frame for recovering pixel data of said missing region; and

- a code for distributing pixel data from said selected frame to all frames of a sequence to which it is possible to extend said recovered pixel data using background motion information.

By reading and understanding the description below, those skilled in the art will understand that the claimed invention may also take other forms. The various steps of the method and system components may be implemented by hardware, software, firmware, or any suitable combination thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

After the above disclosure of the invention, the following is a detailed description of the inventive concept as an example and with reference to the accompanying drawings, which are given only as an illustration and are not intended to limit the scope of the claimed invention or to determine its essential features. In the drawings:

FIG. 1 illustrates the main components of a video processing system according to the invention.

In FIG. 2 shows the basic steps of retouching the background of a video image in accordance with the method of retouching the background of a video image according to the invention.

In FIG. 3 shows an example of the temporary integration of motion in accordance with the method according to the invention.

FIG. 4A-B illustrates the principles of distributing pixel data when propagating to and from the current frame using integrated motion in accordance with the method of the invention.

FIG. 5 illustrates the principles of obtaining background motion according to the invention.

DETAILED DESCRIPTION OF THE INVENTION

The present detailed description is provided to facilitate understanding of the invention. It should be noted that the description refers to exemplary embodiments of the invention, and upon careful reading of the description with reference to the accompanying drawings, other modifications, changes and equivalent replacements in the described objects of the invention may be envisaged by a person skilled in the art. All such obvious modifications, changes and equivalents are considered to be covered by the scope of the claimed invention. The reference numbers and symbols given in the present detailed description, as well as in the attached claims, are not intended to limit or determine the scope of the claimed invention in any way.

The claimed invention relates to a method for retouching a background of a video image working with pixel data, pixel-by-pixel fields of motion vectors and indicating missing areas (i.e., areas to be restored) in pixel data for each frame of a sequence of video data, for example, positions of foreground objects, all of which are mentioned data refer to one video sequence. A video sequence in the context of this document is a sequence of frames that could be shot with a single continuous camera shot without interruptions and instant transitions. The method according to the invention comprises the following steps, in which:

a. get background motion information from a series of frames of a video sequence,

b. for all frames with at least one missing area:

- distribute pixel data from one or more previous frames using background motion information,

- distributing pixel data from one or more subsequent frames using background motion information, and

- combine the above data with partially restored pixel data in one or more of these areas on the current frame;

c. repeat until all missing areas on all frames have been restored, the stages in which:

- choose a frame with at least one remaining missing area;

- perform a spatial retouching procedure on said frame to recover pixel data of said missing region; and

- distributing pixel data from said selected frame to all frames of a sequence into which it is possible to extend said recovered pixel data using background motion information.

In FIG. 1 shows the main components of a generalized system capable of executing the aforementioned algorithm. A program that implements the algorithm is stored in memory (101) along with the data required for the process. It is executed and controlled by one or more processors (100) and outputs the result either to memory or through a suitable display device (103). All data transfer occurs via the data bus (104).

In FIG. 2 shows the main steps of the background retouching algorithm of the video image on which the method according to the invention is based. The algorithm works with sequences of video data that contain one shot scene. In the case of many scenes, it is necessary to divide them into many subsequences and process each of them individually. At step (201), the system that implements the method according to the invention receives all the necessary input data, such as color or grayscale images for all frames in the sequence, information about missing areas for all frames in the sequence (indicated by any suitable means, for example, masks) . Also, for all frames in the sequence, pixel-by-pixel background motion fields (or their estimates) are required. The proposed background motion estimation scheme is explained below, see FIG. 5 for a detailed description. Further, for each frame in the sequence, starting from the first, the system performs steps (203a) and (203b) (which can be performed in parallel). In steps (203a) and (203b), the forward and backward motion fields originating in the current frame are temporarily integrated in the direction of each of the previous and subsequent frames (see the description below with reference to Fig. 3 for a more detailed description of the process) . This provides maps matching background pixels between the current (target) and any of the other frames (sources) in the sequence. If a pixel is marked as missing on the target frame, but there are one or more source frames on which the corresponding pixel is not missing, then the pixel from the source can be filled (“transferred” or “distributed”) with data taken from the said source frame ( for illustration see the top of Fig. 4). After that, perform step (204) mixing the available transferred pixels as follows: if there are two or more possible pixels for the target pixel that are taken from frames from the same search direction (for example, all from previous or subsequent frames), then select a source pixel that is closer in time to the source frame . If there are two possible pixels for the target pixel taken from frames from different search directions (one from the previous frame and one from the next), then mix with weighting factors depending on the time distance of each source pixel. After filling in the target pixel, it is marked as non-missing and the process continues for the next missing pixel. As a result of step (204), a frame is obtained on which at least part of the initially missing pixels are filled. After performing the above steps for each frame in the sequence, the algorithm proceeds to step (205). If some of the frames still have missing pixels, then at step (207) one of these pixels is selected for spatial retouching. Several criteria may be used, for example, based on the total area of the missing pixel data in the frame. Next, in step (208), the missing pixels are retouched by any suitable spatial retouching method, such as the method described in US Pat. No. 7,551,181 B2 (Criminisi et al.) “Image Region Filling by Exemplar-Based Inpainting”. The only requirement is that the selected method should synthesize a visually plausible structure. Therefore, preference is generally given to sample based methods. At step (209), the new information introduced by spatial retouching on the selected frame (source) is distributed to all neighboring (target) frames using the same process as described for steps (203a) and (203b) (see also Fig. 4B for explanation). Steps (205), (207) - (209) are repeated until there are no missing pixels in the data, then the algorithm proceeds to step (206) and ends.

FIG. 3 illustrates the process of temporarily integrating motion fields for three consecutive frames, starting from index N to N + 2 (marked as 300, 301 and 302, respectively). Temporary integration means the selection and addition of successive fields of vectors defined in the same image area, as follows. Suppose that element (303) is a motion vector with coordinates (u ₁ , v ₁ ) originating from the motion field (300) in some position (x, y). Then, the motion field (301) is sampled in the position (x + u ₁ , y + v ₁ ). In the case of subpixel motion accuracy, bilinear interpolation or any other type of interpolation can be used to obtain the motion vector (304) with coordinates (u ₂ , v ₂ ). Addition of these two vectors leads to element (305) - a vector, which can be considered as the integrated pixel offset at position (x, y) in frame (300) compared to frame (302). Exactly the same process is used when integrating backward in time from frame N to frames N-1, etc.

In FIG. 4A-B indicate the main differences and data flow for the distribution processes described in steps (203) and (209) of FIG. 2. In FIG. 4A shows the data stream for step (203), and FIG. 4B shows the data stream for step (209). Elements (400) in both cases represent a source frame or frames, elements (401) (colored black) in both cases correspond to missing areas on the target frames, elements (402) represent source regions, i.e. areas that will be copied to missing areas (401) in both cases, and elements (403) correspond to target areas that were previously absent and were updated with data taken from areas marked as (402). Element (404) is the target frame (or frames) to be updated by an algorithm. In FIG. 4A, data is distributed from all neighboring frames to the current frame, partially filling in various parts of the missing area (for details, see the description of step 203 of FIG. 2). In FIG. 4B, data is distributed from the full current frame to all adjacent frames, partially filling in the missing areas (for details, see the description of step 209 of Fig. 2). Propagation is performed as described above using integrated background motion fields.

FIG. 5 illustrates the basic concepts of the proposed method for obtaining background motion in accordance with the present invention. Reference numeral (500) denotes a full-frame field of motion vectors M ₀ (x, y) determined by the image area, which is used as input data evaluated in any way that provides a pixel-by-pixel motion vector field with the necessary accuracy, see, for example, RF patent No. RU 2012129183. Further, the reference position (501) denotes the edge of the missing region, which consists of regions (502) and (503). The algorithm for obtaining background motion synthesizes its output from three different fields of motion vectors. Outside the area surrounded by the contour (501), the motion vectors are taken equal to M ₀ (x, y). Then, inside the regions (502) and (503), the motion vectors are filled with the estimate M ₁ (x, y) obtained by adapting a certain motion model. For example, you can consider homography between planes defined by the following expressions:

Let (x ', y') = (x, y) + M₀(x, y), then by substituting H at the points formed with the vectors M₀(x, y) taken from region (504), the global field of motion vectors M can be calculated_one(x, y) over the image area in the form of the difference between the source points (x, y) and the points obtained from them by homographic transformation with the substituted parameters H (3 by 3 matrix). Area (504) is a strip widew (taken as a parameter of the algorithm indicated by the reference position (507)), which is at a distance d₂ pixels from path (501) (d ₂ is another algorithm parameter, indicated by (506) in FIG. 5). The purpose of this band is to prevent the introduction of artifacts that may occur when evaluating pixel-by-pixel motion around the edges of an object in global motion estimates. In addition, in the region (503), which is another band ond _one pixels inward from outline (501) (d_one - the parameter indicated by the reference position (505)), use another rating M₂(x, y) fields of motion vectors. The inventors propose the use of diffusion-based methods for assessing movement (retouching) in area (503) using M₀(x, y) as a source for retouching, as described in Telea, A. (2004). An image inpainting technique based on the fast marching method. Journal of graphics tools, 9 (1), 23-34. The result is two fields of motion vectors M_one(x, y), M₂(x, y), which contain estimates of the background motion inside the area surrounded by the contour (501). The next step is to mix them pixel by pixel within the region (503) to obtain the resulting field of motion vectors. This can be realized by choosing the appropriate weight function W (x, y), defined by the same area of the image. Let 0≤W (x, y) ≤1, then the resulting field of vectors is calculated as M (x, y) = (1-W (x, y)) · M_one(x, y) + W (x, y)₂(x, y). The purpose of this is to fix the local features of the movement and ensure a smooth transition from the initial estimates to the estimates of global movement in the frame. In a preferred embodiment, W (x, y) is used, decreasing exponentially (with an acceptable parameter of the decreasing rate in pixels) with a distance from the edge (501) of the missing region inside the missing region and equal to 1 outside the missing region. The result is a pixel-by-pixel estimate of the background motion vectors for the full frame, which generally has the properties of global motion within previously missing regions, but does not have problems associated with loss of continuity around the edges of the missing region.

The above method is applicable to the implementation in systems that convert video from 2D to 3D with the participation of the user, as well as to any other processing for improving video data, which requires retouching the video data. A typical core system with the necessary components that are common to a general purpose computing system has been described in more detail above, see description with reference to FIG. one.

By reading the above description with reference to the drawings, other aspects of the invention may be contemplated by those skilled in the art. One skilled in the art will understand that other embodiments of the invention are possible and that the details of the invention can be changed in various ways without departing from the scope of the inventive concept. Thus, the drawings and description are considered to be illustrative, but not restrictive. The scope of the claimed invention is determined only by the attached claims.

Claims (9)

1. A method of retouching the background of a video image using pixel data, the method comprising the steps of:
a. get background motion information from a series of frames of a video sequence,
b. for all frames with at least one missing area:
- transfer pixel data from one or more previous frames using information about the movement of the background,
- transfer pixel data from one or more subsequent frames using background motion information, and
- combine the above data with partially restored pixel data in one or more of these areas on the current frame;
c. repeat until all missing areas on all frames have been restored, the stages in which:
- choose a frame with at least one remaining missing area;
- perform a spatial retouching procedure on said frame to recover pixel data of said missing region; and
- transferring the restored pixel data from said selected frame to all frames of the sequence to which it is possible to transfer said restored pixel data using background motion information. 2. The method according to p. 1, in which obtaining information about the background motion comprises the steps of:
- evaluate the per-pixel motion for the full frame for a specific frame, and
- apply the spatial motion filling procedure to assess background motion in areas to be restored. 3. The method of claim 2, wherein the spatial motion filling procedure uses an estimate of global motion within missing regions and provides a smooth transition from pixel-by-pixel motion to global motion near the boundaries of the missing region. 4. The method according to claim 1, in which the transfer of pixel data using information about the movement of the background comprises the step of temporarily integrating the full-frame motion fields between the source frames and the target frames and transferring the available pixel data from the source frames to the indicated missing areas in the target frame. 5. A video processing system configured to retouch the background of a video image using pixel data, the system comprising:
at least one processor; and
a memory for storing data related to the processing of video data, as well as computer program instructions;
and when executed by the at least one processor, computer program instructions prompt the processor:
- receive information about the movement of the background from a series of frames of a video sequence;
- for all frames with at least one missing area:
- transfer pixel data from one or more previous frames using background motion information,
- transfer pixel data from one or more subsequent frames using background motion information, and
- combine the mentioned data with partially restored pixel data in one or more of the indicated areas on the current frame;
- select a frame with at least one remaining missing area;
- perform the spatial retouching procedure on said frame to recover pixel data of said missing region; and
- transfer pixel data from said selected frame to all frames of a sequence into which it is possible to transfer said recovered pixel data using background motion information. 6. The system of claim 5, wherein when said at least one processor is executed, computer program instructions further induce at least one processor:
- evaluate pixel-by-pixel motion for a full frame for a specific frame, and
- apply the spatial motion filling procedure to assess background motion in areas to be restored. 7. The system of claim 5, wherein when the at least one processor executes the computer program instructions, the at least one processor further encourages at least one processor to use the global motion estimate in the missing areas and to provide a smooth transition from pixel-by-pixel to global movement near the boundaries of the missing area. 8. The system of claim 5, wherein when the at least one processor executes the computer program instructions, additionally cause the at least one processor to temporarily integrate the full-frame fields of the motion vectors between the source frames and the target frames and transfer the available pixel data from the frame- sources to the indicated missing areas in the target frame. 9. A computer-readable medium on which a computer program is stored, which, when executed by at least one processor, causes the at least one processor to perform a method for retouching the background of the video image using pixel data, the computer program comprising:
- a code for obtaining information about the movement of the background from a series of frames of a video sequence,
- a code for transferring pixel data from one or more previous frames using background motion information,
- a code for transferring pixel data from one or more subsequent frames using background motion information, and
- a code for combining said data with partially restored pixel data in one or more of the indicated areas on the current frame;
- code for selecting a frame with at least one remaining missing area;
- a code for performing a spatial retouching procedure on said frame for recovering pixel data of said missing region; and
- a code for transferring pixel data from said selected frame to all frames of a sequence to which it is possible to transfer said recovered pixel data using background motion information.

Images