CN106851046A - Video dynamic super-resolution processing method and system - Google Patents

Abstract

The present invention relates to a kind of video dynamic super-resolution processing method and system, the processing method includes being obtained according to estimation the motion vector of any pixel in currently pending frame, and calculates the reliability of the motion vector；Motion vector according to the pixel extracts pixel corresponding pixel in previous higher resolution frame, is processed using Kalman filtering according to the corresponding pixel and obtains the first result；Interpolation processing is carried out to the pending frame and obtains second processing result；Fusion is weighted to first result and second processing result using the reliability, super-resolution frame is obtained after fusion.In the present invention, the problems such as realizing more preferable Protect edge information information and improve image border sawtooth, in the case where excessive extra time overhead is not increased, effectively recover more image details；There is preferable robustness simultaneously, it is to avoid the inaccurate disorder for causing super-resolution result of estimation or motion blur.

Description

Video dynamic super-resolution processing method and system

technical field

The invention belongs to video processing technology, in particular to video super-resolution (Super-resolution, SR) technology, in particular to a video dynamic super-resolution processing method and system.

Background technique

With the continuous improvement of the technology of the TV panel industry, the resolution of the display device increases rapidly, but the resolution of the video content does not increase rapidly accordingly. For example, ultra-high-definition TVs are becoming more and more common in the market, but due to the limitation of image acquisition equipment, the resolution of a large number of video sources is still standard-definition or high-definition, which greatly reduces the visual quality of the display. Therefore, it is very important to study the super-resolution (SR) algorithm that uses low-resolution image information to reconstruct high-resolution images.

According to the different information forms used, the current super-resolution technology methods can be roughly divided into two categories: static super-resolution technology (static super-resolution, SSR) method and dynamic super-resolution technology (dynamic super-resolution, DSR). )method. The purpose of the so-called static super-resolution technology is to reconstruct a high-resolution image through some low-quality low-resolution images. If these methods are directly used in video processing, they will often cause discontinuity in the timing of the enlarged video. , thus limiting the application of this type of method in video super-resolution technology, the proposed method of dynamic super-resolution technology is mainly aimed at the problem of video resolution reconstruction.

Static super-resolution technology methods are currently the most studied methods, and are usually divided into the following categories: interpolation-based methods, sample learning-based methods, and reconstruction-based methods. The interpolation-based method actually belongs to the image scaling technology, but many literatures classify this technology as the super-resolution technology. The method of interpolation is mainly to use the pixel values of the original pixel points in the neighborhood and assign different weights to them, and the value after weighted summation is used as the value of the pixel point to be interpolated. Common interpolation methods, such as nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc., are widely used. These methods can only change the size of the original low-resolution image and cannot provide more information or frequency. , the overall resolution of the image is still not improved. In the sample-based method, the selection of the sample library of this method has a great influence on the result, and there must be a separate set of sample library for each magnification, which is too complicated.

Dynamic super-resolution technology is used to solve video super-resolution problems. In fact, static super-resolution methods can be used in video super-resolution problems through expansion, but there are often problems such as high complexity, high cost, and discontinuity between frames. . For example, the online texture synthesis method introduces a super-resolution technology (Database-Free Texture Synthesis, DFTS) that does not require a sample library. Its core idea is to replace the restoration of real signals by improving the sensory resolution of the human eye. This method uses adjacent pixel information at the same position in adjacent frames to synthesize high-frequency information to generate a high-resolution image with sharp edges and rich details, but this method is only suitable for video images with a large number of similar structures , and the synthetic information available is limited. There is also a dynamic super-resolution method using Kalman filtering, which obtains corresponding pixel information through motion compensation in adjacent frames to obtain a more reliable super-resolution image, but this type of method is often restricted by the accuracy of motion estimation , the robustness of the algorithm is not very good. All in all, traditional dynamic super-resolution algorithms generally have the problem of difficult balance between complexity and reconstruction effect, and dynamic super-resolution algorithms using inter-frame motion information generally require motion vectors with higher sub-pixel precision, which cannot be achieved by motion estimation algorithms. Estimating accurate or complex motion videos tends to produce disordered results.

Contents of the invention

In order to solve the above-mentioned problems in the prior art, that is, in order to solve how to improve the image edge while ensuring the robustness of the super-resolution image, the present invention provides a video dynamic super-resolution processing method, including:

Obtain the motion vector of any pixel in the current frame to be processed according to the motion estimation, and calculate the reliability of the motion vector;

Extracting the pixel corresponding to the pixel in the previous high-resolution frame according to the motion vector of the pixel, and obtaining a first processing result by Kalman filtering according to the corresponding pixel;

performing interpolation processing on the frame to be processed to obtain a second processing result;

The reliability is used to perform weighted fusion on the first processing result and the second processing result, and super-resolution frames are obtained after fusion.

Preferably, said obtaining the motion vector of any pixel in the current frame to be processed according to the motion estimation includes:

Motion estimation is performed by using the previous high-resolution frame and the current frame to be processed to obtain the motion vector of any pixel in the current frame to be processed.

Preferably, after obtaining the super-resolution frame, it also includes:

Deblurring is performed on the super-resolution frame to obtain a deblurred high-resolution image.

Preferably, the fusion formula for performing weighted fusion of the first processing result and the second processing result by using the reliability is:

in, For the fusion result, is the reliability of the motion vector, is the first processing result, is the second processing result.

The present invention also provides a video dynamic super-resolution processing system, the processing system comprising:

A motion estimation unit, configured to obtain the motion vector of any pixel in the current frame to be processed according to motion estimation, and calculate the reliability of the motion vector;

The first processing unit is configured to extract the pixel corresponding to the pixel in the previous high-resolution frame according to the motion vector of the pixel, and obtain a first processing result by Kalman filter processing according to the corresponding pixel; Processing the frame to perform interpolation processing to obtain a second processing result;

A weighted fusion unit, configured to use the reliability to perform weighted fusion on the first processing result and the second processing result, and obtain a super-resolution frame after fusion.

Preferably,

The motion estimation unit is specifically configured to use the previous high-resolution frame and the current frame to be processed to perform motion estimation to obtain a motion vector of any pixel in the current frame to be processed.

Preferably, it also includes:

The second processing unit is configured to perform deblurring processing on the super-resolution frame to obtain a deblurred high-resolution image.

Preferably, the fusion formula for the weighted fusion performed by the weighted fusion unit is:

in, For the fusion result, is the reliability of the motion vector, is the first processing result, is the second processing result.

Compared with the prior art, the present invention has at least the following advantages:

Through the design of the present invention, better protection of edge information is achieved to improve image edge jaggedness and other issues, and more image details can be effectively restored without adding too much extra time overhead; at the same time, it has better robustness, Avoid inaccurate motion estimation causing disorder or motion blur in super-resolution results.

Description of drawings

Fig. 1 is a schematic flow chart of the video dynamic super-resolution processing method provided by the present invention;

Fig. 2 is a schematic diagram of the three-dimensional recursive search of the motion estimation algorithm used in the present invention;

Fig. 3 is a motion vector reliability mapping curve diagram used in the present invention;

Fig. 4 is a system architecture diagram of video dynamic super-resolution processing provided by the present invention.

detailed description

Preferred embodiments of the present invention are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present invention, and are not intended to limit the protection scope of the present invention.

The present invention proposes a video dynamic super-resolution processing method. The specific implementation of the present invention will be described in detail below in conjunction with the accompanying drawings.

As shown in Figure 1, the method specifically includes the following steps:

Step 101, obtain the motion vector of any pixel in the current frame to be processed according to the motion estimation, and calculate the reliability of the motion vector.

Wherein, said obtaining the motion vector of any pixel in the current frame to be processed according to the motion estimation includes:

Motion estimation is performed by using the previous high-resolution frame and the current frame to be processed to obtain the motion vector of any pixel in the current frame to be processed.

The motion estimation in this step can use any known motion estimation method, such as full search and three-dimensional recursive search. This embodiment adopts a three-dimensional recursive search algorithm, as shown in FIG. 2 .

Suppose CS represents the motion vector candidate set, with Denote the motion vector candidate sets of the space domain motion estimators Sa and Sb in Fig. 2 respectively, Indicates the coordinate position of the current area block in the two-dimensional image coordinate system, t is time, C _x and C _y represent motion vectors The horizontal and vertical components of , Indicates the estimated region block motion vector, T indicates the time interval between two adjacent frames, Represents the zero motion vector, CS ^max represents the range of candidate motion vectors, and also represents the search range of the three-dimensional recursive search algorithm, where the range of horizontal vectors is [-N,+N], and the range of vertical vectors is [-M,+ M], X Y represents the size of the matching area block, usually 8*8 or 16*16. Represents a random update motion vector, the update value randomly selects an update vector from the lookup table, where with are the horizontal and vertical components of the updated motion vector, respectively. The scope of each of the above collections is as follows:

When calculating the reliability of the motion vector, it specifically includes:

1) Calculate the motion vector variance

Note that MVx _i,j and MVy _i,j are the horizontal and vertical components of the motion vector of the input image at point (i,j) respectively. The variance V _i,j of motion vectors within a point (i,j) and its neighborhood is calculated by the following equation. Among them, the average value mx _i,j ,my _i,j of the horizontal and vertical directions of the motion vector, H and W are the height and width of the local window, and the values are 3 and 3, and the variance Vx _{i of the horizontal and vertical directions of the motion vector, j} ,Vy _i,j :

The variance V _{i, j} of motion vectors in point (i, j) and its neighborhood is expressed as formula:

2) Motion vector reliability curve mapping

The variance of motion vectors can be mapped to motion vector reliability according to FIG. 3 . The threshold and the shape of the curve can be set by yourself. To ensure that the variance of the motion vector is larger, the reliability of the motion vector is considered to be smaller. Figure 3 shows two types of mapping curves. In order to reduce risks, this paper chooses the curve controlled by the soft threshold in the right figure, but the trend of the curve should be designed to be "steeper". Because if the motion vector is not very accurate, then the information obtained from the reference frame and the information fusion of the current frame will inevitably cause motion blur. If the curve is set relatively flat, even if it is fused with the content in the airspace, it will not be greatly improved. , in this case, the image result will degenerate into the interpolation result of the space domain, and for more complex moving image content, the sensitivity of the human eye to sharpness is much lower than that of small motion or still image content. The thresholds for the curves are set to 4 for th ₁ and 6 for th ₂ .

Step 102, extract the pixel corresponding to the pixel in the previous high-resolution frame according to the motion vector of the pixel, and obtain the first processing result according to the corresponding pixel through Kalman filter processing; perform interpolation on the frame to be processed Processing obtains a second processing result.

Among them, when using Kalman filter processing, the final Kalman filter result is obtained by using the following predicted value and updated value where F(t) represents the displacement matrix, is the high-resolution prediction result at the previous moment, K(t) is the gain of Kalman filter, Is the input low-resolution image, D(t) represents downsampling, is the covariance predictor, is the update value of the covariance, which is used for the next frame processing, C _u (t) is the covariance matrix of the image result error caused by motion estimation, and C _w (t) is the covariance matrix of the random noise introduced by the image acquisition device.

Predictive value:

update value:

Step 103 , performing weighted fusion on the first processing result and the second processing result by using the reliability, and obtaining a super-resolution frame after fusion.

When performing weighted fusion, the fusion formula for performing weighted fusion on the first processing result and the second processing result by using the reliability is:

in, For the fusion result, is the reliability of the motion vector, is the first processing result, is the second processing result.

The purpose of fusion is to firstly focus on the first processing results to ensure the clearest super-resolution results under the condition of accurate motion estimation; secondly, the lower the reliability, the first processing results The lower the participation, the super-resolution result degenerates to the second processing result

As shown in Figure 4, the accuracy of the information of the previous generated high-resolution frame directly affects the next generated high-resolution frame. In order to obtain an ideal high-resolution image, a certain number of low-resolution frames are required to be processed by Kalman filtering several times. Therefore, this patent utilizes the obtained edge-guided information to speed up the convergence of the results and improve the accuracy of the results. In order not to add too much complexity, only the first high-resolution image Initialized as a high-resolution image with edge-guided interpolation Instead of edge guidance interpolation for each frame, because the iteration on the time axis will pass the edge guidance information acquired by the current frame to the next frame, and obtain a high-resolution image sequence with faster convergence and higher accuracy, so it is not necessary to Additional calculation and complexity are required.

The accuracy of motion estimation directly affects the processing results of video super-resolution. The motion estimation method used in this patent is a motion estimation method with sub-pixel precision. Therefore, one that affects sub-pixel precision is the design of the motion estimation algorithm itself, and the other is the use of depends on the accuracy of the estimated reference frame information. The motion compensation reference frame selected in this patent is a high-resolution frame estimated from a previous frame, and this high-resolution frame is an image that has not yet been deblurred. The macroscopic object that can be distinguished by the human eye in the image has a certain size, and the motion vector of a point in the object and its surrounding points should be relatively close. If the motion vector difference is large, the motion vector of this point may be inaccurate . For the block (sub-image) corresponding to the completely unreliable (reliability is 0) motion vector, its super-resolution result is directly replaced by the result of the space domain, and the completely reliable (reliability is 1) motion vector its super-resolution The rate results use the results of Kalman filtering, and for the reliability between 0 and 1, the results of spatial interpolation and Kalman filtering are fused according to the reliability.

Further, after obtaining the super-resolution frame, it also includes:

Deblurring is performed on the super-resolution frame to obtain a deblurred high-resolution image.

The adopted deblurring method is a method based on maximum a posteriori probability, but is not limited to this method. The formula for solving the deblurred high-resolution image is:

in Represents the high-resolution image after deblurring, H is low-pass filtering, J(X(t)) is a regularization function, and the regularization term is to provide some prior information to solve the super-resolution ill-conditioned problem, which is used to increase the convergence And to suppress the influence caused by noise and motion estimation error, λ is a regularization parameter used to balance the fidelity term on the left and the regular term on the right in the formula, and the regular term appears in the cost equation as a penalty term. There are many forms of regularization items, and some regularization items are designed for special image processing requirements (such as maintaining edges or removing motion smear noise, etc.).

Use the steepest gradient descent method to solve the minimization problem. The specific calculation form is as follows:

where β is the iteration step size, is true differential. The final termination iteration condition is as follows, where η is the threshold condition for iteration termination, n is the number of current iterations, and iter _max is the maximum number of iterations:

Or n ≥ iter _max .

Based on the same thinking as the method provided by the above-mentioned present invention, the present invention also provides a video dynamic super-resolution processing system, including:

The motion estimation unit is used to obtain the motion vector of any pixel in the current frame to be processed according to the motion estimation, and calculate the reliability of the motion vector; specifically, it is used to perform motion by using the previous high-resolution frame and the current frame to be processed Estimated to obtain the motion vector of any pixel in the current frame to be processed.

A processing unit, configured to extract a pixel corresponding to the pixel in a previous high-resolution frame according to the motion vector of the pixel, and obtain a first processing result by Kalman filter processing according to the corresponding pixel; for the frame to be processed performing interpolation processing to obtain a second processing result;

A weighted fusion unit, configured to use the reliability to perform weighted fusion on the first processing result and the second processing result, and obtain a super-resolution frame after fusion.

The fusion formula for the weighted fusion performed by the weighted fusion unit is:

in, For the fusion result, is the reliability of the motion vector, is the first processing result, is the second processing result.

The second processing unit is configured to perform deblurring processing on the super-resolution frame to obtain a deblurred high-resolution image.

Those skilled in the art should be able to realize that the modules and method steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the electronic hardware and Interchangeability of software. In the above description, the components and steps of each example have been generally described according to their functions. Whether these functions are performed by electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may implement the described functionality using different methods for each particular application, but such implementation should not be considered as exceeding the scope of the present invention.

So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings, but those skilled in the art will easily understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to relevant technical features, and the technical solutions after these changes or substitutions will all fall within the protection scope of the present invention.

Claims (8)

1. A video dynamic super-resolution processing method is characterized by comprising the following steps:

obtaining a motion vector of any pixel in a current frame to be processed according to motion estimation, and calculating the reliability of the motion vector;

extracting a pixel corresponding to the pixel in a previous high-resolution frame according to the motion vector of the pixel, and obtaining a first processing result by utilizing Kalman filtering processing according to the corresponding pixel;

carrying out interpolation processing on the frame to be processed to obtain a second processing result;

and performing weighted fusion on the first processing result and the second processing result by using the reliability, and obtaining a super-resolution frame after fusion.

2. The video dynamic super-resolution processing method according to claim 1, wherein said obtaining a motion vector of any pixel in a current frame to be processed according to motion estimation comprises:

and performing motion estimation by using the previous high-resolution frame and the current frame to be processed to obtain a motion vector of any pixel in the current frame to be processed.

3. The video dynamic super-resolution processing method according to claim 1, further comprising, after obtaining the super-resolution frame:

and carrying out deblurring processing on the super-resolution frame to obtain a deblurred high-resolution image.

4. The video dynamic super-resolution processing method according to claim 1, wherein the fusion formula for performing weighted fusion on the first processing result and the second processing result by using the reliability is as follows:

${\[\widehat{\underset{\‾}{Z}}\left(t\right)\]}_q=\left({\[\underset{\‾}{r}\left(t\right)\]}_q{\[{\widehat{\underset{\‾}{Z}}}_K\left(t\right)\]}_q+0.2*{\[{\widehat{\underset{\‾}{Z}}}_S\left(t\right)\]}_q\right)/({\[\underset{\‾}{r}\left(t\right)\]}_q+0.2)$

wherein,to fuse the results, [ r (t) ]]As a measure of the reliability of the motion vector,as a result of the first processing,is the second processing result.

5. A video dynamic super-resolution processing system, the processing system comprising:

the motion estimation unit is used for obtaining a motion vector of any pixel in the current frame to be processed according to motion estimation and calculating the reliability of the motion vector;

the first processing unit is used for extracting a pixel corresponding to the pixel in a previous high-resolution frame according to the motion vector of the pixel, and obtaining a first processing result by utilizing Kalman filtering processing according to the corresponding pixel; carrying out interpolation processing on the frame to be processed to obtain a second processing result;

and the weighted fusion unit is used for carrying out weighted fusion on the first processing result and the second processing result by utilizing the reliability, and obtaining a super-resolution frame after fusion.

6. The video dynamic super-resolution processing system according to claim 5,

the motion estimation unit is specifically configured to perform motion estimation by using a previous high-resolution frame and a current frame to be processed to obtain a motion vector of any pixel in the current frame to be processed.

7. The video dynamic super-resolution processing system according to claim 5, further comprising:

and the second processing unit is used for carrying out deblurring processing on the super-resolution frame to obtain a deblurred high-resolution image.

8. The video dynamic super-resolution processing system according to claim 5, wherein the fusion formula of the weighted fusion unit for weighted fusion is:

${\[\widehat{\underset{\‾}{Z}}\left(t\right)\]}_q=\left({\[\underset{\‾}{r}\left(t\right)\]}_q{\[{\widehat{\underset{\‾}{Z}}}_K\left(t\right)\]}_q+0.2*{\[{\widehat{\underset{\‾}{Z}}}_S\left(t\right)\]}_q\right)/({\[\underset{\‾}{r}\left(t\right)\]}_q+0.2)$

wherein,as a result of fusionr(t)]As a measure of the reliability of the motion vector,as a result of the first processing,is the second processing result.