CN116208730A - Method, device, equipment and storage medium for improving video clarity - Google Patents

Abstract

The application discloses a method, a device, equipment and a storage medium for improving video definition, wherein the method comprises the following steps: determining optimal computing power and required video of the terminal equipment based on a video request sent by the terminal equipment, and sending a superdivision model and definition level of the determined required video based on the optimal computing power to the terminal equipment; screening out a superdivision list from the required video based on the target definition selected from the definition level by the terminal equipment, wherein the superdivision list records superdivision image blocks needing superdivision in the required video; and sending the superdivision list and the video blocks to the terminal equipment so that the terminal equipment can obtain the high-definition video corresponding to the required video based on the superdivision image blocks recorded by the superdivision list of the superdivision model. In the method, any terminal equipment can obtain the super-division model meeting the self optimal calculation power, and the calculation amount is reduced by reducing the image blocks needing super-division, so that the terminal equipment can provide high-definition quality video for users.

Description

Method, device, equipment and storage medium for improving video clarity

technical field

The present application relates to the technical field of wireless communication, and in particular to a method, device, device and storage medium for improving video clarity.

Background technique

Due to the increasing demand of users to watch high-definition video through terminal equipment, the current network bandwidth is limited, so that the terminal equipment cannot provide high-quality video to users.

At present, the computing power of terminal equipment has been enhanced to convert low-definition video into high-definition video through enhanced computing power, so that terminal equipment can still enable users to watch high-definition video when the network bandwidth is limited. In order to make full use of The CPU (central processing unit, central processing unit) and GPU (graphics processing unit, image processor) and other resources in the terminal device reduce the memory requirements and use the super-resolution model to process the video. The terminal device generally processes the video. Super-resolution operation. However, the super-resolution model is a deep neural model, which requires high computing power. The computing power of some terminal devices cannot meet the super-resolution model. As a result, terminal devices whose computing power does not meet the super-resolution model still cannot provide users with high-definition quality video. .

Contents of the invention

The main purpose of this application is to provide a method, device, device, and storage medium for improving video clarity, aiming to solve the problem that the terminal equipment in the prior art whose computing power does not meet the super-resolution model cannot provide users with high-definition quality video technology question.

In order to achieve the above purpose, the present application provides a method for improving video clarity, and the method for improving video clarity includes:

Based on the video request sent by the terminal device, determine the optimal computing power of the terminal device and the required video, and send the super-resolution model and the determined video of the required video based on the optimal computing power to the terminal device clarity level;

Based on the target definition selected by the terminal device from the definition level, a super-resolution list is selected from the required video, wherein the super-resolution list records the super-resolution image blocks in the required video that need to be super-resolution ;

Sending the super-resolution list and the video blocks to the terminal device, so that the terminal device can super-resolution the super-resolution image blocks recorded in the super-resolution list based on the super-resolution model, and obtain the desired video Corresponding high-definition video.

Optionally, the step of selecting a super-resolution list from the required video based on the target definition selected by the terminal device from the definition level includes:

Acquiring the video blocks whose required video resolution is the target resolution;

Decoding the video blocks at a low-definition frame rate to obtain macroblock decoding information for each of the video blocks;

Based on the decoding information of the macroblock, divide the image block in the video block into an inter image block and an intra image block, and determine the inter frame super-resolution weight of the inter frame image block and the intra frame image block. The intra-frame super-resolution weight;

Based on the inter-frame super-resolution weight, the intra-frame super-resolution weight and a preset search algorithm, determine the super-resolution list of the desired video.

Optionally, based on the macroblock decoding information, the image blocks in the video block are divided into inter image blocks and intra image blocks, and the inter frame super-resolution weight of the inter frame image blocks is determined. steps, including:

determining an inter macroblock within each of said video blocks based on said macroblock decoding information;

defining an image block containing the inter macroblock as an inter image block;

Calculating the degree of dependence and the degree of error diffusion between the image blocks between the frames;

Based on the product of the degree of dependence and the degree of error diffusion, divide the inter-frame image block into multiple inter-frame lists, where each division of the inter-frame list requires recalculating the remaining inter-frame image blocks The degree of dependence and error diffusion between

Based on the degree of dependence, the inter-frame super-resolution weights of the inter-frame image blocks in each of the inter-frame lists are determined.

Optionally, based on the macroblock decoding information, the image blocks in the video block are divided into inter image blocks and intra image blocks, and the intra frame super-resolution weight of the intra frame image blocks is determined steps, including:

determining intra macroblocks within each of the video blocks based on the macroblock decoding information;

defining an image block containing the intra macroblock as an intra image block;

calculating a dependency estimate of the intra-frame image block, and modifying the dependency estimate based on the preset texture complexity of the intra-frame image block to obtain a modified dependency;

Based on the sorting of the modified dependencies, the intra-frame image blocks are divided into multiple intra-frame lists in equal proportions, and the intra-frame super-resolution weight of each of the intra-frame lists is determined.

Optionally, the step of determining the super-resolution list of the desired video based on the inter-frame super-resolution weight, the intra-frame super-resolution weight and a preset search algorithm includes:

Based on the inter-frame super-resolution weight, the intra-frame super-resolution weight and a preset search algorithm, calculate the image with the smallest super-resolution area when the definition of each image block in the video block reaches the target definition. block combination;

The image blocks in the combination of image blocks are defined as super-resolution image blocks, and a super-resolution list is determined based on the super-resolution image blocks.

Optionally, the step of sending the super-resolution list and the video blocks to the terminal device includes:

Packing the super-resolution list and the video blocks into a run-length packet based on a preset run-length encoding;

Comparing the amount of data between the run-length data packet and an ordinary data packet that is not packaged with run-length encoding;

Selecting a sending data packet with a small amount of data from the run length data packet and the normal data packet, and sending the super resolution list and the video block to the terminal device based on the sending data packet.

Optionally, the step of determining the super-resolution model of the desired video based on the optimal computing power includes:

Selecting a super-resolution model whose computing power is less than or equal to the optimal computing power from the super-resolution model group corresponding to the desired video;

Wherein, each of the required videos corresponds to a plurality of super-resolution models with different calculation difficulties.

The present application also provides a device for improving video clarity, and the device for improving video clarity includes:

A determining module, configured to determine the optimal computing power and required video of the terminal device based on the video request sent by the terminal device, and send the required video determined based on the optimal computing power to the terminal device The super-resolution model and definition level;

A selection module, configured to filter out a super-resolution list from the required video based on the target definition selected by the terminal device from the definition level, wherein the super-resolution list records that the required video requires super-resolution The super-resolution image block of ;

A sending module, configured to send the super-resolution list and the video blocks to the terminal device, so that the terminal device can super-resolution the super-resolution image blocks recorded in the super-resolution list based on the super-resolution model, and obtain The high-definition video corresponding to the desired video.

The present application also provides a device for improving video clarity. The device for improving video clarity is a physical node device. The device for improving video clarity includes: a memory, a processor, and an improved device stored on the memory and operable on the processor. The program of the method for video definition, when the program of the method for improving video definition is executed by the processor, can realize the steps of the above-mentioned method for improving video definition.

The present application also provides a storage medium, on which a program for realizing the above-mentioned method for improving video definition is stored, and when the program for improving video definition is executed by a processor, the steps of the above-mentioned method for improving video definition are realized .

This application provides a method, device, device, and storage medium for improving video clarity. Compared with terminal equipment in the prior art whose computing power does not meet the super-resolution model and cannot provide users with high-definition quality video, in this application, Based on the video request sent by the terminal device, determine the optimal computing power of the terminal device and the required video, and send the super-resolution model and the determined video of the required video based on the optimal computing power to the terminal device Clarity level; based on the target definition selected by the terminal device from the clarity level, a super-resolution list is screened from the required video, wherein the super-resolution list records the required video. Super-resolution image blocks; sending the super-resolution list and the video blocks to the terminal device, so that the terminal device can super-resolution super-resolution image blocks recorded in the super-resolution list based on the super-resolution model, to obtain The high-definition video corresponding to the desired video. In this application, according to the optimal computing power of the terminal device, the super-resolution model corresponding to the required video is sent to the terminal device, and then according to the target definition selected by the terminal device from the definition level of the required video, the terminal The device sends the video blocks and the super-resolution list that can reach the target resolution after super-resolution, so that the terminal device can obtain the high-definition video corresponding to the desired video by using the super-resolution image blocks recorded in the super-resolution super-resolution list, that is, in this application In this method, the super-resolution model that satisfies the terminal’s optimal computing power is sent to the terminal device, and the screened super-resolution list is sent to the terminal, so that any terminal device can obtain a super-resolution model that meets its own optimal computing power, and reduce The calculation amount of the terminal device, so that the terminal device can provide users with high-definition quality video.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description serve to explain the principles of the application.

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, In other words, other drawings can also be obtained from these drawings on the premise of not paying creative work.

FIG. 1 is a schematic flow chart of a first embodiment of a method for improving video clarity in the present application;

Fig. 2 is a schematic diagram of a system framework for improving video clarity in the present application;

FIG. 3 is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application;

FIG. 4 is a schematic flowchart of a second embodiment of a method for improving video clarity in the present application.

The realization, functional features and advantages of the present application will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

Detailed ways

It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application.

The embodiment of the present application provides a method for improving video clarity. In the first embodiment of the method for improving video clarity of the present application, referring to FIG. 1, the method for improving video clarity includes:

Step S10, based on the video request sent by the terminal device, determine the optimal computing power and the required video of the terminal device, and send the super video of the required video determined based on the optimal computing power to the terminal device Sub-model and clarity level;

Step S20, based on the target definition selected by the terminal device from the definition level, filter out a super-resolution list from the desired video, wherein the super-resolution list records the super-resolution list of the desired video that needs to be super-resolution sub-image blocks;

Step S30, sending the super-resolution list and the video blocks to the terminal device, so that the terminal device can super-resolution the super-resolution image blocks recorded in the super-resolution list based on the super-resolution model, and obtain the The HD video corresponding to the desired video.

The purpose of this embodiment is to enable each terminal device to obtain a super-resolution model that satisfies its own optimal computing power, so that each terminal device can provide users with high-definition quality video.

In this embodiment, it should be noted that the method for improving video definition can be applied to a device for improving video definition, and the device for improving video definition is subordinate to the device for improving video definition, and the device for improving video definition It belongs to the system of improving video clarity.

In this embodiment, the system for improving video clarity includes a server and a terminal device, wherein the terminal device is used to send a request instruction for the video required by the user to the server; it is also used to receive the super-resolution information sent by the server, and send The server sends a quality request determined based on the super-resolution information, wherein the super-resolution model in the super-resolution information is determined based on its own optimal computing power; it is also used to receive the video block sent by the server and The super-resolution list, based on the super-resolution list and the preset decoder, determines the image blocks to be processed that need to be super-resolution in the video block, and the original video blocks that do not need to be super-resolution; A model that performs super-resolution on the image block to be processed to obtain a super-resolution image block; and is also used to decode the super-resolution image block and the original video based on the sequence of video frames in the decoder and the video block blocks to obtain the high-definition video corresponding to the desired video.

In this embodiment, in the server, each video corresponds to a group of super-resolution model groups, and each group of super-resolution model groups has a plurality of super-resolution model groups with different calculation difficulties, and then according to the optimal Computational power, select a super-resolution model whose computational difficulty meets the optimal computing power of the terminal device from the super-resolution model group, so that each terminal device can obtain the super-resolution model of the required video according to its own computing power, that is, Each terminal device can use the super-resolution model to convert low-definition video into high-definition video and improve the clarity of the video.

The target resolution may be determined by the terminal device according to the current bandwidth, so that the terminal device can play the video with the highest resolution under the current bandwidth and play it stably.

In this embodiment, the terminal device receives the super-resolution model, tightly couples the super-resolution model with the decoder in the terminal device, and uses the super-resolution model to perform super-resolution on the super-resolution image blocks recorded in the super-resolution list, and then uses The decoder decodes the high-definition compressed video obtained after super-resolution to obtain high-definition video. That is, in this embodiment, image blocks that need to be super-resolved by the terminal device are reduced through the super-resolution list, so as to reduce the calculation amount of the terminal device.

It should be noted that the video can be divided into multiple fixed-length video blocks, that is, each video block is composed of the same number of video frames, and the video is composed of multiple video blocks. Finally, the video is divided into video blocks, and by analyzing the image blocks in the video blocks, the super-resolution image blocks that need to be super-resolution are determined, and the corresponding super-resolution image blocks are marked in the super-resolution list for terminal equipment. According to the marks in the super-resolution list, use the super-resolution model to perform super-resolution on the corresponding super-resolution image block.

In this embodiment, after receiving the super-resolution list and video blocks, the terminal device first decodes the super-resolution list, so as to obtain the marks of the super-resolution image blocks from the super-resolution list, and then finds the corresponding Super-resolution image blocks, perform super-resolution on super-resolution image blocks, and convert the remaining image blocks that do not need super-resolution into high-definition compressed image blocks, and finally combine high-definition compressed image blocks and super-resolution image blocks according to the timing of the required video , decoding the high-definition compressed image block and the super-resolution image block to obtain a high-definition video corresponding to the target definition.

In this implementation, refer to FIG. 2 , which is a schematic diagram of a system framework for improving video definition in this application.

Specific steps are as follows:

Step S10, based on the video request sent by the terminal device, determine the optimal computing power and the required video of the terminal device, and send the super video of the required video determined based on the optimal computing power to the terminal device Sub-model and clarity level;

Among them, "SR Models with Different Complexities" in Figure 2 refers to "super-resolution models with different difficulty levels", "Download pretrained SR models" refers to "download super-resolution models", and "Video Chunks and Patch Lists" refers to "video chunks and super-resolution models". Sub-list", "Server Side" is "server side", "Client Side" is "mobile terminal", that is, terminal equipment, "ABR Controller" is "adaptive streaming algorithm", "SR-Codec" is "super resolution Decoder" that is, decoder, "Decode Lists" is "decoding list".

In this embodiment, before the server sends the super-resolution model to the terminal device, it first receives the video request of the terminal device, and determines the optimal computing power of the terminal device according to the video request, and the video requested by the terminal device, that is, the required video .

Wherein, the required video may be a video name or a video code required by the terminal device, which is not specifically limited.

In this embodiment, after the server receives the video request, the server obtains the basic configuration information of the terminal device according to the video request, and infers the optimal computing power of the terminal device based on the basic information, and sends a request to the terminal device according to the optimal computing power. Send the super-resolution model of the required video and all the definition levels of the required video, so that the terminal device can choose one of the definition levels according to its own bandwidth environment, so that the desired video can be played normally, and the definition is high target clarity.

Wherein, the definition level may be a definition level such as low-definition, high-definition, ultra-definition, or 4k, which is not specifically limited.

In this embodiment, the terminal device is provided with a decoder, and after receiving the super-resolution model, the terminal device embeds the super-resolution model in the decoder, so that the decoder and the super-resolution model are tightly coupled.

It should be noted that the server pre-trains proprietary super-resolution models of different complexity for each video, adapting to terminal devices with different computing capabilities, and the complexity of the super-resolution models varies.

Step S20, based on the target definition selected by the terminal device from the definition level, filter out a super-resolution list from the desired video, wherein the super-resolution list records the super-resolution list of the desired video that needs to be super-resolution Divide image blocks.

It should be noted that an adaptive streaming algorithm is set in the terminal device, and after receiving the definition level, the terminal device selects the target definition used in the current bandwidth environment from the definition level through the adaptive streaming algorithm.

In this embodiment, after the terminal device determines the target resolution, it sends the video block request information with the target resolution to the server, so that the server can send the corresponding video block and super-resolution list to the terminal device, and the terminal device receives After arriving at the super-resolution list, first decode the super-resolution list, and block the super-resolution images in the video block that need to be super-resolution according to the super-resolution list and store them in the decoder.

In this embodiment, the server divides the video content corresponding to the definition into video blocks according to the target definition, selects the super-resolution image block that needs super-resolution from the video blocks, and records the encoding corresponding to the super-resolution image block in In the super-resolution list, the number of super-resolution image blocks of the terminal device is reduced, the calculation amount of the terminal device is reduced, and the computing power of the terminal device is saved.

Specifically, the step of screening out the super-resolution list from the desired video based on the target definition selected by the terminal device from the definition level includes:

Step S21, acquiring video blocks whose required video resolution is the target resolution;

Step S22, performing low-definition frame rate decoding on the video block, and obtaining macroblock decoding information of each video block;

Step S23, based on the decoding information of the macroblock, divide the image block in the video block into an inter image block and an intra image block, and determine the inter frame over-resolution weight of the inter image block and the frame The intra-frame super-resolution weight of the inner image block;

Step S24: Determine the super-resolution list of the desired video based on the inter-frame super-resolution weight, the intra-frame super-resolution weight and a preset search algorithm.

Wherein, the macroblock decoding information includes motion vectors, quantization parameters and residuals.

Among them, the motion vector can describe the reference order and frequency between image blocks or macroblocks. Residuals and quantization parameters can describe the error spread and degree of dependence.

Wherein, the degree of dependence may be the degree of quality improvement for subsequent videos when an image block or macroblock is selected for super-resolution. The error diffusion refers to the super-resolution quality loss generated in the process of propagating from the last selected super-resolution image block or macroblock to the current image block or macroblock.

In this embodiment, the server can use the open source code of the decoder to decode the low-definition compressed video in the service to obtain the macroblock decoding information. Macroblock decoding information.

In this embodiment, the image blocks in the video block are divided into inter-frame image blocks and intra-frame image blocks by using the macroblock decoding information, wherein the macroblock decoding information includes inter-frame macroblocks and intra-frame macroblocks, if If the image block contains an intra-frame macroblock, the image block is an intra-frame image block, and if the image block contains an inter-frame macroblock, then the image block is an intra-frame red block.

In this embodiment, the dependent estimation of each intra-frame image block is estimated according to the macroblock decoding information, and the intra-frame image blocks are sorted according to the dependent estimation to determine the intra-frame super-resolution weight of the intra-frame image block, and calculate The dependence degree and error diffusion value of each inter-frame macroblock are obtained, and the inter-frame super-resolution weight of the inter-frame image block is calculated by using the dependence degree and error diffusion value.

In this embodiment, due to the different target resolutions of the video blocks, the number of super-resolution image blocks finally selected is also different. In the process, the images that meet the target definition and need to be super-resolution are screened out quickly, so that the terminal device can reduce the amount of calculation, and enable the terminal device to play the video of the target definition.

Specifically, the step of dividing the image block in the video block into an inter image block and an intra image block based on the macroblock decoding information, and determining the inter frame super-resolution weight of the inter image block ,include:

Step A10, based on the macroblock decoding information, determine an inter-frame macroblock in each of the video blocks;

Step A20, defining the image block containing the inter macroblock as an inter image block;

Step A30, calculating the degree of dependence and the degree of error diffusion between the inter-frame image blocks;

Step A40: Divide the inter-frame image block into multiple inter-frame lists based on the product of the degree of dependence and the degree of error diffusion, wherein each division of one of the inter-frame lists requires recalculation of the remaining inter-frame Dependency and error diffusion between image blocks;

Step A50, based on the degree of dependence, determine the inter-frame super-resolution weights of the inter-frame image blocks in each of the inter-frame lists.

In this embodiment, the degree of dependence of each macroblock is initialized according to the size of the area, and the backward propagation is performed through the decoded reference relationship, and finally the degree of dependence of each macroblock is obtained.

In this embodiment, since the definition of the degree of dependence is to estimate the quality improvement of subsequent video frames by super-dividing an image block or macroblock, however, since inter-frame macroblocks need to be decoded with reference to other inter-frame macroblocks, there will be some After the inter-frame macroblock is decoded, the definition drops to the preset minimum resolution. At this time, it is a waste of computing power to continue to refer to the inter-frame macroblock to decode the next inter-frame macroblock. After the degree of loss reaches a certain level, that is, when the decoded resolution reaches the minimum degree of cleanliness, recalculate the dependence degree and error diffusion degree of the remaining inter-frame image blocks, and determine in the new inter-frame list Inter-frame super-resolution weights. In this embodiment, by replacing the inter-frame super-resolution list, the loss of video quality after super-resolution can be reduced.

In this embodiment, when the product of the error diffusion value and the dependence degree of an inter-frame image block exceeds a set threshold, the over-resolved image block will be selected and added to a new inter-frame list and the error diffusion value will be set to 0. And set different thresholds to obtain different inter-frame lists.

Wherein, different thresholds correspond to different inter-frame lists and different numbers of inter-frame image blocks.

In this city's implementation, since the degree of dependence of macroblocks can be calculated using the reference relationship between macroblocks and decoding information modeling. Due to the intra-frame and inter-frame coding of macroblocks, there are reference relationships between different macroblocks. Since inter macroblocks can only be decoded with reference to already reconstructed macroblocks, the dependencies of inter macroblocks can be propagated backwards to those referenced macroblocks, and the dependencies of these macroblocks can be accumulated.

Specifically, the step of dividing the image block in the video block into an inter image block and an intra image block based on the macroblock decoding information, and determining the intra frame super-resolution weight of the intra frame image block ,include:

Step B10, based on the macroblock decoding information, determine intra-frame macroblocks in each of the video blocks;

Step B20, defining the image block containing the intra-frame macroblock as an intra-frame image block;

Step B30, calculating the dependency estimation of the image block in the frame, and correcting the dependency estimation based on the preset texture complexity of the image block in the frame to obtain the modified dependency;

Step B40, based on the ranking of the corrected dependencies, divide the intra-frame image blocks into multiple intra-frame lists in equal proportions, and determine the intra-frame super-resolution weight of each of the intra-frame lists.

In this embodiment, the dependency estimate of the image block in the frame is added to the image texture complexity for correction to obtain the correction dependency. Since image blocks with complex textures tend to get more high-frequency information after super-resolution, and image blocks with simple textures tend to get less high-frequency information after super-resolution, so the use of texture complexity correction will rely on estimation, which can improve the frame rate. The accuracy rate of the inner overweight weight.

In this embodiment, the mean square error of the low-resolution image block after bilinear interpolation amplification and super-resolution amplification is normalized with the mean square error of the entire picture, and corrected as a coefficient of dependence degree, which can ensure When more useful information is generated by super-resolution, the coefficient will be larger, and the priority of super-resolution will be increased.

In this embodiment, the intra-frame image block sorting is obtained according to the correction dependency, and then the intra-frame image blocks are divided into multiple intra-frame lists in equal proportions. According to the sorting of intra-frame image blocks in each intra-frame list, each Intra-frame super-resolution weights.

Step S30, sending the super-resolution list and the video blocks to the terminal device, so that the terminal device can super-resolution the super-resolution image blocks recorded in the super-resolution list based on the super-resolution model, and obtain the The HD video corresponding to the desired video.

In this embodiment, after sending the super-resolution list and video blocks to the terminal device, the terminal device determines the image blocks to be processed in the video blocks that need super-resolution and the original video blocks that do not need super-resolution according to the super-resolution list and the decoder. block, and then according to the super-resolution model, perform super-resolution on the image block to be processed to obtain the super-resolution image block, and decode the super-resolution image block and the original video block according to the order of the decoder and the video frame in the video block, and obtain the required video corresponding HD video.

In this embodiment, the step of using the decoder to decode the super-resolution image block and the original video block to obtain the high-definition video corresponding to the desired video may be the operation of obtaining the current video frame of the video block by using the frame sequence of the decoder and the video frame The decoding information, and the reference decoding information of the latest decoded video frame, according to the operation decoding information and the reference decoding information, decode the original video frame corresponding to the original video block, or the super-resolution video frame corresponding to the super-resolution image block, to obtain the high-definition video frame, and then According to the frame sequence, the high-definition video frames are integrated to obtain the high-definition video corresponding to the desired video.

In this embodiment, according to the super-resolution list and the preset decoder, the step of determining the image blocks to be processed that require super-resolution and the original video blocks that do not need super-resolution in the video block may be based on the preset decoder Decode the super-resolution list, and determine the super-resolution position information of the image block that needs to be super-resolution in the video block from the super-resolution list, decode the video block according to the decoder, and obtain the low-definition video frame of the video block, according to the super-resolution position information, The image blocks to be processed that require super-resolution are screened out from the low-definition video frames, and the video blocks of the low-definition video frames that do not have image blocks to be processed are defined as original video blocks.

In this embodiment, during the decoding process of the terminal device, the decoding information is used to perform bilinear interpolation and amplification on the intra-frame image blocks and residual information, and the motion vector is enlarged according to the amplification ratio, so as to construct a high-definition reference frame queue . According to the super-resolution list received from the server, the decoder selects image blocks in the super-resolution low-definition reference frame queue and refreshes them at corresponding positions in the high-definition reference frame queue. The decoder can reconstruct high-resolution images according to the high-definition reference frame queue. The tight coupling of decoding and super-resolution reduces computational requirements in the way of decoding.

This application provides a method, device, device, and storage medium for improving video clarity. Compared with terminal equipment in the prior art whose computing power does not meet the super-resolution model and cannot provide users with high-definition quality video, in this application, Based on the video request sent by the terminal device, determine the optimal computing power of the terminal device and the required video, and send the super-resolution model and the determined video of the required video based on the optimal computing power to the terminal device Clarity level; based on the target definition selected by the terminal device from the clarity level, a super-resolution list is screened from the required video, wherein the super-resolution list records the required video. Super-resolution image blocks; sending the super-resolution list and the video blocks to the terminal device, so that the terminal device can super-resolution super-resolution image blocks recorded in the super-resolution list based on the super-resolution model, to obtain The high-definition video corresponding to the desired video. In this application, according to the optimal computing power of the terminal device, the super-resolution model corresponding to the required video is sent to the terminal device, and then according to the target definition selected by the terminal device from the definition level of the required video, the terminal The device sends the video blocks and the super-resolution list that can reach the target resolution after super-resolution, so that the terminal device can obtain the high-definition video corresponding to the desired video by using the super-resolution image blocks recorded in the super-resolution super-resolution list, that is, in this application Among them, the super-resolution model that satisfies the terminal’s optimal computing power is sent to the terminal device, and the screened super-resolution list is sent to the terminal, so that any terminal device can obtain a super-resolution model that meets its own optimal computing power, and reduce The calculation amount of the terminal device, so that the terminal device can provide users with high-definition quality video.

Further, based on the above-mentioned embodiments in this application, another embodiment of this application is provided. In this embodiment, referring to FIG. 4 , the inter-frame over-resolution weight, the intra-frame over-resolution weight and the The established search algorithm determines the steps of the super-resolution list of the desired video, including:

Step S01, based on the inter-frame super-resolution weight, the intra-frame super-resolution weight and the preset search algorithm, calculate the required super-resolution area when the resolution of each image block in the video block reaches the target resolution The smallest combination of image blocks;

Step S02, defining the image blocks in the combination of image blocks as super-resolution image blocks, and determining a super-resolution list based on the super-resolution image blocks.

In this embodiment, after obtaining the intra-frame and inter-frame lists, find a suitable combination of intra-frame and inter-frame image blocks and compress the sequence numbers of the image blocks, and finally output the super-resolution list, so as to use the super-resolution list to ensure that the When the low-definition image blocks are super-divided into the same definition requirements, the terminal device has a lower calculation burden and occupies a lower bandwidth at the same time.

For example, there are M inter lists and N intra lists. Because measuring video quality requires full decoding of the video for the entire video block, this will introduce a certain delay. If the search is exhaustive, the time complexity introduced is "M*N", but the algorithm reduces its time complexity to O(M+N). Because in the two-dimensional space composed of image blocks, when the area of the inter-frame image block or the intra-frame image block increases, the measured video quality will increase. Using this feature, measure the video quality of the current tile combination. When the current video quality is lower than the quality requirement, increase the image block area in the frame. When the current video quality is greater than the quality requirement, reduce the image block area between frames. Through this search method, the image block combination with the smallest area among the M*N discrete points can be found, and the time required for measuring the video quality can be reduced.

In this embodiment, when the resolution of each image block in the video block reaches the target resolution, the combined area of the image blocks that need to be over-resolved is the smallest, and the ratio of the intra-frame image block to the inter-frame image block at this time is It can improve the calculation efficiency, so according to the inter-frame super-resolution weight, intra-frame super-resolution weight and preset search algorithm, when calculating the definition of each image block in the video block to reach the target definition, the image block with the smallest super-resolution area is required Combined to improve the computing efficiency of the terminal equipment, reduce the amount of calculation, and then reduce the computing power requirements of the terminal equipment for super time-sharing, so that the terminal equipment can stably play the highest-definition video under the current bandwidth exchange.

Further, based on the above-mentioned embodiments of the present application, another embodiment of the present application is provided. In this embodiment, the step of sending the super-resolution list and the video blocks to the terminal device includes:

Step C10, based on the preset run-length encoding, pack the super-resolution list and the video blocks into a run-length data packet;

Step C20, comparing the amount of data between the run-length data packet and the normal data packet packed without run-length encoding;

Step C30, selecting a sending data packet with a small amount of data from the run-length data packet and the normal data packet, and sending the super-resolution list and the video block to the terminal device based on the sending data packet.

In this embodiment, when the super-resolution image block needs to be recorded by using the super-resolution list, the encoding mark of the super-resolution image block can be set in the super-resolution list, or the bit of the super-resolution image block can be set to 1, and the Do not mark the encoding of the image blocks that do not require super-resolution, or set the bits of the image blocks that do not require super-resolution to 0, so that the terminal device can identify the super-resolution image blocks from the super-resolution list.

In this embodiment, since the size of the packet data packaged using run-length encoding is relatively small, run-length encoding is used to package the super-resolution image block and video block to obtain a run-length data packet, and a packing method other than run-length encoding is used Pack the super-resolution image blocks and video blocks to obtain ordinary data packets, then compare the data size of the run-length data packets and ordinary data packets, and select the data packets with small data from the run-length data packets and ordinary data packets as sending data packets, That is, from the run-length data packets and ordinary data packets, the data packets with a small amount of data are screened out, and the super-resolution list and video blocks are sent to the terminal device using the data packets, so as to reduce the transmission of the super-resolution list and video blocks. The bandwidth consumed by blocks provides a better bandwidth environment for video terminals.

Referring to FIG. 3 , FIG. 3 is a schematic diagram of the device structure of the hardware operating environment involved in the solution of the embodiment of the present application.

As shown in FIG. 3 , the device for improving video definition may include: a processor 1001 , such as a CPU, a memory 1005 , and a communication bus 1002 . Wherein, the communication bus 1002 is used to realize connection and communication between the processor 1001 and the memory 1005 . The memory 1005 can be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001 .

Optionally, the device for improving video clarity may also include a rectangular user interface, a network interface, a camera, an RF (Radio Frequency, radio frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. The rectangular user interface may include a display screen (Display), an input sub-module such as a keyboard (Keyboard), and the optional rectangular user interface may also include a standard wired interface and a wireless interface. Optionally, the network interface may include a standard wired interface and a wireless interface (such as a WI-FI interface).

Those skilled in the art can understand that the device mechanism for improving video clarity shown in FIG. components, or different component arrangements.

As shown in FIG. 3 , the memory 1005 as a storage medium may include an operating system, a network communication module, and a program for improving video definition. An operating system is a program that manages and controls device hardware and software resources that improve video clarity, supports programs that improve video clarity, and runs other software and programs. The network communication module is used to realize the communication among various components inside the memory 1005, and communicate with other hardware and software in the system for improving video definition.

In the device for improving video definition shown in FIG. 3 , the processor 1001 is configured to execute the program for improving video definition stored in memory 1005 to implement the steps of any one of the methods for improving video definition above.

The specific implementation manners of the device for improving video clarity in the present application are basically the same as the above-mentioned embodiments of the method for improving video clarity, and will not be repeated here.

The present application also provides a device for improving video clarity, and the device for improving video clarity includes:

A determining module, configured to determine the optimal computing power and required video of the terminal device based on the video request sent by the terminal device, and send the required video determined based on the optimal computing power to the terminal device The super-resolution model and definition level;

A selection module, configured to filter out a super-resolution list from the required video based on the target definition selected by the terminal device from the definition level, wherein the super-resolution list records that the required video requires super-resolution The super-resolution image block of ;

A sending module, configured to send the super-resolution list and the video blocks to the terminal device, so that the terminal device can super-resolution the super-resolution image blocks recorded in the super-resolution list based on the super-resolution model, and obtain The high-definition video corresponding to the desired video.

Optionally, the selection module includes:

An acquisition module, configured to acquire video blocks whose required video definition is the target definition;

A decoding module, configured to decode the video blocks at a low-definition frame rate, and obtain macroblock decoding information of each of the video blocks;

A division module, configured to divide the image block in the video block into an inter image block and an intra image block based on the macroblock decoding information, and determine the inter-frame super-resolution weight and the inter-frame image block of the inter-frame image block. Intra-frame super-resolution weight of the image block in the above-mentioned frame;

The first determining submodule is configured to determine the super-resolution list of the desired video based on the inter-frame super-resolution weight, the intra-frame super-resolution weight and a preset search algorithm.

Optionally, the division module includes:

The second determining submodule is configured to determine an inter-frame macroblock in each of the video blocks based on the macroblock decoding information;

A first definition module, configured to define an image block containing the inter macroblock as an inter image block;

The first calculation module is used to calculate the degree of dependence and the degree of error diffusion between the image blocks between the frames;

The first division sub-module is used to divide the inter-frame image block into multiple inter-frame lists based on the product of the dependence degree and the error diffusion degree, wherein each division of the inter-frame list needs to be recalculated The dependence degree and error diffusion degree between the remaining inter-frame image blocks;

The third determining module is configured to determine the inter-frame super-resolution weight of the inter-frame image blocks in each of the inter-frame lists based on the degree of dependence.

Optionally, the division module also includes:

A fourth determining module, configured to determine an intra macroblock in each video block based on the macroblock decoding information;

A second definition module, configured to define the image block containing the intra macroblock as an intra image block;

The second calculation module is used to calculate the dependent estimate of the intra-frame image block, and modify the dependent estimate based on the preset texture complexity of the intra-frame image block to obtain the corrected dependency;

The second division sub-module is configured to divide the intra-frame image blocks into multiple intra-frame lists in equal proportions based on the sorting of the modified dependencies, and determine the intra-frame super-resolution weight of each of the intra-frame lists.

Optionally, the first determining submodule includes:

The calculation sub-module is used to calculate when the definition of each image block in the video block reaches the target definition based on the inter-frame over-resolution weight, the intra-frame over-resolution weight and the preset search algorithm, it needs The combination of image blocks with the smallest super-resolution area;

A determining unit, configured to define the image blocks in the combination of image blocks as super-resolution image blocks, and determine a super-resolution list based on the super-resolution image blocks.

Optionally, the sending module includes:

A packing module, configured to pack the super-resolution list and the video block into a run-length data packet based on preset run-length encoding;

A comparison module, configured to compare the amount of data between the run-length data packet and an ordinary data packet packed without run-length encoding;

A sending submodule, configured to select a sending data packet with a small amount of data from the run length data packet and the normal data packet, and send the super-resolution list and the video block to the terminal based on the sending data packet equipment.

Optionally, the determination module includes:

A selection unit, configured to select a super-resolution model whose computing power is less than or equal to the optimal computing power from the super-resolution model group corresponding to the desired video;

Wherein, each of the required videos corresponds to a plurality of super-resolution models with different calculation difficulties.

The specific implementation manners of the device for improving video clarity in the present application are basically the same as the embodiments of the above-mentioned method for improving video clarity, and will not be repeated here.

An embodiment of the present application provides a storage medium, and the storage medium stores one or more programs, and one or more programs can also be executed by one or more processors to achieve any of the above-mentioned improved video clarity The steps of the degree method.

The specific implementation manners of the storage medium in the present application are basically the same as the above-mentioned embodiments of the method for improving video clarity, and will not be repeated here.

It should be noted that, in this document, the terms "comprising", "comprising" or any other variation thereof are intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, It also includes other elements that are not expressly listed, or that are inherent to the process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising an etc." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

The serial numbers of the above embodiments of the present invention are for description only, and do not represent the advantages and disadvantages of the embodiments.

Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products are stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk, etc.) ) includes several instructions to make a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the method of each embodiment of the present invention.

The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent mechanism or equivalent process transformation made by using the description of the present invention and the contents of the accompanying drawings, or directly or indirectly used in other related technical fields , are all included in the scope of patent protection of the present invention in the same way.

Claims (10)

1. A method for improving video clarity, characterized in that, the method for improving video clarity includes: Based on the video request sent by the terminal device, determine the optimal computing power of the terminal device and the required video, and send the super-resolution model and the determined video of the required video based on the optimal computing power to the terminal device clarity level; Based on the target definition selected by the terminal device from the definition level, a super-resolution list is selected from the required video, wherein the super-resolution list records the super-resolution image blocks in the required video that need to be super-resolution ; Sending the super-resolution list and the video blocks to the terminal device, so that the terminal device can super-resolution the super-resolution image blocks recorded in the super-resolution list based on the super-resolution model, and obtain the desired video Corresponding high-definition video. 2. The method for improving video definition according to claim 1, characterized in that, based on the target definition selected by the terminal device from the definition level, filter out super Checklist steps include: Acquiring the video blocks whose required video resolution is the target resolution; Decoding the video blocks at a low-definition frame rate to obtain macroblock decoding information for each of the video blocks; Based on the decoding information of the macroblock, divide the image block in the video block into an inter image block and an intra image block, and determine the inter frame super-resolution weight of the inter frame image block and the intra frame image block. The intra-frame super-resolution weight; Based on the inter-frame super-resolution weight, the intra-frame super-resolution weight and a preset search algorithm, determine the super-resolution list of the desired video. 3. The method for improving video clarity according to claim 2, wherein, based on the macroblock decoding information, the image blocks in the video block are divided into inter-frame image blocks and intra-frame image blocks , and the step of determining the inter-frame super-resolution weight of the inter-frame image block includes: determining an inter macroblock within each of said video blocks based on said macroblock decoding information; defining an image block containing the inter macroblock as an inter image block; Calculating the degree of dependence and the degree of error diffusion between the image blocks between the frames; Based on the product of the degree of dependence and the degree of error diffusion, divide the inter-frame image block into multiple inter-frame lists, where each division of the inter-frame list requires recalculating the remaining inter-frame image blocks The degree of dependence and error diffusion between Based on the degree of dependence, the inter-frame super-resolution weights of the inter-frame image blocks in each of the inter-frame lists are determined. 4. The method for improving video clarity according to claim 2, wherein, based on the macroblock decoding information, the image blocks in the video block are divided into inter image blocks and intra image blocks , and the step of determining the intra-frame super-resolution weight of the intra-frame image block includes: determining intra macroblocks within each of the video blocks based on the macroblock decoding information; defining an image block containing the intra macroblock as an intra image block; calculating a dependency estimate of the intra-frame image block, and modifying the dependency estimate based on the preset texture complexity of the intra-frame image block to obtain a modified dependency; Based on the sorting of the modified dependencies, the intra-frame image blocks are divided into multiple intra-frame lists in equal proportions, and the intra-frame super-resolution weight of each of the intra-frame lists is determined. 5. The method for improving video clarity as claimed in claim 2, characterized in that, determining the required The steps of the super-score checklist of the video, including: Based on the inter-frame super-resolution weight, the intra-frame super-resolution weight and a preset search algorithm, calculate the image with the smallest super-resolution area when the definition of each image block in the video block reaches the target definition. block combination; The image blocks in the combination of image blocks are defined as super-resolution image blocks, and a super-resolution list is determined based on the super-resolution image blocks. 6. The method for improving video clarity according to claim 1, wherein the step of sending the super-resolution list and the video blocks to the terminal device includes: Packing the super-resolution list and the video blocks into a run-length packet based on a preset run-length encoding; Comparing the amount of data between the run-length data packet and an ordinary data packet that is not packaged with run-length encoding; Selecting a sending data packet with a small amount of data from the run length data packet and the normal data packet, and sending the super resolution list and the video block to the terminal device based on the sending data packet. 7. The method for improving video clarity as claimed in claim 1, wherein the step of determining the super-resolution model of the desired video based on the optimal computing power comprises: Selecting a super-resolution model whose computing power is less than or equal to the optimal computing power from the super-resolution model group corresponding to the desired video; Wherein, each of the required videos corresponds to a plurality of super-resolution models with different calculation difficulties. 8. A device for improving video clarity, characterized in that the device for improving video clarity comprises: A determining module, configured to determine the optimal computing power and required video of the terminal device based on the video request sent by the terminal device, and send the required video determined based on the optimal computing power to the terminal device The super-resolution model and definition level; A selection module, configured to filter out a super-resolution list from the required video based on the target definition selected by the terminal device from the definition level, wherein the super-resolution list records that the required video requires super-resolution The super-resolution image block of ; A sending module, configured to send the super-resolution list and the video blocks to the terminal device, so that the terminal device can super-resolution the super-resolution image blocks recorded in the super-resolution list based on the super-resolution model, and obtain The high-definition video corresponding to the desired video. 9. A device for improving video clarity, characterized in that, the device for improving video clarity includes: a memory, a processor and a program stored on the memory for realizing the method for improving video clarity, memory for storing a program implementing a method for improving video clarity; The processor is used to execute the program for implementing the method for improving video definition, so as to realize the steps of the method for improving video definition according to any one of claims 1-7. 10. A storage medium, characterized in that, the storage medium is stored with a program for implementing a method for improving video clarity, and the program for implementing the method for improving video clarity is executed by a processor to implement any one of claims 1 to 7. The steps of the method for improving video clarity described in the item.

Images