CN117689541A - Multi-region classification video super-resolution reconstruction method with temporal redundancy optimization - Google Patents

Abstract

The invention relates to a multi-region classification video super-resolution reconstruction method with time redundancy optimization, and belongs to the field of computer vision. The present invention proposes an effective method for dividing blocks into multiple regions to determine whether image blocks can provide effective inter-frame information, thereby determining whether to propagate the information. In order to further improve the reconstruction effect of the network, the upsampling process is also improved. Previous upsampling methods used bilinear interpolation to directly amplify by 4 times. This invention uses two cascaded pixelshuffle (×2) layers and uses a two-stage progressive upsampling strategy to reduce direct bilinear interpolation by 4 times. noise in the upsampling process, thereby further improving the performance of video super-resolution reconstruction.

Description

Multi-region classification video super-resolution reconstruction method with temporal redundancy optimization

Technical field

The invention belongs to the field of computer vision and relates to a multi-region classification video super-resolution reconstruction method with time redundancy optimization.

Background technique

Using deep learning technology to solve the video super-resolution (VSR) problem has become an important and hot topic that has attracted much attention. Many scholars have proposed deep learning-based networks to solve such problems. Current research in the field of video super-resolution reconstruction can be divided into two categories based on how inter-frame information is utilized - whether they are aligned or not: aligned methods and non-aligned methods; they can also be divided into sliding methods based on different frameworks. There are two types of window frames and recursive frames. These methods have been widely used and developed in the field of video super-resolution reconstruction.

The alignment method aligns adjacent frames with the current frame by extracting motion information. Commonly used alignment methods are mainly motion estimation and motion compensation (Motion Estimation and Motion Compensation, MEMC) and deformable convolution. Motion estimation and motion compensation are widely studied for video processing. In video super-resolution, motion estimation and motion compensation are mainly used to represent the temporal correlation between consecutive LR (Low Resolution) frames. Normally, the motion estimation module takes two frames as input and generates an optical flow vector field, using optical flow to represent the motion information between the two frames. The motion compensation module is used to perform image transformation between images based on motion information so that adjacent frames are spatially aligned with the current frame.

The Enhanced Deformable Video Restoration (EDVR) network proposed by Xintao Wang et al. is a good example of applying deformable convolution. Optical flow becomes inaccurate when the video to be restored contains occlusions, large motion, and severe blur. Therefore, it proposes two key modules: Pyramid, Cascade and Deformable Alignment Module (PCD) and Spatiotemporal Attention Fusion Module (TSA), which are used to solve large motions in videos and effectively fuse multiple frames respectively. The temporal deformable alignment network (TDAN) proposed by Yapeng Tian et al. applies deformable convolution to the target frame and adjacent frames to obtain the corresponding offset, and then distorts the adjacent frames according to the offset to make the adjacent frames Align to target frame.

Non-aligned methods do not need to align adjacent frames. This type of method mainly uses spatial or spatiotemporal information for feature extraction. Non-aligned methods are mainly divided into four types: 2D convolution methods (2D Conv), 3D convolution methods, recurrent convolutional neural networks and methods based on non-local networks. The temporal attention mechanism (TGA) proposed by Takashi Isobe et al. effectively fuses spatiotemporal information in a hierarchical manner through frame rate groups, using intra-group fusion and inter-group attention mechanisms of 2D dense blocks and 3D units to generate the final High resolution images.

The sliding window framework means that the network inputs a fixed number of frames each time and can only process a few frames at a time. Each frame in the video is restored by using frames within a short time window. Early methods predicted optical flow between low-resolution frames and performed spatial warping for alignment. Later methods took a more sophisticated approach to implicit alignment. For example, TDAN uses deformable convolutions (DCNs) to align frames in different layers. EDVR further uses DCN in a multi-scale manner to achieve more precise alignment. The recursive framework attempts to exploit long-term dependencies by propagating latent features, which can propagate effective information from previous frames to subsequent frames, which is more conducive to the recovery of subsequent video frames. For example, RSDN adopts one-way propagation with recursive detail building blocks and hidden state adaptation modules to enhance robustness to appearance changes and error accumulation. KelvinC.K.Chan et al. proposed Basic VSR, which proved the importance of two-way propagation relative to one-way propagation to better utilize temporal characteristics. BasicVSR++ proposed by Kelvin C.K.Chan and others further improves Basic VSR and proposes deformable alignment of second-order grid propagation and optical flow guidance, achieving a significant increase in network performance with approximately the same amount of parameters.

Contents of the invention

In view of this, the object of the present invention is to provide a multi-region classification video super-resolution reconstruction method with temporal redundancy optimization.

In order to achieve the above objects, the present invention provides the following technical solutions:

Temporal redundancy optimized multi-region classification video super-resolution reconstruction method, the method is:

In the TROMCN network structure, given a low-quality video frame sequence T, H, W, and C respectively represent the length, height, width and number of channels of the video. The goal of video super-resolution is to reconstruct a high-quality video frame sequence. where s represents the scale factor; before feature extraction, the input video frame is divided into 64×64 blocks/> Overlap 8 pixels to take blocks; in the feature extraction module, use the residual Swin Transformer module to extract the feature F ^SF :

F ^SF = H _RSTB (X ^LQ )

Among them, H _RSTB represents the feature extraction module, and then the feature extraction module is obtained through feature alignment and fusion:

F ^MSF = H _MRC (F ^SF )

Among them, H _MRC represents the feature alignment and fusion module. This module uses a second-order grid mode for feature propagation, uses a flow-based deformable alignment method to align features, and then passes the learned features through a cascaded upsampling module. ,get:

F ^UP =H _UP (F ^MSF )

Among them, H _UP represents the upsampling operation of the upsampling module. The upsampling method used is to cascade two × 2 sub-pixel convolution layers (pixelshuffle). F ^UP is the feature obtained after upsampling; then F _UP is input to the reconstruction layer to generate the final super-resolution video frame:

Y ^HQ = _HR (F ^UP ) =H _TROMCN (X ^LQ )

Among them, _HR and H _TROMCN respectively represent the reconstruction layer operation and the TROMCN network proposed by this invention;

For training, use the Charbonnier loss function For optimization, Y ^HQ represents the reconstructed image, Y ^aT represents the real high-resolution image, and ∈ represents a constant.

Optionally, in the feature extraction module, for a video frame x _t , the video frame is divided into blocks of size 64×64 before feature extraction. Overlap 8 pixels, and obtain features after extracting features through the RSTB module; use the residual Swin Transformer module to extract features, capture long-distance dependencies, aggregate high-frequency information, and extract contextually relevant feature representations, using the block-based Image/> As input, after passing through an ordinary convolution layer, L Swin Transformer modules with residual structure and a convolution block are used to extract features to obtain the extracted features. A single STL module represents Swin Transformer Layer, and the formula for the feature extraction stage is:

in, Represents the features extracted by the shallow feature extraction module, HRSTB (·) represents the residual SwinTransformer module, and H _CONV (·) represents an ordinary convolution layer;/> Represents the divided image.

Optionally, in the feature alignment and fusion module, the feature propagation module adopts dynamic propagation, in which each block of the current frame receives information from different frames, using and/> branch to recover valid information from blocks of different frames;/> and/> Represent the hidden states of N overlapping blocks and corresponding blocks respectively;

The average value of optical flow is used to represent the motion state, and its formula is:

where flow(·) represents optical flow estimation, |·| represents absolute value, mean represents average calculation, Indicates reference block/> and corresponding adjacent blocks/> motion state between; by setting the threshold γ and comparing it with the calculated optical flow to determine whether the adjacent blocks of the current block can provide effective inter-frame information; the comparison formula is as follows:

When the calculated optical flow is greater than the threshold, the adjacent blocks are considered to provide useful information to help restore the current block, and their information is adopted and propagated to the current block; conversely, when the calculated optical flow is less than the threshold, the two blocks are considered The movement between pixels is small, there are many overlapping pixels, there is time redundancy, and blur and noise are prone to occur. The information of this frame will be discarded to avoid the disappearance of useful information; set two thresholds γ ₁ = 0.2, γ ₂ =0.3, corresponding to first-order and second-order propagation respectively; change the updated and/> Propagated to the next frame; after feature propagation, the features of N blocks are obtained, an RSTB module is concatenated and the N blocks are resynthesized into a complete feature map h _t .

Optionally, the upsampling is divided into three categories: upsampling based on linear interpolation, upsampling based on deep learning, and Unpooling method.

The beneficial effect of the present invention is that the network proposed by the present invention has more powerful feature extraction capabilities, feature alignment capabilities and the ability to reduce time redundancy, and can reconstruct higher-quality high-resolution images.

Other advantages, objects, and features of the present invention will, to the extent that they are set forth in the description that follows, and to the extent that they will become apparent to those skilled in the art upon examination of the following, or may be derived from This invention is taught by practicing it. The objects and other advantages of the invention may be realized and obtained by the following description.

Description of the drawings

In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be described in detail below in conjunction with the accompanying drawings, in which:

Figure 1 shows the overall network structure of TROMCN;

Figure 2 shows the network structure of RSTB;

Figure 3 shows the network structure of STL;

Figure 4 shows the feature forward propagation structure;

Figure 5 shows cascade upsampling;

Figure 6 is the visual result of DTVIT 046 magnified 4 times;

Figure 7 is the visual result of DTVIT 072 magnified 4 times;

Figure 8 shows the visual results of REDS,000 magnified 4 times;

Figure 9 shows the visual result of REDS,015 magnified 4 times;

Figure 10 shows the visual result of Vid4,city magnified 4 times;

Figure 11 shows the visual result of Vid4, walk magnified 4 times;

Figure 12 shows the visual results of UDM10 and archpeople magnified 4 times;

Figure 13 shows the visual result of UDM10, photography magnified 4 times;

Figure 14 is the visual result of Vimeo, Sequence 00001, Clip 0837 magnified 4 times;

Figure 15 is the visual result of Vimeo, Sequence 00010, Clip 0573 magnified 4 times.

Detailed ways

The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner. The following embodiments and the features in the embodiments can be combined with each other as long as there is no conflict.

The drawings are only for illustrative purposes, and represent only schematic diagrams rather than actual drawings, which cannot be understood as limitations of the present invention. In order to better illustrate the embodiments of the present invention, some components of the drawings will be omitted. The enlargement or reduction does not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

In the drawings of the embodiments of the present invention, the same or similar numbers correspond to the same or similar components; in the description of the present invention, it should be understood that if there are terms "upper", "lower", "left" and "right" The orientation or positional relationship indicated by "front", "rear", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must be It has a specific orientation and is constructed and operated in a specific orientation. Therefore, the terms describing the positional relationships in the drawings are only for illustrative purposes and cannot be understood as limitations of the present invention. For those of ordinary skill in the art, they can determine the specific position according to the specific orientation. Understand the specific meaning of the above terms.

Existing video super-resolution methods focus on the enhancement of feature propagation and temporal and spatial alignment, while ignoring the impact of temporal redundancy on the reconstruction effect. The difference between video super-resolution and image super-resolution is that in video super-resolution, continuous video frames are input into the neural network for learning, and one or more super-resolution images are obtained. There is a large amount of temporal redundancy between multiple consecutive video frames. If the features extracted from all pixels are propagated as effective features indiscriminately, noise will easily be generated, thus causing interference to the reconstruction. Because different components in the image have different frequency characteristics, high-resolution reconstruction is required to distinguish different components in the image. To sum up, in view of the shortcomings of the existing methods, the present invention proposes a temporal redundancy optimized multi-region classification video super-resolution reconstruction method (TemporaryRedundancy Optimization and Multi-Region Classification Network, TROMCN). Perform block processing, divide the blocks into static areas and dynamic areas by judging the optical flow between blocks, and perform different processing to obtain more accurate reconstruction results.

The TROMCN network proposed by this invention first uses RSTB (Residual Swin Transformer Block) instead of traditional residual blocks (Residual Blocks) to extract and reconstruct features. Compared with the residual block based on CNN (Convolutional Neural Network), the residual Swin Transformer module can capture long-distance dependencies and aggregate relevant high-frequency information. Secondly, in order to solve the noise problem caused by the indiscriminate use of features in the image, the present invention proposes an effective method of dividing blocks into multiple regions to determine whether the image blocks can provide effective inter-frame information, thereby determining whether to use the information. spread. In order to further improve the reconstruction effect of the network, the upsampling process is also improved. Previous upsampling methods used bilinear interpolation to directly amplify by 4 times. This invention uses two cascaded pixelshuffle (×2) layers and uses a two-stage progressive upsampling strategy to reduce direct bilinear interpolation by 4 times. noise in the upsampling process, thereby further improving the performance of video super-resolution reconstruction.

The TROMCN network structure proposed by this invention consists of four parts, which are feature extraction module, feature alignment and fusion module, upsampling module and reconstruction module. The entire model structure is shown in Figure 1.

1. Overall process

In the TROMCN network structure, given a low-quality video frame sequence T, H, W, and C respectively represent the length, height, width and number of channels of the video. The goal of video super-resolution is to reconstruct a high-quality video frame sequence. where s represents the scaling factor. Before feature extraction, the input video frame is divided into 64×64 blocks/> (Overlap 8 pixel blocks). In the feature extraction module, the residual Swin Transformer module is used to extract the feature F ^SF ,

F ^SF = H _RSTB (X ^LQ )

Among them, H _RSTB represents the feature extraction module, and then the feature extraction module is obtained through feature alignment and fusion:

F ^MSF = H _MRC (F ^SF )

Among them, H _MRC represents the feature alignment and fusion module. This module uses a second-order grid mode for feature propagation, uses a flow-based deformable alignment method to align features, and then passes the learned features through a cascaded upsampling module. ,get:

F ^UP =H _UP (F ^MAF )

Among them, H _UP represents the upsampling operation of the upsampling module. The upsampling method used here is to cascade two × 2 sub-pixel convolution layers (pixelshuffle), and F ^UP is the feature obtained after upsampling. Then the F _UP is input into the reconstruction layer to generate the final super-resolution video frame:

Y ^HQ = _HR (F ^UP ) =H _TROMCN (X ^LQ )

Among them, _HR and H _{TROMCN respectively} represent the reconstruction layer operation and the TROMCN network proposed by this invention.

For training, the network of the present invention uses the Charbonnier loss function For optimization, Y ^HQ represents the reconstructed image, Y ^GT represents the real high-resolution image, and ∈ represents a small constant.

2. Feature extraction module

Taking a video frame x _t as an example, before feature extraction, the video frame is divided into blocks of size 64×64 (overlapping 8 pixels), the features are obtained after feature extraction by the RSTB module. Existing technical solutions usually use convolutional neural networks, recurrent neural networks, etc. when extracting image features, and both of these networks will lead to performance degradation due to long-term dependency issues. The present invention uses the residual Swin Transformer module to extract features, capture long-distance dependencies, aggregate relevant high-frequency information, and can extract feature representations with contextual relevance, resulting in higher training efficiency. The residual Swin Transformer module structure is shown in Figure 2. This module takes the chunked image/> As input, after passing through an ordinary convolution layer, L Swin Transformer modules with residual structure and a convolution block are used to extract features to obtain the extracted features/> Such a residual design provides equivalent connections from different modules to the reconstruction module, promoting the aggregation of features at different levels. The single STL module in Figure 2 represents the Swin Transformer Layer, and its structure is shown in Figure 3. The formula for this feature extraction stage is:

in, Represents the features extracted by the shallow feature extraction module, H _RSTB (·) represents the residual SwinTransformer module, and H _CONV (·) represents an ordinary convolution layer. /> Represents the divided image.

3. Feature alignment and fusion module

In the network proposed by the present invention, second-order grid propagation is used to enhance features, and spatio-temporal information can be used to more effectively span unaligned video frames. By inputting the features extracted in the previous step into the flow-based deformable alignment module, the problem of offset overflow is overcome while increasing the offset diversity. The feature propagation module proposed by this invention adopts dynamic propagation, in which each block of the current frame can receive information from different frames. In order to achieve this, the feature propagation module containing and/> branch to recover valid information from blocks of different frames. /> and/> represent the hidden states of N overlapping blocks and corresponding blocks respectively. Figure 4 shows the feature forward propagation structure.

This invention proposes a judgment module for detecting temporal redundancy to judge whether image blocks can provide effective inter-frame information, thereby judging whether to propagate this information. Since optical flow is a measure that describes the motion information of an object, the average value of optical flow is used to represent the motion state, and its formula is:

where flow(·) represents optical flow estimation, |·| represents absolute value, mean represents average calculation, Indicates reference block/> and corresponding adjacent blocks/> state of motion. For the calculation of optical flow, the traditional DIS algorithm is used because it only slightly increases the computational cost. By setting a threshold γ and comparing it with the calculated optical flow, it is judged whether the adjacent blocks of the current block can provide effective inter-frame information. The comparison formula is as follows:

When the calculated optical flow is greater than the threshold, it is considered that the adjacent block can provide useful information to help restore the current block, so its information is adopted and propagated to the current block. On the contrary, when the calculated optical flow is less than the threshold, it is considered that the pixel-by-pixel movement between the two blocks is small, there are many overlapping pixels, there is temporal redundancy, and blur and noise are prone to occur, and the information of the frame will be discarded. to avoid the loss of useful information. Since the present invention uses second-order grid propagation, two thresholds γ ₁ =0.2 and γ ₂ =0.3 are set, corresponding to first-order and second-order propagation respectively. Afterwards, the updated and/> propagated to the next frame. Finally, after feature propagation, the features of N blocks are obtained, an RSTB module is concatenated and the N blocks are resynthesized into a complete feature map h _t . After many experiments, it has been shown that this method can improve the performance of the original model and solve the generalization ability well.

4. Cascade sub-pixel convolutional layer upsampling

Upsampling technology is a necessary step for image super-resolution. Judging from previous video super-resolution methods, upsampling is roughly divided into three categories: upsampling based on linear interpolation, upsampling based on deep learning, and Unpooling methods. Previous video super-resolution methods often use bilinear interpolation to directly enlarge the image by 4 times, but it does not consider the interaction between the gray values of adjacent points, which can easily lead to the loss of high-frequency components of the image. The edges become somewhat blurry. Later, many video super-resolution methods used sub-pixel convolution for upsampling operations. This method no longer needs to expand the size of the input through linear interpolation at the beginning, so that smaller convolution kernels can be used to achieve good results. , using convolutional learning can learn a better fitting method than manual design. Improvements have been made on the ordinary method of directly using sub-pixel convolution. A cascade method is used to combine two sub-pixel convolution layers together. The progressive up-sampling strategy at both ends can well reduce the up-sampling process. noise in.

5. Experimental settings

This invention uses the REDS data set and the Vimeo-90K data set to train the network respectively. There are 270 video sequences available in the REDS dataset, each sequence contains one hundred frames. Following the conventional splitting method, the data was split into a training set (266 sequences) and a test set (4 sequences). Vimeo-90K contains 64612 and 7824 video sequences for training and testing respectively. Although both datasets are widely used benchmarks, the two data have different motion conditions. Motion in the Vimeo-90K dataset is generally small, with 99% of pixels having a motion magnitude smaller than 10 (for each clip, motion was measured at frames 4 and 7). The REDS data set has large-scale motion, and at least 20% of the pixels have a motion amplitude greater than 10 (for each segment, the motion of frames 3 and 5 is measured). In addition, the DTVIT data set is used, which includes live broadcasts, TV programs, sports live broadcasts, movies, surveillance cameras, and advertisements.

In order to prove the superiority of the method proposed in this invention, the network proposed in this invention was compared with the existing representative cutting-edge super-resolution reconstruction methods. The comparative documents are shown in Table 1.

Table 1

The objective evaluation results are shown in Table 2. The methods with the best results are bolded. It can be seen from the table that in most cases, the PSNR and SSIM of the present invention are the highest, and the reconstruction effect is significantly better than some existing methods. A very representative cutting-edge video super-resolution reconstruction method. Figures 6 to 15 are subjective visual effect diagrams of the present invention. It can be seen that Figures 6-7 produce sharper and clearer high-resolution video frames, while other methods cannot recover fine textures and details. Figures 8-15 visually present sharper edges and fine details, while other methods produce more blur and lose more details. In summary, whether it is from objective evaluation indicators or subjective visual effect diagrams, it can be proved that the method proposed by the present invention has strong advantages in optimizing time redundancy, feature enhancement, etc.

Table 2

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified. Modifications or equivalent substitutions without departing from the purpose and scope of the technical solution shall be included in the scope of the claims of the present invention.

Claims (4)

1. The multi-region classified video super-resolution reconstruction method for time redundancy optimization is characterized by comprising the following steps of: the method comprises the following steps:

in a TROMCN network structure, a sequence of low quality video frames is givenT, H, W, C denote the length, height, width and channel number, respectively, of a video, the goal of which is to reconstruct a sequence of high quality video framesWherein s represents a scale factor; before feature extraction, the input video frame is divided into 64×64 blocks +.>Overlapping 8 pixels to obtain blocks; in the feature extraction module, the residual Swin transducer module is used to extract feature F ^SF ：

F ^SF ＝H _RSTB (X ^LQ )

Wherein H is _RSTB Representing a feature extraction module, and then obtaining the feature through feature alignment and fusion by the feature extraction module:

F ^MSF ＝H _MRC (F ^SF )

wherein H is _MRC The characteristic alignment and fusion module is used for carrying out characteristic propagation in a mode of a second-order grid, carrying out characteristic alignment in a deformable alignment mode based on flow, and then carrying out cascade upsampling on the learned characteristics to obtain the following components:

F ^UP ＝H _UP (F ^MSF )

wherein H is _UP Representing the upsampling operation of the upsampling module, the upsampling method used is to concatenate two x 2 sub-pixel convolution layers (pixelshuffles), F ^UP Is a feature obtained after upsampling; then F is again carried out _UP Inputting the video frame into a reconstruction layer to generate a final super-resolution video frame:

Y ^HQ ＝H _R (F ^UP )＝H _TROMCN (X ^LQ )

wherein H is _R And H _TROMCN Respectively representing the operation of a reconstruction layer and the TROMCN network proposed by the invention;

for training, a Charbonnier loss function is usedOptimizing Y ^HQ Representing reconstructed image, Y ^GT Representing a true high resolution image, e represents a constant.

2. The method for multi-region classified video super-resolution reconstruction with temporal redundancy optimization according to claim 1, wherein: in the feature extraction module, for a video frame x _t Prior to feature extraction, the video frame is divided into blocks of size 64 x 64Overlapping 8 pixels, and extracting features by an RSTB module to obtain features; extracting features by adopting a residual Swin transducer module, capturing long-distance dependence, aggregating high-frequency information, extracting a feature representation with contextual relevance, and obtaining a segmented image +.>For input, after passing through a common convolution layer, the characteristics are extracted by L Swin transform modules with residual structures and a convolution block, so as to obtain the extracted characteristics->The single STL module represents Swin Transformer Layer and the formula for the feature extraction stage is:

wherein,representing the features extracted by the shallow feature extraction module, H _RSTB (. Cndot.) Swin transducer module, H, representing residual error _CONV (-) represents a common convolution layer; />Representing the segmented image.

3. The method for multi-region classified video super-resolution reconstruction with temporal redundancy optimization according to claim 2, wherein: in the feature alignment and fusion module, the feature propagation module adopts dynamic propagation, wherein each block of the current frame receives information from different frames, and adopts a method comprising the following steps ofAnd->To recover valid information from blocks of different frames; />And->Respectively representing hidden states of N overlapped blocks and corresponding blocks;

the average value of the optical flow is used to represent the motion state, and the formula is:

where flow (·) represents optical flow estimation, |·| represents absolute value, mean represents mean calculation,representing reference block->And corresponding adjacent block->A state of motion therebetween; judging whether the neighboring block of the current block can provide effective inter-frame information by setting a threshold gamma and comparing it with the calculated optical flow; the comparison formula is as follows:

when the calculated optical flow is greater than the threshold value, the adjacent block is considered to provide useful information to help restore the current block, and the information is adopted and transmitted to the current block; conversely, when the calculated optical flow is smaller than the threshold value, the movement amplitude between the two blocks pixel by pixel points is considered to be small, the number of overlapped pixels is large, time redundancy exists, blurring and noise points are easy to occur, and the information of the frame is discarded to avoid the disappearance of useful information; setting two threshold values gamma ₁ ＝0.2，γ ₂ =0.3, corresponding to first and second order propagation, respectively; will be updatedAnd->Propagating to the next frame; through feature propagation, N blocks of features are obtained, one RST is connected in seriesB module and resynthesize N blocks into a complete feature map h _t 。

4. The method for multi-region classified video super-resolution reconstruction with temporal redundancy optimization according to claim 3, wherein: the upsampling is divided into three categories: upsampling based on linear interpolation, upsampling based on deep learning, and the Unpooling method.