CN116664875A - PVT-based saliency target detection method for gating network - Google Patents

Abstract

The invention discloses a saliency object detection method based on a PVT-based gated network, which extracts global features through layer-by-layer fusion of the PVT network; adds a transition layer after four encoders of different scales in PVT, and integrates the transition layer and decoder Multi-level gate units are inserted between layers, and a pyramid pooling module is introduced on the top layer of the encoder to extract advanced semantic information; the advanced semantic information is propagated from top to bottom; the feature aggregation decoder continuously fuses advanced semantic information through element-level addition, and the encoder is effective. information and decoder features at different scales. The invention can extract the content of the target most interesting to human eyes in any given image, pay more attention to the salient area, and suppress the background interference.

Description

Salient Object Detection Method Based on PVT Gating Network

technical field

The invention relates to a salient target detection method, in particular to a salient target detection method based on a PVT gating network, and belongs to the field of computer vision.

Background technique

In recent years, with the rapid development of the new media industry, some short videos, image tweets and other digital information have been integrated into our lives. Every day, tens of thousands of images and video information are transmitted to the Internet, which greatly enriches people's life experience. Live, work and play. The number of digital images uploaded on the WeChat software is as high as 1 billion per day. It can be seen that, but in order to quickly select valuable content from a large number of digital images, it will be difficult to achieve by human sense alone, which poses a greater challenge to the research of computer vision.

Digital image information can very intuitively reflect the content that people want to express, and is widely used in the transmission of network resources. However, a large amount of data information is difficult to be effectively recognized by people. Even if computer resources are used for assistance, it will generate a heavy burden. Therefore, we need to extract important content from digital image information, so as to use limited computing resources to integrate and reduce people's burden of collecting useful information. On the other hand, in the human visual system, they can focus on the important areas of the image, such as the face of the self-portrait of the WeChat circle of friends, the ingredients of the food photo, etc., without paying attention to the useless information in the background area. Fast image processing plays an important role. Therefore, how to apply this ability of focusing on key areas of images to computers to help people complete more complex image processing tasks is the focus of current computer vision researchers.

In recent years, the U-shaped structure has received great attention due to its ability to construct rich feature maps by constructing multi-level top-down paths and achieve good performance. Now, many salient object detection networks adopt U-shaped multi-scale layered encoder-decoder structure as the basic structure of the network. However, the above methods directly use the cross-layer connection structure to directly connect the features of the encoder to the decoder, and there is a lack of interference control between them. This approach will introduce misleading contextual information from the encoder into the decoder, resulting in underutilization of truly useful features. In addition, the existing methods use fully convolutional neural network (FCN) based on deep learning, most of which use pre-trained image classification models such as VGG and ResNet as encoders, focusing on designing effective neural networks by aggregating multi-level features. the decoder. However, CNN's model structure is characterized by modeling local information aggregation, and it is difficult to model long periods. Accurate and complete extraction of salient objects from complex scenes is still a great challenge. The Pyramid Vision Transformer (PVT) opens up a new way to fully extract salient objects.

Contents of the invention

The object of the present invention is to provide a salient object detection method based on PVT gating network.

In order to solve the problems of the technologies described above, the technical solution of the present invention is: a salient object detection method based on a PVT-based gating network, comprising the following steps:

Step 1: Image preprocessing: Resize the input image to a tensor X of preset size;

Step 2: Establish a PVT gating network: the PVT gating network includes the first to fourth feature processing units and the pyramid pooling module PPM; the first to third feature processing units have the same structure; the first feature processing unit includes a feature encoder PVTE _1. Transition layer T ₁ , gate unit G ₁ and decoder feature unit FAD ₁ ; the fourth feature processing unit includes feature encoder PVTE ₄ , transition layer T ₄ , gate unit G ₄ and decoder feature unit FAD ₄ ; tensor After X is sequentially processed by the feature encoder PVTE ₁ -PVTE ₄ , the first to fourth feature tensors are obtained;

The fourth feature tensor is processed by the pyramid pooling module PPM to obtain the fourth high-level semantic information; the fourth feature tensor is processed by the transition layer T ₄ and then spliced with the fourth feature tensor and input to the gate unit G ₄ , and then combined with the transition layer The output of T ₄ is multiplied to obtain the fourth encoder effective information, and the fourth encoder effective information is added to the fourth high-level semantic information and input to the decoder feature device FAD ₄ ; the decoder feature device FAD ₄ outputs the fourth decoding feature vector;

The first feature tensor is concatenated with the second decoded feature vector output by the second feature processing unit FAD ₂ and input to the gate unit G ₁ , the first feature tensor is processed by the transition layer T ₁ and multiplied by the output of the gate unit G ₁ to obtain The first encoder effective information, the first encoder effective information, the fourth high-level semantic information and the second decoding feature vector are added and input to the decoder feature device FAD ₁ ; the first decoding feature vector is output as a saliency map;

Step 3: Detect saliency map: input tensor, get saliency map after processing by PVT gating network.

Further, the size of the tensor is 384×384×3, and the sizes of the first to fourth feature tensors are 96×96×64, 48×48×128, 24×24×320, and 12×12×512, respectively.

Further, the feature encoder PVTE _i and the decoder feature FAD _i+1 are integrated, and then the gate value is calculated through convolution, activation and pooling operations:

In the formula, Cat(·) is the splicing operation between the channel axes, Conv(·) is the convolution operation, S(·) is the element-level sigmoid function, and P(·) is the global average pooling;

The gate value is applied to weight the transition layer features T ₁ -T ₄ , and the transition layer T ₁ -T ₄ is generated by reducing the dimensionality of the feature encoders PVTE ₁ -PVTE ₄ by using the generated 3×3 convolution.

Further, the pyramid pooling module PPM performs adaptive average pooling operations of 4 different scales on the fourth feature tensor to obtain feature maps of 4 different sizes, and the sizes are 1×1, 2×2, 3×3 , 6×6; use 1×1 convolution to reduce the corresponding level channel to 1/4 of the original; obtain the size before unpooling through bilinear interpolation, and then concatenate with the fourth feature tensor in the channel dimension.

Further, the output of the decoder featureizer is:

where the feature aggregation decoder FAD ₁ is a saliency map of the same size as the input image;

The features of the feature encoder PVTE _i are input into the transition layer T _i module; the transition layer T _i reduces the number of channels and feeds it to the gate unit G _i ; the output of the pyramid pooling module PPM is subjected to dimension reduction operation and bilinear interpolation method is used for uplink Sampling to make it the same as the output dimension and scale of T _i ; the effective information of the encoder, the PPM output of the pyramid pooling module and the output of the feature aggregation decoder FAD _i+1 are fused and then passed to the feature aggregation decoder FAD _i .

Furthermore, the feature aggregation decoder FAD first pools the input feature map with three downsampling averages of {2, 4, 8}; then uses bilinear interpolation to upsample to obtain the output feature map of the original size.

Adopt above-mentioned technical scheme, the present invention obtains following technical effect:

By introducing a pyramid visual transformer, the present invention can powerfully model the global dependency and obtain more powerful and robust features; by introducing a gating module between the encoder and the decoder to perform information screening, the more Effective context information is passed into the decoder to make it pay more attention to salient areas and suppress background interference; by introducing a pyramid pooling module PPM on the top layer of the encoder to expand the receptive field to collect advanced semantic information; through top-down The progressive path propagates the high-level semantic information to the pyramid features at all levels to make up for the defect that the top-down signal of the U-shaped network is gradually diluted. And by introducing feature aggregation decoder FAD after continuous fusion of various features in element-level addition, it can better balance and fuse high-level semantic information, effective information of encoder and decoder features of different scales in the top-down path.

Description of drawings

Figure 1 is a block diagram of the present invention.

Fig. 2 is a structural diagram of the door unit module of the present invention.

Fig. 3 is a structural diagram of the feature aggregation decoder FAD of the present invention.

Fig. 4 is an input image of Embodiment 1 of the present invention.

Fig. 5 is a significance diagram detected in Example 1 of the present invention.

Detailed ways

The following examples serve to illustrate the invention.

Example 1

With reference to Fig. 1, a kind of salient object detection method based on PVT gating network, comprises the following steps:

Step 1: Image preprocessing: adjust the size of the input image to a tensor X of a preset size, and the tensor size in this embodiment is 384×384×3;

Step 2: Establish a PVT gating network: the PVT gating network includes the first to fourth feature processing units and the pyramid pooling module PPM; the first to third feature processing units have the same structure; the first feature processing unit includes a feature encoder PVTE _1. Transition layer T ₁ , gate unit G ₁ and decoder feature unit FAD ₁ ; the fourth feature processing unit includes feature encoder PVTE ₄ , transition layer T ₄ , gate unit G ₄ and decoder feature unit FAD ₄ ; tensor After X is sequentially processed by the feature encoder PVTE ₁ -PVTE ₄ , the first to fourth feature tensors are obtained;

The fourth feature tensor is processed by the pyramid pooling module PPM to obtain the fourth high-level semantic information; the fourth feature tensor is processed by the transition layer T ₄ and then spliced with the fourth feature tensor and input to the gate unit G ₄ , and then combined with the transition layer The output of T ₄ is multiplied to obtain the fourth encoder effective information, and the fourth encoder effective information is added to the fourth high-level semantic information and input to the decoder feature device FAD ₄ ; the decoder feature device FAD ₄ outputs the fourth decoding feature vector;

The first feature tensor is concatenated with the second decoded feature vector output by the second feature processing unit FAD ₂ and input to the gate unit G ₁ , the first feature tensor is processed by the transition layer T ₁ and multiplied by the output of the gate unit G ₁ to obtain The first encoder effective information, the first encoder effective information, the fourth high-level semantic information and the second decoding feature vector are added and input to the decoder feature device FAD ₁ ; the first decoding feature vector is output as a saliency map;

Step 3: Detect saliency map: input tensor, get saliency map after processing by PVT gating network.

The dimensions of the first to fourth feature tensors in this embodiment are 96×96×64, 48×48×128, 24×24×320, 12×12×512;

The feature encoder PVTE _i and the decoder feature FAD _i+1 are integrated, and then the gate value is calculated through convolution, activation and pooling operations:

In the formula, Cat(·) is the splicing operation between the channel axes, Conv(·) is the convolution operation, S(·) is the element-level sigmoid function, and P(·) is the global average pooling;

The gate value is applied to weight the transition layer features T ₁ -T ₄ , and the transition layer T ₁ -T ₄ is generated by reducing the dimensionality of the feature encoders PVTE ₁ -PVTE ₄ by using the generated 3×3 convolution. With multi-level gating units, we can suppress and balance the information flowing from different encoder blocks to the decoder. The multi-level gating unit can significantly suppress the perturbation of each encoder block and enhance the contrast between salient and non-salient regions.

The pyramid pooling module PPM is introduced at the top layer of the encoder, and the advanced semantic features learned in the feature encoder PVTE ₄ are input into the pyramid pooling module PPM, and multi-scale pooling features are obtained through different pooling operations to further expand the receptive field and collect the global Contextual information to more accurately capture the exact location of salient objects;

The pyramid pooling module PPM performs adaptive average pooling operations of 4 different scales on the fourth feature tensor to obtain feature maps of 4 different sizes, and the sizes are 1×1, 2×2, 3×3, 6 ×6; use 1×1 convolution to reduce the corresponding level channel to 1/4 of the original; obtain the size before unpooling through bilinear interpolation, and then splicing in the channel dimension to contain the fourth feature tensor, and finally Output a composite feature map that combines multiple scales, so as to achieve the purpose of taking into account the global semantic information and local detail information; continuously integrate the different levels of features of the transition layer T ₁ -T ₄ through element-level addition;

The encoded features obtained by the feature encoder PVTE _i are input into the transition layer T _i module; in order to reduce the number of parameters, the transition layer T _i will reduce the number of channels and feed it to the gate unit G _i ; the PPM features are subjected to dimensionality reduction operations and double-line Upsampling by the linear interpolation method to make it the same dimension and scale as T _i ; after the effective information of the encoder, the output of the pyramid pooling module PPM and the output of the pyramid pooling module FAD _i+1 are fused through the element addition convolution layer Passed into the feature aggregation decoder FAD _i ; the output process of each layer of decoder can be expressed as follows:

Among them, the feature aggregation decoder FAD ₁ is a single-channel feature map with the same size as the input image;

In order to prevent high-level semantic information from being diluted in the top-down path, we directly aggregate the high-level features provided by the pyramid pooling module PPM into the feature map of each feature layer to provide multi-scale information for decoders at all levels;

After the element-level addition continuously fuses various features, the feature aggregation decoder FAD is introduced to encode effective features, high-level features and features at all levels in the top-down path, and convert the fused feature maps into multiple feature spaces. , to obtain local context information of different scales, and then combine these information to ensure that feature maps of different scales can be seamlessly integrated;

The processing of the feature aggregation decoder FAD:

The feature map fused by element-level addition is taken as input, and the fused feature map is converted into multiple feature spaces using pooling techniques. The operation taken is to first use the input feature map with three downsampling average pooling of {2,4,8} to obtain three feature maps of different sizes;

Upsample the feature maps of three different sizes back to the original size to get the same size;

The four pixel branches including the input of the feature aggregation decoder FAD are summed to obtain the integrated feature map;

Send it to the first 3×3 convolutional layer, the scale dimension does not change;

Then reduce the dimension through the second 3×3 convolutional layer, so that the output of the feature aggregation decoder FAD _i i=2,3,4 is the same as the output dimension of the low-order feature T _i-1 of the transition layer, and the feature aggregation decoder FAD _i i=1 dimension is directly reduced to 1.

Finally, the feature aggregation decoder FAD _i i=2, 3, 4 performs upsampling to match the scale of the transition layer T _i−1 using the bilinear interpolation method. FAD _i i=1 performs 4 times upsampling, and the scale becomes 384×384.

Example 2

The difference from Embodiment 1 is: the processing procedure of the feature aggregation decoder FAD:

The feature map fused by element-level addition is taken as input, and the fused feature map is converted into multiple feature spaces using pooling techniques. The operation to be taken is to first pool the input feature map with three downsampling averages of {4, 16, 64} to obtain feature maps of three different sizes; upsample the feature maps of three different sizes back to the original size, and obtain the same size of.

Example 3

The difference from Embodiments 1 and 2 is that the feature aggregation decoder FAD first averages the input feature map with three types of downsampling {2, 4, 8}; then uses the super-resolution method to up-sample to obtain the output of the original size feature map. The edge of the feature map obtained by upsampling by the super-resolution method is clearer.

Claims (8)

1. The utility model provides a PVT-based gate control network's saliency target detection method which is characterized in that: the method comprises the following steps:

step 1: image preprocessing: adjusting the size of an input image to a tensor X with a preset size;

step 2: establishing a PVT gating network: the PVT gating network comprises first to fourth feature processing units and a pyramid pooling module PPM; the first to third feature processing units have the same structure; the first feature processing unit comprises a feature encoder PVTE ₁ Transition layer T ₁ Door unit G ₁ And decoder characterizer FAD ₁ The method comprises the steps of carrying out a first treatment on the surface of the The fourth feature processing unit comprises a feature encoder PVTE ₄ Transition layer T ₄ Door unit G ₄ And decoder characterizer FAD ₄ The method comprises the steps of carrying out a first treatment on the surface of the Tensor X is sequentially processed by PVTE of feature encoder ₁ -PVTE ₄ After the processing, obtaining first to fourth feature tensors;

the fourth feature tensor is subjected to PPM processing by a pyramid pooling module to obtain fourth high-level semantic information; fourth feature tensor transitionLayer T ₄ The processed image is spliced with a fourth characteristic tensor and then is input into a gate unit G ₄ Then is connected with the transition layer T ₄ Is multiplied by the output of the decoder to obtain the fourth encoder effective information, and the fourth encoder effective information is added with the fourth high-level semantic information and then is input into the decoder characterizer FAD ₄ The method comprises the steps of carrying out a first treatment on the surface of the Decoder characterizer FAD ₄ Outputting a fourth decoded feature vector;

first feature tensor and second feature processing unit FAD ₂ The output second decoding eigenvector is input into the gate unit G after being spliced ₁ The first characteristic tensor passes through the transition layer T ₁ After-process AND gate unit G ₁ The output of the first encoder is multiplied to obtain the effective information of the first encoder, the fourth advanced semantic information and the second decoding feature vector are added and then input into a decoder feature device FAD ₁ The method comprises the steps of carrying out a first treatment on the surface of the Outputting the first decoding feature vector as a saliency map;

step 3: detecting a significance map: and inputting tensors, and processing the tensors through a PVT gating network to obtain a saliency map.

2. The method for detecting significance targets in PVT-based gating networks according to claim 1, wherein: the input tensor size is 384×384×3, and the first to fourth feature tensors are 96×96×64, 48×48×128, 24×24×320, 12×12×512, respectively.

3. The method for detecting significance targets in PVT-based gating networks according to claim 1, wherein: feature encoder PVTE _i And decoder feature FAD _i+1 Integrating, and calculating a gate value through convolution, activation and pooling operations:

wherein Cat (·) is a splicing operation between channel axes, conv (·) is a convolution operation, S (·) is an element-level sigmoid function, and P (·) is global average pooling;

applying a threshold to transition layer characteristicsSign T ₁ -T ₄ Weighting, transition layer T ₁ -T ₄ Is a feature encoder PVTE using the generated 3X 3 convolution pair ₁ -PVTE ₄ And (5) carrying out vitamin reduction.

4. The method for detecting significance targets in PVT-based gating networks according to claim 1, wherein: the pyramid pooling module PPM carries out 4 kinds of adaptive average pooling operations with different scales on the fourth characteristic tensor to obtain 4 kinds of characteristic graphs with different sizes, wherein the sizes are respectively 1 multiplied by 1,2 multiplied by 2,3 multiplied by 3 and 6 multiplied by 6; the corresponding level channel is reduced to the original 1/4 by using the convolution of 1 multiplied by 1; the size before non-pooling is obtained through bilinear interpolation, and then the size is spliced with a fourth characteristic tensor in the channel dimension.

5. The method for detecting significance targets in PVT-based gating networks according to claim 1, wherein: the output of the decoder characterizer is:

wherein feature aggregation decoder FAD ₁ A saliency map of the same size as the input image;

feature encoder PVTE _i Feature input transition layer T _i In the module; transition layer T _i Reducing the number of channels and feeding to the gate unit G _i The method comprises the steps of carrying out a first treatment on the surface of the The output of the pyramid pooling module PPM is subjected to dimension reduction operation and up-sampled by adopting a bilinear interpolation method, so that the output is matched with T _i The output dimensions and dimensions of (a) are the same; encoder effective information, pyramid pooling module PPM output and feature aggregation decoder FAD _i+1 Output fused incoming feature aggregation decoder FAD _i Is a kind of medium.

6. The method for detecting significance target of PVT-based gate control network according to claim 5, wherein: the feature aggregation decoder FAD first pools the input feature map with three downsampled averages of {2,4,8 }; and then up-sampling by using a bilinear interpolation method to obtain an output characteristic diagram with the original size.

7. The method for detecting significance target of PVT-based gate control network according to claim 5, wherein: the feature aggregation decoder FAD firstly pools the input feature map with three downsampling averages of {2,4,8 }; and then up-sampling by a super-resolution method to obtain an output characteristic diagram with the original size.

8. The method for detecting significance target of PVT-based gate control network according to claim 5, wherein: the feature aggregation decoder FAD firstly uses three downsampling average pooling of {4,16,64} to obtain three feature graphs with different sizes; and up-sampling the characteristic diagrams with three different sizes to the original size to obtain the same size.