CN107679539B - A Method for Integrating Local Information and Global Information of Single Convolutional Neural Network Based on Local Receptive Field - Google Patents

Abstract

The invention discloses a method for integrating local information and global information of a single convolution neural network based on local perception fields, which comprises the following steps: step 1, a convolutional neural network model is given, and the size of a local perception field of each layer of feature map in an original image is calculated; step 2, selecting a layer to be divided according to the size of the local perception field of each layer so as to balance local information and global information; step 3, comprehensively considering the information amount and the calculated amount, selecting a segmentation mode and the segmentation number, and segmenting the feature map of the selected layer; step 4, carrying out dimension matching on the segmented feature map; and step 5, superposing all layers behind the selected layer, including the loss function layer, to each segmented sub-feature map, and then training. The convolutional neural network reconstruction algorithm provided by the invention can enable local information and global information to be obtained by simultaneously learning in one network model, improves the expression capability of the network model, and only generates less calculation amount increase.

Description

Local information and global information of a single convolutional neural network based on local perceptual field Integration method

technical field

The invention relates to the technical field of image processing, in particular to a method for integrating local information and global information of a single convolutional neural network based on a local perceptual field.

Background technique

In 2012, the convolutional neural network won the championship in the ImageNet large-scale image recognition competition, and achieved an overwhelming advantage over traditional algorithms. In recent years, convolutional neural networks have been increasingly applied to various fields of computer vision, including image recognition, image detection, image segmentation, etc.

Convolutional neural network is a multi-layer neural network model composed of several convolutional layers, pooling layers, and loss function layers. When a picture is input to the convolutional neural network, as the depth increases, a fixed-size region in each layer of feature maps corresponds to a corresponding increase in the size of the local perceptual field in the original image.

Due to the structural characteristics of convolutional neural networks, global information can be well utilized, while local detail information is usually not easy to learn. In order to integrate local information and global information at the same time, researchers currently tend to extract multiple sub-blocks from the original image, and then train a separate convolutional neural network model for each image sub-block. In the test phase, each image and its sub-blocks need to go through the corresponding model to extract features, and then average or stitch all the extracted features as the final features of the image. But this method has many limitations. 1. The feature extraction time will increase linearly with the increase of the number of models, especially in the test phase, too long feature extraction time will affect the performance of the model after deployment; 2. The final accuracy rate will increase with the number of models The increase is getting smaller and smaller. Usually, the accuracy rate increase from one model to two models will be greater than the accuracy rate increase from two models to more than ten models; 3. The selection of sub-blocks requires a lot of manual participation, and the splicing is not suitable sub-blocks will even degrade the final accuracy.

Contents of the invention

Aiming at the limitations of the above-mentioned existing method of integrating local information and global information by using multiple models by dividing the original image, the present invention provides a method for integrating local information and global information of a single convolutional neural network based on the local perceptual field, which can enable The local information and global information are simultaneously learned in the network model, which improves the expressive ability of the network model, and at the same time only produces a small increase in the amount of calculation.

A single convolutional neural network local information and global information integration method based on local perceptual field, including:

Step 1, given the convolutional neural network model, calculate the local perceptual field size corresponding to the feature map of each layer in the original image;

Step 2, according to the size of the local perceptual field of each layer, select a layer segmentation to balance local information and global information;

Step 3, comprehensively considering the amount of information and the amount of calculation, selecting the segmentation method and the number of segmentations, and segmenting the feature map of the selected layer;

Step 4, perform dimension matching on the segmented feature map;

In step 5, all layers behind the selected layer, including the loss function layer, are superimposed on each segmented sub-feature map, and trained to complete the integration of local information and global information.

For the convenience of expression, record the input layer, that is, the image layer, as layer 0, and the subsequent layer with depth i as layer i.

In step 1, the calculation of the size of the local perceptual field can use layer-by-layer iteration:

For a region of size H _N * W _N in the feature map of the Nth layer, it can be conveniently calculated according to the network parameters of the Nth layer such as kernel size and step size to obtain the size H _N corresponding to the feature map of the N-1th layer _-1 *W _N-1 , iterate layer by layer until the input layer of the 0th layer, and obtain H ₀ *W ₀ , which is the desired result.

Among them, H _N and W _N are the length and width of the region described in the feature map of the Nth layer, H _N-1 and W _N-1 are the length and width of the region described in the feature map of the N-1 layer, H ₀ and W ₀ is the length and width of the area described in the feature map of layer 0.

Preferably, in step 2, the present invention selects a convolutional layer located in the middle of the network and followed by a pooling layer.

In step 3, the original feature map can be divided into an appropriate number of sub-feature maps evenly or according to key points.

In step 4, there are many ways to enlarge the feature map for dimension matching, including: linear interpolation, adding a deconvolution layer, etc. For a convolutional neural network model with a specific structure, you can also choose a depooling layer.

The present invention corresponds to the size of the local perception field in the original image by analyzing a fixed-size area in each layer feature map in the convolutional neural network model, selects a suitable layer for segmentation and dimension matching, and then superimposes subsequent layers on the segmentation layer After that, in order to achieve the purpose of integrating local information and global information in a single convolutional neural network, it has the following advantages:

(1) Integrate local information and global information into a single convolutional neural network model;

(2) Since the first few layers remain unchanged, the global information can be preserved, and better results can be achieved compared to splitting the original image;

(3) Since the first few layers remain unchanged and share weights, the amount of calculation increases less than that of splitting the original image.

Description of drawings

Fig. 1 is the flowchart of the method for integrating local information and global information of a single convolutional neural network based on a local perceptual field in the present invention;

Fig. 2 is a schematic structural diagram of a method for integrating local information and global information of a single convolutional neural network based on a local perceptual field in the present invention.

Detailed ways

Below in conjunction with accompanying drawing and example, the present invention will be further introduced.

The method for integrating local information and global information of a single convolutional neural network based on the local perceptual field provided by the present invention is implemented on a Linux system based on the deep learning framework Caffe. The process flow is shown in Figure 1, and the structural diagram is shown in Figure 2. Table 1 shows the basic model of the convolutional neural network used in the experiment of the present invention. Its input size is 100*100, and it is formed by superimposing five groups of similar modules. Each group of modules is composed of two convolutional layers superimposed with a pooling layer , or three convolutional layers. The last column of Table 1 shows that after the feature map of the current layer is evenly divided into 2*2 feature maps, each segmented feature map corresponds to the size of the local perceptual field in the original image. Taking this model as an example, the specific steps of the present invention are described as follows:

(1) Given a convolutional neural network model, calculate the local perceptual field size corresponding to the feature map of each layer in the original image.

Take the Conv22 layer as an example. For the area in the corner of Conv22 whose size is H*W, Conv22/Conv21 is a convolutional layer with a kernel size of 3*3 and a step size of 1, which corresponds to (H+1)*(W+1) in Conv21 The feature map of is corresponding to the feature map of (H+1+1)*(W+1+1) in Pool1.

Table 1

name type Kernel size/step size output size Receptive field size Conv11 convolutional layer 3×3/1 100×100×32 51×51 Conv12 convolutional layer 3×3/1 100×100×64 52×52 Pool1 max pooling layer 2×2/2 50×50×64 52×52 Conv21 convolutional layer 3×3/1 50×50×64 54×54 Conv22 convolutional layer 3×3/1 50×50×128 56×56 Pool2 max pooling layer 2×2/2 25×25×128 58×58 Conv31 convolutional layer 3×3/1 25×25×96 62×62 Conv32 convolutional layer 3×3/1 25×25×192 66×66 Pool3 max pooling layer 2×2/2 13×13×192 70×70 Conv41 convolutional layer 3×3/1 13×13×128 78×78 Conv42 convolutional layer 3×3/1 13×13×256 86×86 Pool4 max pooling layer 2×2/2 7×7×256 94×94 Conv51 convolutional layer 3×3/1 7×7×160 100×100 Conv52 convolutional layer 3×3/1 7×7×320 100×100 Conv5 convolutional layer 7×7/1 1×1×320 100×100 Dropout dropout (40%) 1×1×320 FC10575 fully connected layer 10575 loss Softmax 10575

Pool1 is a pooling layer with a core size of 2*2 and a step size of 2. Conv22 corresponds to the feature map of 2(H+1+1)*2(W+1+1) in Conv12. By analogy, Conv22 corresponds to (2(H+2)+2)*(2(W+2)+2) in the original image. Therefore, after the Conv22 output 50*50 feature map is evenly divided into 2*2, that is, 4 feature maps with a size of 25*25, each segmented feature map corresponds to the local perceptual field size in the original image (2 (25+2)+2)*(2(25+2)+2) is 56*56.

It is worth noting that the derivation above is that the feature map after segmentation is located in the corner, that is, the length and width directions contain the most edge information of the original feature map. If the feature map after segmentation is not located in the corner of the original feature map, the corresponding local perception field Size calculations should be fine-tuned.

(2) According to the local receptive field size of each layer, an appropriate layer segmentation is selected to balance local information and global information.

In this example, we select the Conv22 layer for segmentation. Other layers are also possible.

(3) Comprehensively consider the amount of information and the amount of calculation, select the appropriate segmentation method and the number of segmentation, and segment the feature map of the selected layer.

In this example, since the aspect ratio of the input image and all feature maps is 1:1, in order to avoid the deformed upsampling operation caused by inconsistent length and width, we evenly divide the Conv22 layer feature map into N* with the same size and the same length and width. N sub-feature maps. If N=3, the calculation amount of the modified model is approximately 9 times that of the original model, which will greatly reduce the efficiency, so we choose N=2. The original output size of the Conv22 layer is 50*50, and after this division, 4 sub-feature maps with a size of 25*25 will be obtained.

(4) Perform dimensionality matching on the segmented feature maps.

Dimension matching methods include linear interpolation, adding deconvolution layers, etc. In the implementation, linear interpolation is also achieved by deconvolutional layers with learning rate set to 0 and weights using a specific initialization method. In this example, since the Conv22 convolutional layer is followed by the Pool2 pooling layer, the length and width of the output feature map of the Pool2 layer are one-half the size of the input feature map, which is exactly the same as the segmentation method selected in step (3). , so you can choose to delete the Pool2 layer for dimension matching.

(5) All layers behind the selected layer, including the loss function layer, are superimposed on each segmented sub-feature map, and trained to complete the integration of local information and global information.

The method of the present invention relates to a method for integrating local information and global information of a single convolutional neural network based on a local perceptual field. By modifying the existing convolutional neural network model, the purpose of simultaneously learning local information and global information in a single model can be achieved , with better portability and wide applicability.

In order to prove the effectiveness of the method described in the present invention, a comparative experiment of face verification and face identification was done on the LFW data set standard evaluation index and the BLUFR evaluation index. The face verification problem is given two face images, to determine whether the faces in the two images are the same person. The problem of face identification is to give a face image and a face database, search out the photos in the database that belong to the same person as the face image, and return rejection if it does not exist. The LFW dataset contains tens of thousands of face images captured from the Internet, and the main individual names in each image are manually marked. Under the standard evaluation index, 6000 face images are divided into ten groups, each group contains 600 pairs, of which 300 pairs belong to the same person, and 300 pairs belong to different people. The evaluation algorithm needs to use cross-validation to train on nine sets of data to select thresholds, and then test on the remaining set, repeating ten times. The average accuracy rate on ten sets of data is used as the final effect of the algorithm.

The BLUFR evaluation index utilizes all LFW images and aims to evaluate the open-set face identification accuracy (DIR) of the algorithm under low false positive rate (FAR).

All comparative experiments use the CASIA-webface dataset as the training set for the convolutional neural network model. It contains 494,414 face images captured from the imdb website, which belong to 10,575 individuals who have been manually guaranteed to be mutually exclusive with LFW.

Table 2

As shown in Table 2, it is the result under the standard evaluation index of LFW.

A. Base model. The base model achieves 98.02% accuracy without any segmentation. At the same time, we doubled the number of channels of the basic model conv22 layer and all subsequent convolutional layers to obtain the basic model -2x, to match the calculation amount of the split conv22 layer model for comparison. The model achieved 98.10% accuracy.

B. Split the image layer. Because each segmented feature map in the conv22 layer corresponds to a local perceptual field size of 56*56 in the original image, we conducted two sets of experiments on the segmented image: 1. Evenly divided into four overlapping 56*56 regions; 2. Evenly divided into 4 non-overlapping 50*50 areas. Each sub-block after segmentation is scaled to 100*100 and then input to the base model. Due to the lack of global information, testing the features corresponding to each segmented image alone can only achieve an accuracy rate of about 95%. Taking the maximum operation cannot make good use of all information, and at the same time part of the information is lost, and the accuracy rate decreases. The accuracy of the averaging operation is improved compared to the accuracy of a single feature. The splicing operation preserves the information of all individual features, and the accuracy improvement is greater than that of the averaging operation, exceeding the base model. Due to having more information, the effect of dividing into four 56*56 regions is generally better than that of dividing into four 50*50 regions.

C. Split conv52 layer. Each segmented feature map in the conv52 layer has obtained all the original image information, so testing the features corresponding to each segmented feature map can achieve an accuracy rate of about 98%. But due to the reduced number of connections between conv52 and conv5, they are both less effective than the base model. Taking the maximum and averaging operations does not add additional information, but loses the necessary information, and the accuracy rate is significantly lower than that of a single one. The splicing operation has improved accuracy compared to single features, achieving a similar effect to the basic model.

D. Split conv22 layer. Similar to the segmented image layer, due to the failure to fully utilize all the information of the original image, testing the features corresponding to each segmented conv22 feature map can only achieve an accuracy rate of about 96%, but it is better than the segmented image layer. This is because when the conv22 layer is divided, conv11, conv12 and conv21 remain unchanged, and the shared convolution kernel can indirectly transfer some global information. The accuracy of the maximum operation is greatly reduced due to the loss of part of the information. Compared with the accuracy of a single feature, the accuracy of the average operation is greatly improved, and it is close to the basic model. The splicing operation retains the information of all individual features, and the accuracy rate is greatly improved, which greatly exceeds the effect of the basic model, reaching 99.02%.

At the same time, the last three rows of Table 2 show the effect of using different dimension matching methods (segmenting the conv22 layer). The column of time refers to the time required to extract the features of an image. Both bilinear interpolation and deconvolution have achieved better results than depooling, but the improvement is not obvious. At the same time, the time required for them to extract features is nearly 2 times that of depooling. Therefore, in terms of comprehensive accuracy and efficiency, depooling is a better choice for dimension matching.

table 3

As shown in Table 3, it is the result under the BLUFR evaluation index of LFW. Similar to the results under standard evaluation metrics, splitting the conv22 layer greatly improves the performance of the original model, and reduces the error rate by nearly 50% under the VR metrics.

Table 4

Table 4 shows the effect of the present invention and other current best methods on LFW.

Most of the algorithms that achieve 99% accuracy on LFW use the method of cutting the original image. FaceNet and Baidu are two exceptions, but the training sets they use are both private and much larger than the one used in this invention. CASIA and MM-DFR use the same training set as the present invention, but compared with CASIA, the present invention reduces the error by 3 times; MM-DFR has almost the same accuracy rate as the present invention, but requires more calculation. As far as the inventors know, the present invention is the first method to achieve 99% accuracy in LFW only based on the original image without sub-blocks using the public training set.

The above descriptions are only preferred implementations of the present invention, and the scope of protection of the present invention is not limited to the above-mentioned implementations. All technical solutions belonging to the principle of the present invention belong to the scope of protection of the present invention. For those skilled in the art, some improvements and modifications made without departing from the principles of the present invention should also be regarded as the protection scope of the present invention.

Claims (5)

1. A single convolutional neural network local information and global information integration method based on local perceptual field, including: Step 1, given the convolutional neural network model, calculate the local perceptual field size corresponding to the feature map of each layer in the original image; Step 2, according to the size of the local perceptual field of each layer, select a layer segmentation to balance local information and global information; Step 3, comprehensively considering the amount of information and the amount of calculation, select the segmentation method and the number of segmentations, and segment the feature map of the selected layer. The segmentation method is to evenly divide the feature map of the layer into 2*2 feature maps according to the key points; Step 4, perform dimension matching on the segmented feature map; In step 5, all layers behind the selected layer, including the loss function layer, are superimposed on each segmented sub-feature map, and trained to complete the integration of local information and global information. 2. The single convolutional neural network local information and global information integration method based on the local perceptual field according to claim 1, characterized in that, in step 1, the size of the local perceptual field uses a layer-by-layer iterative calculation method: For a region with a size of H _N * W _N in the feature map of the N-th layer, it is calculated according to the network parameters of the N-th layer to obtain its corresponding size H _N-1 * W _N-1 in the feature map of the N-1 layer, step by step Layer iteration until the input layer of the 0th layer, H ₀ *W ₀ is obtained, which is the required local perceptual field size; Among them, H _N and W _N are the length and width of the region described in the feature map of the Nth layer, H _N-1 and W _N-1 are the length and width of the region described in the feature map of the N-1 layer, H ₀ and W ₀ is the length and width of the area described in the feature map of layer 0. 3. The method for integrating local information and global information of a single convolutional neural network based on a local perception field according to claim 2, wherein said network parameters are kernel size and step size. 4. the single convolutional neural network local information and global information integration method based on the local perceptual field according to claim 1, characterized in that, in step 2, the selected layer is located in the middle of the network and is followed by pooling layers of convolutional layers. 5. the single convolutional neural network local information and global information integration method based on local perception field according to claim 1, is characterized in that, in step 4, described feature map uses linear interpolation or adds deconvolution layer method to perform zoom-in operations for dimension matching.