CN113269226B - Picture selection labeling method based on local and global information - Google Patents

Abstract

The invention discloses a picture selection and marking method based on local and global information, which can learn a model as good as possible by using pictures with marks as less as possible by enabling a learning model to automatically select partial pictures for marking. In order to reduce the requirement of image marking, the method utilizes the feature extraction capability of the depth model to construct the feature representation space of the image sample, and the effect of the sample on model updating is measured based on the local information of the image sample in the feature representation space. Meanwhile, the picture data space is divided into different areas based on the global information of the feature representation space, and the labeling budget is dynamically allocated according to the performance of the model on the different areas, so that the picture marking information is efficiently utilized, and the demand of picture marking is reduced.

Description

An image selection and annotation method based on local and global information

technical field

The invention relates to a picture selection and labeling method based on local and global information. The method can use the local and global information of the feature representation space to efficiently select objects to be labelled in a picture database, and train better pictures with less labeling cost. The classification model belongs to the technical field of computer artificial intelligence data analysis.

Background technique

With the continuous development of the Internet, a large amount of picture data needs to be processed, such as face pictures in face recognition, road pictures in automatic driving, and commodity pictures on e-commerce platforms. The image data structure is relatively complex, so deep models are often used to complete image classification tasks. But training deep models requires a large number of labeled images. Usually, labeling these images requires a lot of manpower and material resources and is expensive. In order to reduce the cost of labeling and improve the efficiency of using labeled pictures, one solution is to let the model automatically select important pictures that need to be labeled, and collect the labels of these pictures to update the model. This is the basic idea of choosing labels. The current selection and labeling methods mainly consider the uncertainty and representativeness of the data when measuring the importance of the data. The lower the confidence of the model in the prediction result of the data, the higher the uncertainty of the data. In addition, the modulo length of the data gradient can also be used to estimate the uncertainty of the data. Since the uncertainty-based method only considers the degree of uncertainty of a single data, it is easy for the model to pick out a batch of data with high uncertainty but redundant. This problem can be mitigated to some extent by considering the representativeness of the data. The representation-based method generally aggregates the features of the data into several clusters, and selects the center point of each cluster as the representative of the cluster. In this way, only a small amount of data can be used to describe the distribution of the entire data. However, since this method does not have model information as a guide, the selected data is not necessarily conducive to model updating.

SUMMARY OF THE INVENTION

Purpose of the invention: In view of the problems and deficiencies in the prior art, the present invention provides a picture selection and annotation method based on local and global information. This method can use the image features to represent the local information in the space, and combine the prediction results of the model to measure the information content of the image, and can avoid similar or redundant images to a certain extent. At the same time, combined with the global information of the feature representation space, the image data is divided into multiple clusters, and the labeling budget is dynamically allocated according to the performance of the model on different clusters, which further improves the utilization efficiency of image labeling and reduces the labeling cost. When using the same number of labeled images, the model trained by this method has better performance than the general selective labeling method.

Technical solution: a method for selecting and labeling pictures based on local and global information, including the following:

First, the user needs to create a picture object library. Next, some image objects are randomly selected from the image object library, and the labels of these image objects are obtained to form an initial training set. The user sets the structure of the depth model, the number of image objects selected in each round, and the total number of iterations.

Next, the deep learning model is trained based on the training set. The image objects in the image object library are converted into feature representations by using the deep model, that is, the features of the pictures in the image object library are extracted. The output of the penultimate layer of the deep model is often used as the feature representation of the corresponding image object. The space composed of these feature representations is called the feature representation space.

Then, in the feature representation space, the information amount of each object is estimated according to the local information calculation method, and the labeling budget is allocated according to the global information budget allocation method. Based on the budget, a batch of image objects with high information content is selected, and the tags of these image objects are collected. Update the marked image object collection and the unmarked image object collection. At the same time, the deep model is retrained using the set of labeled image objects, and the feature representation of the image objects is re-extracted using the new model. These steps are iterated sequentially for the specified number of rounds. The model in the last round is the final deep model.

Finally, in the prediction stage, the user inputs the image object to be tested into the trained deep model, and the deep model returns the prediction result to the user.

Beneficial effects: Compared with the prior art, the present invention avoids selecting redundant pictures by combining the local and global information in the feature representation space, and avoids selecting redundant pictures by considering the local information of the picture object, and assigns the global information of the feature representation space as needed. budget, improve the utilization efficiency of image data tagging, and reduce the cost of tagging.

Description of drawings

Fig. 1 is the flow chart of the present invention;

Fig. 2 is the flow chart of local information calculation method in the present invention;

FIG. 3 is a flow chart of the global information budget allocation method in the present invention.

Detailed ways

Below in conjunction with specific embodiments, the present invention will be further illustrated, and it should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. The modifications all fall within the scope defined by the appended claims of this application.

As shown in Figure 1, the image selection and annotation method based on local and global information includes the following steps:

表示有标记图片对象组成的集合，

表示未标记图片对象组成的集合；In step 100, a picture object library is established as a data set, a small number of objects are randomly selected from the picture object library, and the labels of these objects are obtained to form an initial training set. Note that the number of categories of data in the image object library is C.

Represents a collection of labeled image objects,

Represents a collection of unlabeled image objects;

为模型参数，

为模型的全连接层参数，θ为模型中的其他参数，用户选择每一轮选取的样本数目B与迭代总轮数T；Step 101, the user selects the depth model to be used, and denote the model as f(·;Θ), where

are model parameters,

is the fully connected layer parameter of the model, θ is other parameters in the model, the user selects the number of samples B selected in each round and the total number of iterations T;

训练深度模型，当前轮数t＝1；Step 102, use the marked picture object

Train the deep model, the current number of rounds t=1;

Step 103, input the unlabeled picture object into the depth model, and extract the feature representation r _θ (x) and softmax layer output f (x; Θ) of the picture object according to the depth model;

Step 104, according to the local information calculation method, estimate the amount of information provided by each object for the model, as shown in Figure 2, the specific steps are:

Step 1041, the user selects the range ∈ of the local neighborhood area;

模型f(x；Θ)预测的标记为

为了增加鲁棒性，进行如下的概率平滑：Step 1042, for the unlabeled image object x, the output of the softmax layer is f(x; Θ)=(p ₁ , . . . , p _C ),

The labels predicted by the model f(x;Θ) are

To increase robustness, perform the following probabilistic smoothing:

where g(x; Θ)=(g(x; Θ) ₁ , . . . , g(x; Θ) _C );

与

基于平滑后的概率计算信息量Step 1043, for unmarked picture objects

and

Calculate the amount of information based on the smoothed probability

其中r_θ(x)为图片对象x的特征表示，图片对象x的信息量为

Step 1044, record the adjacent area of the image object x as

where r _θ (x) is the feature representation of the image object x, and the amount of information of the image object x is

计算

并输出。Step 1045, for all unlabeled picture objects

calculate

and output.

Step 105: According to the global information budget allocation method, the unlabeled data is clustered into C clusters in the feature representation space, and the budgets (B ₁ , . . . , B _C ) are allocated in different clusters, where B _j is allocated to the jth The tag budget for each cluster. As shown in Figure 3, the specific steps are:

Step 1051, the temperature parameter τ of the Gibbs distribution is selected by the user;

Step 1052, use the kmeans++ method to gather the feature representations of the unlabeled picture objects into C clusters, and the set of picture objects in the jth cluster is:

Step 1053, estimate the performance of the model on different clusters, and denote the performance of the model on the jth cluster as γ _j

Step 1054, construct a Gibbs distribution of budget α=(α ₁ , . . . , α _C ) according to γ _j

where ∑ _j α _j =1, τ is the temperature parameter used to adjust the smoothness of the Gibbs distribution;

Step ₁₀₅₅ , _perform B times of sampling according to the Gibbs distribution α, obtain the budgets ( _{B 1} , .

中，更新

与

重新训练深度模型；Step 106: In each cluster, according to the corresponding budget B _j j∈[C], select B _j picture objects with the highest amount of information, obtain the marks of these picture objects, and add these picture objects to the set of marked objects.

medium, update

and

Retrain the deep model;

Step 107, if t<T, then t=t+1, jump to step 103;

Step 108, using the model obtained by the T-th round of training as the final model. For the object to be tested, output the label predicted by the model.

Claims (1)

1. a picture selection labeling method based on local and global information, is characterized in that, comprises the following content: First build a picture object library; then randomly select some picture objects from the picture object library, obtain the labels of these picture objects, and form an initial training set; set the structure of the deep model, the number of picture objects selected in each round, and the total number of iterations. number; Next, the deep learning model is trained based on the training set; the image objects in the image object library are converted into feature representations by using the deep model, that is, the features of the pictures in the image object library are extracted; the space formed by the feature representation is called the feature representation space; Then, in the feature representation space, the information amount of each object is estimated according to the local information calculation method, and the labeling budget is allocated according to the global information budget allocation method; Labeling of picture objects; update the set of labeled picture objects and the set of unlabeled picture objects; at the same time, use the set of labeled picture objects to retrain the deep model, and use the new model to re-extract the feature representation of picture objects; iterate the specified number of rounds; the last round The model is the final deep model; Finally, in the prediction stage, the user inputs the image object to be tested into the trained deep model, and the deep model returns the prediction result to the user; Note that the number of categories of data in the image object library is C;

Represents a collection of labeled image objects,

Represents a collection of unlabeled image objects; the selected depth model is denoted as f( ;Θ), where

are model parameters,

is the fully connected layer parameter of the model, θ is other parameters in the model, the user selects the number of samples B selected in each round and the total number of iterations T; use marked image objects

Training the depth model, the current number of rounds t=1; input the unlabeled image object into the depth model, and extract the feature representation of the image object according to the depth model r _θ (x) and the softmax layer output f (x; Θ); Using probability smoothing and local information to calculate the amount of object information, the specific steps are: Step 1041, select the range ∈ of the local neighborhood area; Step 1042, for the unlabeled image object x, the output of the softmax layer is f(x; Θ)=(p ₁ ,...,p _C ),

The labels predicted by the model f(x;Θ) are

Probabilistic smoothing is performed as follows:

where g(x; Θ)=(g(x; Θ) ₁ ,...,g(x; Θ) _C ); Step 1043, for unmarked picture objects

and

Calculate the amount of information based on the smoothed probability

Step 1044, record the adjacent area of the image object x as

where r _θ (x) is the feature representation of the image object x, and the amount of information of the image object x is

Step 1045, for all unlabeled picture objects

calculate

and output; According to the global information budget allocation method, the unlabeled data is clustered into C clusters in the feature representation space, and the budgets (B ₁ ,...,B _C ) are allocated in different clusters, where B _j is allocated to the jth cluster , the specific steps are: Step 1051, the temperature parameter τ of the Gibbs distribution is selected by the user; Step 1052, use the kmeans++ method to gather the feature representations of the unlabeled picture objects into C clusters, and the set of picture objects in the jth cluster is:

Step 1053, estimate the performance of the model on different clusters, and denote the performance of the model on the jth cluster as γ _j

Step 1054, construct the Gibbs distribution of budget α=(α ₁ ,...,α _C ) according to γ _j

τ is the temperature parameter used to adjust the smoothness of the Gibbs distribution; Step ₁₀₅₅ : _Perform B sampling according to the Gibbs distribution α to obtain and output the allocated budgets ( _{B 1} , .

Images