CN118351399A - Sample generation method, image recognition model training method and corresponding device - Google Patents

Abstract

The disclosure relates to a sample generation method, a training method of an image recognition model and a corresponding device, and relates to the technical field of data processing, wherein the method comprises the following steps: determining a plurality of image sets; clustering a plurality of images corresponding to each image set to obtain a plurality of intra-class clusters, and selecting images from different intra-class clusters to form positive sample pairs; clustering the plurality of image sets to obtain a plurality of inter-class clusters, and selecting images corresponding to different image sets from the same inter-class cluster to form a negative sample pair; a triplet of sample pairs is generated from the plurality of image sets, the positive sample pairs, and the negative sample pairs. By selecting images with lower similarity in different intra-class clusters to form positive sample pairs, the situation that invalid positive sample pairs are generated due to the fact that the same images with higher similarity are selected can be reduced. The images corresponding to different image sets are selected from the same inter-class cluster to form the negative sample pair with higher similarity, so that the condition that invalid negative sample pairs are generated due to the selection of different images with lower similarity can be reduced.

Description

Sample generation method, image recognition model training method and corresponding device

Technical Field

The present disclosure relates to the field of data processing technology, and in particular, to a sample generation method, an image recognition model training method and corresponding devices.

Background technique

In-vehicle image recognition mainly applies image recognition technology to vehicles. When image recognition technology is installed in a vehicle, the vehicle can scan the in-vehicle image to identify the corresponding identity information, so as to realize personalized services such as door unlocking, ignition start, seat adjustment and route planning, as well as in-vehicle payment and other functions.

In the related art, a large number of images of drivers are collected and used as training samples to train an image recognition model, and then the trained image recognition model is used to identify the driver's identity information to achieve personalized vehicle services and functions.

Summary of the invention

The purpose of the present disclosure is to provide a sample generation method, an image recognition model training method and corresponding devices to solve the technical problems existing in the above-mentioned related technologies.

In order to achieve the above objectives, in a first aspect, the present disclosure provides a sample generation method, comprising:

Determine a plurality of image sets, each of the image sets including a plurality of images corresponding to the same identity information;

For each of the image sets, clustering the multiple images corresponding to the image set to obtain multiple intra-class clusters, selecting images from different intra-class clusters to form positive sample pairs, wherein the similarity between the features corresponding to different images in the same intra-class cluster is less than a first similarity threshold;

Clustering the multiple image sets to obtain multiple inter-class clusters, selecting images corresponding to different image sets in the same inter-class cluster to form negative sample pairs, wherein the similarity between the features corresponding to the different image sets in the same inter-class cluster is less than a second similarity threshold;

A triplet of sample pairs is generated according to the multiple image sets, the positive sample pairs, and the negative sample pairs.

Optionally, clustering the multiple images corresponding to the image set to obtain multiple intra-class clusters includes:

Calculating the similarity between every two images in the image set;

Considering the images in the image set as nodes, connecting the nodes corresponding to the two images whose similarity is less than the first similarity threshold, to obtain a first undirected graph;

The image corresponding to each subgraph in the first undirected graph is determined as an intra-cluster of a class, so as to obtain a plurality of intra-cluster of the class.

Optionally, selecting images from different in-class clusters to form positive sample pairs includes:

Randomly selecting a first image from a first in-class cluster and randomly selecting a second image from a second in-class cluster, wherein the first in-class cluster and the second in-class cluster are different in-class clusters among the multiple in-class clusters;

The first image and the second image constitute a positive sample pair.

Optionally, clustering the multiple image sets to obtain multiple inter-class clusters includes:

Calculating the similarity between the central features of every two image sets in the plurality of image sets, the central feature of the image being the image in the image set that meets a preset condition;

The central features of the image sets are regarded as nodes, and the nodes corresponding to the central features of the two image sets whose similarities are less than the second similarity threshold are connected to obtain a second undirected graph;

The image set corresponding to each subgraph in the second undirected graph is determined as an inter-class cluster to obtain the multiple inter-class clusters.

Optionally, selecting images corresponding to different image sets in the same inter-class cluster to form negative sample pairs includes:

Randomly selecting a third image and a fourth image from the same between-class cluster, wherein the third image and the fourth image are images corresponding to different image sets;

The third image and the fourth image form a negative sample pair.

Optionally, selecting images corresponding to different image sets in the same inter-class cluster to form negative sample pairs includes:

For each image set in the same between-class cluster, randomly selecting a fifth image from the image set;

Randomly select a sixth image from each of the multiple intra-class clusters of the remaining image set, wherein the remaining image set is the image set remaining after excluding the image set in the same inter-class cluster;

The fifth image and the sixth image constitute the negative sample pair.

Optionally, determining a plurality of image sets comprises:

A plurality of image sets captured by the vehicle-mounted image capture device are determined.

In a second aspect, the present disclosure provides a method for training an image recognition model, the method comprising:

The first image recognition model is trained according to the triplet sample pairs generated by the sample generation method according to any one of the first aspects of the present disclosure.

Optionally, before training the first image recognition model, the method further includes:

A vehicle-mounted image set in a vehicle-mounted image acquisition device is obtained, and a second image recognition model is trained according to the vehicle-mounted image set by using a normalized exponential softmax loss function to obtain the first image recognition model.

Optionally, before training the second image recognition model by using a normalized exponential softmax loss function to obtain the first image recognition model, the method further includes:

Get RGB image set;

Performing grayscale processing on the RGB image set to obtain a grayscale image set;

The third image recognition model is pre-trained according to the grayscale image set to obtain the second image recognition model.

In a third aspect, the present disclosure provides a non-temporary computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods provided in the first and second aspects of the present disclosure.

In a fourth aspect, the present disclosure provides a controller, comprising:

a memory having a computer program stored thereon;

A processor is used to execute the computer program in the memory to implement the steps of any one of the methods provided in the first aspect and the second aspect of the present disclosure.

In a fifth aspect, the present disclosure provides a vehicle, comprising the controller provided in the fourth aspect of the present disclosure.

In a sixth aspect, the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any one of the methods provided in the first and second aspects of the present disclosure.

Through the above technical solution, the similarity of multiple images corresponding to the same intra-class cluster is high, and the similarity of images corresponding to different intra-class clusters is low. Selecting images with lower similarity in different intra-class clusters to form positive sample pairs can reduce the situation where invalid positive sample pairs are generated due to selecting the same image with higher similarity. In addition, the similarity of images of multiple image sets corresponding to the same inter-class cluster is high. Selecting images corresponding to different image sets in the same inter-class cluster to form negative sample pairs with higher similarity can reduce the situation where invalid negative sample pairs are generated due to selecting different images with lower similarity. Therefore, a large number of triple sample pairs can be generated based on the positive sample pairs, negative sample pairs, and image sets. Furthermore, when the triple sample pairs are used to train the first image recognition model, the convergence stability and recognition accuracy of the first image recognition model can be improved.

Other features and advantages of the present disclosure will be described in detail in the following detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

FIG. 1 is a flow chart of vehicle-mounted face recognition in the related art.

FIG. 2 is a flow chart of a training model in the related art.

FIG. 3 is a schematic diagram showing a sample generating method according to an exemplary embodiment of the present disclosure.

FIG. 4 is a schematic diagram showing a first undirected graph according to an exemplary embodiment of the present disclosure.

FIG5 is a schematic flow chart showing a method for training an image recognition model according to an exemplary embodiment of the present disclosure.

FIG. 6 is a schematic diagram showing a process of selecting a triplet sample pair according to an exemplary embodiment of the present disclosure.

FIG. 7 is a schematic diagram showing a sample generating device according to an exemplary embodiment of the present disclosure.

FIG. 8 is a block diagram showing a vehicle according to an exemplary embodiment.

Detailed ways

The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

Image recognition technology is mainly a recognition technology that analyzes and processes images through computer vision technology. However, in related technologies, when performing image recognition, it is impossible to train the image recognition model with a small number of samples, or it is impossible to obtain a target image recognition model with stable convergence. This embodiment takes face recognition technology in image recognition technology as an example for explanation as follows.

When the face recognition technology is installed in the vehicle, as shown in FIG1 , the vehicle-mounted face recognition technology mainly includes three processes: training model, registering identity, and identifying. In the process of training the model, a large number of driver's face images are collected through the vehicle-mounted camera, and the face images are preprocessed. Then, the face recognition model is trained through the deep learning algorithm based on the preprocessed face images to obtain the target face recognition model. In the process of registering identity, the driver enters his own identity information and records his own face image through the vehicle-mounted camera, wherein the face image includes the front and multi-angle face images of the driver. In the process of identifying, the real-time face image of the driver is collected through the vehicle-mounted camera, and when the quality of the real-time face image is qualified, the matching rate between the real-time face image and the recorded face image is calculated through the target face recognition model, and the identity information of the corresponding driver is obtained according to the matching rate, and returned to the terminal, so that the corresponding vehicle control can be realized.

As shown in Figure 2, in the process of training the model, a large number of driver's face images are first collected, and then the face detection model is used to extract the face area image from the face image, and the face area image is affine transformed to align the face area image to the area at the same position as the preset face template. Then, the aligned face image is sent to the face recognition model, and by designing a reasonable loss function, the face recognition model is converged to obtain the target face recognition model, which can then enable the target face recognition model to distinguish drivers with different identity information.

During the model training process, the loss function in the face recognition model is trained mainly through the loss functions provided in the first related technology and the second related technology.

In the first related technology, it is mainly based on the softmax (normalized exponential function) and its variant loss functions based on margin (the distance between the training samples closest to the decision boundary and the force boundary), such as CosFace (cosine face), SphereFace (spherical face), ArcFace (arc face), etc. This type of loss function regards a large number of input face images as classification tasks, and regards face images corresponding to the same identity information as the same type of face images. When face images collected from drivers with different identity information are used as training samples, the face recognition model outputs a certain number of types of face images after training. The ultimate goal of training is to ensure that a large number of input face images are correctly classified.

In the second related technology, the loss function is mainly based on metric learning, such as triplet loss, contrastive loss, etc. This type of method constructs triplet sample pairs so that the distance between face images with the same identity information is smaller than the distance between face images with different identity information. This method can combine a small number of face images with different strategies to form a large number of training samples, which is more suitable for application scenarios with fewer face images.

When the triplet loss function is used as the loss function in the face recognition model to train face images, the triplet loss can be expressed by the following calculation formula.

in, is the input triplet sample, a is the anchor, p is the positive, and n is the negative. And a and p can represent different face images corresponding to the same identity information, and a and n can represent face images corresponding to different identity information.

In the second related technology, triple sample pairs are selected by the following method. In each round of iterative training of the face recognition model, triple sample pairs are generated online using the collected face images and the face recognition model. The specific method includes: in each round of iteration, all face images corresponding to p different identity information are randomly extracted from a large number of face images, and k face images are randomly extracted from the face images corresponding to each of the p identity information, and the k extracted face images constitute the current sample. Then, for each face image in each identity information, the face image and other face images corresponding to the identity information constitute a positive sample pair, and then a positive sample pair can be generated. positive sample pairs. Then, for each positive sample pair, the positive sample pair is combined with each face image in other identity information to form a negative sample pair, so a total of triplet sample pairs .

However, the inventors found that when using the loss function in the first related technology to train a face recognition model, a large number of face image samples are required, but the face images obtained from the vehicle are a small number of samples, and thus it is impossible to train a face recognition model on the face images obtained from the vehicle.

When the face recognition model is trained by the triplet loss function in the second related technology, the face images in the current sample are randomly extracted, and there may be face images with great differences such as men and women, which cannot form valuable negative sample pairs, resulting in the inability to converge the trained face recognition model, which is an invalid sample. Positive sample pairs are formed by randomly extracting from the face images corresponding to each identity information. The positive sample pairs may include a large number of face images with high similarity, and thus the trained face recognition model cannot converge, which is an invalid sample. And by the method of randomly extracting positive sample pairs and negative sample pairs, it may be impossible to extract positive sample pairs with low similarity and negative sample pairs with high similarity, which makes the accuracy of the trained target face recognition model uncertain, and may make the convergence of the target face recognition model unstable.

In view of this, the present disclosure provides a sample generation method, an image recognition model training method and corresponding devices to solve the problems existing in the above-mentioned related technologies.

As shown in FIG3 , FIG3 is a schematic diagram showing a sample generation method according to an exemplary embodiment of the present disclosure, and referring to FIG3 , the method includes:

S301: Determine a plurality of image sets, each of the image sets including a plurality of images corresponding to the same identity information;

S302: for each of the image sets, clustering multiple images corresponding to the image set to obtain multiple intra-class clusters, selecting images from different intra-class clusters to form positive sample pairs, wherein the similarity between features corresponding to different images in the same intra-class cluster is less than a first similarity threshold;

S303: clustering the multiple image sets to obtain multiple inter-class clusters, selecting images corresponding to different image sets in the same inter-class cluster to form negative sample pairs, wherein the similarity between the features corresponding to different image sets in the same inter-class cluster is less than a second similarity threshold;

S304: Generate triplet sample pairs according to the multiple image sets, the positive sample pairs, and the negative sample pairs.

Through the above technical solution, the similarity of multiple images corresponding to the same intra-class cluster is high, and the similarity of images corresponding to different intra-class clusters is low. Selecting images with lower similarity in different intra-class clusters to form positive sample pairs can reduce the situation where invalid positive sample pairs are generated due to selecting the same image with higher similarity. In addition, the similarity of images of multiple image sets corresponding to the same inter-class cluster is high. Selecting images corresponding to different image sets in the same inter-class cluster to form negative sample pairs with higher similarity can reduce the situation where invalid negative sample pairs are generated due to selecting different images with lower similarity. Therefore, a large number of triple sample pairs can be generated based on the positive sample pairs, negative sample pairs, and image sets. When the triple sample pairs are used to train the first image recognition model, the convergence stability and recognition accuracy of the first image recognition model can be improved.

In addition, compared with the method of training face recognition models through a large number of face images in related technologies, the technical solution provided by the present invention does not require a large number of images for training, and can achieve a large number of triple sample pairs composed of a small number of images, and train the first image recognition model through the triple sample pairs.

In order to enable those skilled in the art to better understand the sample generation method provided by the present disclosure, the above steps are described in detail with examples below.

For example, an image set may be a plurality of images corresponding to a piece of identity information collected. Multiple image sets may be a plurality of images corresponding to multiple pieces of identity information. Among them, an image set may be an image collected by a vehicle-mounted camera, which is not limited in the embodiments of the present disclosure. An image set may be a face image set, and the multiple images included in the image set may be multiple face images, which is not specifically limited in the embodiments of the present disclosure.

For example, clustering can be to divide multiple data into several categories, so that the data in the same category are similar to each other, and the data between different categories are relatively different. And the same image set can be multiple images under the same identity information, among which there may be images with high similarity and images with low similarity. In this way, multiple images in the image set can be clustered, and images with high similarity can be divided into an intra-class cluster, so that multiple intra-class clusters can be obtained, and the similarity of images between different intra-class clusters is low. Images can be selected from different intra-class clusters to form positive sample pairs, so that the similarity of the positive sample pairs can be low, and the occurrence of positive sample pairs with high similarity can be avoided. Among them, the similarity between the features corresponding to different images in the same intra-class cluster is less than the first similarity threshold, so that the similarity of different images in the same intra-class cluster can be high. Among them, the first similarity threshold can be a critical threshold for the similarity between two images.

By clustering multiple images in the image set, the similarity of multiple images corresponding to the same cluster is higher, and the similarity of images corresponding to different clusters is lower. Images with lower similarity are selected from different clusters to form positive sample pairs, which can reduce the situation of invalid positive sample pairs caused by selecting the same image with higher similarity.

In a possible manner, clustering the multiple images corresponding to the image set to obtain multiple intra-class clusters includes:

Calculating the similarity between every two images in the image set;

Considering the images in the image set as nodes, connecting the nodes corresponding to the two images whose similarity is less than the first similarity threshold, to obtain a first undirected graph;

The image corresponding to each subgraph in the first undirected graph is determined as an intra-cluster of a class, so as to obtain a plurality of intra-cluster of the class.

It should be understood that the similarity can be the Euclidean distance, which is not specifically limited in the embodiments of the present disclosure. When the similarity is the Euclidean distance, the Euclidean distance between every two images in the image set can be expressed by the following calculation formula.

Where N can be the feature dimension of multiple images in the image set, and m can be the index value. can be the Euclidean distance, , Both can be eigenvalues in the matrix vector formed by multiple images in the image set.

When the Euclidean distance is less than the first similarity threshold, it can represent that the similarity between the two images is high, and the two images can be classified into the same intra-class cluster. When the Euclidean distance is greater than the first similarity threshold, it can represent that the similarity between the two images is low, and the two images may not belong to the same intra-class cluster. The first undirected graph can be a mathematical structure composed of vertices and edges connecting these vertices.

Then, each image in the image set can be regarded as a node, and the nodes corresponding to the two images whose Euclidean distance is less than the first similarity threshold are connected to obtain a first undirected graph. In the first undirected graph, multiple nodes connected together can be regarded as a subgraph. Then, in the first undirected graph, multiple subgraphs can be formed or one subgraph can be formed. Among them, each subgraph can include two nodes, three nodes, four nodes and other nodes of varying numbers. When the subgraph includes two nodes, it can represent that the similarity of the two images corresponding to the nodes is high. When the subgraph includes three nodes, it can represent that the similarity of the images corresponding to the three nodes is high, and so on. Then, the subgraph in the first undirected graph can be determined as an intra-class cluster, and multiple intra-class clusters can be obtained.

As shown in FIG. 4, FIG. 4 is a schematic diagram showing a first undirected graph according to an exemplary embodiment of the present disclosure. In the first undirected graph, a first subgraph connected by eight nodes a-h, a second subgraph connected by three nodes l-n, and a third subgraph connected by three nodes i-k are included. In the first subgraph, the similarity of images corresponding to any two of the eight nodes a-h is high, and the first subgraph can be determined as an inner cluster of a class. In the second subgraph, the similarity of images corresponding to the three nodes l-n is high, and the second subgraph can be determined as an inner cluster of a class. In the third subgraph, the similarity of images corresponding to the three nodes i-k is high, and the third subgraph can be determined as an inner cluster of a class. Then, one image is randomly selected from the first subgraph and the second subgraph, and the similarity of the two images is low. One image is randomly selected from the second subgraph and the third subgraph, and the similarity of the two images is low. One image is randomly selected from the first subgraph and the third subgraph, and the similarity of the two images is low. Thus, the three subgraphs can be determined as three inner clusters in the first undirected graph.

According to the similarity, multiple images in the image set are divided into multiple intra-class clusters, so that the similarity of images in multiple image sets corresponding to the same inter-class cluster is higher. In the same inter-class cluster, images corresponding to different image sets are selected to form negative sample pairs with higher similarity, which can reduce the situation of invalid negative sample pairs generated by selecting different images with lower similarity.

In a possible manner, selecting images from different in-class clusters to form positive sample pairs includes:

Randomly selecting a first image from a first in-class cluster and randomly selecting a second image from a second in-class cluster, wherein the first in-class cluster and the second in-class cluster are different in-class clusters among the multiple in-class clusters;

The first image and the second image constitute a positive sample pair.

It should be understood that when forming a positive sample pair, two images with low similarity in the same image set need to be formed into a positive sample pair. Thus, a first image can be randomly selected from a first-class cluster, and a second image can be randomly selected from a second-class cluster, and the first-class cluster and the second-class cluster belong to different in-class clusters among multiple in-class clusters. In other words, the first image and the second image are images with low similarity, and then the first image and the second image can be formed into a positive sample pair, which is a positive sample pair with low similarity. This can avoid selecting negative sample pairs with high similarity. Thus, when the image set includes M in-class clusters, it can be formed. Positive sample pairs.

Selecting images with lower similarity from clusters within different classes to form positive sample pairs can reduce the situation where invalid positive sample pairs are generated due to selecting the same image with higher similarity. And when the positive sample pairs are used for subsequent training of the first image recognition model, the situation where invalid training occurs can be reduced.

For example, an inter-class cluster can be a set of multiple images corresponding to different identity information with high similarity. In multiple image sets, there may be images with different identity information but similarity. Therefore, multiple image sets can be clustered, and multiple image sets with high similarity can be divided into an inter-class cluster, and multiple inter-class clusters can be obtained. After that, images corresponding to different image sets can be selected in the same inter-class cluster to form negative sample pairs. In this way, negative sample pairs with high similarity can be formed, and negative sample pairs with low similarity can be avoided.

By dividing multiple image sets into multiple inter-class clusters through a clustering method, and selecting images corresponding to different image sets in each inter-class cluster to form negative sample pairs, negative sample pairs with high similarity can be obtained. The situation of generating invalid negative sample pairs due to selecting different images with low similarity can be reduced. Furthermore, when the negative sample pairs are used to train the first image recognition model, the situation of invalid training can be reduced.

In a possible manner, clustering the multiple image sets to obtain multiple inter-class clusters includes:

Calculating the similarity between the central features of every two image sets in the plurality of image sets, the central feature of the image being the image in the image set that meets a preset condition;

The central features of the image sets are regarded as nodes, and the nodes corresponding to the central features of the two image sets whose similarities are less than the second similarity threshold are connected to obtain a second undirected graph;

The image set corresponding to each subgraph in the second undirected graph is determined as an inter-class cluster to obtain the multiple inter-class clusters.

It should be understood that the central feature of the image set may be the images in the image set that meet the preset conditions.

When determining the central feature of an image set, the sum of the distances between each image in the image set and the remaining images can be calculated, and the image corresponding to the minimum distance sum can be determined as the central feature of the image set. Then the preset condition can be that the sum of the distances between the images in the image set and the remaining images reaches the minimum value. The central feature of the image set can be calculated by the following calculation formula.

Among them, K is the number of images in the image set, C is the central feature of the image set, is the Euclidean distance between the features corresponding to two images in the image set, can be the distance and, Can be the minimum distance and.

In this embodiment, the similarity may be the Euclidean distance, which is not specifically limited in the embodiment of the present disclosure. When the similarity is the Euclidean distance, the Euclidean distance between the central features of every two image sets in the plurality of image sets may be calculated based on the central features of the image sets. The Euclidean distance may be expressed by the following calculation formula.

in, is the Euclidean distance between the central features of each two image sets, N is the feature dimension of the central feature of the image set, and m is the index value. , are the central features of the image set.

When the Euclidean distance is less than the second similarity threshold, it can represent that the images corresponding to the two image sets corresponding to the similarity are highly similar; otherwise, it can represent that the images corresponding to the two image sets corresponding to the similarity are relatively low similarity. The central feature corresponding to each image set can be regarded as a node, and the nodes corresponding to the central features of the two image sets whose Euclidean distance is less than the second similarity threshold are connected to obtain a second undirected graph. In the second undirected graph, one subgraph or multiple subgraphs can be included. In each subgraph, one image set corresponding to a node can be included, or image sets corresponding to multiple nodes can be included. The images in different image sets in each subgraph have a high similarity. In other words, each subgraph can be an image corresponding to different identity information with a high similarity. Each subgraph is determined as an inter-class cluster, and then multiple inter-class clusters can be obtained according to the second undirected graph.

By clustering the central features of multiple image sets, the multiple image sets are divided into multiple inter-class clusters, and the inter-class clusters can be used in the subsequent determination of negative sample pairs, thereby avoiding the occurrence of negative sample pairs with low similarity.

In a possible manner, selecting images corresponding to different image sets in the same inter-class cluster to form negative sample pairs includes:

Randomly selecting a third image and a fourth image from the same between-class cluster, wherein the third image and the fourth image are images corresponding to different image sets;

The third image and the fourth image form a negative sample pair.

It should be understood that in each inter-class cluster, multiple image sets with high similarity can be included. The third image and the fourth image can be selected from the same inter-class cluster, and the third image and the fourth image belong to images corresponding to different image sets. In other words, the third image and the fourth image belong to images with different identity information but high similarity. Furthermore, the third image and the fourth image can be formed into a negative sample pair, so that the similarity of the negative sample pair is high, and the occurrence of negative sample pairs with low similarity can be reduced.

By selecting negative sample pairs with higher similarity in the same inter-class cluster, the situation of generating invalid negative sample pairs due to selecting different images with lower similarity can be reduced. Furthermore, when the negative sample pairs are used for subsequent training of the first image recognition model, the situation of invalid training can be reduced, and the training accuracy of the first image recognition model can be improved.

In a possible manner, selecting images corresponding to different image sets in the same inter-class cluster to form negative sample pairs includes:

For each image set in the same between-class cluster, randomly selecting a fifth image from the image set;

Randomly select a sixth image from each of the multiple intra-class clusters of the remaining image set, wherein the remaining image set is the image set remaining after excluding the image set in the same inter-class cluster;

The fifth image and the sixth image constitute the negative sample pair.

It should be understood that for the same inter-class cluster, the fifth image can be randomly selected from each image set in the inter-class cluster. Then, multiple intra-class clusters in the remaining image set can be determined, and the sixth image is randomly selected from each intra-class cluster of the multiple intra-class clusters. The remaining image set is the image set remaining after the image set is removed from the same inter-class cluster. The fifth image set and the sixth image set constitute a negative sample pair, and a negative sample pair with a high similarity can be obtained.

For example, when the inter-class cluster includes four image sets from the first image set to the fourth image set, when calculating the negative sample pairs formed by the first image set, a fifth image is randomly selected from the first image set. When the second image set includes 4 intra-class clusters, the third image set includes 5 intra-class clusters, and the fourth image set includes 8 intra-class clusters, it can be calculated that the total number of intra-class clusters included in the second image set to the fourth image set is 17. A sixth image can be randomly selected from each of the 17 intra-class clusters, and 17 sixth images can be obtained. The fifth image can be respectively combined with the 17 sixth images to form 17 negative sample pairs. The method for calculating the negative sample pairs formed from the second image set to the fourth image set is the same as that formed by the first image set, and will not be repeated here.

By combining the images corresponding to each image set in the same inter-class cluster with the images corresponding to each intra-class cluster in the remaining image set as negative sample pairs, the selected negative sample pairs can be negative sample pairs with higher similarity, thereby avoiding the occurrence of negative sample pairs with lower similarity. And by selecting valuable negative sample pairs based on the similarity between images and training the first image recognition model based on the negative sample pairs, the problem of oscillation of the first image recognition model caused by randomly selecting sample pairs can be avoided, and the occurrence of invalid training can be reduced.

For example, a triplet sample pair may be generated based on multiple image sets, positive sample pairs, and negative sample pairs. When generating the triplet sample pair, each positive sample pair in each image set may be combined with a negative sample pair formed by the inter-class cluster of the image set into a triplet sample pair.

For example, when the image set includes A intra-class clusters, the image set can form positive sample pairs. When the inter-class cluster of the image set includes 5 image sets, the intra-class clusters of each image set except the current image set include P1, P2, P3 and P4 intra-class clusters respectively. An image is randomly selected from each of these intra-class clusters, and a total of P1+P2+P3+P4 images can be extracted. Each positive sample pair in the image set is combined with the multiple images, and a total of For each image set, the above method is used to calculate the number of triple sample pairs, and a large number of triple sample pairs can be formed from a small number of images.

When the positive sample pairs with lower similarity and the negative sample pairs with higher similarity are combined into a triple sample pair, and the triple sample pair is used for subsequent training of the first image recognition model, the situation where the positive sample pairs with lower similarity and the negative sample pairs with higher similarity cannot be selected when the triple sample pairs are randomly selected can be reduced. The problem of oscillation in the training of the first image recognition model caused by the random selection of positive sample pairs and negative sample pairs can also be reduced, and the accuracy of the first image recognition model obtained by training can be made more accurate, thereby making the convergence stability of the first image recognition model better.

In a possible manner, the determining of the plurality of image sets includes:

A plurality of image sets captured by the vehicle-mounted image capture device are determined.

It should be understood that the vehicle-mounted image acquisition device may be a vehicle-mounted camera for acquiring images of the vehicle driver. The vehicle-mounted camera may be an in-vehicle monitoring camera, a panoramic camera, or the like, which is not specifically limited in the embodiments of the present disclosure. A small number of vehicle-mounted image sets may be acquired through the vehicle-mounted image acquisition device, and then feature data in the vehicle-mounted image sets may be extracted through an image recognition model to obtain multiple image sets.

Based on the same concept, this embodiment also discloses a method for training an image recognition model, the method comprising:

The first image recognition model is trained based on the triplet sample pairs generated by the sample generation method disclosed in this embodiment.

For example, the sample production method disclosed in this embodiment can generate a large number of triple sample pairs based on a small number of image sets, and use the large number of triple sample pairs to train the first image recognition model. The first image recognition model can be a face recognition model, which is not specifically limited in the embodiment of the present disclosure. Compared with the related art of obtaining a large number of face images to train the face recognition model, the training method of the first image recognition model provided in this embodiment does not need to provide a large number of images, and the first image recognition model can be trained to obtain a target image recognition model.

In a possible manner, before training the first image recognition model, the method further includes:

A vehicle-mounted image set in a vehicle-mounted image acquisition device is obtained, and a second image recognition model is trained according to the vehicle-mounted image set by using a normalized exponential softmax loss function to obtain the first image recognition model.

It should be understood that a vehicle-mounted image set collected by a vehicle-mounted image acquisition device is obtained, wherein the vehicle-mounted image acquisition device may be a vehicle-mounted camera, and the vehicle-mounted camera may be an in-vehicle monitoring camera, a panoramic camera, or other cameras, to which the embodiments of the present disclosure do not specifically limit. Afterwards, the second image recognition model may be trained by the vehicle-mounted image set through a normalized exponential softmax loss function to obtain a first image recognition model. Afterwards, the first image recognition model is trained by triplet sample pairs. Among them, the training process of training the second image recognition model by a normalized exponential softmax loss function is a training process in the related art, which will not be described in detail here.

First, the second image recognition model is trained by the normalized exponential softmax loss function, and then the first image recognition model is trained by the triple sample pairs, which can avoid the overfitting phenomenon that occurs when the second image recognition model is trained only by the softmax loss function, and can avoid the low generalization of the first image recognition model. The triple sample pairs can expand a small number of vehicle images into a large number of triple sample pair data, thereby avoiding the problem of being unable to train due to the small number of images in the vehicle image set, and can reduce the difficulty of sample acquisition.

The following operations are performed during each iteration: extract multiple image sets from the vehicle image set through the first image recognition model, train the first image recognition model according to the multiple image sets through the training method of the above-mentioned image recognition model, obtain a new first image recognition model and a loss value, and judge whether the loss value is less than the preset threshold. If the loss value is greater than the preset threshold, extract multiple new image sets from the vehicle image set through the new first image recognition model until the loss value obtained is less than the preset threshold, and stop the iteration. The first image recognition model obtained from the last iteration is used as the target image recognition model, and the target image recognition model is used to identify images in the vehicle. Among them, the preset threshold can represent the threshold at which the model reaches the convergence condition.

By extracting multiple image sets from the vehicle-mounted image set through the second image recognition model, and grouping the multiple image sets into triplet sample pairs, and performing iterative optimization training, the accuracy of the target image recognition model can be improved and the convergence stability of the target image recognition model can be improved.

In a possible manner, before training the second image recognition model by using the normalized exponential softmax loss function to obtain the first image recognition model, the method further includes:

Get RGB image set;

Performing grayscale processing on the RGB image set to obtain a grayscale image set;

The third image recognition model is pre-trained according to the grayscale image set to obtain the second image recognition model.

It should be understood that the RGB image set may be a large number of various image sets directly downloaded from the Internet. The RGB image set may be gray-scaled by the following calculation formula to obtain a gray-scale image set.

in, For the converted The pixel value of the point, is the pixel value of the pixel point in the RGB image set, is the R channel point before transformation The pixel value of G channel point before conversion The pixel value of is the B channel point before conversion The pixel value of .

Afterwards, the third image recognition model is pre-trained with a grayscale image set to obtain a second image recognition model, wherein the specific training process of pre-training the third image recognition model with a grayscale image set is the training process in the related art and will not be described here. The second image recognition model can then be applied to subsequent fine-tuning with an on-board image set to obtain a first image recognition model. By pre-training the third image recognition model with a large number of grayscale image sets after grayscale processing from a large number of RGB images, the convergence speed and generalization of the first image recognition model can be improved.

5 , which is a flow chart showing a method for training an image recognition model according to an exemplary embodiment of the present disclosure. As shown in FIG5 , the process of the image recognition model includes the following steps.

S500: Train a third image recognition model using an RGB image set to obtain a second image recognition model.

S501: Train the second image recognition model through a normalized exponential softmax loss function to obtain a first image recognition model.

S502: Generate triplet sample pairs.

S503: Train a first image recognition model.

S504: Iterate optimization and return to step S502.

S505: Determine whether the loss value reaches a preset threshold, if so, execute step S506, otherwise execute step S504.

S506: Obtain a target image recognition model.

Referring to Fig. 6, Fig. 6 is a schematic diagram showing a process of selecting a triplet sample pair according to an exemplary embodiment of the present disclosure. As shown in Fig. 6, the process of the method for selecting a triplet sample pair includes the following steps.

S600: extract multiple image sets through the first image recognition model, and execute step S601 and step S603 respectively.

S601: Clustering multiple images corresponding to each image set.

S602: Select positive sample pairs within clusters in different classes and execute step S606.

S603: Calculate the central feature of each image set.

S604: Clustering the central features of all image sets.

S605: Select negative sample pairs within the same inter-class cluster and execute step S606.

S606: Construct triplet sample pairs.

The specific implementation methods of the above-mentioned process steps have been illustrated in detail above and will not be repeated here. In addition, it should be understood that for the above-mentioned system embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited to the order of actions described above. Secondly, those skilled in the art should also know that the embodiments described above belong to preferred embodiments, and the steps involved are not necessarily required by the present disclosure.

Through the above technical solution, by selecting images with lower similarity in different intra-class clusters to form positive sample pairs, the situation of generating invalid positive sample pairs due to selecting the same image with higher similarity can be reduced. By selecting different images in the same inter-class cluster to form negative sample pairs, the situation of generating invalid negative sample pairs due to selecting different images with lower similarity can be reduced. Thus, according to the positive sample pairs, negative sample pairs, and image sets, a triplet sample pair is generated to train the first image recognition model, which can improve the convergence stability and recognition accuracy of the first image recognition model. And the problem of oscillation in the training of the first image recognition model caused by randomly selecting positive sample pairs and negative sample pairs can be reduced.

Based on the same concept, this embodiment further provides a sample generating device 700. Referring to FIG. 7 , FIG. 7 is a schematic diagram showing a sample generating device 700 according to an exemplary embodiment of the present disclosure. As shown in FIG. 7 , the sample generating device 700 includes:

A first determining module 701 is used to determine a plurality of image sets, each of which includes a plurality of images corresponding to the same identity information;

A first selection module 702 is used to cluster multiple images corresponding to each of the image sets to obtain multiple intra-class clusters, and select images from different intra-class clusters to form positive sample pairs, wherein the similarity between features corresponding to different images in the same intra-class cluster is less than a first similarity threshold;

A second selection module 703 is used to cluster the multiple image sets to obtain multiple inter-class clusters, and select images corresponding to different image sets in the same inter-class cluster to form negative sample pairs, wherein the similarity between the features corresponding to different image sets in the same inter-class cluster is less than a second similarity threshold;

The generating module 704 is configured to generate triplet sample pairs according to the multiple image sets, the positive sample pairs and the negative sample pairs.

Optionally, the first selection module 702 includes:

A first calculation module, used for calculating the similarity between every two images in the image set;

A first connection module, configured to regard the images in the image set as nodes, and connect the nodes corresponding to the two images whose similarity is less than the first similarity threshold, to obtain a first undirected graph;

The second determining module is used to determine the image corresponding to each subgraph in the first undirected graph as an intra-cluster of a class to obtain multiple intra-clusters.

Optionally, the first selection module 702 includes:

A first selection submodule is used to randomly select a first image from a first in-class cluster and randomly select a second image from a second in-class cluster, wherein the first in-class cluster and the second in-class cluster are different in-class clusters among the multiple in-class clusters;

The first forming module is used to form a positive sample pair with the first image and the second image.

Optionally, the second selection module 703 includes:

A second calculation module is used to calculate the similarity between the central features of every two image sets in the multiple image sets, where the central feature of the image is the image in the image set that meets a preset condition;

A second connection module, configured to regard the central features of the image sets as nodes, and connect the nodes corresponding to the central features of two image sets whose similarity is less than the second similarity threshold, to obtain a second undirected graph;

The third determination module is used to determine the image set corresponding to each subgraph in the second undirected graph as an between-class cluster to obtain the multiple between-class clusters.

Optionally, the second selection module 703 includes:

A second selection submodule is used to randomly select a third image and a fourth image from the same between-class cluster, wherein the third image and the fourth image are images corresponding to different image sets;

The second forming module is used to form a negative sample pair with the third image and the fourth image.

Optionally, the second selection module 703 includes:

A third sub-selection module is used for randomly selecting a fifth image from each image set in the same between-class cluster;

a fourth sub-selection module, configured to randomly select a sixth image from each of the multiple intra-class clusters of the remaining image set, wherein the remaining image set is the image set remaining after excluding the image set in the same inter-class cluster;

The third forming module is used to form the negative sample pair with the fifth image and the sixth image.

Optionally, the first determining module is further configured to:

Determine the image set collected by the vehicle-mounted image acquisition device.

Based on the same concept, this embodiment also discloses a training device for an image recognition model, the training device comprising:

The first training module is used to train the first image recognition model according to the triplet sample pairs generated by the sample generation method disclosed in this embodiment.

Optionally, the training device further comprises:

The second training module is used to obtain a vehicle-mounted image set in a vehicle-mounted image acquisition device, and train a second image recognition model based on the vehicle-mounted image set through a normalized exponential softmax loss function to obtain the first image recognition model.

Optionally, the training device further comprises:

Image acquisition module, used to obtain RGB image sets;

An image processing module, used for performing grayscale processing on the RGB image set to obtain a grayscale image set;

The third training module is used to pre-train the third image recognition model according to the grayscale image set to obtain the second image recognition model.

Based on the same concept, this embodiment also provides a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the sample generation method and the image recognition model training method provided in this embodiment are implemented.

Based on the same concept, this embodiment also provides a controller, including:

a memory having a computer program stored thereon;

The processor is used to execute the computer program in the memory to implement the steps of the sample generation method and the image recognition model training method provided in this embodiment.

In another exemplary embodiment, a controller is also provided. The controller may be an integrated circuit (IC) or a chip, wherein the integrated circuit may be an IC or a collection of multiple ICs; the chip may include but is not limited to the following types: GPU (Graphics Processing Unit), CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), SOC (System on Chip, SoC), etc. The above-mentioned integrated circuit or chip can be used to execute executable instructions (or codes) to implement the above-mentioned sample generation method and image recognition model training method. The executable instructions can be stored in the integrated circuit or chip, or can be obtained from other devices or equipment, for example, the integrated circuit or chip includes a processor, a memory, and an interface for communicating with other devices. The executable instructions can be stored in the memory, and when the executable instructions are executed by the processor, the above-mentioned sample generation method and image recognition model training method are implemented; alternatively, the integrated circuit or chip can receive the executable instructions through the interface and transmit them to the processor for execution, so as to implement the above-mentioned sample generation method and image recognition model training method.

Based on the same concept, this embodiment also provides a vehicle, including the controller provided by this embodiment.

8 is a block diagram of a vehicle 800 according to an exemplary embodiment. For example, the vehicle 800 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicles. The vehicle 800 may be an autonomous vehicle or a semi-autonomous vehicle.

8 , vehicle 800 may include various subsystems, such as an infotainment system 810, a perception system 820, a decision control system 830, a drive system 840, and a computing platform 850. Vehicle 800 may also include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and each component of vehicle 800 may be interconnected by wire or wireless means.

In some embodiments, the infotainment system 810 may include a communication system, an entertainment system, and a navigation system, etc.

The perception system 820 may include several sensors for sensing information about the environment around the vehicle 800. For example, the perception system 820 may include a global positioning system (the global positioning system may be a GPS system, or a Beidou system or other positioning systems), an inertial measurement unit (IMU), a laser radar, a millimeter wave radar, an ultrasonic radar, and a camera.

The decision control system 830 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

The drive system 840 may include components that provide powered motion for the vehicle 800. In one embodiment, the drive system 840 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of an internal combustion engine, an electric motor, and an air compression engine. The engine is capable of converting energy provided by the energy source into mechanical energy.

Some or all functions of the vehicle 800 are controlled by a computing platform 850. The computing platform 850 may include at least one processor 851 and a memory 852, and the processor 851 may execute instructions 853 stored in the memory 852.

The processor 851 may be any conventional processor, such as a commercially available CPU. The processor may also include a graphics processor (Graphic Process Unit, GPU), a field programmable gate array (Field Programmable Gate Array, FPGA), a system on chip (System on Chip, SOC), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) or a combination thereof.

The memory 852 may be implemented by any type of volatile or nonvolatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

In addition to instructions 853 , memory 852 may also store data, such as road maps, route information, vehicle location, direction, speed, etc. The data stored in memory 852 may be used by computing platform 850 .

In the embodiment of the present disclosure, the processor 851 can execute instruction 853 to complete all or part of the steps of the above-mentioned sample generation method and image recognition model training method.

In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned sample generation method and the training method of the image recognition model are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 852 including program instructions, and the above-mentioned program instructions can be executed by the processor 851 of the vehicle 800 to complete the above-mentioned sample generation method and the training method of the image recognition model. In another exemplary embodiment, a computer program product is also provided, which includes a computer program that can be executed by a programmable device, and the computer program has a code portion for executing the above-mentioned sample generation method and the training method of the image recognition model when executed by the programmable device.

In another exemplary embodiment, a computer program product is also provided, which includes a computer program that can be executed by a programmable device, and the computer program has a code portion for executing the above-mentioned sample generation method and image recognition model training method when executed by the programmable device.

The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims (14)

1. A method of generating a sample, comprising:

Determining a plurality of image sets, wherein each image set comprises a plurality of images corresponding to the same identity information;

clustering a plurality of images corresponding to each image set to obtain a plurality of intra-class clusters, and selecting images from different intra-class clusters to form positive sample pairs, wherein the similarity between features corresponding to different images in the same intra-class cluster is smaller than a first similarity threshold;

Clustering the plurality of image sets to obtain a plurality of inter-class clusters, and selecting images corresponding to different image sets from the same inter-class cluster to form a negative sample pair, wherein the similarity between the features corresponding to the different image sets in the same inter-class cluster is smaller than a second similarity threshold;

and generating a triplet sample pair according to the plurality of image sets, the positive sample pair and the negative sample pair.

2. The method for generating a sample according to claim 1, wherein clustering the plurality of images corresponding to the image set to obtain a plurality of intra-class clusters comprises:

calculating the similarity between every two images in the image set;

Taking the images in the image set as nodes, and connecting the nodes corresponding to the two images with the similarity smaller than the first similarity threshold to obtain a first undirected graph;

and determining the image corresponding to each sub-graph in the first undirected graph as an intra-class cluster to obtain a plurality of intra-class clusters.

3. The method of claim 1, wherein selecting images in different of the intra-class clusters forms positive sample pairs, comprising:

Randomly selecting a first image from a first class inner cluster and randomly selecting a second image from a second class inner cluster, wherein the first class inner cluster and the second class inner cluster are different class inner clusters in the plurality of class inner clusters;

the first image and the second image are formed into a positive sample pair.

4. The method of generating samples according to claim 1, wherein the clustering the plurality of image sets to obtain a plurality of inter-class clusters includes:

Calculating the similarity between the central characteristics of each two image sets in the plurality of image sets, wherein the central characteristics of the image sets are images which reach preset conditions in the image sets;

Taking the central characteristics of the image sets as nodes, and connecting the nodes corresponding to the central characteristics of the two image sets with the similarity smaller than the second similarity threshold value to obtain a second undirected graph;

And determining the image set corresponding to each sub-graph in the second undirected graph as an inter-class cluster to obtain the plurality of inter-class clusters.

5. The method of generating samples according to claim 1, wherein selecting images corresponding to different image sets in the same inter-class cluster forms a negative pair of samples, comprising:

Randomly selecting a third image and a fourth image from the same inter-class cluster, wherein the third image and the fourth image are images corresponding to different image sets;

and forming a negative sample pair by the third image and the fourth image.

6. The method of generating samples according to claim 1, wherein selecting images corresponding to different image sets in the same inter-class cluster forms a negative pair of samples, comprising:

randomly selecting a fifth image from the image sets for each image set in the same inter-class cluster;

randomly selecting a sixth image from each of a plurality of intra-class clusters of a residual image set, wherein the residual image set is a residual image set obtained by dividing the image set in the same inter-class cluster;

and forming the negative sample pair by the fifth image and the sixth image.

7. The sample generation method of any of claims 1-6, wherein the determining a plurality of image sets comprises:

and determining a plurality of image sets acquired by the vehicle-mounted image acquisition device.

8. A method of training an image recognition model, the method comprising:

A pair of triplets of samples generated by the sample generation method of any of claims 1-7, training the first image recognition model.

9. The method of training an image recognition model of claim 8, wherein prior to training the first image recognition model, the method further comprises:

And acquiring a vehicle-mounted image set in the vehicle-mounted image acquisition device, and training a second image recognition model through a normalized index softmax loss function according to the vehicle-mounted image set to obtain the first image recognition model.

10. The method of training an image recognition model of claim 9, further comprising, prior to training a second image recognition model with a normalized exponential softmax loss function to obtain the first image recognition model:

acquiring an RGB image set;

gray processing is carried out on the RGB image set to obtain a gray image set;

And pre-training a third image recognition model according to the gray level image set to obtain the second image recognition model.

11. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that the program when executed by a processor realizes the steps of the method according to any of claims 1-10.

12. A controller, comprising:

A memory having a computer program stored thereon;

a processor for executing the computer program in the memory to implement the steps of the method of any one of claims 1-10.

13. A vehicle comprising the controller of claim 12.

14. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any one of claims 1-10.