CN115731530A - Model training method and device - Google Patents

Abstract

The application discloses a model training method, which comprises the following steps: sampling a sample image to obtain a first image block and a second image block, respectively extracting the features of the first image block and the second image block through a feature extraction network based on the fact that the cross-over ratio of the areas of the first image block and the second image block on the sample image is larger than a threshold value, determining loss according to the difference between an obtained first feature map and an obtained second feature map, and updating the feature extraction network based on the loss. According to the method, the cross-over ratio is used as a mode for measuring the relative distance between the two image blocks, the semantic consistency in the local area is guaranteed by requiring the cross-over ratio of the image blocks to be larger than a given threshold value so as to restore the basic assumption of contrast learning, the image blocks are limited in one local area, therefore, the global inconsistency is converted into the local consistency, and the data processing accuracy of the updated feature extraction network is further improved.

Description

A model training method and device thereof

technical field

The present application relates to the field of artificial intelligence, in particular to a model training method and device thereof.

Background technique

Computer vision is an integral part of various intelligent/autonomous systems in various application fields such as manufacturing, inspection, document analysis, medical diagnosis, and military. What we need is the knowledge of the data and information of the subject being photographed. Figuratively speaking, it is to install eyes (camera or video camera) and brain (algorithm) on the computer to replace the human eye to identify, track and measure the target, so that the computer can perceive the environment. Because perception can be thought of as extracting information from sensory signals, computer vision can also be thought of as the science of how to make artificial systems "perceive" from images or multidimensional data. In general, computer vision is to use various imaging systems to replace the visual organs to obtain input information, and then use the computer to replace the brain to complete the processing and interpretation of these input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

The perceptual network can be a neural network model that processes and analyzes images and obtains processing results. Currently, the perceptual network can perform more and more functions, such as image classification, 2D detection, semantic segmentation, key point detection, Linear object detection (such as lane line or stop line detection in autonomous driving technology), drivable area detection, etc. In addition, the visual perception system has the characteristics of low cost, non-contact, small size and large amount of information. With the continuous improvement of the accuracy of visual perception algorithms, it has become the key technology of many artificial intelligence systems today, and has been more and more widely used, such as: advanced driving assistant system (advanced driving assistant system, ADAS) and automatic driving system (autonomous In the driving system (ADS), the recognition of dynamic obstacles (people or cars) and static objects (traffic lights, traffic signs or traffic cones) on the road surface, through the identification of human body masks in the camera beauty function of terminal vision Mask and key points to achieve slimming effect, etc.

In recent years, a major technological breakthrough has been made in self-supervised representation learning based on contrastive learning. The basic assumption is that image blocks from the same image describe the same semantic information, and their features should be as close as possible, while image blocks from different images describe different semantic information, so their features should be as dissimilar as possible. Although this method does not require manual annotation, it still requires an additional single-instance assumption, which requires that there is only one instance in each image, and this instance occupies the central main part of the image. In this way, the image blocks cut from the same image can be regarded as positive samples, while the image blocks cut from different images are negative samples. However, in autonomous driving scenarios, street view images often contain multiple instances such as pedestrians and vehicles, which have strong global inconsistency and do not meet the above single-instance assumptions. Therefore, existing methods are often limited to datasets with single-instance assumptions. , it is difficult to make full use of more realistic multi-instance autonomous driving datasets.

Contents of the invention

In a first aspect, the present application provides a model training method, the method comprising:

Obtain a sample image; the sample image can be a multi-instance image (such as a street view image) in the above-mentioned automatic driving scene. In a possible implementation, the sample image can include multiple objects, and the objects include people, vehicles, traffic At least one of a sign, a lane marking, a plant. Exemplarily, the objects may include but not limited to at least one of the following examples: dynamic obstacles (pedestrians, cyclists (Cyclist), tricycles (Tricycle), cars (Car), trucks (Truck), buses ( Bus)), static obstacles (TrafficCone, TrafficStick, Fire Hydrant, Motocycle, Bicycle), traffic signs (TrafficSign), guide signs ( GuideSign), Billboard, Red Traffic Light (TrafficLight_Red) / Yellow Traffic Light (TrafficLight_Yellow) / Green Traffic Light (TrafficLight_Green) / Black Traffic Light (TrafficLight_Black), Road Sign (RoadSign)).

Sampling the sample image to obtain a first image block and a second image block, where the first image block and the second image block are different image blocks on the sample image;

Wherein, after the sample image is acquired, the sample image can be sampled to obtain a plurality of partial images (such as a first image block and a second image block), and the first image block and the second image block can be small samples obtained by sampling the sample image. image (here small refers to the image area of the first image block and the second image block being smaller relative to the size of the sample image).

Wherein, the above-mentioned sampling can be understood as cropping. In one sampling process, a rectangular area can be randomly determined on the sample image, and the image in the determined rectangular area can be used as the first image block. In another sampling process, it can be A rectangular area is randomly determined on the sample image, and an image within the determined rectangular area is used as a second image block.

Wherein, the above-mentioned sampling may be random sampling, and the so-called random sampling may be understood as that a sampling position and/or a sampling size are randomly determined.

Wherein, the first image block and the second image block may be rectangular images.

Based on the intersection ratio of the first image block and the second image block on the sample image where the area is greater than a threshold, the first image block and the second image block are respectively analyzed through a feature extraction network Perform feature extraction to obtain a first feature map and a second feature map;

Wherein, there may be a common area (overlapping area) between the area where the first image block is located on the sample image and the area where the second image block is located on the sample image, the area of the common area is the same as that of the first image block and the second image block The ratio between all areas occupied by the image blocks on the sample image is an intersection over union (IoU) between the areas where the first image block and the second image block are located on the sample image;

In a possible implementation, the threshold may be a value greater than or equal to 0.4, for example, the threshold may be 0.4, 0.45, 0.5, 0.55 and so on. It should be understood that the threshold value of the intersection ratio cannot be set too low, but at the same time, it is not as high as possible, because it is not expected that the image blocks are irrelevant, but they are not expected to be exactly the same, so the selection of the threshold value of the intersection ratio It is actually also controlling the balance between data noise and data complexity in multi-instance self-supervised learning.

A loss is determined according to the difference between the first feature map and the second feature map, and the feature extraction network is updated based on the loss to obtain an updated feature extraction network.

This application uses the intersection and union ratio as a way to measure the relative distance between two image blocks. By requiring the intersection and union ratio of the image block to be greater than a given threshold to ensure the semantic consistency in the local area, the basic assumption of contrastive learning is restored, and the image block is limited to a local area. Inside the region, the global inconsistency is transformed into local consistency, thereby improving the data processing accuracy of the updated feature extraction network. At the same time, the choice of threshold can effectively control the balance between data noise and data complexity in different scenarios.

In a possible implementation, the threshold is a value greater than or equal to 0.4.

In a possible implementation, before determining the loss according to the difference between the first feature map and the second feature map, the method further includes: analyzing the first image block and the second feature map Two image blocks are aligned to obtain an aligned first image block and an aligned second image block; said determining the loss according to the difference between the first feature map and the second feature map includes: A loss is determined based on the difference between the aligned first feature map and the aligned second feature map.

In a possible implementation, the sample image includes a target area, where the target area is an overlapping area where the first image block and the second image block are located on the sample image, and the pair of Aligning the first image block with the second image block includes: determining, according to the target area, a first sub-feature map corresponding to the target area in the first feature map and a sub-feature map in the second feature map A second sub-feature map corresponding to the target area; performing upsampling on the first sub-feature map to obtain the aligned first image block; performing up-sampling on the second sub-feature map to obtain The aligned second image block is obtained, wherein the aligned first image block has the same size as the aligned second image block.

In order to distinguish multi-instance features, it is necessary to abandon the global pooling layer after the backbone network to maintain the two-dimensional structure and position information of the feature map, but it also brings additional features that are not aligned. One-to-one correspondence with the same relative position. In the embodiment of the present application, two different ways are provided for feature alignment: region of interest (region of interest) alignment uses the overlapping parts of the image blocks as the regions of interest of the two image blocks, for example, RoI Align can be used to extract only the features of the overlapping part For subsequent calculations, this method is intuitive but does not make full use of the information of the non-overlapping parts.

In a possible implementation, the first feature map and the second feature map have the same size, the first feature map includes M first feature points, and the second feature map includes M first feature points Two feature points, the M first feature points correspond to M first pixel points in the sample image, and the M second feature points correspond to M second pixel points in the sample image, The M first pixels are in one-to-one correspondence with the M second pixels, and the method further includes: according to the M first pixels and the M second pixels, to obtain a third A feature map, the size of the third feature map is the same as that of the second feature map, the third feature map includes M third feature points, and each third feature point is based on the first pixel with a corresponding relationship The pixel position difference between the point and the second pixel point is obtained; the third feature map is fused with the first feature map to obtain the aligned first image block, and the second feature map is used as the aligned second image block.

In a possible implementation, the fusing the third feature map with the first feature map includes:

Splicing the third feature map and the first feature map in a depth direction.

Among them, the displacement alignment takes each pair of pixels at the same relative position, calculates their coordinate displacement in the original image and concatenates them with the feature map (stitching in the depth direction), and provides them as additional side information to the prediction network for implicit The feature alignment to help the subsequent feature prediction makes full use of the feature information of the non-overlapping area, and at the same time makes the subsequent similarity measurement more flexible.

In a possible implementation, the determining the loss according to the difference between the first feature map and the second feature map includes: using a target prediction network to perform a A feature point is processed to obtain the predicted value of each first feature point; based on the target clustering algorithm, the M second feature points of the first feature map are clustered to update the M second feature points The feature value of the feature point, wherein the feature value of each updated second feature point is the cluster center feature value of the cluster category where it is located; according to the predicted value of each first feature point and each update After the difference between the eigenvalues of the second eigenpoints, the loss is determined.

Among them, the embodiment of the present application uses image clustering. On the one hand, the network has the ability to distinguish the characteristics of different instances; Deploying a self-attention module introduces a global perspective to obtain more accurate prediction results.

In a possible implementation, the method also includes:

Based on the loss, the target prediction network is updated.

In a possible implementation, the sample image includes multiple objects, and the objects include at least one of people, vehicles, traffic signs, lane lines, and plants.

In a possible implementation, the method also includes:

Obtaining a target network and an image to be processed, the target network including the updated feature extraction network and a downstream task network; processing the image to be processed through the target network to obtain a processing result.

In a second aspect, the present application provides a model training device, the device comprising:

An acquisition module, used to acquire sample images;

a sampling module, configured to sample the sample image to obtain a first image block and a second image block, the first image block and the second image block being different image blocks on the sample image;

A feature extraction module, configured to extract the first image block and the second image block respectively through a feature extraction network based on the intersection ratio between the areas where the first image block and the second image block are located on the sample image is greater than a threshold performing feature extraction on the second image block to obtain a first feature map and a second feature map;

A model updating module, configured to determine a loss according to the difference between the first feature map and the second feature map, and update the feature extraction network based on the loss to obtain an updated feature extraction network.

In a possible implementation, the threshold is a value greater than or equal to 0.4.

In a possible implementation, the device also includes:

an alignment module, configured to align the first image block and the second image block before determining the loss according to the difference between the first feature map and the second feature map, so as to obtain the aligned first image block and the aligned second image block;

The model update module is specifically used for:

A loss is determined based on the difference between the aligned first feature map and the aligned second feature map.

In a possible implementation, the sample image includes a target area, where the target area is an overlapping area where the first image block and the second image block are located on the sample image, and the alignment module, Specifically for:

According to the target area, determining a first sub-feature map corresponding to the target area in the first feature map and a second sub-feature map corresponding to the target area in the second feature map;

Upsampling the first sub-feature map to obtain the aligned first image block;

Up-sampling the second sub-feature map to obtain the aligned second image block, wherein the aligned first image block has the same size as the aligned second image block.

In a possible implementation, the first feature map and the second feature map have the same size, the first feature map includes M first feature points, and the second feature map includes M first feature points Two feature points, the M first feature points correspond to M first pixel points in the sample image, and the M second feature points correspond to M second pixel points in the sample image, The M first pixels are in one-to-one correspondence with the M second pixels, and the alignment module is specifically used for:

According to the M first pixel points and the M second pixel points, a third feature map is obtained, the size of the third feature map is the same as that of the second feature map, and the third feature map Including M third feature points, each third feature point is obtained based on the pixel position difference between the first pixel point and the second pixel point with corresponding relationship;

The third feature map is fused with the first feature map to obtain the aligned first image block, and the second feature map is used as the aligned second image block.

In a possible implementation, the alignment module is specifically used for:

Splicing the third feature map and the first feature map in a depth direction.

In a possible implementation, the model updating module is specifically used for:

Processing the M first feature points of the first feature map through the target prediction network to obtain a predicted value of each first feature point;

Based on the target clustering algorithm, cluster the M second feature points of the first feature map to update the feature values of the M second feature points, wherein the feature of each updated second feature point The value is the cluster center feature value of the cluster category;

The loss is determined according to the difference between the predicted value of each first feature point and the feature value of each updated second feature point.

In a possible implementation, the model update module is also used to:

Based on the loss, the target prediction network is updated.

In a possible implementation, the sample image includes multiple objects, and the objects include at least one of people, vehicles, traffic signs, lane lines, and plants.

In a possible implementation, the device also includes:

A data processing module, configured to acquire a target network and an image to be processed, the target network including the updated feature extraction network and a downstream task network;

The image to be processed is processed through the target network to obtain a processing result.

In the third aspect, the embodiment of the present application provides a model training device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to perform the above-mentioned first aspect. And any optional method of the first aspect.

In the fourth aspect, the embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when it is run on a computer, the computer executes the above-mentioned first aspect and the first Aspect any optional method.

In the fifth aspect, the embodiment of the present application provides a computer program, which, when running on a computer, causes the computer to execute the above-mentioned first aspect and any optional method thereof.

In a sixth aspect, the present application provides a chip system, which includes a processor, configured to support an execution device or a training device to implement the functions involved in the above aspect, for example, send or process the data involved in the above method; or, information. In a possible design, the chip system further includes a memory, and the memory is used for storing necessary program instructions and data of the execution device or the training device. The system-on-a-chip may consist of chips, or may include chips and other discrete devices.

An embodiment of the present application provides a model training method, the method comprising: acquiring a sample image; sampling the sample image to obtain a first image block and a second image block, the first image block and the second image block The image block is a different image block on the sample image; based on the union ratio between the first image block and the second image block on the sample image where the area is greater than a threshold, the feature extraction network is respectively used for performing feature extraction on the first image block and the second image block to obtain a first feature map and a second feature map; determining a loss according to the difference between the first feature map and the second feature map , and update the feature extraction network based on the loss to obtain an updated feature extraction network. Using the intersection ratio as a measure of the relative distance between two image blocks, by requiring the image block intersection ratio to be greater than a given threshold to ensure semantic consistency in the local area to restore the basic assumption of contrastive learning, the image block is limited to a local area , so as to transform the global inconsistency into local consistency, and then improve the data processing accuracy of the updated feature extraction network. At the same time, the choice of threshold can effectively control the balance between data noise and data complexity in different scenarios.

Description of drawings

Fig. 1 is a kind of structural schematic diagram of main frame of artificial intelligence;

Fig. 2 is the application scenario of the embodiment of the present application;

FIG. 3 is a schematic diagram of the structure of the convolutional neural network used in the embodiment of the present application;

FIG. 4 is a schematic structural representation of the convolutional neural network used in the embodiment of the present application;

FIG. 5 is a schematic diagram of a system architecture provided by an embodiment of the present application;

FIG. 6 is a schematic diagram of a model training method provided in an embodiment of the present application;

FIG. 7 is a schematic diagram of a parallel ratio calculation provided by the embodiment of the present application;

FIG. 8 is a schematic diagram of an image block sampling provided by an embodiment of the present application;

FIG. 9 is a schematic diagram of an image block sampling provided by an embodiment of the present application;

FIG. 10 is a schematic diagram of an image block sampling provided by an embodiment of the present application;

FIG. 11 is a schematic diagram of an image block sampling provided by an embodiment of the present application;

FIG. 12 is a schematic structural representation of a backbone network;

FIG. 13 is a schematic diagram of image block alignment in an embodiment of the present application;

FIG. 14 is a schematic diagram of image block alignment in an embodiment of the present application;

FIG. 15 is a schematic diagram of clustering in the embodiment of the present application;

FIG. 16 is a schematic diagram of a similarity calculation in an embodiment of the present application;

Fig. 17 is a schematic diagram of a model training method provided by the embodiment of the present application;

Fig. 18 is a schematic diagram of a model training method provided by the embodiment of the present application;

Fig. 19 is a structural representation of a downstream task network;

Fig. 20 is a structural representation of a head;

Figure 21 is a schematic diagram of a clustering result;

Fig. 22 is a schematic diagram of a model training device provided by the embodiment of the present application;

Fig. 23 is a schematic structural diagram of the execution device provided by the embodiment of the present application;

Fig. 24 is a schematic structural diagram of a training device provided by an embodiment of the present application;

FIG. 25 is a schematic structural diagram of a chip provided by an embodiment of the present application.

Detailed ways

Embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention. The terms used in the embodiments of the present invention are only used to explain specific examples of the present invention, and are not intended to limit the present invention.

Embodiments of the present application are described below in conjunction with the accompanying drawings. Those of ordinary skill in the art know that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

The terms "first", "second" and the like in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, and this is merely a description of the manner in which objects with the same attribute are described in the embodiments of the present application. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, product, or apparatus comprising a series of elements is not necessarily limited to those elements, but may include elements not expressly included. Other elements listed explicitly or inherent to the process, method, product, or apparatus.

First, describe the overall workflow of the artificial intelligence system. Please refer to Figure 1. Figure 1 shows a schematic structural diagram of the main framework of artificial intelligence. The following is from the "intelligent information chain" (horizontal axis) and "IT value chain" ( Vertical axis) to illustrate the above artificial intelligence theme framework in two dimensions. Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensed process of "data-information-knowledge-wisdom". "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of artificial intelligence, information (provided and processed by technology) to the systematic industrial ecological process.

(1) Infrastructure

The infrastructure provides computing power support for the artificial intelligence system, realizes communication with the outside world, and realizes support through the basic platform. Communicate with the outside through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing framework and network and other related platform guarantees and supports, which can include cloud storage and Computing, interconnection network, etc. For example, sensors communicate with the outside to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

(2) data

Data from the upper layer of the infrastructure is used to represent data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional equipment, including business data of existing systems and sensory data such as force, displacement, liquid level, temperature, and humidity.

(3) Data processing

Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

Among them, machine learning and deep learning can symbolize and formalize intelligent information modeling, extraction, preprocessing, training, etc. of data.

Reasoning refers to the process of simulating human intelligent reasoning in a computer or intelligent system, and using formalized information to carry out machine thinking and solve problems according to reasoning control strategies. The typical functions are search and matching.

Decision-making refers to the process of decision-making after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

(4) General ability

After the above-mentioned data processing is performed on the data, some general capabilities can be formed based on the results of data processing, such as algorithms or a general system, such as translation, text analysis, computer vision processing, speech recognition, image processing identification, etc.

(5) Smart products and industry applications

Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. It is the packaging of the overall solution of artificial intelligence, which commercializes intelligent information decision-making and realizes landing applications. Its application fields mainly include: intelligent terminals, intelligent transportation, Smart healthcare, autonomous driving, smart cities, etc.

The following is a brief introduction to application scenarios such as ADAS/ADS visual perception system, mobile phone beautification, image classification, and product classification.

Application Scenario 1: ADAS/ADS Visual Perception System

As shown in Figure 2, in ADAS and ADS, real-time multi-type 2D target detection is required, including: dynamic obstacles (pedestrians, cyclists, tricycles, cars, trucks) (Truck, Bus (Bus)), static obstacles (Traffic Cone, Traffic Stick, Fire Hydrant, Motorcycle, Bicycle), traffic signs ( (TrafficSign), Guide Signs (GuideSign), Billboards (Billboard), Red Traffic Lights (TrafficLight_Red)/Yellow Traffic Lights (TrafficLight_Yellow)/Green Traffic Lights (TrafficLight_Green)/Black Traffic Lights (TrafficLight_Black), Road Signs (RoadSign)). In addition, in order to accurately obtain the area occupied by the dynamic obstacle in the 3D space, it is also necessary to perform 3D estimation on the dynamic obstacle and output a 3D frame. In order to integrate with the laser radar data, it is necessary to obtain the Mask of the dynamic obstacle, so as to filter out the laser point cloud hit on the dynamic obstacle; in order to carry out accurate parking space, it is necessary to detect 4 key points of the parking space at the same time ; In order to perform composition positioning, it is necessary to detect key points of static objects. Using the model trained by the technical solution provided by the embodiment of the present application (for example, the target network), all or part of the above-mentioned functions can be completed in the target network.

Application Scenario 2: Mobile phone beauty function

In the mobile phone, the model (such as the target network) trained by the technical solution provided by the embodiment of the present application detects the Mask and key points of the human body, and can zoom in and out on the corresponding parts of the human body, such as performing waist and buttock enhancement operations, thereby Output beautiful images.

Application scenario 3: Image classification scenario:

After obtaining the image to be classified, the model (such as the target network) trained by the technical solution provided by the embodiment of the application can determine the category of the object in the image to be classified, and then it can be classified according to the category of the object in the image to be classified Images are classified. For photographers, many photos are taken every day, including animals, people, and plants. Using the method of the present application, photos can be quickly classified according to the content in the photos, which can be divided into photos containing animals, photos containing people and photos containing plants.

Application Scenario 4: Commodity Classification:

After the image containing the commodity is acquired, the category of the commodity in the image of the commodity can be determined by using the model (such as the target network) trained by the technical solution provided by the embodiment of the present application, and then the commodity is classified according to the category of the commodity. For a wide variety of goods in large shopping malls or supermarkets.

Since the embodiment of the present application involves the application of a large number of neural networks, for ease of understanding, the following first introduces related terms and neural network related concepts involved in the embodiment of the present application.

(1) neural network

The neural network can be composed of neural units, and the neural unit can refer to an operation unit that takes xs (ie input data) and intercept 1 as input, and the output of the operation unit can be:

Wherein, s=1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple above-mentioned single neural units, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected with the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

(2) Object detection, using image processing and related methods such as machine learning and computer graphics, object detection can determine the category of image objects and determine the detection frame used to locate the object.

(3) Convolutional neural network (CNN) is a deep neural network with a convolutional structure. The convolutional neural network contains a feature extractor composed of a convolutional layer and a subsampling layer, which can be regarded as a filter. The convolutional layer refers to the neuron layer that performs convolution processing on the input signal in the convolutional neural network. In the convolutional layer of a convolutional neural network, a neuron can only be connected to some adjacent neurons. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. Neural units of the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract features independent of position. The convolution kernel can be formalized as a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

CNN is a very common neural network. The structure of CNN will be introduced in detail below in conjunction with Figure 3. As mentioned in the introduction to the basic concepts above, the convolutional neural network is a deep neural network with a convolutional structure, and it is a deep learning architecture. The deep learning architecture refers to the algorithm through machine learning. Multiple levels of learning are performed on the abstraction level. As a deep learning architecture, CNN is a feed-forward artificial neural network in which individual neurons can respond to images input into it.

As shown in FIG. 3 , a convolutional neural network (CNN) 200 may include an input layer 210 , a convolutional/pooling layer 220 (where the pooling layer is optional), and a fully connected layer 230 .

Convolutional layer/pooling layer 220:

Convolution layer:

As shown in Figure 3, the convolutional layer/pooling layer 220 may include layers 221-226 as examples, for example: in one implementation, the 221st layer is a convolutional layer, the 222nd layer is a pooling layer, and the 223rd layer is a volume Layers, 224 are pooling layers, 225 are convolutional layers, and 226 are pooling layers; in another implementation, 221 and 222 are convolutional layers, 223 are pooling layers, and 224 and 225 are convolutional layers Layer, 226 is a pooling layer. That is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or it can be used as the input of another convolutional layer to continue the convolution operation.

The following will take the convolutional layer 221 as an example to introduce the inner working principle of one convolutional layer.

The convolution layer 221 may include many convolution operators, which are also called kernels, and their role in image processing is equivalent to a filter for extracting specific information from the input image matrix. The convolution operators are essentially It can be a weight matrix. This weight matrix is usually pre-defined. During the convolution operation on the image, the weight matrix is usually one pixel by one pixel (or two pixels by two pixels) along the horizontal direction on the input image. ...It depends on the value of the stride) to complete the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image. During the convolution operation, the weight matrix will be extended to The entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolutional output with a single depth dimension, but in most cases instead of using a single weight matrix, multiple weight matrices of the same size (row×column) are applied, That is, multiple matrices of the same shape. The output of each weight matrix is stacked to form the depth dimension of the convolution image, where the dimension can be understood as determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract image edge information, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to filter unwanted noise in the image. Do blurring etc. The multiple weight matrices have the same size (row×column), and the feature maps extracted by the multiple weight matrices of the same size are also of the same size, and then the extracted multiple feature maps of the same size are combined to form the convolution operation. output.

The weight values in these weight matrices need to be obtained through a lot of training in practical applications, and each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions .

When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (such as 221) often extracts more general features, which can also be referred to as low-level features; As the depth of the network 200 deepens, the features extracted by the later convolutional layers (such as 226) become more and more complex, such as features such as high-level semantics, and features with higher semantics are more suitable for the problem to be solved.

Pooling layer:

Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolutional layer. In the layers 221-226 as shown in 220 in Figure 3, one layer of convolutional layers can be followed by one layer The pooling layer can also be a multi-layer convolutional layer followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and/or a maximum pooling operator for sampling an input image to obtain an image of a smaller size. The average pooling operator can calculate the pixel values in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value within a specific range as the result of maximum pooling. Also, just like the size of the weight matrix used in the convolutional layer should be related to the size of the image, the operators in the pooling layer should also be related to the size of the image. The size of the image output after being processed by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average or maximum value of the corresponding sub-region of the image input to the pooling layer.

Fully connected layer 230:

After being processed by the convolutional layer/pooling layer 220, the convolutional neural network 200 is not enough to output the required output information. Because as mentioned earlier, the convolutional layer/pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (required class information or other relevant information), the convolutional neural network 200 needs to use the fully connected layer 230 to generate one or a group of outputs with the required number of classes. Therefore, the fully connected layer 230 may include multiple hidden layers (231, 232 to 23n as shown in FIG. Pre-trained, for example, the task type can include image recognition, image classification, image super-resolution reconstruction, etc...

After the multi-layer hidden layer in the fully connected layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240. The output layer 240 has a loss function similar to the classification cross entropy, and is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (as shown in Fig. 3, the propagation from 210 to 240 direction is forward propagation) is completed, the backpropagation (as shown in Fig. 3, the propagation from 240 to 210 direction is back propagation) will Start to update the weights and biases of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the output of the convolutional neural network 200 through the output layer and the ideal result.

It should be noted that the convolutional neural network 200 shown in FIG. 3 is only an example of a convolutional neural network. In a specific application, the convolutional neural network can also exist in the form of other network models. For example, only Including a part of the network structure shown in FIG. 3 , for example, the convolutional neural network used in the embodiment of the present application may only include an input layer 210 , a convolutional layer/pooling layer 220 and an output layer 240 .

It should be noted that the convolutional neural network 100 shown in FIG. 3 is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models, for example, as Multiple convolutional layers/pooling layers shown in FIG. 4 are parallelized, and the extracted features are input to the fully connected layer 230 for processing.

(4) Deep Neural Network

其中，

是输入向量，

是输出向量，

是偏移向量，W是权重矩阵(也称系数)，α()是激活函数。每一层仅仅是对输入向量

经过如此简单的操作得到输出向量

由于DNN层数多，则系数W和偏移向量

的数量也就很多了。这些参数在DNN中的定义如下所述：以系数W为例：假设在一个三层的DNN中，第二层的第4个神经元到第三层的第2个神经元的线性系数定义为

上标3代表系数W所在的层数，而下标对应的是输出的第三层索引2和输入的第二层索引4。总结就是：第L-1层的第k个神经元到第L层的第j个神经元的系数定义为

需要注意的是，输入层是没有W参数的。在深度神经网络中，更多的隐含层让网络更能够刻画现实世界中的复杂情形。理论上而言，参数越多的模型复杂度越高，“容量”也就越大，也就意味着它能完成更复杂的学习任务。训练深度神经网络的也就是学习权重矩阵的过程，其最终目的是得到训练好的深度神经网络的所有层的权重矩阵(由很多层的向量W形成的权重矩阵)。Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers, and there is no special metric for "many" here. According to the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in the middle are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, it is actually not complicated in terms of the work of each layer. In simple terms, it is the following linear relationship expression:

in,

is the input vector,

is the output vector,

Is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just the input vector

After such a simple operation to get the output vector

Due to the large number of DNN layers, the coefficient W and the offset vector

The number is also a lot. The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the fourth neuron of the second layer to the second neuron of the third layer is defined as

The superscript 3 represents the layer number of the coefficient W, and the subscript corresponds to the output third layer index 2 and the input second layer index 4. The summary is: the coefficient of the kth neuron of the L-1 layer to the jth neuron of the L layer is defined as

It should be noted that the input layer has no W parameter. In deep neural networks, more hidden layers make the network more capable of describing complex situations in the real world. Theoretically speaking, a model with more parameters has a higher complexity and a greater "capacity", which means that it can complete more complex learning tasks. Training the deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by the vector W of many layers).

(5) Loss function

In the process of training the deep neural network, because it is hoped that the output of the deep neural network is as close as possible to the value you really want to predict, you can compare the predicted value of the current network with the target value you really want, and then according to the difference between the two to update the weight vector of each layer of the neural network (of course, there is usually an initialization process before the first update, that is, to pre-configure parameters for each layer in the deep neural network), for example, if the predicted value of the network If it is high, adjust the weight vector to make it predict lower, and keep adjusting until the deep neural network can predict the real desired target value or a value very close to the real desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function (loss function) or objective function (objective function), which are used to measure the difference between the predicted value and the target value important equation. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. Then the training of the deep neural network becomes a process of reducing the loss as much as possible.

(6) Back propagation algorithm

The convolutional neural network can use the back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, passing the input signal forward until the output will generate an error loss, and updating the parameters in the initial super-resolution model by backpropagating the error loss information, so that the error loss converges. The backpropagation algorithm is a backpropagation movement dominated by error loss, aiming to obtain the parameters of the optimal super-resolution model, such as the weight matrix.

(7) Self-supervised learning (self-supervised learning): refers to the use of certain attributes of the data itself (such as rotation angle and image block distribution order) as the "label" of the data to do unsupervised pre-training in a supervised manner, Finally, we take the backbone network parameters as the initialization of network parameters for various downstream tasks. Here upstream refers to the pre-training process, while downstream refers to various practical visual problems. Among them, contrastive learning (contrastive learning) is a recent research hotspot. It constructs contrastive tasks by mining the consistency of the data itself, and believes that different image blocks from the same image should have similar features. The feature representation learned in this way It even achieves a transfer effect that surpasses supervised pre-training.

(7) Domain-specific pre-training: For downstream tasks in a specific domain, use the upstream data set of the same data domain for pre-training, so as to reduce the data difference between the upstream and downstream to reduce the downstream task. Difficulty of optimization when fine-tuning. In the present invention, the data sets used upstream and downstream are all multi-instance automatic driving data sets.

(9) Backbone: It is the basic network structure for feature extraction of data, and it is also the main learning object of pre-training. On the basis of the backbone network, we can deploy different dedicated network structures according to different tasks; the backbone network can be shared and migrated, but the dedicated network is often not migrated. For images, the commonly used backbone network is a convolutional neural network, and its final feature representation is a two-dimensional feature map (2D feature map). Existing comparative learning models often deploy an additional global pooling layer (global pooling layer), such as averaging in the spatial dimension, and processing the two-dimensional feature map into a one-dimensional feature vector (1D feature vector). In the present invention, the two-dimensional feature map is directly modeled by abandoning the global pooling layer.

Next, a more detailed architecture of the execution subject that executes the model training method in the embodiment of the present application is introduced.

The system architecture provided by the embodiment of the present application will be described in detail below with reference to FIG. 5 . FIG. 5 is a schematic diagram of a system architecture provided by an embodiment of the present application. As shown in FIG. 5 , the system architecture 500 includes an execution device 510 , a training device 520 , a database 530 , a client device 540 , a data storage system 550 and a data acquisition system 560 .

The execution device 510 includes a calculation module 511 , an I/O interface 512 , a preprocessing module 513 and a preprocessing module 514 . The calculation module 511 may include the target model/rule 501, and the preprocessing module 513 and the preprocessing module 514 are optional.

The data collection device 560 is used to collect training samples. The training samples may be image data, etc. In the embodiment of the present application, the training samples may be sample images. After collecting the training samples, the data collection device 560 stores these training samples in the database 530 .

The training device 520 can train the feature extraction network based on the training samples maintained in the database 530 to obtain the target model/rule 501 . In the embodiment of the present application, the target model/rule 501 may be an updated feature extraction network. It should be understood that the above-mentioned process of training the feature extraction network may be a pre-training process. After the updated feature extraction network is obtained, the updated feature extraction network may be fine-tuned for the target task in combination with the downstream task data set.

It should be noted that, in practical applications, the training samples maintained in the database 530 are not necessarily collected by the data collection device 560, and may also be received from other devices. In addition, it should be noted that the training device 520 does not necessarily perform the training of the target model/rule 501 based entirely on the training samples maintained by the database 530, and it is also possible to obtain training samples from the cloud or other places for model training. Limitations of the Examples.

The target model/rule 501 trained according to the training device 520 can be applied to different systems or devices, such as the execution device 510 shown in FIG. 5, which can be a terminal, such as a mobile terminal, a tablet computer, a notebook A computer, an augmented reality (augmented reality, AR)/virtual reality (virtual reality, VR) device, a vehicle terminal, etc., may also be a server or a cloud, etc.

Specifically, the training device 520 may transfer the data to the executing device 510 .

In FIG. 5 , an execution device 510 is configured with an input/output (input/output, I/O) interface 512 for data interaction with an external device, and a user may input data to the I/O interface 512 through a client device 540 (such as this The data to be processed in the application examples).

The preprocessing module 513 and the preprocessing module 514 are configured to perform preprocessing according to the input data received by the I/O interface 512 . It should be understood that there may be no preprocessing module 513 and preprocessing module 514 or only one preprocessing module. When the preprocessing module 513 and the preprocessing module 514 do not exist, the calculation module 511 may be used directly to process the input data.

When the execution device 510 preprocesses the input data, or in the calculation module 511 of the execution device 510 performs calculation and other related processing, the execution device 510 can call the data, codes, etc. in the data storage system 550 for corresponding processing , the correspondingly processed data and instructions may also be stored in the data storage system 550 .

Finally, the I/O interface 512 presents the processing result (for example, the processing result in the embodiment of the present application) to the client device 540, thereby providing it to the user.

In the situation shown in FIG. 5 , the user can manually specify input data, and the “manually specify input data” can be operated through the interface provided by the I/O interface 512 . In another case, the client device 540 can automatically send the input data to the I/O interface 512 . If the client device 540 is required to automatically send the input data to obtain the user's authorization, the user can set the corresponding authority in the client device 540 . The user can view the results output by the execution device 510 on the client device 540, and the specific presentation form may be specific ways such as display, sound, and action. The client device 540 can also be used as a data collection terminal, collecting input data from the input I/O interface 512 and output results from the output I/O interface 512 as new sample data, and storing them in the database 530 . Of course, it is also possible to collect without the client device 540, but the I/O interface 512 directly uses the input data input to the I/O interface 512 as shown in the figure and the output result of the output I/O interface 512 as a new sample The data is stored in database 530 .

It should be noted that FIG. 5 is only a schematic diagram of a system architecture provided by the embodiment of the present application, and the positional relationship between devices, devices, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG. 5, the data The storage system 550 is an external memory relative to the execution device 510 , and in other cases, the data storage system 550 may also be placed in the execution device 510 . It should be understood that the above execution device 510 may be deployed in the client device 540 .

From the inference side of the model:

In the embodiment of the present application, the calculation module 511 of the execution device 520 can obtain the code stored in the data storage system 550 to implement the model feed-forward process in the embodiment of the present application.

In the embodiment of the present application, the calculation module 511 of the execution device 520 may include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, digital signal processing (digital signal processing, DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits, for example, the training device 520 can be a hardware system with the function of executing instructions, such as CPU, DSP, etc., Or a hardware system that does not have the function of executing instructions, such as ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems that do not have the function of executing instructions and hardware systems that have the function of executing instructions.

Specifically, the computing module 511 of the execution device 520 may be a hardware system capable of executing instructions, the data processing method provided in the embodiment of the present application may be software codes stored in the memory, and the computing module 511 of the execution device 520 may read from the memory The software code is obtained, and the obtained software code is executed to realize the model feed-forward process in the embodiment of the present application.

It should be understood that the calculation module 511 of the execution device 520 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. The computing module 511 is implemented by a hardware system that does not have the function of executing instructions, and is not limited here.

From the training side of the model:

In the embodiment of the present application, the above-mentioned training device 520 can obtain the code stored in the memory (not shown in FIG. 5, which can be integrated into the training device 520 or deployed separately from the training device 520) to implement the model training method in the embodiment of the present application. .

In the embodiment of the present application, the training device 520 may include a hardware circuit (such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (digital signal processing, DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits, for example, the training device 520 can be a hardware system with the function of executing instructions, such as CPU, DSP, etc., or a hardware system without execution A hardware system with an instruction function, such as ASIC, FPGA, etc., or a combination of the above-mentioned hardware system without the function of executing instructions and a hardware system with the function of executing instructions.

Specifically, the training device 520 can be a hardware system capable of executing instructions, and the data processing method provided in the embodiment of the present application can be a software code stored in a memory, and the training device 520 can obtain the software code from the memory, and execute the acquisition The obtained software code is used to implement the model training method provided in the embodiment of the present application.

It should be understood that the training device 520 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. The instruction function is implemented by a hardware system, which is not limited here.

Referring to Fig. 6, Fig. 6 is a schematic diagram of an embodiment of a model training method provided by the embodiment of the present application. The model training method provided by the embodiment of the present application can be applied to training equipment, and the training equipment can be a mobile phone, a tablet, or a notebook Terminal devices such as computers and smart wearable devices, and training devices can also be devices with data processing capabilities such as servers and chips. As shown in Figure 6, a model training method provided by the embodiment of the present application may include:

601. Acquire a sample image.

In autonomous driving scenarios, street view images often contain multiple instances (or objects) such as pedestrians and vehicles, which have strong global inconsistency and do not meet the single instance assumption (image blocks from the same image describe the same semantics information, their features should be as close as possible, and image blocks from different images describe different semantic information, so their features should be as different as possible), so existing methods are often limited to datasets with single-instance assumptions, It is difficult to take full advantage of more realistic multi-instance autonomous driving datasets.

In the embodiment of the present application, the sample image may be a multi-instance image (such as a street view image) in the above-mentioned automatic driving scene.

In a possible implementation, the sample image may include multiple objects, and the objects include at least one of people, vehicles, traffic signs, lane lines, and plants. Exemplarily, the objects may include but not limited to at least one of the following examples: dynamic obstacles (pedestrians, cyclists (Cyclist), tricycles (Tricycle), cars (Car), trucks (Truck), buses ( Bus)), static obstacles (TrafficCone, TrafficStick, Fire Hydrant, Motocycle, Bicycle), traffic signs (TrafficSign), guide signs ( GuideSign), Billboard, Red Traffic Light (TrafficLight_Red) / Yellow Traffic Light (TrafficLight_Yellow) / Green Traffic Light (TrafficLight_Green) / Black Traffic Light (TrafficLight_Black), Road Sign (RoadSign)).

602. Sampling the sample image to obtain a first image block and a second image block, where the first image block and the second image block are different image blocks on the sample image.

Wherein, after the sample image is acquired, the sample image can be sampled to obtain a plurality of partial images (such as a first image block and a second image block), and the first image block and the second image block can be small samples obtained by sampling the sample image. image (here small refers to the image area of the first image block and the second image block being smaller relative to the size of the sample image).

Wherein, the above-mentioned sampling can be understood as cropping. In one sampling process, a rectangular area can be randomly determined on the sample image, and the image in the determined rectangular area can be used as the first image block. In another sampling process, it can be A rectangular area is randomly determined on the sample image, and an image within the determined rectangular area is used as a second image block.

Wherein, the above-mentioned sampling may be random sampling, and the so-called random sampling may be understood as that a sampling position and/or a sampling size are randomly determined.

Wherein, the first image block and the second image block may be rectangular images.

603. Based on the fact that the parallelism ratio between the regions where the first image block and the second image block are located on the sample image is greater than a threshold value, respectively extract the first image block and the second image block through a feature extraction network. Feature extraction is performed on the image block to obtain a first feature map and a second feature map.

Wherein, there may be a common area (overlapping area) between the area where the first image block is located on the sample image and the area where the second image block is located on the sample image, the area of the common area is the same as that of the first image block and the second image block The ratio between all areas occupied by image blocks on the sample image is the intersection over union (IoU) (intersection over union, IoU) between the areas where the first image block and the second image block are located on the sample image ( For example, it can be referred to as shown in FIG. 7).

In a possible implementation, the threshold may be a value greater than or equal to 0.4.

Among them, the root cause of global inconsistency in multi-instance scenes is that randomly cut image blocks may represent completely different scenes and semantic information, but if two image blocks can be controlled to be "close" to each other enough, then in the local area according to the image Continuity, two image patches can be considered to describe similar semantic information, thus restoring the basic assumption of contrastive learning. In the embodiment of the present application, the intersection ratio between two image blocks can be used as a way to measure the distance, where the intersection ratio refers to the ratio of the overlapping area of the two image blocks to the total coverage area (see the left side of Figure 7). It is required that the subsequent feature extraction and similarity calculation will be performed only when the intersection ratio of two randomly generated image blocks is greater than a given threshold (see the right of Figure 7).

It should be understood that the threshold value of the intersection ratio cannot be set too low, but at the same time, it is not as high as possible, because it is not expected that the image blocks are irrelevant, but they are not expected to be exactly the same, so the selection of the threshold value of the intersection ratio It is actually also controlling the balance between data noise and data complexity in multi-instance self-supervised learning.

Exemplarily, referring to FIG. 8, FIG. 8 shows an example of a sample image. Referring to FIG. 9, FIG. 9 is a schematic diagram of a first image block and a second image block. The intersection ratio IOU between the second image blocks is 0.3. Referring to FIG. 10, FIG. 10 is a schematic diagram of the first image block and the second image block. Between the first image block and the second image block shown in FIG. 10 The parallel and intersection ratio IOU between them is 0.5. Referring to Figure 11, Figure 11 is a schematic diagram of the first image block and the second image block, and the parallel and intersection ratio between the first image block and the second image block shown in Figure 11 The IOU is 0.7.

In this embodiment of the present application, based on the fact that the parallelism ratio between the regions where the first image block and the second image block are located on the sample image is greater than a threshold, the feature extraction network can be used to separately analyze the first image block and performing feature extraction with the second image block to obtain a first feature map and a second feature map.

Next, introduce the feature extraction network in the embodiment of this application:

In a possible implementation, the feature extraction network can be a backbone network, and the backbone network is used to receive input images (such as the first image block and the second image block in the embodiment of the present application), and The image is convolved to generate multiple feature maps.

It should be noted that "convoluting the input image" here should not be understood as only performing convolution processing on the input image. In some implementations, the input image can be Perform convolution processing and other processing.

It should be noted that the "convolution process on the first image to generate multiple feature maps" here should not only be understood as performing multiple convolution processes on the image, and each convolution process can be Generate a feature map, that is, it should not be understood that each feature map is obtained based on the convolution of the image, but, on the whole, the image is the source of multiple feature maps; in one implementation, you can Convolute the image to obtain a feature map, and then perform convolution processing on the generated feature map to obtain another feature map, and so on, to obtain multiple feature maps.

It should be noted that a series of convolution processing may be performed on the input image, specifically, each convolution processing may be performed on the feature map obtained by the previous convolution processing, and then A feature map is obtained. Through the above method, multiple feature maps can be obtained.

It should be noted that multiple feature maps can be feature maps with multi-scale resolution, that is, multiple feature maps are not feature maps with the same resolution. In an optional implementation, multiple feature maps can constitute a feature pyramid.

Referring to FIG. 12 , FIG. 12 is a schematic structural diagram of a feature extraction network provided by an embodiment of the present application. As shown in Figure 12, the backbone network is used to receive the input image, and perform convolution processing on the input image, and output feature maps with different resolutions corresponding to the image (feature map C1, feature map C2, feature map C3 , feature map C4); that is to say, feature maps of different sizes corresponding to the image are output, and the backbone network completes the extraction of basic features to provide corresponding features for subsequent detection.

Specifically, the backbone network can perform a series of convolution processing on the input image to obtain feature maps (feature maps) at different scales (with different resolutions). These feature maps will provide the basic features for subsequent detection modules. The backbone network can take many forms, such as visual geometry group (VGG), residual neural network (resnet), GoogLeNet's core structure (Inception-net), etc.

The backbone network can perform convolution processing on the input image to generate several convolutional feature maps of different scales. Each feature map is a matrix of H*W*C, where H is the height of the feature map and W is the width of the feature map. , C is the number of channels of the feature map.

Backbone can use a variety of existing convolutional network frameworks, such as VGG16, Resnet50, Inception-Net, etc. The following uses Resnet18 as Backbone as an example to illustrate.

Assume that the resolution of the input image is H*W*3 (height H, width W, and the number of channels is 3, that is, three channels of RBG). The input image can be convoluted through a convolutional layer Res18-Conv1 of Resnet18 to generate Featuremap (feature map) C1. This feature map has been down-sampled twice relative to the input image, and the number of channels has been expanded to 64. Therefore, the feature map of C1 The resolution is H/4*W/4*64. C1 can be convolved by Res18-Conv2 of Resnet18 to obtain Featuremap C2. The resolution of this feature map is consistent with C1; C2 continues to be convolved by Res18-Conv3 to generate Featuremap C3. This feature map is further downsampled relative to C2. , the number of channels is doubled, and its resolution is H/8*W/8*128; finally, C3 undergoes convolution operation through Res18-Conv4 to generate Featuremap C4, whose resolution is H/16*W/16*256.

It should be noted that the backbone network in the embodiment of the present application may also be referred to as a backbone network, which is not limited here.

It should be noted that the backbone network shown in FIG. 12 is only an implementation manner and does not limit the present application.

Wherein, in the embodiment of the present application, the feature extraction network is used to extract the features of the first image block and the second image block respectively, so as to obtain the first feature map and the second feature map, and the first feature map can be obtained by The multiple feature maps (feature pyramids) obtained by the feature extraction network processing the first image block can also be a feature map obtained after a certain convolution operation. Similarly, the second feature map can be obtained through feature extraction A plurality of feature maps (feature pyramids) obtained by processing the second image block by the network may also be a feature map obtained after a certain convolution operation.

Wherein, the size of the first feature map and the second feature map are the same.

604. Determine a loss according to the difference between the first feature map and the second feature map, and update the feature extraction network based on the loss to obtain an updated feature extraction network.

In the embodiment of the present application, based on the fact that the parallelism ratio between the regions where the first image block and the second image block are located on the sample image is greater than a threshold value, the feature extraction network is used to separately analyze the first image block After performing feature extraction with the second image block to obtain the first feature map and the second feature map, the loss can be determined according to the difference between the first feature map and the second feature map, and based on the The loss is used to update the feature extraction network to obtain the updated feature extraction network.

In a possible implementation, the first image block and the second image block may be aligned first, so as to obtain an aligned first image block and an aligned second image block;

In one implementation, the sample image may include a target area, and the target area is an overlapping area where the first image block and the second image block are located on the sample image, and the target area may be , determine the first sub-feature map corresponding to the target area in the first feature map and the second sub-feature map corresponding to the target area in the second feature map; for the first sub-feature map performing upsampling to obtain the aligned first image block; performing upsampling to the second sub-feature map to obtain the aligned second image block, wherein the aligned first image block It is consistent with the size of the aligned second image block.

In order to distinguish multi-instance features, it is necessary to abandon the global pooling layer after the backbone network to maintain the two-dimensional structure and position information of the feature map, but it also brings additional features that are not aligned. One-to-one correspondence with the same relative position. In the embodiment of the present application, two different ways are provided for feature alignment: region of interest (region of interest) alignment uses the overlapping parts of the image blocks as the regions of interest of the two image blocks, for example, RoI Align can be used to extract only the features of the overlapping part For subsequent calculations, this method is intuitive but does not make full use of the information of non-overlapping parts (for example, refer to FIG. 13 ).

In one implementation, the first feature map and the second feature map have the same size, the first feature map includes M first feature points, and the second feature map includes M second feature points points, the M first feature points correspond to M first pixel points in the sample image, the M second feature points correspond to M second pixel points in the sample image, the The M first pixels are in one-to-one correspondence with the M second pixels, and a third feature map can be obtained according to the M first pixels and the M second pixels, and the third The size of the feature map is the same as that of the second feature map, and the third feature map includes M third feature points, and each third feature point is based on a corresponding relationship between the first pixel point and the second pixel point The pixel position difference between them is obtained; the third feature map is fused with the first feature map to obtain the aligned first image block, and the second feature map is used as the aligned the second image block. Optionally, the third feature map and the first feature map may be spliced in a depth direction.

Among them, the displacement alignment takes each pair of pixels at the same relative position, calculates their coordinate displacement in the original image and concatenates them with the feature map (stitching in the depth direction), and provides them as additional side information to the prediction network for implicit feature alignment to help subsequent feature prediction (for example, as shown in Figure 14), making full use of the feature information of non-overlapping regions, and making the subsequent similarity measurement more flexible.

After the aligned first feature map and the aligned second feature map are obtained, a loss may be determined according to a difference between the aligned first feature map and the aligned second feature map.

The following describes how to obtain the difference between the aligned first feature map and the aligned second feature map:

In a possible implementation, the M first feature points of the first feature map can be processed through the target prediction network to obtain the predicted value of each first feature point; wherein, the target prediction network can include Convolution operation (eg 1*1 convolution kernel).

Wherein, the above-mentioned step of processing the M first feature points of the first feature map through the target prediction network can be called an online branch, and the online branch predicts the clustering center result of the target branch through the prediction network, Optionally, the online branch can additionally deploy a spatial dimension self-attention module, fully considering the context information to make more accurate predictions. Specifically, if the image online branch input is R, then the prediction result Q of the final online branch is:

where H and W are the length and width of R and R′, q _θ ( ) is the prediction network, and sim( , ) is defined as:

sim(R _i,j ,R _i′,j′ )=(max(cos(R _i,j ,R _i′,j′ ),0)) ² ;

In a possible implementation, the M second feature points of the first feature map may be clustered based on the target clustering algorithm, so as to update the feature values of the M second feature points, where each The updated eigenvalues of the second feature points are the eigenvalues of the cluster center of the cluster category in which they are located.

Wherein, based on the target clustering algorithm, the step of clustering the M second feature points of the first feature map may be referred to as a target branch. The target clustering algorithm can be but not limited to K-means (k-means), mean shift clustering, density-based clustering method, agglomerative hierarchical clustering, graph community detection (graph community detection), Gaussian mixture model k-means ( Gaussian mixture model kmeans, GMM k-means) and other clustering methods.

Among them, there is a hierarchical structure of clustering in multi-instance scenarios, including category clustering, instance clustering, and pixel clustering from top to bottom; at the same time, the definition of clustering is a relative concept, that is, images of the same pixel in different contexts The blocks may belong to different clusters (see the two image blocks shown in the lower right of Fig. category clustering). Based on this observation, the features of different instances can be distinguished by clustering within the image, so that the features of the same clustered pixels are as close as possible, while the features of different clustered pixels are as different as possible, which promotes the consistency of the clustering analysis results of the two image blocks. consistency. The specific method can be shown in Figure 16. Optionally, K-means clustering can be used on the target branch to obtain the cluster center (mean, smooth) feature of each point.

Among them, the embodiment of the present application uses image clustering. On the one hand, the network has the ability to distinguish the characteristics of different instances; Deploying a self-attention module introduces a global perspective to obtain more accurate prediction results.

After obtaining the predicted value of each first feature point and the feature value of each updated second feature point, based on the predicted value of each first feature point and each updated second feature point The difference between the feature values of the two feature points determines the loss, where the loss can represent the difference between the predicted value of each first feature point and the feature value of each updated second feature point (or describe for the similarity). The two branches maximize the overall information of the two-dimensional feature map in different ways, so that the similarity between two two-dimensional feature maps can be better measured.

In one implementation, similarity can be defined as:

Among them, the input of the online branch and the target branch are R and R' respectively, Q _i,j is the output of the online branch, and Kmeans(R′ _i,j ) is the output of the target branch.

After the loss is obtained, the online branch (for example, including the target prediction network and the feature extraction network) can be updated according to the loss. For example, gradient descent can be used to update the online branch parameters, and at the same time, the updated target branch parameters are the moving average of the online branch parameters, and the above training process is iterated until the network converges.

Referring to Figure 17 and Figure 18, Figure 17 and Figure 18 are a multi-instance self-supervised learning framework (MultiSiam) data enhancement, feature extraction and network training process provided by the embodiment of the present application:

For each input image, randomly cut to obtain two image blocks, check whether the intersection ratio of the two is greater than a given threshold, if not, re-cut until the intersection ratio threshold is exceeded, and if so, continue to scale adjustment and color texture enhancement; Use the backbone network to extract the two-dimensional feature maps of the two image blocks; use the projection network to reduce the dimensionality of the two-dimensional feature maps of the two image blocks to reduce the amount of subsequent calculations; through the feature alignment module, use the region of interest alignment or displacement alignment to Restore the one-to-one correspondence on the same relative position of the two-dimensional feature map. Note that the feature alignment module does not change the spatial scale of the input feature map; run the K-means clustering algorithm on the target branch (left) feature map to obtain each The clustering results of the pixels and the corresponding clustering center point features, and in order to prevent network degradation, a gradient truncation operation is added at the end of the target branch; the online branch (right) clusters the target branch through the self-attention module and the prediction network Predict the results, calculate the two-dimensional cluster similarity according to the definition, and do a weighted sum with the one-dimensional feature similarity (for example, the weight of both can be 0.5 by default) to obtain the final feature similarity measure; use gradient descent to update online branch parameters , while updating the target branch parameters to be the moving average of the online branch parameters, iteratively operate until the network converges.

It should be understood that the above-mentioned process of training the feature extraction network may be a pre-training process. After the updated feature extraction network is obtained, the updated feature extraction network may be fine-tuned for the target task in combination with the downstream task data set.

For example, the above-mentioned updated feature extraction network can be connected to the downstream task network, and the above-mentioned updated feature extraction network and the downstream task network can be fine-tuned based on the downstream task data set to obtain the target network, which can be deployed on the execution device Perform a feed-forward process.

Exemplarily, taking the downstream task as an example of target detection, the downstream task network may include one or more heads, as shown in Figure 19, one or more heads are used to extract the feature map output by the feature extraction network, for a The task object in the task is detected, and the 2D frame of the area where the task object is located and the confidence corresponding to each 2D frame are output; optionally, multiple heads can be set in parallel, and each head can complete the detection of different task objects ; Wherein, the task object is an object that needs to be detected in the task; the higher the confidence level, the greater the probability that the object corresponding to the task exists in the 2D frame corresponding to the confidence level.

In the embodiment of this application, different heads can complete different 2D detection tasks. For example, one of the multiple heads can complete the detection of the car, and output the 2D frame and confidence of Car/Truck/Bus; head1 of the multiple heads It can complete the detection of people, and output the 2D frame and confidence of Pedestrian/Cyclist/Tricyle; the head of multiple heads can complete the detection of traffic lights, and output the 2D frame and confidence of Red_Trafficlight/Green_Trafficlight/Yellow_TrafficLight/Black_TrafficLight.

In the embodiment of the present application, the downstream task network may include multiple serial heads; the serial head is connected to a parallel head; what needs to be emphasized here is that, in fact, the serial head is not necessary, and it is only necessary to detect 2D box scenario, there is no need to include the serial head.

Wherein, the serial head can be used to: use the 2D frame of the task object of the task provided by the connected parallel head to extract the features of the region where the 2D frame is located on one or more feature maps on the feature extraction network , predicting the 3D information, Mask information or Keypiont information of the task object of the task according to the characteristics of the area where the 2D frame is located. The serial head is optionally connected in series behind the parallel head, and on the basis of detecting the 2D frame of the task, the 3D/Mask/Keypoint detection of the object inside the 2D frame is completed. For example, the serial 3D_head0 completes the estimation of the orientation, center of mass and length, width and height of the vehicle, thereby outputting the 3D frame of the vehicle; the serial Mask_head0 predicts the fine mask of the vehicle, thereby separating the vehicle; the serial Keypont_head0 completes the key points of the vehicle estimate. The serial head is not necessary. Some tasks do not require 3D/Mask/Keypoint detection, so there is no need to connect the serial head. For example, the detection of traffic lights only needs to detect the 2D frame, and there is no need to connect the serial head. In addition, some tasks can choose to connect one or more serial heads in series according to the specific needs of the task, such as parking lot (Parkingslot) detection, in addition to the need to obtain the 2D frame, but also the key points of the parking space, so in this task Only one serial Keypoint_head needs to be connected in series, and the 3D and Mask heads are not required.

In the embodiment of this application, the header is connected to the feature extraction network, and the header can complete the detection of the 2D frame of a task according to the feature map provided by the feature extraction network, and output the 2D frame of the object of this task and the corresponding confidence, etc., then Next, describe a schematic diagram of a header structure. Referring to FIG. 20, FIG. 20 is a schematic diagram of a header. As shown in FIG. 20, the header includes a candidate region generation network (Region Proposal Network, RPN), ROI-ALIGN and RCNN Three modules.

Among them, the RPN module can be used to predict the area where the task object is located on one or more third feature maps provided by the feature extraction network, and output a candidate 2D frame matching the area; or it can be understood that the RPN is in the feature The region where the task object may exist is predicted on one or more horizontal images output by the extraction network, and the frames of these regions are given, and these regions are called candidate regions (Proposal). For example, when the head is responsible for detecting a car, its RPN layer predicts a candidate frame that may contain a car; when the head is responsible for detecting a person, its RPN layer predicts a candidate frame that may contain a person. Of course, these Proposals are inaccurate. On the one hand, they do not necessarily contain the objects of the task, and on the other hand, these frames are not compact.

The 2D candidate area prediction process can be implemented by the RPN module of the head, which predicts the areas where the task object may exist according to the feature map provided by the feature extraction network, and gives the candidate frames of these areas (also called candidate areas, Proposal) . In this embodiment, if the head is responsible for detecting cars, its RPN layer predicts candidate boxes that may contain cars.

The RPN layer can generate a feature map RPN Hidden by, for example, a 3*3 convolution on the third feature map provided by the feature extraction network. The RPN layer of the head behind will predict the Proposal from the RPN Hidden. Specifically, the RPN layer of the head predicts the coordinates and confidence of the Proposal at each position of the RPN Hidden through a 1*1 convolution. The higher the confidence, the greater the probability that the Proposal exists in the object of the task. For example, the greater the score of a Proposal in the head, the greater the probability of its existence. Proposals predicted by each RPN layer need to go through the Proposal Merging Module, remove redundant Proposals according to the degree of overlap between Proposals (this process can be used but not limited to the NMS algorithm), and select the largest score among the remaining K Proposals N(N<K) Proposals are used as candidate areas where objects may exist. These Proposals are inaccurate. On the one hand, they do not necessarily contain the objects of the task, and on the other hand, these boxes are not compact. Therefore, the RPN module is only a rough detection process, which requires the subsequent RCNN module to be subdivided. When the RPN module returns the coordinates of the Proposal, it does not directly return the absolute value of the coordinates, but returns the coordinates relative to the Anchor. When these anchors match the actual object, the higher the probability that the RPN can detect the object.

The ROI-ALIGN module is used to deduct the features of the region where the candidate 2D frame is located from a feature map provided by the feature extraction network according to the region predicted by the RPN module; that is, the ROI-ALIGN module is mainly based on The Proposal provided by the RPN module extracts the features of the area where each Proposal is located on a certain feature map, and resizes to a fixed size to obtain the features of each Proposal. It can be understood that the ROI-ALIGN module can use but is not limited to ROI-POOLING (region of interest pooling)/ROI-ALIGN (region of interest extraction)/PS-ROIPOOLING (position-sensitive region of interest pooling)/ PS-ROIALIGN (Position Sensitive Region of Interest Extraction) and other feature extraction methods.

The RCNN module is used to perform convolution processing on the features of the region where the candidate 2D frame is located through the neural network to obtain the confidence that the candidate 2D frame belongs to each object category; adjust the coordinates of the 2D frame in the candidate region through the neural network , so that the adjusted 2D candidate frame matches the shape of the actual object more than the candidate 2D frame, and selects the adjusted 2D candidate frame with a confidence greater than a preset threshold as the 2D frame of the region. In other words, the RCNN module mainly refines the features of each Proposal proposed by the ROI-ALIGN module, and obtains the confidence of each Proposal belonging to each category (for example, for the task of car, it will give Backgroud/Car/Truck /Bus 4 points), and adjust the coordinates of the Proposal's 2D box to output a more compact 2D box. These 2D boxes are combined by nonmaximum suppression (NMS) and output as the final 2D box.

The subdivision of 2D candidate regions is mainly implemented by the RCNN module of the head in Figure 20, which further regresses the more compact 2D frame coordinates according to the features of each Proposal extracted by the ROI-ALIGN module, and classifies the Proposal at the same time. Output the confidence that it belongs to each class. There are many realizable forms of RCNN. The feature size output by the ROI-ALIGN module can be N*14*14*256 (Feature of proposals), which is first processed by the convolution module 4 (Res18-Conv5) of Resnet18 in the RCNN module. The output feature size is N*7*7*512, and then it is processed through a Global Avg Pool (average pooling layer), and the 7*7 features in each channel of the input feature are averaged to obtain N*512 Features, where each 1*512-dimensional feature vector represents the feature of each Proposal. Next, the exact coordinates of the box are returned through two fully connected layers FC (output N*4 vector, these 4 numerical sub-tables represent the x/y coordinates of the center point of the box, the width and height of the box), and the confidence of the box category degree (in head0, need to give the score that this box is Backgroud/Car/Truck/Bus). Finally, through the frame merging operation, several frames with the largest scores are selected, and the repeated frames are removed through the NMS operation, so as to obtain a compact frame output.

In some practical application scenarios, the perception network can also include other heads, which can further perform 3D/Mask/Keypoint detection on the basis of the detected 2D frame. Exemplarily, taking 3D as an example, the ROI-ALIGN module extracts the features of the area where each 2D frame is located on the feature map output by the feature extraction network according to the accurate 2D frame provided by the head, assuming that the number of 2D frames is M , then the feature size output by the ROI-ALIGN module is M*14*14*256, which is first processed by Res18-Conv5 of Resnet18, and the output feature size is N*7*7*512, and then passed through a Global AvgPool (average pool layer) for processing, and average the 7*7 features of each channel in the input features to obtain M*512 features, where each 1*512-dimensional feature vector represents the feature of each 2D box. Next, the orientation angle (orientation, M*1 vector) and centroid point coordinates (centroid, M*2 vector, these two values represent the x/y coordinates of the centroid) and Length, width and height (dimention).

It should be noted that the header shown in FIG. 19 and FIG. 20 is only an implementation manner, and does not constitute a limitation to the present application.

Taking two large-scale multi-instance autonomous driving datasets as upstream pre-training datasets as an example, in the selection of experimental data, upstream self-supervised pre-training can use multi-instance autonomous driving datasets including Waymo public datasets and SODA-5M data set. The Waymo public dataset contains nearly 790,000 unlabeled images, ranging in size from (1920, 968) to (1920, 1280), and is currently the largest open-source autonomous driving dataset; the SODA-5M dataset contains 500 10,000 high-quality autonomous driving images with image scales of (1920, 1280). The downstream migration data set mainly considers Cityscapes and BDD100K, two widely used autonomous driving semantic segmentation benchmark tasks, among which Cityscpaes contains annotations of 8 semantic categories, including 2975 training set images and 500 validation set images collected from 27 cities; BDD100K contains annotations for 9 semantic categories, and contains 70,000 training set images and 10,000 validation set images collected from 4 cities.

Due to the lack of manual annotation in the upstream data set, the performance of the algorithm cannot be directly evaluated. Therefore, the self-supervised pre-training model is used to initialize the backbone network of the downstream task and then fine-tune it. The average union ratio mIOU of the final prediction results on the downstream task verification set is used as the evaluation. The indicator judges the quality of the pre-trained model, and the higher the mIOU value, the better. First, self-supervised pre-training is performed on the upstream autonomous driving dataset according to the proposed framework. After the network converges, the upstream task-specific modules such as the projection network and the prediction network are discarded, and only the backbone network parameters are retained and transferred to the downstream tasks; use the downstream task training Fine-tune the network parameters, take the fine-tuned model to predict on the verification set, and report the mIOU of the final prediction result as the evaluation standard.

Use ResNet-50 as the default backbone network. In order to better adapt to large batch training, the LARS optimizer and cosine learning rate decay are used to stabilize the network training process. In order to better compare the pre-training results of the models on different datasets, we keep the GPU time of training on different datasets consistent. Specifically, 325 epoch and 55 epoch pre-training are performed on Waymo and SODA-5M respectively to make a fair comparison with ImageNet 200 epoch pre-training.

By pre-training on the public self-driving dataset Waymo, the method proposed in the embodiment of this application successfully obtained a significant performance improvement of 4.7% on Cityscapes and 3.9% on BDD100K on the basis of the benchmark model BYOL, and successfully pre-trained on Waymo The current optimal result has been achieved during the training. It can be seen that the embodiment of the present application can better use multi-instance data sets for feature learning. However, considering that Waymo has fewer images (0.79Mvs1.28M) than ImageNet, and has a strong front-background imbalance, in fact Waymo is not as good as ImageNet in terms of the number and quality of images, and the pre-training results are also It is difficult to get beyond ImageNet pre-training. Therefore, the self-driving data set SODA-5M was additionally used for pre-training, which successfully surpassed the migration results of ImageNet pre-training, demonstrating the effectiveness of in-domain pre-training.

It is worth noting that due to the existence of the single-instance assumption, to collect more images similar to the ImageNet data set, it is necessary to artificially filter and clean the data, and the multi-instance scene gets rid of the single-instance assumption, which can be obtained at a very low cost. A large-scale pre-training data set is collected, so multi-instance self-supervised learning is more practical for industrial-level data sets.

Referring to Table 1, Table 1 is the pre-training result of the embodiment of the present application on the multi-instance automatic driving data set. Compared with the benchmark model, the embodiment of the present application obtains obvious performance improvement; Successfully surpassed ImageNet pre-training, demonstrating the effectiveness of in-domain pre-training.

Table 1

Exemplarily, visual analysis is performed on the finally learned features, and the last two-dimensional feature map of the pre-trained backbone network is used for K-means clustering analysis. The clustering results are shown in Figure 21. Compared with the existing technology , the embodiment of the present application can not only judge the information of the foreground and the background, but also accurately distinguish the features of different instances. At the same time, the feature clustering result of the embodiment of the present application is smoother, which means that the translation in the learned feature representation Invariance and feature robustness are enhanced.

In addition, the solution provided by the embodiment of this application is deployed on a single instance for self-supervised learning to verify the scalability of this framework. Except for the experimental data, the rest of the settings are consistent with the technical scheme 1. In the selection of experimental data, the upstream data set can use the ImageNet data set with a single-instance hypothesis, which contains 1,000 categories of natural images, and contains a total of 1.28 million training set pictures; here, manual labeling is not used, but directly used Image data for self-supervised representation learning. The downstream data set additionally considers two general-purpose detection data sets, VOC and COCO. The VOC data set contains 20 semantic category annotation information, a total of 10,000 training images and 4,900 test images; the COCO data set contains 80 semantic category annotation information. , a total of about 118,000 training images and 5,000 validation set images.

It has been verified that although the embodiment of the present application is designed for multi-instance scenes, it can still effectively use single-instance scene images to obtain high-quality feature representations, and obtain the current optimal migration results on multiple downstream tasks (see Table 2). This shows the versatility and scalability of the framework of the embodiment of the present application.

Table 2

An embodiment of the present application provides a model training method, the method comprising: acquiring a sample image; sampling the sample image to obtain a first image block and a second image block, the first image block and the second image block The image block is a different image block on the sample image; based on the union ratio between the first image block and the second image block on the sample image where the area is greater than a threshold, the feature extraction network is respectively used for performing feature extraction on the first image block and the second image block to obtain a first feature map and a second feature map; determining a loss according to the difference between the first feature map and the second feature map , and update the feature extraction network based on the loss to obtain an updated feature extraction network. Using the intersection ratio as a measure of the relative distance between two image blocks, by requiring the image block intersection ratio to be greater than a given threshold to ensure semantic consistency in the local area to restore the basic assumption of contrastive learning, the image block is limited to a local area , so as to transform the global inconsistency into local consistency, and then improve the data processing accuracy of the updated feature extraction network. At the same time, the choice of threshold can effectively control the balance between data noise and data complexity in different scenarios.

Referring to Fig. 22, Fig. 22 is a schematic structural diagram of a model training device provided in the embodiment of the present application. As shown in Fig. 22, the device 2200 provided in the embodiment of the present application includes:

An acquisition module 2201, configured to acquire sample images.

Wherein, for the specific description of the obtaining module 2201, reference may be made to the description of step 601 in the above-mentioned embodiment, and details are not repeated here.

A sampling module 2202, configured to sample the sample image to obtain a first image block and a second image block, where the first image block and the second image block are different image blocks on the sample image;

Wherein, for the specific description of the sampling module 2202, reference may be made to the description of step 602 in the above-mentioned embodiment, which will not be repeated here.

The feature extraction module 2203 is configured to, based on the intersection ratio between the regions where the first image block and the second image block are located on the sample image, being greater than a threshold, respectively extract the first image block through a feature extraction network performing feature extraction with the second image block to obtain a first feature map and a second feature map;

Wherein, for the specific description of the feature extraction module 2203, reference may be made to the description of step 603 in the above embodiment, and details are not repeated here.

The model updating module 2204 is configured to determine a loss according to the difference between the first feature map and the second feature map, and update the feature extraction network based on the loss to obtain an updated feature extraction network.

Wherein, for the specific description of the model update module 2204, reference may be made to the description of step 604 in the above-mentioned embodiment, and details are not repeated here.

In a possible implementation, the threshold is a value greater than or equal to 0.4.

In a possible implementation, the device also includes:

an alignment module, configured to align the first image block and the second image block before determining the loss according to the difference between the first feature map and the second feature map, so as to obtain the aligned first image block and the aligned second image block;

The model update module is specifically used for:

A loss is determined based on the difference between the aligned first feature map and the aligned second feature map.

In a possible implementation, the sample image includes a target area, where the target area is an overlapping area where the first image block and the second image block are located on the sample image, and the alignment module, Specifically for:

According to the target area, determining a first sub-feature map corresponding to the target area in the first feature map and a second sub-feature map corresponding to the target area in the second feature map;

Upsampling the first sub-feature map to obtain the aligned first image block;

Up-sampling the second sub-feature map to obtain the aligned second image block, wherein the aligned first image block has the same size as the aligned second image block.

In a possible implementation, the first feature map and the second feature map have the same size, the first feature map includes M first feature points, and the second feature map includes M first feature points Two feature points, the M first feature points correspond to M first pixel points in the sample image, and the M second feature points correspond to M second pixel points in the sample image, The M first pixels are in one-to-one correspondence with the M second pixels, and the alignment module is specifically used for:

According to the M first pixel points and the M second pixel points, a third feature map is obtained, the size of the third feature map is the same as that of the second feature map, and the third feature map Including M third feature points, each third feature point is obtained based on the pixel position difference between the first pixel point and the second pixel point with corresponding relationship;

The third feature map is fused with the first feature map to obtain the aligned first image block, and the second feature map is used as the aligned second image block.

In a possible implementation, the alignment module is specifically used for:

Splicing the third feature map and the first feature map in a depth direction.

In a possible implementation, the model updating module is specifically used for:

Processing the M first feature points of the first feature map through the target prediction network to obtain a predicted value of each first feature point;

Based on the target clustering algorithm, cluster the M second feature points of the first feature map to update the feature values of the M second feature points, wherein the feature of each updated second feature point The value is the cluster center feature value of the cluster category;

The loss is determined according to the difference between the predicted value of each first feature point and the feature value of each updated second feature point.

In a possible implementation, the model update module is also used to:

Based on the loss, the target prediction network is updated.

In a possible implementation, the sample image includes multiple objects, and the objects include at least one of people, vehicles, traffic signs, lane lines, and plants.

In a possible implementation, the device also includes:

A data processing module, configured to acquire a target network and an image to be processed, the target network including the updated feature extraction network and a downstream task network;

The image to be processed is processed through the target network to obtain a processing result.

Next, an execution device provided by the embodiment of the present application is introduced. Please refer to FIG. 23. FIG. 23 is a schematic structural diagram of the execution device provided by the embodiment of the present application. Tablets, laptops, smart wearable devices, monitoring data processing equipment or servers, etc., are not limited here. Specifically, the execution device 2300 includes: a receiver 2301, a transmitter 2302, a processor 2303, and a memory 2304 (the number of processors 2303 in the execution device 2300 may be one or more, and one processor is taken as an example in FIG. 23 ) , where the processor 2303 may include an application processor 23031 and a communication processor 23032 . In some embodiments of the present application, the receiver 2301 , the transmitter 2302 , the processor 2303 and the memory 2304 may be connected through a bus or in other ways.

The memory 2304 may include read-only memory and random-access memory, and provides instructions and data to the processor 2303 . A part of the memory 2304 may also include a non-volatile random access memory (non-volatile random access memory, NVRAM). The memory 2304 stores processors and operating instructions, executable modules or data structures, or their subsets, or their extended sets, wherein the operating instructions may include various operating instructions for implementing various operations.

The processor 2303 controls the operations of the execution device. In a specific application, various components of the execution device are coupled together through a bus system, where the bus system may include not only a data bus, but also a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the various buses are referred to as bus systems in the figures.

The methods disclosed in the foregoing embodiments of the present application may be applied to the processor 2303 or implemented by the processor 2303 . The processor 2303 may be an integrated circuit chip, which has a signal processing capability. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 2303 or instructions in the form of software. The above-mentioned processor 2303 may be a general-purpose processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and may further include an application specific integrated circuit (ASIC), a field programmable gate Field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 2303 may implement or execute various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory 2304, and the processor 2303 reads the information in the memory 2304, and completes the steps of the above method in combination with its hardware.

The receiver 2301 can be used to receive input digital or character information, and generate signal input related to performing device related settings and function control. The transmitter 2302 can be used to output digital or character information through the first interface; the transmitter 2302 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 2302 can also include display devices such as a display screen .

In the embodiment of the present application, in one case, the processor 2303 is configured to execute the model obtained through the above-mentioned model training method in FIG. 6 .

The embodiment of the present application also provides a training device. Please refer to FIG. 24. FIG. 24 is a schematic structural diagram of the training device provided in the embodiment of the present application. The image described in the embodiment corresponding to FIG. 22 can be deployed on the training device 2400 The training device is used to implement the functions of the image training device in the embodiment corresponding to FIG. 22. Specifically, the training device 2400 is implemented by one or more servers. The training device 2400 may have relatively large differences due to different configurations or performances, and may include One or more central processing units (central processing units, CPU) 2424 (for example, one or more processors) and memory 2432, one or more storage media 2430 for storing application programs 2442 or data 2444 (for example, one or more mass storage devices). Wherein, the memory 2432 and the storage medium 2430 may be temporary storage or persistent storage. The program stored in the storage medium 2430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the training device. Furthermore, the central processing unit 2424 may be configured to communicate with the storage medium 2430 , and execute a series of instruction operations in the storage medium 2430 on the training device 2400 .

The training device 2400 can also include one or more power supplies 2426, one or more wired or wireless network interfaces 2450, one or more input and output interfaces 2458; or, one or more operating systems 2441, such as Windows Server™, Mac OS X™ , UnixTM, LinuxTM, FreeBSDTM and so on.

In the embodiment of the present application, the central processing unit 2424 is configured to execute the model training method in FIG. 6 above.

The embodiment of the present application also provides a computer program product, which, when running on a computer, causes the computer to perform the steps performed by the aforementioned execution device, or enables the computer to perform the steps performed by the aforementioned training device.

An embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a program for signal processing, and when it is run on a computer, the computer executes the steps performed by the aforementioned executing device , or, causing the computer to perform the steps performed by the aforementioned training device.

The execution device, training device or terminal device provided in the embodiment of the present application may specifically be a chip. The chip includes: a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input/output interface, pins or circuits etc. The processing unit can execute the computer-executed instructions stored in the storage unit, so that the chips in the execution device execute the data processing methods described in the above embodiments, or make the chips in the training device execute the data processing methods described in the above embodiments. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit may also be a storage unit located outside the chip in the wireless access device, such as only Read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (random access memory, RAM), etc.

Specifically, please refer to FIG. 25. FIG. 25 is a schematic structural diagram of a chip provided by an embodiment of the present application. The chip can be represented as a neural network processor NPU 2500, and the NPU 2500 is mounted on the main CPU (Host CPU) as a coprocessor. CPU), the tasks are assigned by the Host CPU. The core part of the NPU is the operation circuit 2503, and the operation circuit 2503 is controlled by the controller 2504 to extract matrix data in the memory and perform multiplication operations.

In some implementations, the operation circuit 2503 includes multiple processing units (Process Engine, PE). In some implementations, arithmetic circuit 2503 is a two-dimensional systolic array. The arithmetic circuit 2503 may also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 2503 is a general-purpose matrix processor.

For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit fetches the data corresponding to the matrix B from the weight memory 2502, and caches it in each PE in the operation circuit. The operation circuit takes the data of matrix A from the input memory 2501 and performs matrix operation with matrix B, and the obtained partial or final results of the matrix are stored in the accumulator (accumulator) 2508 .

The unified memory 2506 is used to store input data and output data. The weight data directly accesses the controller (Direct Memory Access Controller, DMAC) 2505 through the storage unit, and the DMAC is transferred to the weight storage 2502 . The input data is also transferred to the unified memory 2506 through the DMAC.

The BIU is a Bus Interface Unit, that is, the bus interface unit 2510 , which is used for the interaction between the AXI bus, the DMAC and the instruction fetch buffer (Instruction Fetch Buffer, IFB) 2509 .

The bus interface unit 2510 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 2509 to obtain instructions from the external memory, and for the storage unit access controller 2505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

The DMAC is mainly used to move the input data in the external memory DDR to the unified memory 2506 , to move the weight data to the weight memory 2502 , or to move the input data to the input memory 2501 .

The vector computing unit 2507 includes a plurality of computing processing units, and if necessary, further processes the output of the computing circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison and so on. It is mainly used for non-convolutional/fully connected layer network calculations in neural networks, such as Batch Normalization (batch normalization), pixel-level summation, and upsampling of feature planes.

In some implementations, vector computation unit 2507 can store the vector of the processed output to unified memory 2506 . For example, the vector calculation unit 2507 can apply a linear function; or, a nonlinear function to the output of the operation circuit 2503, such as performing linear interpolation on the feature plane extracted by the convolution layer, and then for example, a vector of accumulated values to generate activation values. In some implementations, the vector computation unit 2507 generates normalized values, pixel-level summed values, or both. In some implementations, the vector of processed outputs can be used as an activation input to operational circuitry 2503, eg, for use in subsequent layers in a neural network.

An instruction fetch buffer (instruction fetch buffer) 2509 connected to the controller 2504 is used to store instructions used by the controller 2504;

The unified memory 2506, the input memory 2501, the weight memory 2502 and the fetch memory 2509 are all On-Chip memories. External memory is private to the NPU hardware architecture.

Wherein, the processor mentioned above can be a general-purpose central processing unit, microprocessor, ASIC, or one or more integrated circuits for controlling the execution of the above-mentioned programs.

In addition, it should be noted that the device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be A physical unit can be located in one place, or it can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the device embodiments provided in the present application, the connection relationship between the modules indicates that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines.

Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course it can also be realized by special hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, Special components, etc. to achieve. In general, all functions completed by computer programs can be easily realized by corresponding hardware, and the specific hardware structure used to realize the same function can also be varied, such as analog circuits, digital circuits or special-purpose circuit etc. However, for this application, software program implementation is a better implementation mode in most cases. Based on this understanding, the essence of the technical solution of this application or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a readable storage medium, such as a floppy disk of a computer , U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including several instructions to make a computer device (which can be a personal computer, training device, or network device, etc.) execute the instructions described in various embodiments of the present application method.

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product.

The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present application will be generated in whole or in part. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transferred from a website, computer, training device, or data The center transmits to another website site, computer, training device or data center via wired (eg, coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be stored by a computer, or a data storage device such as a training device or a data center integrated with one or more available media. The available medium may be a magnetic medium (for example, a floppy disk, a hard disk, or a magnetic tape), an optical medium (for example, DVD), or a semiconductor medium (for example, a solid state disk (Solid State Disk, SSD)).

Claims (23)

1. A method of model training, the method comprising:

acquiring a sample image;

sampling the sample image to obtain a first image block and a second image block, wherein the first image block and the second image block are different image blocks on the sample image;

respectively extracting the features of the first image block and the second image block through a feature extraction network based on the fact that the cross-over ratio of the first image block to the second image block on the sample image is larger than a threshold value, so as to obtain a first feature map and a second feature map;

determining loss according to the difference between the first feature map and the second feature map, and updating the feature extraction network based on the loss to obtain an updated feature extraction network.

2. The method of claim 1, wherein the threshold is a value greater than or equal to 0.4.

3. The method according to claim 1 or 2, wherein before determining the loss from the difference between the first profile and the second profile, the method further comprises:

aligning the first image block and the second image block to obtain an aligned first image block and an aligned second image block;

said determining a loss from a difference between said first feature map and said second feature map comprises:

determining a loss according to a difference between the aligned first feature map and the aligned second feature map.

4. The method according to claim 3, wherein the sample image includes a target area, the target area is an overlapping area where the first image block and the second image block are located on the sample image, and aligning the first image block and the second image block includes:

according to the target area, determining a first sub-feature map corresponding to the target area in the first feature map and a second sub-feature map corresponding to the target area in the second feature map;

performing upsampling on the first sub-feature map to obtain the aligned first image block;

and upsampling the second sub-feature map to obtain the aligned second image block, wherein the size of the aligned first image block is consistent with that of the aligned second image block.

5. The method of claim 3, wherein the first feature map and the second feature map are the same in size, the first feature map includes M first feature points, the second feature map includes M second feature points, the M first feature points correspond to M first pixel points in the sample image, the M second feature points correspond to M second pixel points in the sample image, the M first pixel points correspond to the M second pixel points one-to-one, and the method further comprises:

obtaining a third feature map according to the M first pixel points and the M second pixel points, wherein the third feature map and the second feature map have the same size, the third feature map comprises M third feature points, and each third feature point is obtained based on a pixel position difference between the first pixel point and the second pixel point having a corresponding relationship;

and fusing the third feature map and the first feature map to obtain the aligned first image block, wherein the second feature map is used as the aligned second image block.

6. The method of claim 5, wherein fusing the third feature map with the first feature map comprises:

and splicing the third characteristic map and the first characteristic map in the depth direction.

7. The method of any of claims 1 to 6, wherein determining the loss based on the difference between the first profile and the second profile comprises:

processing the M first feature points of the first feature map through a target prediction network to obtain a predicted value of each first feature point;

clustering M second feature points of the first feature map based on a target clustering algorithm to update feature values of the M second feature points, wherein the feature value of each updated second feature point is a clustering center feature value of a clustering category where the updated second feature point is located;

and determining loss according to the predicted value of each first characteristic point and the difference between the characteristic values of each updated second characteristic point.

8. The method of claim 7, further comprising:

and updating the target prediction network according to the loss.

9. The method of any one of claims 1 to 8, wherein the sample image comprises a plurality of objects, the objects comprising at least one of a person, a vehicle, a traffic sign, a lane line, a plant.

10. The method according to any one of claims 1 to 9, further comprising:

acquiring a target network and an image to be processed, wherein the target network comprises the updated feature extraction network and a downstream task network;

and processing the image to be processed through the target network to obtain a processing result.

11. A model training apparatus, the apparatus comprising:

the acquisition module is used for acquiring a sample image;

the sampling module is used for sampling the sample image to obtain a first image block and a second image block, wherein the first image block and the second image block are different image blocks on the sample image;

the characteristic extraction module is used for respectively extracting the characteristics of the first image block and the second image block through a characteristic extraction network based on the fact that the cross-over ratio of the first image block and the second image block between the areas of the sample image is larger than a threshold value so as to obtain a first characteristic diagram and a second characteristic diagram;

and the model updating module is used for determining loss according to the difference between the first feature map and the second feature map and updating the feature extraction network based on the loss so as to obtain an updated feature extraction network.

12. The apparatus of claim 11, wherein the threshold is a value greater than or equal to 0.4.

13. The apparatus of claim 11 or 12, further comprising:

an alignment module, configured to align the first image block and the second image block before determining a loss according to a difference between the first feature map and the second feature map, so as to obtain an aligned first image block and an aligned second image block;

the model update module is specifically configured to:

determining a loss according to a difference between the aligned first feature map and the aligned second feature map.

14. The apparatus according to claim 13, wherein the sample image includes a target area, the target area is an overlapping area where the first tile and the second tile are located on the sample image, and the alignment module is specifically configured to:

according to the target area, determining a first sub-feature map corresponding to the target area in the first feature map and a second sub-feature map corresponding to the target area in the second feature map;

performing upsampling on the first sub-feature map to obtain the aligned first image block;

and upsampling the second sub-feature map to obtain the aligned second image block, wherein the size of the aligned first image block is consistent with that of the aligned second image block.

15. The apparatus according to claim 13, wherein the first feature map and the second feature map have the same size, the first feature map includes M first feature points, the second feature map includes M second feature points, the M first feature points correspond to M first pixel points in the sample image, the M second feature points correspond to M second pixel points in the sample image, the M first pixel points correspond to the M second pixel points one-to-one, and the alignment module is specifically configured to:

obtaining a third feature map according to the M first pixel points and the M second pixel points, wherein the third feature map and the second feature map have the same size, the third feature map comprises M third feature points, and each third feature point is obtained based on a pixel position difference between the first pixel point and the second pixel point having a corresponding relationship;

and fusing the third feature map and the first feature map to obtain the aligned first image block, wherein the second feature map is used as the aligned second image block.

16. The apparatus according to claim 15, wherein the alignment module is specifically configured to:

and splicing the third feature map and the first feature map in the depth direction.

17. The apparatus according to any one of claims 11 to 16, wherein the model update module is specifically configured to:

processing the M first feature points of the first feature map through a target prediction network to obtain a predicted value of each first feature point;

clustering M second feature points of the first feature map based on a target clustering algorithm to update feature values of the M second feature points, wherein the feature value of each updated second feature point is a clustering center feature value of a clustering category where the updated second feature point is located;

and determining the loss according to the difference between the predicted value of each first characteristic point and the characteristic value of each updated second characteristic point.

18. The apparatus of claim 17, wherein the model update module is further configured to:

and updating the target prediction network according to the loss.

19. The apparatus of any of claims 11 to 18, wherein the sample image comprises a plurality of objects, the objects comprising at least one of a person, a vehicle, a traffic sign, a lane line, a plant.

20. The apparatus of any one of claims 11 to 19, further comprising:

the data processing module is used for acquiring a target network and an image to be processed, wherein the target network comprises the updated feature extraction network and a downstream task network;

and processing the image to be processed through the target network to obtain a processing result.

21. A model training apparatus, the apparatus comprising a memory and a processor; the memory stores code, and the processor is configured to retrieve the code and perform the method of any of claims 1 to 10.

22. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 10.

23. A computer program product, characterized in that it comprises code for implementing the steps of the method of any one of claims 1 to 10 when said code is executed.

Images