CN116824342A - Image processing method and device - Google Patents

Abstract

The embodiment of the application provides an image processing method and device, which relate to the technical field of deep learning, and the method comprises the following steps: when the image needs to be processed, a preset model library file is called; the preset model library file comprises: a target model file of a target deep learning model corresponding to each of the plurality of preset image dimension information; the target model file contains weight parameters of the target deep learning model; according to the memory size required by the target deep learning model to process the image with the largest preset image dimension information, distributing a target memory space for processing each image to be processed; for each image to be processed, determining a target model file corresponding to preset image dimension information matched with the image to be processed from preset model library files; and processing the image to be processed based on the determined target model file and the target memory space. In this way, the efficiency of image processing can be improved.

Description

An image processing method and device

Technical field

This application relates to the field of deep learning technology, and in particular to an image processing method and device.

Background technique

In related technologies, image processing based on a deep learning model may include the following steps: step 1: memory allocation; step 2: model loading; step 3: forward inference, and step 4: memory release. The allocated memory is used to store the input data, output data, and deep learning model of the deep learning model.

When processing images based on a deep learning model, the size of the memory space required is related to the image dimension information of the input image. Therefore, when the input image to be processed corresponds to different image dimension information, the image processing device needs to execute the above in a loop. From step one to step four, the image processing device needs to perform memory allocation and memory release frequently and many times to adapt to the images to be processed with different image dimension information. The process of memory application and memory release will consume more time, which will further reduce the efficiency of image processing.

Contents of the invention

The purpose of the embodiments of the present application is to provide an image processing method and device that can improve the efficiency of image processing. The specific technical solutions are as follows:

In a first aspect, in order to achieve the above objectives, embodiments of the present application disclose an image processing method, which method includes:

When an image needs to be processed, a preset model library file is called; wherein the preset model library file includes: a target model file of a target deep learning model corresponding to each of multiple preset image dimension information; the target model file Contains the weight parameters of the target deep learning model;

According to the memory size required by the target deep learning model to process the image with the largest preset image dimension information, allocate the target memory space for processing each image to be processed;

For each image to be processed, determine a target model file corresponding to the preset image dimension information matching the image to be processed from the preset model library file;

Based on the determined target model file and the target memory space, the image to be processed is processed.

Optionally, before determining the target model file corresponding to the preset image dimension information matching the image to be processed from the preset model library file, the method further includes:

For each image to be processed, determine whether there is first image dimension information consistent with the image dimension information of the image to be processed in each preset image dimension information;

If the first image dimension information exists, it is determined that the first image dimension information matches the image to be processed;

If the first image dimension information does not exist, it is determined that the second image dimension information in the plurality of preset image dimension information matches the image to be processed; wherein, among the plurality of preset image dimension information, the The second image dimension information is not less than the image dimension information of the image to be processed, and the difference between the second image dimension information and the image dimension information of the image to be processed is the smallest.

Optionally, the image dimension information includes the number of batches, the number of channels, and image height and image width;

Determining that the second image dimension information among the plurality of preset image dimension information matches the image to be processed includes:

From the plurality of preset image dimension information, determine the image dimension information that is not smaller than the image to be processed in the number of batches, the number of channels, and the dimensions of image height and image width, as the third image dimension information;

Second image dimension information matching the image to be processed is determined from the third image dimension information based on differences in the dimensions of the batch number, the number of channels, and the image height and image width with the image to be processed.

Optionally, the third image matching the image to be processed is determined from the third image dimension information based on the difference between the image to be processed in the batch number, the number of channels, and the dimensions of image height and image width. Two image dimension information, including:

From the third image dimension information, determine the image dimension information with the smallest difference in the batch number dimension from the image to be processed as the fourth image dimension information;

From the fourth image dimension information, determine the image dimension information with the smallest difference in the dimension of the number of channels from the image to be processed as the fifth image dimension information;

Among the fifth image dimension information, the image dimension information with the smallest difference in the dimensions of image height and image width from the image to be processed is determined as the second image dimension information matching the image to be processed.

Optionally, the construction process of the preset model library file includes:

Obtain the dimension information of each preset image and the initial model file representing the weight parameters of the initial deep learning model;

For each preset image dimension information, according to the initial model file, a model file of the target deep learning model matching the preset image dimension information is generated as a target model file; wherein, matching the preset image dimension information The matched target deep learning model represents: a deep learning model obtained by optimizing the initial deep learning model based on the preset image dimension information;

Based on encapsulating each target model file, the preset model library file is obtained.

Optionally, the preset model library file is obtained by encapsulating each target model file, including:

Each target model file and the model loading instruction are encapsulated to obtain the preset model library file; wherein the model loading instruction is used to call the target model file that needs to be run from the preset model library file; so The above-mentioned default model library file contains the common parts in each target model file, and the private parts in each target model file.

Optionally, the preset model library file is a dynamic library file or a static library file.

In a second aspect, in order to achieve the above object, an embodiment of the present application discloses an image processing device, which includes:

The preset model library file calling module is used to call the preset model library file when the image needs to be processed; wherein the preset model library file includes: target deep learning models corresponding to multiple preset image dimension information. The target model file; the target model file contains the weight parameters of the target deep learning model;

The target memory space allocation module is used to allocate the target memory space for processing each image to be processed according to the memory size required for processing the image with the largest preset image dimension information according to the target deep learning model;

A target model file determination module, configured to determine, for each image to be processed, the target model file corresponding to the preset image dimension information matching the image to be processed from the preset model library file;

The image processing module is used to process the image to be processed based on the determined target model file and the target memory space.

Optionally, the device also includes:

A judgment module configured to judge each preset image dimension for each image to be processed before determining the target model file corresponding to the preset image dimension information matching the image to be processed from the preset model library file. In the information, whether there is first image dimension information consistent with the image dimension information of the image to be processed;

A first processing module configured to, if the first image dimension information exists, determine that the first image dimension information matches the image to be processed;

A second processing module configured to, if the first image dimension information does not exist, determine that the second image dimension information in the plurality of preset image dimension information matches the image to be processed; wherein, in the plurality of preset image dimension information, It is assumed that among the image dimension information, the second image dimension information is not less than the image dimension information of the image to be processed, and the difference between the second image dimension information and the image dimension information of the image to be processed is the smallest.

Optionally, the image dimension information includes the number of batches, the number of channels, and image height and image width;

The second processing module includes:

The third image dimension information determination submodule is used to determine, from multiple preset image dimension information, image dimension information that is no less than the image to be processed in the number of batches, the number of channels, and the dimensions of image height and image width. , as the third image dimension information;

The second image dimension information determination submodule is used to determine the difference between the image to be processed and the image to be processed from the third image dimension information based on the differences in the dimensions of the number of batches, the number of channels, and the image height and image width. The image matches the second image dimension information.

Optionally, the second image dimension information determination sub-module is specifically configured to determine, from the third image dimension information, the image dimension information that has the smallest difference with the image to be processed in the dimension of the batch number, as the third image dimension information. Four image dimension information;

From the fourth image dimension information, determine the image dimension information with the smallest difference in the dimension of the number of channels from the image to be processed as the fifth image dimension information;

Among the fifth image dimension information, the image dimension information with the smallest difference in the dimensions of image height and image width from the image to be processed is determined as the second image dimension information matching the image to be processed.

Optionally, the device also includes:

The preset model library file construction module is used to obtain the dimension information of each preset image and the initial model file representing the weight parameters of the initial deep learning model;

For each preset image dimension information, according to the initial model file, a model file of the target deep learning model matching the preset image dimension information is generated as a target model file; wherein, matching the preset image dimension information The matched target deep learning model represents: a deep learning model obtained by optimizing the initial deep learning model based on the preset image dimension information;

Based on encapsulating each target model file, the preset model library file is obtained.

Optionally, the preset model library file construction module is specifically used to encapsulate each target model file and a model loading instruction to obtain the preset model library file; wherein the model loading instruction is used to obtain the preset model library file from The target model file that needs to be run is called in the preset model library file; the preset model library file includes the common parts in each target model file and the private parts in each target model file.

Optionally, the preset model library file is a dynamic library file or a static library file.

In another aspect of the implementation of the present application, in order to achieve the above object, an embodiment of the present application also discloses an electronic device. The electronic device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the The communication interface, the memories complete mutual communication through the communication bus;

The memory is used to store computer programs;

The processor is configured to implement any of the above image processing methods when executing a program stored on the memory.

In yet another aspect of the implementation of the present application, a computer-readable storage medium is also provided. A computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, any of the above-mentioned methods are implemented. Image processing methods.

Embodiments of the present application also provide a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the above image processing methods.

Beneficial effects of the embodiments of this application:

The image processing method provided by the embodiment of the present application calls a preset model library file when an image needs to be processed; wherein the preset model library file includes: the target deep learning model corresponding to each of the plurality of preset image dimension information. Model file; the target model file contains the weight parameters of the target deep learning model; according to the memory size required by the target deep learning model to process the image with the largest preset image dimension information, allocate the target for processing each image to be processed Memory space; for each image to be processed, determine the target model file corresponding to the preset image dimension information that matches the image to be processed from the preset model library file; based on the determined target model file, and the target memory space , process the image to be processed.

Based on the above processing, since the allocated target memory space is determined according to the memory size required to process the image with the largest preset image dimension information, therefore, for any image input that is smaller than the largest preset image dimension information, the The target memory space can meet the memory size required to process the image. That is, only one memory allocation is needed to process multiple input images of different sizes that are smaller than the largest preset image dimension information. There is no need to perform frequent memory allocation and release, which can improve image quality. processing efficiency.

Of course, implementing any product or method of the present application does not necessarily require achieving all the above-mentioned advantages simultaneously.

Description of the drawings

In order to explain the embodiments of the present application or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other embodiments can be obtained based on these drawings.

Figure 1 is a flow chart of an image processing method provided by an embodiment of the present application;

Figure 2 is a flow chart of another image processing method provided by an embodiment of the present application;

Figure 3A is a flow chart of another image processing method provided by an embodiment of the present application;

Figure 3B is a flow chart of another image processing method provided by an embodiment of the present application;

Figure 4 is a schematic diagram of the principle of image processing provided by an embodiment of the present application;

Figure 5 is a flow chart of image processing provided by an embodiment of the present application;

Figure 6 is a flow chart for determining the preset image dimension information closest to the input image provided by an embodiment of the present application;

Figure 7 is a structural diagram of an image processing device provided by an embodiment of the present application;

Figure 8 is a structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art based on this application fall within the scope of protection of this application.

The embodiment of the present application provides an image processing method, which can be applied to an electronic device, and the electronic device can process images based on a deep learning model. The deep learning model involved in the embodiments of this application may be a CNN (Convolutional Neural Networks) model or an RNN (Rerrent Neural Network) model, but is not limited thereto. Correspondingly, processing the image can be to identify the category of the object (for example, a person, a vehicle, etc.) in the image, or it can also be to identify the image area to which the object in the image belongs, or it can also be to identify whether there is a specified object in the image. objects, but are not limited to this.

Referring to Figure 1, Figure 1 is a flow chart of an image processing method provided by an embodiment of the present application. The method may include the following steps:

S101: When the image needs to be processed, call the preset model library file.

The preset model library file includes: target model files of target deep learning models corresponding to multiple preset image dimension information. The target model file contains the weight parameters of the target deep learning model.

S102: According to the memory size required by the target deep learning model to process the image with the largest preset image dimension information, allocate the target memory space for processing each image to be processed.

S103: For each image to be processed, determine the target model file corresponding to the preset image dimension information matching the image to be processed from the preset model library file.

S104: Based on the determined target model file and the target memory space, process the image to be processed.

Based on the image processing method provided by the embodiment of the present application, since the allocated target memory space is determined according to the memory size required to process the image with the largest preset image dimension information, therefore, for the input smaller than the largest preset image dimension For any image of information, the target memory space can satisfy the memory size required to process the image. That is, only one memory allocation is needed to process multiple input images of different sizes that are smaller than the largest preset image dimension information. There is no need to perform frequent memory allocation and release, which can improve image quality. processing efficiency.

Regarding step S101, each target model file may also include the relationship between each network layer in the corresponding target deep learning model, and the network parameters of each network layer.

Multiple preset image dimension information can be set according to requirements. For example, it can be set to the image dimension information of images commonly used in current business.

In one embodiment, image dimension information includes batch number, channel number, and image height and image width. That is, the image dimension information may include N (Num, batch number), C (Channel, channel), H (High, height), and W (Width, width). N represents the number of batch processing, that is, the number of (image frames/feature maps) processed simultaneously per unit time based on the deep learning model, C represents the number of channels, H represents the height of the image frame, and W represents the width of the image frame.

Regarding step S102, when processing an image based on a deep learning model, it is necessary to obtain input data (ie, the input image), and output data (ie, the processing result), as well as intermediate cache data (for example, the feature map generated by the network layer) . Therefore, before processing images based on the deep learning model, memory space needs to be allocated to the deep learning model in advance to store the above data. The memory space required by the deep learning model to process images is related to the image dimension information of the input image. That is, the greater the image dimension information of the input image, the greater the memory space required; the image dimension of the input image The smaller the information, the smaller the memory space required.

In the embodiment of the present application, the memory size required for processing the image with the largest preset image dimension information based on the deep learning model can be calculated, and the memory space (ie, the target memory space) is allocated according to the memory size. Furthermore, for any input image that is smaller than the largest preset image dimension information, the target memory space can satisfy the memory space required for processing the image.

In one embodiment, before the above step S102, the electronic device may also perform other initialization processing, such as image loading, etc.

Regarding step S103 and step S104, the image to be processed represents an input image that needs to be processed, and can be an image collected by other devices, or it can also be a feature map.

After allocating the target memory space, the electronic device can process multiple images to be processed with different image dimension information based on the target memory space.

In one implementation, the electronic device can process each acquired image to be processed in sequence. For example, for each image to be processed, the electronic device can determine the image dimension information that matches the image to be processed from each preset image dimension information, that is, the image dimension that matches the image dimension information of the image to be processed. information. Furthermore, the electronic device can obtain the target model file corresponding to the determined image dimension information from the preset model library file, and then the electronic device can load the target model file to run the target deep learning model corresponding to the image dimension information, and Based on the allocated target memory space, the image to be processed is processed. Based on the above processing, memory space multiplexing can be achieved when images are processed multiple times.

In one embodiment, after completing processing of each image to be processed, the electronic device can release the target memory space to avoid wasting memory space.

In one embodiment, referring to Figure 2, based on Figure 1, before the above step S103, the method may also include the following steps:

S105: For each image to be processed, determine whether there is first image dimension information consistent with the image dimension information of the image to be processed in each preset image dimension information; if yes, perform step S106; if not, perform step S106. S107.

S106: Determine that the first image dimension information matches the image to be processed.

S107: Determine that the second image dimension information among the plurality of preset image dimension information matches the image to be processed.

Among the plurality of preset image dimension information, the second image dimension information is not less than the image dimension information of the image to be processed, and the difference between the second image dimension information and the image dimension information of the image to be processed is the smallest.

In the embodiment of the present application, if there is image dimension information that is consistent with the image to be processed (ie, the first image dimension information) in the plurality of preset image dimension information, the electronic device can determine that the first image dimension information is consistent with the image dimension information to be processed. The image matches. That is to say, the image to be processed matches the target deep learning model corresponding to the first image dimension information. Furthermore, when processing the processed image, the electronic device can load the target model file corresponding to the first image dimension information to run the target deep learning model corresponding to the first image dimension information to process the image to be processed. In this way, the efficiency of the target deep learning model in processing the image to be processed can be improved.

If the first image dimension information does not exist, the electronic device determines that the second image dimension information among the plurality of preset image dimension information matches the image to be processed. Since the second image dimension information is not less than the image dimension information of the image to be processed, and the difference between the second image dimension information and the image dimension information of the image to be processed is the smallest, that is, the second image dimension information and the image dimension information of the image to be processed closest. Therefore, when processing the processed image, the electronic device can load the target model file corresponding to the second image dimension information to run the target deep learning model corresponding to the second image dimension information to process the image to be processed. . In this way, when the target deep learning model corresponding to the second image dimension information processes the image to be processed, the electronic device only needs to perform minor optimization processing, which can improve the efficiency of the target deep learning model to process the image to be processed. efficiency.

In one embodiment, the image dimension information includes a batch number, a channel number, as well as image height and image width. Correspondingly, referring to Figure 3A, based on Figure 2, the above step S107 may include the following steps:

S1071: From multiple preset image dimension information, determine the image dimension information that is not smaller than the image to be processed in the dimensions of batch number, channel number, image height and image width, as the third image dimension information.

S1072: Determine the second image dimension information matching the image to be processed from the third image dimension information based on the differences in the dimensions of the batch number, the number of channels, and the image height and image width of the image to be processed. .

The third image dimension information may be one or multiple. If the third image dimension information is one, the electronic device directly determines that the third image dimension information matches the image to be processed.

If there are multiple third image dimension information, each third image dimension information is not less than the image dimension information of the image to be processed in the four dimensions of batch number, channel number, image height and image width. Therefore, the electronic device can select one image dimension information from each third image dimension information to process the image to be processed based on the corresponding target deep learning model.

For example, the electronic device can compare each preset image dimension information with the image dimension information of the image to be processed, and filter out the images that are no less than the number of batches, the number of channels, and the dimensions of image height and image width. The preset image dimension information of the image to be processed is used to obtain the third image dimension information.

In one implementation, for each third image dimension information, the electronic device can calculate the difference between it and the image to be processed in the four dimensions of batch number, channel number, image height and image width, and then, according to The above four differences are used to calculate the total difference between the third image dimension information and the image dimension information of the image to be processed. For example, the electronic device may calculate the weighted sum of the above four differences as the total difference. Furthermore, the electronic device can determine the third image dimension information corresponding to the smallest total difference value as the second image dimension information matching the image to be processed.

In another implementation, after determining each third image dimension information, the electronic device can also randomly select one image dimension information from each third image dimension information, and perform deep learning based on the target corresponding to the selected image dimension information. The model processes the image to be processed.

In one implementation, after determining the third image dimension information, the electronic device may also determine the second image dimension information matching the image to be processed from the third image dimension information in a specified dimensional order.

In one embodiment, the specified dimension order can represent the number of batches, the number of channels, image height, and image width. Correspondingly, see Figure 3B. Based on Figure 3A, the above step S1072 can include the following steps:

S10721: From the third image dimension information, determine the image dimension information with the smallest difference in the batch number dimension from the image to be processed as the fourth image dimension information.

S10722: From the fourth image dimension information, determine the image dimension information with the smallest difference in the dimension of the number of channels from the image to be processed as the fifth image dimension information.

S10723: Determine the image dimension information that has the smallest difference in the dimensions of image height and image width with the image to be processed among the fifth image dimension information, as the second image dimension information matching the image to be processed.

In one implementation, since the number of batch processes has the greatest impact on the memory occupied, the electronic device can determine the second image dimension information in the order of the number of batch processes, the number of channels, the image height and the image width.

For example, the electronic device determines, from the third image dimension information, the image dimension information that has the smallest difference in the dimension of the batch number with the image to be processed (ie, the fourth image dimension information). If the fourth image dimension information is one, the electronic device directly determines that the fourth image dimension information matches the image to be processed.

If there are multiple fourth image dimension information, the electronic device determines from the fourth image dimension information the image dimension information with the smallest difference in the dimension of the number of channels from the image to be processed (that is, the fifth image dimension information). If the fifth image dimension information is one, the electronic device directly determines that the fifth image dimension information matches the image to be processed.

If there are multiple fifth image dimension information, the electronic device determines that among the fifth image dimension information, the image dimension information with the smallest difference between the image height and the image width of the image to be processed matches the image to be processed. .

Based on the above processing, it is not necessary to list all the image dimension information. That is, the preset model library file does not need to contain the target model files corresponding to all image dimension information. Through dynamic matching, the nearest neighbor can be selected. Image dimension information is used for forward reasoning, which can improve the efficiency of image processing while realizing memory space reuse.

In addition, the specified dimension order can also be the number of batches, image height, image width, and number of channels, or it can also be the image height, image width, number of channels, and number of batches, but is not limited to this. Correspondingly, the method of determining the second image dimension information may refer to the above steps S10721-S10723.

Because the determined second image dimension information is not consistent with the image dimension information of the image to be processed, that is, the second image dimension information is greater than the image dimension information of the image to be processed. Therefore, when processing the image to be processed based on the target deep learning model corresponding to the second image dimension information, optimization processing can be performed.

For example, if the second image dimension information is greater than the image dimension information of the image to be processed in the batch number dimension, the input data can be supplemented (for example, image frames containing data of 0 are supplemented). Correspondingly, after obtaining the output data, the output data corresponding to the supplementary data can be trimmed to obtain the final output result.

For example, if the second image dimension information is greater than the image dimension information of the image to be processed in the dimensions of image height and image width, then filling processing can be performed on each image frame in the image to be processed (for example, the data of the filling part is 0 ). Correspondingly, after obtaining the output data, the output data corresponding to the padding part can be cropped to obtain the final output result.

Based on the above processing, since the difference between the second image dimension information and the image dimension information of the image to be processed is minimal, the electronic device only needs to perform less processing when processing the image to be processed based on the target deep learning model corresponding to the second image dimension information. The optimization process can improve the efficiency of the target deep learning model in processing the image to be processed.

In one embodiment, the construction process of the preset model library file includes:

Step 1: Obtain the dimension information of each preset image and the initial model file representing the weight parameters of the initial deep learning model.

Step 2: For each preset image dimension information, according to the initial model file, generate a model file of the target deep learning model that matches the preset image dimension information as a target model file.

Step 3: Obtain the preset model library file based on encapsulation of each target model file.

Wherein, the target deep learning model matching the preset image dimension information represents: a deep learning model obtained by optimizing the initial deep learning model based on the preset image dimension information.

In this embodiment of the present application, the default model library file may be pre-generated for the electronic device, or may be pre-generated for other devices.

In one implementation, the electronic device can pre-train the deep learning model of the initial structure based on the training sample set (ie, the sample image) to obtain the initial deep learning model. Then, the weight parameters of this initial deep learning model can be obtained.

In order to improve the efficiency of processing images to be processed based on the target deep learning model, each preset image dimension information can be set according to the preset step size. For example, in each preset image dimension information, the number of batch processing can be set to 1, 2, 4, 8, 16, 24, and 32. The image height and image width can be set to commonly used resolutions (such as 540P, 720P, 1080P). For example, the maximum preset image dimension information can be 32*3*1080*1920, and the maximum supported image processing size is 32*3*1080*1920.

In one implementation, the electronic device can generate a preset model library file through a model generation engine. Accordingly, each preset image dimension information can be recorded in the preset json file to facilitate loading by the model generation engine. In addition, the json file can also record the data format of the input image, for example, it can be Int8 format, or it can be FP16 format, or it can also be Float format.

Furthermore, for each preset image dimension information, the model generation engine can optimize the initial deep learning model based on the initial model file. In one implementation, the electronic device can fuse the network layers in the initial deep learning model based on the above data. For example, fuse the Conv layer (convolution layer), BN (Batch Normalization, batch normalization) and Relu (Rectified Linear Unit, linear rectification function) layer; and/or, fuse the Concat (fully connected) layer and convolution layers are merged.

Based on the above processing, it is possible to realize the weight sharing of each network layer in the deep learning model, reduce the space occupied by each target model file, and reduce the space occupied by the preset model library file. Correspondingly, if image processing is performed based on NPU (Nneural-network Processing Units, embedded neural network processor), the utilization of NPU on-chip resources can also be improved.

In addition, for each preset image dimension information, the electronic device can also determine the dimensions of the input and output data of each network layer in the corresponding target deep learning model, and determine the relationship between each network layer based on the relationship between each network layer. Memory reuse method, record the memory reuse method in the corresponding target model file.

In one embodiment, for each preset image dimension information, the electronic device can determine the dimensions of the input and output data of each network layer in the corresponding target deep learning model, and combined with the current hardware characteristics (such as on-chip memory size), determine How data is loaded for each network layer. For example, for each network layer, if the dimension of the input data of the network layer is greater than the threshold, the loading method of the data recorded in the corresponding target model file can be block loading processing. This threshold can be determined based on the on-chip memory size mentioned above. For example, load 1/4 of the input data each time, or load 1/8 of the input data each time. If the dimension of the input data of the network layer is less than the threshold, the data loading method recorded in the corresponding target model file can be full loading, that is, the entire input data is loaded directly.

In one embodiment, the model generation engine can also support user-defined plug-ins, and generate a preset model library file together with user-defined network layer configuration information (for example, weight information, implementation methods, etc.), thereby improving The flexibility of the generated preset model library files can also improve the flexibility of image processing. Among them, the implementation method may include the memory multiplexing method between network layers and the data loading method of the network layer.

In one embodiment, in order to further reduce the space occupied by the preset model library file, the above-mentioned step three may include the following steps: encapsulate each target model file and the model loading instruction to obtain the preset model library file.

Among them, the model loading instruction is used to call the target model file that needs to be run from the preset model library file. The default model library file contains the common parts in each target model file and the private parts in each target model file.

In the embodiment of the present application, the target deep learning model corresponding to each preset image dimension information has the same part. Therefore, in order to further reduce the space occupied by the preset model library file, in the preset model library file, for each Only one copy of the same part (that is, the common part) in the target model file can be recorded. Correspondingly, parts of each target model file that are different from other target model files (ie, private parts) can be recorded separately.

For example, if the common part represents the same network layer in multiple target model files, then in the default model library file, only one copy of the same network layer is recorded for the multiple target model files, and the network layer is marked as Common parts of multiple target model files. Correspondingly, in the default model library file, network layers other than the same network layer in the multiple target model files can also be recorded, and the target model files to which such network layers belong respectively can be marked.

In one implementation, for the above-mentioned public part and private part, the electronic device can read in binary mode and write it into a c file. Then, the model loading instruction (for example, vload instruction) can be encapsulated together with the above-mentioned data. Get the preset model library file. For example, the electronic device can encapsulate the above data in a Fatbin manner. Alternatively, for the above-mentioned public and private parts, the electronic device can also read in binary form and write it into a C++ file to generate a preset model library file. In this application, the file types of the written files are not limited to c files and c++ files, and can also be other types.

In one embodiment, the preset model library file may be a dynamic library file, that is, the electronic device may compile the above data into a dynamic library file to obtain the preset model library file.

For example, you can use the following instructions to generate a dynamic library file:

gcc-shared–fpic XXX.c lib XXX.so

In the above instructions, shared indicates that the output result is a shared library type; fpic indicates that the output file is generated using address-independent code; XXX indicates the name of the generated dynamic library file.

In one implementation, the code corresponding to the process of forward inference (that is, processing the input image) can be encapsulated into a library file to obtain an inference library. Electronic devices can implement forward reasoning by calling the reasoning library. If the preset model library file is a dynamic library file, during the process of calling the inference library, the electronic device can link the above preset model library file through dynamic linking to process the image based on the target deep learning model. Based on the dynamic link method, only the entry address of the preset model library file needs to be recorded in the inference library, and the entire preset model library file does not need to be recorded. Therefore, the memory space occupied by the inference library can be saved.

In one embodiment, the preset model library file may be a static library file, that is, the electronic device may compile the above data into a static library file to obtain the preset model library file.

For example, you can use the following instructions to generate a static library file:

gcc–c XXX.c

ar–cr XXX.a XXX.o

In the above instructions, ar–cr means to package multiple compiled files into a static library file, and XXX means the name of the generated static library file.

Referring to Figure 4, Figure 4 is a schematic diagram of the principle of image processing provided by an embodiment of the present application.

The offline implementation part includes the process of generating preset model library files before the deep learning model is put online. For example, the model generation engine can be used to generate model files corresponding to each preset image dimension information based on the json file package (recorded with each preset image dimension information) and the weight file. Pack each model file and model loading instructions to generate an so library (i.e. dynamic library) to obtain the preset model library file. In addition, when generating model files, the model generation engine can also support user-defined plug-ins and generate preset model library files together with user-defined network layer configuration information (such as weight information, implementation methods, etc.), and then, Able to improve the flexibility of generated preset model library files.

The forward reasoning part includes the process of processing images after the deep learning model is online. For example, after obtaining the input data (for example, an image), you can call the inference library interface and call the inference library (that is, a library that encapsulates the code corresponding to the forward inference process) to process the input data. Specifically, during processing, the generated so library can be called to process the input data based on the model file that matches the input data and obtain the output result.

Referring to Figure 5, Figure 5 is a flow chart of image processing provided by an embodiment of the present application.

Loading the model means calling a preset model library file. The preset model library file includes multiple model files corresponding to preset image dimension information. Furthermore, the maximum memory space is allocated, that is, the memory space is allocated according to the memory size required to process the image with the largest preset image dimension information. Then, it can be determined whether there is preset image dimension information consistent with the input image in each preset image dimension information.

If it exists, based on the allocated memory space, the input image is processed with the model file corresponding to the consistent preset image dimension information.

If it does not exist, based on the allocated memory space, the model file corresponding to the preset image dimension information closest to the input image is selected to process the input image. The preset image dimension information that is closest to the input image, that is, the preset image dimension information that is not smaller than the input image and has the smallest difference with the image dimension information of the input image. In addition, the output data can also be cropped, that is, when processing the input image, the input image can be filled in, and/or the input data can be supplemented. Correspondingly, after the output data is obtained, the input image can be processed. The output data corresponding to the padding part and/or the supplementary part is cropped.

After outputting the processing results, the allocated memory space can be released.

Referring to FIG. 6 , FIG. 6 is a flow chart for determining the preset image dimension information closest to the input image provided by an embodiment of the present application.

When obtaining the image dimension information of the input image (i.e. N, C, H, W), you can compare N, C, H, W one by one to determine whether there are four dimensions in each preset image dimension information that are not smaller than the input The third image dimension information of the image dimension information of the image.

If it does not exist, it is determined that processing of the input image is not supported.

If it exists, determine the fourth image dimension information with the smallest difference in N dimensions from the input image from the third image dimension information. Furthermore, from the fourth image dimension information, determine the fifth image dimension information that has the smallest difference with the input image in the C dimension, and then determine the fifth image dimension information that has the smallest difference with the input image in the H and W dimensions. The image dimension information with the smallest difference in dimension is used as the preset image dimension information closest to the input image.

Based on the same inventive concept, an embodiment of the present application also provides an image processing device. See Figure 7. Figure 7 is a structural diagram of an image processing device provided by an embodiment of the present application. The device may include:

The preset model library file calling module 701 is used to call the preset model library file when an image needs to be processed; wherein the preset model library file includes: target deep learning corresponding to each of multiple preset image dimension information. The target model file of the model; the target model file contains the weight parameters of the target deep learning model;

The target memory space allocation module 702 is used to allocate the target memory space for processing each image to be processed according to the memory size required by the target deep learning model to process the image with the largest preset image dimension information;

The target model file determination module 703 is configured to determine, for each image to be processed, the target model file corresponding to the preset image dimension information matching the image to be processed from the preset model library file;

The image processing module 704 is used to process the image to be processed based on the determined target model file and the target memory space.

Optionally, the device also includes:

A judgment module configured to judge each preset image dimension for each image to be processed before determining the target model file corresponding to the preset image dimension information matching the image to be processed from the preset model library file. In the information, whether there is first image dimension information consistent with the image dimension information of the image to be processed;

A first processing module configured to, if the first image dimension information exists, determine that the first image dimension information matches the image to be processed;

A second processing module configured to, if the first image dimension information does not exist, determine that the second image dimension information in the plurality of preset image dimension information matches the image to be processed; wherein, in the plurality of preset image dimension information, It is assumed that among the image dimension information, the second image dimension information is not less than the image dimension information of the image to be processed, and the difference between the second image dimension information and the image dimension information of the image to be processed is the smallest.

Optionally, the image dimension information includes the number of batches, the number of channels, and image height and image width;

The second processing module includes:

The third image dimension information determination submodule is used to determine, from multiple preset image dimension information, image dimension information that is no less than the image to be processed in the number of batches, the number of channels, and the dimensions of image height and image width. , as the third image dimension information;

The second image dimension information determination submodule is used to determine the difference between the image to be processed and the image to be processed from the third image dimension information based on the differences in the dimensions of the number of batches, the number of channels, and the image height and image width. The image matches the second image dimension information.

Optionally, the second image dimension information determination sub-module is specifically configured to determine, from the third image dimension information, the image dimension information that has the smallest difference with the image to be processed in the dimension of the batch number, as the third image dimension information. Four image dimension information;

From the fourth image dimension information, determine the image dimension information with the smallest difference in the dimension of the number of channels from the image to be processed as the fifth image dimension information;

Among the fifth image dimension information, the image dimension information with the smallest difference in the dimensions of image height and image width from the image to be processed is determined as the second image dimension information matching the image to be processed.

Optionally, the device also includes:

The preset model library file construction module is used to obtain the dimension information of each preset image and the initial model file representing the weight parameters of the initial deep learning model;

For each preset image dimension information, according to the initial model file, a model file of the target deep learning model matching the preset image dimension information is generated as a target model file; wherein, matching the preset image dimension information The matched target deep learning model represents: a deep learning model obtained by optimizing the initial deep learning model based on the preset image dimension information;

Based on encapsulating each target model file, the preset model library file is obtained.

Optionally, the preset model library file construction module is specifically used to encapsulate each target model file and a model loading instruction to obtain the preset model library file; wherein the model loading instruction is used to obtain the preset model library file from The target model file that needs to be run is called in the preset model library file; the preset model library file includes the common parts in each target model file and the private parts in each target model file.

Optionally, the preset model library file is a dynamic library file or a static library file.

The embodiment of the present application also provides an electronic device, as shown in Figure 8, including a processor 801, a communication interface 802, a memory 803, and a communication bus 804. The processor 801, the communication interface 802, and the memory 803 communicate through the communication bus 804. complete mutual communication,

Memory 803, used to store computer programs;

The processor 801 is used to execute the program stored in the memory 803 to implement the following steps:

When an image needs to be processed, a preset model library file is called; wherein the preset model library file includes: a target model file of a target deep learning model corresponding to each of multiple preset image dimension information; the target model file Contains the weight parameters of the target deep learning model;

According to the memory size required by the target deep learning model to process the image with the largest preset image dimension information, allocate the target memory space for processing each image to be processed;

For each image to be processed, determine a target model file corresponding to the preset image dimension information matching the image to be processed from the preset model library file;

Based on the determined target model file and the target memory space, the image to be processed is processed.

The communication bus mentioned in the above-mentioned electronic equipment may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the above-mentioned electronic devices and other devices.

The memory may include random access memory (Random Access Memory, RAM) or non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far away from the aforementioned processor.

The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (Network Processor, NP), etc.; it can also be a digital signal processor (Digital SignalProcessor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

In yet another embodiment provided by the present application, a computer-readable storage medium is also provided. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, any of the above image processing methods can be implemented. A step of.

In yet another embodiment provided by this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the image processing methods in the above embodiments.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, e.g., the computer instructions may be transferred from a website, computer, server, or data center Transmission to another website, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), etc.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations are mutually exclusive. any such actual relationship or sequence exists between them. Furthermore, the terms "comprises," "comprises," or any other variations thereof are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also those not expressly listed other elements, or elements inherent to the process, method, article or equipment. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article, or apparatus that includes the stated element.

Each embodiment in this specification is described in a related manner. The same and similar parts between the various embodiments can be referred to each other. Each embodiment focuses on its differences from other embodiments. In particular, the device, electronic equipment, computer-readable storage medium, and computer program product embodiments are described simply because they are basically similar to the method embodiments. For relevant details, please refer to the partial description of the method embodiments.

The above descriptions are only preferred embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application are included in the protection scope of this application.

Claims (10)

1. An image processing method, characterized in that the method includes: When an image needs to be processed, a preset model library file is called; wherein the preset model library file includes: a target model file of a target deep learning model corresponding to each of multiple preset image dimension information; the target model file Contains the weight parameters of the target deep learning model; According to the memory size required by the target deep learning model to process the image with the largest preset image dimension information, allocate the target memory space for processing each image to be processed; For each image to be processed, determine a target model file corresponding to the preset image dimension information matching the image to be processed from the preset model library file; Based on the determined target model file and the target memory space, the image to be processed is processed. 2. The method according to claim 1, characterized in that, before determining the target model file corresponding to the preset image dimension information matching the image to be processed from the preset model library file, the method Also includes: For each image to be processed, determine whether there is first image dimension information consistent with the image dimension information of the image to be processed in each preset image dimension information; If the first image dimension information exists, it is determined that the first image dimension information matches the image to be processed; If the first image dimension information does not exist, it is determined that the second image dimension information in the plurality of preset image dimension information matches the image to be processed; wherein, among the plurality of preset image dimension information, the The second image dimension information is not less than the image dimension information of the image to be processed, and the difference between the second image dimension information and the image dimension information of the image to be processed is the smallest. 3. The method according to claim 2, wherein the image dimension information includes the number of batches, the number of channels, and image height and image width; Determining that the second image dimension information among the plurality of preset image dimension information matches the image to be processed includes: From the plurality of preset image dimension information, determine the image dimension information that is not smaller than the image to be processed in the number of batches, the number of channels, and the dimensions of image height and image width, as the third image dimension information; Second image dimension information matching the image to be processed is determined from the third image dimension information based on differences in the dimensions of the batch number, the number of channels, and the image height and image width with the image to be processed. 4. The method according to claim 3, characterized in that, based on the difference between the image to be processed in the dimensions of batch number, number of channels, and image height and image width, from the third image dimension information Determine the second image dimension information matching the image to be processed, including: From the third image dimension information, determine the image dimension information with the smallest difference in the batch number dimension from the image to be processed as the fourth image dimension information; From the fourth image dimension information, determine the image dimension information with the smallest difference in the dimension of the number of channels from the image to be processed as the fifth image dimension information; Among the fifth image dimension information, the image dimension information with the smallest difference in the dimensions of image height and image width from the image to be processed is determined as the second image dimension information matching the image to be processed. 5. The method according to claim 1, characterized in that the construction process of the preset model library file includes: Obtain the dimension information of each preset image and the initial model file representing the weight parameters of the initial deep learning model; For each preset image dimension information, according to the initial model file, a model file of the target deep learning model matching the preset image dimension information is generated as a target model file; wherein, matching the preset image dimension information The matched target deep learning model represents: a deep learning model obtained by optimizing the initial deep learning model based on the preset image dimension information; Based on encapsulating each target model file, the preset model library file is obtained. 6. The method according to claim 5, characterized in that said obtaining the preset model library file based on encapsulating each target model file includes: Each target model file and the model loading instruction are encapsulated to obtain the preset model library file; wherein the model loading instruction is used to call the target model file that needs to be run from the preset model library file; so The above-mentioned default model library file contains the common parts in each target model file, and the private parts in each target model file. 7. The method according to any one of claims 1 to 6, characterized in that the preset model library file is a dynamic library file or a static library file. 8. An image processing device, characterized in that the device includes: The preset model library file calling module is used to call the preset model library file when the image needs to be processed; wherein the preset model library file includes: target deep learning models corresponding to multiple preset image dimension information. The target model file; the target model file contains the weight parameters of the target deep learning model; The target memory space allocation module is used to allocate the target memory space for processing each image to be processed according to the memory size required for processing the image with the largest preset image dimension information according to the target deep learning model; A target model file determination module, configured to determine, for each image to be processed, the target model file corresponding to the preset image dimension information matching the image to be processed from the preset model library file; The image processing module is used to process the image to be processed based on the determined target model file and the target memory space. 9. An electronic device, characterized in that it includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the method steps described in any one of claims 1-7 when executing a program stored in the memory. 10. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method steps of any one of claims 1-7 are implemented.