CN114359289A - An image processing method and related device - Google Patents

Abstract

The embodiment of the application discloses an image processing method, which is applied to the field of artificial intelligence and comprises the following steps: acquiring an image to be processed; processing the image to be processed through a first network to obtain a first feature, wherein the first network is configured to at least extract the feature for image enhancement; processing the image to be processed through a second network to obtain a second feature, wherein the second network is configured to extract at least a semantic segmentation feature; generating a third feature according to the first feature and the second feature; obtaining a semantic segmentation result of an image to be processed; generating a fourth feature according to the third feature and the semantic segmentation result of the image to be processed; and carrying out image reconstruction on the fourth characteristic to obtain a target image. By introducing the semantic features and semantic segmentation results of the image in the image enhancement processing process, different image enhancement strengths can be adopted for different semantic regions, the texture details can be accurately kept, and the reality of the texture details after the image enhancement is improved.

Description

An image processing method and related device

technical field

The present application relates to the technical field of artificial intelligence, and in particular, to an image processing method and a related device.

Background technique

Artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that responds in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

Deep learning methods, a key driving force for the development of artificial intelligence in recent years, have achieved remarkable results in various tasks of computer vision. In the field of image enhancement (also known as image quality enhancement), deep learning-based methods have surpassed traditional methods.

However, the current image enhancement network based on deep learning has an unnatural enhancement effect on the image, and the image texture details obtained after processing by the image enhancement network are unreal.

SUMMARY OF THE INVENTION

Embodiments of the present application provide an image processing method and a related device, which are used to improve an image enhancement effect.

A first aspect of the present application provides an image processing method, including: acquiring an image to be processed, for example, the image to be processed may be an image that needs to be enhanced; processing the image to be processed through a first network to obtain a first feature , the first network is configured to at least extract features used for image enhancement, and the features used for image enhancement may be, for example, low-level image features; the second network processes the to-be-processed image to obtain second features, The second network is configured to extract at least a semantic segmentation feature, for example, the semantic segmentation feature may be a high-level image feature; generate a third feature according to the first feature and the second feature; acquire the to-be-processed image the semantic segmentation result; generate a fourth feature according to the third feature and the semantic segmentation result of the to-be-processed image; perform image reconstruction on the fourth feature to obtain a target image.

In this solution, by introducing the semantic features of the image and the semantic segmentation results in the image enhancement process, and merging the semantic features and the semantic segmentation results with the features used for image enhancement, different image enhancement strengths can be used for different semantic regions. , accurately maintain texture details, and improve the authenticity of texture details after image enhancement.

Optionally, in a possible implementation manner, the acquiring the semantic segmentation result of the to-be-processed image includes: processing the third feature through a third network to obtain the semantic segmentation of the to-be-processed image result.

Because the third feature is a feature after fusion of low-level image features related to image enhancement and high-level image features related to semantic segmentation. Therefore, by processing the third feature, the low-level image features can be introduced on the basis of the high-level image features related to semantic segmentation, that is, the semantic segmentation results of the to-be-processed image can be obtained based on the features of different levels, and the obtained semantics can be improved. The precision of the segmentation result.

Optionally, in a possible implementation manner, the generating a third feature according to the first feature and the second feature includes: performing feature fusion on the first feature and the second feature processing to obtain the third feature; generating a fourth feature according to the third feature and the semantic segmentation result of the image to be processed, including: semantic segmentation of the third feature and the image to be processed As a result, feature fusion processing is performed to obtain the fourth feature.

Optionally, in a possible implementation manner, the feature fusion processing includes summation processing, multiplication processing, cascade processing, or cascade processing and convolution processing.

Optionally, in a possible implementation manner, before generating the fourth feature according to the third feature and the semantic segmentation result of the to-be-processed image, the method further includes: processing the third feature , to obtain a fifth feature; the generating a fourth feature according to the third feature and the semantic segmentation result of the image to be processed includes: generating a fourth feature according to the fifth feature and the semantic segmentation result of the image to be processed Fourth feature.

That is to say, after the image processing apparatus performs feature extraction on the third feature, feature fusion is performed based on the fifth feature obtained by further feature extraction and the semantic segmentation result of the image to be processed. By performing further feature extraction processing on the third feature obtained after feature fusion, finer-grained features can be extracted on the basis of the third feature, so as to improve the accuracy of the fourth feature obtained by subsequent feature fusion.

Optionally, in a possible implementation manner, the step of processing the image to be processed through the second network to obtain the second feature includes: preprocessing the image to be processed to obtain the preprocessing feature; performing down-sampling processing on the pre-processing features to obtain down-sampling features; processing the down-sampling features through the second network to obtain sixth features; performing up-sampling processing on the sixth features to obtain the to-be-processed features The second feature of the image. Through the downsampling operation, the resolution of the preprocessing feature is reduced, the computation amount for extracting the semantic segmentation feature can be reduced, and the computing power requirement on the image processing device can be reduced.

Optionally, in a possible implementation manner, the method is used to implement at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image dehazing, image deblurring, image contrast Enhancement, Image Demosaicing, Image Deraining, Image Color Enhancement, Image Brightness Enhancement, Image Detail Enhancement, and Image Dynamic Range Enhancement.

A second aspect of the present application provides a model training method, including: acquiring a training sample pair, the training sample pair including a first image and a second image, the quality of the first image is lower than that of the second image; The image processing model is trained to process the first image to obtain a predicted image, wherein the image processing model to be trained is used to obtain the image to be processed; the first image is processed by the first network to obtain the first feature, the The first network is configured to at least extract features for image enhancement; the first image is processed by the second network to obtain second features, and the second network is configured to extract at least semantic segmentation features; a feature and the second feature to generate a third feature; obtain the semantic segmentation result of the first image; generate a fourth feature according to the third feature and the semantic segmentation result of the first image; Image reconstruction is performed on the fourth feature to obtain a predicted image; a first loss is obtained according to the second image in the training sample pair and the predicted image, and the first loss is used to describe the second image and the predicted image. Predicting the difference between the images; updating the model parameters of the image processing model to be trained according to at least the first loss, until the model training conditions are met, and an image processing model is obtained.

Optionally, in a possible implementation manner, the image processing model to be trained is further configured to process the third feature through a third network to obtain a semantic segmentation prediction result of the first image.

Optionally, in a possible implementation manner, the image processing model to be trained is further used to: obtain the real result of semantic segmentation of the first image; according to the prediction result of semantic segmentation and the real result of semantic segmentation , obtain a second loss, the second loss is used to describe the difference between the predicted result of the semantic segmentation and the real result of the semantic segmentation; The model parameters of the image processing model are updated until the model training conditions are met, and an image processing model is obtained.

Optionally, in a possible implementation manner, the image processing model to be trained is further configured to: perform feature fusion processing on the first feature and the second feature to obtain the third feature; the Generating a fourth feature according to the third feature and the semantic segmentation result of the first image includes: performing feature fusion processing on the third feature and the semantic segmentation result of the first image to obtain the fourth feature. feature.

Optionally, in a possible implementation manner, the feature fusion processing includes summation processing, cascade processing, or cascade processing and convolution processing; the feature fusion processing includes summation processing, cascade processing, Or cascade processing and convolution processing.

Optionally, in a possible implementation manner, the image processing model to be trained is further configured to process the third feature to obtain a fifth feature; according to the third feature and the first image Semantic segmentation results, generate the fourth feature.

Optionally, in a possible implementation manner, the image processing model to be trained is further used to preprocess the first image to obtain preprocessing features; perform downsampling processing on the preprocessing features to obtain down-sampling features; processing the down-sampling features through the second network to obtain sixth features; performing up-sampling processing on the sixth features to obtain the second features of the first image.

Optionally, in a possible implementation manner, the image processing model is used to implement at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image dehazing, image deblurring, Image contrast enhancement, image demosaicing, image deraining, image color enhancement, image brightness enhancement, image detail enhancement, and image dynamic range enhancement.

A third aspect of the present application provides an image processing device, comprising: an acquisition unit and a processing unit; the acquisition unit is configured to acquire an image to be processed; the processing unit is configured to process the image to be processed through a first network, obtaining a first feature, the first network is configured to extract at least features for image enhancement; processing the to-be-processed image by a second network to obtain a second feature, the second network is configured to at least extract semantics segmentation feature; generating a third feature according to the first feature and the second feature; the obtaining unit is further configured to obtain the semantic segmentation result of the image to be processed; the processing unit is further configured to obtain the semantic segmentation result according to the The third feature and the semantic segmentation result of the to-be-processed image are used to generate a fourth feature; and image reconstruction is performed on the fourth feature to obtain a target image.

Optionally, in a possible implementation manner, the processing unit is further configured to process the third feature through a third network to obtain a semantic segmentation result of the image to be processed.

Optionally, in a possible implementation manner, the processing unit is further configured to perform feature fusion processing on the first feature and the second feature to obtain the third feature; The feature and the semantic segmentation result of the image to be processed are subjected to feature fusion processing to obtain the fourth feature.

Optionally, in a possible implementation manner, the feature fusion processing includes at least one of summation processing, multiplication processing, concatenated processing and concatenated convolution processing.

Optionally, in a possible implementation manner, the processing unit is further configured to process the third feature to obtain a fifth feature; according to the fifth feature and the semantic segmentation of the image to be processed As a result, a fourth feature is generated.

Optionally, in a possible implementation manner, the processing unit is further configured to perform preprocessing on the to-be-processed image to obtain preprocessing features; perform downsampling processing on the preprocessing features to obtain downsampling. feature; process the down-sampling feature through the second network to obtain a sixth feature; perform up-sampling processing on the sixth feature to obtain the second feature of the image to be processed.

Optionally, in a possible implementation manner, the image processing device is configured to implement at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image dehazing, image deblurring, Image contrast enhancement, image demosaicing, image deraining, image color enhancement, image brightness enhancement, image detail enhancement, and image dynamic range enhancement.

A fourth aspect of the present application provides a model training device, comprising: an acquisition unit and a training unit; the acquisition unit is configured to acquire a training sample pair, the training sample pair includes a first image and a second image, the first image The quality of the image is lower than that of the second image; the training unit is configured to process the first image through the image processing model to be trained to obtain a predicted image, wherein the image processing model to be trained is used to obtain the image to be processed ; Process the first image through a first network to obtain a first feature, and the first network is configured to extract at least features for image enhancement; Process the first image through a second network to obtain a second feature feature, the second network is configured to extract at least a semantic segmentation feature; generate a third feature according to the first feature and the second feature; obtain a semantic segmentation result of the first image; according to the third feature feature and the semantic segmentation result of the first image to generate a fourth feature; image reconstruction is performed on the fourth feature to obtain a predicted image; according to the second image in the training sample pair and the predicted image, obtain a first loss, the first loss is used to describe the difference between the second image and the predicted image; the model parameters of the image processing model to be trained are updated at least according to the first loss until the Model training conditions to obtain an image processing model.

Optionally, in a possible implementation manner, the training unit is further configured to process the third feature through a third network to obtain a semantic segmentation prediction result of the first image.

Optionally, in a possible implementation manner, the training unit is further configured to obtain the real result of semantic segmentation of the first image; loss, the second loss is used to describe the difference between the predicted result of semantic segmentation and the real result of semantic segmentation; at least according to the first loss and the second loss on the image processing model to be trained The model parameters are updated until the model training conditions are met, and an image processing model is obtained.

Optionally, in a possible implementation manner, the training unit is further configured to perform feature fusion processing on the first feature and the second feature to obtain the third feature; Perform feature fusion processing with the semantic segmentation result of the first image to obtain the fourth feature.

Optionally, in a possible implementation manner, the feature fusion processing includes at least one of summation processing, multiplication processing, concatenated processing and concatenated convolution processing.

Optionally, in a possible implementation manner, the training unit is further configured to process the third feature to obtain a fifth feature; according to the third feature and the semantic segmentation result of the first image , to generate the fourth feature.

Optionally, in a possible implementation manner, the training unit is further configured to preprocess the first image to obtain preprocessing features; perform downsampling processing on the preprocessing features to obtain downsampling features. ; Process the down-sampling feature through the second network to obtain a sixth feature; and perform an up-sampling process on the sixth feature to obtain the second feature of the first image.

Optionally, in a possible implementation manner, the image processing model is used to implement at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image dehazing, image deblurring, Image contrast enhancement, image demosaicing, image deraining, image color enhancement, image brightness enhancement, image detail enhancement, and image dynamic range enhancement.

A fifth aspect of the present application provides an image processing apparatus, which may include a processor, the processor is coupled with a memory, the memory stores program instructions, and the method described in the first aspect is implemented when the program instructions stored in the memory are executed by the processor . For the steps in each possible implementation manner of the first aspect performed by the processor, reference may be made to the first aspect for details, and details are not repeated here.

A sixth aspect of the present application provides a model training device, which may include a processor, the processor is coupled to a memory, the memory stores program instructions, and the method described in the second aspect is implemented when the program instructions stored in the memory are executed by the processor . For the steps in each possible implementation manner of the second aspect performed by the processor, reference may be made to the second aspect for details, and details are not repeated here.

A seventh aspect of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when it runs on a computer, causes the computer to execute the method described in the first aspect.

An eighth aspect of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when it runs on a computer, causes the computer to execute the method described in the second aspect.

A ninth aspect of the present application provides a circuit system, the circuit system comprising a processing circuit configured to perform the method of the above-mentioned first aspect.

A tenth aspect of the present application provides a circuit system, the circuit system includes a processing circuit configured to perform the method described in the second aspect above.

An eleventh aspect of the present application provides a computer program, which, when executed on a computer, causes the computer to execute the method described in the first aspect above.

A twelfth aspect of the present application provides a computer program that, when executed on a computer, causes the computer to execute the method described in the second aspect above.

A thirteenth aspect of the present application provides a chip system, the chip system includes a processor for supporting a server or a threshold value acquisition device to implement the functions involved in the above aspects, for example, sending or processing the above methods. data and/or information. In a possible design, the chip system further includes a memory for storing necessary program instructions and data of the server or the communication device. The chip system may be composed of chips, or may include chips and other discrete devices.

Description of drawings

1 is a schematic structural diagram of an artificial intelligence main frame provided by an embodiment of the present application;

FIG. 2a is an image processing system provided by an embodiment of the present application;

FIG. 2b is another image processing system provided by an embodiment of the present application;

FIG. 2c is a schematic diagram of a related device for image processing provided by an embodiment of the present application;

FIG. 3a is a schematic diagram of the architecture of a system 100 provided by an embodiment of the present application;

3b is a schematic diagram of an image semantic segmentation provided by an embodiment of the present application;

4 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

5 is a schematic structural diagram of a densely connected atrous convolutional network provided by an embodiment of the present application;

FIG. 6a is a schematic diagram of an architecture for image processing provided by an embodiment of the present application;

FIG. 6b is a schematic diagram of a network structure for image processing provided by an embodiment of the present application;

7 is a schematic diagram of an objective index comparison provided in the embodiment of the present application;

Fig. 8 is another objective index comparison schematic diagram provided by the embodiment of the present application;

9 is a schematic diagram of image comparison provided by the embodiment of the present application;

FIG. 10 is a schematic flowchart of a model training method provided by an embodiment of the present application

FIG. 11 is a schematic structural diagram of an image processing apparatus according to an embodiment of the application;

12 is a schematic structural diagram of a model training apparatus provided by an embodiment of the application;

13 is a schematic structural diagram of an execution device provided by an embodiment of the present application;

14 is a schematic structural diagram of a training device provided by an embodiment of the application;

FIG. 15 is a schematic structural diagram of a chip provided by an embodiment of the present application.

Detailed ways

The embodiments of the present invention will be described below with reference to the accompanying drawings in the embodiments of the present invention. The terms used in the embodiments of the present invention are only used to explain specific embodiments of the present invention, and are not intended to limit the present invention.

The embodiments of the present application will be described below with reference to the accompanying drawings. Those of ordinary skill in the art know that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

The terms "first", "second" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, and this is only a distinguishing manner adopted when describing objects with the same attributes in the embodiments of the present application. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, product or device comprising a series of elements is not necessarily limited to those elements, but may include no explicit or other units inherent to these processes, methods, products, or devices.

First, the overall workflow of the artificial intelligence system will be described. Please refer to Figure 1. Figure 1 shows a schematic structural diagram of the main frame of artificial intelligence. The above-mentioned artificial intelligence theme framework is explained in two dimensions (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data has gone through the process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecological process of the system.

(1) Infrastructure

The infrastructure provides computing power support for artificial intelligence systems, realizes communication with the outside world, and supports through the basic platform. Communication with the outside world through sensors; computing power is provided by smart chips (hardware acceleration chips such as CPU, NPU, GPU, ASIC, FPGA); the basic platform includes distributed computing framework and network-related platform guarantee and support, which can include cloud storage and computing, interconnection networks, etc. For example, sensors communicate with external parties to obtain data, and these data are provided to the intelligent chips in the distributed computing system provided by the basic platform for calculation.

(2) Data

The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, and text, as well as IoT data from traditional devices, including business data from existing systems and sensory data such as force, displacement, liquid level, temperature, and humidity.

(3) Data processing

Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

Among them, machine learning and deep learning can perform symbolic and formalized intelligent information modeling, extraction, preprocessing, training, etc. on data.

Reasoning refers to the process of simulating human's intelligent reasoning method in a computer or intelligent system, using formalized information to carry out machine thinking and solving problems according to the reasoning control strategy, and the typical function is search and matching.

Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

(4) General ability

After the above-mentioned data processing, some general capabilities can be formed based on the results of data processing, such as algorithms or a general system, such as translation, text analysis, computer vision processing, speech recognition, image identification, etc.

(5) Smart products and industry applications

Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall solution of artificial intelligence, and the productization of intelligent information decision-making to achieve landing applications. Its application areas mainly include: intelligent terminals, intelligent transportation, Smart healthcare, autonomous driving, safe city, etc.

Next, several application scenarios of the present application are introduced.

FIG. 2a is an image processing system provided by an embodiment of the present application, where the image processing system includes a user equipment and a data processing device. The user equipment includes smart terminals such as mobile phones, personal computers, or information processing centers. The user equipment is the initiator of image processing. As the initiator of the image enhancement request, the user usually initiates the request through the user equipment.

The above-mentioned data processing device may be a device or server with data processing functions, such as a cloud server, a network server, an application server, and a management server. The data processing device receives the image enhancement request from the intelligent terminal through the interactive interface, and then performs image processing in the form of machine learning, deep learning, search, reasoning, and decision-making through the memory for storing data and the processor for data processing. The memory in the data processing device may be a general term, including local storage and a database for storing historical data. The database may be on the data processing device or on other network servers.

In the image processing system shown in FIG. 2a, the user equipment can receive instructions from the user, for example, the user equipment can acquire an image input/selected by the user, and then initiate a request to the data processing equipment, so that the data processing equipment can target the data obtained by the user equipment. The image is subjected to image enhancement processing applications (such as image super-resolution reconstruction, image denoising, image dehazing, image deblurring, and image contrast enhancement, etc.), thereby obtaining corresponding processing results for the image. Exemplarily, the user equipment may acquire an image input by the user, and then initiate an image denoising request to the data processing device, so that the data processing device performs image denoising on the image, thereby obtaining a denoised image.

In Fig. 2a, the data processing device may execute the image processing method of the embodiment of the present application.

Fig. 2b is another image processing system provided by the embodiment of the application. In Fig. 2b, the user equipment is directly used as a data processing device, and the user equipment can directly obtain the input from the user and directly process it by the hardware of the user equipment itself, The specific process is similar to that of FIG. 2a, and the above description can be referred to, and details are not repeated here.

In the image processing system shown in Fig. 2b, the user equipment can receive instructions from the user, for example, the user equipment can acquire an image selected by the user in the user equipment, and then the user equipment can execute an image processing application ( For example, image super-resolution reconstruction, image denoising, image dehazing, image deblurring, and image contrast enhancement, etc.), so as to obtain corresponding processing results for the image.

In FIG. 2b, the user equipment itself can execute the image processing method of the embodiment of the present application.

FIG. 2c is a schematic diagram of a related device for image processing provided by an embodiment of the present application.

The user equipment in the above-mentioned FIGS. 2a and 2b may specifically be the local device 301 or the local device 302 in FIG. 2c, and the data processing device in FIG. 2a may specifically be the execution device 210 in FIG. 2c, wherein the data storage system 250 may be To store the data to be processed by the execution device 210, the data storage system 250 may be integrated on the execution device 210, or may be set on the cloud or other network servers.

The processors in Figures 2a and 2b may perform data training/machine learning/deep learning through a neural network model or other model (eg, a support vector machine-based model), and use the data to finally train or learn the model to execute on the image Image processing application, so as to obtain the corresponding processing results.

FIG. 3a is a schematic diagram of the architecture of a system 100 provided by an embodiment of the present application. In FIG. 3a, the execution device 110 is configured with an input/output (I/O) interface 112, which is used for data interaction with external devices, The user may input data to the I/O interface 112 through the client device 140, and the input data may include various tasks to be scheduled, callable resources, and other parameters in this embodiment of the present application.

When the execution device 110 preprocesses the input data, or the calculation module 111 of the execution device 110 performs computation and other related processing (for example, performing the function realization of the neural network in this application), the execution device 110 may call the data storage system 150 The data, codes, etc. in the corresponding processing can also be stored in the data storage system 150 .

Finally, the I/O interface 112 returns the processing results to the client device 140 for provision to the user.

It is worth noting that the training device 120 can generate corresponding target models/rules based on different training data for different goals or tasks, and the corresponding target models/rules can be used to achieve the above-mentioned goals or complete the above-mentioned tasks. , which provides the user with the desired result. The training data may be stored in the database 130 and come from training samples collected by the data collection device 160 .

In the case shown in FIG. 3 a , the user can manually specify input data, which can be operated through the interface provided by the I/O interface 112 . In another case, the client device 140 can automatically send the input data to the I/O interface 112 . If the user's authorization is required to request the client device 140 to automatically send the input data, the user can set the corresponding permission in the client device 140 . The user can view the result output by the execution device 110 on the client device 140, and the specific presentation form can be a specific manner such as display, sound, and action. The client device 140 can also be used as a data collection terminal to collect the input data of the input I/O interface 112 and the output result of the output I/O interface 112 as new sample data as shown in the figure, and store them in the database 130 . Of course, it is also possible not to collect through the client device 140, but the I/O interface 112 directly uses the input data input into the I/O interface 112 and the output result of the output I/O interface 112 as shown in the figure as a new sample The data is stored in database 130 .

It is worth noting that FIG. 3a is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG. 3a, the data The storage system 150 is an external memory relative to the execution device 110 , and in other cases, the data storage system 150 may also be placed in the execution device 110 . As shown in FIG. 3a, the neural network can be obtained by training according to the training device 120.

An embodiment of the present application also provides a chip, where the chip includes a neural network processor NPU. The chip can be set in the execution device 110 as shown in FIG. 3 a to complete the calculation work of the calculation module 111 . The chip can also be set in the training device 120 as shown in FIG. 3a to complete the training work of the training device 120 and output the target model/rule.

The neural network processor NPU, the NPU is mounted on the main central processing unit (central processing unit, CPU) (host CPU) as a co-processor, and the main CPU assigns tasks. The core part of the NPU is an arithmetic circuit, and the controller controls the arithmetic circuit to extract the data in the memory (weight memory or input memory) and perform operations.

In some implementations, the arithmetic circuit includes a plurality of process engines (PE) inside. In some implementations, the arithmetic circuit is a two-dimensional systolic array. The arithmetic circuit may also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit is a general-purpose matrix processor.

For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit fetches the data corresponding to the matrix B from the weight memory, and buffers it on each PE in the operation circuit. The arithmetic circuit fetches the data of matrix A from the input memory and performs matrix operation on matrix B, and stores the partial result or final result of the matrix in an accumulator.

The vector calculation unit can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison and so on. For example, the vector computing unit can be used for network computation of non-convolutional/non-FC layers in neural networks, such as pooling, batch normalization, local response normalization, etc.

In some implementations, the vector computation unit can store the processed output vector to a unified buffer. For example, the vector computing unit may apply a nonlinear function to the output of the arithmetic circuit, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit generates normalized values, merged values, or both. In some implementations, the vector of processed outputs can be used as activation input to an operational circuit, such as for use in subsequent layers in a neural network.

Unified memory is used to store input data as well as output data.

The weight data directly transfers the input data in the external memory to the input memory and/or the unified memory through the memory unit access controller (direct memory access controller, DMAC), stores the weight data in the external memory into the weight memory, and transfers the unified memory store the data in the external memory.

The bus interface unit (bus interface unit, BIU) is used to realize the interaction between the main CPU, the DMAC and the instruction fetch memory through the bus.

The instruction fetch buffer connected to the controller is used to store the instructions used by the controller;

The controller is used for invoking the instructions cached in the memory to realize and control the working process of the operation accelerator.

Generally, the unified memory, input memory, weight memory and instruction fetch memory are all on-chip memories, and the external memory is the memory outside the NPU, and the external memory can be double data rate synchronous dynamic random access memory (double data rate synchronous random access memory). rate synchronous dynamic random access memory, DDRSDRAM), high bandwidth memory (high bandwidth memory, HBM) or other readable and writable memory.

Since the embodiments of the present application involve a large number of neural network applications, for ease of understanding, related terms and neural networks and other related concepts involved in the embodiments of the present application are first introduced below.

(1) Neural network

A neural network can be composed of neural units, and a neural unit can refer to an operation unit that takes xs and intercept 1 as inputs, and the output of the operation unit can be:

Among them, s=1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is an activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected with the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be an area composed of several neural units.

来描述：从物理层面神经网络中的每一层的工作可以理解为通过五种对输入空间(输入向量的集合)的操作，完成输入空间到输出空间的变换(即矩阵的行空间到列空间)，这五种操作包括：1、升维/降维；2、放大/缩小；3、旋转；4、平移；5、“弯曲”。其中1、2、3的操作由

完成，4的操作由+b完成，5的操作则由a()来实现。这里之所以用“空间”二字来表述是因为被分类的对象并不是单个事物，而是一类事物，空间是指这类事物所有个体的集合。其中，W是权重向量，该向量中的每一个值表示该层神经网络中的一个神经元的权重值。该向量W决定着上文所述的输入空间到输出空间的空间变换，即每一层的权重W控制着如何变换空间。训练神经网络的目的，也就是最终得到训练好的神经网络的所有层的权重矩阵(由很多层的向量W形成的权重矩阵)。因此，神经网络的训练过程本质上就是学习控制空间变换的方式，更具体的就是学习权重矩阵。The work of each layer in a neural network can be expressed mathematically

To describe: From the physical level, the work of each layer in the neural network can be understood as the transformation from the input space to the output space (that is, the row space of the matrix to the column space) through five operations on the input space (set of input vectors). ), the five operations include: 1. Dimension raising/lowering; 2. Enlarging/reducing; 3. Rotation; 4. Translation; 5. "Bending". Among them, the operations of 1, 2, and 3 are determined by

Complete, the operation of 4 is completed by +b, and the operation of 5 is realized by a(). The reason why the word "space" is used here is because the object to be classified is not a single thing, but a type of thing, and space refers to the collection of all individuals of this type of thing. Among them, W is the weight vector, and each value in the vector represents the weight value of a neuron in the neural network of this layer. The vector W determines the space transformation from the input space to the output space described above, that is, the weight W of each layer controls how the space is transformed. The purpose of training the neural network is to finally obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by the vectors W of many layers). Therefore, the training process of the neural network is essentially learning the way to control the spatial transformation, and more specifically, learning the weight matrix.

Because we want the output of the neural network to be as close as possible to the value you really want to predict, you can compare the predicted value of the current network with the target value you really want, and then update each layer of the neural network according to the difference between the two. (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network), for example, if the predicted value of the network is high, adjust the weight vector to make it predict low Some, keep adjusting until the neural network can predict the real desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which is used to measure the difference between the predicted value and the target value. important equation. Among them, taking the loss function as an example, the higher the output value of the loss function (loss), the greater the difference, then the training of the neural network becomes the process of reducing the loss as much as possible.

(2) Back propagation algorithm

The neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the input signal is passed forward until the output will generate error loss, and the parameters in the initial neural network model are updated by back-propagating the error loss information, so that the error loss converges. The back-propagation algorithm is a back-propagation movement dominated by error loss, aiming to obtain the parameters of the optimal neural network model, such as the weight matrix.

(3) Image enhancement

Image enhancement refers to processing the brightness, color, contrast, saturation, dynamic range, etc. of an image to meet certain specific indicators. Simply put, in the process of image processing, by purposefully emphasizing the overall or local characteristics of the image, the original unclear image becomes clear or some interesting features are emphasized, and the difference between the features of different objects in the image is enlarged. It can improve the image quality and enrich the amount of image information, and can strengthen the image interpretation and recognition effect to meet the needs of some special analysis. Exemplarily, image enhancement may include, but is not limited to, image super-resolution reconstruction, image denoising, image dehazing, image deblurring, and image contrast enhancement.

(4) Image Semantic Segmentation

Image semantic segmentation refers to subdividing the image into different categories according to certain rules (such as illumination, category). To put it simply, the goal of image semantic segmentation is to label each pixel in the image with a label, that is, to label the object category to which each pixel in the image belongs. These labels can include people, animals, cars, flowers, furniture, etc. . Referring to FIG. 3b, FIG. 3b is a schematic diagram of an image semantic segmentation provided by an embodiment of the present application. As shown in Figure 3b, through image semantic segmentation, the image can be divided into different sub-regions according to categories at the pixel level, such as sub-regions such as buildings, sky, and plants.

The method provided by the present application will be described below from the training side of the neural network and the application side of the neural network.

The neural network training method provided in the embodiment of the present application involves the processing of images, and can be specifically applied to data processing methods such as data training, machine learning, deep learning, etc., to symbolize and form the training data (such as the images in the present application). In addition, the image processing method provided in the embodiment of the present application can use the above-mentioned trained image processing model to convert input data (such as The image to be processed in this application) is input into the trained image processing model to obtain output data (such as the target image in this application). It should be noted that the training method and the image processing method of the image processing model provided in the embodiments of this application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process : such as model training phase and model application phase.

Referring to FIG. 4 , FIG. 4 is a schematic flowchart of an image processing method provided by an embodiment of the present application. As shown in FIG. 4 , an image processing method provided by an embodiment of the present application includes the following steps:

Step 401: Acquire an image to be processed.

In this embodiment, the image processing apparatus may acquire an image to be processed, and the image to be processed may be, for example, an image that needs to be enhanced.

It can be understood that when the image processing device is deployed in the unmanned vehicle, the image processing device can obtain the street view image collected by the unmanned vehicle during the driving process through the camera. When the image processing device is deployed in the robot, the image processing device can acquire a real-world image of the environment where the robot is located in real time. When the image processing apparatus is deployed in a security device (such as a surveillance camera), the image processing apparatus can acquire the real-world image captured by the surveillance camera in real time. When the image processing apparatus is deployed on a handheld device such as a mobile phone or a tablet computer, the image processing apparatus can acquire photos taken by the user or pictures downloaded from a website, and these images can be used as images to be processed.

Step 402: Process the to-be-processed image through a first network to obtain a first feature, where the first network is configured to extract at least features for image enhancement.

In this embodiment, the first network may be a backbone network related to image enhancement, such as a convolutional neural network, and the first network is configured to extract at least features for image enhancement, such as low-level image features (low level feature). Exemplarily, low-level image features may refer to some small details in the image, such as edges, corners, colors, pixels, gradients, and textures. frequency details.

It can be understood that for different image enhancement tasks, different networks can be used to suit the needs of the image enhancement task. For example, when the image enhancement task is image super-resolution reconstruction, the first network may use Residual Network (ResNet); when the image enhancement task is image contrast enhancement, the first network may use Unet network.

Specifically, the image processing method provided in this embodiment can be applied to different image enhancement tasks, such as, but not limited to, image super-resolution reconstruction, image denoising, image dehazing, image deblurring, image contrast enhancement, image Image enhancement tasks such as demosaicing, image deraining, image color enhancement, image brightness enhancement, image detail enhancement, and image dynamic range enhancement.

In a possible embodiment, before the image to be processed is processed through the first network, the image to be processed may also be preprocessed through a preprocessing network to obtain preprocessing features. Then, the obtained preprocessing features are processed through the first network to obtain the first features. The preprocessing network can be, for example, a convolutional neural network. By preprocessing the to-be-processed image, irrelevant information in the to-be-processed image can be eliminated, useful real information can be recovered, the detectability of the relevant information can be enhanced, and the data can be simplified to the greatest extent, thereby improving the reliability of feature extraction.

It can be understood that the preprocessing network may also be included in the first network, that is, the first network includes the preprocessing network, and the image to be processed is processed through the first network. , the first feature can be obtained.

Step 403: Process the to-be-processed image through a second network to obtain a second feature, where the second network is configured to extract at least semantic segmentation features.

In this embodiment, the second network may be a backbone network related to image semantic segmentation, such as a convolutional neural network, configured to extract at least features for image semantic segmentation, such as image high level features. Exemplarily, the high-level image features may refer to features that can reflect the semantic information of the image based on the low-level features of the image. Generally, image high-level features can be used to identify and detect the shape of objects or objects in images, and have richer semantic information.

In a possible embodiment, the second network may be, for example, a densely connected atrous convolutional network. Among them, the atrous convolutional network can increase the receptive field, and the atrous convolutional network can obtain multi-scale information based on dense connections. Through the joint action of the two, accurate high-level image feature information related to semantic segmentation can be generated.

The dilated convolutional network actually introduces the dilation rate (also called the number of holes) in the standard convolutional network. This parameter defines the spacing of each value when the convolution kernel processes the data, so as to increase the receptive field. Generally, the receptive field is used to represent the size of the receptive range of different neurons inside the network to the original image, or, in other words, the pixels on the feature map output by each layer of the convolutional network are mapped on the original image. area size. By increasing the receptive field, the pixels on the feature map can be made to respond to a large enough area in the image to capture information about large objects, so that accurate semantic information can be obtained.

Referring to FIG. 5, FIG. 5 is a schematic structural diagram of a densely connected atrous convolutional network provided by an embodiment of the present application. As shown in Figure 5, the densely connected dilated convolutional network includes multiple layers of networks, each of which includes a dilated convolutional network (dilated conv) and a linear activation function (leaky relu). For each layer of the densely connected atrous convolutional network, its output is used as the input of each layer after the layer to achieve feature reuse. By directly outputting the features of the low-level network to each subsequent high-level network for aggregation, the lost features caused by passing through the intermediate-level network are reduced, so that the features of the low-level network can be better utilized.

It can be understood that, in the embodiments of this application, the second network is a densely connected hole convolutional network as an example for description. In actual situations, the second network may also be other neural networks, which are not specifically limited here. .

In a possible embodiment, the processing of the to-be-processed image through the second network to obtain the second feature may specifically include: an image processing apparatus pre-processing the to-be-processed image to obtain the pre-processed feature, for example, by The preprocessing network described in the above step 402 preprocesses the image to be processed; performs downsampling processing on the preprocessing features to obtain downsampling features; processes the downsampling features through the second network, Obtain a sixth feature; perform up-sampling processing on the sixth feature to obtain the second feature of the image to be processed.

In this embodiment, by performing down-sampling processing on the pre-processing features, the down-sampling features with reduced resolution can be output; after processing the down-sampling features through the second network to obtain the sixth feature; The sixth feature is subjected to an upsampling process to generate a second feature with the same resolution as the preprocessing feature, that is, the resolution of the restored feature.

In practical applications, the multiple of downsampling can be determined according to the expected processing accuracy and the computing power of the target hardware platform, which is not specifically limited here. Generally speaking, the larger the multiple of downsampling, the lower the processing accuracy and the smaller the amount of calculation, that is, the lower the computing power requirement; the smaller the multiple of downsampling, the higher the processing accuracy, but the greater the amount of calculation. High, that is, higher computing power requirements. The multiple of upsampling needs to be consistent with the multiple of downsampling to ensure the resolution of the recovered features. The methods that can be used to perform upsampling include but are not limited to methods such as deconvolution, bilinear interpolation upsampling, and nearest neighbor interpolation upsampling, and the upsampling method is not specifically limited herein.

Step 404: Generate a third feature according to the first feature and the second feature.

In a possible embodiment, the image processing apparatus may perform feature fusion processing on the first feature and the second feature to obtain the third feature. Exemplarily, the feature fusion processing may include at least one of fusion processing operations such as summation processing, multiplication processing, concatenation processing, and concatenated convolution processing, wherein the concatenated convolution processing means performing concatenated processing and Convolution processing. In an actual situation, a corresponding feature fusion processing manner may be adopted according to actual needs, which is not specifically limited here.

In this embodiment, by performing fusion processing on the first feature and the second feature, the low-level image features related to image enhancement and the high-level image features related to semantic segmentation can be effectively fused to realize the complementarity of features at different levels. to improve the robustness of the network.

Step 405: Obtain the semantic segmentation result of the image to be processed.

In a possible embodiment, the image processing apparatus may process the third feature to obtain a semantic segmentation result of the image to be processed. Because the third feature is a feature after fusion of low-level image features related to image enhancement and high-level image features related to semantic segmentation. Therefore, by processing the third feature, the low-level image features can be introduced on the basis of the high-level image features related to semantic segmentation, that is, the semantic segmentation results of the to-be-processed image can be obtained based on the features of different levels, and the obtained semantics can be improved. The precision of the segmentation result. Exemplarily, the image processing apparatus may perform a convolution operation on the third feature through a convolution network to obtain the semantic segmentation result of the image to be processed.

In another possible embodiment, the image processing apparatus may process the second feature output by the second network to obtain a semantic segmentation result of the to-be-processed image, that is, obtain an image directly based on features related to semantic segmentation semantic segmentation results. Exemplarily, the image processing apparatus may also perform a convolution operation on the second feature through a convolution network to obtain the semantic segmentation result of the image to be processed.

Step 406: Generate a fourth feature according to the third feature and the semantic segmentation result of the image to be processed.

In a possible embodiment, the image processing apparatus may perform feature fusion processing on the third feature and the semantic segmentation result of the image to be processed to obtain the fourth feature. The specific method of the feature fusion processing may refer to the description in step 404, which will not be repeated here.

In a possible embodiment, after obtaining the third feature, the image processing apparatus may process the third feature, for example, perform further feature extraction on the third feature to obtain the fifth feature. The image processing apparatus generates a fourth feature according to the fifth feature and the semantic segmentation result of the image to be processed. That is to say, after the image processing apparatus performs feature extraction on the third feature, feature fusion is performed based on the fifth feature obtained by further feature extraction and the semantic segmentation result of the image to be processed. By performing further feature extraction processing on the third feature obtained after feature fusion, finer-grained features can be extracted on the basis of the third feature, so as to improve the accuracy of the fourth feature obtained by subsequent feature fusion.

Step 407: Perform image reconstruction on the fourth feature to obtain a target image.

In this embodiment, after the image processing apparatus obtains the fourth feature after two feature fusion processes, the image processing apparatus may perform image reconstruction on the fourth feature, for example, after convolving the fourth feature The processing operation is performed to obtain a target image, and the target image is an image obtained after image enhancement.

In this embodiment, in the process of image enhancement processing, the semantic segmentation features and the semantic segmentation results are fused with the related image enhancement features through two feature fusion processing, so as to realize the complementarity of feature information, so that different semantic regions can be targeted. Different image enhancement strengths are used to accurately maintain texture details and improve the authenticity of texture details after image enhancement.

It should be understood that the execution subject (ie, the image processing device) of steps 401 to 407 can be a terminal device or a server on the cloud side, and steps 401 to 407 can also be obtained through data processing and interaction between the terminal device and the server.

For ease of understanding, the following will describe in detail how the image processing method provided in this embodiment implements image dehazing with reference to specific examples.

6a and 6b, FIG. 6a is a schematic diagram of an architecture for image processing provided by an embodiment of the present application; FIG. 6b is a schematic diagram of a network structure for image processing provided by an embodiment of the present application. As shown in Figures 6a and 6b, the architecture can include:

The preprocessing unit 100 is configured to receive a low-contrast image with fog, and preprocess the image to generate a preprocessing feature F. Wherein, the preprocessing unit 100 can be, for example, a convolutional network, which generates a preprocessing feature F by performing a convolution operation on a received image (eg, an image with a resolution of 12 megapixels and a resolution of 3000*4000). The resolution of feature F is the same as the resolution of the image, which is 3000*4000 resolution.

The first feature extraction unit 101 is configured to perform feature extraction on the preprocessing feature F, for example, performing extraction of low-level image features on the preprocessing feature F to obtain the first feature _FL . The first feature extraction unit 101 may use a backbone network related to the dehazing task, for example, a multi-level cascaded convolutional network + instance normalization (Instance Normalization, IN) network. Based on the IN network, a highly nonlinear contrast normalization effect can be learned, so that the deviation in the appearance of the image such as brightness, color, and style will not affect the final prediction result, and it can also improve the low-level features of the extracted image and subsequent Compatibility issues of extracted image high-level features. The number of cascaded layers N can be determined according to the desired processing accuracy and the computing power of the target hardware platform. Generally speaking, the larger the number of cascaded layers, N, the higher the accuracy of feature extraction and the greater the amount of computation.

The down-sampling unit 200 is configured to perform a pre-processing operation on the pre-processing feature F, and perform down-sampling processing to obtain a down-sampling feature F _down with reduced resolution. Exemplarily, the downsampling unit 200 may be, for example, a pooling layer network, and performs a downsampling operation of k*k average pooling (average pooling) on the preprocessing feature F to obtain the downsampling feature F _down . The multiple k of downsampling can be determined according to the desired processing accuracy and the computing power of the target hardware platform. Generally speaking, the smaller the number of cascaded layers N, the higher the accuracy of feature extraction and the greater the amount of computation. Exemplarily, the value of k may be 4, that is, the width and height of the feature are simultaneously downsampled by 4 times.

The second feature extraction unit 201 is configured to perform feature extraction processing on the down-sampling feature F _down , for example, perform image high-level feature extraction on the down-sampling feature F _down to obtain a sixth feature F _down-seg . Exemplarily, the second feature extraction unit 201 may be a densely connected atrous convolutional network.

The up-sampling unit 202 is configured to perform up-sampling processing on the second feature F _down-seg , so as to obtain the second feature F _H-seg with the same resolution as the original input image. The sampling method adopted by the upsampling unit 202 may be, for example, deconvolution, bilinear interpolation upsampling, and nearest neighbor interpolation upsampling, and the upsampling multiple is the same as the downsampling multiple.

The first feature fusion unit 102 is configured to perform feature fusion processing on the first feature FL and the second feature F _{down-seg ,} for example, perform a cascade operation on the first feature FL and the second feature F down-seg to obtain a fusion The third feature F _fusion1 .

The third feature extraction unit 103 is used to perform further feature extraction on the third feature F _fusion1 , for example, perform convolution processing on the third feature F _fusion1 through a convolution network to extract fine-grained features, that is, the fifth feature F _fine .

The semantic result prediction unit 203 is used for predicting the semantic segmentation result of the third feature F _fusion1 , for example, performing a post-processing operation on the third feature Ffusion1 through a convolutional network to obtain a semantic segmentation result corresponding to the input image.

The second feature fusion unit 104 is used to perform feature fusion processing on the semantic segmentation result corresponding to the fifth feature F _fine and the input image, for example, a cascade operation is performed on the semantic segmentation result corresponding to the fifth feature F _fine and the input image, to A fused fourth feature F _{fusion2 is obtained} .

The image reconstruction unit 105 is configured to perform image reconstruction processing on the fused fourth feature F _fusion2 , for example, perform a post-processing operation on the fourth feature F _fusion2 through a convolutional network to obtain a dehazed image.

Taking the method of this embodiment for image dehazing as an example, this embodiment is tested on an open source simulation data set to compare the method of this embodiment with the existing dehazing algorithm.

Referring to FIG. 7 , FIG. 7 is a schematic diagram of objective index comparison provided in this embodiment of the present application. It can be seen from FIG. 7 that, compared with various existing dehazing algorithms, the image processing method provided by the embodiment of the present application has higher peak signal-to-noise ratio (Peak Signal to Noise Ratio, PSNR) and structural similarity (Structural SIMilarity, SSIM).

where PSNR is an engineering term that expresses the ratio of the maximum possible power of a signal to the power of destructive noise that affects the accuracy of its representation. PSNR is usually used to measure the quality of signal reconstruction in fields such as image processing, and is generally defined by mean square error. In general, the higher the PSNR, the smaller the gap with the true value.

SSIM is an index to measure the similarity of two images, which is mainly based on brightness, contrast and structure to evaluate the similarity of images.

In addition, when different levels of noise are added to the open source simulation data set, the method of this embodiment has higher PSNR and SSIM than the existing dehazing methods, that is, the method of this embodiment has stronger robustness Stability and stability. Specifically, reference may be made to FIG. 8 , which is a schematic diagram of another objective index comparison provided by this embodiment of the present application.

This embodiment is tested on a real foggy data set, and it can be known that the method of this embodiment can obtain clearer and more transparent results without artifact distortion. The existing dehazing algorithms have the problem that the dehazing level is not enough, and the contrast of the image after dehazing is low; or the dehazing is too much, and the texture details are lost in some local areas. Specifically, reference may be made to FIG. 9 , which is a schematic diagram of image comparison provided by this embodiment of the present application. It can be seen from FIG. 9 that the method of this embodiment in the lower right corner, on the basis of defogging, well maintains the details of green plants, floors, ground and other areas, the picture is transparent and natural, and the visual effect is the best.

Referring to FIG. 10 , FIG. 10 is a schematic flowchart of a model training method provided by an embodiment of the present application. As shown in FIG. 10 , a model training method provided by an embodiment of the present application includes the following steps:

Step 1001: Acquire a training sample pair, where the training sample pair includes a first image and a second image, and the quality of the first image is lower than that of the second image.

In this embodiment, before the image training apparatus performs model training, a pair of samples for training may be obtained. The first image and the second image are two images in the same scene, and the image quality of the first image is lower than that of the second image. Image quality refers to one or more of color, brightness, saturation, contrast, dynamic range, resolution, texture detail, sharpness, etc. For example, the first image is an image with fog, the second image is an image without fog, and the brightness, contrast, and clarity of the first image are lower than those of the second image.

Step 1002: Process the first image through the image processing model to be trained to obtain a predicted image, wherein the image processing model to be trained is used to obtain the image to be processed; process the first image through the first network to obtain the first image. a feature, the first network is configured to extract at least features for image enhancement; the first image is processed by a second network to obtain a second feature, the second network is configured to extract at least semantic segmentation features ; Generate a third feature according to the first feature and the second feature; Obtain the semantic segmentation result of the first image; Generate a fourth feature according to the third feature and the semantic segmentation result of the first image feature; perform image reconstruction on the fourth feature to obtain a predicted image.

Step 1003: Obtain a first loss according to the second image in the training sample pair and the predicted image, where the first loss is used to describe the difference between the second image and the predicted image.

In this embodiment, after the predicted image is obtained, a first loss corresponding to the second image and the predicted image may be obtained based on a preset loss function to determine the difference between the second image and the predicted image.

In a possible implementation manner, the first loss corresponding to the second image and the predicted image may be obtained based on a reconstruction loss function (reconstruction loss) and a gradient loss function (gradient loss), so as to ensure that after the enhancement The images can meet the requirements of objective indicators and subjective indicators.

Exemplarily, the reconstruction loss function may be to obtain the pixel-level loss of the predicted image and the second image using the L1 norm. The reconstruction loss function can be as shown in Equation 1:

Among them, L _rec represents the reconstruction loss, || represents the L1 norm, GT represents the true value, that is, the value of the pixel of the second image, output represents the value of the pixel of the predicted image, and P is the number of pixels. Wherein, the L1 normal form refers to calculating the difference between the value of the pixel of the second image and the value of the pixel of the predicted image, and calculating the sum of the absolute values of the difference values corresponding to each pixel.

Illustratively, the gradient loss function may be a loss representing the average gradient of the predicted image and the second image in the x/y direction. The gradient loss function can be as shown in Equation 2:

L _grad = |grad(GT)-grad(output)| Equation 2

Among them, L _grad represents the gradient loss, || represents the L1 normal form, GT represents the true value, that is, the value of the pixel of the second image, output represents the value of the pixel of the predicted image, and grad( ) represents the average value of the image in the x/y direction gradient.

Based on the reconstruction loss function and the gradient loss function, the first loss corresponding to the second image and the predicted image can be obtained. Exemplarily, the function for obtaining the first loss may be as shown in Equation 3:

L _total =L _rec +α*L _grad Formula 3

Among them, L _total represents the first loss, and α is a hyperparameter used to adjust the weight of the gradient loss.

Step 1004: Update the model parameters of the image processing model to be trained according to at least the first loss, until the model training conditions are met, and an image processing model is obtained.

For the image processing model obtained after training in step 1004, reference may be made to the description in the embodiment corresponding to FIG. 4, and details are not repeated here.

Optionally, in a possible implementation manner, the image processing model to be trained is further configured to process the third feature through a third network to obtain a semantic segmentation prediction result of the first image.

Optionally, in a possible implementation manner, the image processing model to be trained is further used to: obtain the real result of semantic segmentation of the first image; according to the prediction result of semantic segmentation and the real result of semantic segmentation , obtain a second loss, the second loss is used to describe the difference between the predicted result of the semantic segmentation and the real result of the semantic segmentation; The model parameters of the image processing model are updated until the model training conditions are met, and an image processing model is obtained.

That is, in the model training process, the semantic segmentation loss function may be used to control the semantic segmentation prediction result of the first image, so that the semantic segmentation result generated by the model can be more accurate.

Exemplarily, the semantic segmentation loss function used to obtain the second loss of the semantic segmentation prediction result and the semantic segmentation real result may be a cross-entropy loss function. For example, the semantic segmentation loss function can be shown in Equation 4:

s_i用于表示语义分割预测结果在像素i位置针对语义类别z的概率，

用于表示语义分割真实结果在像素i位置针对语义类别z的概率，log()用于表示求对数。where L _seg is the second loss, p is the number of pixels in the image,

s _i is used to represent the probability of the semantic segmentation prediction result for the semantic category z at the pixel i position,

It is used to represent the probability that the true result of semantic segmentation is for the semantic category z at the position of pixel i, and log() is used to represent the logarithm.

Based on the above-mentioned first loss and second loss, a third loss can be obtained, and the model parameters of the image processing model to be trained are updated according to the third loss until the model training conditions are satisfied, so as to obtain the image processing model.

Exemplarily, the formula for calculating the third loss can be shown in formula 5:

L _total =L _rec +α*L _seg +β*L _grad Formula 5

Among them, L _total is used to represent the third loss, α is the first hyperparameter, β is the second hyperparameter, and α and β are used to adjust the weight of semantic segmentation loss and gradient loss, respectively.

Optionally, in a possible implementation manner, the image processing model to be trained is further configured to: perform feature fusion processing on the first feature and the second feature to obtain the third feature; the Generating a fourth feature according to the third feature and the semantic segmentation result of the first image includes: performing feature fusion processing on the third feature and the semantic segmentation result of the first image to obtain the fourth feature. feature.

Optionally, in a possible implementation manner, the feature fusion processing includes at least one of summation processing, multiplication processing, concatenated processing and concatenated convolution processing.

Optionally, in a possible implementation manner, the image processing model to be trained is further configured to process the third feature to obtain a fifth feature; according to the third feature and the first image Semantic segmentation results, generate the fourth feature.

Optionally, in a possible implementation manner, the image processing model to be trained is further used to preprocess the first image to obtain preprocessing features; perform downsampling processing on the preprocessing features to obtain down-sampling features; processing the down-sampling features through the second network to obtain sixth features; performing up-sampling processing on the sixth features to obtain the second features of the first image.

Optionally, in a possible implementation manner, the image processing model is used to implement at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image dehazing, image deblurring, Image contrast enhancement, image demosaicing, image deraining, image color enhancement, image brightness enhancement, image detail enhancement, and image dynamic range enhancement.

Referring to FIG. 11 , FIG. 11 is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application. As shown in FIG. 11, an image processing apparatus provided by an embodiment of the present application includes: an acquisition unit 1101 and a processing unit 1102; the acquisition unit 1101 is used to acquire an image to be processed; the processing unit 1102 is used to A network processes the to-be-processed image to obtain a first feature, the first network is configured to extract at least features for image enhancement; the to-be-processed image is processed by a second network to obtain a second feature, the The second network is configured to extract at least a semantic segmentation feature; generate a third feature according to the first feature and the second feature; the acquiring unit 1101 is further configured to acquire the semantic segmentation result of the image to be processed ; The processing unit 1102 is further configured to generate a fourth feature according to the third feature and the semantic segmentation result of the image to be processed; perform image reconstruction on the fourth feature to obtain a target image.

Optionally, in a possible implementation manner, the processing unit 1102 is further configured to process the third feature through a third network to obtain a semantic segmentation result of the image to be processed.

Optionally, in a possible implementation manner, the processing unit 1102 is further configured to perform feature fusion processing on the first feature and the second feature to obtain the third feature; The three features and the semantic segmentation result of the image to be processed are subjected to feature fusion processing to obtain the fourth feature.

Optionally, in a possible implementation manner, the feature fusion processing includes at least one of summation processing, multiplication processing, concatenated processing and concatenated convolution processing.

Optionally, in a possible implementation manner, the processing unit 1102 is further configured to process the third feature to obtain a fifth feature; according to the fifth feature and the semantics of the image to be processed Segment the result to generate a fourth feature.

Optionally, in a possible implementation manner, the processing unit 1102 is further configured to perform preprocessing on the to-be-processed image to obtain preprocessing features; perform downsampling processing on the preprocessing features to obtain the following: Sampling features; processing the down-sampling features through the second network to obtain sixth features; performing up-sampling processing on the sixth features to obtain the second features of the to-be-processed image.

Optionally, in a possible implementation manner, the image processing device is configured to implement at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image dehazing, image deblurring, Image contrast enhancement, image demosaicing, image deraining, image color enhancement, image brightness enhancement, image detail enhancement, and image dynamic range enhancement.

Referring to FIG. 12, FIG. 12 is a schematic structural diagram of a model training apparatus provided by an embodiment of the present application. As shown in FIG. 12 , a model training apparatus provided by an embodiment of the present application includes: an acquisition unit 1201 and a training unit 1202 ; the acquisition unit 1201 is configured to acquire a training sample pair, and the training sample pair includes a first image and a second image, the quality of the first image is lower than that of the second image; the prediction unit is configured to process the first image through the image processing model to be trained to obtain a predicted image, wherein the to-be-trained image processing model The image processing model is used to obtain the image to be processed; the first image is processed through a first network to obtain first features, and the first network is configured to extract at least features for image enhancement; The first image is processed to obtain a second feature, and the second network is configured to extract at least semantic segmentation features; generate a third feature according to the first feature and the second feature; obtain the first image Semantic segmentation result; generate a fourth feature according to the third feature and the semantic segmentation result of the first image; perform image reconstruction on the fourth feature to obtain a predicted image; Two images and the predicted image, obtain a first loss, where the first loss is used to describe the difference between the second image and the predicted image; process the to-be-trained image at least according to the first loss The model parameters of the model are updated until the model training conditions are met, and an image processing model is obtained.

Optionally, in a possible implementation manner, the training unit 1202 is further configured to process the third feature through a third network to obtain a semantic segmentation prediction result of the first image.

Optionally, in a possible implementation manner, the training unit 1202 is further configured to obtain the real result of semantic segmentation of the first image; Two losses, the second loss is used to describe the difference between the predicted result of semantic segmentation and the real result of semantic segmentation; the image processing model to be trained is processed at least according to the first loss and the second loss The model parameters are updated until the model training conditions are met, and the image processing model is obtained.

Optionally, in a possible implementation manner, the training unit 1202 is further configured to perform feature fusion processing on the first feature and the second feature to obtain the third feature; The feature and the semantic segmentation result of the first image are subjected to feature fusion processing to obtain the fourth feature.

Optionally, in a possible implementation manner, the feature fusion processing includes at least one of summation processing, multiplication processing, concatenated processing and concatenated convolution processing.

Optionally, in a possible implementation manner, the training unit 1202 is further configured to process the third feature to obtain a fifth feature; according to the third feature and the semantic segmentation of the first image As a result, a fourth feature is generated.

Optionally, in a possible implementation manner, the training unit 1202 is further configured to preprocess the first image to obtain preprocessing features; perform downsampling processing on the preprocessing features to obtain downsampling feature; processing the down-sampling feature through the second network to obtain a sixth feature; performing up-sampling processing on the sixth feature to obtain the second feature of the first image.

Optionally, in a possible implementation manner, the image processing model is used to implement at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image dehazing, image deblurring, Image contrast enhancement, image demosaicing, image deraining, image color enhancement, image brightness enhancement, image detail enhancement, and image dynamic range enhancement.

Next, an execution device provided by an embodiment of the present application is introduced. Please refer to FIG. 13 , which is a schematic structural diagram of the execution device provided by the embodiment of the present application. Smart wearable devices, servers, etc., are not limited here. The data processing apparatus described in the embodiment corresponding to FIG. 13 may be deployed on the execution device 1300 to implement the function of data processing in the embodiment corresponding to FIG. 13 . Specifically, the execution device 1300 includes: a receiver 1301, a transmitter 1302, a processor 1303, and a memory 1304 (wherein the number of processors 1303 in the execution device 1300 may be one or more, and one processor is taken as an example in FIG. 13 ) , wherein the processor 1303 may include an application processor 13031 and a communication processor 13032 . In some embodiments of the present application, the receiver 1301, the transmitter 1302, the processor 1303, and the memory 1304 may be connected by a bus or otherwise.

Memory 1304 may include read-only memory and random access memory, and provides instructions and data to processor 1303 . A portion of memory 1304 may also include non-volatile random access memory (NVRAM). The memory 1304 stores processors and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions may include various operating instructions for implementing various operations.

The processor 1303 controls the operation of the execution device. In a specific application, various components of the execution device are coupled together through a bus system, where the bus system may include a power bus, a control bus, a status signal bus, and the like in addition to a data bus. However, for the sake of clarity, the various buses are referred to as bus systems in the figures.

The methods disclosed in the above embodiments of the present application may be applied to the processor 1303 or implemented by the processor 1303 . The processor 1303 may be an integrated circuit chip, which has signal processing capability. In the implementation process, each step of the above-mentioned method can be completed by an integrated logic circuit of hardware in the processor 1303 or an instruction in the form of software. The above-mentioned processor 1303 may be a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and may further include an application specific integrated circuit (ASIC), a field programmable gate Field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 1303 may implement or execute the methods, steps, and logical block diagrams disclosed in the embodiments of this application. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory 1304, and the processor 1303 reads the information in the memory 1304, and completes the steps of the above method in combination with its hardware.

The receiver 1301 can be used to receive input numerical or character information, and to generate signal input related to performing the relevant setting and function control of the device. The transmitter 1302 can be used to output digital or character information through the first interface; the transmitter 1302 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1302 can also include a display device such as a display screen .

In the embodiment of the present application, in one case, the processor 1303 is configured to execute the image processing method executed by the execution device in the embodiment corresponding to FIG. 4 .

This embodiment of the present application also provides a training device. Please refer to FIG. 14 . FIG. 14 is a schematic structural diagram of the training device provided by the embodiment of the present application. Specifically, the training device 1400 is implemented by one or more servers. The training device 1400 Large differences may occur due to differences in configuration or performance, and may include one or more central processing units (CPUs) 1414 (eg, one or more processors) and memory 1432, one or more storage applications 1442 or storage medium 1430 for data 1444 (eg, one or more mass storage devices). Among them, the memory 1432 and the storage medium 1430 may be short-term storage or persistent storage. The program stored in the storage medium 1430 may include one or more modules (not shown in the figure), and each module may include a series of instructions to operate on the training device. Further, the central processing unit 1414 may be configured to communicate with the storage medium 1430 to execute a series of instruction operations in the storage medium 1430 on the training device 1400 .

The training device 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458; or, one or more operating systems 1441, such as Windows Server ^™ , Mac OS X ^TM , Unix ^TM , Linux ^TM , FreeBSD ^TM and many more.

Specifically, the training device may perform the steps in the embodiment corresponding to FIG. 10 .

Embodiments of the present application also provide a computer program product that, when running on a computer, causes the computer to perform the steps performed by the aforementioned execution device, or causes the computer to perform the steps performed by the aforementioned training device.

Embodiments of the present application further provide a computer-readable storage medium, where a program for performing signal processing is stored in the computer-readable storage medium, and when it runs on a computer, the computer executes the steps performed by the aforementioned execution device. , or, causing the computer to perform the steps as performed by the aforementioned training device.

The execution device, training device, or terminal device provided in this embodiment of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, the processing unit may be, for example, a processor, and the communication unit may be, for example, an input/output interface, pins or circuits, etc. The processing unit can execute the computer executable instructions stored in the storage unit, so that the chip in the execution device executes the data processing method described in the above embodiments, or the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit may also be a storage unit located outside the chip in the wireless access device, such as only Read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM), and the like.

Specifically, please refer to FIG. 15. FIG. 15 is a schematic structural diagram of a chip provided by an embodiment of the present application. The chip may be represented as a neural network processor NPU 1500, and the NPU 1500 is mounted on the main CPU (Host CPU) as a co-processor. CPU), tasks are allocated by the Host CPU. The core part of the NPU is the arithmetic circuit 1503, which is controlled by the controller 1504 to extract the matrix data in the memory and perform multiplication operations.

In some implementations, the arithmetic circuit 1503 includes multiple processing units (Process Engine, PE). In some implementations, the arithmetic circuit 1503 is a two-dimensional systolic array. The arithmetic circuit 1503 may also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 1503 is a general-purpose matrix processor.

For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit fetches the data corresponding to the matrix B from the weight memory 1502 and buffers it on each PE in the arithmetic circuit. The arithmetic circuit fetches the data of matrix A and matrix B from the input memory 1501 to perform matrix operation, and stores the partial result or final result of the matrix in the accumulator 1508 .

Unified memory 1506 is used to store input data and output data. The weight data is directly passed through a storage unit access controller (Direct Memory Access Controller, DMAC) 1505 , and the DMAC is transferred to the weight memory 1502 . Input data is also moved into unified memory 1506 via the DMAC.

The BIU is the Bus Interface Unit, that is, the bus interface unit 1510 , which is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (Instruction Fetch Buffer, IFB) 1509 .

The bus interface unit 1510 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 1509 to obtain instructions from the external memory, and also for the storage unit access controller 1505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

The DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 1506 or the weight data to the weight memory 1502 or the input data to the input memory 1501 .

The vector calculation unit 1507 includes a plurality of operation processing units, and further processes the output of the operation circuit 1503, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc., if necessary. It is mainly used for non-convolutional/fully connected layer network computation in neural networks, such as Batch Normalization, pixel-level summation, and upsampling of feature planes.

In some implementations, the vector computation unit 1507 can store the vector of processed outputs to the unified memory 1506 . For example, the vector calculation unit 1507 may apply a linear function; or a non-linear function to the output of the operation circuit 1503, such as linear interpolation of the feature plane extracted by the convolution layer, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 1507 generates normalized values, pixel-level summed values, or both. In some implementations, the vector of processed outputs can be used as activation input to the arithmetic circuit 1503, such as for use in subsequent layers in a neural network.

an instruction fetch buffer 1509 connected to the controller 1504 for storing instructions used by the controller 1504;

The unified memory 1506, the input memory 1501, the weight memory 1502 and the instruction fetch memory 1509 are all On-Chip memories. External memory is private to the NPU hardware architecture.

Wherein, the processor mentioned in any one of the above may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

In addition, it should be noted that the device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be A physical unit, which can be located in one place or distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. In addition, in the drawings of the device embodiments provided in the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

From the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware. Special components, etc. to achieve. Under normal circumstances, all functions completed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or special circuit, etc. However, a software program implementation is a better implementation in many cases for this application. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that make contributions to the prior art. The computer software products are stored in a readable storage medium, such as a floppy disk of a computer. , U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including several instructions to enable a computer device (which may be a personal computer, training device, or network device, etc.) to execute the various embodiments of the application. method.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product.

The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be retrieved from a website, computer, training device, or data Transmission from the center to another website site, computer, training facility or data center via wired (eg coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be stored by a computer, or a data storage device such as a training device, a data center, or the like that includes an integration of one or more available media. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), and the like.

Claims (17)

1. An image processing method, comprising:

acquiring an image to be processed;

processing the image to be processed through a first network to obtain a first feature, wherein the first network is configured to extract at least a feature for image enhancement;

processing the image to be processed through a second network to obtain a second feature, wherein the second network is configured to extract at least a semantic segmentation feature;

generating a third feature according to the first feature and the second feature;

obtaining a semantic segmentation result of the image to be processed;

generating a fourth feature according to the third feature and the semantic segmentation result of the image to be processed;

and carrying out image reconstruction on the fourth characteristic to obtain a target image.

2. The image processing method according to claim 1, wherein the obtaining of the semantic segmentation result of the image to be processed comprises:

and processing the third features through a third network to obtain a semantic segmentation result of the image to be processed.

3. The image processing method according to claim 1 or 2, wherein the generating a third feature from the first feature and the second feature comprises:

performing feature fusion processing on the first feature and the second feature to obtain a third feature;

generating a fourth feature according to the third feature and the semantic segmentation result of the image to be processed, wherein the fourth feature comprises:

and performing feature fusion processing on the third feature and the semantic segmentation result of the image to be processed to obtain the fourth feature.

4. The image processing method according to claim 3, wherein the feature fusion process includes at least one of a summation process, a multiplication process, a concatenation process, and a concatenation convolution process.

5. The image processing method according to any one of claims 1 to 4, wherein before generating a fourth feature according to the third feature and a semantic segmentation result of the image to be processed, the method further comprises:

processing the third characteristic to obtain a fifth characteristic;

generating a fourth feature according to the third feature and the semantic segmentation result of the image to be processed, wherein the fourth feature comprises: and generating a fourth feature according to the fifth feature and the semantic segmentation result of the image to be processed.

6. The image processing method according to any one of claims 1 to 5, wherein the processing the image to be processed through the second network to obtain a second feature comprises:

preprocessing the image to be processed to obtain preprocessing characteristics;

performing down-sampling processing on the preprocessing characteristic to obtain a down-sampling characteristic;

processing the downsampled features through the second network to obtain sixth features;

and performing upsampling processing on the sixth feature to obtain a second feature of the image to be processed.

7. The image processing method according to any of claims 1 to 6, wherein the method is used to implement at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image defogging, image deblurring, image contrast enhancement, image demosaicing, image raining, image color enhancement, image brightness enhancement, image detail enhancement and image dynamic range enhancement.

8. A method of model training, comprising:

acquiring a training sample pair, wherein the training sample pair comprises a first image and a second image, and the quality of the first image is lower than that of the second image;

processing the first image through an image processing model to be trained to obtain a predicted image, wherein the image processing model to be trained is used for obtaining an image to be processed; processing the first image through a first network resulting in first features, the first network configured to extract at least features for image enhancement; processing the first image through a second network to obtain a second feature, wherein the second network is configured to extract at least a semantic segmentation feature; generating a third feature according to the first feature and the second feature; obtaining a semantic segmentation result of the first image; generating a fourth feature according to the third feature and the semantic segmentation result of the first image; carrying out image reconstruction on the fourth feature to obtain a predicted image;

obtaining a first loss according to a second image in the training sample pair and the predicted image, wherein the first loss is used for describing the difference between the second image and the predicted image;

and updating the model parameters of the image processing model to be trained at least according to the first loss until model training conditions are met, so as to obtain the image processing model.

9. The model training method of claim 8, wherein the to-be-trained image processing model is further configured to process the third feature through a third network to obtain a semantic segmentation prediction result of the first image.

10. The model training method of claim 9, wherein the image processing model to be trained is further configured to:

obtaining a semantic segmentation real result of the first image;

obtaining a second loss according to the semantic segmentation prediction result and the semantic segmentation real result, wherein the second loss is used for describing the difference between the semantic segmentation prediction result and the semantic segmentation real result;

and updating the model parameters of the image processing model to be trained at least according to the first loss and the second loss until model training conditions are met, so as to obtain the image processing model.

11. The model training method according to any one of claims 8 to 10, wherein the image processing model to be trained is further configured to:

performing feature fusion processing on the first feature and the second feature to obtain a third feature;

and performing feature fusion processing on the third feature and the semantic segmentation result of the image to be processed to obtain the fourth feature.

12. The model training method of claim 11, wherein the feature fusion process comprises at least one of a summation process, a multiplication process, a concatenation process, and a concatenation convolution process.

13. The model training method according to any one of claims 8 to 12, wherein the to-be-trained image processing model is further configured to process the third feature to obtain a fifth feature; and generating a fourth feature according to the third feature and the semantic segmentation result of the first image.

14. The model training method according to any one of claims 8 to 13, wherein the image processing model to be trained is further configured to preprocess the first image to obtain a preprocessing feature; performing down-sampling processing on the preprocessing characteristic to obtain a down-sampling characteristic; processing the downsampled features through the second network to obtain sixth features; and performing upsampling processing on the sixth feature to obtain a second feature of the first image.

15. The model training method of any one of claims 8 to 14, wherein the image processing model is configured to perform at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image defogging, image deblurring, image contrast enhancement, image demosaicing, image raining, image color enhancement, image brightness enhancement, image detail enhancement and image dynamic range enhancement.

16. An image processing apparatus, characterized in that the apparatus comprises a memory and a processor; the memory stores code, the processor is configured to execute the code, and when executed, the image processing apparatus performs the method of any of claims 1 to 15.

17. A computer storage medium, characterized in that the computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to implement the method of any of claims 1 to 15.

Images